Xgboost Dask

[Kim-Jongchan]

Is XGBoost with Dask truly scalable in practice — especially regarding dataset size relative to RAM capacity?

I’m testing distributed training with xgboost.dask and noticed that during DaskDMatrix initialization, it appears to load the entire dataset into memory across the workers. This seems to limit scalability, as even with a Dask cluster, the dataset size cannot exceed total cluster RAM by much.

My understanding was that Dask+XGBoost would allow “out-of-core” or streaming-like training behavior, but it looks like the data still needs to be materialized in memory first.

So, a few sub-questions:

1. Does DaskDMatrix or DaskQuantileDMatrix actually load all data into memory on each worker before training starts?

2. Is there any truly out-of-core training mode supported (similar to XGBoost’s single-machine external memory mode)?

3. In practical terms, how large can a dataset be (relative to total RAM across workers) before Dask-XGBoost starts failing due to memory errors?

4. Are there known best practices to train on datasets larger than memory — e.g., chunked training, Arrow-based streaming, or saving intermediate histograms?

[trivialfis]

Yes, it needs to load the dataset into the main memory. It's horizontal scaling only (using more workers).

External memory supports distributed training, just not through dask, please find the demo in the document.