You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 271017a
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/Tips/Tips_for_parallel_execution.md
-36Lines changed: 0 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -87,42 +87,6 @@ Median aggregation and `return_by_separate_ensembles=True` retain the full matri
87
87
88
88
For estimators with read-only prediction, `fit(..., persist_prediction_state=False)` lets lazy models be loaded and released without rewriting their files. The default is True, preserving custom estimators whose prediction-time mutations must persist. This choice is independent of streaming aggregation. The standalone `LazyLoadingEstimator` offers the corresponding `dump_on_read` option and also defaults to True. Performance comparisons explicitly disable repeated writes for the tested forests/XGBoost models.
89
89
90
-
### Bound predictor batches within temporal queries
91
-
92
-
Streaming ensemble aggregation bounds the parent's reduction arrays. It does not
93
-
bound a worker's temporal-window covariates: a wide prediction window is otherwise
94
-
materialized once in each concurrent worker. For pointwise estimators such as
95
-
XGBoost, Random Forests and their local beta wrappers, opt into bounded batches:
96
-
97
-
```python
98
-
pred = model.predict_proba(prediction_path, n_jobs=30,
99
-
temporal_window_prequery=True, streaming=True,
100
-
prediction_batch_size=100_000)
101
-
```
102
-
103
-
Each worker still issues one query per temporal window. DuckDB yields Arrow
104
-
record batches and only the current batch becomes a Pandas covariate table;
105
-
stixel geometry, probabilities, overlapping-window means and coverage rules are
106
-
retained. Pandas and Parquet inputs are also supported. No prediction copy of the
107
-
full covariate dataset is saved to scratch. The option is independent of the
108
-
training backend, lazy loading and total number of ensembles.
109
-
110
-
The row limit is per worker, not a total RAM cap. DuckDB caches, fitted models,
111
-
ensemble results, indexes and parent output arrays still consume memory. Smaller
112
-
batches reduce input memory but can increase model loads and prediction calls;
113
-
compare 25,000, 50,000 and 100,000 on the target workload. Keep `None` (the default)
114
-
for custom batch-dependent or prediction-stateful estimators, because splitting
115
-
their prediction calls can change their behavior. Custom ensemble prediction
116
-
overrides, including SphereAdaSTEM, reject this option explicitly rather than
117
-
silently ignoring it. Median and separate-ensemble returns still require their
118
-
full result matrix even when input batches are bounded.
119
-
120
-
An isolated, repeatable comparison is available with
Copy file name to clipboardExpand all lines: resources/PIPELINE_OPTIMIZATION.md
-66Lines changed: 0 additions & 66 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,72 +4,6 @@ This second pass on `codex-rewrite` adds streaming ensemble means and standard d
4
4
5
5
The measured gains depend on the workload. In three repeated lazy-forest runs, prediction was about 31% faster and complete-job time about 10% lower than the first rewrite. Small XGBoost jobs changed little. Streaming greatly reduces the aggregation buffer, but does not bound estimator workspace or guarantee a comparable reduction in complete-job peak memory.
6
6
7
-
## Bounded prediction inputs, October 10, 2026
8
-
9
-
The HPC occurrence log showed successful training and held-out prediction, followed
10
-
by SIGKILL during annual prediction before any ensemble finished. That is consistent
11
-
with a memory-limit kill, although Slurm accounting is needed to confirm its cause.
12
-
The log's repeated 192,909.8 MiB peak belongs to the entire workflow; it is not a
13
-
prediction-stage peak. Summed process RSS also counts shared pages repeatedly.
14
-
15
-
The previous streaming reducer already avoided an N-by-ensemble matrix in the
16
-
parent. Each worker still materialized a full, wide temporal window as Pandas data.
17
-
The new opt-in `prediction_batch_size` bounds that conversion while retaining
18
-
`temporal_window_prequery=True`: one query per window yields successive Arrow
19
-
batches, each passed through the same rotation, half-open stixels and base models.
20
-
Workers remain parallel. Missing models are skipped before constructing their
21
-
stixel covariate copies. No full prediction-input copy is written to scratch.
22
-
DuckDB documents the [Arrow batch reader](https://duckdb.org/docs/lts/guides/python/export_arrow);
23
-
the separate parent-side reducer follows the same incremental-output principle
24
-
described in [joblib's generator example](https://joblib.readthedocs.io/en/stable/auto_examples/parallel_generator.html).
25
-
26
-
A synthetic wide-input comparison used 1,000,000 prediction rows, 116 random
27
-
covariates plus longitude, latitude and DOY, four ensembles, two loky workers,
28
-
lazy models and local ridge-100 beta calibration around XGBoost. Models had
29
-
10 trees per calibration fold and were fitted once on 10,000 rows; identical
30
-
model files were reused for every prediction variant. This measures execution
31
-
equivalence and prediction costs, not whether the calibration is scientifically
32
-
optimal. Each variant ran three times in fresh processes, alternating order.
33
-
34
-
| Temporal-query batch | Median prediction time | Median peak process-tree physical footprint | Median peak summed RSS |
35
-
| --- | ---: | ---: | ---: |
36
-
| Full window | 7.73 s | 3.29 GiB | 5.76 GiB |
37
-
| 100,000 rows | 9.03 s | 1.85 GiB | 4.51 GiB |
38
-
| 50,000 rows | 10.35 s | 1.69 GiB | 3.99 GiB |
39
-
40
-
The 100,000-row choice reduced the median sampled footprint by **43.8%** at a
41
-
**16.8%** time cost. The 50,000-row choice saved 48.8% but took 33.9% longer.
42
-
All probabilities and standard deviations agreed within 5.56e-17; missing masks
43
-
and coverage were identical. These are Apple Silicon measurements (Python
0 commit comments