Skip to content

Commit 271017a

Browse files
committed
Revert "Bound temporal prediction batches while retaining parallel execution"
This reverts commit 1bcbd5d.
1 parent 1bcbd5d commit 271017a

10 files changed

Lines changed: 22 additions & 1330 deletions

File tree

‎data/00_benchmarks/prediction_batches/calibrated.json‎

Lines changed: 0 additions & 179 deletions
This file was deleted.

‎docs/Tips/Tips_for_parallel_execution.md‎

Lines changed: 0 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -87,42 +87,6 @@ Median aggregation and `return_by_separate_ensembles=True` retain the full matri
8787

8888
For estimators with read-only prediction, `fit(..., persist_prediction_state=False)` lets lazy models be loaded and released without rewriting their files. The default is True, preserving custom estimators whose prediction-time mutations must persist. This choice is independent of streaming aggregation. The standalone `LazyLoadingEstimator` offers the corresponding `dump_on_read` option and also defaults to True. Performance comparisons explicitly disable repeated writes for the tested forests/XGBoost models.
8989

90-
### Bound predictor batches within temporal queries
91-
92-
Streaming ensemble aggregation bounds the parent's reduction arrays. It does not
93-
bound a worker's temporal-window covariates: a wide prediction window is otherwise
94-
materialized once in each concurrent worker. For pointwise estimators such as
95-
XGBoost, Random Forests and their local beta wrappers, opt into bounded batches:
96-
97-
```python
98-
pred = model.predict_proba(prediction_path, n_jobs=30,
99-
temporal_window_prequery=True, streaming=True,
100-
prediction_batch_size=100_000)
101-
```
102-
103-
Each worker still issues one query per temporal window. DuckDB yields Arrow
104-
record batches and only the current batch becomes a Pandas covariate table;
105-
stixel geometry, probabilities, overlapping-window means and coverage rules are
106-
retained. Pandas and Parquet inputs are also supported. No prediction copy of the
107-
full covariate dataset is saved to scratch. The option is independent of the
108-
training backend, lazy loading and total number of ensembles.
109-
110-
The row limit is per worker, not a total RAM cap. DuckDB caches, fitted models,
111-
ensemble results, indexes and parent output arrays still consume memory. Smaller
112-
batches reduce input memory but can increase model loads and prediction calls;
113-
compare 25,000, 50,000 and 100,000 on the target workload. Keep `None` (the default)
114-
for custom batch-dependent or prediction-stateful estimators, because splitting
115-
their prediction calls can change their behavior. Custom ensemble prediction
116-
overrides, including SphereAdaSTEM, reject this option explicitly rather than
117-
silently ignoring it. Median and separate-ensemble returns still require their
118-
full result matrix even when input batches are bounded.
119-
120-
An isolated, repeatable comparison is available with
121-
`python scripts/00_benchmarks/compare_prediction_batches.py --calibrated`.
122-
It checks predictions, standard deviations and missing-value masks against the
123-
whole-window route, reports sampled prediction memory and time, and removes its
124-
temporary databases, model files and prediction arrays afterward.
125-
12690
### Monitor a running workflow
12791

12892
Install the optional monitoring dependency in the environment that runs the model:

‎resources/PIPELINE_OPTIMIZATION.md‎

Lines changed: 0 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -4,72 +4,6 @@ This second pass on `codex-rewrite` adds streaming ensemble means and standard d
44

55
The measured gains depend on the workload. In three repeated lazy-forest runs, prediction was about 31% faster and complete-job time about 10% lower than the first rewrite. Small XGBoost jobs changed little. Streaming greatly reduces the aggregation buffer, but does not bound estimator workspace or guarantee a comparable reduction in complete-job peak memory.
66

7-
## Bounded prediction inputs, October 10, 2026
8-
9-
The HPC occurrence log showed successful training and held-out prediction, followed
10-
by SIGKILL during annual prediction before any ensemble finished. That is consistent
11-
with a memory-limit kill, although Slurm accounting is needed to confirm its cause.
12-
The log's repeated 192,909.8 MiB peak belongs to the entire workflow; it is not a
13-
prediction-stage peak. Summed process RSS also counts shared pages repeatedly.
14-
15-
The previous streaming reducer already avoided an N-by-ensemble matrix in the
16-
parent. Each worker still materialized a full, wide temporal window as Pandas data.
17-
The new opt-in `prediction_batch_size` bounds that conversion while retaining
18-
`temporal_window_prequery=True`: one query per window yields successive Arrow
19-
batches, each passed through the same rotation, half-open stixels and base models.
20-
Workers remain parallel. Missing models are skipped before constructing their
21-
stixel covariate copies. No full prediction-input copy is written to scratch.
22-
DuckDB documents the [Arrow batch reader](https://duckdb.org/docs/lts/guides/python/export_arrow);
23-
the separate parent-side reducer follows the same incremental-output principle
24-
described in [joblib's generator example](https://joblib.readthedocs.io/en/stable/auto_examples/parallel_generator.html).
25-
26-
A synthetic wide-input comparison used 1,000,000 prediction rows, 116 random
27-
covariates plus longitude, latitude and DOY, four ensembles, two loky workers,
28-
lazy models and local ridge-100 beta calibration around XGBoost. Models had
29-
10 trees per calibration fold and were fitted once on 10,000 rows; identical
30-
model files were reused for every prediction variant. This measures execution
31-
equivalence and prediction costs, not whether the calibration is scientifically
32-
optimal. Each variant ran three times in fresh processes, alternating order.
33-
34-
| Temporal-query batch | Median prediction time | Median peak process-tree physical footprint | Median peak summed RSS |
35-
| --- | ---: | ---: | ---: |
36-
| Full window | 7.73 s | 3.29 GiB | 5.76 GiB |
37-
| 100,000 rows | 9.03 s | 1.85 GiB | 4.51 GiB |
38-
| 50,000 rows | 10.35 s | 1.69 GiB | 3.99 GiB |
39-
40-
The 100,000-row choice reduced the median sampled footprint by **43.8%** at a
41-
**16.8%** time cost. The 50,000-row choice saved 48.8% but took 33.9% longer.
42-
All probabilities and standard deviations agreed within 5.56e-17; missing masks
43-
and coverage were identical. These are Apple Silicon measurements (Python
44-
3.13.15, NumPy 2.5.1, Pandas 3.0.5, DuckDB 1.5.5, scikit-learn 1.9.0, XGBoost
45-
3.3.0), sampled every 50 ms. Mac physical footprint and Linux PSS/Slurm charges
46-
are different measurements. Shared filesystem model loading and larger stixels
47-
can change the speed trade-off on HPC, so these percentages are not a server
48-
memory guarantee. Model workspace, DuckDB caches and ensemble output buffers
49-
remain additional allocations; the row limit is not a total job RAM cap.
50-
51-
The package defaults to `prediction_batch_size=None` for compatibility with
52-
custom batch-dependent and stateful prediction methods. The project occurrence
53-
runner defaults to 100,000 for its pointwise XGBoost/beta models; `--prediction-batch-size 0`
54-
restores whole windows. Smaller batches can repeat lazy-model loads, which
55-
explains part of the time cost. Custom ensemble overrides reject batching
56-
explicitly; separate-ensemble/median returns retain their full result matrix.
57-
58-
Raw results: [calibrated.json](../data/00_benchmarks/prediction_batches/calibrated.json).
59-
The comparison removes temporary databases, models and prediction arrays.
60-
Validation passed 76 checks on Python 3.13 covering batching, streaming, calibration
61-
and both project runners, plus six focused Python 3.14 checks. The batch reader
62-
tests cover the older DuckDB API fallback, early reader cleanup, one query per
63-
window, private estimator row workspaces, nonconsecutive/reversed observation IDs,
64-
overlapping windows, intentional dummy predictions and matching missing coverage.
65-
66-
```bash
67-
python scripts/00_benchmarks/compare_prediction_batches.py \
68-
--rows 1000000 --features 116 --ensembles 4 --jobs 2 \
69-
--calibrated --batches 0 50000 100000 --repeat 3 \
70-
--output data/00_benchmarks/prediction_batches/calibrated.json
71-
```
72-
737
## Public choices
748

759
Continue passing `base_model` to `AdaSTEMClassifier` or `AdaSTEMRegressor`. Tuning and calibration are independent:

0 commit comments

Comments
 (0)