[cosmicbboy]

The round-13 1M run (upnz22h5cdt6gdgrtshv) succeeded at 31,704 rows of a requested 1,000,000 — 78 surviving archetypes × 400 variants, plus 504 measured calibration rows. It reported success because require_target=False turns shipping short into a warning.

Four mechanisms are fixed here, all of them post-mortem findings.

1. The wave loop could not stop itself

Wave 1 ran 19.5h against a 15h generation deadline, because the deadline was only tested after asyncio.gather returned. The run generated once, spent its whole budget in a single un-adapted wave, and never reached wave 2 — the adaptive sizing that was supposed to respond to the observed keep rate never executed at all.

The deadline is now checked inside build_archetype, wave sizing is bounded by the time actually left (plan_wave, extracted as a pure function and unit tested), and wave 1 is deliberately small so there is a keep-rate observation to size wave 2 with.

2. Work kept going to dead teachers

277 archetypes died on HTTP 504 after exhausting all three retries — and the histogram is the tell: 291 first retries, 290 second, 288 third. Essentially every retry failed too, because the endpoints were persistently unavailable rather than hiccuping. Each burned 5+10+20s of backoff first: ~2.8 hours of pure sleeping against the concurrency limit for zero recovered work.

TeacherPool now round-robins over healthy endpoints only and trips a breaker after N consecutive transport failures (content rejections don't count — that's a prompt problem, and dropping the endpoint wouldn't fix it), never the last one standing, with a half-open probe each wave since these apps scale from zero and do come back. Per-call retries drop from 3/5s to 2/2s.

3. Broken code cost 7 pods to discover

The largest rejection bucket was curated out: execution failed — 462 pods of code that passed the AST screen, imported fine, and raised at runtime.

A viability probe now runs the cheapest calibration point first and stops there if the code cannot execute. It costs nothing when it passes, because that point is a measurement we needed anyway. It fires only on execution/pod failure, never on a point merely landing out of bounds — that's a property of the parameter point, not the code, and rejecting there would throw away archetypes the probe was never meant to judge.

4. A dozen new libraries had one line of guidance each

Round 13 admitted polars/duckdb/dask/jax/lightning/faiss/langgraph/llama-index with a sentence apiece, which names a library but pins none of the call signatures that decide whether a module runs. STACK_PITFALLS adds a narrow API contract per stack — polars group_by, lightgbm's removed verbose_eval, networkx from_numpy_array, faiss float32 contiguity, langgraph recursion limits, lightning's writes-to-cwd defaults, transformers hidden_size % num_heads — and the prompt now says outright that the module is executed at its range lows first and discarded if it raises there.

Sizing follows from the measured unit cost

upnz22h5 ran ~1,550 oracle pods in 19.5h at concurrency 24: ~17 minutes of wall clock per pod, of which only ~1–2 min is the workload. The rest is node scale-up and pulling the (much fatter) round-13 harness image. That cost isn't going away, so the levers are width and pod count:

• oracle_concurrency 24 → 160 (pods are one per t3a.xlarge on a 3–250 node pool — 24 was using a tenth of the available fan-out)
• variants_per_archetype 400 → 1200 (~834 archetypes for 1M: 10× what round 13 shipped, and well inside both gates that bound this axis — near-duplication is measured over archetype code, so variants don't affect it, and concentration admits 2% per archetype = 20,000 rows at 1M)
• 150s peak-hold dropped from the rotation (paid by every calibration pod)
• generation deadline 16h → 20h, task runtime 24h → 36h, so a release that generates to its deadline still has time to ship

Also fixes an int default on a float task parameter that was aborting flyte deploy partway through the task list.

Tests

23 new tests over the wave planner, the circuit breaker, the probe classifier and the stack contracts (tests/test_wave_throughput.py); 209 pass overall.

Research log

Adds research_log/2026-09-09-round-12-14-corpus-at-scale.md covering rounds 12–14 — the quality-gate work (including two gate bugs that failed a perfectly good corpus, recovered via flyte fork), the stack axis and teacher tiers, and the post-mortem above.

Run

uqjsqswf4fntdjrq9qzq is the first 1M attempt on this code, launched 2026-09-09 and running now. The ~14–16h projection is an estimate from one prior run's unit cost, not a measurement.

🤖 Generated with Claude Code

https://claude.ai/code/session_016huuSPKsdzj5YwzV846E8e

[cosmicbboy]

Relaunched as us86v7zcfphfz76gjrdw after aborting the first attempt.

The first run (uqjsqswf4fntdjrq9qzq) started with only qwen38-27b: the three frontier teachers each needed GPU nodes and were still scaling when the 300s grace expired. All four were ACTIVE ~30 min later — by which point the release was locked to a single teacher for its whole 20-hour life, throwing away the multi-family diversity that is the point of running them.

Grace raised 300s → 1800s (commit 8027b49). That 300s was tuned when the pool was two similar models, and waiting longer is much safer now than it was: a teacher that wakes and then goes bad gets dropped mid-run by the new circuit breaker rather than absorbing its share of archetypes forever.