[cosmicbboy]

Round 15 asks what a full fine-tune does with a real training budget on a real corpus. Round 11's r11-fullft-4b beat LoRA on fit (79% vs 53%) on 300 steps; this arm gets 26,000 steps on the 1,016,896-row round-15 corpus — the 1M target round 12–14 left open is met.

The training run (udrtv5nj7dcth86qdfn6) is in flight. Nothing about full FT vs LoRA is claimed in this PR — what it ships is the arm plus the three fixes each dead launch exposed.

The arm

AMBITIOUS_FULLFT_4B_1D — Qwen3-4B full FT, 26,000 steps, group 4, prompt 1,536, batch 2 × accum 2, one L40S, 48Gi. Every number is derived from a prior run's measurement; the arithmetic and both OOM calibration points are recorded in the profile comment.

The three fixes

Fix	What broke
_sample_train_records replaces the whole-corpus pd.read_parquet	usnnr9xmpnnq4ftjqgzn OOMKilled 4× on host RAM before a step ran
gradient_accumulation_steps added to TunerProfile, wired into GRPOConfig (which never set it)	the knob that keeps group size + prompt budget while halving the logits tensor
vram_estimate_gib/preflight_vram now model full-FT grads + 8-bit Adam state	umnzhngvwbf9d6fbhwfz sailed through preflight and CUDA-OOMed in the step, 4×

• Bounded corpus read. Pass 1 reads only the split column to choose k rows with a fixed seed; pass 2 re-reads in row-group batches keeping just those. Memory scales with train_contexts, not corpus size. harness_code — eval's executable twin, never opened by training — is not read at all. Materializing 10⁶ rows to train on 26,000 was always waste: GRPO consumes one prompt per step.
• Preflight. Gradients (bf16) and the 8-bit Adam moments allocate on the first optimizer step, long after the check passes — ~15 GiB for a 4B model. The estimator now adds that state plus a 1.25× margin measured, not padded: the observed failure wanted ~45.8 GiB, and logits×4.5 + state explains only ~84% of it; the residual is activations and the generation KV cache, which the function still does not model. It now rejects the config that died and cleared the fixed one with 7.2 GiB to spare on the live device.

Findings worth carrying forward

flyte run re-registers trainer resources, not just flyte deploy. train_resources() reads RT_TRAIN_GPU/RT_TRAIN_MEMORY at import, so the env is serialized from whichever process registers it. Setting the vars on the deploy alone put the first run on the 10Gi/T4 defaults while the deployment said L40S/48Gi. Export RT_TRAIN_* for both commands. The profile's train_gpu/train_memory are advisory — preflight compares against them, they do not request anything.

Step time tracks sequences per optimizer step, not nominal batch size. The original 18,000-step budget was extrapolated from a 3.57 s/step measurement taken at batch 8 / ngen 8. This arm puts four sequences through a step and measured 2.92 s/step over 839 steps — so 18,000 was ~16h, not a day. Budget is now 26,000.

Tests

tests/test_corpus_sampling.py (8 cases) pins the bounded read — including, precisely, that pd.read_parquet is never called, since a memory assertion in a unit test would report 0.4MB for the path that killed a 10Gi pod. tests/test_round11_arms.py gains the TRL divisibility check, the VRAM-fit check, and a regression that the batch-4 config can never pass preflight again. 263 pass.

Still open

• The run itself: does this move median overprovision below the rule baseline's 25–28%? That clause is the one every arm has failed.
• Artifact cards (shipped in 6654000) are still unverified on a live artifact — the first real card will be this run's tuner-checkpoint.
• The Qwen3-14B full-FT checkpoint from uk9pj886sxmw9sqnhgrt remains unevaluated (carried from round 12–14).

🤖 Generated with Claude Code