[ydnyshhh]

Summary

This PR adds the reproducible generation and publication side of Synthetic Workspace Gym (SWG). It owns deterministic task generation, immutable curriculum construction, grounding and reward validation, secure execution-protocol projection, Hugging Face export, and Prime image builds. The native Verifiers V1 taskset is submitted separately in #26.

SWG covers tabular transformation, Python script repair, data pipelines, local-document retrieval, and composite workflows. The six V5 manifests contain 1,328 assignments spanning D1-D3 SFT/mid-training data, held-out SFT validation, D4-D5 RL data, and evaluation panels.

Architecture

The export has three trust domains:

1. A split-scoped agent image contains only visible workspaces.
2. A clean, answer-free execution runtime runs model-controlled code and emits observations/artifacts.
3. A dedicated trusted grader receives expected answers and computes rewards, but never executes model-controlled code.

Repository layout

• generators/synthetic_workspace_gym/: generators, schemas, evaluator-identical local validation, secure protocol projection, export, and publication library.
• configs/synthetic_workspace_gym/: compact V5 release pointer with the pinned dataset revision, export integrity SHA-256, and six reproducible source-manifest hashes. Generated JSON is not committed.
• scripts/synthetic_workspace_gym/: curriculum/calibration entrypoints, image builder assets, and architecture diagram.

V5 curricula

Manifest	Tasks	Intended use
sft-easy-v5	512	Family-calibrated D1-D3 teacher traces / mid-training
sft-validation-v5	96	Scenario- and seed-disjoint D1-D3 validation
rl-hard-v5	512	D4-D5 atomic and composite RL training
rl-eval-v5	120	Held-out D4-D5 scenarios, seeds, fixtures, and compositions
eval-d1-d4-paired-panel-48-v5	48	Within-family paired difficulty calibration
eval-d5-family-calibration-40-v5	40	D5 calibration by family and retrieval profile

Review hardening

• Generator calibration now uses the same precomputed execution protocol and scoring implementations as the shipped grader.
• Baselines are centrally measured after secure-protocol projection; the D4 composite baseline is 5/8 = 0.625, making every untouched composite D4 row normalize to 0.0.
• Nondeterministic command output cannot earn reward, and structure-only output is capped at the documented empty-output bar.
• Observation probes send a framed child-process payload that the trusted wrapper writes only after candidate shutdown, preventing atexit replacement of the destination.
• Exports require and embed the full public generator source revision.
• Exact-JSON retrieval tasks retain generation-time grounding validation with explicit derivation exemptions.
• Every generated task validates its reference solution and untouched-reward limit.

Immutable V5 artifacts

• Public dataset: ydnysh/synthetic-workspace-gym
• Dataset revision: 3ba83f647b96556be8c5084584a2d86e93225684
• Exporting public commit: 3bd13bb949846e7b7393afe81727f1127e984475
• Export integrity SHA-256: 128c3c984d91e7d46918e0f2f77cab4543943a25e11083bf18d0ad00dcf9f7f8
• Public Prime image tag: v5-128c3c98

All older artifact revisions remain unchanged. The V5 dataset and images were not republished: rebuilding the six source manifests in an ignored build directory reproduced their pinned SHA-256 hashes exactly. Evaluator provenance swg-capability-evaluators-v4 is independent of curriculum V5 and was bumped from v3 in the exporting commit.

Validation performed

uv run ruff check .
uv run ruff format --check .
uv run pytest -n auto tests -v
uv build generators/synthetic_workspace_gym

• Ruff and formatting passed.
• All 6 public-repository package tests passed.
• Generator sdist and wheel built successfully. All six rebuilt V5 source-manifest hashes matched the release pointer.
• 1,328/1,328 reference solutions scored 1.0.
• All 1,328 tasks were checked against an untouched-reward limit; zero violations occurred and the corpus maximum was 0.013497.
• All 60 composite D4 rows scored 0.0 untouched after shipped-evaluator normalization.
• Focused regressions passed for nondeterministic-output gating, structure-only scoring, and observation atexit tampering.

Out of scope

• This PR does not launch training jobs.
• The installable Verifiers V1 taskset is kept in Add Synthetic Workspace Gym taskset #26 for focused review.

Note

Add Synthetic Workspace Gym generators, evaluators, and dataset tooling

Adds the full swg_generators package under generators/synthetic_workspace_gym, covering five environment families (tabular, script repair, pipeline, retrieval workspace, composite workspace) with a shared base.py generator framework, D5 composition profiles, and per-family evaluators.

• Includes dataset pipeline tooling: export_artifacts for deterministic gzipped task export, publish_dataset for Hugging Face upload, and SHA-256 provenance fingerprinting in provenance.py.
• Adds a secure evaluation protocol: secure_protocol.py instruments assertEqual/assertTrue/assertRaises calls via AST to record labeled observations, and local_protocol.py runs evaluator plans in temporary workspaces.
• Adds calibration and validation scripts: D5 oracle monotonicity reports, retrieval surface validation, export validation, training-manifest builders, and Prime image build tooling in build_images.py.
• Risk: D5CalibratedEvaluator in registry.py cubes scores for unsuccessful difficulty-5 evaluations, changing reward values; evaluators outside difficulty 5 are unaffected.

^{Macroscope summarized 1e22f57.}

[snimu]

[major] The curriculum data belongs in the published dataset, not committed to the repo

configs/synthetic_workspace_gym/ adds ~87K lines of curriculum JSON, including four frozen revisions (v2–v4 plus the live v5) of each curriculum. The taskset already consumes this data from a pinned HF dataset revision, and this PR publishes exactly there (publish_dataset.py), so the checked-in copies are a redundant channel — and the dead revisions are pure bloat: the repo's own history serves the provenance role the README cites. Please follow the convention the other sandbox-image environments use: keep the data in the published dataset, commit only a small pointer that pins the dataset revision and the sha256 (or the generation_fingerprint you already compute), and drop the committed JSONs.

[written by prime-agent, checked by snimu]

[ydnyshhh]

Addressed in 5e99e4e: removed all 24 checked-in curriculum JSON files (86,592 lines). The six active V5 source manifests are now generated into an ignored build directory, and configs/synthetic_workspace_gym/release.json is the small pointer to the pinned HF dataset commit, export integrity SHA-256, exporting commit, and source-manifest SHA-256 hashes. I rebuilt the six manifests and verified all six hashes match the previously committed bytes. The published V5 dataset and images were not changed.

[snimu]

[minor] EVALUATOR_VERSION misreports the V5 protocol

EVALUATOR_VERSION = "swg-capability-evaluators-v4" ships in every exported task manifest's provenance while the curriculum family is V5. Provenance metadata that misreports its own version undermines exactly what it exists to guarantee — bump it alongside the protocol changes.

[written by prime-agent, checked by snimu]

[ydnyshhh]

Clarified in 5e99e4e. The two version numbers track different things: V5 is the curriculum/release revision; swg-capability-evaluators-v4 is the evaluator-protocol revision. Commit 3bd13bb (the exact exporting generator commit pinned by V5) explicitly bumped the evaluator label from v3 to v4 while introducing the V5 hardening. Thus the frozen task manifests correctly say v4. Renaming it to v5 now would change newly generated task-blob bytes without changing the already-published V5 artifacts, so I preserved the accurate label and documented the distinction in provenance.py and the generator README.