Add Synthetic Workspace Gym generators

configs/synthetic_workspace_gym/README.md

@@ -0,0 +1,5 @@
+# Synthetic Workspace Gym configurations
+
+The frozen V5 curricula live in the pinned Hugging Face dataset, not in this repository. `release.json` records the dataset commit, export integrity hash, exporting generator commit, and SHA-256 hashes of the six source manifests. Earlier published dataset revisions remain frozen on Hugging Face.
+
+To reproduce the source manifests without adding generated data to Git, run `scripts/synthetic_workspace_gym/build_qwen35_training_manifests.py`. It writes to the ignored `generators/synthetic_workspace_gym/build/manifests/` directory by default. Compare those files with the hashes in `release.json` before attempting to reproduce the pinned export. The installable taskset loads the published curriculum rows from the pinned dataset commit.

configs/synthetic_workspace_gym/release.json

@@ -0,0 +1,14 @@
+{
+  "dataset_repo": "ydnysh/synthetic-workspace-gym",
+  "dataset_revision": "3ba83f647b96556be8c5084584a2d86e93225684",
+  "export_integrity_sha256": "128c3c984d91e7d46918e0f2f77cab4543943a25e11083bf18d0ad00dcf9f7f8",
+  "exporting_generator_revision": "3bd13bb949846e7b7393afe81727f1127e984475",
+  "source_manifests_sha256": {
+    "eval-d1-d4-paired-panel-48-v5.json": "cb1271ce51a9677b3be9da90c9585798a2134fbb769a4e70b0a1cb3b393609fa",
+    "eval-d5-family-calibration-40-v5.json": "4c34bfc894be1d1c883a4fb9a892de6b48a94b14f354ec9c564984801d607aef",
+    "rl-eval-v5.json": "56b857976dc30a405089bc5c1c90d8d8cd36be267f827c8c08b82be4c42790bc",
+    "rl-hard-v5.json": "fa152e4c73517c3c76bb01e6183c66f19e05842a357e0f877b427e12e6e8d87a",
+    "sft-easy-v5.json": "c604b98c302d8434369dc40f60071525a9bfebff23d747c1dc493844ff35763f",
+    "sft-validation-v5.json": "5f30a82295e61e678adf89d8806ba8e7967069b1ef62a1f33d0683e3fac54509"
+  }
+}

generators/synthetic_workspace_gym/README.md

@@ -0,0 +1,59 @@
+# Synthetic Workspace Gym generators
+
+This directory contains the deterministic generation, provenance, and validation library for the Synthetic Workspace Gym taskset. It is deliberately separate from the installable environment package.
+
+The six V5 curricula and evaluation panels are published in the immutable dataset; `configs/synthetic_workspace_gym/release.json` pins their release and hashes the reproducible source manifests. The curricula calibrate untouched baselines through the shipped execution protocol and harden structure-only and nondeterministic rewards:
+
+- `sft-easy-v5`
+- `sft-validation-v5`
+- `rl-hard-v5`
+- `rl-eval-v5`
+- `eval-d1-d4-paired-panel-48-v5`
+- `eval-d5-family-calibration-40-v5`
+
+Standalone generation and calibration entrypoints live in `scripts/synthetic_workspace_gym/`. From the repository root, rebuild the curricula with:
+
+```bash
+uv run --project generators/synthetic_workspace_gym python scripts/synthetic_workspace_gym/build_qwen35_training_manifests.py
+```
+
+Generate the review-safe artifact layout from all 1,328 assignments, validating each reference solution while exporting:
+
+```bash
+uv run --project generators/synthetic_workspace_gym swg-export-dataset \
+  --manifest-dir generators/synthetic_workspace_gym/build/manifests \
+  --output-dir generators/synthetic_workspace_gym/dist \
+  --generator-revision "$(git rev-parse HEAD)"
+```
+
+`--generator-revision` must be the full committed source revision used for the export. It is embedded in the export metadata and every task manifest. Generation writes canonical UTF-8/LF bytes so the same revision produces identical task fingerprints on Windows and Linux. The command above creates a **new** export from the current commit; to reproduce the already-published V5 bytes, check out the `exporting_generator_revision` recorded in `configs/synthetic_workspace_gym/release.json` and use that revision. Do not overwrite the frozen dataset or image tags.
+
+The source-manifest hashes in `configs/synthetic_workspace_gym/release.json` describe the already-published V5 export. The `swg-capability-evaluators-v4` provenance label is a separate evaluator revision, not a curriculum number: V5 was exported after the evaluator moved from v3 to v4. Changing that label would change task-blob bytes and requires a new immutable export; it must not silently rewrite the pinned V5 release.
+
+The export deliberately separates trust domains. `dist/dataset/` contains lightweight curriculum indexes and one deterministic grader-only blob per unique task. `dist/images/<curriculum>/` contains only the visible workspaces for that curriculum. `integrity.json` records every exported file digest. Hidden files and visible workspaces are never combined in one distributable archive.
+
+Publish the dataset directory privately and retain the immutable commit returned by Hugging Face. Because the dataset contains grader-only payloads, public visibility requires the explicit `--public` flag:
+
+```bash
+uv run --project generators/synthetic_workspace_gym swg-publish-dataset \
+  --dataset-dir generators/synthetic_workspace_gym/dist/dataset \
+  --repo-id <organization>/<dataset>
+```
+
+Build the six split-isolated agent images and the dedicated grader image with Prime's server-side builder. The Dockerfiles pin their Python base image by digest, and the generated map resolves every pushed image to an immutable digest:
+
+```bash
+uv run --project generators/synthetic_workspace_gym \
+  python scripts/synthetic_workspace_gym/build_images.py \
+  --artifacts generators/synthetic_workspace_gym/dist \
+  --tag <artifact-version> \
+  --output generators/synthetic_workspace_gym/dist/task_images.json
+```
+
+Run static checks and build the generator package with:
+
+```bash
+uv run ruff check generators/synthetic_workspace_gym scripts/synthetic_workspace_gym
+uv run ruff format --check generators/synthetic_workspace_gym scripts/synthetic_workspace_gym
+uv build generators/synthetic_workspace_gym
+```

generators/synthetic_workspace_gym/pyproject.toml

@@ -0,0 +1,29 @@
+[project]
+name = "synthetic-workspace-gym-generators"
+version = "0.1.0"
+description = "Generation, provenance, and validation code for Synthetic Workspace Gym."
+readme = "README.md"
+requires-python = ">=3.12,<3.14"
+dependencies = ["huggingface-hub>=0.28,<2"]
+
+[project.scripts]
+swg-export-dataset = "swg_generators.export_dataset:main"
+swg-publish-dataset = "swg_generators.publish_dataset:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["swg_generators"]
+
+[dependency-groups]
+dev = [
+    "ruff>=0.12.1",
+]
+
+[tool.ruff]
+line-length = 120
+
+[tool.ruff.lint]
+select = ["F", "I"]

generators/synthetic_workspace_gym/swg_generators/__init__.py

@@ -0,0 +1,5 @@
+"""Synthetic Workspace Gym generation and validation code."""
+
+from swg_generators.export_dataset import export_artifacts
+
+__all__ = ["export_artifacts"]

generators/synthetic_workspace_gym/swg_generators/analysis/__init__.py

@@ -0,0 +1 @@
+"""Generator calibration helpers."""

generators/synthetic_workspace_gym/swg_generators/analysis/calibration.py

@@ -0,0 +1,60 @@
+from __future__ import annotations
+
+from collections.abc import Mapping, Sequence
+
+
+def build_monotonicity_report(
+    *,
+    task_id: str,
+    ordered_states: Sequence[tuple[str, float]],
+    capability_count: int | None = None,
+    semantic_dependency_depth: int | None = None,
+) -> dict[str, object]:
+    """Describe an ordered oracle staircase and any reward regressions."""
+
+    if not ordered_states:
+        raise ValueError("At least one calibration state is required.")
+    states = {name: round(float(score), 6) for name, score in ordered_states}
+    violations: list[dict[str, object]] = []
+    for (before_name, before), (after_name, after) in zip(ordered_states, ordered_states[1:]):
+        if float(after) + 1e-9 < float(before):
+            violations.append(
+                {
+                    "before": before_name,
+                    "before_reward": round(float(before), 6),
+                    "after": after_name,
+                    "after_reward": round(float(after), 6),
+                }
+            )
+    full_name, full_reward = ordered_states[-1]
+    if full_name != "full" or abs(float(full_reward) - 1.0) > 1e-9:
+        violations.append(
+            {
+                "state": full_name,
+                "expected": 1.0,
+                "actual": round(float(full_reward), 6),
+                "reason": "final state must be named full and score exactly 1.0",
+            }
+        )
+    return {
+        "task_id": task_id,
+        "states": states,
+        "monotonic": not violations,
+        "violations": violations,
+        "capability_count": capability_count,
+        "semantic_dependency_depth": semantic_dependency_depth,
+    }
+
+
+def capability_pass_rates(rows: Sequence[Mapping[str, object]]) -> dict[str, float]:
+    totals: dict[str, float] = {}
+    counts: dict[str, int] = {}
+    for row in rows:
+        for name, value in dict(row.get("subscores", {})).items():
+            name = str(name)
+            if not name.startswith("capability_"):
+                continue
+            capability = name.removeprefix("capability_")
+            totals[capability] = totals.get(capability, 0.0) + float(value)
+            counts[capability] = counts.get(capability, 0) + 1
+    return {name: round(totals[name] / counts[name], 6) for name in sorted(totals) if counts[name]}

generators/synthetic_workspace_gym/swg_generators/analysis/oracle_calibration.py

@@ -0,0 +1,65 @@
+from __future__ import annotations
+
+import shutil
+from collections.abc import Mapping, Sequence
+from pathlib import Path
+
+from swg_generators.analysis.calibration import build_monotonicity_report
+from swg_generators.evaluators.base import BaseEvaluator
+from swg_generators.schemas import EnvironmentManifest
+
+
+def evaluate_ordered_oracle_states(
+    *,
+    task_id: str,
+    correct_files: Mapping[str, str],
+    selected_bugs: Sequence[Mapping[str, object]],
+    visible_template: Path,
+    scratch_root: Path,
+    evaluator: BaseEvaluator,
+    manifest: EnvironmentManifest,
+    hidden_root: Path,
+    capability_count: int,
+    semantic_dependency_depth: int,
+) -> dict[str, object]:
+    """Evaluate five useful reference states without enumerating a full lattice."""
+
+    labels = [str(bug["label"]) for bug in selected_bugs]
+    if len(labels) == 2:
+        fixed_sets = [
+            ("untouched", frozenset()),
+            ("one_capability", frozenset(labels[:1])),
+            ("all_but_one", frozenset(labels[:-1])),
+            ("two_capabilities", frozenset(labels[:2])),
+            ("full", frozenset(labels)),
+        ]
+    else:
+        fixed_sets = [
+            ("untouched", frozenset()),
+            ("one_capability", frozenset(labels[:1])),
+            ("two_capabilities", frozenset(labels[:2])),
+            ("all_but_one", frozenset(labels[:-1])),
+            ("full", frozenset(labels)),
+        ]
+    ordered_scores: list[tuple[str, float]] = []
+    for index, (state_name, fixed) in enumerate(fixed_sets):
+        workspace = scratch_root / f"{index:02d}-{state_name}"
+        shutil.copytree(visible_template, workspace)
+        candidate_files = dict(correct_files)
+        for bug in selected_bugs:
+            if str(bug["label"]) in fixed:
+                continue
+            target_path = str(bug["target_path"])
+            candidate_files[target_path] = bug["apply"](candidate_files[target_path])
+        for relative_path, content in candidate_files.items():
+            target = workspace / relative_path
+            target.parent.mkdir(parents=True, exist_ok=True)
+            target.write_text(str(content), encoding="utf-8")
+        result = evaluator.evaluate(workspace, manifest, hidden_root)
+        ordered_scores.append((state_name, float(result.score)))
+    return build_monotonicity_report(
+        task_id=task_id,
+        ordered_states=ordered_scores,
+        capability_count=capability_count,
+        semantic_dependency_depth=semantic_dependency_depth,
+    )

generators/synthetic_workspace_gym/swg_generators/evaluators/__init__.py

@@ -0,0 +1,3 @@
+from .registry import get_evaluator, list_evaluators
+
+__all__ = ["get_evaluator", "list_evaluators"]

generators/synthetic_workspace_gym/swg_generators/evaluators/base.py

@@ -0,0 +1,12 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from pathlib import Path
+
+from swg_generators.schemas import EnvironmentManifest, EvaluatorResult
+
+
+class BaseEvaluator(ABC):
+    @abstractmethod
+    def evaluate(self, workspace_path: Path, manifest: EnvironmentManifest, hidden_root: Path) -> EvaluatorResult:
+        raise NotImplementedError

generators/synthetic_workspace_gym/swg_generators/evaluators/capabilities.py

@@ -0,0 +1,45 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class CapabilityScore:
+    """One independently observable evaluator capability."""
+
+    name: str
+    value: float
+    weight: float
+    diagnostic: str | None = None
+
+    def __post_init__(self) -> None:
+        if not self.name.strip():
+            raise ValueError("Capability names must not be empty.")
+        if self.weight < 0:
+            raise ValueError("Capability weights must be non-negative.")
+
+    @property
+    def clamped_value(self) -> float:
+        return max(0.0, min(1.0, float(self.value)))
+
+
+def weighted_capability_score(capabilities: list[CapabilityScore]) -> float:
+    """Return a normalized weighted score without semantic floors or caps."""
+
+    total_weight = sum(item.weight for item in capabilities)
+    if total_weight <= 0:
+        raise ValueError("Capability weights must sum to a positive value.")
+    score = sum(item.clamped_value * item.weight for item in capabilities)
+    return round(max(0.0, min(1.0, score / total_weight)), 6)
+
+
+def capability_subscores(
+    capabilities: list[CapabilityScore],
+) -> dict[str, float]:
+    return {f"capability_{item.name}": item.clamped_value for item in capabilities}
+
+
+def capability_diagnostics(
+    capabilities: list[CapabilityScore],
+) -> dict[str, str]:
+    return {item.name: item.diagnostic for item in capabilities if item.diagnostic is not None}

generators/synthetic_workspace_gym/swg_generators/evaluators/composite_workspace.py

@@ -0,0 +1,7 @@
+from __future__ import annotations
+
+from .retrieval_workspace import RetrievalWorkspaceEvaluator
+
+
+class CompositeWorkspaceEvaluator(RetrievalWorkspaceEvaluator):
+    """Capability-scored evaluator for retrieval-guided pipeline tasks."""