A coding agent built on pi, re-engineered for long-horizon terminal tasks.
Terminal-Bench 2.1 pass@1 from 0.539 to 0.773 — on par with Claude Code on the same model.
English · 简体中文
| Agent | Terminal-Bench 2.1 pass@1 |
|---|---|
| Crux | 0.773 0.793 on a rerun |
| Claude Code, same model | 0.730 |
| pi, upstream | 0.539 |
Self-hosted Qwen3.8-27B, identical weights for every agent · 89 tasks · pass@1 over whole runs.
All changes are in the agent layer; the model is untouched.
- Context engine. Compaction requests are sized to fit the window, retried smaller on rejection, and summarize single-task sessions in full; the task's original wording is carried verbatim through every compaction. Compaction success: 14% → 100%.
- Agent loop. Recovery paths live in one explicit state machine: two-phase recovery from truncated output, the reasoning tail handed back after a cut, stall detection, and pacing against each task's own deadline. Truncated outputs: 18.0% → 0.7%.
- Runtime. A stream idle watchdog, a default command timeout, process-tree reaping and resume after an OOM kill keep multi-hour runs alive. Idle time in timed-out runs: 94% → 3%.
- Tools. A failed edit shows the nearest matching region of the file, long command output keeps both its head and its tail, and the model keeps short visible notes beside its tool calls.
benchmark/ holds the harness behind these numbers: it runs the
agent in task containers via harbor, classifies every trial's outcome, and
compares versions task by task with a paired sign test.
cd benchmark && uv sync
harbor run --dataset terminal-bench/terminal-bench-2-1 \
--agent crux.pi_agent:CruxPiAgent --model openai/<model>Built on pi by Earendil Works
(earendil-works/pi). Licensed as
upstream; see LICENSE.