Skip to content

AGENTS.md: rewards section (bounds, binary by default) - #39

Merged
snimu merged 1 commit into
mainfrom
chore/rewards-section
Oct 30, 2026
Merged

snimu merged 1 commit into
mainfrom
chore/rewards-section

Conversation

@snimu

@snimu snimu commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a ### Rewards subsection to Environment Design — the same section as RL-Residency/residency-environments#83, for this repo's copy of the AGENTS.md:

  • Bounds: rewards are floats in [0, 1], inclusive; each reward function returns [0, 1] and weights are non-negative. Multi-environment training in prime-rl does no reward re-scaling, so a reward outside [0, 1] silently changes the relative weighting of that environment's gradients; GRPO's group normalization makes advantages negative or positive from group statistics, so the reward range never needs to encode direction.
  • Binary by default: 1 only when the task is genuinely complete, verified by the most authoritative mechanism available (kernel, simulator, upstream evaluator), else 0. Partial or shaped rewards are the exception and need a stated reason in the environment's PR, with every threshold documented; observability-only partial signal belongs in metrics, not rewards.

Note

Low Risk
Documentation-only change to contributor guidance; no runtime, auth, or training code is modified.

Overview
Adds a new ### Rewards block under Environment Design in AGENTS.md, right after the tool-description prompting rules.

It standardizes reward bounds: values and per-function outputs stay in [0, 1], weights are non-negative, with rationale tied to multi-env prime-rl training (no re-scaling) and GRPO (direction comes from advantages, not the reward sign/range).

It also sets binary completion rewards as the default—1 only when completion is verified by the strongest check available, otherwise 0—and requires explicit PR justification and documented thresholds for any partial/shaped rewards, keeping observability-only signal in metrics instead of rewards.

Reviewed by Cursor Bugbot for commit be0d9c8. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add rewards guidance section to AGENTS.md (0-1 bounds, binary by default)

New section in AGENTS.md sets reward design rules for agents: rewards must be floats from 0 to 1, weights non-negative, and binary rewards tied to authoritative verification are the default.

  • Partial or shaped rewards require PR justification and documentation; observability-only signals belong in metrics, not rewards.
  • Includes rationale on reward scaling, GRPO group normalization, and reward hacking.

Macroscope summarized be0d9c8.

@snimu
snimu requested a review from a team as a code owner October 30, 2026 12:47
@snimu
snimu requested a review from xeophon October 30, 2026 12:48
@snimu
snimu merged commit 524a22f into main Oct 30, 2026
7 checks passed
@snimu
snimu deleted the chore/rewards-section branch October 30, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants