Repository navigation
Fix/judge framework tool awareness - #126
Merged
Merged
Conversation
Introduce HARNESS_TOOL_GUIDANCE so trace metrics can consistently treat harness meta-tools as neutral infrastructure rather than task attempts. Co-authored-by: Cursor <cursoragent@cursor.com>
Preserve harness-prefixed tool names verbatim and score meta-tool arguments on their own merits instead of penalizing calls that do not advance the task. Co-authored-by: Cursor <cursoragent@cursor.com>
Consulting learned guidance before acting is deliberate preparation, not hesitation or strategy abandonment. Co-authored-by: Cursor <cursoragent@cursor.com>
Harness calls are operational infrastructure, not steps of the agent's task plan, so they should not count as deviations or extraneous actions. Co-authored-by: Cursor <cursoragent@cursor.com>
Plans describe task work; omitting harness meta-tools is expected and this prompt never sees tool calls directly. Co-authored-by: Cursor <cursoragent@cursor.com>
Judge efficiency over task-directed steps only; harness state checks are sanctioned preparation, not redundant or superfluous steps. Co-authored-by: Cursor <cursoragent@cursor.com>
…letion. Distinguish harness preparation from task attempts, score partial progress fairly, and reserve zero scores for outcomes that show no progress. Co-authored-by: Cursor <cursoragent@cursor.com>
Preserve harness-prefixed tool names verbatim and exclude meta-tools from mis-selection and relevance penalties. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Provide a brief summary of the changes and the motivation behind them.
Related Issues
Link any related issues below (use
Closes #123to auto-close on merge).Type of Change
Contributor Checklist
make lintpasses with no errorsmake test-unitpassesScreenshots / Logs
If applicable, add screenshots or log output to help reviewers.
Note
Medium Risk
Changes only LLM judge prompt text across multiple eval metrics, so scores and pass/fail thresholds may shift for harness-instrumented traces; no runtime API or auth impact.
Overview
Trace evaluation prompts were penalizing
harness_*meta-tool calls (reading rules, notices, prior traces) as bad tool use, wasted steps, or failed task work. This PR adds sharedHARNESS_TOOL_GUIDANCEinmetrics/utils.pyand threads it through the trace metric templates so judges treat those calls as neutral infrastructure, score only task-directed actions, and preserve exact tool names in extraction.Argument correctness and tool correctness now keep verbatim
harness_names and avoid marking harness args or selection wrong when the call is only consulting harness state. Confidence, step efficiency, and plan adherence no longer count harness consultation as hesitation, redundancy, or plan deviation; plan extraction explicitly omits harness steps from the task plan. Task completion separates harness prep from task outcomes in extraction and verdict (no progress credit, but not a “failed attempt”); partial traces score incomplete progress rather than defaulting to zero. Plan quality documents that harness tools are out of scope for the plan text (no shared block, since it never sees raw tool calls). The Docker entrypoint banner version is bumped to v0.5.0.Reviewed by Cursor Bugbot for commit 054bd6b. Bugbot is set up for automated code reviews on this repo. Configure here.