proposer_solver: the proposer's contract answer is never verified, so false or ambiguous problems earn max learnability #2947

What happens

In environments/proposer_solver, the proposer's JSON contract answer becomes the solvers' ground truth without any check. The docstrings describe it as "the proposer's verified answer", but SolveTask.from_trace only checks that answer is a JSON integer, and finalize rewards the proposer purely on solve rate (4p(1-p)).

So a wrong label and a genuinely hard problem look identical to the reward:

Obviously false easy problems mostly earn 0, because solvers all disagree with the label. The practical shortcut is an ambiguous or underspecified problem with an arbitrary answer: solvers split between readings for reasons unrelated to difficulty, and with n=4 a 50/50 split gives expected learnability ~0.75 without constructing or verifying anything hard.

Because train_solver defaults to True, solver rollouts in those episodes are also trained toward the wrong label (the solvers who answered correctly get 0).

Reproduce

Deterministic, no model calls; uses the env's real from_trace, correct and finalize (tested on main at 8859385):

import asyncio, json
import verifiers.v1 as vf
from verifiers.v1.graph import MessageNode
from verifiers.v1.types import AssistantMessage, UserMessage
from proposer_solver.taskset import ProposerSolverEnv, ProposerSolverTaskset, SolveTask


def trace(task, user, reply, agent):
    nodes = [MessageNode(parent=None, message=UserMessage(content=user), sampled=False),
             MessageNode(parent=0, message=AssistantMessage(content=reply), sampled=True)]
    return vf.Trace(agent=vf.AgentInfo(name=agent, config=vf.AgentConfig()),
                    task=vf.TraceTask(type=type(task).__name__, data=task.data), nodes=nodes)


async def episode(seed, problem, answer, replies):
    proposer = trace(seed, seed.data.prompt, "I verified it.\n" + json.dumps({"problem": problem, "answer": answer}), "proposer")
    solve = SolveTask.from_trace(proposer)
    solvers = [trace(solve, solve.data.prompt, r, "solver") for r in replies]
    for t in solvers:
        await solve.score(t, None)
    env = object.__new__(ProposerSolverEnv)  # finalize only uses the static _solve_rate
    await env.finalize(seed, vf.Episode(task=vf.TraceTask(type="ProposeTask", data=seed.data), traces=[proposer, *solvers]))
    return [t.rewards["correct"].score for t in solvers], proposer.rewards["learnability"].score


async def main():
    seed = list(ProposerSolverTaskset(vf.TasksetConfig()))[0]
    for problem, answer, replies in [("What is 1+1?", 3, ["2", "2", "3", "3"]),
                                     ("If x^2 = 9, what is x?", -3, ["3", "3", "-3", "-3"]),
                                     ("What is 1+1?", 2, ["2", "2", "2", "2"])]:
        print(problem, "->", answer, await episode(seed, problem, answer, replies))


asyncio.run(main())

What is 1+1? -> 3 ([0.0, 0.0, 1.0, 1.0], 1.0)
If x^2 = 9, what is x? -> -3 ([0.0, 0.0, 1.0, 1.0], 1.0)
What is 1+1? -> 2 ([1.0, 1.0, 1.0, 1.0], 0.0)

Impact

This is the repo's self-play task-generation recipe. As shipped, the proposer is paid for divisive problems rather than correct ones, so a policy can drift toward mislabeled or ambiguous problems, and with train_solver=True those labels feed straight into solver training.

Possible fixes

• Make verification part of the contract, e.g. a solution_code field the env executes in the proposer's runtime; give the proposer 0 (or reject the episode) if it doesn't produce answer.
• And/or gate on a reference solver: if the non-matching solvers agree with each other on a different answer, treat the label as wrong or the problem as ambiguous.
• Skip solver training on episodes whose label wasn't independently confirmed.
• Until then, drop "verified" from the docstrings so the behaviour is clear.

Happy to help test a fix. Found while auditing public RL environments.