jevSweeper

Jev — TypeSafe's System One model — plays Minesweeper on https://www.minesweeper.cn/, driven through a real browser.

The point is not that a bot can play Minesweeper. The point is that Minesweeper is one of the few settings where the correct probability for every guess is exactly computable, so a model's probability judgment can be scored against ground truth instead of against somebody's opinion. That turns "does this model reason well?" into a number.

What came out of it

Two results, one about the model and one about how we were using it.

1. Jev's mine probabilities are worse than random. Over 25 stuck intermediate positions, asking one Noul question per candidate cell:

measurement	value
mean Spearman ρ (Jev's P(mine) vs exact P(mine))	−0.274
mean absolute error	0.317
spread of Jev's own numbers	0.237
spread of the true probabilities	0.704
picked the exact-optimal cell	1/25 (4%) — chance is 16.9%
picked a top-3 cell	9/25 (36%) — chance is 50.7%

A negative ρ means the ordering carries no information — and ordering is the only thing a policy actually needs. The compressed spread is the signature of it: on one position the true risks ranged 0.06 → 0.94 and Jev answered 0.770, 0.770, 0.770, 0.770. It was not estimating badly; it was not estimating.

2. But 86% of the time we should not have been asking at all. The escalation rule was "local rules are stuck → ask the model". On expert, 31 of 36 stuck positions were still logically decidable — they needed global counting, not the single-neighbourhood rules:

	before	after	model calls
beginner	92%	95%	8 → 7
intermediate	90%	92%	29 → 13
expert	10%	37%	74 → 51

Expert win rate nearly quadrupled, because the bot had been handing the decision to a model that was worse than the coin it replaced.

Baselines with no model at all: beginner 95%, intermediate 93.3%, expert 47.5%.

Why this is measurable at all

Two oracles, both free:

1. The true mine layout — read straight out of the live game's memory.
2. The true P(mine) of every cell — by exhaustively counting the mine layouts consistent with what is visible.

With those, every guess can be scored three ways: was the chosen cell actually safe, what was its real risk, and was a safer cell available. enumerate.py is cross-checked against brute-force enumeration over every consistent layout on small boards (tests/test_engine.py); if that test ever fails, every calibration number downstream is junk.

Quickstart

git clone https://github.com/lvk901/jevSweeper
cd jevSweeper
pip install -e ".[jev]"

# put your key in .env  ->  TYPESAFE_API_KEY=sk-...
python -m jev_sweeper.eval.runner --check

Then either watch it play, or play it yourself.

# you pick the level on the page, then press START
python scripts/manual.py

# or fully automatic, headless
python -m jev_sweeper.eval.runner --site --policy jev --level intermediate --games 5

# baselines need no API key
python -m jev_sweeper.eval.runner --policy exact --level expert --games 30

The panel over the page shows the current decision, the candidate cells, the latency, tokens and cost. START/STOP (or S / X) hand control over and take it back. A browser window stays open between games.

Measure things:

python scripts/calibrate.py --level intermediate --positions 40   # ρ, error, pick quality
python scripts/peek.py --level expert --seed 7                    # one position, side by side
python -m jev_sweeper.viz.build                                   # replay a recorded game

How it works

minesweeper.cn  <-- CDP -->  bridge/  <-- Board -->  agent/  --> real mouse events
   (canvas, no                reads the game's own    rules first, model
    DOM cells)                JS globals; real clicks  only when undecidable

bridge/ — the board is a <canvas>, so there are no DOM cells to scrape. We read the game's state out of JS memory instead: exact, no image processing, and it hands us the mine layout for free. An injected PRNG seeds Math.random, so a given --seed reproduces the same board.

engine/ — the rules, a deterministic solver (singles, subsets, propagation), and the exact probability oracle.

agent/ — one Policy interface, several implementations. Deterministic rules run first, then the exact counter, and only what neither can settle reaches the model. The model is asked one narrow question per candidate cell ("is this a mine?") against a small local window.

eval/ — runs games and records every decision to JSONL. viz/ — replays that JSONL as one self-contained HTML file.

Ground truth never reaches a policy. Board.is_mine is used for scoring and for the UI only.

Why the model is asked narrow questions

Jev is a decision model, not a reasoning model — its own documentation says it is for gut-check determinations, and that anything needing extended reasoning should be decomposed. Guessing in Minesweeper is a global constraint problem, which is the wrong shape of task. What a human expert does when stuck is local pattern recognition — "1, 2, 1, so that one is a mine" — which is a gut check. So the model sees a small window with the cells under consideration marked by letter, and all the arithmetic, weighting and thresholds stay in code.

Notes on minesweeper.cn

Reverse-engineered from new.6.31.js. State lives in globals: g[y][x] = [cell, mine, adjacent, scratch] where cell is 0 covered / 1 revealed / 2 flagged, mine is 1 for a mine, and adjacent is the number shown. Also m width, h height, n mines, o status (0 ready, 1 playing, 2 won, 3 lost), E mines remaining, R cells left to reveal. Cells are a hard-coded 25 px.

Gotchas that cost real debugging time:

• The first click relocates mines out of its 3×3 (u()), so any mine layout read before the first click is provisional. It only guarantees the clicked cell is safe — unlike most implementations it does not guarantee a zero-opening, which makes win rates here lower than the published numbers for optimal play.
• Mouse handling is rr on mousedown and fr on mouseup, gated by two preferences that default to making only mousedown act. A browser profile with those changed silently double-fires every move.
• The full-screen level derives its board size from the window, so anything that resizes the page deals a new board. See set_viewport for why growing the viewport to fit a large board is a trap.
• On a loss the clicked mine is marked revealed without decrementing R.

Honest limitations

• The calibration sample is 25 positions, intermediate, 8 candidates. Enough to be clear, not enough to be a paper.
• This may be our question design, not the model's ceiling. The obvious next experiments are a Choice over candidate cells instead of per-cell Noul, and putting the remaining mine count into the state. If ρ turns positive, the model was fine and we asked badly.
• The exact counter is not always available: it falls back to a uniform prior when a position is too tangled. enumerate.STATS.fallback_rate reports the share, and calibration should exclude those positions. It also cannot be trusted to be fast on pathological frontiers.
• Jev's confidence is computed from the shape of a probability distribution, not an independent signal, and Noul does not return one at all. Calibration work must use probabilities.
• TypeSafe's published latency for Jev is ~114 ms; we measure ~750 ms per call, against ~5 ms for a whole bot step.

Performance

one bot step (read board, solve, record, update panel)	~5 ms
CDP round trips per step	4
one model call	~750 ms
cost per game	~$0.00003

The bridge is not the bottleneck and cannot be made meaningfully faster; the model is. The only optimisation that matters is calling it less, which — given ρ = −0.274 — also improves play.

Layout

src/jev_sweeper/
  config.py          .env and environment lookup for the API key
  engine/            rules, deterministic solver, exact probability oracle
  agent/             policies: random, exact, jev
  bridge/            CDP client, minesweeper.cn adapter, on-page decision panel
  eval/              game runner and JSONL recorder
  viz/               replay page and its builder
scripts/             manual console, calibration, position inspector, watcher
tests/               engine tests incl. the oracle cross-check, and secret guards

Development notes

Things that were not obvious and are worth keeping:

• A wrong move and a lost move look identical unless you record enough to tell them apart. Every step stores the grid, the ground truth, the candidates and the chosen cell's true risk, which is what made the analysis above possible.
• The bot must verify the board is still the same board before it clicks. A resize or a level button redeals it, and deducing against stale evidence produces a bot that clicks mines while reporting it never had to guess. BoardChanged aborts instead.
• A loss with zero guesses is impossible unless something else ended the game, so those are labelled external rather than counted as model failures. Quietly accepting them would have poisoned every win rate.
• Loops that never block on a fixed height need a search budget. The exact counter hangs on a loose frontier without one.

Tests

python -m pytest tests -q

test_exact_estimates_match_brute_force is the important one. The rest guard the API key and the browser profile from ever being committed.

Licence

MIT. Not affiliated with TypeSafe AI or minesweeper.cn.