Speculative decoding proposes tokens with a cheap draft model and verifies them with the expensive target model. The acceptance rule decides which proposals survive. It determines two things at once: how many tokens a round emits per target forward (throughput) and what distribution the emitted stream follows (correctness). This document states which rule each code path runs, what each one guarantees, and how a request reports what speculation did.
Write p for the target model's effective next-token distribution at a verify
position (after token bias, penalties, top-k / top-p / min-p filtering on the
untempered distribution, XTC on the renormalised filtered row, and the final
temperature scaling) and q for the distribution the drafter actually drew
its proposal from.
Accept the drafted token iff it equals a fresh draw from p; emit that draw on
both branches. Lossless: every emitted token is a fresh conditional draw
from the target, so the stream is exactly a target-only sample. Its acceptance
probability is sum_x p(x) q(x).
Sampler-match where the sampler happens to be argmax. Lossless by the same
argument; it is exactly greedy target decoding.
Accept iff the drafted token equals argmax(target_logits), regardless of
temperature; emit the argmax on mismatch. Not lossless at temperature > 0:
the served stream is the target's greedy stream, not a sample from p. Its
acceptance probability is q(argmax p).
The Muse Glimmer assistant drafter (#1343) runs on the same DFlash round loop
under this rule, with no dispatch difference from the Qwen 3.5 DFlash drafter:
its proposals are sampled per masked slot under the request's sampler and
accepted against the target's argmax. At temperature == 0 it is exactly
greedy target decoding, subject to the block-versus-chain exactness probe
below, including past the 2048-token sliding window of the target's rotating
caches.
Its verify width is the one adaptive width in the DFlash loop, so the probe
runs more than once: the round loop warms up at the drafter's declared depth
of four rows and widens to the requested ceiling only when a measurement
window pays for it, and the gate clears both widths before the burst engages.
Probing the ceiling alone would admit the warm-up width unmeasured, which is
not a formality: the forward width selects which quantized-matmul kernel MLX
dispatches, and docs/benchmarks.md records the same comparison reading
20.6 percent disagreement at width 8 and 0.0 percent at width 32.
The LFM2 DSpark drafter runs on the DFlash round loop and therefore this rule,
with one difference in dispatch: it is greedy-only. The drafter's Markov head
proposes a chain trained against greedy targets, and the server declines any
request whose sampler is not argmax (temperature > 0 with top_k != 1) to
classic decode with one warning, rather than serving the biased greedy stream
under a sampling request. At temperature == 0 it is exactly greedy target
decoding, subject to the block-versus-chain exactness probe below.
For a q-distributed proposal and a p-distributed emission, the largest
probability the two can coincide is sum_x min(p(x), q(x)), the
maximal-coupling bound, equivalently 1 - TV(p, q). Every
distribution-preserving rule is capped there:
| Rule | Lossless | Acceptance probability |
|---|---|---|
| Sampler-match | yes | sum_x p(x) q(x) |
| Argmax-against-argmax | no | q(argmax p) (not bounded by the ceiling) |
q(argmax p) is under no such constraint: with p(a) = 0.3 and q(a) = 0.95
the argmax rule accepts at 0.95 where the ceiling for any correct rule is about
0.30. So no distribution-preserving rule can strictly improve mean accepted
length against argmax acceptance in general.
Modified rejection sampling (Leviathan et al. 2023, Chen et al. 2023) attains the ceiling. An opt-in implementation shipped on the classic draft-model loop and was removed with it in v0.8.0 (#2754); the full-model drafter that replaced that loop (#2757) runs the sampler-match rule of the engine's token-only loop, which is lossless, and does not restore the acceptance-optimal rule: see the next section.
| Path | temperature == 0 |
temperature > 0 |
|---|---|---|
PromptLookupDrafter on the engine's token-only loop (mlxcel generate --prompt-lookup) |
greedy argmax (lossless) | sampler-match (lossless, and acceptance-optimal here) |
PromptLookupDrafter in the server scheduler (--draft-kind prompt-lookup, the same Engine::verify_round) |
greedy argmax (lossless) | sampler-match (lossless, and acceptance-optimal here) |
FullModelDrafter on the engine's token-only loop (mlxcel generate --draft-model <ordinary model>, mlxcel-server --draft-model <ordinary model>, #2757) |
greedy argmax, no probe gate, not byte-identical to plain decode (the same class and the same streams as prompt lookup: see Offline routing) | sampler-match (lossless): the drafter proposes its own argmax, the target's draw decides |
| Gemma 4 MTP round loop | argmax (lossless here) | argmax-against-argmax (biased) |
| DFlash round loop (Qwen 3.5 DFlash drafter) | argmax (lossless here) | argmax-against-argmax (biased) |
| DFlash round loop (LFM2 DSpark drafter) | argmax (lossless here, probe-gated) | declines to classic decode |
| DFlash round loop (Muse Glimmer assistant drafter) | argmax (lossless here, probe-gated) | argmax-against-argmax (biased) |
"Classic decode" on this page means plain non-speculative decoding. The MTP and DFlash verify paths select the target token with
argmax_per_position / argmax_logits_to_array regardless of the sampler, and
their drafters return token ids without the distribution they were drawn from
(Drafter::draft_block returns Vec<i32>). Making them lossless requires
widening both the drafter interface and MtpTarget::verify_forward's output.
Prompt lookup proposes deterministically, so its q is one-hot on the drafted
token d. The two acceptance probabilities then coincide at p(d)
(sum_x p(x) q(x) = sum_x min(p(x), q(x)) = p(d)), and on a rejection both
rules emit a draw from p conditioned on not being d. Sampler-match is
therefore already the acceptance-optimal rule for that path.
The CUDA server's prompt-lookup governor additionally pauses low-yield rows
when the EMA of accepted proposals plus one falls below four tokens after at
least two drafted rounds and the latest round is also below the floor. A
latest round meeting the floor can repay startup EMA debt without pausing.
An isolated zero-accept round after an accepted-token EMA at least equal to
the floor receives one round of grace; partial low-yield acceptance does not.
Grace never extends beyond the first consecutive low-yield round; pausing
still requires the emitted-token EMA to be below the floor. This measured
field-boundary heuristic leaves the acceptance rule itself unchanged.
This is a proposal-cost decision, not a change to
the sampler-match acceptance rule. Paused rows still observe emitted tokens
and run shadow lookups; seven matching shadow tokens restore verification.
Plain asks, retracted proposals, and gated-out rows add no verify evidence.
The CLI has no break-even floor and retains its two-token shadow confirmation.
adaptive = false ignores the floor, and the graded policy is unchanged.
mlxcel generate --draft-model <dir> runs one of three round loops: MTP (explicit
--draft-kind mtp, or an MTP model_type), DFlash (a DFlash drafter directory,
explicit or auto-detected, on a target with an offline DFlash arm, today Laguna),
or the full-model drafter (#2757): an ordinary smaller checkpoint, auto-detected
when the directory carries no DFlash-family marker and no MTP model_type, or
selected with --draft-kind full-model. The full-model drafter is the engine's
token-only loop, the one --prompt-lookup runs, with the draft model's own KV
caches rolled back next to the target's; it needs a drafter that shares the
target's tokenizer, and a draft vocabulary larger than the target's is refused
before the load. Anything else fails before any drafter load, with an error
naming --draft-kind mtp, --draft-kind dflash, --draft-kind full-model and
--prompt-lookup. A target or a drafter whose state a cache trim cannot roll
back (recurrent, hybrid and model-owned families) is refused by the one rule
Engine::verify_rounds_unsupported applies to prompt lookup (#2495), on
mlxcel-server too. The CLI prints the drafter's acceptance and draft-forward
counts on stderr after the run.
Greedy exactness is the token-only loop's, which does not reach the bar the MTP
and DFlash paths reach through their block-versus-chain probe: at temperature 0
the stream is the target's own argmax at every position, but it is not guaranteed
byte-identical to plain decode, and nothing gates on the difference. On the
M1 Ultra record for #2757, 32 of 36 real-pair runs matched plain decode. The other
four come from the target alone: --prompt-lookup, which has no draft model and
no draft caches, emits the same divergent streams on the same prompts, and a
teacher-forced replay of the target over its own plain stream reproduces the flip
once its KV history is built by multi-token forwards. The pull request for #2757 holds that evidence.
A drafter's proposals cannot move the stream on their own, because the
target decides every position from its own caches; a misaligned draft cache costs
acceptance only. The sampled stream passes the chi-square and Markov guards.
A DFlash or DSpark drafter on a target without an offline arm gets the same
refusal plus a pointer to mlxcel-server, which drives the DFlash round loop for
more targets (#1168). These checkpoints ship no embed_tokens and no lm_head,
because they borrow both from the target when they bind. The LFM2 DSpark drafters
(#1339) and the Muse Glimmer assistant (#1343) have the same shape.
The discriminator is structural (a dflash_config block and/or
architectures: ["DFlashDraftModel"] or ["Lfm2DSparkDraftModel"] in the
drafter's config.json, or the Muse assistant's own
model_type: muse_glimmer_assistant / ["MuseGlimmerAssistantModel"]), not
the resolved DrafterKind. An ordinary full model auto-resolves to dflash too, so the resolved kind cannot tell
the two apart. The same structural check runs inside get_model_type, so
mlxcel generate -m <dflash-dir> is rejected as "not a standalone model" too.
Setting RUST_LOG installs a subscriber for the offline CLI commands (writing to
stderr, so piped stdout stays clean), which surfaces the drafter auto-detection
line. Default output is unchanged when RUST_LOG is unset. mlxcel serve is
unaffected: it installs its own subscriber. mlxcel generate --prompt-lookup
prints its drafter statistics on stderr after the run.
mlxcel-server reports what speculation did for one request on that
request's own response, so a client can answer "did the drafter help me" without
access to the server's logs (#1314). The block appears only when a drafter
executed at least one verify round; a classic-decode deployment, and a
speculative request that finished inside prefill, answer the body they always
did.
On the native /completion and /completions routes the counters ride the
existing timings object:
"timings": {"cache_n": 0, "prompt_n": 12, "predicted_n": 64, "...": "...",
"draft_n": 72, "draft_n_accepted": 55, "draft_rounds": 9,
"draft_kind": "dflash"}draft_n and draft_n_accepted are llama-server's own optional pair, spelled
as upstream spells them and appended the way upstream appends them, so a client
that already reads llama-server timings reads mlxcel's unchanged.
draft_rounds and draft_kind are mlxcel additions: upstream has neither a
round count nor a drafter-kind concept, and without the round count the mean
accepted length per round is not recoverable from the pair.
/v1/chat/completions carries the whole block as a top-level timings object,
on the non-streaming response and on the streaming chunk that carries
finish_reason. b10621 puts a timings object on its own OpenAI chat
responses, built from the same struct its native route uses, so mlxcel's is that
same object rather than the draft_* half alone: a client that probes for the
key and then reads predicted_per_second off it finds the key it expects. The
one difference that remains is presence, upstream emits the block on every chat
completion while mlxcel emits it only for a drafted request, and it is recorded
against both chat routes in compat/llama-server/b10621/routes.json.
Only the finish_reason chunk carries it: the counters are the run's totals,
and a mid-stream frame reporting a running draft_n would be reporting a total
that is not one yet. For the same reason the native route's per-token
timings_per_token frames carry the nine base keys and none of the four.
The base keys mean the same thing on a drafted request as on a classic one
(#1592). prompt_ms covers queue wait plus the target prefill, and
predicted_ms covers every verify round from the moment the prefill finished,
so predicted_per_second is the drafted decode rate and is comparable to the
classic figure for the same prompt. The burst runs to completion before it
streams, so it stamps the first-token instant at its prefill end rather than
when the finished tokens are replayed; before the fix that replay was the
stamp, prompt_ms absorbed the whole burst, and predicted_ms reported the
sub-millisecond replay (95000 tokens per second on a 96-token request). The
first drafted request after startup also pays the drafter's disk load, which
is a once-per-process cost and is excluded from every timings field; it
stays on the Drafter loaded log line. See the timings note in
docs/llama-server-compat.md.
The mean accepted length per round is (draft_n_accepted + draft_rounds) / draft_rounds: every round emits its accepted drafts plus one bonus token. It
is left to the client rather than reported, so a client that wants a different
aggregate is not stuck with this one. draft_kind carries a --draft-kind
name: dflash, mtp or prompt-lookup.
The block covers the three paths that own a request's whole round loop: the DFlash B=1 burst, the MTP B=1 burst, and the tick-cooperative MTP slice (whose counters are the session totals across every slice, not one slice's), plus prompt lookup's per-tick verify rounds, whose counters are the drafter's totals for the request (a proposal retracted because no forward followed it is not counted). The default-off B>1 batched burst reports nothing, because its round loops return per-row tokens without per-row acceptance counters.
The disaggregated router front reports nothing either. It assembles its stream
from the prefill/decode handoff protocol rather than from a finished
GenerationResult, and that protocol carries a generated-token count and no
acceptance counters, so surfacing the block there is a handoff-protocol change
and not a response-shaping one.
In src/lib/mlxcel-core/src/engine/speculative_distribution_tests.rs, all on the
engine's token-only verify loop:
speculative_stream_matches_the_target_distribution(and_at_block_size_one): chi-square goodness of fit of the emitted stream against the target's exact categorical, over 12000 tokens (MLXCEL_SPEC_DIST_SAMPLESoverrides), dof 5, rejected above 25.7448 (alpha = 1e-4). The target is context-free, so these two guard the sampler path into the stream and cannot see a wrong accept/reject decision.markov_stream_keeps_the_target_conditional: the same statistic over(previous, next)on a target whose distribution depends on the previous token, dof 30. This is the stochastic guard on the accept/reject decision.chi_square_rejects_known_wrong_acceptance_rules: power calibration on host draws. The same statistic must reject emitting the proposal untested and an argmax-against-argmax rule.filtered_target_support_is_never_violated: zero tolerance. One emitted token outside atop_k = 2target's support fails.temperature_zero_stream_is_byte_identical_to_greedy_target_only: zero tolerance on greedy output.acceptance_accounting_is_exact_in_both_deterministic_regimes: theSpeculativeRoundscounters in the all-reject and all-accept regimes.
engine/verify_round_rollback_tests.rs (kv_cache_length_is_exact_in_both_acceptance_regimes,
kv_cache_stays_consistent_across_mixed_accept_reject_rounds) pins that after
every round the state holds the prompt plus every emitted token except the
pending one, on every layer. src/lib/mlxcel-core/src/cache/kv_trim_tests.rs pins KVCache::trim, and
src/commands/generate_tests.rs pins the offline routing matrix.