Skip to content

README.md

Latest commit

 

History

History
70 lines (54 loc) · 3.23 KB

README.md

File metadata and controls

70 lines (54 loc) · 3.23 KB
Raw

Did One Session Account For Most Of The Increase?

From the repository checkout, with Go installed:

sh examples/investigate.sh --sessions

The v0.3.1 report starts with:

1 of 8 tracked sessions flagged for review: 90.91% of attributed tokens.

No Docker, provider account, API key, or traffic capture is needed. The script builds fleetdiff and reads two checked-in synthetic summary files. With a verified v0.3.1 binary already installed, skip the build:

fleetdiff investigate --before examples/sessions/data/before \
  --after examples/sessions/data/after --expected app

The Scenario

Each window covers 15 seconds. Before: four users, each with one session and one model attempt reporting 80 input and 20 output tokens. After: three of those users make one ordinary attempt each in new sessions; a new user makes ten attempts in one session, each reporting 250 input and 50 output tokens.

Question Expected answer
Did usage rise? 400 to 3,300 reported tokens; 4 to 13 model attempts
More attempts, or more tokens per attempt? About +1,592 and +1,308 tokens respectively, totaling +2,900
Did one session dominate? One new session has bounds [3000, 3000] and share [90.91%, 90.91%]; only it is flagged
Which user accounts for that volume? One new pseudonymous user has the same 3,000-token bounds
Are observations complete? Both declared intervals are complete, with no missing numeric usage
Did a provider report those counts? cannot_determine: no provenance declarations were supplied

The text report says flagged for review; JSON retains runaway_candidate. The session's share lower bound exceeds the review threshold. Shares cover attributed weight; this small fixture retains every key, so the bounds are exact.

Item aliases are local to each measurement. The scenario defines the user/session relationship; matching item-1 labels alone do not join identities.

Reproduce The Files

The generator composes llm-sketchkit hashing, frequent-items, HLL and summary APIs. These deterministic library-generated fixtures have synthetic accounting and extraction markers. Keep them separate from production exports.

GOWORK=off go run ./examples/sessions -out /tmp/fleetdiff-sessions-new
diff -ru examples/sessions/data /tmp/fleetdiff-sessions-new
GOWORK=off go test ./examples/sessions ./internal/cli

The output directory must not exist. The generator uses fixed timestamps and a public synthetic hashing key, never suitable for real traffic. Tests compare generated bytes to the checked-in files, check the CLI's answers and absence of raw sentinel values, and assert that other sessions are not flagged. The release binary smoke test also runs this scenario.

For your own application, enable collector v0.3.0's topk_keys for user_key and session_key with token weights, then collect two complete windows. Keep the hashing key private and the same across the comparison. Use the LiteLLM recipe when collecting through a gateway, including its provider-origin reporting.