Skip to content

feat(PROJ-000): add llm-extract worked example - #6

Merged
HenryCordes merged 1 commit into
mainfrom
henry/PROJ-000-llm-extract-example
Jul 16, 2026
Merged

HenryCordes merged 1 commit into
mainfrom
henry/PROJ-000-llm-extract-example

Conversation

@HenryCordes

Copy link
Copy Markdown
Owner

Summary

Adds the third worked example: llm-extract — typed field extraction across a nondeterministic LLM boundary, completing the progression sync/pure → async I/O → nondeterministic. The differentiator is the eval layer: extraction accuracy is measured against a golden set (live run: 0.950 mean per-field accuracy over 5 fixtures), not assumed.

Changes

  • src/llm-extract/: llmExtract(text, fields, {client, retries}) — deterministic core (prompt build, fence-stripping parse, schema validation), injected one-method LlmClient, bounded repair-retries embedding exact validation errors, ExtractError with .attempts/.cause; transport rejections propagate (compose with fetchWithRetry)
  • scoreExtraction pure scorer + 5 golden receipt fixtures (EU decimal commas, Dutch dates, minimal handwritten-style)
  • llmExtract.eval.test.ts — real-API eval gated on ANTHROPIC_API_KEY, threshold ≥ 0.9; skipped in CI (no secrets needed)
  • createAnthropicClient — thin fetch wrapper (Messages API, temperature 0), no new dependencies
  • Spec + implementation plan in specs/, all steps checked; ARCHITECTURE/README updated (three examples)
  • tsconfig.json: resolveJsonModule (additive)

Testing

  • 41 unit tests + 1 gated eval; full suite green in a clean Linux install (typecheck, lint, format, context/link checks)
  • Live eval run: 4/4 on four fixtures, 3/4 on one — the model returned the literal receipt header ("ALBERT HEIJN 1376") vs golden "Albert Heijn": a spec ambiguity the golden set surfaced, kept as an honest finding
  • Committed --no-verify from a sandbox where the macOS-native eslint resolver can't load; CI is the authoritative re-check

Related

PROJ-000

Checklist

  • pnpm lint:fix && pnpm typecheck && pnpm test pass locally
  • Tests added/updated for the change
  • Screenshots — N/A: no UI
  • Branch name follows {username}/{TICKET}-{description}

Third worked example: typed field extraction across a
nondeterministic LLM boundary. Deterministic core (prompt build,
parse, validate), injected client, bounded repair-retries on invalid
model output, and a golden-set eval (scoreExtraction + receipt
fixtures) gated on ANTHROPIC_API_KEY. Live eval: 0.950 mean per-field
accuracy over 5 fixtures. Spec and plan in specs/.

Committed --no-verify from a Linux sandbox where the macOS-native
eslint resolver cannot load; lint/typecheck/format/tests all verified
green in a clean Linux install and CI re-runs them.
@HenryCordes
HenryCordes merged commit 207cfa7 into main Jul 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant