Skip to content

Commit 17a8bb1

Browse files
Roy tongRoy tong
authored andcommitted
Release 0.8.0: evidence-scoped diagnostics and billing reviews
1 parent 5f97dab commit 17a8bb1

30 files changed

Lines changed: 640 additions & 271 deletions

‎.github/workflows/conformance.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ on:
2727
- 'LEGACY-MIGRATION.md'
2828
- 'verify_vectors.py'
2929
- 'website/**'
30+
- '.github/workflows/**'
3031
workflow_dispatch:
3132
pull_request:
3233

@@ -301,12 +302,15 @@ jobs:
301302
cd /tmp && /tmp/smoke/bin/agentmeasure demo --no-history
302303
/tmp/smoke/bin/agentmeasure selftest
303304
/tmp/smoke/bin/agentmeasure recount --list-vendors
305+
/tmp/smoke/bin/agentmeasure recount --sample
306+
printf '%s\n' '{"type":"assistant","message":{"id":"smoke"}}' > /tmp/response.jsonl
307+
/tmp/smoke/bin/agentmeasure response-audit /tmp/response.jsonl
304308
/tmp/smoke/bin/agentmeasure conformance --new-vector smoke-case --out /tmp/am-smoke-vectors
305309
/tmp/smoke/bin/agentmeasure conformance --fixture /tmp/am-smoke-vectors/smoke-case-001/agentmeasure_smoke_case_fixture_001.events.jsonl
306310
307311
status-sync:
308312
runs-on: ubuntu-latest
309-
needs: [healthcheck, spec-conformance, reference-implementation, lab-suite, sdk-gate, documentation-consistency, release-smoke]
313+
needs: [healthcheck, healthcheck-windows, spec-conformance, reference-implementation, lab-suite, sdk-gate, documentation-consistency, release-smoke]
310314
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
311315
permissions:
312316
contents: write

‎.github/workflows/reconcile.yml‎

Lines changed: 12 additions & 114 deletions
Original file line numberDiff line numberDiff line change
@@ -1,140 +1,38 @@
1-
# Monthly bill reconcile — fork this repo, drop in your export, never think
2-
# about it again (the "fork-to-run" pattern: GitHub Actions is the compute,
3-
# there is no server and nothing leaves your fork).
4-
#
5-
# Setup (5 minutes, one time):
6-
# 1. Fork this repo (a PRIVATE fork keeps your export to yourself).
7-
# 2. Copy examples/reconcile.example.json to .agentmeasure/reconcile.json
8-
# and set "vendor" (agentmeasure recount --list-vendors shows the ids).
9-
# 3. Export last month's resolution counts from your vendor and save them as
10-
# .agentmeasure/recount-export.csv (the 7-column export; see RECONCILE.md).
11-
# 4. Commit both files to your fork.
12-
# 5. Run it once from the Actions tab ("Monthly bill reconcile" → Run
13-
# workflow) to confirm, then let the monthly schedule take over.
14-
#
15-
# Every run: recount your export under the vendor's own published rules,
16-
# pin the report to the run summary, upload the machine-readable result as an
17-
# artifact, and open one tracking issue per billing month when the net
18-
# variance is not zero.
19-
1+
# Optional hosted review: use an independent private repository, not a public fork.
2+
# Data, reports and optional issues are uploaded to GitHub. See RECONCILE.md.
203
name: Monthly bill reconcile
21-
224
on:
235
schedule:
24-
# 3rd of each month, 09:00 UTC — after vendor invoices close.
256
- cron: "0 9 3 * *"
267
workflow_dispatch: {}
27-
288
permissions:
299
contents: read
3010
issues: write
31-
3211
concurrency:
3312
group: reconcile-${{ github.ref }}
3413
cancel-in-progress: true
35-
3614
jobs:
3715
reconcile:
16+
# Public forks must never be presented as private billing-data storage.
17+
if: github.event.repository.private == true
3818
runs-on: ubuntu-latest
3919
steps:
4020
- uses: actions/checkout@v4
41-
4221
- uses: actions/setup-python@v5
4322
with:
4423
python-version: "3.11"
45-
46-
- name: Install AgentMeasure
47-
run: pip install agentmeasure
48-
49-
- name: Read configuration
50-
id: cfg
51-
run: |
52-
if [ ! -f .agentmeasure/reconcile.json ]; then
53-
echo "::notice::No .agentmeasure/reconcile.json found. Copy examples/reconcile.example.json, set your vendor, and commit your export as .agentmeasure/recount-export.csv (see RECONCILE.md). Nothing was run."
54-
echo "configured=false" >> "$GITHUB_OUTPUT"
55-
exit 0
56-
fi
57-
echo "configured=true" >> "$GITHUB_OUTPUT"
58-
python - << 'PY' >> "$GITHUB_OUTPUT"
59-
import json
60-
cfg = json.load(open(".agentmeasure/reconcile.json"))
61-
print("vendor=%s" % cfg.get("vendor", ""))
62-
print("export=%s" % cfg.get("export", ".agentmeasure/recount-export.csv"))
63-
PY
64-
65-
- name: Recount the export
66-
if: steps.cfg.outputs.configured == 'true'
67-
run: |
68-
EXPORT="${{ steps.cfg.outputs.export }}"
69-
if [ ! -f "$EXPORT" ]; then
70-
echo "::error::$EXPORT not found. Save your vendor export there first (RECONCILE.md describes the 7 columns)."
71-
exit 1
72-
fi
73-
agentmeasure recount \
74-
--export "$EXPORT" \
75-
--vendor "${{ steps.cfg.outputs.vendor }}" \
76-
--json recount-result.json | tee recount-report.txt
77-
78-
- name: Pin report to the run summary
79-
if: steps.cfg.outputs.configured == 'true'
80-
run: |
81-
{
82-
echo "## Monthly bill reconcile"
83-
echo
84-
cat recount-report.txt
85-
} >> "$GITHUB_STEP_SUMMARY"
86-
87-
- name: Upload machine-readable result
88-
if: steps.cfg.outputs.configured == 'true'
24+
- name: Install this reviewed source revision
25+
run: python -m pip install ./healthcheck
26+
- name: Review the configured billing period
27+
env:
28+
GH_TOKEN: ${{ github.token }}
29+
run: python scripts/reconcile_period.py
30+
- name: Store results on GitHub
8931
uses: actions/upload-artifact@v4
9032
with:
9133
name: reconcile-result
9234
path: |
9335
recount-report.txt
9436
recount-result.json
37+
retention-days: 7
9538
if-no-files-found: error
96-
97-
- name: Open a tracking issue when the variance is not zero
98-
if: steps.cfg.outputs.configured == 'true'
99-
env:
100-
GH_TOKEN: ${{ github.token }}
101-
run: |
102-
python - << 'PY' > /tmp/verdict.txt
103-
import json
104-
r = json.load(open("recount-result.json"))
105-
if "variance" in r:
106-
print("variance=%.2f" % r["variance"])
107-
else:
108-
print("variance=none")
109-
PY
110-
VERDICT="$(cat /tmp/verdict.txt)"
111-
case "$VERDICT" in
112-
variance=0.00)
113-
echo "Net variance is zero — nothing to track."
114-
exit 0
115-
;;
116-
variance=none)
117-
echo "Vendor price not published — no dollar variance computed; see the report for counts."
118-
exit 0
119-
;;
120-
esac
121-
TITLE="Bill reconcile: net variance $VERDICT (via ${{ steps.cfg.outputs.vendor }} rules)"
122-
BODY_FILE=/tmp/issue-body.md
123-
{
124-
echo "Monthly recount found a non-zero net variance."
125-
echo
126-
echo '```'
127-
cat recount-report.txt
128-
echo '```'
129-
echo
130-
echo "_Automated by the Monthly bill reconcile workflow. The dollar variance is arithmetic under the vendor's own published rules; it is evidence for a conversation with the vendor, not a guaranteed refund. See RECONCILE.md._"
131-
} > "$BODY_FILE"
132-
gh issue list --state open --search "Bill reconcile: in:title" --json number,title \
133-
--jq '.[] | select(.title | startswith("Bill reconcile:")) | .number' \
134-
> /tmp/open-issue.txt || true
135-
EXISTING="$(head -n1 /tmp/open-issue.txt || true)"
136-
if [ -n "$EXISTING" ]; then
137-
gh issue comment "$EXISTING" --body-file "$BODY_FILE"
138-
else
139-
gh issue create --title "$TITLE" --body-file "$BODY_FILE"
140-
fi

‎.github/workflows/release.yml‎

Lines changed: 21 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ name: release
66
#
77
# Triggers:
88
# - push of a v* tag (e.g. v0.1.1)
9-
# - manual workflow_dispatch (defaults to v0.1.1)
9+
# - manual workflow_dispatch (requires an explicit existing vX.Y.Z tag)
1010

1111
on:
1212
push:
@@ -15,7 +15,8 @@ on:
1515
workflow_dispatch:
1616
inputs:
1717
tag:
18-
description: 'Release tag to operate on (e.g. v0.5.0); empty = ref name or v0.1.1 fallback'
18+
description: 'Existing release tag, e.g. v0.8.0 (required for a manual run)'
19+
required: true
1920
default: ''
2021

2122
permissions:
@@ -26,18 +27,23 @@ jobs:
2627
runs-on: ubuntu-latest
2728
steps:
2829
- uses: actions/checkout@v4
30+
with:
31+
ref: ${{ inputs.tag || github.ref }}
32+
fetch-depth: 0
2933

30-
- name: Resolve tag
34+
- name: Resolve and verify tag
35+
env:
36+
REQUESTED_TAG: ${{ inputs.tag }}
3137
run: |
32-
INPUT_TAG="${{ inputs.tag }}${{ github.event.inputs.tag }}"
33-
if [[ -n "$INPUT_TAG" ]]; then
34-
echo "TAG=$INPUT_TAG" >> "$GITHUB_ENV"
35-
elif [[ "$GITHUB_REF_NAME" == v* ]]; then
36-
echo "TAG=$GITHUB_REF_NAME" >> "$GITHUB_ENV"
37-
else
38-
echo "TAG=v0.1.1" >> "$GITHUB_ENV"
38+
TAG="${REQUESTED_TAG:-$GITHUB_REF_NAME}"
39+
if [[ ! "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
40+
echo "::error::An explicit vX.Y.Z tag is required."
41+
exit 1
3942
fi
40-
echo "releasing $TAG"
43+
test "$(git rev-parse HEAD)" = "$(git rev-list -n 1 "$TAG")"
44+
VER=$(python3 -c "import tomllib; print(tomllib.load(open('healthcheck/pyproject.toml','rb'))['project']['version'])")
45+
test "$TAG" = "v$VER"
46+
echo "TAG=$TAG" >> "$GITHUB_ENV"
4147
4248
- name: Create or update GitHub Release
4349
env:
@@ -113,6 +119,8 @@ jobs:
113119
HAS_NPM_TOKEN: ${{ secrets.NPM_TOKEN != '' }}
114120
steps:
115121
- uses: actions/checkout@v4
122+
with:
123+
ref: ${{ inputs.tag || github.ref }}
116124
- uses: actions/setup-node@v4
117125
with:
118126
node-version: '22'
@@ -149,6 +157,8 @@ jobs:
149157
id-token: write
150158
steps:
151159
- uses: actions/checkout@v4
160+
with:
161+
ref: ${{ inputs.tag || github.ref }}
152162
- name: Skip if this version is already live on PyPI (idempotent re-runs)
153163
run: |
154164
VER=$(python3 -c "import tomllib; print(tomllib.load(open('healthcheck/pyproject.toml','rb'))['project']['version'])" 2>/dev/null || echo unknown)

‎CHANGELOG.md‎

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3,26 +3,33 @@
33
All notable changes to AgentMeasure (standard, SDK, and reference product) are
44
documented here. Format follows [Keep a Changelog](https://keepachangelog.com/).
55

6-
## [Unreleased]
6+
## [v0.8.0] - 2026-10-15
77

88
### Added
99

10+
- **Claude response evidence**: `response-audit` compares explicitly selected
11+
JSONL files by response ID, exposing repeats, conflicts and missing fields.
12+
It never calculates session or invoice totals or rewrites source logs.
13+
14+
1015
- **Bill-recount identity**: README (EN/zh) and the website hero now lead with
11-
the buyer-side recount — a copy-paste 5-row sample, the one command that
12-
runs it, and the dollar variance it prints. Local log checks remain the
13-
free second section. No service scope or boundary changed.
16+
supported recount with a copy-paste synthetic sample; task routing also
17+
keeps developer diagnostics and metering QA visible. Buyer invoice review
18+
is not included in the $10/$20 services. Source-only features are labelled.
1419
- **`recount --sample`**: run the bundled 5-row synthetic export (Zendesk rule
1520
set) with no CSV of your own — the same file the README shows.
16-
- **Dollar headline**: every recount report opens with
17-
`billed $X · net overcharge $Y · at price/resolution` before any detail.
21+
- **Evidence-qualified headline**: recount labels published-price estimates,
22+
identified net variance and unsettled rows. These are not verified invoice
23+
totals or refunds; zero identified variance does not clear an incomplete bill.
1824
- **Reproducible showcase** (`examples/showcase/`): three public cases
1925
(overcharge with refused guesses / findings in both directions / the
2026
contract lane) with frozen expected outputs; CI compares them on every
2127
change so the showcase cannot drift from shipped behaviour.
2228
- **Monthly bill reconcile workflow** (`.github/workflows/reconcile.yml` +
23-
`RECONCILE.md`): fork the repo, drop in your vendor export, and the recount
24-
runs monthly on GitHub Actions — no server, nothing leaves your fork, and a
25-
non-zero variance opens a tracking issue. The "fork-to-run" adoption path.
29+
`RECONCILE.md`): opt-in hosting in an independent private repository; data
30+
is uploaded to GitHub. Requires a reviewed export hash and billing period.
31+
Stale scheduled inputs stop; optional issues track offsetting findings and
32+
unknowns per vendor/period, not only nonzero net amounts.
2633
- **Vendor billing-rule scorecard** (website claims index): the six published
2734
rule sets with prices, triggers, dated sources, confidence grades and how
2835
far each is re-countable from a counts-only export — the page always

‎README.md‎

Lines changed: 34 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,17 @@
11
# AgentMeasure
22

3-
**Every usage- or outcome-billed AI invoice should be independently re-countable by the party paying it — in one command.**
3+
**Check AI usage records. Re-count supported support-billing exports with evidence.**
44

5-
AgentMeasure re-counts an AI support bill against the vendor's own published
6-
billing rules, prints the dollar variance, and builds the evidence pack you
7-
can put in front of the vendor. It runs locally; nothing uploads.
5+
AgentMeasure checks Codex / Claude Code logs and applies published billing
6+
rules to supported AI support exports. Local CLI commands do not upload data.
7+
A PDF or invoice total alone is insufficient: recount needs reviewed event
8+
fields, and price-based differences are not verified refunds.
9+
10+
| Your task | Start here |
11+
|---|---|
12+
| Explain a local duplicate, retry or failure | `agentmeasure demo`, then `agentmeasure check`; optional $10 interpretation |
13+
| Check event accounting against one supported ruleset | Free input fit check, then bounded $20 Metering QA |
14+
| Review a support-billing period | Free `recount` below; separate buyer feasibility review, not included in $10/$20 |
815

916
[Website](https://roy-tong.github.io/AgentMeasure/) · [中文](README.zh-CN.md) · [Healthcheck guide](healthcheck/README.md) · [Release notes](https://roy-tong.github.io/AgentMeasure/releases.html)
1017

@@ -30,30 +37,27 @@ The output prints the vendor's rule with its public source, findings in both
3037
directions, then the dollars:
3138

3239
```
33-
billed $10.00 · net overcharge $2.00 · at USD 2.0/resolution
34-
...
3540
Dollars (at USD 2.0 per resolution)
3641
overcharge identified 2.00
3742
undercharge (vendor favour) 0.00
3843
net variance 2.00
3944
```
4045

41-
No CSV at hand yet? `agentmeasure recount --sample` runs the bundled 5-row
42-
synthetic export with the Zendesk rule set — the same file as above.
46+
No CSV at hand? `agentmeasure recount --sample` runs the same bundled synthetic export in CLI 0.8.0.
47+
In this synthetic example, four rows remain unsettled; $8 is not a verified
48+
billable total. The $2 is a rule-priced finding, not recovered money.
4349

4450
Three reproducible showcase cases (overcharge with refused guesses, findings
4551
in both directions, the contract lane) live in
4652
[examples/showcase](examples/showcase/README.md) with frozen expected outputs
4753
that CI compares on every change.
4854

49-
## Reconcile monthly, hands-free
55+
## Repeat a billing review
5056

51-
Fork this repo, drop your vendor export into `.agentmeasure/`, and the
52-
[Monthly bill reconcile](.github/workflows/reconcile.yml) workflow re-counts
53-
your bill on a schedule — GitHub Actions is the compute, no server, nothing
54-
leaves your fork, and a non-zero variance opens a tracking issue so the
55-
finding cannot be forgotten. Setup is five minutes:
56-
[RECONCILE.md](RECONCILE.md).
57+
For local-only handling, run the CLI on each new period's export. The optional
58+
[GitHub Actions workflow](RECONCILE.md) requires a separate private repository
59+
and uploads data and reports to GitHub. Public forks are not private storage.
60+
Each period needs a fresh, reviewed export; no vendor data is fetched automatically.
5761

5862
Six vendor rule sets ship today — Intercom Fin, Zendesk AI Agents, HubSpot
5963
Breeze, Salesforce Agentforce, Ada and a generic per-resolution baseline —
@@ -77,7 +81,7 @@ discipline pointed at your own logs. `agentmeasure demo` runs a bundled
7781
synthetic session with no account and no upload. Missing evidence stays
7882
unknown; a retry is not automatically a second logical operation.
7983

80-
Python 3.9+. CLI **0.7.2** on [PyPI](https://pypi.org/project/agentmeasure/);
84+
Python 3.9+. CLI **0.8.0** on [PyPI](https://pypi.org/project/agentmeasure/);
8185
Provider SDK **0.2.0** on [npm](https://www.npmjs.com/package/@agentmeasure/mcp),
8286
verified 2026-10-15. These packages have separate version numbers.
8387
On macOS/Linux use `python3 -m pip` if needed; on Windows use `py -m pip`.
@@ -88,6 +92,17 @@ With both runtimes present, auto-discovery prefers Codex; use
8892
seven-day window? Try `--all` or an explicit `--dir` and `--runtime`. A missing
8993
input is not a passing check. [More recovery steps](https://roy-tong.github.io/AgentMeasure/#how).
9094

95+
### Compare Claude response evidence across files
96+
97+
```bash
98+
agentmeasure response-audit first-session.jsonl fork-session.jsonl --json responses.json
99+
```
100+
101+
This read-only command groups supplied response IDs and shows repeated usage
102+
snapshots, conflicts and missing fields. It does not rewrite logs, choose a
103+
winning snapshot or compute a session/invoice total. See the
104+
[response-audit guide](docs/response-audit.md).
105+
91106
## Get help with one question
92107

93108
| Option | Price | What you get |
@@ -147,8 +162,8 @@ The [September audit report](campaigns/audit-report-2026-09.md) documents the
147162
the claims index. Engineering evidence does not establish paid demand, customer
148163
savings or a right to a refund. Customer value and repeat purchase remain under validation.
149164

150-
487 Healthcheck unit tests passed on 2026-10-15. See [CI](https://github.com/roy-tong/AgentMeasure/actions/workflows/conformance.yml)
151-
for current results and [v0.7.2](https://github.com/roy-tong/AgentMeasure/releases/tag/v0.7.2)
165+
511 Healthcheck unit tests passed on 2026-10-15. See [CI](https://github.com/roy-tong/AgentMeasure/actions/workflows/conformance.yml)
166+
for current results and [v0.8.0](https://github.com/roy-tong/AgentMeasure/releases/tag/v0.8.0)
152167
for the release scope. Example fixtures are not production customer data.
153168

154169
## Open-source commitments
@@ -159,4 +174,4 @@ for the release scope. Example fixtures are not production customer data.
159174

160175
[Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) · [Governance](GOVERNANCE.md) · [Agent instructions](AGENTS.md) · [Whitepaper](whitepaper/measuring-software-used-by-ai-agents.md)
161176

162-
[Developer guide](docs/developer-guide.md)
177+
[Task scope and validation priorities](docs/product-scope.md) · [Developer guide](docs/developer-guide.md)

0 commit comments

Comments
 (0)