Skip to content

Commit 5b2761c

Browse files
Roy tongRoy tong
Roy tong
authored and
Roy tong
committed
Make the buyer-side recount reproducible: sample, showcase, monthly reconcile
- recount --sample runs the bundled 5-row synthetic export (README shows the same file); every recount report opens with a dollar headline - examples/showcase: three public cases with frozen expected outputs, guarded by CI so the showcase cannot drift from shipped behaviour - Monthly bill reconcile workflow (fork-to-run): GitHub Actions re-counts a committed vendor export on a schedule; non-zero variance opens a tracking issue; nothing leaves the fork (RECONCILE.md) - claims index gains a vendor billing-rule scorecard: price, trigger, dated source, confidence grade, counts-only re-countability per vendor - 490 tests green (3 new showcase freeze tests); v0.8.0 notes staged, tag follows the coordinated release window
1 parent 9832f68 commit 5b2761c

21 files changed

Lines changed: 578 additions & 1 deletion

‎.github/workflows/reconcile.yml‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Monthly bill reconcile — fork this repo, drop in your export, never think
2+
# about it again (the "fork-to-run" pattern: GitHub Actions is the compute,
3+
# there is no server and nothing leaves your fork).
4+
#
5+
# Setup (5 minutes, one time):
6+
# 1. Fork this repo (a PRIVATE fork keeps your export to yourself).
7+
# 2. Copy examples/reconcile.example.json to .agentmeasure/reconcile.json
8+
# and set "vendor" (agentmeasure recount --list-vendors shows the ids).
9+
# 3. Export last month's resolution counts from your vendor and save them as
10+
# .agentmeasure/recount-export.csv (the 7-column export; see RECONCILE.md).
11+
# 4. Commit both files to your fork.
12+
# 5. Run it once from the Actions tab ("Monthly bill reconcile" → Run
13+
# workflow) to confirm, then let the monthly schedule take over.
14+
#
15+
# Every run: recount your export under the vendor's own published rules,
16+
# pin the report to the run summary, upload the machine-readable result as an
17+
# artifact, and open one tracking issue per billing month when the net
18+
# variance is not zero.
19+
20+
name: Monthly bill reconcile
21+
22+
on:
23+
schedule:
24+
# 3rd of each month, 09:00 UTC — after vendor invoices close.
25+
- cron: "0 9 3 * *"
26+
workflow_dispatch: {}
27+
28+
permissions:
29+
contents: read
30+
issues: write
31+
32+
concurrency:
33+
group: reconcile-${{ github.ref }}
34+
cancel-in-progress: true
35+
36+
jobs:
37+
reconcile:
38+
runs-on: ubuntu-latest
39+
steps:
40+
- uses: actions/checkout@v4
41+
42+
- uses: actions/setup-python@v5
43+
with:
44+
python-version: "3.11"
45+
46+
- name: Install AgentMeasure
47+
run: pip install agentmeasure
48+
49+
- name: Read configuration
50+
id: cfg
51+
run: |
52+
if [ ! -f .agentmeasure/reconcile.json ]; then
53+
echo "::notice::No .agentmeasure/reconcile.json found. Copy examples/reconcile.example.json, set your vendor, and commit your export as .agentmeasure/recount-export.csv (see RECONCILE.md). Nothing was run."
54+
echo "configured=false" >> "$GITHUB_OUTPUT"
55+
exit 0
56+
fi
57+
echo "configured=true" >> "$GITHUB_OUTPUT"
58+
python - << 'PY' >> "$GITHUB_OUTPUT"
59+
import json
60+
cfg = json.load(open(".agentmeasure/reconcile.json"))
61+
print("vendor=%s" % cfg.get("vendor", ""))
62+
print("export=%s" % cfg.get("export", ".agentmeasure/recount-export.csv"))
63+
PY
64+
65+
- name: Recount the export
66+
if: steps.cfg.outputs.configured == 'true'
67+
run: |
68+
EXPORT="${{ steps.cfg.outputs.export }}"
69+
if [ ! -f "$EXPORT" ]; then
70+
echo "::error::$EXPORT not found. Save your vendor export there first (RECONCILE.md describes the 7 columns)."
71+
exit 1
72+
fi
73+
agentmeasure recount \
74+
--export "$EXPORT" \
75+
--vendor "${{ steps.cfg.outputs.vendor }}" \
76+
--json recount-result.json | tee recount-report.txt
77+
78+
- name: Pin report to the run summary
79+
if: steps.cfg.outputs.configured == 'true'
80+
run: |
81+
{
82+
echo "## Monthly bill reconcile"
83+
echo
84+
cat recount-report.txt
85+
} >> "$GITHUB_STEP_SUMMARY"
86+
87+
- name: Upload machine-readable result
88+
if: steps.cfg.outputs.configured == 'true'
89+
uses: actions/upload-artifact@v4
90+
with:
91+
name: reconcile-result
92+
path: |
93+
recount-report.txt
94+
recount-result.json
95+
if-no-files-found: error
96+
97+
- name: Open a tracking issue when the variance is not zero
98+
if: steps.cfg.outputs.configured == 'true'
99+
env:
100+
GH_TOKEN: ${{ github.token }}
101+
run: |
102+
python - << 'PY' > /tmp/verdict.txt
103+
import json
104+
r = json.load(open("recount-result.json"))
105+
if "variance" in r:
106+
print("variance=%.2f" % r["variance"])
107+
else:
108+
print("variance=none")
109+
PY
110+
VERDICT="$(cat /tmp/verdict.txt)"
111+
case "$VERDICT" in
112+
variance=0.00)
113+
echo "Net variance is zero — nothing to track."
114+
exit 0
115+
;;
116+
variance=none)
117+
echo "Vendor price not published — no dollar variance computed; see the report for counts."
118+
exit 0
119+
;;
120+
esac
121+
TITLE="Bill reconcile: net variance $VERDICT (via ${{ steps.cfg.outputs.vendor }} rules)"
122+
BODY_FILE=/tmp/issue-body.md
123+
{
124+
echo "Monthly recount found a non-zero net variance."
125+
echo
126+
echo '```'
127+
cat recount-report.txt
128+
echo '```'
129+
echo
130+
echo "_Automated by the Monthly bill reconcile workflow. The dollar variance is arithmetic under the vendor's own published rules; it is evidence for a conversation with the vendor, not a guaranteed refund. See RECONCILE.md._"
131+
} > "$BODY_FILE"
132+
gh issue list --state open --search "Bill reconcile: in:title" --json number,title \
133+
--jq '.[] | select(.title | startswith("Bill reconcile:")) | .number' \
134+
> /tmp/open-issue.txt || true
135+
EXISTING="$(head -n1 /tmp/open-issue.txt || true)"
136+
if [ -n "$EXISTING" ]; then
137+
gh issue comment "$EXISTING" --body-file "$BODY_FILE"
138+
else
139+
gh issue create --title "$TITLE" --body-file "$BODY_FILE"
140+
fi

‎CHANGELOG.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,31 @@
33
All notable changes to AgentMeasure (standard, SDK, and reference product) are
44
documented here. Format follows [Keep a Changelog](https://keepachangelog.com/).
55

6+
## [Unreleased]
7+
8+
### Added
9+
10+
- **Bill-recount identity**: README (EN/zh) and the website hero now lead with
11+
the buyer-side recount — a copy-paste 5-row sample, the one command that
12+
runs it, and the dollar variance it prints. Local log checks remain the
13+
free second section. No service scope or boundary changed.
14+
- **`recount --sample`**: run the bundled 5-row synthetic export (Zendesk rule
15+
set) with no CSV of your own — the same file the README shows.
16+
- **Dollar headline**: every recount report opens with
17+
`billed $X · net overcharge $Y · at price/resolution` before any detail.
18+
- **Reproducible showcase** (`examples/showcase/`): three public cases
19+
(overcharge with refused guesses / findings in both directions / the
20+
contract lane) with frozen expected outputs; CI compares them on every
21+
change so the showcase cannot drift from shipped behaviour.
22+
- **Monthly bill reconcile workflow** (`.github/workflows/reconcile.yml` +
23+
`RECONCILE.md`): fork the repo, drop in your vendor export, and the recount
24+
runs monthly on GitHub Actions — no server, nothing leaves your fork, and a
25+
non-zero variance opens a tracking issue. The "fork-to-run" adoption path.
26+
- **Vendor billing-rule scorecard** (website claims index): the six published
27+
rule sets with prices, triggers, dated sources, confidence grades and how
28+
far each is re-countable from a counts-only export — the page always
29+
matches the shipped registry.
30+
631
## [v0.7.2] - 2026-10-15
732

833
- Add `inspect-input`: read-only JSONL format routing, explicit NOT-RUN validation/verdict, invalid-row counts and safe next steps.

‎README.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,12 +30,31 @@ The output prints the vendor's rule with its public source, findings in both
3030
directions, then the dollars:
3131

3232
```
33+
billed $10.00 · net overcharge $2.00 · at USD 2.0/resolution
34+
...
3335
Dollars (at USD 2.0 per resolution)
3436
overcharge identified 2.00
3537
undercharge (vendor favour) 0.00
3638
net variance 2.00
3739
```
3840

41+
No CSV at hand yet? `agentmeasure recount --sample` runs the bundled 5-row
42+
synthetic export with the Zendesk rule set — the same file as above.
43+
44+
Three reproducible showcase cases (overcharge with refused guesses, findings
45+
in both directions, the contract lane) live in
46+
[examples/showcase](examples/showcase/README.md) with frozen expected outputs
47+
that CI compares on every change.
48+
49+
## Reconcile monthly, hands-free
50+
51+
Fork this repo, drop your vendor export into `.agentmeasure/`, and the
52+
[Monthly bill reconcile](.github/workflows/reconcile.yml) workflow re-counts
53+
your bill on a schedule — GitHub Actions is the compute, no server, nothing
54+
leaves your fork, and a non-zero variance opens a tracking issue so the
55+
finding cannot be forgotten. Setup is five minutes:
56+
[RECONCILE.md](RECONCILE.md).
57+
3958
Six vendor rule sets ship today — Intercom Fin, Zendesk AI Agents, HubSpot
4059
Breeze, Salesforce Agentforce, Ada and a generic per-resolution baseline —
4160
each citing its source and confidence grade

‎README.zh-CN.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,29 @@ agentmeasure recount --export export.csv --vendor zendesk
2828
输出先是厂商规则及其公开出处，然后是双向核对结果，最后是钱数：
2929

3030
```
31+
billed $10.00 · net overcharge $2.00 · at USD 2.0/resolution
32+
...
3133
Dollars (at USD 2.0 per resolution)
3234
overcharge identified 2.00
3335
undercharge (vendor favour) 0.00
3436
net variance 2.00
3537
```
3638

39+
手头没有 CSV？`agentmeasure recount --sample` 直接跑内置的 5 行合成导出
40+
（Zendesk 规则）——与上面的文件完全相同。
41+
42+
三个可复现 showcase 案例（拒绝猜测的超收判定 / 双向核对 / 合同航道）在
43+
[examples/showcase](examples/showcase/README.md)，期望输出已冻结、CI 每次
44+
变更都会比对，展示与现实不会悄悄分叉。
45+
46+
## 每月自动对账，无需值守
47+
48+
Fork 本仓库、把厂商导出放进 `.agentmeasure/`，
49+
[Monthly bill reconcile](.github/workflows/reconcile.yml) 工作流就会按月
50+
自动复算你的账单——GitHub Actions 当算力，无服务器，数据不出你的 fork，
51+
差额非零时自动开一个跟踪 issue，让发现不会被遗忘。配置五分钟：
52+
[RECONCILE.md](RECONCILE.md)。
53+
3754
今天内置六家厂商规则——Intercom Fin、Zendesk AI Agents、HubSpot Breeze、
3855
Salesforce Agentforce、Ada，以及一个通用按结果基线——每条规则都注明出处
3956
与置信级（`agentmeasure recount --list-vendors`）。复算与账单对不上时：

‎RECONCILE.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Reconcile your vendor's bill every month, hands-free
2+
3+
This repo ships a GitHub Actions workflow (`Monthly bill reconcile`) that
4+
re-counts an AI support bill under the vendor's own published rules on a
5+
monthly schedule. Fork the repo, drop in your export, and the recount happens
6+
by itself: GitHub Actions is the compute, there is no server, and nothing
7+
leaves your fork.
8+
9+
## Setup (5 minutes, one time)
10+
11+
1. **Fork this repo.** A private fork keeps your export file to yourself.
12+
2. **Copy the config template**:
13+
- `examples/reconcile.example.json` → `.agentmeasure/reconcile.json`
14+
- set `"vendor"` to your vendor's id — `agentmeasure recount --list-vendors`
15+
prints every id with its rule source (intercom, zendesk, hubspot, ada,
16+
salesforce, generic).
17+
3. **Save your export** as `.agentmeasure/recount-export.csv` — the vendor's
18+
resolution-counts export with these columns (5 required, 2 optional):
19+
20+
```
21+
conversation_id, human_agent_participated, issue_addressed,
22+
customer_recontacted_within_window[, customer_recontacted_at, closed_at],
23+
vendor_billed
24+
```
25+
26+
`agentmeasure prepare --export <native export> --vendor <id>` maps a native
27+
Intercom or Zendesk export onto these columns; judgement columns stay empty
28+
for a human to review — the tool never invents a value.
29+
4. **Commit both files** to your fork.
30+
5. **Run it once manually**: your fork → Actions → *Monthly bill reconcile* →
31+
*Run workflow*. Confirm the report looks right, then let the schedule take
32+
over (3rd of each month, 09:00 UTC).
33+
34+
## What each run does
35+
36+
- Re-counts your export under the vendor's own published billing rules.
37+
- Pins the full report to the run summary and uploads the machine-readable
38+
result (`recount-result.json`) as an artifact.
39+
- When the net variance is not zero, opens (or updates) one tracking issue
40+
per state, so the finding cannot be forgotten.
41+
42+
## What it deliberately does not do
43+
44+
- It does not guess. Rows without enough evidence land in `cannot settle` and
45+
are removed from the claim, never zeroed.
46+
- It does not call the vendor, request a refund, or assert wrongdoing. The
47+
variance is arithmetic under the vendor's own published rules — evidence for
48+
a conversation with the vendor, not a guaranteed refund. When you want the
49+
negotiable pack, `agentmeasure dispute` builds it.
50+
- It does not upload anything anywhere. The report and the artifact live in
51+
your fork only.
52+
53+
## Cost
54+
55+
Zero. The workflow installs the CLI from PyPI on GitHub's runners and exits in
56+
under a minute; the free tier's monthly minutes cover it comfortably.

‎docs/RELEASE-v0.8.0.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# v0.8.0 — Your AI bill, re-countable in one command
2+
3+
This release makes the buyer-side recount the product's front door.
4+
5+
`agentmeasure recount --sample` runs the bundled 5-row synthetic export under the Zendesk rule set with zero preparation — the README shows the same file, the command prints the same report. Every recount report now opens with a dollar headline (`billed $10.00 · net overcharge $2.00 · at USD 2.0/resolution`) before any detail.
6+
7+
Three reproducible showcase cases ship in `examples/showcase/` — an overcharge that refuses to guess, findings in both directions, and the contract lane as renewal leverage — each with a frozen expected output that CI compares on every change, so the showcase cannot drift from shipped behaviour.
8+
9+
Fork this repo, drop your vendor export into `.agentmeasure/`, and the new Monthly bill reconcile workflow re-counts your bill every month on GitHub Actions: no server, nothing leaves your fork, and a non-zero variance opens a tracking issue so the finding cannot be forgotten. Setup is five minutes in RECONCILE.md; six vendor rule sets ship today (intercom, zendesk, hubspot, ada, salesforce, generic), each printed with its public source and confidence grade.
10+
11+
The website claims index gains a vendor billing-rule scorecard: published price, billing trigger, dated rule source and how far each vendor is re-countable from a counts-only export — always matching the shipped registry.
12+
13+
What did not change: the free/paid boundary (the CLI answers IF; paid human services answer WHAT/WHY/NEXT), the evidence discipline (rows without enough evidence are `cannot settle`, removed from the claim and never zeroed), and no promise that a variance is a guaranteed refund.
14+
15+
Upgrade: `python -m pip install --upgrade agentmeasure`.
16+
17+
Validation: full Healthcheck suite plus the new showcase freeze tests; installed-wheel smoke covers `recount --sample`. SDK remains 0.2.0. Research feedback and test results do not establish payment or PMF.

‎examples/reconcile.example.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"vendor": "intercom",
3+
"export": ".agentmeasure/recount-export.csv"
4+
}

‎examples/showcase/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Reproducible bill-recount showcase
2+
3+
Three public cases. Each case is an input CSV (synthetic data), the one command
4+
that runs it, and a frozen expected output. If the tool in your hand produces
5+
a different report than the one frozen here, that is a bug — open an issue.
6+
7+
The pattern is borrowed from projects that earn trust by being reproducible:
8+
the showcase is not a claim, it is a runnable expectation.
9+
10+
## Case A — overcharge, and the rows we refuse to guess
11+
12+
```bash
13+
agentmeasure recount --export case-a-overcharge-zendesk.csv --vendor zendesk
14+
```
15+
16+
One conversation a human agent finished was still billed (`$2.00` overcharge).
17+
Four of five rows land in `cannot settle` because a counts-only export cannot
18+
carry Zendesk's LLM verdict — those rows are removed from the claim, not
19+
zeroed. Read: [expected output](case-a-overcharge-zendesk.expected.txt).
20+
21+
## Case B — findings run both directions
22+
23+
```bash
24+
agentmeasure recount --export case-b-both-directions-intercom.csv --vendor intercom
25+
```
26+
27+
Two overcharged lines (a reopen inside the documented window, a human
28+
takeover), one line the vendor under-billed in your favour. Undercharge is
29+
netted before any dollar is claimed: `$1.98 − $0.99 = $0.99`. Read:
30+
[expected output](case-b-both-directions-intercom.expected.txt).
31+
32+
## Case C — the contract lane is leverage, never netted into the claim
33+
34+
```bash
35+
agentmeasure recount --export case-c-contract-lane-intercom.csv --vendor intercom \
36+
--contract case-c-contract-lane-intercom.contract.json
37+
```
38+
39+
The buyer's contract (72h reopen window, human takeover disqualifies) is a
40+
second lane: its findings are renewal leverage, listed separately and never
41+
netted into the Tier 1 billing claim. Rows without timestamps stay
42+
`unprovable` — a missing recontact time is never read as "no recontact".
43+
Read: [expected output](case-c-contract-lane-intercom.expected.txt).
44+
45+
## Guarantees
46+
47+
- All data is synthetic. No customer data appears here.
48+
- The vendor rule sets and their sources are printed in every report
49+
(`agentmeasure recount --list-vendors` for the full registry).
50+
- The frozen expected outputs are compared by CI on every change
51+
(`healthcheck/tests/test_showcase.py`), so the showcase cannot silently
52+
drift from the shipped behaviour.

‎examples/showcase/case-a-overcharge-zendesk.csv‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
conversation_id,human_agent_participated,issue_addressed,customer_recontacted_within_window,vendor_billed
2+
c-001,N,Y,N,Y
3+
c-002,N,Y,N,Y
4+
c-003,N,Y,N,Y
5+
c-004,Y,Y,N,Y
6+
c-005,N,Y,Y,Y

0 commit comments

Comments
 (0)