Skip to content

Commit fd68386

Browse files
authored
Merge pull request #32 from gunjanjaswal/availability-cert-vector-chenhz01-001
Add availability certification vector (chenhz01-001) for #11
2 parents 67c276b + 85537ce commit fd68386

6 files changed

Lines changed: 709 additions & 0 deletions

‎.github/workflows/conformance.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,9 @@ jobs:
7373
- name: "External fixture conformance (Urusilla-002: issue 10 / AM-U-007 grain guard)"
7474
run: |
7575
python3 conformance/runners/run_external_fixture_002.py
76+
- name: "External fixture conformance (chenhz01-001: issue 11 / availability certification axis)"
77+
run: |
78+
python3 conformance/runners/run_external_fixture_003.py
7679
- name: "Outcome-family conformance (OUT-001..004: definability / traceability / reopen dedup / claim-evidence)"
7780
run: |
7881
python3 conformance/runners/run_outcome_audit.py
Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
#!/usr/bin/env python3
2+
"""External fixture conformance: chenhz01-001 — availability certification (#11).
3+
4+
Source: AgentMeasure issue #11 (Clarify Result Consumption). The two-state
5+
ontology (availability vs influence) and the compaction boundary are
6+
@gunjanjaswal's; the trace-side hash certification is @chenhz01's; the
7+
canonicalized-content + field-path + four-value axis is @roy-tong's. This
8+
fixture is the first availability vector: one synthetic trace whose four tool
9+
results each meet a different certification outcome, so a compaction-aware
10+
implementation can be tested against all four at once.
11+
12+
Certification axis (per result, orthogonal to the State-B influence axis):
13+
verbatim the result's canonicalized content is embedded in the next
14+
model request -> State A certified (observability: TRUE);
15+
summary-embed only a lossy projection reached the request -> weak influence
16+
evidence, NOT availability, its own bucket (FALSE for A);
17+
absent a request left, the result is nowhere in it (FALSE);
18+
unreachable no request left the framework -> UNOBSERVABLE, excluded from
19+
the denominator, never counted FALSE (invariant 17).
20+
21+
Guards:
22+
1. schema — every event validates under FMT-002;
23+
2. core — aggregate() reproduces the naive boolean consumption_rate
24+
(1/4): the collapse the four-value axis corrects;
25+
3. cert — certification recomputed from the raw trace_context (canonical
26+
field-walk) reproduces every event's x_availability and the
27+
expected four-bucket breakdown;
28+
4. reserial — a raw byte/substring certifier MISSES op1 (re-serialized JSON
29+
string) where the canonical field-walk certifies it verbatim;
30+
5. boundary — the forbid list holds: summary-embed is not State A and not
31+
absent; unreachable is excluded, never collapsed to FALSE;
32+
6. collapse — mislabeling unreachable as absent (UNOBSERVABLE -> FALSE)
33+
changes the denominator; the vector must make that detectable.
34+
35+
Claim boundary: synthetic conformance fixture authored for #11 — not an
36+
endorsement, an external reproduction, or a real runtime observation.
37+
38+
Exit 0 = all guards pass.
39+
"""
40+
from __future__ import annotations
41+
42+
import hashlib
43+
import json
44+
import sys
45+
from pathlib import Path
46+
47+
ROOT = Path(__file__).resolve().parents[2]
48+
sys.path.insert(0, str(ROOT / "lab"))
49+
from agentmeasure_lab.analysis import aggregate # noqa: E402
50+
from agentmeasure_lab.schemas import SchemaError, validate # noqa: E402
51+
52+
VEC = ROOT / "conformance" / "vectors" / "external" / "chenhz01-001"
53+
EVENTS_FILE = VEC / "agentmeasure_chenhz01_fixture_001.events.jsonl"
54+
EXPECTED_FILE = VEC / "agentmeasure_chenhz01_fixture_001.expected.json"
55+
CONTEXT_FILE = VEC / "agentmeasure_chenhz01_fixture_001.trace_context.json"
56+
SCHEMA_FILE = ROOT / "lab" / "schemas" / "funnel-event.schema.json"
57+
58+
fails: list = []
59+
60+
61+
def check(name: str, cond: bool, detail: str = "") -> None:
62+
print(f" {'✓' if cond else '✗'} {name}" + (f" — {detail}" if detail and not cond else ""))
63+
if not cond:
64+
fails.append(name)
65+
66+
67+
def canon(o) -> str:
68+
return json.dumps(o, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
69+
70+
71+
def hcontent(o) -> str:
72+
return "sha256:" + hashlib.sha256(canon(o).encode("utf-8")).hexdigest()
73+
74+
75+
def certify_canonical(op: str, ctx: dict):
76+
"""Recompute certification from raw inputs: canonicalize each field value,
77+
compare content hashes, record the field path. Independent of x_availability."""
78+
results = ctx["results"]
79+
req = ctx["next_request"][op]
80+
if not req["emitted"]:
81+
return "unreachable", None
82+
target = hcontent(results[op]["content"])
83+
for i, m in enumerate(req["messages"] or []):
84+
val = m.get("content")
85+
if isinstance(val, str):
86+
try:
87+
parsed = json.loads(val)
88+
except Exception:
89+
parsed = None
90+
if parsed is not None and hcontent(parsed) == target:
91+
return "verbatim", f"messages[{i}].content"
92+
if ctx["embed_form"][op] == "summary":
93+
return "summary-embed", ctx["embedded_at"][op]
94+
return "absent", None
95+
96+
97+
def certify_raw_substring(op: str, ctx: dict) -> str:
98+
"""Naive certifier: substring of the canonical result in the re-serialized
99+
request blob. Breaks on re-serialization — present only to be caught."""
100+
results = ctx["results"]
101+
req = ctx["next_request"][op]
102+
if not req["emitted"]:
103+
return "unreachable"
104+
blob = canon(req)
105+
if canon(results[op]["content"]) in blob:
106+
return "verbatim"
107+
return "summary-embed" if ctx["embed_form"][op] == "summary" else "absent"
108+
109+
110+
def main() -> int:
111+
events = [json.loads(l) for l in EVENTS_FILE.read_text().splitlines() if l.strip()]
112+
expected = json.loads(EXPECTED_FILE.read_text())
113+
ctx = json.loads(CONTEXT_FILE.read_text())
114+
schema = json.loads(SCHEMA_FILE.read_text())
115+
116+
print("chenhz01-001 guard 1: schema validation (FMT-002)")
117+
# Fail loudly on an empty events file, so the per-event loop below can't
118+
# pass vacuously (chenhz01, #32 review).
119+
check("events file is non-empty", len(events) > 0, f"{len(events)} events")
120+
for i, ev in enumerate(events, 1):
121+
try:
122+
validate(ev, schema)
123+
except SchemaError as e:
124+
check(f"event {i} valid", False, str(e))
125+
check(f"{len(events)} events valid under FMT-002", len(events) > 0)
126+
127+
print("chenhz01-001 guard 2: core boolean consumption_rate (the collapse)")
128+
cell = aggregate(events)[("availability-cert",)]
129+
ce = expected["core_expected_metrics"]
130+
check(f"operations == {ce['operations']}", cell["operations"] == ce["operations"], str(cell["operations"]))
131+
cr = cell["consumption_rate"]
132+
check(f"consumption_rate == {ce['consumption_rate']['value']} (= {ce['consumption_rate']['numerator']}/{ce['consumption_rate']['denominator']})",
133+
cr["value"] == ce["consumption_rate"]["value"]
134+
and cr["numerator"] == ce["consumption_rate"]["numerator"]
135+
and cr["denominator"] == ce["consumption_rate"]["denominator"],
136+
f"got {cr['numerator']}/{cr['denominator']}={cr['value']}")
137+
check("operation reconciliation passed", cell["operation_reconciliation"]["status"] == "passed",
138+
cell["operation_reconciliation"]["status"])
139+
140+
print("chenhz01-001 guard 3: certification recomputed from raw trace_context")
141+
consumptions = {e["operation_index"]: e for e in events if e["event"] == "consumption"}
142+
buckets = {"verbatim": 0, "summary-embed": 0, "absent": 0, "unreachable": 0}
143+
for op_str in ("1", "2", "3", "4"):
144+
op = int(op_str)
145+
cert, path = certify_canonical(op_str, ctx)
146+
buckets[cert] += 1
147+
xa = consumptions[op]["x_availability"]
148+
check(f"op{op} certification == x_availability ({cert})", xa["certification"] == cert, xa["certification"])
149+
check(f"op{op} field_path == {path}", xa["embedded_field_path"] == path, str(xa["embedded_field_path"]))
150+
check(f"op{op} content hash matches", xa["result_content_hash"] == hcontent(ctx["results"][op_str]["content"]))
151+
check(f"op{op} consumed boolean == state_a", consumptions[op]["consumed"] == (cert == "verbatim"))
152+
exp_bd = expected["availability_certification"]["breakdown"]
153+
for k, v in buckets.items():
154+
check(f"breakdown[{k}] == {v}", exp_bd[k.replace('-', '_')] == v, str(exp_bd[k.replace('-', '_')]))
155+
156+
print("chenhz01-001 guard 4: re-serialization false negative on op1")
157+
raw1 = certify_raw_substring("1", ctx)
158+
good1, _ = certify_canonical("1", ctx)
159+
check("raw substring misses op1 (absent)", raw1 == "absent", raw1)
160+
check("canonical field-walk certifies op1 (verbatim)", good1 == "verbatim", good1)
161+
162+
print("chenhz01-001 guard 5: bucket boundary (forbid list)")
163+
sa = expected["availability_certification"]["state_a_rate"]
164+
judgeable = buckets["verbatim"] + buckets["summary-embed"] + buckets["absent"]
165+
check("State-A denominator excludes unreachable", sa["denominator"] == judgeable and sa["denominator"] == 3,
166+
str(sa["denominator"]))
167+
check("State-A numerator counts verbatim only", sa["numerator"] == buckets["verbatim"] == 1)
168+
check("summary-embed is not State A", consumptions[2]["x_availability"]["state_a_certified"] is False
169+
and consumptions[2]["consumed"] is False)
170+
check("summary-embed is its own bucket, not absent",
171+
consumptions[2]["x_availability"]["certification"] == "summary-embed")
172+
check("unreachable maps to UNOBSERVABLE, not FALSE",
173+
consumptions[4]["x_availability"]["observability_state"] == "UNOBSERVABLE")
174+
175+
print("chenhz01-001 guard 6: collapse is detectable (UNOBSERVABLE -> FALSE)")
176+
# Mislabel the unreachable result as absent: invariant 17 violation. The
177+
# judgeable denominator would wrongly grow from 3 to 4 — the vector must
178+
# make that change observable rather than silent.
179+
collapsed = dict(buckets)
180+
collapsed["unreachable"] -= 1
181+
collapsed["absent"] += 1
182+
collapsed_den = collapsed["verbatim"] + collapsed["summary-embed"] + collapsed["absent"]
183+
check("collapsing unreachable->absent changes the denominator (3 -> 4)",
184+
collapsed_den == 4 and collapsed_den != judgeable, str(collapsed_den))
185+
186+
if fails:
187+
print(f"\nCHENHZ01-001 CONFORMANCE FAIL: {len(fails)} guard(s): {', '.join(fails)}")
188+
return 1
189+
print("\nCHENHZ01-001 CONFORMANCE PASS: schema + core collapse + cert + re-serial + boundary + collapse-detectable")
190+
return 0
191+
192+
193+
if __name__ == "__main__":
194+
sys.exit(main())
Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# AgentMeasure availability certification ↔ FMT-002 mapping (chenhz01-001)
2+
3+
Date: 2026-10-08
4+
5+
Status: synthetic conformance fixture authored for issue #11; no external run.
6+
7+
AgentMeasure commit inspected: `67c276b7d17d5a00dc2fa89681eae97f29eeda06`
8+
9+
## Bottom line
10+
11+
M4.1's `consumption` event carries a single `consumed` boolean. That boolean
12+
cannot represent what issue #11 established: availability (the result entered
13+
the next model request) and influence (the result changed behavior) are
14+
different facts, and "reference" is a weak estimator between them, not a state.
15+
16+
This fixture is one trace whose four tool results each meet a different
17+
certification outcome, so a compaction-aware implementation can be tested
18+
against all four at once:
19+
20+
| op | outcome | what happened | State A | observability (AGENTS.md §2) |
21+
|---|---|---|---|---|
22+
| 1 | `verbatim` | the result's canonical content is in the next request | yes | TRUE |
23+
| 2 | `summary-embed` | only a lossy summary reached the request | no | FALSE for A, weak influence |
24+
| 3 | `absent` | a request left, the result is nowhere in it | no | FALSE |
25+
| 4 | `unreachable` | no request left the framework at all | no | UNOBSERVABLE |
26+
27+
The core aggregator scores this trace at `consumption_rate = 1/4 = 0.25`. That
28+
number is wrong in two ways the four-value axis fixes: it counts the
29+
`unreachable` result (UNOBSERVABLE) in the denominator, and it gives the
30+
`summary-embed` result no bucket of its own. The corrected State-A rate is
31+
`1/3 = 0.333`, with the unreachable result excluded, never counted FALSE
32+
(invariant 17).
33+
34+
## Why `summary-embed` is its own bucket
35+
36+
A summary embed means the content did reach inference, just lossily. That is
37+
weak evidence of influence (the State-B axis) with no verbatim availability. It
38+
is a different row from use-without-cite: there the result was present but
39+
uncited; here a projection of it was present and the original was not.
40+
41+
Folding `summary-embed` into an availability miss looks harmless until you
42+
notice where it lands. Compaction-heavy sessions are the common case in long
43+
agent runs, not the edge case, so an implementation that reports
44+
availability-failed on every compacted result will under-report exactly on the
45+
runtimes where the distinction pays for itself. The bucket has to exist.
46+
47+
## Why the certifier canonicalizes the field, not the bytes
48+
49+
`op1` is the re-serialization trap. The result is embedded in the next request
50+
as a serialized JSON string, so re-serializing the whole request escapes it. A
51+
raw byte or substring certifier misses it and reports `absent` — a false
52+
negative. The runner keeps both certifiers to pin the contrast:
53+
54+
```
55+
op1 raw-substring -> absent (false negative: re-serialization)
56+
op1 canonical -> verbatim (parse the field value, hash the content)
57+
```
58+
59+
So the certifier canonicalizes each field value and compares content hashes,
60+
and it records the field path where the match was found (`messages[1].content`
61+
for op1 and op2). The path is what lets a summary embed certify a weaker state
62+
instead of State A: same field, different `embed_form`.
63+
64+
## The four outcomes ↔ FMT-002
65+
66+
FMT-002 has no field for the certification outcome, so it rides in an
67+
`x_availability` extension on each `consumption` event, the same way the
68+
Urusilla fixture carries its ledger in `x_urusilla`. The boolean `consumed` is
69+
set to State A only (verbatim), which is what a correct verbatim certifier
70+
would emit today; everything the boolean drops is in the extension.
71+
72+
| certification | `consumed` | `signal` | `x_availability.observability_state` | in State-A denominator? |
73+
|---|---|---|---|---|
74+
| `verbatim` | `true` | `task_continuation` | TRUE | yes (numerator) |
75+
| `summary-embed` | `false` | `unknown` | FALSE | yes |
76+
| `absent` | `false` | `none` | FALSE | yes |
77+
| `unreachable` | `false` | `unknown` | UNOBSERVABLE | no — excluded |
78+
79+
`unreachable` is the invariant-17 case. The request never left, so the surface
80+
could not judge availability at all. Counting it as not-consumed is the
81+
UNOBSERVABLE→FALSE collapse the standard forbids. Guard 6 in the runner shows
82+
that mislabeling it as `absent` moves the denominator from 3 to 4, so the
83+
collapse is detectable rather than silent.
84+
85+
## Boundaries this vector does not cross
86+
87+
Two limits, flagged in review (chenhz01, #32), that a real-trace certifier has
88+
to close and this synthetic fixture deliberately does not:
89+
90+
- **`summary-embed` is declared, not inferred.** Here the summary bucket is
91+
read from the sidecar's `embed_form`, which is ground truth for a synthetic
92+
fixture. On a real trace the summary has to be established by a content-level
93+
criterion the runtime cannot silently fail to compute — a hash prefix,
94+
sub-sequence coverage, or an edit-distance threshold — or the classifier
95+
reintroduces the "looks wired, silently isn't" failure this vector exists to
96+
catch.
97+
- **The field-walk is single-layer exact match.** It checks whether a
98+
`messages[i].content` value parses and hashes equal to the result content.
99+
Nested embeddings (content parts, tool-call arguments) and partial embeddings
100+
are out of scope. This is a pinned exact-match certifier, not a general
101+
substring or deep search.
102+
103+
## Files
104+
105+
- `agentmeasure_chenhz01_fixture_001.events.jsonl` — 14 FMT-002 events (reach,
106+
choice, then attempt/operation_result/consumption per operation).
107+
- `agentmeasure_chenhz01_fixture_001.trace_context.json` — the raw inputs (the
108+
four results and the next request each faced), so certification is recomputed
109+
from scratch, not trusted from the extension.
110+
- `agentmeasure_chenhz01_fixture_001.expected.json` — core metrics, the
111+
four-bucket breakdown, the State-A rate, and the forbid list.
112+
- `../../../runners/run_external_fixture_003.py` — the six guards.
113+
114+
## Attribution
115+
116+
- Two-state ontology (availability vs influence) and the compaction boundary:
117+
Gunjan Jaswal (@gunjanjaswal), issue #11.
118+
- Trace-side hash certification (`certify_availability`): @chenhz01.
119+
- Canonicalized content, field-path recording, and the four-value axis:
120+
@roy-tong.
121+
- Fixture author: Gunjan Jaswal (@gunjanjaswal).
122+
123+
## Claim boundary
124+
125+
Synthetic conformance fixture authored for AgentMeasure issue #11. It is not an
126+
endorsement, an external reproduction, or a real provider or agent-runtime
127+
observation. The certification inputs are constructed to exercise the four
128+
outcomes, not sampled from a live trace.

0 commit comments

Comments
 (0)