Repository navigation
Pre-release offline agent-usage auditor: seeking one owner for a local validation review #23581
tz1335
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi LlamaIndex folks — I'm building Agent Token Waste Auditor, a pre-release offline prototype for examining sanitized per-call metadata from agent runs. It reports cost coverage, attributes child LLM calls without treating parent containers as extra billable calls, and flags exact repeated-context, retry, loop, and token-growth patterns when the needed fields exist. Costs without complete, verifiable evidence remain unknown. These flags are prompts for review, not proof of waste or verified savings.
I'm seeking one developer with authority over their own LlamaIndex or similar agent workflow who would be willing to review a short, content-free metadata field checklist. If code-distribution permission is cleared later, an optional test would run on your machine against one sanitized session, with you judging findings and misses against the actual task outcome. The current tool accepts normalized JSONL and supported sanitized OTLP GenAI/TraceLab exports; it has no LlamaIndex-native importer. No source checkout or installer is being offered at this stage.
Please don't post or send raw traces, prompts, reasoning, tool inputs/outputs, code, credentials, or identifiable logs. If interested, reply with only your export format and whether you have authority over the session metadata. Authorization for local analysis and any sharing of results would be separate choices; you can stop at the field review.
All reactions