Looking for help debugging an ai live benchmark

[mixal1279]

Hi everyone! I'm working on the AI system for my Discord bot, CzituchBot, and I'm looking for developers who could help me identify the root cause of some issues with answer validation.

Repository: https://github.com/mixal1279/CzituchBot

Branch: "arena/9034b5a4-czituchbot"

The problem

I ran the AI Live Benchmark on five questions. The two latest runs produced nearly identical, poor results:

• Answer accuracy: 20%
• Completeness accuracy: 20%
• Factual accuracy: 0%
• Four out of five test cases ended with the answer being rejected in both runs.

The logs suggest that the system can retrieve evidence and verify supporting quotes, yet the final answer is still rejected by subsequent validation stages.

Some of the reported errors include:

• "evidence_not_verified"
• "answer_not_covered_by_claims"
• "claims_too_coarse_for_answer"

There are also API rate-limit errors (HTTP 429), validator timeouts, and fallback budget exhaustion.

What I need help with

I'd appreciate help from developers experienced in RAG systems, LLM answer validation, and multi-stage AI pipelines.

Specifically, I'd like to understand:

1. Which pipeline stage is actually responsible for rejecting the answers, and whether those rejections are justified or false positives.
2. Why verified evidence and supporting quotes are not sufficient for the pipeline to accept an answer.
3. Whether the root cause lies in generation, evidence validation, claim coverage checks, answer repair, or fallback handling.
4. How to fix the underlying logic without compromising factual reliability or allowing the model to hallucinate.
5. How to design regression tests that verify the fix without breaking other parts of the system.

I'm not looking to artificially improve the benchmark score by weakening every validation rule. I want to identify the actual root cause and fix it properly, following established engineering practices.

Benchmark runs and logs

• "AI Live Benchmark — Run Refactor response utils with a ResponseBuilder  #126" (https://github.com/mixal1279/CzituchBot/actions/runs/39491625181)
• "AI Live Benchmark — Run [version] bump version to 0.1.5 #127" (https://github.com/mixal1279/CzituchBot/actions/runs/39498184752)
• "CzituchBot repository" (https://github.com/mixal1279/CzituchBot)

Any help reviewing the code, tracing the validation pipeline, identifying the specific failure point, or suggesting a robust fix would be greatly appreciated.

Thanks!

[mixal1279]

ai-live-benchmark-4a208281a3504c6e09cec776da31b71ad4034109.zip

[nicholas-greenwood]

i'd leave the factual checks alone for now and look at the answer text. run #127 accepts why-001 with groq, then the quality check throws it out as unfinished, before repair runs into quota errors and timeouts. no wonder the logs look contradictory

walidujOdpowiedzAI strips evidence labels from its own copy, while the guard receives the original odp. the logged sentence passes offline, but adding (EVIDENCE CANDIDATE 1) triggers the rejection and matches the 23-character gap. the raw text isn't saved, so i'd log both versions and freeze them in 1 test before calling it the cause

if confirmed, moving that cleanup into oczyscOdpowiedzPrzedWalidacja would give both checks the same text, including after repair. a regression test could check that truncated answers, wrong numbers, changed negation and unsupported causes still fail. who-225 needs a separate coverage test

[mixal1279]

Okay, I will test it. Thank you for your response.