Handling contradicting retrieved chunks — relevant to LlamaIndex's retrieval layer?

[raa82]

Hi all — I'm a database reliability engineer (Postgres/MySQL/TimescaleDB background) who got frustrated that every retriever I've used treats retrieved chunks as equally trustworthy, even when two of them flatly contradict each other.

I built TekmerDB — open source, Apache 2.0, single Rust binary — as a storage layer that runs an NLI contradiction check on every insert. If a new fact conflicts with something already stored, both get flagged, confidence drops on the outlier, and the conflict is tracked with full source provenance. It ships with an MCP server too, so an agent can query it directly.

Given LlamaIndex is the data/retrieval layer for a lot of production RAG stacks, I'm curious whether this is a problem people building indices and custom retrievers here have actually hit — conflicting source documents, stale vs. current data, that kind of thing. Not pitching a replacement for anything in the stack, more wondering if it's worth exploring as something that plugs in alongside an existing index/retriever.

Happy to share a quick demo (a real vs. fabricated energy-stats example) if it's useful context. Genuinely just looking for whether this resonates with people using LlamaIndex day to day.

[fsystemweb]

Yes, this is a real problem. In production RAG.

I'd suggest a more creative solution, like this strategy:

• At ingestion, when a chunk arrives, check it against similar chunks already stored. If they disagree, save that information as metadata on the chunk (not inside the text, that would break search, new field for that).

• At query time, after retrieval, use that metadata to decide what to do.

Useful metadata fields

a group id shared by chunks that disagree with each other
a status: current / outdated / uncertain
a date the fact was stated
the source

The group id matters most. If two chunks from the same group both come back, you know they are competing versions of the same fact, and you can keep only one.

Three options at query time

Remove the outdated chunk
Lower its rank, so it has less influence
Keep both and tell the model they disagree

I would make the third one the default. If your check makes a mistake and you delete a good chunk, that's worse than the original problem. Showing both: "Source A says 82 GW (2024), Source B says 67 GW (2021)", is usually a better answer anyway.