Entity resolution at ingestion — where do you dedupe entities before they hit the graph store?

[jules-gd-dev]

We've been building a GraphRAG pipeline with LlamaIndex's PropertyGraphIndex → Neo4j, and kept hitting the same wall: the extractor is great at finding entities, but it has no memory of which ones are the same node. Every run, "Apple Inc.", "Apple", and "AAPL" land as three separate nodes, and the graph slowly fragments.

How are others handling entity resolution at the ingestion boundary today — inline Cypher MERGE, a post-processing pass, or leaving it to retrieval time?

What we landed on (sharing for feedback):

A 3-layer short-circuit before anything reaches the graph store:

1. Deterministic (rapidfuzz): exact / alias / lexical core match — 0 tokens.
2. Semantic: cosine on pre-computed embeddings (we feed all-MiniLM-L6-v2 vectors in via node properties) — 0 tokens.
3. LLM arbiter: a single litellm call only for residual ambiguous pairs (e.g. "J. Dupont" vs "Jean Dupont").

The goal is to avoid sending every entity pair to an LLM. On a 500-doc / 3,000-mention / 63-identity benchmark (real run, real Groq pricing), this resolved 99.4% recall / 100% precision with 7 LLM calls and 1,566 tokens total, vs ~661k tokens ($0.0334) for full-LLM ER.

Honest limitation: ~18 of the 3,000 mentions were missed merges without an alias map (~4 with one), because Layer 1 can't bridge abbreviations it hasn't seen yet. That's the next thing I'm working on.

I open-sourced the approach as AutoGraft (MIT, one-line wrapper for the Neo4j store) — linking it as a reference, not a pitch. What I'd really like to hear: how do you all handle this? Is anyone doing ER after the graph is built (query-time or periodic merge job) rather than at ingestion? The trade-offs feel non-obvious.

[rafayaar]

@jules-gd-dev

I'd lean toward doing the primary resolution at ingestion time.

Once Apple, Apple Inc. and AAPL become separate nodes, it's not just retrieval that gets noisy. Edges, degree counts, communities, multi-hop traversal... all of that inherits the mess too. Deduping only at query time kind of papers over the duplicates in the final answer without actually fixing the graph underneath.

Honestly I'd probably want both: ingestion-time resolution as the main mechanism, plus a periodic reconciliation pass, since new documents can surface evidence that two previously separate entities are actually the same thing.

[jules-gd-dev]

Thanks for the feedback! Makes total sense, doing it at ingestion definitely keeps the graph metrics clean downstream. The periodic reconciliation pass is a great point too for when new context comes in later.

To be honest I'm 16 and still trying to figure out how to get actual hands-on feedback on projects like this. If you ever have a bit of time or a test dataset to run AutoGraft on, I'd really love to know how it holds up on your side or what breaks!

[rafayaar]

That’s really cool, especially at 16, you’re already thinking about some pretty interesting problems.
I don’t have a dataset ready to test it on right now, but I’ll keep AutoGraft in mind if I come across a good use case. I’d also be curious to see how it evolves as you test it on more varied data.
Good luck with it!

[amasen02]

We’ve run into this exact fragmentation problem in production GraphRAG setups with Neo4j and PropertyGraphIndex. Relying on query-time deduping quickly turns multi-hop Cypher traversals into a performance nightmare because every synonym path duplicates graph traversal branches and fragments Louvain/Leiden community clustering.

Here is the architectural pattern that has worked best for us in high-throughput Graph pipelines:

1. Synchronous Ingestion: Deterministic Canonicalization

For the primary ingestion pipeline, you want zero LLM latency and deterministic guarantees:

• Canonical Key Normalization: Lowercase, strip punctuation/legal suffixes (Inc., LLC, Corp), and resolve known tickers/identifiers via a fast pre-loaded hash table.
• Cypher MERGE with Alias Sets: Instead of creating duplicate nodes, maintain an aliases list property on the canonical node. When a variant mention arrives:

MERGE (n:Entity {id: $canonical_id})
ON CREATE SET 
    n.name = $primary_name,
    n.aliases = [$mention],
    n.createdAt = datetime()
ON MATCH SET 
    n.aliases = apoc.coll.toSet(n.aliases + [$mention]),
    n.mentionCount = coalesce(n.mentionCount, 0) + 1

This guarantees that trivial variants ("Apple", "Apple Inc.") collapse immediately into a single node with all edge connections aggregated cleanly.

2. Ambiguous Pairs: Soft SAME_AS Links Instead of Blind Collapsing

When similarity falls into the fuzzy zone (e.g., rapidfuzz score between 0.75 and 0.90, or cosine similarity on embeddings ~0.82):

• Do not blindly merge or delete nodes inline: If an LLM or fuzzy matcher makes a false-positive merge at ingestion time, recovering split node relationships later is painful.
• Create an explicit SAME_AS edge:

MATCH (a:Entity {id: $idA}), (b:Entity {id: $idB})
MERGE (a)-[r:SAME_AS]->(b)
SET r.confidence = $score, r.provenance = "embedding_knn"

During retrieval, your Cypher queries can expand across SAME_AS relationships (with a 0-hop cost) without corrupting the underlying node lineage.

3. Asynchronous Reconciliation (The Periodic Sweep)

Don't block the document ingestion workers waiting for embedding nearest-neighbor lookups or LLM arbitration.
Instead, dump newly created ambiguous node IDs into a queue (or query nodes created in the last $N$ hours with low mention counts):

• Run batch MiniLM vector clustering or cross-encoder reranking in a background worker.
• For high-confidence matches ($>0.95$), perform an automated topological merge in Neo4j using APOC:

MATCH (source:Entity {id: $sourceId}), (target:Entity {id: $targetId})
CALL apoc.refactor.mergeNodes([target, source], {
    properties: {
        aliases: "combine",
        description: "combine",
        `.*`: "discard"
    },
    mergeRels: true
})
YIELD node
RETURN node;

apoc.refactor.mergeNodes automatically redirects all incoming and outgoing edges to the unified node and deletes the duplicate without leaving broken relationships.

This hybrid model gives you sub-second ingestion with deterministic MERGE, zero-loss fuzzy linking via SAME_AS, and clean automated maintenance via background APOC reconciliation.