Repository navigation
Implement metadata filtering post retrieval #15477
Replies: 1 comment
|
At that point in your pipeline, filter the retrieved query_bundle = QueryBundle(query_str=query)
retrieved_nodes = self.fusion_retriever.retrieve(query_bundle)
namespace = self.namespace
if not isinstance(namespace, str):
raise ValueError("namespace must be a string")
filtered_nodes = [
hit for hit in retrieved_nodes
if hit.node.metadata.get("namespace") == namespace
]
recency_nodes = self.recency_postprocessor.postprocess_nodes(
filtered_nodes, query_bundle=query_bundle
)
rerank_nodes = self.postprocessor.postprocess_nodes(
nodes=recency_nodes, query_bundle=query_bundle
)
return rerank_nodesThis implements scalar, exact namespace equality and excludes nodes missing that metadata. It preserves the original nodes and retrieval scores. One tradeoff: if fusion returns 10 candidates and only 2 match, post-filtering leaves 2; it cannot recover matching nodes that never entered the candidate set. Prefer passing the namespace filter into each supporting underlying retriever when possible, and evaluate candidate limits if you must post-filter. If namespace represents a tenant/security boundary, apply the restriction to every retrieval branch before retrieval, as well as any defensive final check. Post-filtering after an external reranker/LLM has seen unauthorized content would be too late. Reference: |
Uh oh!
There was an error while loading. Please reload this page.
I meant is implement a metadata filter to filter based on this:
All reactions