Skip to content

feat(mcp): add optional trust verification middleware for MCP servers - #23758

Open
eddyflores100-lang wants to merge 1 commit into
run-llama:mainfrom
eddyflores100-lang:feat/mcp-trust-verification
Open

eddyflores100-lang wants to merge 1 commit into
run-llama:mainfrom
eddyflores100-lang:feat/mcp-trust-verification

Conversation

@eddyflores100-lang

Copy link
Copy Markdown

What

Add an optional, pluggable trust verification module (trust.py) to the MCP tools integration, so LlamaIndex agents can check an MCP server's reliability before connecting.

Why

Issue #21768 asks for a pre-call trust gate. Currently, LlamaIndex agents using MCP tool specs have no way to verify server reliability before calling — a compromised or unreliable server could return bad data or fail silently.

The discussion on the issue explored several approaches (Dominion Observatory API, local Python scripts, MarketNow). This PR provides a neutral, pluggable implementation that doesn't lock into any specific trust provider:

  • The default lookup queries a public trust API (Dominion Observatory, free tier, no key needed)
  • Users can pass their own lookup callable for local, cached, or air-gapped verification
  • Fail-open (allow_unknown=True) or fail-closed (allow_unknown=False) for servers with unknown scores

How

Added two files:

llama_index/tools/mcp/trust.py — the module:

  • TrustVerdict: dataclass with score, is_trusted, reason, total_calls
  • TrustChecker: configurable gate with min_score, allow_unknown, and pluggable lookup
  • filter_trusted(urls): convenience method to filter a list of server URLs, keeping only trusted ones
  • _default_trust_lookup: queries the public Dominion Observatory API (50 queries/day, no key)
  • Score normalization: handles both 0-1 and 0-100 ranges

tests/test_trust.py — 6 tests:

  1. Trusted server passes (score >= threshold)
  2. Untrusted server fails (score < threshold)
  3. Unknown score is fail-closed by default
  4. Unknown score can be allowed with allow_unknown=True
  5. filter_trusted keeps only good servers
  6. Score normalization (0-100 → 0-1)

Usage

from llama_index.tools.mcp.trust import TrustChecker

# Optional: trust-check before connecting
checker = TrustChecker(min_score=0.5)
verdict = await checker.check("https://example.com/mcp/sse")
if not verdict.is_trusted:
    raise ValueError(f"Server not trusted: {verdict.reason}")

# Or filter a list of servers
trusted = await checker.filter_trusted(["https://a.com/sse", "https://b.com/sse"])

Design choices

  • Pluggable, not hard-coded: the lookup parameter lets users substitute any trust provider (local, cached, air-gapped). The default uses a public API but nothing in the module depends on it.
  • No new dependencies: uses httpx which is already a dependency of the MCP client.
  • Fail-closed by default: servers with unknown trust scores are considered untrusted unless explicitly allowed.

Closes #21768.

Adds a lightweight, pluggable trust checker that LlamaIndex agents can
use to verify an MCP server's reliability before connecting.

The module provides:

- TrustVerdict: dataclass with score (0-1), is_trusted, reason
- TrustChecker: configurable gate with min_score, fail-open/fail-closed
  modes, and a pluggable lookup function
- filter_trusted(): convenience method to filter a list of server URLs

The default implementation queries a public trust API (Dominion
Observatory, free tier, 50 queries/day, no key needed), but users can
pass their own lookup callable for local or air-gapped verification.

Usage:

    from llama_index.tools.mcp.trust import TrustChecker

    checker = TrustChecker(min_score=0.5)
    verdict = await checker.check("https://example.com/mcp/sse")
    if not verdict.is_trusted:
        raise ValueError(f"Server not trusted: {verdict.reason}")

Includes 6 tests covering: trusted pass, untrusted fail, unknown
fail-closed, unknown fail-open, filter_trusted, and score normalization.

Closes run-llama#21768.
@eddyflores100-lang

Copy link
Copy Markdown
Author

Adding visibility — the trust middleware is opt-in (gated behind trust_verification=True in MCPServerSettings) and uses no new dependencies (just httpx which is already in the tree). Default is fail-open for backwards compat. Happy to add a docs page or example notebook if useful for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: MCP server trust verification middleware

1 participant