Feature proposal: intra-judge reliability (repeated-trial self-consistency for feedback)

Problem

A TruLens feedback function is a single-trial LLM judge. At the default temperature the same judge, given identical inputs, does not always return the same verdict, and the pipeline currently records only that single draw. There is no built-in way to (a) reduce that run-to-run variance by repeated sampling, or (b) surface how reliable a given score was.

Jury already addresses a related but different axis: it ensembles several different judges (inter-judge diversity) to reduce bias. It does not run the same judge repeatedly, and it reports no run-to-run stability signal. Intra-judge variance is orthogonal to inter-judge bias and is not currently covered.

This matters because a single-trial score can flip on re-run with nothing about the inputs changed, which makes leaderboards and pass/fail gates built on one draw brittle.

Proposal

Add a callable wrapper, complementary to Jury and using the same plug-in contract, that runs one judge N times, aggregates the repeated trials, and attaches a reliability summary. Working name SelfConsistency (open to naming).

Like Jury, it exposes the same parameter names as the wrapped provider method, so it drops into Metric(implementation=...) with no change to Metric, Selector, or the evaluation pipeline, and it returns (score, {...}) so the detail flows into FeedbackCall.meta and the dashboard with no UI change.

SDK direction

from trulens.core import Metric
from trulens.feedback import SelfConsistency
from trulens.providers.openai import OpenAI

judge = SelfConsistency(
    provider=OpenAI(model_engine="gpt-4o-mini"),
    method="relevance",
    n_trials=5,
    aggregation="mean",  # or "median", "trimmed_mean", "majority_vote", or a callable
)

metric = Metric(implementation=judge, name="Relevance (self-consistent)").on_input_output()

Aggregated score

• Pointwise methods: aggregate the N scalar scores with the chosen strategy (reuse the Jury strategies: mean, median, trimmed_mean, majority_vote).
• The aggregated value is what the metric returns, so downstream scoring, storage, and dashboards are unchanged.

Reliability summary (attached to the result dict, not a new column)

Computed from the N trials and stored under meta, so it is visible in OTEL spans and the dashboard without schema changes:

• n_trials, and the per-trial scores.
• score_std (dispersion of the pointwise draws).
• For discretized or categorical verdicts, flip_rate = 1 - max(count)/N and outcome_entropy = -sum p*log2 p.

Deferred to follow-ups

To keep this first change small and reviewable, two pieces are intentionally left for follow-up PRs:

• A recommended-trial-count helper (Monte-Carlo majority-vote reliability curve), offline and advisory.
• Position randomization and a position-bias signal for pairwise or comparative methods.

Grounding

The metric definitions (flip rate, outcome entropy, and the reliability curve for trial-count) come from published work measuring exactly this run-to-run variance in LLM judges (arXiv:2606.13685). In that study, pairwise judge verdicts flipped on average 13.6 percent of the time across repeated identical evaluations, with a subset of items far higher, and a majority vote needed roughly 11 trials to recover a stable verdict. This proposal operationalizes those findings inside TruLens. (Disclosure: I am the author of that paper.)

Non-goals

• No change to Metric, Selector, or the storage schema.
• Not a replacement for Jury; the two compose (a self-consistent juror is possible later, but is not in scope here).
• No automatic multi-trial by default; opt-in per metric.
• No new dashboard page; reliability rides in existing meta.

Acceptance criteria

• A wrapped judge runs N trials and returns the aggregated score.
• The reliability summary (trials, std, and, where applicable, flip rate and entropy) is present in the result meta and visible in the dashboard.
• Drops into Metric(implementation=...) with no pipeline changes, exactly like Jury.
• Deterministic given a fixed set of trial outputs (unit-testable with a stubbed provider).

Tests

• Aggregation strategies over known trial scores (mean, median, majority_vote).
• Flip rate and entropy on constructed categorical outcomes.
• Reliability summary shape and presence in the returned meta.
• Plug-in parity: a SelfConsistency judge and the raw provider method produce the same argument-binding behavior under Metric.

Docs

Extend docs/component_guides/evaluation/llm_judge_alignment.md, which already routes "one judge is too noisy or biased" to Jury, with an intra-judge reliability entry, and add a short cookbook.

Prototype

A working reference implementation with unit tests is on a branch: https://github.com/Abelo9996/trulens/tree/explore/judge-consistency (SelfConsistency in src/feedback/trulens/feedback/self_consistency.py, exported in trulens.feedback, plus tests/unit/test_self_consistency.py). It reuses the Jury plug-in pattern and verifies end to end inside a real Metric. Happy to open it as a PR if the direction looks right.