What the stamp means
Every response from the assistant carries a confidence stamp
visible in the UI and in the HTTP header X-Confidence-Level. It
has three values:
| Stamp | Meaning | When it appears |
|---|---|---|
| high | The assistant has chunks with rerank_score ≥ 0.6 and verified citations | Queries with clear regulatory anchor (articles, BOE, annexes) |
| medium | Relevant chunks with rerank_score between 0.4 and 0.6, or single citation | Queries where the corpus covers the topic but synthesis is required |
| low | No chunks with rerank ≥ 0.4, or extractive fallback used | Out-of-domain queries, ambiguous, or where the model exhausted the budget |
The stamp is conservative by design: when in doubt, go one level down.
The five signals that feed it
The backend computes the stamp by combining five deterministic signals:
- Max rerank score: the reranker’s confidence on the best chunk. Thresholds calibrated against the human evaluation set.
- Corpus coverage: percentage of in-response chunks the numeric
verifier could back. Below this number we drop to
mediumeven if rerank is high. - Self-grounded check: a second model call asking “is this
response supported by the cited sources?”. If it fails, stamp →
low. - Refusal detection: if the response contains canonical refusal
phrases (e.g. “no puedo afirmar…”, “no dispongo de datos…”; the
assistant’s canonical phrases are in Spanish, the corpus language),
stamp →
lowautomatically. - Fallback path: if synthesis fell back to extractive (timeout,
retry exhausted), stamp →
lowregardless of the rest.
Each signal is versioned and the thresholds are pinned in the backend test suite so any change breaks visibly.
How the calibration is validated
A miscalibrated stamp is worse than no stamp: it turns a transparency signal into marketing. So we apply three ongoing validations:
1. Human evaluation set
The evaluation set contains 15 hand-curated cases with
ground truth faithfulness: 0.0 | 1.0. Against that set we measure
the Pearson correlation between confidence_level and
faithfulness.
| Metric | Expected | Acceptable | Blocking |
|---|---|---|---|
Pearson r | > 0.7 | > 0.5 | < 0.5 |
| Cohen κ vs multi-judge panel | > 0.6 | > 0.4 | < 0.4 |
If correlation drops below 0.5 across two consecutive baselines, the stamp falls back to deterministic mode — rerank and coverage only, without the self-grounded check — until it is recalibrated.
2. Agreement between judges
Three independent evaluator models assess the same 40 responses. When
dispersion between judges exceeds 0.3 in faithfulness, the case is
flagged unreliable and excluded from the calibration metric. This
stops a noisy judge from shifting the thresholds.
3. Drift watch (post-market monitoring)
The post-market monitoring plan defines R / A / G thresholds over
the daily stamp distribution. If the high rate rises > 10 % in a
week without a prompt/model change, that’s a calibration drift
symptom and triggers manual review.
Honest limitations
- The stamp is not a probability. It’s an ordinal class calibrated
against human evaluation, not
P(response correct) ∈ [0, 1]. Treating it as probability leads to overconfidence. - It does not cover prompt injection. If the user injects instructions in the query, the stamp still evaluates grounding against the corpus, not coherence with the original benign intent.
- It does not cover freshness. For queries with “now” data (real- time flows, reservoirs) the stamp evaluates the citation, not the freshness of the datum. Check the source timestamp for freshness.
Changes
| Date | Change | Version |
|---|---|---|
| 2026-05-27 | Initial page. Documents v2 calibration post Tier 3.3. | 1.0 |