Ascentis AI · Measurement note 001

A retrieval system can cite its sources and still make things up

Our support-resolution engine cited a real, retrieved document on 95.8% of cases. On the same run, 9.2% of its answers contained specifics that came from nowhere in the retrieved set. Both numbers were correct.

The question

If a system cites a real document for every answer it gives, has it stopped hallucinating?

The intuitive answer is yes, and citation rate is the metric most retrieval systems report. We measured both citation rate and fabrication rate on the same run of the same corpus. They disagreed sharply, and the gap between them turned out to be a bug we had been shipping for weeks.

The system

Cortex is our self-hosted support-resolution engine. These numbers come from its knowledge-retrieval path running against a technical support corpus for vacuum deposition and thin-film equipment — a domain where the same two hundred terms appear in nearly every document, and where a resolution has to name specific components, power settings and voltages to be worth anything.

• 120 cases, of which 92 carry labelled gold references.
• 64 knowledge-base articles, 498 indexed chunks.
• Self-hosted end to end: embeddinggemma embeddings, Qdrant, a BM25 and dense shortlist fused with reciprocal rank fusion, a bge-reranker-v2-m3 cross-encoder, and a locally served 30B model at temperature 0.

The client is not named and no client-identifying values appear below. Where a specific token matters to the argument it is redacted as PART-A, W-A, V-A.

Two measurements, same run

Retrieval quality alongside these: RR@5 (measurable-any) 83.3%. The engine was finding the right documents and the answers were pointing at them.

The first number says retrieval is working. The second says that in eleven of 120 cases the model asserted technical values that came from nowhere in the retrieved set.

Most retrieval systems never produce that second number, because computing it means auditing every asserted specific against the documents actually supplied to the model. Citation rate is easy to compute and reassuring. It is also not the question a customer is asking.

The eleven failures were not scattered. Every one emitted the same cluster of tokens — a part number, a wattage, a voltage — inside a single equipment failure category.

The diagnosis

The prompt was being built from titles.

_fetch_kb_context, the function assembling the recommended procedures and technical specifications sections of the model’s prompt, used retrieved article titles and their relevance scores. The content field that retrieval had already populated on every hit was sitting in the result object, unused.

So the model was handed a list of headings and asked to write a specific technical recommendation. It did what a language model does when asked for specifics it has not been given: it supplied them from training.

Retrieval had been right for weeks. Its output was not reaching the prompt.

How we proved it

A suspicion is not a finding. What made this one was a token-level provenance check.

PART-A appears in exactly one article in the corpus. One case cited that article and grounded PART-A, W-A and V-A correctly. Five other cases did not cite it, and emitted the identical tokens anyway.

There is no retrieval path by which those five could have seen those values. They came from the model’s own weights, not from the prompt. That asymmetry is the difference between the model hallucinates and the model was starved.

The fix

Three changes, no schema migration:

• Emit article and chunk bodies alongside titles. Articles capped at 4,000 characters — the corpus 90th percentile is 6,875, so this trims the longest ten per cent. Chunks are already section-bounded by the splitter and pass through whole.
• Persist the assembled prompt section at generation time, so the audit replays exactly what the model saw.
• Forward that section through the pipeline stage, which selectively copies fields and would otherwise have dropped it.

Cost: +2,900 tokens on average, +5,000 worst case. Comfortably inside the context window.

The two unchanged rows are the point. Retrieval did not move, because the fix does not touch retrieval — it changes only what the model is allowed to see. Grounding improved while retrieval stayed exactly where it was. That is what makes these two independent dimensions rather than one metric measured twice.

What this does not show

What we would now do by default

Measure linkage and consistency separately

Citation rate answers did it point at a real source? A grounding audit answers are its specifics actually in that source? A system can score 95.8% on the first and fail 9.2% on the second. Reporting only the first is how that goes unnoticed for weeks.

Persist what the model saw

Record the assembled prompt verbatim at generation time and replay it at audit time. Any evaluation that reconstructs context afterwards is measuring a different artefact than the one the model produced.