On the tasks most businesses actually run, turning reasoning on can make the output less
reliable, not more. When Vectara replaced its hallucination leaderboard in November 2025 with
a harder benchmark built on more than 7,700 articles, the leading model came in around 3.3% — and
several reasoning variants exceeded 10%. Same vendors. Same underlying families. The models told to
think harder did worse at staying inside the source they were given.
That is not a bug report. It is a clue about what reasoning is for.
For the wider picture, our guide covers when reasoning mode earns its cost and when it does not.
Two objectives that pull against each other
Grounded summarisation asks a model for restraint: say only what the supplied evidence supports,
and say nothing else. Reasoning is the opposite instinct. It infers, connects, extrapolates and fills
gaps. That is precisely the capability you want when a problem needs several steps of analysis — and
precisely the capability that produces an unsupported sentence when the job was to stay inside a
contract, a datasheet or a policy document.
So a reasoning model doing worse on a faithfulness benchmark is not evidence that it is a weaker
model. It is evidence that faithfulness and inference are different objectives, and that the
setting you flicked on to get better answers optimises the wrong one.
The faithfulness section of our guide to understanding AI in 2026 makes a
related point that matters more than the headline: “what is the hallucination rate?” is not a
well-formed question unless you also specify the task, the dataset, the model, the scoring
method and the date. A grounded summary and an open factual answer are different problems. Compressing
them into one universal percentage is how vendors end up quoting numbers that mean nothing about your
system.
What the benchmarks do and do not tell you
The Stanford AI Index 2026 reports high error rates across frontier models on some open factuality
benchmarks. Vectara’s benchmark reports much lower rates on grounded summarisation. It is tempting to
divide one by the other and announce that grounding cuts hallucination by some tidy multiple. Resist
it. The two use different datasets, different prompts and often different model sets, so the ratio is
an artefact of the comparison, not a property of grounding.
The defensible conclusion is stronger anyway, and it does not need a multiplier: grounding
changes the failure mode. Instead of depending on what the model recalls from its weights, you
hand it evidence a human can inspect. When it goes wrong, you can see where — which is the difference
between a system you can improve and one you can only apologise for.
Where reasoning genuinely earns its cost
None of this is an argument against reasoning models. It is an argument against leaving reasoning on
by default across every task in your stack. Reasoning pays for itself on multi-step planning, on
mathematics, on ambiguous analysis where the right answer depends on weighing several considerations,
and on agent loops that have to decide what to do next. Those are jobs where the inference is
the work.
They are not the jobs most production AI does. Extraction, classification, formatting, routing and
grounded question answering over your own documents make up the bulk of real deployments, and every one
of them rewards restraint over invention.
What to do on Monday
This one is unusually actionable, because the fix is a configuration change rather than a project.
- Audit which of your tasks have reasoning enabled by default. In several stacks it is
on unless you say otherwise, and nobody made a decision about it. - Turn it off for the grounded ones — extraction, classification, formatting, and any
retrieval-augmented answering where the model should quote the source rather than reason around it. - Measure faithfulness before and after. Not vibes: take fifty to a hundred real
questions with known answers, and check whether each material claim is supported by the evidence
supplied. - Check what it costs you. Reasoning tokens are billed. If a setting is making output
both slower and less faithful, it is paying you twice to be wrong.
The broader habit worth building is separating the two questions your evaluation should answer.
Factual correctness asks whether a claim is true in the world. Faithfulness asks
whether it is supported by the evidence you provided. A response can be true but unsupported, or
perfectly faithful to a source that is itself wrong. Most teams measure neither, and then argue about
whether the system is “accurate”.
If you want the full framework — what to measure, how to rate evidence rather than eloquence, and how
to audit a synthesis against the sources it compressed — it is set out in the
faithfulness chapter of Understanding AI in 2026.