We added a new component to one of our AI retrieval pipelines and tested it on its own. The result almost made us bin it. It turned out to be the single most important brick in the stack.
What a reranker actually does
Standard retrieval-augmented generation (RAG) embeds your query and your documents separately, then matches them by cosine similarity, so the model never actually sees the question and the document together. It is comparing two summaries and hoping they line up. A cross-encoder reranker reads the query and a candidate document as a pair and scores how well they truly match. Slower, because it cannot pre-compute the way embeddings do, but far more accurate, because it reads the question against the answer instead of guessing from a distance. Ours is a 568-million-parameter cross-encoder: about 1.5GB of VRAM, adding roughly 150 milliseconds to a query. Modest on paper. The effect on answer quality was anything but.
The benchmark that nearly fooled us
We ran it against our 92-case, gold-labelled benchmark. Accuracy moved from 70.1% to 73.6%. Decent, not life-changing. Then we added two techniques every RAG tutorial recommends, BM25 keyword matching and Reciprocal Rank Fusion, and ran the same benchmark again. Accuracy dropped 35 percentage points. Not a regression: a complete mess, worse than day one. We nearly stopped there.
Then we tried one last thing: all three together. Result: 78.2%, the biggest single jump we had ever measured on that corpus.
The lesson: it is a gatekeeper, not a modest improver
Alone, the reranker just tidies a list that was already mostly right. Combined with broader, noisier retrieval, it extracts the real signal from the noise. The two “unhelpful” techniques only became useful because something downstream was clever enough to filter them. We run it in two stages: a fast vector search casts a wide net and pulls the top twenty to thirty candidates, then the reranker picks the real top five to ten. On a specialised technical corpus, where the same two hundred terms appear in every document, that gatekeeper is what makes hybrid search work at all.
So if you are building a RAG system and have not added a reranker, do that before you tune chunking or argue about which embedding model to use (and mind your fusion parameters while you are there). The smartest engineering decision we made that month was not adding the reranker; it was running the four-way test that showed us what the reranker was actually for.
What is the most useful technical brick you have almost thrown away because it tested poorly on its own?