The Reranking Bake-Off

Somewhere in our own archive sits a paragraph explaining why a reduced answer has to be labelled as reduced. Ask that archive a plain question — why must a reduced-quality answer be visibly marked as reduced? — and ordinary vector search puts the paragraph nineteenth. Nineteenth might as well be nowhere. A generation step reads the top few chunks, not the top twenty, so the answer gets built without the one chunk that answers it. The paragraph exists. The index works. The retrieval still fails where it counts.

That gap, between retrieved and actually read, is what a reranker is sold to close. Vendor numbers for it come from public benchmarks, which tell you little about your corpus. So we ran the comparison on the one corpus we can publish without asking anyone's permission: our own posts.

The rig

The corpus is 36 published posts from this blog, body text only, split into 230 chunks with a median length of 165 words. The question set is 24 questions, written and frozen before any retrieval ran. Each question has exactly one chunk that answers it, pinned by a phrase that occurs in that chunk and in no other chunk of its post. Eight questions are paraphrases, deliberately reworded so they share little vocabulary with their answer. Eight are exact, carrying a distinctive term or figure from the chunk. Eight are confusable: the answer lives in one post, and a sibling post covers a neighbouring topic in similar words.

Stage one is vector search with all-MiniLM-L6-v2, a bi-encoder that maps each chunk to a 384-dimensional vector; the question is embedded the same way and ranked by cosine similarity. Stage two takes stage one's top 20 and rescores each pair with ms-marco-MiniLM-L-6-v2, a cross-encoder. The difference is mechanical, and the Retrieve & Re-Rank pattern in the Sentence Transformers documentation states it plainly: a cross-encoder receives the query and the document together and outputs a single relevance score, so attention can cross between the question's words and the chunk's words. A bi-encoder never gets that look. It commits each chunk to one vector before it knows what will be asked.

We scored both stages on whether the answer chunk reached rank 1 and the top 5 (recall), and on mean reciprocal rank at 10. Latency was measured on the small shared CPU that ran the test, no GPU involved. Treat our milliseconds as a ratio between the stages, not as a spec sheet. One thing we did not measure: generated answers. This test scores the gate, not what walks through it. If the answer chunk is in the context, a competent model has what it needs; if it is not, no prompt will save the answer.

What the second stage bought

Vector search alone put the answer chunk first on 13 of 24 questions and inside the top five on 19. Adding the reranker moved those to 18 of 24 and 22 of 24. Mean reciprocal rank went from 0.65 to 0.82. On this corpus, in aggregate, the second stage is a clear win.

The bill is the other half of the result. The vector stage's median was 14 milliseconds per question, query embedding included. Rescoring 20 pairs took a median of 3.6 seconds on the same machine, roughly 250 times as much. Production hardware shrinks both numbers; it does not change the shape. One stage is a lookup against vectors computed in advance. The other is twenty pair scorings per question, every question, forever. A reranker is not a setting you turn on. It is a per-query cost you agree to pay.

The rescues

Paraphrase questions are where the money showed. The answer chunk reached the top three on 4 of 8 under vector search and on 8 of 8 after reranking. The buried paragraph from the opening went from rank 19 to rank 1. That is the cross-encoder's mechanism doing exactly what it promises: the question and the chunk share few words, but read together, they plainly match.

Exact questions improved too, which we did not expect. The chunk naming the three properties in Meta's Agents Rule of Two sat at rank 12 under vector search, because much of the corpus talks about sessions, tools and permissions in the same vocabulary. The reranker put it first. A chunk naming the benchmarks in a retrieval paper went from 8 to 1. Where a question carries a rare term, the embedding seems to spread its vote across every chunk that shares the surrounding topic; reading the pair directly concentrates it.

Where it lost ground

The confusable class went backwards. First place fell from 6 of 8 to 5 of 8, and mean reciprocal rank from 0.77 to 0.66. Three runs show the shape of the problem.

On the receipts question (what must be attached to a claim so a reader can check it), the answer chunk fell from rank 7 to rank 12. Look at the reranked top five and little seems wrong: three of its five chunks come from the receipts post itself, and the other two from its nearest neighbours. At document level the system looks healthy. At chunk level, the one chunk that answers the question dropped out of reading range, and chunk level is the level a generator consumes. On the cancelling question, a chunk vector search had ranked first fell to fifth. A general-purpose scorer, trained on MS MARCO passage ranking, has no way to know the distinctions our neighbouring posts draw between similar machinery. Faced with five plausible chunks, it picks by surface fit, and surface fit is what made them confusable in the first place.

The third failure is structural, and no reranker can fix it. On the circuit-breaker question, the answer chunk sat at rank 21, one place below the candidate window. The reranker never saw it. A second stage reorders what the first stage fetches; it cannot rescue what was never fetched. Across the set, vector search placed 23 of 24 answer chunks inside its top 20. That missing one belongs to the first stage's budget: a wider candidate window, hybrid search, or better chunking, not a better reranker.

What we would ship

Rerank when your questions arrive as paraphrases, which is most human traffic: support queues, internal search, anyone asking in their own words rather than the document's. That is the class where the second stage paid for itself several times over here.

Score per question class, never only in aggregate. Our aggregate says reranking wins. Our confusable class says it ships a regression on exactly the questions where two documents disagree, which are usually the questions worth answering carefully. An aggregate-only eval would have hidden that completely.

Widen before you rank. Candidate depth sets the ceiling for everything downstream, and the one total miss in this run was a depth failure. If you can afford only one improvement, buy first-stage recall before you buy a larger reranker.

And keep the latency honest in the design doc. If your p95 cannot absorb a second stage, the answer is not to pretend the cost away; it is to spend the same money on the first stage and accept the ranking you get.

The limits of this test

Twenty-four questions is small. One question is worth about four points of recall, so differences of one or two questions are noise; we trust the class pattern, not any single number. The corpus is our own prose, 36 posts on neighbouring topics, which makes the confusable class denser than most enterprise collections and the vocabulary more consistent. The method travels. These specific numbers describe this corpus, not yours. Both models are general-purpose, and neither was trained or tuned on this corpus; a larger or domain-tuned reranker may well handle the confusable class better, and that is a test worth running on your own text rather than assuming. Finally, we scored strictly, against one gold chunk per question, even where a sibling chunk would partly answer. That understates both systems, equally.

Work with us

If your system keeps answering from the wrong chunks, the place to start is not the prompt or the model. It is a measurement: which chunks your questions actually need, and where those chunks rank today. Bring the corpus and the questions it fails through Trida AI's intake and we will run that measurement with you.