Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is reranking? Rerankers and cross-encoders in RAG

Reranking rescores a retriever's shortlist with a cross-encoder that reads the query and each document together. Why it beats embeddings alone, with Python.

What is reranking? Rerankers and cross-encoders in RAG

Reranking is a second retrieval stage: a fast first-stage search pulls a shortlist of candidates (often 50 to 150), and a slower, more accurate model rescores each candidate against the query and reorders them, so only the best few go into the LLM prompt. The model doing the rescoring is usually a cross-encoder, a transformer that reads the query and one document together as a single input and outputs a relevance score.

The pattern goes back to Nogueira and Cho’s “Passage Re-ranking with BERT” (January 2019), which fine-tuned BERT to score query and passage pairs and took the top spot on the MS MARCO passage ranking leaderboard, beating the previous best by 27% relative in MRR@10. In RAG systems it is the step that turns “the right chunk is somewhere in the top 50” into “the right chunk is in the top 5”.

Why do you need a reranker if you already have embeddings?

Because the first stage is built for speed, and speed costs accuracy in a specific way.

An embedding model is a bi-encoder: it encodes the query and each document separately into vectors, and relevance is the cosine similarity of the two vectors. Document vectors are computed once at index time, so a search is one query encoding plus a nearest-neighbor lookup. The price is that the document’s vector is fixed before anyone asks a question. It has to summarize everything the document might be relevant to, in a few hundred or thousand numbers, without knowing the query.

A cross-encoder takes [query] [SEP] [document] as one sequence. Every token of the query can attend to every token of the document inside the transformer, so the model can see that the query asks about Python to Rust and the document is about Rust to Python, or that a document says “does not”. The Sentence Transformers documentation puts the trade simply: “Cross-Encoder achieve better performances than Bi-Encoders”, but “A Cross-Encoder does not produce a sentence embedding”, so nothing can be precomputed. Every query needs one forward pass per candidate.

That is why you can’t use a cross-encoder for the first stage. The same docs give a clustering example: scoring all pairs of 10,000 sentences with a cross-encoder means about 50 million combinations and “about 65 hours”, against 5 seconds to embed them with a bi-encoder.

How does a two-stage retrieve and rerank pipeline work?

The Sentence Transformers retrieve and re-rank guide describes the standard shape: first “use a retrieval system that retrieves a large list of e.g. 100 possible hits”, then a cross-encoder “scores the relevancy of all candidates for the given search query”.

In a RAG pipeline that becomes:

  1. Retrieve a wide shortlist from BM25, vectors or both (hybrid RAG). Optimize this stage for recall: the right chunk just has to be somewhere in the list.
  2. Rerank the shortlist with a cross-encoder. Optimize this stage for precision at the top.
  3. Cut to the top 5 to 20 and put them in the prompt.

Anthropic’s contextual retrieval experiments used this shape with Cohere’s reranker: retrieve the top 150 chunks, rerank, keep the top 20. Adding the reranker took their reduction in top-20 retrieval failures from 49% (contextual embeddings plus contextual BM25) to 67%. They also note the cost side: “There is an inherent trade-off between reranking more chunks for better performance vs. reranking fewer” for latency and cost.

What does reranking look like in code?

A real cross-encoder needs a model download, so this script uses a toy reranker that shares the important property: it reads the query and document together and scores the pair, so it can see word order and negation. The first stage is a bag-of-words overlap, which, like a single vector per document, is blind to order.

# rerank.py: two-stage retrieval. A cheap first stage, then a pairwise reranker.
# Python 3 stdlib only. The reranker is a toy: it reads query and document together
# (word order and negation), which a bag-of-words vector cannot see.
import re

docs = {
    "a": "How to migrate a Python service to Rust, step by step.",
    "b": "How to migrate a Rust service to Python, step by step.",
    "c": "Rust or Python: which language should a new service use?",
    "d": "Python service does not need Rust for this migration.",
    "e": "Migrating Java services to Kotlin.",
}

def words(t): return re.findall(r"[a-z]+", t.lower())

def first_stage(query, k=4):  # bag-of-words overlap: order-blind, one vector per doc
    q = set(words(query))
    scored = sorted(docs, key=lambda d: len(q & set(words(docs[d]))), reverse=True)
    return [(d, len(q & set(words(docs[d])))) for d in scored[:k]]

def rerank_score(query, doc):  # pairwise: looks at both texts at once
    q, d = words(query), words(doc)
    q_bi, d_bi = set(zip(q, q[1:])), set(zip(d, d[1:]))
    score = len(set(q) & set(d)) + 2 * len(q_bi & d_bi)  # reward matching word order
    if "not" in d and "not" not in q:
        score -= 3  # the document negates what the query asks for
    return score

query = "migrate python service to rust"
candidates = first_stage(query)
print("stage 1 (bag of words):", ", ".join(f"{d}={s}" for d, s in candidates))
reranked = sorted(candidates, key=lambda x: rerank_score(query, docs[x[0]]), reverse=True)
print("stage 2 (pairwise)    :", ", ".join(f"{d}={rerank_score(query, docs[d])}" for d, _ in reranked))
print("top answer            :", docs[reranked[0][0]])
print()

# Why two stages: count the expensive model calls per query.
corpus, shortlist = 1_000_000, 100
print(f"cross-encoder over the whole corpus: {corpus:,} forward passes per query")
print(f"bi-encoder retrieval + rerank top {shortlist}: 1 query encode + {shortlist} forward passes")

Output (Python 3.14.5):

stage 1 (bag of words): a=5, b=5, c=3, d=3
stage 2 (pairwise)    : a=11, b=7, c=3, d=2
top answer            : How to migrate a Python service to Rust, step by step.

cross-encoder over the whole corpus: 1,000,000 forward passes per query
bi-encoder retrieval + rerank top 100: 1 query encode + 100 forward passes

The first stage ties a and b at 5: same words, opposite meaning. It also can’t tell that d argues against the migration. The pairwise scorer separates a from b by matching word order (python service, to rust) and pushes d to the bottom for the negation. A trained cross-encoder learns these signals and many subtler ones from millions of labeled query and passage pairs instead of from my two hand-written rules.

The last two lines are the reason for the whole architecture. Running the accurate model on every document is a million forward passes per query. Running it on a shortlist is a hundred.

Which rerankers are there?

Three broad kinds, in rough order of cost:

  • Open cross-encoders you host yourself, such as the MS MARCO cross-encoders listed in the Sentence Transformers model docs. Small ones run on a CPU for short shortlists.
  • Hosted rerank APIs. Cohere’s Rerank takes a query, a list of documents and top_n, and returns each document’s index with a relevance_score. Its docs currently list Rerank 3.5, 4.0 and 5 models (read 2026-10-31). Voyage also offers one; Anthropic mentions it in the contextual retrieval post without testing it.
  • LLM-as-reranker. Ask a chat model to score or order the candidates. Flexible and slow; useful when relevance needs reasoning, such as “which of these contract clauses limits liability”.

Late-interaction models such as ColBERT sit between bi-encoders and cross-encoders: they keep one vector per token, so document side work is precomputed, but scoring still compares query and document token by token.

Where does reranking break?

  • It can’t recover what wasn’t retrieved. A reranker only reorders the shortlist. If the right chunk is at position 300 and you rerank the top 100, it’s gone. Measure first-stage recall at the shortlist size before tuning the reranker.
  • Latency scales with the shortlist. Each candidate is a forward pass over query plus chunk. Long chunks and big shortlists add up, which is the trade-off Anthropic calls out.
  • Domain shift. A cross-encoder trained on web search queries may rank code, legal text or Gujarati poorly. Test it on your own judged queries.
  • Scores aren’t calibrated across queries. A relevance score of 0.9 on one query and 0.9 on another don’t mean the same thing. Thresholding on a fixed score to decide “no good answer” needs checking per domain.
  • Chunk truncation. Rerankers have an input length limit. Text past it is ignored, so a relevant sentence at the end of a long chunk can be missed.

What I would do

In any RAG pipeline where answer quality matters, I’d add a reranker before tuning anything else in the prompt. The setup I’d start from:

  1. Hybrid first stage (BM25 plus vectors, fused with RRF), keeping 50 to 150 candidates.
  2. A cross-encoder over the fused list, keeping the top 10 to 20 for the prompt.
  3. A judged query set to compare recall at the shortlist size (first stage) and precision at the top (after rerank) separately. How to evaluate RAG covers the metrics.
  4. Latency budget per query decides the shortlist size, not the other way round.

Rerank-then-cut also lets you send fewer chunks to the LLM, which saves input tokens on every call and offsets part of the reranker’s cost. Whether it covers all of it depends on your models and prices, so do the arithmetic for your own traffic.

FAQ

What is a cross-encoder?

A transformer that takes a query and a document as one input and outputs a relevance score. It is more accurate than comparing separate embeddings because query and document tokens attend to each other, and slower because nothing can be precomputed.

What is the difference between a bi-encoder and a cross-encoder?

A bi-encoder embeds query and document separately and compares the vectors; document vectors are computed once, so search is fast. A cross-encoder scores each query and document pair jointly; it is more accurate and needs one model call per pair.

How many documents should I rerank?

Enough that the right one is almost always in the shortlist, and few enough to meet your latency budget. 50 to 150 is a common range; Anthropic reranked the top 150 down to 20.

Do I still need a reranker with a strong embedding model?

Usually yes. Anthropic’s tests showed reranking reducing failures further even after contextual embeddings and BM25. Measure on your data.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.