GuidesUnderstanding AI

Reranking: rereading results to keep the good ones

By the Banzai Labs teamUpdated on 2 min read

50 candidatesRerankerquestion + passage0.940.880.81Reread the top candidates with the question, and keep the truly good ones
The reranker rereads each candidate with the question and keeps only the most relevant.

Two ways to compare

Embeddings or reranker
Embeddings
  • Question and passage encoded separately
  • Comparing two vectors: very fast
  • Over millions of passages
Reranker
  • Question and passage read together
  • Finer judgment, slower
  • Over a few dozen candidates

Its place in a RAG pipeline

  1. 1Retrieval50 candidates, by keywords and meaning.
  2. 2RerankingEach candidate scored with the question.
  3. 3SelectionThe top 5, above a threshold.
  4. 4AnswerThe model reads only these passages.

The trade-off

Aspect Effect
Quality Fewer off-topic passages, more accurate answers
Main model cost Less text to read
Latency One more step
Complexity One more service or model to maintain

Measure the gain on your test set: evaluating a RAG system. Before reranking: hybrid search.

Frequently asked questions

Why not rerank the whole corpus directly?
Because it is slow: the reranker processes every question-passage pair. You apply it to the few dozen candidates already retrieved.
Which models exist?
Services such as Cohere Rerank or Voyage offer rerankers, and open source models such as bge-reranker can run on your own servers.
How much time does it add?
Usually from a few tens to a few hundred milliseconds, depending on the model and the number of candidates. Measure on your data.

Read next

All guides