What is reranking and when do you need it?
Reranking is the second stage in a RAG pipeline that reorders candidate documents by relevance. When does it actually add value, and when is it unnecessary complexity?
Reranking is a second-stage selection step in a RAG pipeline that reassesses the documents returned by the initial search and orders them by relevance to the exact user query. Vector search or hybrid retrieval first retrieves a relatively broad set of candidate documents; a reranker then compares each candidate more precisely with the query before the best results are passed to the language model.
You mainly need reranking when first-stage retrieval consistently finds documents that are semantically related but not precise enough to produce reliable answers. For small, well-structured knowledge bases with clearly differentiated documents and straightforward questions, reranking can be unnecessary complexity. For larger RAG systems, collections containing highly similar documents, technical or legal knowledge, and queries where subtle distinctions matter, reranking can become an important part of the retrieval architecture.
What reranking actually does
A conventional RAG system normally starts with retrieval. Documents are split into chunks and converted into embeddings: numerical representations of their semantic content. The user query is also converted into an embedding, after which a vector database searches for chunks whose vectors are close to the query vector.
This process normally relies on a bi-encoder. Queries and documents are encoded independently. Document embeddings can therefore be calculated in advance, making it practical to search large vector collections efficiently. The trade-off is that the model never evaluates the query and document together.
A reranker often uses a cross-encoder instead. Rather than comparing two independently generated vectors, the model receives the query and candidate document as a single input, and directly estimates how relevant that particular document is to that particular query. This requires more computation but allows for a much more detailed relevance judgment. A cross-encoder is therefore generally not used to search the entire corpus — the first retrieval stage creates a manageable candidate set, and the reranker only evaluates those candidates.
Conceptually: query → vector/hybrid retrieval → candidate chunks → reranker → best chunks → LLM. Reranking does not replace retrieval — it refines it, as also described in hybrid retrieval in production.
Why first-stage retrieval is often not enough
Embedding models are good at identifying semantic similarity, but semantic similarity is not the same thing as relevance. Imagine a knowledge base containing documents about:
- termination of employment contracts;
- termination of service agreements;
- notice periods;
- contract dissolution;
- conditions for early termination.
A question about the notice period for one specific type of agreement may be semantically close to all of these documents. A vector database may therefore rank several of them highly. For a RAG application, that creates a problem: the language model may receive the correct document together with several passages that are related to the topic but do not actually apply to the user's case.
The first retrieval stage should therefore primarily provide good recall: making sure the relevant material gets into the candidate set at all. The reranker can then improve precision: deciding which of those candidates are actually most useful for answering the query. A reranker cannot recover a document that the initial retriever never found — if the correct source is missing from the candidate set, the retrieval stage itself needs improvement.
When do you need reranking?
One strong signal is evaluation data showing that the correct document is usually retrieved but frequently appears below less relevant documents. That is exactly the type of ranking error a reranker is designed to address. It is also useful for collections with many similar documents — policies, contracts, technical manuals and procedures may share almost identical vocabulary while differing in a few crucial passages.
More complex queries are another good use case, for example: "Which procedure applies if supplier X still has access to personal data after the contract has ended?" The question combines several concepts at once — supplier, termination, access and personal data — and a cross-encoder can evaluate how these concepts interact because it sees the query and passage together. Reranking is also easier to justify when retrieval errors have meaningful consequences: in a general internal search engine, an imperfect ranking may simply inconvenience the user, while in legal or compliance systems an irrelevant passage can become the basis for an incorrect generated answer.
When is reranking overkill?
Not every RAG system needs a reranker. For a small knowledge base with cleanly structured and clearly distinct documents, ordinary vector retrieval may already be sufficient — if evaluation shows that the relevant passages consistently appear at the top, reranking mainly adds infrastructure and latency. Broad informational queries may also benefit less: a question such as "what are our main IT security policies?" can legitimately have several relevant passages, and highly precise ordering matters less than for a question about one specific exception in a procedure.
Reranking also does not automatically fix poor chunking, and poor retrieval remains poor retrieval — if the relevant document never reaches the candidate set, the reranker cannot select it. A sensible sequence is therefore: measure retrieval → improve retrieval → determine whether reranking is still necessary.
Latency and practical trade-offs
The main technical costs of reranking are compute and latency. With vector search, document embeddings are precomputed, so queries can be compared efficiently against a large collection. A cross-encoder, by contrast, must process multiple query-document pairs for each search request: the more candidates you rerank, the more inference work is required. Your first-stage retriever therefore needs to return enough candidates to give a high probability of including the correct document, without sending an unnecessarily large candidate set through a more expensive model.
Reranking does not necessarily require an external API. Cross-encoder models can also be deployed locally or on your own infrastructure — ecosystems such as Sentence Transformers provide models designed specifically for scoring query-document pairs. That makes self-hosted reranking relevant when documents should not leave company-controlled infrastructure, or when the entire RAG system needs to operate on-premises. Self-hosting still comes with operational costs: your team becomes responsible for model deployment, compute capacity, monitoring, updates and scaling — the real decision is therefore not simply "paid API versus free local model," but a choice between different infrastructure and operational models.
Reranking does not replace good retrieval
A common architectural mistake is to treat reranking as a universal fix for retrieval problems. A robust RAG pipeline contains several interacting layers: document parsing → chunking → metadata → embeddings → retrieval → filtering → reranking → context selection → generation. If document extraction is poor, reranking has little useful information to work with. If chunks are poorly sized or split at the wrong boundaries, the reranker may evaluate incomplete context. Reranking works best when the first-stage retrieval system is already reasonably good — it should be viewed as the final precision layer of retrieval, not as a repair mechanism for a fundamentally weak search architecture.
Rule of thumb: should you add reranking?
Use this checklist before adding a reranker to your RAG stack:
- Does the correct document usually appear somewhere in the initial search results?
- Does it frequently rank below less relevant documents?
- Does your knowledge base contain many semantically similar documents or passages?
- Do subtle differences in meaning materially affect the answer?
- Is retrieval accuracy more important than achieving the lowest possible latency?
- Can your infrastructure support the additional inference step?
- Have you measured retrieval quality rather than judging it from a few examples?
If the answer to the first question is no, improve your retrieval layer first. If the answer to the first question is yes, and several of the remaining answers are also yes, reranking is a logical next step. The simplest rule is: retrieval gets the right document into the candidate set; reranking gets the right document to the top. Add reranking when your measurements show that the second problem actually exists.