Published 2025-09-10
Tags: #computing
TL;DR. Read how Anthropic did it, and for more tips, read Chapter 6 of the AI engineering book. I’m just condensing tips that I’ll personally use.
I have been looking into various RAG resources, from Llama Index CEO [1], Anthropic [2] and this O’Reilly AI engineering book [3]. Here are the various ways you can improve your RAG pipeline. I organize them in the order in which I would personally try them, by weighing the foreseen benefit and the implementation and monetary cost.
Table of content
Anthropic [3] recommended Gemini and Voyager.
If you have a crappy query like “This doesn’t work” and feed it directly to the RAG, then you lose all hope in getting meaningful matches. [2] suggested rewriting the user query first. Here are more snippets from the book:
Example scenario
User: When was the last time John Doe bought something from us? AI: John last bought a Fruity Fedora hat from us two weeks ago, on January 3, 2030. User: How about Emily Doe?
The last question, “How about Emily Doe?”, is ambiguous without context. If you use this query verbatim to retrieve documents, you’ll likely get irrelevant results. You need to rewrite this query to reflect what the user is actually asking. The new query should make sense on its own. In this case, the query should be rewritten to “When was the last time Emily Doe bought something from us?”
Example prompt “Given the following conversation, rewrite the last user input to reflect what the user is actually asking”.
From [1] you can supposedly improve your PDF-to-text performance using LlamaParse. The benefit of this step depends on how complicated your PDF is. If your PDFs are simple, then this step is probably not that important.
Preface. Everything below here I would not do unless I have invested in an evaluation framework, because they’re more expensive, and so you would want to invest in keeping these additions only if you have data points to show that (1) your performance currently sucks and (2) these heavy investments do help in a significant way
Set up a robust evaluation framework. Track precision and recall. You can use AI as a judge to auto-generate a dataset for you like so: Start with a bunch of queries and a fixed document library. For each query, use AI to label which docs are relevant vs not. With an evaluation framework, you can gain more confidence in your changes.
LlamaIndex [1] recommended starting with per-page chunking as a baseline, because in some media, information is optimized to not be broken across page boundaries. [2] recommended the following:
Instead of just using a chunk of raw text as the vector store key, Anthropic [3] suggested prepending chunk-specific explanatory context.
E.g.
original_chunk = “The company’s revenue grew by 3% over the previous quarter.”
contextualized_chunk = “This chunk is from an SEC filing on ACME corp’s performance in Q2 2023; the previous quarter’s revenue was $314 million. The company’s revenue grew by 3% over the previous quarter.”
Here is their example prompt
Here is the chunk we want to situate within the whole document
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.
It is worth noting that other approaches to using context to improve retrieval have been proposed in the past. Other proposals include: adding generic document summaries to chunks (we experimented and saw very limited gains), hypothetical document embedding, and summary-based indexing (we evaluated and saw low performance). These methods differ from what is proposed in this post.
From Anthropic [3], you can divide your retrieval into multiple phases: (1) A high-recall, cheap retrieval phase from possibly multiple approaches — e.g. vector store and BM25 followed by (2) A more accurate, more expensive re-ranking step. As a baseline, [2] mentioned RRF [4], but Anthropic [3] mentioned there are commercial solutions from Cohere and Voyage. Finally, you can add custom re-ranking logic if you have additional metadata like each source’s trustworthiness, and train your own ranker using traditional ML methods.
Vector-based retrievers can surprisingly fail at matching exact keywords like exact error codes, or long scientific names. Anthropic [3] recommended using BM25-based retriever in addition to the vector-based retriever, then merging the results using a re-ranking step.
[1] Building Production RAG Over Complex Documents — Jerry Liu, Co-founder and CEO, LlamaIndex
[2] AI engineering Chapter 6 — Chip huyen
[3] Introducing Contextual Retrieval — Anthropic