Building Production-Grade RAG Pipelines: Beyond Simple Vector Search

Implementing a basic Retrieval-Augmented Generation (RAG) system has become trivial. With frameworks like LangChain or LlamaIndex and a few lines of code, you can load a PDF, store it in a vector database, and query it using an LLM.

However, moving a naive RAG system to production is a completely different challenge. Naive RAG systems typically hit a performance ceiling where retrieval accuracy hovers around 60–70%, leading to hallucinations, irrelevant responses, and user frustration.

To build an enterprise-grade RAG bot that delivers 95%+ accuracy, you must move beyond simple cosine similarity lookups. Here is the engineering blueprint we use at Axrio Labs to design robust, production-grade retrieval architectures.

1. Smart Chunking: Beyond Fixed-Size Splits

The performance of any RAG pipeline is fundamentally bound by how you split your source documents. Splitting text into fixed blocks of 500 characters often breaks paragraphs in half, separating key nouns from the verbs that give them context.

Parent-Child Chunking

Instead of feeding the exact same chunks used for search directly to the LLM, decouple the retrieval chunk from the synthesis chunk:

• Child Chunks (100-200 tokens): Small, highly focused segments optimized for dense vector search matching.
• Parent Chunks (1000-2000 tokens): Larger, contextually complete blocks containing the child chunks.

When the database matches a child chunk, the system automatically fetches the parent chunk and passes it to the LLM. This provides the LLM with full context while keeping vector search highly granular.

Semantic Chunking

Rather than splitting on character counts, split documents at points where the semantic meaning changes. This is achieved by:

1. Splitting the document into individual sentences.
2. Calculating the embedding vector for each sentence.
3. Measuring the cosine distance between consecutive sentences.
4. Setting a threshold (e.g., the 95th percentile of differences) to mark a split and create a new chunk.

2. Multi-Stage Retrieval: Hybrid Search + Reranking

Relying solely on dense embeddings for retrieval is a common mistake. Dense vectors excel at matching semantic concepts, but they perform poorly when looking up exact product IDs, database columns, or specific names.

Implementing Hybrid Search

Combine dense semantic search with sparse keyword search (BM25) to leverage the strengths of both:

• Dense Vector Search: Captures synonymy and conceptual matches (e.g., matching "billing queries" to "invoice issues").
• Sparse BM25 Search: Captures exact keywords and alphanumeric matches (e.g., matching serial codes or specific package names).

To combine these search results, use Reciprocal Rank Fusion (RRF). RRF scores documents based on their relative ranking in each separate search list rather than their raw score:

def reciprocal_rank_fusion(dense_results, sparse_results, k=60):
    rrf_scores = {}
    
    # Process dense results
    for rank, doc_id in enumerate(dense_results):
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
        
    # Process sparse results
    for rank, doc_id in enumerate(sparse_results):
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
        
    # Sort documents by RRF score descending
    sorted_docs = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
    return sorted_docs

Reranking with Cross-Encoders

Dense search models are typically bi-encoders (encoding query and document independently to save time). While fast, this limits detail matching.

Optimize your pipeline by querying your hybrid database for the top 50 documents, and then running them through a Cross-Encoder reranking model (like Cohere Rerank or BGE-Reranker). Cross-encoders perform joint query-document attention, scoring alignment with extreme accuracy. Select the top 5 reranked documents for your final LLM prompt context.

3. Advanced Query Transformations

Users rarely formulate queries in a way that matches your database schema. Before executing retrieval, preprocess the query to maximize matching probability.

• Query Rewriting: Ask a lightweight LLM to rewrite the user's input into multiple distinct search queries, or expand abbreviations into full terms.
• HyDE (Hypothetical Document Embeddings): Use an LLM to generate a hypothetical "ideal answer" to the query, and then use the embedding of that hypothetical answer to search the vector database. This matches text structured as answers to other answers, which often improves retrieval alignment.

4. Setting Up Continuous Evaluation

You cannot optimize what you do not measure. In production, we implement automated evaluation tools like Ragas or TruLens to run nightly validation suites against our retrieval loop.

Track these three core metrics:

1. Context Precision: Out of all the context retrieved, how much was actually relevant to answering the query? (Measures retrieval quality).
2. Context Recall: Did the retrieval step gather all the pieces of information necessary to formulate the answer? (Checks if information was missed).
3. Faithfulness / Groundedness: Is the generated response strictly derived from the retrieved documents? If the LLM generates facts not present in the context, flag it as a hallucination.

By treating RAG as an engineering pipeline with measurable unit-like tests, you can confidently iterate on your chunking layouts, index strategies, and prompts without breaking production behaviors.