Building RAG Pipelines That Actually Work: Lessons from Production Systems

Retrieval-augmented generation is now the default architecture for enterprise AI systems that need to reason over private data. The pattern is well understood. The implementation is less forgiving than most people expect.

We have built RAG systems over clinical notes, legal contracts, financial reports, and automotive databases. The problems that show up in production are usually not the ones that show up in the LlamaIndex quickstart tutorial.

The Three Phases Where RAG Fails

Phase 1: Ingestion

Bad ingestion is invisible until it is not. If your PDFs are scanned images without OCR, your chunks contain gibberish. If you split documents on fixed token counts without regard for sentence boundaries, you fragment the context that makes individual chunks useful. If you ingest without metadata — document type, date, author, section header — you lose the ability to filter retrievals meaningfully.

Invest in ingestion. Run OCR with confidence scoring. Chunk semantically, not mechanically. Extract and store rich metadata. Clean data retrieved well is worth more than any reranker you can throw at dirty data.

Phase 2: Retrieval

Vector similarity search alone is not enough. Pure semantic search returns contextually similar chunks — but similar is not the same as relevant. Hybrid retrieval combines dense vector search with sparse BM25 keyword search, giving you the best of both worlds.

Beyond hybrid retrieval, add a reranking step. A cross-encoder reranker takes your top-K candidates and rescores them based on the full query-chunk pair. This catches retrievals that are semantically related but not actually useful for the specific question. We typically retrieve 20 candidates and rerank down to 5.

Phase 3: Generation

The model receives retrieved context and generates an answer. Most hallucinations happen here, not because the model is broken, but because the prompt does not constrain it properly. If you ask a model to answer based on provided context and it cannot find a good answer, it will often make one up rather than say it does not know.

Fix this at the prompt level. Explicitly instruct the model to refuse to answer if the context does not contain sufficient information. Add an explicit citation requirement — every factual claim must reference a specific chunk ID. Then validate programmatically that citations exist and point to real content.

Chunking Strategies That Work

• •Semantic chunking: Split on sentence boundaries and paragraph breaks, not token counts. Aim for 200-500 tokens per chunk with 50-token overlaps.
• •Hierarchical chunking: Store both fine-grained chunks (for retrieval) and parent chunks (for context). Retrieve small, generate with large.
• •Document-aware splitting: Different document types need different strategies. A legal contract has clauses. A clinical note has SOAP sections. A financial report has tables. Treat them differently.
• •Metadata tagging: Every chunk gets document ID, creation date, section heading, document type, and confidence score from your ingestion pipeline.

Vector Databases: Which One and Why

For most teams, pgvector inside PostgreSQL is the right starting point. You already have Postgres. It supports hybrid search. It integrates with your existing data pipelines. The operational overhead is zero if you already run Postgres.

At scale — hundreds of millions of vectors, sub-10ms latency requirements — Pinecone or OpenSearch with the k-NN plugin become compelling. But most enterprise use cases do not start there, and teams often over-engineer their vector database choice before they have figured out their chunking strategy.

Measuring RAG Quality

You need an evaluation framework before you go to production. We use three metrics: faithfulness (does the answer contain only claims supported by retrieved context), answer relevance (does the answer actually address the question), and context precision (are the retrieved chunks actually relevant to the question). Tools like RAGAS make this measurable.

Set thresholds. If faithfulness drops below 0.85 in your evaluation set, do not ship. If it drops below that threshold in production monitoring, alert someone.

The Architecture That Works

Query rewriting, hybrid retrieval with BM25 plus dense vectors, cross-encoder reranking, structured prompt with explicit citation requirements, programmatic citation validation, and evaluation monitoring in production. This is not glamorous. It works.