Building a Production RAG Pipeline: Chunking, Hybrid Search & Cost Controls
Most Retrieval-Augmented Generation demos follow the same script: split text every 500 characters, embed into a vector database, grab the top 5 chunks, and stuff them into a prompt. In production, users ask about exact error codes, product SKUs, or documents they should not have permission to see — and the naive pipeline breaks.
Why pure vector search misses exact matches
Dense embeddings excel at semantic similarity ("how to reset password" matches "account recovery"), but they struggle with exact identifiers like invoice numbers, API error codes, or acronyms. Combining full-text keyword search (BM25 / PostgreSQL tsvector) with vector similarity via Reciprocal Rank Fusion (RRF) fixes both failure modes.
Challenge
Enforce permissions before retrieval, never after
Never retrieve top-K chunks globally and ask the LLM to "only show data the user owns." Always filter by tenantId and user ACLs inside the database query before chunks ever enter the prompt context.
Structure-aware chunking beats fixed character windows
- Split documents along natural boundaries (headings, sections, table rows) rather than mid-sentence character counts.
- Attach metadata to every chunk: document title, section heading, source URL, updatedAt timestamp and tenantId.
- Keep chunk sizes moderate (300–600 tokens) with a small overlap so the retriever stays precise while preserving local context.
sql-- Hybrid retrieval with mandatory tenant isolation (PostgreSQL + pgvector)
WITH semantic AS (
SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank
FROM document_chunks
WHERE tenant_id = $2
ORDER BY embedding <=> $1
LIMIT 20
),
keyword AS (
SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, plainto_tsquery('english', $3)) DESC) AS rank
FROM document_chunks
WHERE tenant_id = $2 AND tsv @@ plainto_tsquery('english', $3)
LIMIT 20
)
SELECT c.id, c.content, c.metadata,
COALESCE(1.0 / (60 + s.rank), 0) + COALESCE(1.0 / (60 + k.rank), 0) AS rrf_score
FROM document_chunks c
LEFT JOIN semantic s ON s.id = c.id
LEFT JOIN keyword k ON k.id = c.id
WHERE s.id IS NOT NULL OR k.id IS NOT NULL
ORDER BY rrf_score DESC
LIMIT 6;Keep latency and token costs bounded
Passing 15,000 tokens of raw context into every user question quickly turns into a slow, expensive feature. Rerank or cap context to the highest-scoring chunks, cache embeddings for repeated queries in Redis, and route simple classification or query-rewriting steps to smaller, faster models.
Note
Cite sources and allow "I do not know"
Instruct the model to return structured citations referencing chunk IDs and explicitly state when the retrieved context does not contain the answer. Grounded answers with clickable citations build far more user trust than confident hallucinations.
Takeaways
- Use hybrid search (vector + BM25) fused with RRF so both semantic queries and exact terms match reliably.
- Apply tenant and role filters inside the database query prior to LLM generation.
- Chunk by document structure and preserve section metadata for accurate citations.
- Cap context size, cache frequent queries in Redis, and enforce structured output schemas.
Read our companion guide on Integrating the Claude API Into Production, explore our AI & LLM Integration services, or start a project plan in the Project Builder.
Working on something like this?
I build NestJS backends and AI integrations for SaaS, fintech and crypto products. Happy to talk through your architecture.
Get in touch →