07-02-2026, 08:52 PM
Built a RAG system over a private document corpus (technical docs, ~4000 pages). Sharing what actually mattered vs what the tutorials made sound important.
Chunking strategy matters more than the embedding model. I started with naive fixed-size chunks (512 tokens) and got mediocre retrieval. Switching to semantic chunking that respects section boundaries improved answer quality noticeably. The embedding model was less important than getting the chunks right.
Hybrid search beats pure vector search. Combining BM25 (keyword) with vector similarity via reciprocal rank fusion caught things vector search missed - especially exact product names, error codes, and technical terms with unusual semantics.
The reranker is not optional. Cross-encoder reranking on the top-K retrieved chunks before passing to the LLM made a bigger difference than almost anything else. Cohere Rerank or a local cross-encoder both work.
Eval is the hard part. I spent 20% of the time building and 80% figuring out whether it was actually working. RAGAS helped but you still need human-evaluated golden datasets.
What I would do differently: Start with eval. Build your test questions first, then build the system to pass them.
Happy to go deeper on any part of this.
Chunking strategy matters more than the embedding model. I started with naive fixed-size chunks (512 tokens) and got mediocre retrieval. Switching to semantic chunking that respects section boundaries improved answer quality noticeably. The embedding model was less important than getting the chunks right.
Hybrid search beats pure vector search. Combining BM25 (keyword) with vector similarity via reciprocal rank fusion caught things vector search missed - especially exact product names, error codes, and technical terms with unusual semantics.
The reranker is not optional. Cross-encoder reranking on the top-K retrieved chunks before passing to the LLM made a bigger difference than almost anything else. Cohere Rerank or a local cross-encoder both work.
Eval is the hard part. I spent 20% of the time building and 80% figuring out whether it was actually working. RAGAS helped but you still need human-evaluated golden datasets.
What I would do differently: Start with eval. Build your test questions first, then build the system to pass them.
Happy to go deeper on any part of this.