Start with recursive character chunking at roughly 500–1000 tokens with 10–20% overlap, tuned to your embedding model's preferred input length. Use document-structure-aware chunking for anything that has headings or sections. Reach for semantic, hierarchical, or contextual chunking only after you've measured a retrieval-quality problem that simpler chunking can't fix. The biggest mistake is treating chunking as a "set it and forget it" choice — it's the most under-instrumented part of most RAG pipelines.
Most production RAG failures get blamed on the embedding model or the LLM. Research across enterprise deployments points to retrieval as the primary failure point — poor source data quality and suboptimal chunking are among the leading contributors to those retrieval failures. The LLM didn't hallucinate because the model is dumb; it hallucinated because the relevant fact lived halfway across two chunks, neither of which embedded cleanly enough to be retrieved. The embedding model didn't underperform; it was given a 4,000-token blob to embed when its training distribution was 256 tokens of focused content.
Chunking sits at the seam between two systems — the document store and the retriever — and it's the place where the most invisible quality losses happen. Below are the six chunking strategies that matter in production, when each one is the right move, and the mistakes that quietly tank retrieval before you even get to the interesting parts of the pipeline.
What Chunking Actually Does (and Why It's Harder Than It Looks)
Chunking is the step where a document is split into smaller pieces before being embedded into a vector database. Each chunk becomes one row in your index: an embedding vector representing the chunk's semantic content, the original text, and some metadata. At query time, the user's question gets embedded into the same vector space, and the top-K most similar chunks are pulled back and fed to the LLM as context.
The deceptively simple framing — "just split the document into pieces" — hides three real constraints.
- Chunks have to be small enough to embed well. Embedding models compress everything in the input into a single fixed-length vector. The more diverse meaning you cram into the input, the more the embedding averages out into noise. A 2,000-token chunk that contains three loosely related topics embeds to a vector that's not strongly close to any of them.
- Chunks have to be large enough to be self-contained. If the answer to "what's our refund policy for premium customers?" needs three sentences of context to make sense, and your chunking puts those three sentences in three different chunks, retrieval will surface one chunk that says "30 days" without context for whom it applies. The LLM either guesses wrong or refuses to answer.
- Chunks have to land at meaningful boundaries. A chunk that starts mid-sentence and ends mid-list is a degraded version of a chunk that respects the document's structure. Embedding models don't penalize mid-sentence chunks explicitly, but the resulting vectors are weaker because the meaning is fragmented.
The whole craft of chunking is trading these three constraints off against each other. The strategies below are different ways to make that trade-off.
The 6 Chunking Strategies That Matter
Fixed-size chunking (with overlap)
Split the document into chunks of a fixed token count (e.g., 500 tokens) with a fixed overlap between consecutive chunks (e.g., 50 tokens). The simplest possible strategy and the right default for ~80% of RAG systems.
Why it works: Predictable chunk sizes mean predictable embedding behavior. Overlap defends against the worst case where the answer lands exactly at a chunk boundary. Fast to compute at indexing time, easy to debug at retrieval time, and good enough for most domains.
Recursive character chunking
Split the document by trying a hierarchy of separators in order: paragraph breaks first (\n\n), then sentence breaks (. ), then word breaks, then character breaks — stopping when each chunk is below the target size. This is the strategy LangChain calls RecursiveCharacterTextSplitter and it's a small but meaningful upgrade over fixed-size.
Why it works: By preferring paragraph and sentence boundaries when they exist, the resulting chunks are far more likely to be semantically coherent. Embeddings are higher quality. Retrieval ranking is more stable.
Document-structure-aware chunking
For documents with explicit structure — Markdown with headings, HTML with sections, JSON with fields — split along those structural boundaries instead of treating the document as flat text. A Markdown chunker uses headings as anchors; an HTML chunker splits on <section> or <article>; a code chunker splits on function or class boundaries.
Why it works: Authors already organized the document into meaningful units. Use them. A heading is a strong human-curated signal that says "everything below me is about this thing." Chunks that respect headings get embeddings that line up cleanly with topic queries.
Semantic chunking
Use an embedding model to detect topic shifts within the document, and split at those shifts. Concretely: embed each sentence, walk through them looking for big drops in similarity between consecutive sentences, split at those drops. The resulting chunks are coherent units of meaning rather than fixed-size blocks.
Why it works: When it works, it's the highest-quality chunking you can do for narrative documents — each chunk is exactly one topic, embedded cleanly, ranks well at retrieval. When it doesn't work, you've spent a lot of compute at indexing and gotten chunks that are wildly variable in size, hard to budget against context windows, and hard to debug.
Hierarchical / parent-document chunking
Store two layers of chunks: small chunks for retrieval (e.g., 200 tokens) and large chunks or full documents for context delivery (e.g., the full section the small chunk came from). Retrieval matches against the small chunks because they embed precisely; the LLM gets the larger parent context so it can actually answer the question.
Why it works: Solves the central tension in chunking — the chunk size that's best for retrieval is usually too small to be useful as LLM context. Decoupling retrieval-time chunks from context-delivery chunks lets you optimize both independently.
Contextual chunking (LLM-augmented)
Before embedding each chunk, use an LLM to generate a short summary of how the chunk fits into the overall document, and prepend that summary to the chunk's text. The embedding then captures both the local content and the document-level context.
Why it works: Solves the "this chunk references something earlier in the document" problem. If chunk N+5 says "this policy applies only to the customers described above," embedding chunk N+5 alone produces a vague vector; embedding chunk N+5 with the summary "context: section about premium customer refunds" produces a vector that retrieves correctly.
How to Choose: A Decision Path
The honest decision tree for a new RAG system looks like this:
- Start with recursive character chunking, ~500–800 tokens, ~10–20% overlap. Build a small evaluation set of representative queries with known correct answers from your corpus. Measure retrieval recall@K on that set. This is your baseline.
- If your documents have structure (headings, sections, JSON), upgrade to document-structure-aware chunking. This is almost always a free win for Markdown, HTML, and code corpora.
- If your eval set shows that the answer often spans chunks, increase chunk size or overlap. Don't immediately reach for semantic chunking; just give the chunks more room.
- If your eval set shows that chunks retrieve correctly but the LLM gives bad answers because context is too thin, try hierarchical chunking. Keep small chunks for retrieval, deliver larger parents to the LLM.
- If your eval set shows that chunks routinely reference context defined elsewhere ("this applies to the customers mentioned above"), try contextual chunking — but only on the highest-value, lowest-update part of your corpus.
- Reach for semantic chunking last. It's the strategy with the biggest gap between "sounds elegant" and "actually pays off in production."
Five Mistakes That Quietly Tank RAG Quality
- Never evaluating retrieval separately from generation. If you only measure end-to-end answer quality, you can't tell whether the LLM is hallucinating because of bad retrieval or bad prompting. Always build a retrieval-only eval set and measure recall@K.
- Embedding chunks that exceed the embedding model's training distribution. Different models have different sweet spots; embedding 2,000-token chunks with a 256-token-trained model produces noisy vectors. Read the embedding model's docs.
- Treating all document types in one corpus the same way. A corpus that mixes Markdown technical docs, transcripts, and PDFs needs different chunking strategies per document type. One strategy applied to all of them is a guaranteed quality ceiling.
- Throwing metadata away. Document title, section heading, source URL, last-updated date — all of this is high-value at retrieval time, both for ranking and for the LLM's context. Store metadata alongside the chunk; don't discard it during chunking.
- Setting chunking once and never revisiting it. Embedding model changes, query distribution shifts, new document types — any of these can invalidate your chunking choices. Treat the strategy as a hypothesis you re-test quarterly, not a settled decision.
If you're building production RAG systems and looking for roles that go deep on this kind of work, the strongest engineering teams hiring for this are listed under our AI / ML engineering roles and the AI Skills hub. The companies with the best RAG infrastructure are usually the ones with strong engineering-driven cultures — where the people closest to the system get to make the calls about how to chunk, retrieve, and serve.
Frequently Asked Questions
Looking for engineering roles working on RAG / AI infra?
Browse open AI/ML engineering jobs at companies with engineering-driven cultures — the kind where the people building the system get to make the architectural calls.
See AI/ML Roles → AI Skills Hub →