Chunking is harder than people expect

Ask a team building an AI retrieval system where the risk is, and they will point at the model, the prompt, maybe the vector database. Almost nobody points at chunking — the humble step of splitting documents into pieces before embedding them. And almost nobody expects how much damage it can do.

Chunking is harder than people expect. It looks like a plumbing detail; it is one of the biggest determinants of whether an AI system gives good answers or confident nonsense.

Bad chunking destroys retrieval

The most common approach is also the most damaging: split every document into fixed-size pieces — say, every 500 characters — and embed each one. It is simple, it is fast, and it shreds meaning.

Take a 300-page PDF split blindly every 500 characters. Somewhere in the middle, a sentence gets cut in half, and the system stores a chunk that reads:

“…must be signed by…”

When a user asks who has to approve a payment, that fragment is what retrieval finds. What they needed was:

“The contract must be signed by the CFO before payment approval.”

The full sentence carries the answer. The fragment carries nothing — no subject, no object, no meaning to embed. The model either misses the answer entirely or, worse, fills the gap with a plausible guess. Every other part of the system can be flawless and it will still fail, because the knowledge was destroyed before it ever reached the model.

That is the trap: chunking failures are invisible. Nothing errors. The pipeline runs green. The answers are just quietly wrong.

Good chunking respects the document

The fix is to stop treating a document as a stream of characters and start treating it as what it is — a structured thing with meaning built into its shape. Good chunking respects:

• Headings — a heading tells you where one idea ends and the next begins. Cutting across them, or orphaning content from the heading that gives it context, throws away the document’s own map.
• Paragraphs — a paragraph is usually a complete thought. It is the most natural unit to keep whole, and splitting one mid-way is how you get “…must be signed by…”.
• Tables — a cell means nothing without its row and column headers. Split a table blindly and “€2,000,000” ends up divorced from “Approval limit — CFO”. Tables need to be kept together, or serialised so each row keeps its headings.
• Sections — clauses, articles, chapters. In contracts and policies especially, the section is the unit that carries a self-contained rule. Respect the boundaries the author already drew.
• Semantic boundaries — even within prose, meaning shifts. Chunk where the topic changes, not where an arbitrary character count lands.

Get this right and each chunk is a coherent, self-contained idea — something that embeds cleanly and retrieves precisely.

Why it is trickier than it looks

Respecting structure sounds obvious. Doing it well is not, and this is where the difficulty hides:

• The Goldilocks problem is real. Too small and a chunk loses the context needed to answer. Too large and its embedding becomes an average of several topics — unfocused, and a weak match for any specific query. There is no universal magic number; the right size depends on your documents and your embedding model.
• Real documents are messy. Scanned PDFs, inconsistent formatting, multi-column layouts, tables that break across pages, headers and footers repeating on every page. The structure you need to respect is often not cleanly encoded — extracting it reliably is half the battle.
• Context has to travel with the chunk. A well-sized chunk is still weaker if it has lost its place in the document. Sensible overlap between chunks, and enriching each chunk with metadata — its heading, section, source and date — keeps meaning intact and lets retrieval be filtered and ranked.
• It can only be tuned by measuring. The only way to know your chunking is working is to test retrieval against real questions with known answers, and adjust. Guessing at a chunk size and hoping is how projects end up quietly wrong.

The quiet decision that decides everything

Chunking gets skipped because it is unglamorous and because, when it goes wrong, nothing looks broken. But it is one of the quiet decisions that decides whether an AI system earns trust or loses it — and it is a large part of why ambitious RAG projects fail.

When we build AI retrieval, we treat chunking as a first-class engineering problem: structure-aware, right-sized, metadata-enriched, and tuned against a real evaluation harness — because the cleverest model in the world cannot answer from a fragment that never held the answer.

---

Building AI retrieval, or wondering why yours returns confident nonsense? Get in touch.