What is Chunking?

Chunking is the process of splitting large documents into smaller text segments before indexing them in a retrieval system. It is a foundational step in building retrieval-augmented generation (RAG) pipelines, where documents are too long to be stored or searched as a whole.

When a user submits a query, the system searches across these smaller segments – called chunks – rather than full documents. This makes retrieval faster and more precise, since a focused passage is more likely to match a specific question than an entire document would be.

How chunking works

Documents are divided into chunks using a defined strategy, typically based on token count, sentence boundaries, or paragraph structure. Each chunk is then converted into a numerical representation – called an embedding – and stored in a vector database where it can be retrieved by similarity to a user’s query.

The size of each chunk involves a tradeoff: smaller chunks preserve specific details but lose surrounding context, while larger chunks capture broader meaning but may dilute the relevance signal for precise queries. Because of this, choosing the right chunk size is one of the most important – and often underestimated – decisions when building a RAG system. Research has shown that no single chunk size works best for all queries, and that different questions benefit from different levels of granularity.

What chunking is used for

Chunking is used in any system that retrieves information from a document corpus before generating a response. Common applications include enterprise search, document Q&A, customer support bots, and knowledge management tools. More advanced implementations index the same documents at multiple chunk sizes simultaneously, allowing the retrieval system to surface the most relevant granularity depending on the nature of each query.