Chunking
Chunking means splitting longer documents into smaller sections (chunks) that are converted into embeddings individually and made searchable – a central quality factor in RAG systems.
In practice
This is necessary because embedding models can only process a limited amount of text at once. One widely used model accepts at most 8,191 tokens, longer documents would simply be cut off. Common approaches are fixed section sizes with overlap, cuts at sentence or paragraph boundaries, and semantic methods that keep related statements together.
Microsoft recommends roughly 512 tokens per section with a quarter of overlap as a starting point – beyond that, only measuring against your own material helps. Sections that are too small lose their context, sections that are too large dilute the individual statement. A useful rule of thumb: if an excerpt makes sense to a human without its surroundings, it usually does for the model too.
Microsoft recommends roughly 512 tokens per section with a quarter of overlap as a starting point – beyond that, only measuring against your own material helps. Sections that are too small lose their context, sections that are too large dilute the individual statement. A useful rule of thumb: if an excerpt makes sense to a human without its surroundings, it usually does for the model too.