GuidesAI development

Chunking documents for a RAG

By the Banzai Labs teamUpdated on 3 min read

The document§1§2§3§4Overlapping passagesTitle: …Section: …Each passage keeps its context
The document is split into overlapping passages, each with its title and section.

The four rules

  1. 1Split at headingsOne section, one passage, as long as it stays short.
  2. 2Cap the sizeA section that is too long is split again at paragraphs.
  3. 3OverlapA few sentences shared between neighbouring passages.
  4. 4Keep the contextDocument and section titles added to each passage.

By document type

Document Method
Web pages, Markdown Split by headings
Text PDF Extract, then split by headings or paragraphs
Scanned PDF Text recognition first
Tables Kept whole, in Markdown
FAQ One question and its answer per passage

The metadata to keep

  • The source and page, to cite.
  • The language and date, to filter.
  • Access rights, so a passage is never shown to someone who must not see it.

Testing your chunking

Take twenty real questions and look at the five passages returned for each. If the right answer is not there, change the chunking before touching the model.

Next: a RAG with pgvector and embeddings.

Frequently asked questions

What passage size should I choose?
A few hundred tokens is a common starting point. Too short and the passage loses its meaning; too long and it dilutes the search. Test on your real questions.
Why overlap passages?
So that an idea cut in two is found whole in at least one passage. An overlap of 10 to 20% is usually enough.
How should tables be handled?
Keep them whole, converted to Markdown, with the sentence that introduces them. Cutting a table in the middle makes it unreadable.

Read next

All guides