"Chunk size" can mean characters, words, or tokens, and these are not interchangeable — an embedding model has a maximum token input length, not a character length, and English averages roughly 4 characters per token. A naive character-based splitter can silently overshoot an embedding model's token limit on token-dense text (code, non-English text, or heavy punctuation), causing the embedding call to truncate the chunk without warning. Production splitters typically count length with the actual tokenizer of the target embedding model (e.g., tiktoken for OpenAI models) rather than raw character count, precisely to avoid this mismatch.

Just cuts the text every N characters, based on a separator like a space.

Why chunk_overlap? If you cut text at exactly character 100 with zero overlap, you might slice a sentence right in half — chunk 1 ends with "...the claim will be processed with" and chunk 2 starts with "in 30 days if...". Neither chunk alone makes sense. A small overlap re-includes the last few words of the previous chunk at the start of the next one, so each chunk still reads sensibly on its own.

The problem: it cuts blindly based on character count, with no awareness of sentence or paragraph boundaries. It can — and often does — split mid-sentence or mid-word, producing chunks that read like garbage fragments and embed poorly. That's why it performs poorly in most RAG pipelines despite being the simplest option.

This splitter tries to split along the natural structure of the text, in priority order:

It tries the first separator. If a resulting piece is still too big, it recursively tries the next separator on that piece, and so on, until every chunk fits within chunk_size.

Each chunk is a complete, sensible thought — much better than a random character cutoff.

This is the default go-to splitter for most RAG projects — a good balance of simplicity and quality.

When the source format has explicit structure, splitting along that structure beats guessing at separators:

• Markdown — a header-aware splitter (e.g. MarkdownHeaderTextSplitter) splits on #/##/### boundaries first, and attaches the header hierarchy itself as metadata on each chunk (so a chunk under "## Refund Policy > ### Exceptions" carries that path even if the words "refund policy" never appear in the chunk's own text).
• HTML — splitting on tag boundaries (sections, list items, table rows) avoids cutting through a structural element the way a plain text splitter would.
• Code — a code-aware splitter (e.g. RecursiveCharacterTextSplitter.from_language()) uses language-specific separators (function and class boundaries) so a chunk doesn't end mid-function — critical for code-search RAG, since a truncated function body is often worse than no context at all.

A table split by any generic text splitter loses the row/column relationship — a row of numbers with no header context is meaningless once separated from the header row above it. Two common fixes: keep each table as a single, unsplit chunk (with its header row always included, even if this means an oversized chunk), or convert the table to a compact markdown or key-value textual representation per row (e.g., "Region: West, Q1: $120k, Q2: $135k") so each row is independently meaningful when retrieved in isolation.

Even recursive splitting has a blind spot: a single paragraph can still cover two different topics, and recursive splitting has no way to detect that — it only looks at punctuation and structure, not meaning. Semantic chunking works differently:

• Break the text into individual sentences
• Generate an embedding (vector) for each sentence
• Compare the similarity between each pair of consecutive sentences
• Wherever similarity drops sharply — a "breakpoint," signaling the topic has shifted — cut the chunk there

The breakpoint_threshold_type controls how a "sharp drop" is defined: percentile cuts wherever the similarity drop is in, say, the top 5% of all drops observed in the document; standard_deviation cuts wherever the drop exceeds some number of standard deviations from the mean drop; interquartile uses the interquartile range instead, which is more robust to a few extreme outlier sentences skewing the threshold. There's no universally correct choice — it's another parameter to validate against an eval set, same as chunk size.

Trade-off: this requires embedding every sentence just to decide where to cut — more embedding calls and more time and cost at indexing. Worth it for high-value, complex documents; overkill for simple or short ones.

Two techniques address the same underlying problem — a chunk taken out of its document loses context a human reader would have had:

• Contextual retrieval — before embedding, an LLM prepends a short, chunk-specific summary of where this chunk sits in the larger document (e.g., "This chunk is from the Q2 2024 earnings report, in the section discussing international revenue" prepended to a chunk that just says "Revenue grew 3% in this segment"). This directly fixes the case where a pronoun or an implicit subject in the original text ("this segment," "the company") becomes ambiguous once the chunk is isolated from its surrounding paragraphs.
• Propositional chunking — instead of chunking by fixed size or structure, an LLM first rewrites the document into a list of self-contained, atomic factual statements ("propositions"), each stripped of pronouns and made fully independent, and each proposition becomes its own retrievable unit. This produces the most semantically clean and independently-meaningful chunks possible, at the cost of an LLM pass over the entire corpus at indexing time.

Both trade extra indexing-time compute for retrieval-time precision — reasonable when the corpus is stable and re-indexed infrequently, harder to justify for a corpus that changes constantly.

There's no universal magic number — it depends on your documents and questions. The correct approach is empirical, not guesswork:

• Pick a few candidate chunk sizes (e.g., 200, 500, 1000 tokens)
• Build a small labeled test set of (question → expected answer/source)
• Run the pipeline with each chunk size and measure retrieval accuracy (see Evaluation)
• Pick the size that performs best empirically, and re-check it any time the document mix changes materially

Rough starting points: 300-500 tokens for question-answering RAG, 800-1200 tokens if the task is closer to summarization. Overlap is typically 10-20% of chunk size. These are starting points for the search, not a substitute for it.