汤不热吧在构建RAG（检索增强生成）系统时，很多人把精力集中在选择更好的向量模型或更大的LLM上，却忽略了一个最基础却影响深远的环节——文档分块（Chunking）。分块策略的好坏直接决定了检索阶段能否找到真正相关的内容，进而影响最终生成答案的质量。本文将从实际工程角度出发，详解几种主流分块策略的原理与实现，帮助你为自己的RAG系统选择最合适的方案。
为什么分块策略如此重要
RAG系统的核心流程是：先将知识库文档切分成小块（chunk），为每个块生成向量并存储，用户提问时检索最相关的块，再拼接上下文交给LLM生成答案。如果分块太大，向量表示会变得模糊，检索精度下降；分块太小，则丢失上下文语义，答案不完整。
举一个直观的例子：一篇技术文档中有一段关于”MySQL索引优化”的完整说明，如果按固定500字切块，可能正好把这段内容一分为二，导致检索时无法召回完整信息。因此，选择合理的分块策略是RAG工程化的第一步。
固定大小分块：简单但有局限
最简单的分块方式是按固定字符数或token数切分，通常设置一个重叠窗口（overlap）来缓解语义断裂问题。
2
3
4
5
6
7
8
9
10
11
<span class="n">text_splitter</span> <span class="o">=</span> <span class="n">CharacterTextSplitter</span><span class="p">(</span>
<span class="n">chunk_size</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span>
<span class="n">chunk_overlap</span><span class="o">=</span><span class="mi">50</span><span class="p">,</span>
<span class="n">separator</span><span class="o">=</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span>
<span class="p">)</span>
<span class="n">chunks</span> <span class="o">=</span> <span class="n">text_splitter</span><span class="o">.</span><span class="n">split_text</span><span class="p">(</span><span class="n">document_content</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"共切分为 </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">chunks</span><span class="p">)</span><span class="si">}</span><span class="s2"> 个块"</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"第1个块前100字: </span><span class="si">{</span><span class="n">chunks</span><span class="p">[</span><span class="mi">0</span><span class="p">][:</span><span class="mi">100</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
这种方式实现简单、速度快，适合结构化程度不高的文本（如聊天记录、日志）。但缺点也很明显：它不关心句子或段落的边界，可能把一句话从中间截断。
递归字符分块：尊重文本结构
递归字符分块是LangChain推荐的默认策略。它按层级分隔符（段落 → 换行 → 句号 → 空格）依次尝试切分，优先在自然断点处分块。
2
3
4
5
6
7
8
9
10
11
12
13
<span class="n">splitter</span> <span class="o">=</span> <span class="n">RecursiveCharacterTextSplitter</span><span class="p">(</span>
<span class="n">chunk_size</span><span class="o">=</span><span class="mi">800</span><span class="p">,</span>
<span class="n">chunk_overlap</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span>
<span class="n">separators</span><span class="o">=</span><span class="p">[</span><span class="s2">"</span><span class="se">\n\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"。"</span><span class="p">,</span> <span class="s2">"，"</span><span class="p">,</span> <span class="s2">" "</span><span class="p">,</span> <span class="s2">""</span><span class="p">]</span>
<span class="p">)</span>
<span class="n">chunks</span> <span class="o">=</span> <span class="n">splitter</span><span class="o">.</span><span class="n">split_text</span><span class="p">(</span><span class="n">document</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">chunks</span><span class="p">[:</span><span class="mi">3</span><span class="p">]):</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"--- Chunk </span><span class="si">{</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="si">}</span><span class="s2"> (长度: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">chunk</span><span class="p">)</span><span class="si">}</span><span class="s2">) ---"</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">chunk</span><span class="p">[:</span><span class="mi">150</span><span class="p">])</span>
<span class="nb">print</span><span class="p">()</span>
这种方法在大多数中文技术文档场景下表现良好，是快速上手RAG的首选方案。
语义分块：按含义切分
语义分块的核心思想是：相邻句子如果语义相似则归入同一块，语义差异大则切开。实现方式是先按句子拆分，计算相邻句子的向量余弦相似度，在相似度骤降处切分。
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
<span class="kn">from</span><span class="w"> </span><span class="nn">sentence_transformers</span><span class="w"> </span><span class="kn">import</span> <span class="n">SentenceTransformer</span>
<span class="k">def</span><span class="w"> </span><span class="nf">semantic_chunking</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">model_name</span><span class="o">=</span><span class="s2">"BAAI/bge-small-zh-v1.5"</span><span class="p">,</span> <span class="n">threshold</span><span class="o">=</span><span class="mf">0.5</span><span class="p">):</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">SentenceTransformer</span><span class="p">(</span><span class="n">model_name</span><span class="p">)</span>
<span class="c1"># 按中文句号拆分</span>
<span class="n">sentences</span> <span class="o">=</span> <span class="p">[</span><span class="n">s</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">text</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"。"</span><span class="p">)</span> <span class="k">if</span> <span class="n">s</span><span class="o">.</span><span class="n">strip</span><span class="p">()]</span>
<span class="n">embeddings</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">sentences</span><span class="p">)</span>
<span class="n">chunks</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">current_chunk</span> <span class="o">=</span> <span class="p">[</span><span class="n">sentences</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">sentences</span><span class="p">)):</span>
<span class="n">sim</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">embeddings</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">embeddings</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="o">/</span> \
<span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">norm</span><span class="p">(</span><span class="n">embeddings</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">norm</span><span class="p">(</span><span class="n">embeddings</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
<span class="k">if</span> <span class="n">sim</span> <span class="o"><</span> <span class="n">threshold</span><span class="p">:</span>
<span class="n">chunks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s2">"。"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">current_chunk</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"。"</span><span class="p">)</span>
<span class="n">current_chunk</span> <span class="o">=</span> <span class="p">[</span><span class="n">sentences</span><span class="p">[</span><span class="n">i</span><span class="p">]]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">current_chunk</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">sentences</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
<span class="k">if</span> <span class="n">current_chunk</span><span class="p">:</span>
<span class="n">chunks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s2">"。"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">current_chunk</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"。"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">chunks</span>
<span class="n">chunks</span> <span class="o">=</span> <span class="n">semantic_chunking</span><span class="p">(</span><span class="s2">"你的长文本内容..."</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"语义分块结果: 共 </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">chunks</span><span class="p">)</span><span class="si">}</span><span class="s2"> 个块"</span><span class="p">)</span>
语义分块的检索质量通常最优，但计算成本较高，适合对精度要求严格且文档量不大的场景。
基于文档结构的分块
对于有明确结构的文档（Markdown、HTML、PDF），利用标题层级进行分块是最自然的方式。每个标题及其下属内容作为一个完整的块，既保留了语义完整性，又附带了标题作为元数据。
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<span class="k">def</span><span class="w"> </span><span class="nf">markdown_chunking</span><span class="p">(</span><span class="n">md_text</span><span class="p">,</span> <span class="n">max_size</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""按Markdown标题层级分块"""</span>
<span class="n">sections</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="sa">r</span><span class="s1">'(?=^#{1,3} )'</span><span class="p">,</span> <span class="n">md_text</span><span class="p">,</span> <span class="n">flags</span><span class="o">=</span><span class="n">re</span><span class="o">.</span><span class="n">MULTILINE</span><span class="p">)</span>
<span class="n">sections</span> <span class="o">=</span> <span class="p">[</span><span class="n">s</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">sections</span> <span class="k">if</span> <span class="n">s</span><span class="o">.</span><span class="n">strip</span><span class="p">()]</span>
<span class="n">chunks</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">section</span> <span class="ow">in</span> <span class="n">sections</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">section</span><span class="p">)</span> <span class="o"><=</span> <span class="n">max_size</span><span class="p">:</span>
<span class="n">chunks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">section</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># 超长段落递归切分</span>
<span class="n">sub_splitter</span> <span class="o">=</span> <span class="n">RecursiveCharacterTextSplitter</span><span class="p">(</span>
<span class="n">chunk_size</span><span class="o">=</span><span class="n">max_size</span><span class="p">,</span> <span class="n">chunk_overlap</span><span class="o">=</span><span class="mi">100</span>
<span class="p">)</span>
<span class="n">chunks</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">sub_splitter</span><span class="o">.</span><span class="n">split_text</span><span class="p">(</span><span class="n">section</span><span class="p">))</span>
<span class="k">return</span> <span class="n">chunks</span>
<span class="c1"># 使用示例</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"technical_doc.md"</span><span class="p">,</span> <span class="s2">"r"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">doc</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">chunks</span> <span class="o">=</span> <span class="n">markdown_chunking</span><span class="p">(</span><span class="n">doc</span><span class="p">)</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">chunks</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">c</span><span class="p">[:</span><span class="mi">80</span><span class="p">],</span> <span class="s2">"..."</span><span class="p">)</span>
分块策略对比与选型建议
| 策略 | 优点 | 缺点 | 适用场景 |
|---|---|---|---|
| 固定大小 | 实现简单、速度快 | 语义断裂 | 日志、聊天记录 |
| 递归字符 | 平衡效果与速度 | 依赖分隔符选择 | 通用技术文档 |
| 语义分块 | 检索精度最高 | 计算开销大 | 高精度问答系统 |
| 结构化分块 | 保留文档结构 | 需要结构化输入 | Markdown/HTML文档 |
在实际项目中，建议采用混合策略：先用结构化分块处理有标题的文档，对超长段落再用递归字符分块兜底。同时设置合理的
1 | chunk_overlap |
（通常为块大小的10%-20%），避免边界处信息丢失。
实践中的优化技巧
最后分享几个工程实践中验证有效的优化点：
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">chunks</span><span class="p">):</span>
<span class="n">metadata</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"source"</span><span class="p">:</span> <span class="s2">"技术文档.pdf"</span><span class="p">,</span>
<span class="s2">"chunk_index"</span><span class="p">:</span> <span class="n">i</span><span class="p">,</span>
<span class="s2">"total_chunks"</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">chunks</span><span class="p">),</span>
<span class="s2">"word_count"</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">chunk</span><span class="p">)</span>
<span class="p">}</span>
<span class="c1"># 2. 使用Parent-Child策略：小块检索，大块喂给LLM</span>
<span class="n">small_chunks</span> <span class="o">=</span> <span class="n">splitter_small</span><span class="o">.</span><span class="n">split_text</span><span class="p">(</span><span class="n">doc</span><span class="p">)</span> <span class="c1"># 200字，用于检索</span>
<span class="n">big_chunks</span> <span class="o">=</span> <span class="n">splitter_big</span><span class="o">.</span><span class="n">split_text</span><span class="p">(</span><span class="n">doc</span><span class="p">)</span> <span class="c1"># 1000字，用于生成</span>
<span class="c1"># 3. 定期评估分块质量</span>
<span class="c1"># 可用检索命中率、答案准确率作为指标，迭代优化chunk_size</span>
分块没有万能方案，关键是根据你的文档类型和业务场景做实验、看数据、持续优化。