nitonitori
1ba326279f
- 重新实现Markdown追加分片，确保每个分片为完整独立的顶级区块序列，满足update_document追加模式要求 - 修复分片过程中丢失换行符导致标题等Markdown元素识别错误的问题 - 解决超大表格和代码块被截断在单元格或代码围栏中间的问题 - 修正读取回写验证，避免与服务器未接收的内容比较导致验证失败 - 统一多条Markdown写入路径，统一分片逻辑与分片大小限制（30000字符） - doc update --index参数在内容需分片时改为失败，避免不可预期的插入位置 - 添加分片写入过程中的降级变更通知字段，明确分片对文档结构的影响 - 优化写入流水线，增强超长内容自动分片支持和错误处理逻辑 - 补充分片相关的单元测试，保障内容完整性及边界拆分行为
2.8 KiB
2.8 KiB
category
| category |
|---|
| Fixed |
- Markdown append chunking rewritten around safe split positions — long markdown is now split so that every chunk is a complete, self-contained top-level block sequence, which is what
update_document mode=appendrequires: the server inserts a brand new structure per call and cannot continue the previous one. Split points are chosen strictly by how much they change the rendered document — fully safe boundaries (blank lines, block starts that interrupt a paragraph) before boundaries that need repair (a table's rows now carry a re-emitted header and delimiter row; a fenced code block is closed and reopened with its original marker and info string) before boundaries that merely restructure (long paragraphs, list items) before a hard character cut. Within a tier the latest boundary in the window wins, since all chunks land in the same document. Every boundary that changes the rendered structure is reported in a newdegradationsfield instead of being applied silently. - Fixed markdown chunking dropping a newline — the previous splitter rebuilt block text from lines and lost one
\nwhenever the content's last line began a heading, table or code fence, so"para\n# Title"was written as"para# Title"and the heading stopped being a heading. Roughly one in five randomly generated documents was affected. The new splitter slices by offset and never rebuilds text, making content preservation structural. - Fixed oversized tables and code blocks being cut mid-cell and mid-fence — the hard-split path never received the block type, so it cut at arbitrary character boundaries despite claiming to preserve table and code block integrity.
- Fixed readback verification comparing against content the server never receives —
doc +create/doc +updateverified the readback against the raw input, so any repaired boundary (and, previously, any paragraph split) failed verification on large documents. Verification now compares against the document the chunk plan says the server should hold. - Unified four markdown write paths onto one splitter —
doc create/doc update,doc +create/doc +updateanddoc +checkpoint-updatenow sharehelpers.SplitMarkdownForAppendand one limit constant (30000 runes), replacing two independent implementations plus one path that never chunked at all.doc +checkpoint-updateaccepts@fileand stdin content, so oversized input was reachable there while the equivalentdoc +updatechunked.doc +doc-appendtakes--textfrom argv only and now rejects oversized input with a pointer todoc +updaterather than sending one oversized call. doc update --indexnow fails closed when the content requires chunking — each chunk creates an unpredictable number of blocks, so the insertion point for later chunks is unknowable; the flag was previously accepted and silently ignored.