Cargo Features
TextSplitter has no features set by default.
[dependencies]
text-splitter = { version = "0.25.1", features = ["code", "markdown", "tiktoken-rs", "tokenizers"] }- code
-
Enables tree-sitter
- markdown
-
Enables pulldown-cmark
- tiktoken-rs
-
Enables tiktoken-rs
- tokenizers
-
Enables onig of tokenizers
TextSplitter has 4 features without comments.