Cargo Features
TextSplitter has no features set by default.
[dependencies]
text-splitter = { version = "0.18.1", features = ["code", "markdown", "rust-tokenizers", "tiktoken-rs", "tokenizers"] }- code
-
Enables tree-sitter
- markdown
-
Enables pulldown-cmark
- rust-tokenizers
-
Enables rust_tokenizers
- tiktoken-rs
-
Enables tiktoken-rs
- tokenizers
-
Enables onig of tokenizers
TextSplitter has 5 features without comments.