Cargo Features
CodeSplitter has no features set by default.
[dependencies]
code-splitter = { version = "0.1.5", features = ["tiktoken-rs", "tokenizers"] }- tiktoken-rs
-
Enables tiktoken-rs ^0.5.9
- tokenizers
-
Enables tokenizers ^0.19.1
CodeSplitter has 2 features without comments.