1 unstable release
| new 0.0.0 | Dec 8, 2024 |
|---|
#42 in #discrete
2KB
Lightweight, blazingly fast, zero-copy tokenizer designed to efficiently break down text streams into discrete tokens based on developer-defined rules
| new 0.0.0 | Dec 8, 2024 |
|---|
#42 in #discrete
2KB
Lightweight, blazingly fast, zero-copy tokenizer designed to efficiently break down text streams into discrete tokens based on developer-defined rules