1 unstable release
0.1.0 | Oct 7, 2024 |
---|
#1119 in Algorithms
150 downloads per month
165KB
1K
SLoC
OpenAI Byte Pair Encoders
Fast tokenizers for OpenAI token sets based on the bpe crate.
Serialized BPE instances are generated during build and lazily loaded at runtime as static values.
The overhead of loading the tokenizers is small because it happens only once per process and only requires deserialization (as opposed to actually building the internal data structures).
For convencience it re-exports the bpe
crate so that depending on this crate is enough to use these tokenizers.
Supported token sets:
- r50k
- p50k
- cl100k
- o200k
Usage
Add a dependency by running
cargo add bpe-openai
or by adding the following to Cargo.toml
[dependencies]
bpe-openai = "0.1"
Counting tokens is as simple as:
use bpe_openai::cl100k;
fn main() {
let bpe = cl100k();
let count = bpe.count("Hello, world!");
println!("{tokens}");
}
For more detailed documentation we refer to bpe.
Dependencies
~1.3–4MB
~51K SLoC