Cargo Features
BPE has no features set by default.
[dependencies]
bpe = { version = "0.2.0", features = ["rand", "tiktoken"] }- rand
-
Enables rand
Affects
byte_pair_encoding::find_hash_factor_for_dictionary,byte_pair_encoding::create_test_string,byte_pair_encoding::create_test_string_with_predicate,byte_pair_encoding::select_test_string,byte_pair_encoding::create_test_bytes,byte_pair_encoding::find_hash_factor_for_tiktoken… - tiktoken
-
Enables base64
Affects
byte_pair_encoding::read_tiktoken,byte_pair_encoding::find_hash_factor_for_tiktoken…