Code review of the Tokenizers crate // Lib.rs

This review is from cargo-vet. To add your review, set up cargo-vet and submit your URL to its registry.

The current version of Tokenizers is 0.21.0.

0.20.1 (older version) ub-risk-3

From google/supply-chain copy of google/rust-crate-audits. By Manish Goregaokar.

Reviewed in CL 684450749 Issues found:

underdocumented safety invariants in cases that need more documentation (PR in https://github.com/huggingface/tokenizers/pull/1651)

cargo-vet does not verify reviewers' identity. You have to fully trust the source the audits are from.

ub-risk-4

Extreme unsoundness.

Full description of the audit criteria can be found at https://github.com/google/rust-crate-audits/blob/main/auditing_standards.md#ub-risk-4

ub-risk-3 (implies ub-risk-4)

Mild unsoundness or suboptimal soundness.

Full description of the audit criteria can be found at https://github.com/google/rust-crate-audits/blob/main/auditing_standards.md#ub-risk-3

Lib.rs has been able to verify that all files in the crate's tarball, except Cargo.lock, are in the crate's repository with a git tag matching the version. Please note that this check is still in beta, and absence of this confirmation does not mean that the files don't match.

Crates in the crates.io registry are tarball snapshots uploaded by crates' publishers. The registry is not using crates' git repositories, so there is a possibility that published crates have a misleading repository URL, or contain different code from the code in the repository.

To review the actual code of the crate, it's best to use cargo crev open tokenizers. Alternatively, you can download the tarball of tokenizers v0.21.0 or view the source online.