Lib.rs

› Science › Machine learning
#nlp #tokenizer #llm #python-packages #codegeex

bin+lib tokengeex

TokenGeeX is an efficient tokenizer for code based on UnigramLM and TokenMonster

by rojas-diego

11 releases (3 stable)

1.1.0 Jun 3, 2024
1.0.0 Apr 28, 2024
0.7.1 Apr 4, 2024
0.7.0 Mar 27, 2024

#573 in Machine learning

Download history 44/week @ 2024-03-28 141/week @ 2024-04-04 93/week @ 2024-04-11 147/week @ 2024-04-25 9/week @ 2024-05-02 133/week @ 2024-05-16 7/week @ 2024-05-23 144/week @ 2024-05-30 27/week @ 2024-06-06 3/week @ 2024-06-13 9/week @ 2024-06-27 56/week @ 2024-07-04

65 downloads per month

Apache-2.0

160KB
4K SLoC

Rust 3K SLoC // 0.1% comments Python 802 SLoC // 0.1% comments Shell 10 SLoC

TokenGeeX - Efficient Tokenizer for CodeGeeX

This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for CodeGeeX aimed at code and Chinese. It is based on UnigramLM (Taku Kudo 2018).

Dependencies

~9–18MB
~269K SLoC