#nlp #tokenizer #llm #python-packages #codegeex

bin+lib tokengeex

TokenGeeX is an efficient tokenizer for code based on UnigramLM and TokenMonster

10 releases (2 stable)

new 1.0.1 May 17, 2024
1.0.0 Apr 28, 2024
0.7.1 Apr 4, 2024
0.7.0 Mar 27, 2024
0.1.0 Feb 18, 2024

#389 in Machine learning

Download history 115/week @ 2024-02-12 222/week @ 2024-02-19 37/week @ 2024-02-26 6/week @ 2024-03-04 3/week @ 2024-03-11 227/week @ 2024-03-18 106/week @ 2024-03-25 146/week @ 2024-04-01 100/week @ 2024-04-08 115/week @ 2024-04-22 41/week @ 2024-04-29 123/week @ 2024-05-13

279 downloads per month

Apache-2.0

155KB
4K SLoC

Rust 3K SLoC // 0.1% comments Python 1K SLoC // 0.1% comments Shell 10 SLoC

TokenGeeX - Efficient Tokenizer for CodeGeeX

This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for CodeGeeX aimed at code and Chinese. It is based on UnigramLM (Taku Kudo 2018).

Dependencies

~9–20MB
~288K SLoC