11 releases

0.4.2 Apr 1, 2021
0.4.1 Apr 1, 2021
0.4.0 Aug 4, 2020
0.3.0 May 28, 2020
0.1.4 Jan 8, 2020

#870 in Algorithms

Download history 142/week @ 2024-06-16 154/week @ 2024-06-23 123/week @ 2024-06-30 119/week @ 2024-07-07 179/week @ 2024-07-14 189/week @ 2024-07-21 305/week @ 2024-07-28 185/week @ 2024-08-04 224/week @ 2024-08-11 211/week @ 2024-08-18 258/week @ 2024-08-25 131/week @ 2024-09-01 149/week @ 2024-09-08 209/week @ 2024-09-15 202/week @ 2024-09-22 436/week @ 2024-09-29

1,003 downloads per month
Used in textspan

MIT license

27KB
270 lines

Robust and Fast tokenizations alignment library for Rust and Python

creates.io pypi Actions Status

sample

Demo: demo
Rust document: docs.rs
Blog post: How to calculate the alignment between BERT and spaCy tokens effectively and robustly

Usage (Python)

  • Installation
$ pip install -U pip # update pip
$ pip install pytokenizations
  • Install from source

This library uses maturin to build the wheel.

$ git clone https://github.com/tamuhey/tokenizations
$ cd tokenizations/python
$ pip install maturin
$ maturin build

Now the wheel is created in python/target/wheels directory, and you can install it with pip install *whl.

get_alignments

def get_alignments(a: Sequence[str], b: Sequence[str]) -> Tuple[List[List[int]], List[List[int]]]: ...

Returns alignment mappings for two different tokenizations:

>>> tokens_a = ["å", "BC"]
>>> tokens_b = ["abc"] # the accent is dropped (å -> a) and the letters are lowercased(BC -> bc)
>>> a2b, b2a = tokenizations.get_alignments(tokens_a, tokens_b)
>>> print(a2b)
[[0], [0]]
>>> print(b2a)
[[0, 1]]

a2b[i] is a list representing the alignment from tokens_a to tokens_b.

Usage (Rust)

See here: docs.rs

Dependencies

~1MB
~29K SLoC