#pytorch #deep-learning #machine-learning #tokenizer #sentencizer

nnsplit

A tool to split text using a neural network. For sentence boundary detection, compound splitting and more.

28 releases

new 0.5.8 Jul 23, 2021
0.5.7 Mar 16, 2021
0.5.5 Feb 13, 2021
0.5.2 Nov 1, 2020
0.2.2 Feb 26, 2020

#26 in Machine learning

Download history 36/week @ 2021-04-08 112/week @ 2021-04-15 5/week @ 2021-04-22 33/week @ 2021-04-29 1/week @ 2021-05-06 34/week @ 2021-05-13 6/week @ 2021-05-27 15/week @ 2021-06-03 27/week @ 2021-06-10 8/week @ 2021-06-17 16/week @ 2021-06-24 9/week @ 2021-07-01 1/week @ 2021-07-08 64/week @ 2021-07-15 32/week @ 2021-07-22

74 downloads per month

MIT license

30KB
665 lines

NNSplit

PyPI Crates.io npm CI License

A tool to split text using a neural network. The main application is sentence boundary detection, but e. g. compound splitting for German is also supported.

Features

  • Robust: Not reliant on proper punctuation, spelling and case. See the metrics.
  • Small: NNSplit uses a byte-level LSTM, so weights are small (< 4MB) and models can be trained for every unicode encodable language.
  • Portable: NNSplit is written in Rust with bindings for Rust, Python, and Javascript (Browser and Node.js). See how to get started in the usage section.
  • Fast: Up to 2x faster than Spacy sentencization, see the benchmark.
  • Multilingual: NNSplit currently has models for 9 different languages (German, English, French, Norwegian, Swedish, Simplified Chinese, Turkish, Russian and Ukrainian). Try them in the demo.

Documentation has moved to the NNSplit website: https://bminixhofer.github.io/nnsplit.

License

NNSplit is licensed under the MIT license.

Dependencies

~2–4.5MB
~95K SLoC