#key-value-store #fst #search #big-data #string-matching #fuzzy-string

sys keyvi

Keyvi - the key value index. It is an in-memory FST-based data structure highly optimized for size and lookup performance.

13 releases

Uses old Rust 2015

0.5.6 Jul 22, 2022
0.5.3 May 19, 2021
0.5.1 Mar 20, 2021
0.5.0 Dec 12, 2020
0.3.0 Mar 28, 2018

#337 in Data structures

Download history 315/week @ 2024-03-13 339/week @ 2024-03-20 279/week @ 2024-03-27 233/week @ 2024-04-03 242/week @ 2024-04-10 257/week @ 2024-04-17 308/week @ 2024-04-24 347/week @ 2024-05-01 337/week @ 2024-05-08 228/week @ 2024-05-15 226/week @ 2024-05-22 272/week @ 2024-05-29 364/week @ 2024-06-05 313/week @ 2024-06-12 306/week @ 2024-06-19 241/week @ 2024-06-26

1,291 downloads per month

Apache-2.0

1.5MB
39K SLoC

C++ 38K SLoC // 0.1% comments Ruby HTML 883 SLoC Shell 341 SLoC // 0.1% comments Rust 277 SLoC // 0.3% comments KV Language 33 SLoC INI 7 SLoC Forge Config 3 SLoC JavaScript 2 SLoC

BuildBadge C++ PythonVersions PythonImpl PythonFormat PyPIVersion

Keyvi

Keyvi - the short form for "Key value index" is a key value store (KVS) optimized for size and lookup speed. The usage of shared memory makes it scalable and resistant. The biggest difference to other stores is the underlying data structure based on finite state machine. Storage is very space efficient, fast and by design makes various sorts of approximate matching be it fuzzy string matching or geo highly efficient. The immutable FST data structure can be used stand-alone for static datasets. If you need online writes, you can use keyvi index, a near realtime index. The index can be used as embedded key value store, e.g. if you already have a network stack in your application. A out of the box network enabled store is available with keyvi-server.

Introduction

Install

Quick

Precompiled binary wheels are available for OS X and Linux on PyPi. To install use:

pip install keyvi

From Source

The core part is a C++ header-only library, which can be used stand-alone. For more information check the Readme file in the keyvi subfolder.

The python extension can be compiled standalone, check the Readme file in the python subfolder for more information.

Usage

Internals

If you like to go deep down in the basics, keyvi is inspired by the following 2 papers:

Release procedure

License

keyvi is licensed under Apache License 2.0("ALv2"), see license for details, all 3rdparty libraries ship with their own license. Except Boost, Snappy and zlib all 3rdparty code can be exclusively found in the 3rdparty folder. The following licenses are used for the 3rdparty code (last updated: 0.5.0, provided without warranty).

C++ dependencies

Dependency License
Boost Boost Software License
moodycamel::ConcurrentQueue Simplified BSD License
md5 RSA MD5 License
msgpack-c Boost Software License
RapidJSON MIT License
Snappy BSD
Zlib Zlib License

Python dependencies

The python version ships with the same 3rdparty dependencies as the C++ code and additionaly depends on:

Dependency License
msgpack (for python) Apache License, Version 2.0

Credits

Thanks go to:

Dependencies

~0.7–3.5MB
~72K SLoC