#filter #json #big

datapan

datapan filters (big) files

1 unstable release

0.1.1 Feb 14, 2020

#280 in #filter

GPL-3.0 license

13KB

datapan

Build Lifecycle PyPI Crates.io


This is still a test bed. It is not useful

datapan sifts through enormous files in parallelized Rust to only grab the data you want as quickly and memory-efficiently as possilbe.


Installation

## create/activate venv
# sudo apt-get install python3-venv
# python3 -m venv datapan_env
# source datapan_env/bin/activate
# python -m pip install --upgrade pip

## install datapan
pip install datapan

Usage

import datapan

some_dir = ""

test = datapan.hello_rust(some_dir)

print(test)

Developer Version

  • Rust (nightly)
curl https://sh.rustup.rs -sSf | sh
# rustup default nightly
rustup update nightly
  • Poetry
pip install poetry
make install
make test

Dependencies

~8.5MB
~166K SLoC