#data-format #data-science #machine-learning #apache-arrow #data-analytics

bin+lib lance

A columnar data format that is 100x faster than Parquet for random access

14 releases

new 0.3.19 Mar 27, 2023
0.3.8 Feb 24, 2023
0.1.4 Jan 27, 2023
0.0.1-alpha0 Jul 28, 2022

#48 in Machine learning

Download history 1/week @ 2022-12-02 1/week @ 2022-12-09 1/week @ 2022-12-16 1/week @ 2022-12-23 68/week @ 2023-01-06 24/week @ 2023-01-13 14/week @ 2023-01-20 34/week @ 2023-01-27 78/week @ 2023-02-03 124/week @ 2023-02-10 81/week @ 2023-02-17 32/week @ 2023-02-24 7/week @ 2023-03-03 4/week @ 2023-03-10 13/week @ 2023-03-17

59 downloads per month
Used in parquet2lance


12K SLoC

Rust Implementation of Lance Data Format

⚠️ Under heavy development

Lance Logo

A new columnar data format for data science and machine learning


Install using cargo:

cargo install lance


Create dataset

Suppose batches is an Arrow RecordBatchBuffer:

use ::lance::dataset::WriteParams;
use ::lance::dataset::Dataset;

let mut write_params = WriteParams::default();
let mut reader: Box<dyn RecordBatchReader> = Box::new(batches);
Dataset::write(&mut reader, test_uri, Some(write_params)).await?;


let dataset = Dataset::open(path).await.unwrap();
let mut scanner = dataset.scan();
let batches: Vec<RecordBatch> = scanner
    .map(|b| b.unwrap())


let values: Result<RecordBatch> = dataset.take(&[200, 199, 39, 40, 100], &projection).await;

Vector index

Assume "embeddings" is a FixedSizeListArray

use ::lance::index::vector::VectorIndexParams;

let params = VectorIndexParams::default();
params.num_partitions = 256;
params.num_sub_vectors = 16;

// this will Err if list_size(embeddings) / num_sub_vectors does not meet simd alignment
dataset.create_index(&["embeddings"], IndexType::Vector, None, &params, true).await;


Why do we need a new format for data science and machine learning?

1. Reproducibility is a must-have

Versioning and experimentation support should be built into the dataset instead of requiring multiple tools.
It should also be efficient and not require expensive copying everytime you want to create a new version.
We call this "Zero copy versioning" in Lance. It makes versioning data easy without increasing storage costs.

2. Cloud storage is now the default

Remote object storage is the default now for data science and machine learning and the performance characteristics of cloud are fundamentally different.
Lance format is optimized to be cloud native. Common operations like filter-then-take can be order of magnitude faster using Lance than Parquet, especially for ML data.

3. Vectors must be a first class citizen, not a separate thing

The majority of reasonable scale workflows should not require the added complexity and cost of a specialized database just to compute vector similarity. Lance integrates optimized vector indices into a columnar format so no additional infrastructure is required to get low latency top-K similarity search.

4. Open standards is a requirement

The DS/ML ecosystem is incredibly rich and data must be easily accessible across different languages, tools, and environments. Lance makes Apache Arrow integration its primary interface, which means conversions to/from is 2 lines of code, your code does not need to change after conversion, and nothing is locked-up to force you to pay for vendor compute. We need open-source not fauxpen-source.


~2.5M SLoC