Lib.rs

› Authentication
#lance #parquet #object-store

bin+lib parquet2lance

Convert parquet files to lance

by Xin Hao

29 releases

0.5.0 Jan 24, 2025
0.4.4 Jul 30, 2024
0.4.2 Jun 6, 2024
0.4.1 Mar 7, 2024
0.0.3 Mar 29, 2023

#261 in Authentication

Download history 2/week @ 2024-12-03 65/week @ 2024-12-10 105/week @ 2025-01-21 6/week @ 2025-01-28 6/week @ 2025-02-04

117 downloads per month

Custom license

38KB
383 lines

  • Convert parquet files to lance

  • Working with GCS

    • Using gcloud auth application-default login to generate Application Default Credentials (ADC)
parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/ \
  -o test.lance -O

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/us-states.parquet \
  -o test.lance -O
  • From GCS to GCS
parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/ \
  -o gs://your-bucket-name/bigquery/us-states/

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/us-states.parquet \
  -o gs://your-bucket-name/bigquery/us-states/us-states.lance
  • Working with local FS
gsutil cp gs://cloud-samples-data/bigquery/us-states/us-states.parquet test.parquet

parquet2lance -i test.parquet -o test.lance -O

Dependencies

~83–115MB
~2M SLoC