Lib.rs

› Authentication
#parquet #lance #object-store

bin+lib parquet2lance

Convert parquet files to lance

by Xin Hao

28 releases

0.4.4 Jul 30, 2024
0.4.2 Jun 6, 2024
0.4.1 Mar 7, 2024
0.4.0 Dec 22, 2023
0.0.3 Mar 29, 2023

#283 in Authentication

Download history 132/week @ 2024-07-30 18/week @ 2024-09-10 7/week @ 2024-09-17 16/week @ 2024-09-24 4/week @ 2024-10-01

1,410 downloads per month

Custom license

35KB
383 lines

  • Convert parquet files to lance

  • Working with GCS

    • Using gcloud auth application-default login to generate Application Default Credentials (ADC)
parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/ \
  -o test.lance -O

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/us-states.parquet \
  -o test.lance -O
  • From GCS to GCS
parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/ \
  -o gs://your-bucket-name/bigquery/us-states/

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/us-states.parquet \
  -o gs://your-bucket-name/bigquery/us-states/us-states.lance
  • Working with local FS
gsutil cp gs://cloud-samples-data/bigquery/us-states/us-states.parquet test.parquet

parquet2lance -i test.parquet -o test.lance -O

Dependencies

~95MB
~2M SLoC