#parquet #lance #object-store

bin+lib parquet2lance

Convert parquet files to lance

26 releases

0.4.2 Jun 6, 2024
0.4.1 Mar 7, 2024
0.4.0 Dec 22, 2023
0.3.5 Nov 15, 2023
0.0.3 Mar 29, 2023

#262 in Authentication

Download history 172/week @ 2024-02-20 9/week @ 2024-02-27 128/week @ 2024-03-05 27/week @ 2024-03-12 3/week @ 2024-03-26 9/week @ 2024-04-02 190/week @ 2024-06-04

190 downloads per month

Custom license

31KB
383 lines

  • Convert parquet files to lance

  • Working with GCS

    • Using gcloud auth application-default login to generate Application Default Credentials (ADC)
parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/ \
  -o test.lance -O

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/us-states.parquet \
  -o test.lance -O
  • From GCS to GCS
parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/ \
  -o gs://your-bucket-name/bigquery/us-states/

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/us-states.parquet \
  -o gs://your-bucket-name/bigquery/us-states/us-states.lance
  • Working with local FS
gsutil cp gs://cloud-samples-data/bigquery/us-states/us-states.parquet test.parquet

parquet2lance -i test.parquet -o test.lance -O

Dependencies

~81MB
~1.5M SLoC