parquet2lance

25 releases

0.4.1	Mar 7, 2024
0.4.0	Dec 22, 2023
0.3.5	Nov 15, 2023
0.2.6	May 31, 2023
0.0.3	Mar 29, 2023

#224 in Authentication

39 downloads per month

Custom license

31KB
383 lines

Convert parquet files to lance
Working with GCS
- Using gcloud auth application-default login to generate Application Default Credentials (ADC)

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/ \
  -o test.lance -O

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/us-states.parquet \
  -o test.lance -O

From GCS to GCS

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/ \
  -o gs://your-bucket-name/bigquery/us-states/

parquet2lance \
  -i gs://cloud-samples-data/bigquery/us-states/us-states.parquet \
  -o gs://your-bucket-name/bigquery/us-states/us-states.lance

Working with local FS

gsutil cp gs://cloud-samples-data/bigquery/us-states/us-states.parquet test.parquet

parquet2lance -i test.parquet -o test.lance -O

Dependencies

~78MB
~1.5M SLoC