#thrift #parquet #hadoop #temporary #compiled #async #define #read #apache #reader-writer

parquet-format-async-temp

Temporary crate containing thrift library + parquet definitions compiled to support read+write async

5 unstable releases

0.3.1 Jun 14, 2022
0.3.0 Mar 15, 2022
0.2.0 Aug 28, 2021
0.1.1 Aug 10, 2021
0.1.0 Aug 8, 2021

#1512 in Database interfaces

Download history 305/week @ 2025-03-24 262/week @ 2025-03-31 140/week @ 2025-04-07 316/week @ 2025-04-14 298/week @ 2025-04-21 287/week @ 2025-04-28 127/week @ 2025-05-05 118/week @ 2025-05-12 44/week @ 2025-05-19 116/week @ 2025-05-26 162/week @ 2025-06-02 160/week @ 2025-06-09 178/week @ 2025-06-16 253/week @ 2025-06-23 173/week @ 2025-06-30 409/week @ 2025-07-07

1,024 downloads per month

Apache-2.0

465KB
10K SLoC

parquet-format-async-temp

This is a temporary crate containing a subset of rust's thirft library and parquet to support native async parquet read and write.

Specifically, it:

  • supports async read API (via futures)
  • supports async write API (via futures)
  • the write API returns the number of written bytes

It must be used with the fork of thrift's compiler available at https://github.com/jorgecarleitao/thrift/tree/write_size .

Why

To read and write files with thrift (e.g. parquet) without commiting to a particular runtime (e.g. tokio, hyper, etc.), the protocol needs to support AsyncRead + AsyncSeek and AsyncWrite respectively.

To not require Seek and AsyncSeek on write, the protocol must return the number of written bytes on its write_* API.

This crate addresses these two concerns for parquet. It is essentially:

Dependencies

~1–1.7MB
~34K SLoC