8 breaking releases
0.12.0 | Sep 22, 2024 |
---|---|
0.10.0 | Jun 24, 2024 |
0.7.0 | Mar 7, 2024 |
0.6.0 | Nov 14, 2023 |
#1937 in Parser implementations
431 downloads per month
44KB
559 lines
Parquet code generation for Rust
This project provides tools for generating Rust code to work with Parquet files using the Rust implementation of Arrow.
It includes both a code generation crate (parquetry-gen
) and a small runtime library required by the generated code (parquetry
).
Please note that this software is not "open source", but the source is available for use and modification by individuals, non-profit organizations, and worker-owned businesses (see the license section below for details).
Table of contents
Example
Given a schema like this:
message user {
required int64 id (integer(64, false));
required int64 ts (timestamp(millis, true));
optional int32 status;
optional group user_info {
required byte_array screen_name (string);
optional group user_name_info {
required byte_array name (string);
optional group user_profile_info {
required int64 created_at (timestamp(millis, true));
required byte_array location (string);
required byte_array description (string);
optional byte_array url (string);
required int32 followers_count;
required int32 friends_count;
required int32 favourites_count;
required int32 statuses_count;
optional group withheld_in_countries (list) {
repeated group list {
required byte_array element (string);
}
}
}
}
}
}
The code generator will produce the following Rust structs:
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
pub struct User {
pub id: u64,
#[serde(with = "chrono::serde::ts_milliseconds")]
pub ts: chrono::DateTime<chrono::Utc>,
pub status: Option<i32>,
pub user_info: Option<UserInfo>,
}
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
pub struct UserInfo {
pub screen_name: String,
pub user_name_info: Option<UserNameInfo>,
}
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
pub struct UserNameInfo {
pub name: String,
pub user_profile_info: Option<UserProfileInfo>,
}
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
pub struct UserProfileInfo {
#[serde(with = "chrono::serde::ts_milliseconds")]
pub created_at: chrono::DateTime<chrono::Utc>,
pub location: String,
pub description: String,
pub url: Option<String>,
pub followers_count: i32,
pub friends_count: i32,
pub favourites_count: i32,
pub statuses_count: i32,
pub withheld_in_countries: Option<Vec<String>>,
}
It will also generate an instance of the parquetry::Schema
trait for User
with the code for reading and writing values to Parquet files.
Dependencies
All usage requires the use of the parquetry
, parquet
, chrono
, and lazy_static
crates as runtime dependencies.
If the serde_support
flag is enabled in configuration (which it is by default),
you will also need a dependency on serde
with the derive
feature enabled.
If the tests
flag is enabled in configuration (also the default),
you will need to add bincode
(a recent 2.0.0-rc.x
with the serde
feature enabled),
tempdir
, and quickcheck
to your dev-dependencies
.
Usage
The example
directory provides a fairly minimal example, and the generated code is checked in there.
In most cases a build.rs
like the following should be all you need:
use std::{fs::File, io::Write};
fn main() -> Result<(), parquetry_gen::error::Error> {
for schema in parquetry_gen::ParsedFileSchema::open_dir(
"src/schemas/",
Default::default(),
Some(".parquet.txt"),
)? {
println!("cargo:rerun-if-changed={}", schema.absolute_path_str()?);
let mut output = File::create(format!("src/{}.rs", schema.name))?;
write!(output, "{}", schema.code()?)?;
}
Ok(())
}
By default the generated code is formatted with prettyplease
and is annotated to indicate that it should not be formatted by Rustfmt,
but if you'd prefer to use Rustfmt yourself, you can set format
to false in the configuration.
Testing
The default configuration will generate test code that uses QuickCheck to generate arbitrary values and confirm that they serialize and deserialize correctly.
The test code generation currently does not support some types.
Specifically, sequences of floating point numbers, fixed-length byte arrays, and timestamps are not supported.
If your schema contains any of these types, the generated test code will simply not compile, and you will have to disable the tests
flag in your configuration
(you can also open an issue if you'd like).
The generated test code does not produce NaN
values for floating point types.
If you want to confirm that your system handles these values correctly, you'll have to do that manually.
By default the generated arbitrary values for your types may be very large.
You may want to set the QUICKCHECK_GENERATOR_SIZE
environment variable to a small value (e.g. 10
or 20
) if your tests are too slow.
Status and scope
These tools support schemas with most physical and logical types, and with arbitrary nestings of lists, optional fields, and structures.
Missing features that I might add at some point:
- 8- and 16-bit logical integer types (trivial, I just haven't needed them)
DATE
,TIME
,INTERVAL
, andUUID
(same as previous)DECIMAL
(at least for theINT32
andINT64
representations)ENUM
(not really useful in this context since the schema doesn't enumerate the variants?)- Maps (a little more work but probably worth having)
Features that will probably never be supported:
- The
INT96
physical type (which has been deprecated) - Timestamps with local semantics (
isAdjustedToUTC = false
) - Legacy list shapes
- Legacy map shapes
- A way to avoid the
chrono
dependency or support for other time libraries
This project differs from parquet_derive
in a few ways:
- Both generate reading and writing code, but this project generates Rust structs from the schema, as opposed to the reverse.
- This project does not use
parquet::record::RecordWriter
(it just didn't seem all that useful and I wanted more flexibility). - This project supports nested structures.
In general the two projects have different use cases, and if you just want to store some Rust values in Parquet, I'd recommend choosing parquet_derive
.
Warnings
Name collisions
There's currently no special handling for field names that collide with Rust keywords, names from the standard library, etc. Group names should also be unique within a schema. The code generator also produces non-public structs with names that could in theory collide with user-generated code.
In most of these cases the problems should be immediately obvious, since the generated code will generally just not compile. It wouldn't be hard to check for these collisions and provide better errors, or to allow more user control over naming to avoid these issues, but this hasn't been a priority for me.
Constructors
The generated code includes fn new
constructors for each struct that will truncate the precision of any DateTime<Utc>
to the number of subsecond digits
supported by the column representation. These constructors will also check whether any string arguments contain null bytes, and will return an error if they do.
If you don't want either behavior, you can construct the structs manually, as all fields are always public.
Serde instances
By default the generated code will include derived Serde instances for serialization.
These instances will use the time unit specified by the schema (millisecond or microsecond) for fields with the type DateTime<Utc>
or Option<DateTime<Utc>>
,
but fields of type Vec<DateTime<Utc>>
will use the default Serde serialization encoding for DateTime<Utc>
.
There's no particular reason for this beyond the fact that chrono::serde
only provides e.g. ts_milliseconds
and ts_milliseconds_option
functions,
and the runtime library could easily provide its own ts_milliseconds_vec
that would be used in these cases.
Performance
I haven't made any attempt to optimize the generated code and probably won't until performance becomes an issue for my use cases.
License
This software is published under the Anti-Capitalist Software License (v. 1.4).
Dependencies
~48MB
~1M SLoC