Transfer data between the Arrow memory format and JSON line-delimited records.

See the module level documentation for the reader and writer for usage examples.

Binary Data uses Base16 Encoding

As per RFC7159 JSON cannot encode arbitrary binary data. This crate works around that limitation by encoding/decoding binary data as a hexadecimal string (i.e. Base16 encoding).

Note that Base16 only has 50% space efficiency (i.e., the encoded data is twice as large as the original). If that is an issue, we recommend to convert binary data to/from a different encoding format such as Base64 instead. See the following example for details.

Base64 Encoding Example

Base64 is a common binary-to-text encoding scheme with a space efficiency of 75%. The following example shows how to use the arrow_cast crate to encode binary data to Base64 before converting it to JSON and how to decode it back.

use arrow_cast::base64::{b64_decode, b64_encode, BASE64_STANDARD};
#
// The data we want to write
let input = BinaryArray::from(vec![b"\xDE\x00\xFF".as_ref()]);

// Base64 encode it to a string
let encoded: StringArray = b64_encode(&BASE64_STANDARD, &input);

// Write the StringArray to JSON
let batch = RecordBatch::try_from_iter([("col", Arc::new(encoded) as _)]).unwrap();
let mut buf = Vec::with_capacity(1024);
let mut writer = LineDelimitedWriter::new(&mut buf);
writer.write(&batch).unwrap();
writer.finish().unwrap();

// Read the JSON data
let cursor = Cursor::new(buf);
let mut reader = ReaderBuilder::new(batch.schema()).build(cursor).unwrap();
let batch = reader.next().unwrap().unwrap();

// Reverse the base64 encoding
let col: BinaryArray = batch.column(0).as_string::<i32>().clone().into();
let output = b64_decode(&BASE64_STANDARD, &col).unwrap();

assert_eq!(input, output);