libflatterer. Making JSON flatterer

Introduction

An opinionated JSON to CSV/XLSX/SQLITE/PARQUET converter which tries to make a useful relational output for data analysis.

Doucmentation For Flatterer, the python CLI/library built of this library. See these docs for high level information about flatterer.

Docs.rs Documentation

lib.rs:

libflatterer - Library to make JSON flatterer.

Mainly used for flatterer, which is a python library/cli using bindings to this library. Read flatterer documentation to give high level overview of how the flattening works. Nonetheless can be used as a standalone Rust library.

High level usage, use flatten function, supply a BufReader, an output directory and the Options struct (generated with the builder pattern):

use tempfile::TempDir;
use std::fs::File;
use libflatterer::{flatten, Options};
use std::io::BufReader;

let tmp_dir = TempDir::new().unwrap();
let output_dir = tmp_dir.path().join("output");
let options = Options::builder().xlsx(true).sqlite(true).parquet(true).table_prefix("prefix_".into()).build();

flatten(
   BufReader::new(File::open("fixtures/basic.json").unwrap()), // reader
   output_dir.to_string_lossy().into(), // output directory
   options, // options
).unwrap();

Lower level usage, use the FlatFiles struct directly and supply options.

 use tempfile::TempDir;
 use std::fs::File;
 use libflatterer::{FlatFiles, Options};
 use std::io::BufReader;
 use serde_json::json;

 let myjson = json!({
     "a": "a",
     "c": ["a", "b", "c"],
     "d": {"da": "da", "db": "2005-01-01"},
     "e": [{"ea": "ee", "eb": "eb2"},
           {"ea": "ff", "eb": "eb2"}],
 });

 let tmp_dir = TempDir::new().unwrap();
 let output_dir = tmp_dir.path().join("output");
 let options = Options::builder().xlsx(true).sqlite(true).parquet(true).table_prefix("prefix_".into()).build();

 // Create FlatFiles struct
 let mut flat_files = FlatFiles::new(
     output_dir.to_string_lossy().into(), // output directory
     options
 ).unwrap();

 // process JSON to memory
 flat_files.process_value(myjson.clone(), vec![]);

 // write processed JSON to disk. Do not need to do this for every processed value, but it is recommended.
 flat_files.create_rows();

 // copy the above two lines for each JSON object e.g..
 flat_files.process_value(myjson.clone(), vec![]);
 flat_files.create_rows();

 // ouput the selected formats
 flat_files.write_files();