#json #smoothing #flatten #parquet #json-object #unnest

smooth-json

smooth-json is an opinionated, customizable utility to flatten serde_json Value variants into serde_json Objects ready for use in columnar or table-like usages

8 releases

new 0.2.6 Apr 12, 2024
0.2.5 Apr 2, 2024
0.2.4 Mar 19, 2024
0.1.0 Mar 15, 2024

#256 in Data structures

Download history 331/week @ 2024-03-11 150/week @ 2024-03-18 153/week @ 2024-04-01

634 downloads per month

MIT license

21KB
460 lines

smooth-json

Publish Status

This crate allows for flattening JSON objects into objects ready for use in Parquet, CSV, or other data formats.

The flattening is similar to ElasticSearch's ingestion flattening or what would be needed for VAST's DB and Table integrations.

Features

  • Flatten serde_json's Value variants into structures suitable for use with applications that are expecting table formatted data or columnar data formats.
  • Pass a custom separator by instantiating a Flattener and passing the separator.
  • Use an opinionated flattening format that places values in an array if the object is nested inside an array

Examples

Standard Usage

use smooth_json;
use serde_json::json;

fn main() {
    let flattener = smooth_json::Flattener::new();

    let example = json!({
        "name": "John Doe",
        "age": 43,
        "address": {
            "street": "10 Downing Street",
            "city": "London"
        },
        "phones": [
            "+44 1234567",
            "+44 2345678"
        ]
    });

    let flattened_example = flattener.flatten(&example);
    
    println!("{}", flattened_example);
    /*
    {
        "address.city": "London",
        "address.street": "10 Downing Street",
        "age": 43,
        "name": "John Doe",
        "phones": [
            "+44 1234567",
            "+44 2345678"
        ]
    }
    */
}

Custom Separator Usage

use serde_json::json;
use smooth_json;

fn main() {
    let flattener = smooth_json::Flattener{ 
        separator: "$", 
        ..Default::default()
    };

    let example = json!({
        "a": {
            "b": 1
        }});

    let flattened_example = flattener.flatten(&example);

    println!("{}", flattened_example);
    /*
    {
        "a$b": 1
    }
    */
}

Opinionated Flattening

If an object is present in an array, the result will be in an array when flattened.

use serde_json::json;
use smooth_json;

fn main() {
    let flattener = smooth_json::Flattener{ 
        alt_array_flattening: true,
        ..Default::default()
    };

    let example = json!({
          "a": [
            ["b", "c"],
            { "d": "e" },
            ["f", "g"],
            [
                { "h": "i" },
                { "d": "j" },
            ],
            ["k", "l"],
          ]
        });

    let flattened_example = flattener.flatten(&example);

    println!("{}", flattened_example);
    /*
    {
        "a": ["b", "c", "f", "g", "k", "l"],
        "a.d": ["e", "j"],
        "a.h": ["i"],
    }
    */
}

Completely Flat JSON (keep array position in key)

use serde_json::json;
use smooth_json;

fn main() {
    let flattener = smooth_json::Flattener{ 
        preserve_arrays: true,
        ..Default::default()
    };

    let example: Value = json!({
        "a": [
            "b",
            ["c", "d"],
        ]
    });

    let flattened_example = flattener.flatten(&example);
    println!("{}", flattened_example);
    /*
    {
        "a.0": "b",
        "a.1.0": "c",
        "a.1.1": "d"
    }
    */

Dependencies

~360–760KB
~17K SLoC