#fluvio #json #java

fluvio-jolt

Fluvio JSON to JSON transformation library

2 releases

Uses new Rust 2021

0.1.1 Nov 22, 2022
0.1.0 Aug 3, 2022

#434 in Development tools

Download history 3/week @ 2022-08-11 1/week @ 2022-08-18 3/week @ 2022-08-25 4/week @ 2022-09-08 1/week @ 2022-09-22 3/week @ 2022-09-29 2/week @ 2022-10-06 1352/week @ 2022-10-13 1868/week @ 2022-10-20 875/week @ 2022-10-27 689/week @ 2022-11-03 940/week @ 2022-11-10 792/week @ 2022-11-17 98/week @ 2022-11-24

2,743 downloads per month

Apache-2.0

38KB
771 lines

Fluvio Jolt JSON library

JSON to JSON transformation Rust library

Overview

JSON to JSON transformation where the "specification" for the transform is itself a JSON document.

Port of Java Jolt library written in Rust.

Usage Example

Add fluvio-jolt crate to your Cargo.toml file:

[dependencies]
fluvio-jolt = { version = "0.1"}

Then, for example, if you want to repack your JSON record, you can do the following:

use serde_json::{json, Value};
use fluvio_jolt::{transform, TransformSpec};

let input: Value = serde_json::from_str(r#"
    {
        "id": 1,
        "name": "John Smith",
        "account": {
            "id": 1000,
            "type": "Checking"
        }
    }
"#).unwrap();

let spec: TransformSpec =
serde_json::from_str(r#"[
    {
      "operation": "shift",
      "spec": {
        "name": "data.name",
        "account": "data.account"
      }
    }
  ]"#).unwrap();

let output = transform(input, &spec);

assert_eq!(output, json!({
    "data" : {
      "name": "John Smith",
      "account": {
        "id": 1000,
        "type": "Checking"
      }
    }
}));

Supported Operations

  1. shift: copy data from the input tree and put it the output tree
  2. default: apply default values to the tree
  3. remove: remove data from the tree

Specification

Composes a list of operation specifications. Each operation has its own DSL (Domain Specific Language) in order to facilitate its narrow job.

use fluvio_jolt::TransformSpec;

let spec: TransformSpec =
serde_json::from_str(r#"[
    {
      "operation": "shift",
      "spec": {
        "name": "data.name",
        "account": "data.account"
      }
    }
  ]"#).unwrap();

Shift operation

Specifies where the data from the input JSON should be placed in the output JSON, or in other words, how the input JSON/data should be shifted around to make the output JSON/data.

At a base level, a single shift operation is a mapping from an input path to an output path, similar to the mv command in Unix, mv /var/data /var/backup/data.

The input path is a JSON tree structure, and the output path is flattened "dot notation" path notation.

For example, given this simple input JSON:

{
    "id": 1,
    "name": "John Smith",
    "account": {
        "id": 1000,
        "type": "Checking"
    }
}

A simple spec could be constructed by copying that input, and modifying it to supply an output path for each piece of data:

{
    "id": "data.id",
    "name": "data.name",
    "account": "data.account"
}

would produce the following output JSON:

{
    "data" : {
        "id": 1,
        "name": "John Smith",
        "account": {
            "id": 1000,
            "type": "Checking"
        }
    }
}

Wildcards

The shift specification on the keys level supports wildcards and conditions:
1. * - match everything
2. name1|name2|nameN - match any of the specified names

& lookup allows referencing the values captured by the * or |.
It allows for specs to be more compact. For example, for this input:

{
    "id": 1,
    "name": "John Smith",
    "account": {
        "id": 1000,
        "type": "Checking"
    }
}

to get the output:

{
    "data" : {
        "id": 1,
        "name": "John Smith",
        "account": {
            "id": 1000,
            "type": "Checking"
        }
    }
}

the spec with wildcards would be:

{
    "*": "data.&0"
}

If you want only id and name in the output, the spec is:

{
    "id|name": "data.&0"
}

& wildcard also allows to dereference any level of the path of given node:

{
    "foo": {
        "bar" : {
            "baz": "new_location.&0.&1.&2" // &0 = baz, &1 = bar, &2 = foo
            }
        }
    }
}

for the input:

{
    "foo": {
      "bar": {
        "baz": "value"
      }
    }
  }

will produce:

{
    "new_location": {
      "baz": {
        "bar": {
          "foo": "value"
        }
      }
    }
}

Default operation

Applies default values if the value is not present in the input JSON.

For example, given this simple input JSON:

{
    "phones": {
        "mobile": 01234567,
        "country": "US"
    }
}

with the following specification for default operation:

{
    "phones": {
        "mobile": 0000000,
        "code": "+1"
    }
}

the output JSON will be:

{
    "phones": {
        "mobile": 01234567,
        "country": "US",
        "code": "+1"
    }
}

As you can see, the field mobile remains not affected while the code has a default '+1' value.

Remove operation

Removes content from the input JSON. The spec structure matches the input JSON structure. The value of fields is ignored.

For example, given this simple input JSON:

{
    "phones": {
        "mobile": 01234567,
        "country": "US"
    }
}

you can remove the country by the following specification for remove operation:

{
    "phones": {
        "country": ""
    }
}

the output JSON will be:

{
    "phones": {
        "mobile": 01234567
    }
}

Contributing

If you'd like to contribute to the project, please read our Contributing guide.

License

This project is licensed under the Apache license.

Dependencies

~1–1.9MB
~40K SLoC