YABE (YAml Base Extractor) - Multi-layer YAML organizer

The idea comes from the need to manage huge amount of YAML files in a GitOps environment. Especially when using ArgoCD multi-source apps with some common values and some overrides. The tool helps to compute the common base configuration among multiple YAML files and generate differences for each file, reducing the duplication of configuration values. It also provides the ability to sort YAML content based on user-defined configuration.

Features

• Compute diffs: Detect differences between YAML files.
• Merge YAML files: Combine YAML files with a base YAML, either from an existing file or dynamically computed.
• Quorum-based diffing: Extract common base YAML based on a quorum percentage.
• Sort YAML content: Sort keys in YAML files based on user-defined configuration.
• Helm Values Integration: Merge input YAML files with Helm values files.
• In-place modification or output to new files.

Usage

Usage: yabe [OPTIONS] <INPUT_FILES>...

Arguments:
  <INPUT_FILES>...  Input YAML files

Options:
  -r, --read-base <READ_BASE>                (Optional) Read-only base for values deduplication
  -b, --base <WRITE_BASE>                    (Optional) Common values of all input files, if not provided, will be computed
  -i, --in-place                             Modify the original input files with diffs
  -o, --out <OUT_FOLDER>                     Output folder for diff files [default: ./out]
      --debug                                Enable debug logging
  -q, --quorum <QUORUM>                      Quorum percentage (0-100) [default: 51]
      --base-out-path <BASE_OUT_PATH>        (Optional) Base file output path [default: ./base.yaml]
      --sort-config-path <SORT_CONFIG_PATH>  (Optional) Sort configuration file path [default: ./sort-config.yaml], if not provided, will not sort
  -h, --help                                 Print help
  -V, --version                              Print version

Basic Usage

Run the tool with the YAML override files:

./yabe file1.yaml file2.yaml file3.yaml

This will compute the differences among the override files and generate:

• base.yaml: The common base configuration.
• file1_diff.yaml, file2_diff.yaml, file3_diff.yaml: The differences for each file.

Inplace Modification

Use the -i or --inplace flag to modify the original override files with their differences:

./yabe -i -r helm_values.yaml file1.yaml file2.yaml file3.yaml

Enable Debug Logging

Use the --debug flag to enable detailed debug logging:

./yabe --debug -r helm_values.yaml file1.yaml file2.yaml file3.yaml

Examples

Sample Input Files

helm_values.yaml

settings:
  theme: dark
  notifications: true
  advanced:
    mode: auto
    level: 5

file1.yaml

settings:
  theme: dark
  notifications: true
  advanced:
    mode: auto
    level: 5

file2.yaml

settings:
  theme: light
  notifications: true
  advanced:
    mode: manual
    level: 5

file3.yaml

settings:
  theme: dark
  notifications: false
  advanced:
    mode: auto
    level: 7

Running the Tool

./yabe -r helm_values.yaml file1.yaml file2.yaml file3.yaml

Expected Output

base.yaml

settings:
  advanced:
    level: 5

file1_diff.yaml (Empty file or not generated since there are no differences)

file2_diff.yaml

settings:
  theme: light
  advanced:
    mode: manual

file3_diff.yaml

settings:
  notifications: false
  advanced:
    level: 7

Inplace Modification Example

Running with the -i flag:

./yabe -i -r helm_values.yaml file1.yaml file2.yaml file3.yaml

Testing

The project includes a suite of tests to verify functionality. To run the tests:

cargo test

Ensure all tests pass to verify that the tool is functioning correctly.

Project Structure

• src/
  • lib.rs: The library module containing core functionality.
  • main.rs: The main executable entry point.
  • diff.rs: Functions for computing diffs and common bases.
  • deep_equal.rs: Utility function for deep comparison of YAML values.
  • sorter.rs: Functions for sorting YAML content.
• tests/
  • test_deep_equal.rs: Tests for the deep_equal function.
  • test_diff.rs: Tests for compute_diff and diff_and_common_multiple functions.
  • test_common.rs: Common tests for the project.
  • test_sorter.rs: Tests for the sorter functions.
• Cargo.toml: Project configuration file.
• sort-config.yaml: Configuration file for sorting YAML content.