lq - low overhead yq/tq/jq cli

A lightweight and portable jq style cli for doing jq queries/filters/maps/transforms on YAML/TOML/JSON documents by converting to JSON and passing data to jq. Output is raw jq output which can optionally be mapped to TOML or YAML.

Installation

Via cargo:

cargo install lq

or download a prebuilt from releases either manually, or via binstall:

cargo binstall lq

Note: Requires jq.

Why / Why Not

jq compatibility

• arbitrary jq usage on any input format (yaml/toml/json) by going through json and jq
• same syntax, same filters, types, operators, conditionals, regexes, assignment, modules, etc
• matches jq's cli interface (only some extra input/output format controlling flags)
• supports jq output formatters such as -c, -r, and -j (compact, raw, joined output resp)

Extra Features

• supports multidoc yaml input, handles yaml merge keys (expanding tags)
• supports multidoc document splitting into expression based filenames
• supports in-place edits of documents
• maintains key order even while roundtripping between formats
• reads from stdin xor file (file if last arg is a file)
• filetype format inference when passing files
• quick input/output flags: -y (YAML out) or -t (TOML out), -T (TOML in), -J (JSON in)

Portable yq replacement

• ~1MB in binary (for small CI images / binstalled ci actions)
• 99% replacement of python-yq (with yq named/linked to lq)

Limitations

• Shells out to jq (not standalone - for now)
• Expands YAML tags (input is singleton mapped -> recursively, then merged) - so tags are not preserved in the output
• Does not preserve indentation (unsupported in serde_yaml)
• Halts on duplicate keys in the input document
• Formats require a serde implementation.
• Limited format support. No XML/CSV/RON support (or other more exotic formats). KDL wanted.

Usage

YAML

Use as jq either via stdin:

$ lq '.[3].kind' -r < test/deploy.yaml
Service

$ lq -y '.[3].metadata' < test/deploy.yaml
labels:
  app: controller
name: controller
namespace: default

or from a file arg (at the end):

$ lq '.[3].kind' -r test/deploy.yaml
$ lq -y '.[3].metadata' test/deploy.yaml

TOML

Infers input format from extension, or set explicitly via -T or --input=toml.

$ lq '.package.categories[]' -r Cargo.toml
command-line-utilities
parsing

convert jq output back into toml (-t):

$ lq -t '.package.metadata' Cargo.toml
[binstall]
bin-dir = "lq-{ target }/{ bin }{ format }"
pkg-url = "{ repo }/releases/download/{ version }lq-{ target }{ archive-suffix }"

convert jq output to yaml (-y) and set explicit toml input when using stdin (-T):

$ lq -Ty '.dependencies.clap' < Cargo.toml
features:
- cargo
- derive
version: 4.4.2

jq style compact output:

$ lq '.profile' -c Cargo.toml
{"release":{"lto":true,"panic":"abort","strip":"symbols"}}

To shortcut passing input formats, you can add alias tq='lq --input=toml' in your .bashrc / .zshrc (etc).

JSON Input

Infers input format from extension, or set explicitly via -J or --input=json.

$ lq -Jy '.ingredients | keys' < test/guacamole.json
- avocado
- coriander
- cumin
- garlic
- lime
- onions
- pepper
- salt
- tomatoes

Using JSON input is kind of like talking to jq directly, with the benefit that you can change output formats, or do inplace edits.

Formats

Default is going from yaml input to jq output to allow further pipes into jq.

• Input flags are upper case :: -J json input, -T toml input (shorthands for --input=FORMAT)
• Output flags are lower case :: -y yaml output, -t toml output (shorthands for --output=FORMAT)

Ex;

• lq :: yaml -> jq output
• lq -t :: yaml -> toml
• lq -y :: yaml -> yaml
• lq -Tt :: toml -> toml
• lq -Jy :: json -> yaml
• jq -Ty :: toml -> yaml
• jq -Jt :: json -> toml

Output formatting such as -y for YAML or -t for TOML will require the output from jq to be parseable json. If you pass on -r,-c or -c for raw/compact output, then this output may not be parseable as json.

Advanced Features

Two things you cannot do in jq:

Multidoc Splits

Split a bundle of yaml files into a yaml file per Kubernetes .metadata.name key:

mkdir -p crds
curl -sSL https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.82.1/stripped-down-crds.yaml \
  | lq . -y --split '"crds/" + (.metadata.name) + ".yaml"'

In Place Edits

Patch a json file (without multiple pipes):

lq -i '.SKIP_HOST_UPDATE=true' ~/.config/discord/settings.json

Advanced jq

Any weird things you can do with jq works. Some common (larger) examples:

Selects

Select on yaml multidoc:

$ lq '.[] | select(.kind == "Deployment") | .spec.template.spec.containers[0].ports[0].containerPort' test/deploy.yaml
8000

Escaping keys with slashes etc in them:

lq '.updates[] | select(.["package-ecosystem"] == "cargo") | .groups' .github/dependabot.yml

Modules

You can import jq modules e.g. k.jq:

$ lq 'include "k"; .[] | gvk' -r -L$PWD/test/modules < test/deploy.yaml
v1.ServiceAccount
rbac.authorization.k8s.io/v1.ClusterRole
rbac.authorization.k8s.io/v1.ClusterRoleBinding
v1.Service
apps/v1.Deployment

Debug Logs

The project respects RUST_LOG when set, and sends these diagnostic logs to stderr:

$ RUST_LOG=debug lq '.version' test/circle.yml
2023-09-18T23:17:04.533055Z DEBUG lq: args: Args { input: Yaml, output: Jq, yaml_output: false, toml_output: false, in_place: false, jq_query: ".version", file: Some("test/circle.yml"), compact_output: false, raw_output: false, join_output: false, modules: None }
2023-09-18T23:17:04.533531Z DEBUG lq: found 1 documents
2023-09-18T23:17:04.533563Z DEBUG lq: input decoded as json: {"definitions":{"filters":{"on_every_commit":{"tags":{"only":"/.*/"}},"on_tag":{"branches":{"ignore":"/.*/"},"tags":{"only":"/v[0-9]+(\\.[0-9]+)*/"}}},"steps":[{"step":{"command":"chmod a+w . && cargo build --release","name":"Build binary"}},{"step":{"command":"rustc --version; cargo --version; rustup --version","name":"Version information"}}]},"jobs":{"build":{"docker":[{"image":"clux/muslrust:stable"}],"environment":{"IMAGE_NAME":"lq"},"resource_class":"xlarge","steps":["checkout",{"run":{"command":"rustc --version; cargo --version; rustup --version","name":"Version information"}},{"run":{"command":"chmod a+w . && cargo build --release","name":"Build binary"}},{"run":"echo versions"}]},"release":{"docker":[{"image":"clux/muslrust:stable"}],"resource_class":"xlarge","steps":["checkout",{"run":{"command":"rustc --version; cargo --version; rustup --version","name":"Version information"}},{"run":{"command":"chmod a+w . && cargo build --release","name":"Build binary"}},{"upload":{"arch":"x86_64-unknown-linux-musl","binary_name":"${IMAGE_NAME}","source":"target/x86_64-unknown-linux-musl/release/${IMAGE_NAME}","version":"${CIRCLE_TAG}"}}]}},"version":2.1,"workflows":{"my_flow":{"jobs":[{"build":{"filters":{"tags":{"only":"/.*/"}}}},{"release":{"filters":{"branches":{"ignore":"/.*/"},"tags":{"only":"/v[0-9]+(\\.[0-9]+)*/"}}}}]},"version":2}}
2023-09-18T23:17:04.533650Z DEBUG lq: jq args: [".version"]
2023-09-18T23:17:04.538606Z DEBUG lq: jq stdout: 2.1

2.1

lq as yq

Because yaml is the default input language, you can use it as your top level yq executable with a symlink or alias:

# globally make yq be lq
ln -s $(which lq) /usr/local/bin/yq

# alias yq to lq in shell environment only
alias yq=lq

It is mostly compatible with python-yq (which uses jq syntax) but differs from go yq (which invents its own syntax).

(This use-case was the first use-case for this tool, i.e. to get rid of heavy python deps in CI images)