cargo-aprz

A cargo tool to appraise the quality of Rust dependencies.

• Background
• Installation
• Quick Start
• Data Sources
• Crates and Dependencies
  • Dependency Types
  • Package & Feature Selection
  • Tokens
• Reports
• Configuration and Expressions
  • Expression Checks in CI
• Troubleshooting
• Collected Metrics
  • Metadata Metrics
  • Usage Metrics
  • Stability Metrics
  • Community Metrics
  • Activity Metrics
  • Documentation Metrics
  • Advisory Metrics
  • Code Metrics
  • Trustworthiness Metrics

Background

Building modern applications usually involves integrating a large number of third-party dependencies. While these dependencies can provide valuable functionality and accelerate development, they also introduce risks related to quality, security vulnerabilities, future compatibility, and more.

Before taking a dependency in your project, it's useful to vet whether that dependency meets some baseline quality standards. For example, maybe you believe in having excellent unit test coverage for your projects, but if you pull in some dependency which has no tests, it can undermine the overall quality of your application.

cargo-aprz lets you appraise the quality of dependencies. For any given crate, it collects a large number of metrics, such as the number of open issues, the frequency of releases, the existence of security advisories, the number of examples, the code coverage percentage, and many more. You can view nice reports showing you all of these metrics in an easy to consume form.

You can also use cargo-aprz to automatically evaluate whether a crate meets your quality standards. You do this by writing a set of expressions that operate on the collected metrics. For example, you can have an expression that says "if code coverage is less than 20%, treat this crate as not being acceptable as a dependency".

You can run cargo-aprz by specifying a set of crates to evaluate, or you can run it on the full transitive set of dependencies of an existing Rust project.

Installation

cargo install --locked cargo-aprz

Quick Start

1. Generate a default configuration file:

   cargo aprz init
   

   This creates aprz.toml which lets you control various options. This is where you define expressions that let you evaluate the relative quality of a crate by inspecting its metrics.

2. Get the metrics associated with the latest version of a crate:

   cargo aprz crates tokio
   

   The first time you run this command, it will take a while as it needs to download a large database from crates.io along with the RustSec advisory database. This data is cached such that subsequent runs will be much faster.

3. Get the metrics associated with the dependencies of a Rust project:

   cargo aprz deps
   

4. Get the metrics for specific versions of crates:

   cargo aprz crates tokio@1.40.0 serde@1.0.0
   

5. Get the metrics for a crate and produce an HTML report instead of outputting to the console:

   cargo aprz crates tokio@1.40.0 --html report.html
   

Data Sources

cargo-aprz collects data from these sources:

• crates.io: Provides metadata and download statistics for each crate.

• GitHub or Codeberg: Provide information about the popularity of a crate, the number of issues and pull requests, the frequency of commits, and more. This is also where cargo-aprz gets source code in order to analyze the code quality of a crate.

• RustSec Advisory Database: Provides information about known vulnerabilities in Rust crates.

• docs.rs: Provides information about the quality of documentation for a crate, such as the presence of examples, the number of items with documentation comments, and more.

• codecov.io: Provides code coverage information.

Crates and Dependencies

cargo-aprz can be used to appraise the quality of specific crates, or the quality of the dependencies of an existing Rust project.

When you run cargo-aprz crates, you specify a set of crates to appraise with or without a version number. For example:

cargo aprz crates tokio serde@1.0.1

When you run cargo-aprz deps, it will appraise the quality of the dependencies of the Rust project in the current directory.

cargo aprz deps --dependency-types standard

Dependency Types

The --dependency-types option accepts a comma-separated list of dependency types to include in the appraisal. Possible values are:

• standard: Only include the standard dependencies of the project.
• dev: Only include the development dependencies of the project.
• build: Only include the build dependencies of the project.

Package & Feature Selection

When using the deps command, you can use the usual cargo options to control precisely which package and feature to consider. The available options include:

• --manifest-path <PATH>: Path to the Cargo.toml file of the project to analyze. By default, it looks for Cargo.toml in the current directory.
• --features: A comma-separated list of features to activate.
• --no-default-features: Do not activate the default feature.
• --all-features: Activate all available features.
• --package: Appraise the dependencies of a specific package in a workspace.
• --workspace: Appraise the dependencies of all packages in a workspace.

Tokens

cargo-aprz accesses he GitHub or Codeberg API to collect data about a crate. Although these APIs can be used without any form of authentication, this results in very low rate limits. If cargo-aprz detects it is being throttled by the API, it will enter a retry loop where it will wait until it is safe to try the operation again.

When using the deps command on a large project, it's likely you'll hit these rate limits, which can make the process take hours to complete fully. In such a case, you can provide a GitHub or Codebarg token on the command-line or through environment variables, which gives you substantially higher rate limits.

cargo aprz deps --github-token <GITHUB_TOKEN> --codeberg-token <CODEBERG_TOKEN>

You can also set the GITHUB_TOKEN and CODEBERG_TOKEN environment variables, which cargo-aprz will automatically pick up.

Reports

When you run cargo-aprz, it collects the many metrics listed below and then proceeds to generate a report that shows all the collected metrics. The report can be in a variety of formats including HTML and JSON. By default, the report is simply printed to the console.

cargo aprz crates tokio                     # Terminal output (default)
cargo aprz crates tokio --console           # Terminal output (explicit)
cargo aprz crates tokio --html report.html  # HTML report
cargo aprz crates tokio --json report.json  # JSON data
cargo aprz crates tokio --csv report.csv    # CSV file
cargo aprz crates tokio --excel report.xlsx # Excel spreadsheet

Configuration and Expressions

You can configure cargo-aprz by creating an aprz.toml file in the current directory. This file lets you define the set of expressions that the tool uses in order to assess whether a crate is acceptable or not acceptable to use as a dependency. The --config option lets you specify an arbitrary path to the configuration file instead of the default.

cargo-aprz uses the CEL expression language. This is a flexible, general-purpose expression language that allows you to write potentially complex boolean expressions that operate on the value of collected metrics. Expressions are divided into two buckets:

• high_risk: All expressions must evaluate to true. If any evaluates to false, the crate is flagged as high risk.

• eval: Each expression has a point value (default 1). All expressions are evaluated and a score is computed as granted_points / total_points * 100. The score is compared against configurable thresholds (medium_risk_threshold and low_risk_threshold) to determine whether the crate is low, medium, or high risk.

These buckets are evaluated in order. If no expressions are defined, then all crates are considered low risk.

Within these expressions, you can refer to any of the collected metrics. For example, you could write an expression that says "the crate must have 100 or fewer open issues to avoid being flagged as high risk":

[[high_risk]]
name = "Open Issues"
description = "Crate must not have too many open issues."
expression = "activity.open_issues <= 100"   

Any of the metric listed in Collected Metrics below can be used in these expressions, which gives you a lot of flexibility in defining what you consider to be an acceptable or unacceptable crate.

You can also use duration() in expressions for time-based comparisons. You can assign higher point values to more important expressions using the points field:

[[eval]]
name = "Established Crate"
description = "Accepts if the crate version was created more than 6 months ago."
expression = "stability.version_created_at < (stability.version_updated_at - duration('4320h'))"  # 4320 hours = 180 days
points = 5

By default, crates scoring below 30 are high risk, between 30 and 70 are medium risk, and 70 or above are low risk. You can customize these thresholds:

medium_risk_threshold = 30.0
low_risk_threshold = 70.0

Expression Checks in CI

If you want to use cargo-aprz in a CI pipeline to detect if any unsavory dependencies are being added to your project, you can use the --error-if-high-risk option to make cargo-aprz return a non-zero exit code if any of the crates being appraised are flagged as high risk based on the configured expressions. Similarly, --error-if-medium-risk returns a non-zero exit code if any crate is flagged as medium or high risk.

You can exempt specific crates from triggering these error exit codes by adding them to the allow list in your configuration file. Allowed crates are still evaluated and reported normally, but they won't cause a non-zero exit code. Each entry specifies a crate name and a semver version requirement:

[[allow_list]]
name = "some-crate"
version = "=1.2.3"

[[allow_list]]
name = "another-crate"
version = "^2.0"

Version requirements use standard semver syntax such as "*" (any version), "=1.2.3" (exact), "^1.2" (compatible), "~1.2" (patch-level), or ">=1.0, <2.0" (range).

Troubleshooting

The crates and deps commands both let you specify a logging level using the --log-level option. Turning on logging can be useful to troubleshooting connectivity problems. When logging is enabled, then normal console output is suspended.

Collected Metrics

The sections below show the full set of metrics collected.

Metadata Metrics

Usage Metrics

Stability Metrics

Community Metrics

Activity Metrics

Documentation Metrics

Advisory Metrics

Code Metrics

Trustworthiness Metrics