5 releases (3 stable)

1.0.2 Jun 6, 2022
1.0.1 Jan 12, 2022
0.1.1 Aug 25, 2021
0.1.0 May 11, 2021

#185 in Machine learning

Download history 12/week @ 2024-07-01 18/week @ 2024-07-08 18/week @ 2024-07-15 38/week @ 2024-07-22 110/week @ 2024-07-29 23/week @ 2024-08-05 9/week @ 2024-08-12 55/week @ 2024-08-19 19/week @ 2024-08-26 69/week @ 2024-09-02 17/week @ 2024-09-09 8/week @ 2024-09-16 23/week @ 2024-09-23 8/week @ 2024-09-30 13/week @ 2024-10-07 24/week @ 2024-10-14

68 downloads per month

MIT/Apache

81KB
1.5K SLoC

eval-metrics

Evaluation metrics for machine learning

crates.io License


Design

The goal of this library is to provide an intuitive collection of functions for computing evaluation metrics commonly encountered in machine learning. Metrics are separated into modules for either classification or regression, with the classification module supporting both binary and multi-class tasks. This distinction between binary and multi-class classification is made explicit to underscore the fact that there are subtle differences in certain metrics between the two cases (i.e. multi-class metrics often require averaging methods). Metrics can often fail to be defined for a variety of numerical reasons, and in these cases Result types are used to make this fact apparent.

Supported Metrics

Metric Task Description
Accuracy Binary Classification Binary Class Accuracy
Precision Binary Classification Binary Class Precision
Recall Binary Classification Binary Class Recall
F-1 Binary Classification Harmonic Mean of Precision and Recall
MCC Binary Classification Matthews Correlation Coefficient
ROC Curve Binary Classification Receiver Operating Characteristic Curve
AUC Binary Classification Area Under ROC Curve
PR Curve Binary Classification Precision-Recall Curve
AP Binary Classification Average Precision
Accuracy Multi-Class Classification Multi-Class Accuracy
Precision Multi-Class Classification Multi-Class Precision
Recall Multi-Class Classification Multi-Class Recall
F-1 Multi-Class Classification Multi-Class F1
Rk Multi-Class Classification K-Category Correlation Coefficient as described by Gorodkin (2004)
M-AUC Multi-Class Classification Multi-Class AUC as described by Hand and Till (2001)
RMSE Regression Root Mean Squared Error
MSE Regression Mean Squared Error
MAE Regression Mean Absolute Error
R-Square Regression Coefficient of Determination
Correlation Regression Linear Correlation Coefficient

Usage

Binary Classification

The BinaryConfusionMatrix struct provides functionality for computing common binary classification metrics.

use eval_metrics::error::EvalError;
use eval_metrics::classification::BinaryConfusionMatrix;

fn main() -> Result<(), EvalError> {
    // note: these scores could also be f32 values
    let scores = vec![0.5, 0.2, 0.7, 0.4, 0.1, 0.3, 0.8, 0.9];
    let labels = vec![false, false, true, false, true, false, false, true];
    let threshold = 0.5;

    // compute confusion matrix from scores and labels
    let matrix = BinaryConfusionMatrix::compute(&scores, &labels, threshold)?;

    // counts
    let tpc = matrix.tp_count;
    let fpc = matrix.fp_count;
    let tnc = matrix.tn_count;
    let fnc = matrix.fn_count;

    // metrics
    let acc = matrix.accuracy()?;
    let pre = matrix.precision()?;
    let rec = matrix.recall()?;
    let f1 = matrix.f1()?;
    let mcc = matrix.mcc()?;

    // print matrix to console
    println!("{}", matrix);
    Ok(())
}
                            o=========================o
                            |          Label          |
                            o=========================o
                            |  Positive  |  Negative  |
o==============o============o============|============o
|              |  Positive  |     2      |     2      |
|  Prediction  |============|------------|------------|
|              |  Negative  |     1      |     3      |
o==============o============o=========================o

In addition to the metrics derived from the confusion matrix, ROC curves and PR curves can be computed, providing metrics such as AUC and AP.

use eval_metrics::error::EvalError;
use eval_metrics::classification::{RocCurve, RocPoint, PrCurve, PrPoint};

fn main() -> Result<(), EvalError> {
    // note: these scores could also be f32 values
    let scores = vec![0.5, 0.2, 0.7, 0.4, 0.1, 0.3, 0.8, 0.9];
    let labels = vec![false, false, true, false, true, false, false, true];

    // construct roc curve
    let roc = RocCurve::compute(&scores, &labels)?;
    // compute auc
    let auc = roc.auc();
    // inspect roc curve points
    roc.points.iter().for_each(|point| {
        let tpr = point.tp_rate;
        let fpr = point.fp_rate;
        let thresh = point.threshold;
    });

    // construct pr curve
    let pr = PrCurve::compute(&scores, &labels)?;
    // compute average precision
    let ap = pr.ap();
    // inspect pr curve points
    pr.points.iter().for_each(|point| {
        let pre = point.precision;
        let rec = point.recall;
        let thresh = point.threshold;
    });
    Ok(())
}

Multi-Class Classification

The MultiConfusionMatrix struct provides functionality for computing common multi-class classification metrics. Additionally, averaging methods must be explicitly provided for several of these metrics.

use eval_metrics::error::EvalError;
use eval_metrics::classification::{MultiConfusionMatrix, Averaging};

fn main() -> Result<(), EvalError> {
    // note: these scores could also be f32 values
    let scores = vec![
        vec![0.3, 0.1, 0.6],
        vec![0.5, 0.2, 0.3],
        vec![0.2, 0.7, 0.1],
        vec![0.3, 0.3, 0.4],
        vec![0.5, 0.1, 0.4],
        vec![0.8, 0.1, 0.1],
        vec![0.3, 0.5, 0.2]
    ];
    let labels = vec![2, 1, 1, 2, 0, 2, 0];

    // compute confusion matrix from scores and labels
    let matrix = MultiConfusionMatrix::compute(&scores, &labels)?;

    // get counts
    let counts = &matrix.counts;

    // metrics
    let acc = matrix.accuracy()?;
    let mac_pre = matrix.precision(&Averaging::Macro)?;
    let wgt_pre = matrix.precision(&Averaging::Weighted)?;
    let mac_rec = matrix.recall(&Averaging::Macro)?;
    let wgt_rec = matrix.recall(&Averaging::Weighted)?;
    let mac_f1 = matrix.f1(&Averaging::Macro)?;
    let wgt_f1 = matrix.f1(&Averaging::Weighted)?;
    let rk = matrix.rk()?;

    // print matrix to console
    println!("{}", matrix);
    Ok(())
}
                           o===================================o
                           |               Label               |
                           o===================================o
                           |  Class-1  |  Class-2  |  Class-3  |
o==============o===========o===========|===========|===========o
|              |  Class-1  |     1     |     1     |     1     |
|              |===========|-----------|-----------|-----------|
|  Prediction  |  Class-2  |     1     |     1     |     0     |
|              |===========|-----------|-----------|-----------|
|              |  Class-3  |     0     |     0     |     2     |
o==============o===========o===================================o

In addition to these global metrics, per-class metrics can be obtained as well.

use eval_metrics::error::EvalError;
use eval_metrics::classification::{MultiConfusionMatrix};

fn main() -> Result<(), EvalError> {
    // note: these scores could also be f32 values
    let scores = vec![
        vec![0.3, 0.1, 0.6],
        vec![0.5, 0.2, 0.3],
        vec![0.2, 0.7, 0.1],
        vec![0.3, 0.3, 0.4],
        vec![0.5, 0.1, 0.4],
        vec![0.8, 0.1, 0.1],
        vec![0.3, 0.5, 0.2]
    ];
    let labels = vec![2, 1, 1, 2, 0, 2, 0];

    // compute confusion matrix from scores and labels
    let matrix = MultiConfusionMatrix::compute(&scores, &labels)?;
    
    // per-class metrics
    let pca = matrix.per_class_accuracy();
    let pcp = matrix.per_class_precision();
    let pcr = matrix.per_class_recall();
    let pcf = matrix.per_class_f1();
    let pcm = matrix.per_class_mcc();
    
    // print per-class metrics to console
    println!("{:?}", pca);
    println!("{:?}", pcp);
    println!("{:?}", pcr);
    println!("{:?}", pcf);
    println!("{:?}", pcm);
    Ok(())
}
[Ok(0.5714285714285714), Ok(0.7142857142857143), Ok(0.8571428571428571)]
[Ok(0.3333333333333333), Ok(0.5), Ok(1.0)]
[Ok(0.5), Ok(0.5), Ok(0.6666666666666666)]
[Ok(0.4), Ok(0.5), Ok(0.8)]
[Ok(0.09128709291752773), Ok(0.3), Ok(0.7302967433402215)]

In addition to the metrics derived from the confusion matrix, the M-AUC (multi-class AUC) metric as described by Hand and Till (2001) is provided as a standalone function:

let mauc = m_auc(&scores, &labels)?;

Regression

All regression metrics operate on a pair of scores and labels.

use eval_metrics::error::EvalError;
use eval_metrics::regression::*;

fn main() -> Result<(), EvalError> {

    // note: these could also be f32 values
    let scores = vec![0.4, 0.7, -1.2, 2.5, 0.3];
    let labels = vec![0.2, 1.1, -0.9, 1.3, -0.2];

    // root mean squared error
    let rmse = rmse(&scores, &labels)?;
    // mean squared error
    let mse = mse(&scores, &labels)?;
    // mean absolute error
    let mae = mae(&scores, &labels)?;
    // coefficient of determination
    let rsq = rsq(&scores, &labels)?;
    // pearson correlation coefficient
    let corr = corr(&scores, &labels)?;
    Ok(())
}

Dependencies