#black-box-benchmarking #black-box-optimization

bin+lib kurobako

A black-box optimization benchmarking framework

42 releases

0.2.10 Apr 20, 2022
0.2.9 Feb 28, 2021
0.2.8 Jan 12, 2021
0.2.6 Sep 20, 2020
0.0.7 Mar 28, 2019

#180 in Science

Download history 46/week @ 2023-11-06 2/week @ 2023-11-13 44/week @ 2023-11-20 134/week @ 2023-11-27 42/week @ 2023-12-04 42/week @ 2023-12-11 5/week @ 2023-12-18 84/week @ 2023-12-25 1/week @ 2024-01-08 42/week @ 2024-01-22 84/week @ 2024-01-29 2/week @ 2024-02-05 131/week @ 2024-02-12 715/week @ 2024-02-19

932 downloads per month

MIT license

385KB
10K SLoC

kurobako

kurobako Documentation Actions Status Coverage Status License: MIT Docker Hub

kurobako is a command-line tool to benchmark black-box optimization algorithms.

Features:

  • It has the capability to support wide range of optimization problems:
    • various search sapce:
      • Continuous numerical, discrete numerical and categorical
      • Uniform distribution and log uniform distribution
      • Conditional
    • Constrainted problems
    • Multi-objective problems
  • Generating a markdown report and PNG plots from benchmarking results
  • Easy to add user-defined optimization problems and solvers
  • Simulating a concurrent environment in which an optimization process is executed by multiple workers simultaneously
  • Reproducible

Installation

Precompiled binaries

Precompiled binaries for Linux are available in the releases page.

$ curl -L https://github.com/optuna/kurobako/releases/download/${VERSION}/kurobako-${VERSION}.linux-amd64 -o kurobako
$ chmod +x kurobako
$ ./kurobako -h

Using Cargo

If you have already installed Cargo, you can install kurobako by executing the following command:

$ cargo install kurobako

Dependencies

If you want to use kurobako plot command, you need to have installed gnuplot package on your environment.

Usage Example

// Define solver.
$ kurobako solver random | tee solver.json
{"random":{}}

// Define problem.
$ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz
$ tar xf fcnet_tabular_benchmarks.tar.gz && cd fcnet_tabular_benchmarks/
$ kurobako problem hpobench fcnet_protein_structure_data.hdf5 | tee problem.json
{"hpobench":{"dataset":"fcnet_protein_structure_data.hdf5"}}

// Run benchmark.
$ kurobako studies --solvers $(cat solver.json) --problems $(cat problem.json) | kurobako run > result.json
(ALL) [00:00:01] [STUDIES     10/10 100%] [ETA  0s] done

// Report the benchmark result.
$ cat result.json | kurobako report
...abbrev...

// Plot the benchmark result.
$ cat result.json | kurobako plot curve
(PLOT) [00:00:01] [1/1 100%] [ETA  0s] done (dir="images/curve/")

Build-in Solvers and Problems

Solvers:

Problems:

Where does the name come from?

"kurobako" is a Japanese translation of "black box".

References

Dependencies

~14–26MB
~388K SLoC