#partitioning #task #worker #hash #count #sets #data

task_partitioner

helper lib that allows partitioning named tasks by count or hash

2 releases

0.1.1 Nov 25, 2022
0.1.0 Nov 25, 2022

#1572 in Algorithms

Download history 49/week @ 2024-11-13 129/week @ 2024-11-20 91/week @ 2024-11-27 188/week @ 2024-12-04 152/week @ 2024-12-11 101/week @ 2024-12-18 60/week @ 2024-12-25 51/week @ 2025-01-01 95/week @ 2025-01-08 107/week @ 2025-01-15 66/week @ 2025-01-22 96/week @ 2025-01-29 172/week @ 2025-02-05 203/week @ 2025-02-12 350/week @ 2025-02-19 93/week @ 2025-02-26

830 downloads per month
Used in laze

MIT/Apache

10KB
232 lines

Introduction

This Rust library allows partitioning a list of tasks to multiple workers, by count or by hash. It was cut out from the awesome nextest.

License

task_partitioner is licensed under the terms of the Apache License (Version 2.0), or MIT.


lib.rs:

Support for partitioning task runs across several machines.

At the moment this only supports simple hash-based and count-based sharding. In the future it could potentially be made smarter: e.g. using data to pick different sets of binaries and tests to run, with an aim to minimize total build and test times.

Dependencies

~0.4–0.9MB
~20K SLoC