#cli #data #csv

app datamars

A Rust rewrite of https://www.gnu.org/software/datamash/

7 unstable releases (3 breaking)

0.4.0 Aug 20, 2022
0.3.0 Aug 3, 2022
0.2.3 Jul 25, 2022
0.2.1 Jun 26, 2022
0.1.0 Jun 6, 2022

#1037 in Math

35 downloads per month


547 lines


My own copy of GNU datamash.

This software is pre-alpha and it's missing quite a lot of features.


If you want the latest version, use cargo install datamars.

Note: the executable is called ms.

Implementation details

the algorithm basically goes down like this:

args = parse_args()
# parse ops
ops = build_operation_objects(args.ops)

groups = dict(str->list[ops])

for line in file: 
    parts = line.split(args.delimiter)
    group = groupby(parts) if grouping is defined else default
    if group not in groups:
        # build new objects for every group
        groups.add_group(group, ops)
    for op in groups[group]:
for group in groups:
    print(group, [op.result() for op in group])

There is a set of operations for every group. Operations keep the running state for a group. The operations update the state for every new entry in the group - but only the state that is needed, eg no calculation of squared deviations. Should be pretty fast.


~51K SLoC