|0.2.0||Mar 3, 2021|
|0.1.2||Sep 8, 2020|
|0.1.1||Sep 5, 2020|
|0.1.0||Sep 4, 2020|
#513 in Science
Ssam, short for split sampler, splits one or more text-based input files into multiple sets using random sampling. This is useful for splitting data into a training, test and development sets, or whatever sets you desire.
- Split input into multiple sets, user can specify the size of each set in either absolute numbers or a relative fraction.
- Supports both sampling without replacement (default) or with replacement.
- Defaults to line-based sampling, but a custom delimiter can be configured to sample larger blocks of variable size.
- Can handle multiple input files that will be considered dependent. Useful for splitting and sampling for instance parallel corpora.
- By default ordering is preserved, use
--shufflefor more randomness.
- Specify a seed for the random number generator to create reproducible samples.
Install it using Rust's package manager:
cargo install ssam
No cargo/rust on your system yet? Do
sudo apt install cargo on Debian/ubuntu based systems,
brew install rust on mac, or use rustup.
ssam --help for extensive usage information.
Suppose you have a text file
sentences.txt with one sentence per line, and you want to sample the sentences into a test, development and
train set using respectively 10% (
0.1), 10% (
0.1) and the remainder (
*) of the sentences:
$ ssam --sizes "0.1,0.1,*" --names "test,dev,train" sentences.txt
This will output three files:
sentences.dev.txt. If you don't specify
any names explicitly the infix will simply be
Suppose you have the same sentences in German in a file called
sätze.txt and the sentences are aligned up nicely with
the ones in
sentences.txt (i.e. the same line numbers correspond and contain translations). You can now make a
dependent split as follows:
$ ssam --shuffle --sizes "0.1,0.1,*" --names "test,dev,train" sentences.txt sätze.txt
The sentences will still correspond in each of the output sets. We also added
--shuffle for more randomness in the
output order, as by default ssam preserves order.
Ssam can also read from stdin (provided you want to supply only one input document). If you're only doing one sample (rather than three as shown above), then it will simply output to stdout.
Rather than using lines as units, you can specify a delimiter manually. For example, set
--delimiter "" (empty
delimiter) if you want empty lines to be the delimiter, such as for instance those often used to separate paragraphs.
Alternative you can set it to an explicit marker in your input, like
--delimiter "<utt>" for example.
In loving memory of our cat Sam, 2009-2019.