#mnist #matrix #idx #file-format #database #multidimensional #ones

nightly idx_parser

Parse IDX files such as the ones used in MNIST database files

3 unstable releases

0.3.0 Nov 18, 2021
0.2.1 Nov 11, 2021
0.2.0 Nov 11, 2021
0.1.1 Nov 6, 2021
0.1.0 Nov 6, 2021

#1992 in Parser implementations

GPL-2.0-or-later WITH Bison-exception-2…

15KB
392 lines

IDX Parser

IDX data file parser written in Rust.

THE IDX FILE FORMAT

the IDX file format is a simple format for vectors and multidimensional matrices of various numerical types.

The basic format is

magic number
size in dimension 0
size in dimension 1
size in dimension 2
.....
size in dimension N
data

The magic number is an integer (MSB first). The first 2 bytes are always 0.

The third byte codes the type of the data:

0x08: unsigned byte
0x09: signed byte
0x0B: short (2 bytes)
0x0C: int (4 bytes)
0x0D: float (4 bytes)
0x0E: double (8 bytes)

The 4-th byte codes the number of dimensions of the vector/matrix: 1 for vectors, 2 for matrices....

The sizes in each dimension are 4-byte integers (MSB first, high endian, like in most non-Intel processors).

The data is stored like in a C array, i.e. the index in the last dimension changes the fastest.

(taken from http://yann.lecun.com/exdb/mnist/ )

No runtime deps