Verilization

Verilization is a serilization description language designed for defining binary file formats. Unlike other serialization tools such as Protocol Buffers, serialized Verilization data is not forward or backward compatible. Instead, conversions from older format versions are made easy, allowing for more compact data and more control over the underlying shape of the data.

Goals

Verilization has the following primary goals.

• Give maximum control of the file format to the user
• Define the format in a language-independent manner
• Support easy conversions from older versions of the format

Other less high-level goals.

• Support bigint types
• Embedable in other languages without using native binaries

Types

The following types are supported.

Structs

A struct type is defined with multiple versions. Each version defines a list of fields.

struct Rectangle {
    version 1 {
        width: u32;
        height: u32;
    }
}

Enums

An enum type is defined with multiple versions. Each version defines a list of fields used as cases. An enum value consists of exactly one of these fields.

struct StringOrInt {
    version 1 {
        str: string;
        num: int;
    }
}

Externs

An extern type is defined in user code. The type definition, conversions, and codecs must be implemented in the target language.

An extern type may declare what literals may be used for the type.

extern MyString {
    literal {
        string;
    }
}

The following literal specifications are supported.

Runtime Library Types

There are a number of extern types provided by the runtime library.

The encodings for int and nat define a sequence of bits in little-endian order. The highest bit in each byte is set if there are more bytes in the number.

This encoding is a sequence of bytes [B₀, ..., B_n] where B_i,7 = 1 when i < n and B_n,7 = 0. This sequence of bytes is equivalent to a sequence of bits [B_0,0, ... B_0,6, ..., B_n-1,0, ..., B_n-1,6] = [b₀, ..., b_m-1] where m = 6n. Essentially, the sequence of bits removes the flag bits that are used to determine when the sequence has reached the end and orders the remaining bits in each byte from least to most significant. The sequence of bits is mapped as follows:

• For the int type, if b_m-1 = 0, then k = b₀ * 2⁰ + ... + b_{m - 2} * 2^m-2
• For the int type, if b_m-1 = 1, then k = -(b₀ * 2⁰ + ... + b_{m - 2} * 2^m-2) - 1
• For the nat type, k = b₀ * 2⁰ + ... + b_{m - 1} * 2^m-1

Versioning

In the following example, a user has a username and birth date.

struct Person {
    version 1 {
        name: Name;
        dob: Date;
    }
}

struct Name {
    version 1 {
        firstName: string;
        middleName: option string;
        lastName: string;
    }
}

However, not everyone has 2 or 3 names. In order to accomodate this, we can create a new version that allows for an arbitrary number of names.

struct Name {
    version 1 {
        ...
    }
    version 2 {
        names: list string;
    }
}

This change to Name means that in version 2 of the format, the name field of Person will now use version 2 of Name. However, since there are no direct changes to Person, version 2 is automaticly created. In the generated code, the user is expected to provide code that can upgrade Name from version 1 to version 2. However, there is no need to provide such code for upgrading Person. Person can be upgraded automaticially using the upgrade code for its fields.

Final

Versioned types can be declared as final to indicate that no new versions of the type will be added. This restricts the type to the last explicitly declared version, preventing newer versions from being automaticially generated. Final types may only include fields of final or non-versioned types.

final struct FormatVersion {
    version 1 {
        major: nat;
    }
}

Generics

Generic types allow a type to be parameterized.

final struct Pair(A, B) {
    version 1 {
        left: A;
        right: B;
    }
}

Constants

Constants allow for values to be defined that are shared between any generated languages.

Command Line

Verilization has a command line interface. The following options are supported.

Language Generators

The following languages are supported.

• TypeScript
• Java
• Scala

Compiler Bindings

The verilization compiler is written in Rust. It can be compiled into WebAssembly for use in other languages. This has the advantage that a tool can be distributed (for example, as an NPM package, a standalone JAR, etc) without requiring any native binaries. These bindings expose both an interface that can be used directly from the runtime, as well as a command line interface that depends only on the associated runtime.

Currently, there are bindings for the following runtimes.

• Node

lib.rs:

The Verilization compiler generates serialization code for various languages.