Lib.rs

Why?

We like CBOR for the following:

1. It's IETF standard
2. It's self-descriptive
3. It's compact enough
4. It's extensive including mind-boggling things
5. It has implementations in most languages

What we lack in CBOR:

1. No schema definition, i.e. can't check interoperability between systems
2. Transmitting/storing lots of objects is expensive because keys are encoded for every object
3. No standard way to cast "object" (i.e. a map or a dict) into native typed object

Comparison

This section roughly compares similar projects to see second our arguments in "Why?" section. Individual arguments may not be very convincing by they are reasonable enough in total.

Probor vs Capnproto

Capnproto has ugly and complex serialization format which is useful for mapping values directly into memory without decoding. But its more complex to implement correctly than what we target for. We also wanted compact encoding which Capnproto has but it's built on top of already hard to understand encoding and complicates things even more.

Capnproto like other relies on code generation with ugly protocol objects as result of decoding, but we wanted native types.

Look-a-Like

For example, here is schema:

struct SearchResults {
    total_results @0 :int
    results @1 :array Page
}
struct Page {
    url @0 :text,
    title @1 :text,
    snippet @2 :optional text,
}

Note the following things:

• We use generic type names like int (integer), not fixed width (see FAQ)
• We give each field a number, they are similar to ones used in other IDL's (like protobuf, thrift or capnproto)

The structure serialized with probor will look like (displaying json for clarity, in fact you will see exact this data if decode CBOR and encode with JSON):

[1100, [
     ["http://example.com", "Example Com"],
     ["http://example.org", "Example Org", "Example organization"]]]

Obviously when unpacked, it looks more like (in javascript):

new SearchResults({"total_results": 1100,
                   "results": [new Page({"url": "http://example.com",
                                         "title": "Example Com"}),
                               new Page({"url": "http://example.org",
                                         "title": "Example Org",
                                         "snippet": "Example organization"})]}

Actually the object can be serialized like this:

{"total_results": 1100,
 "results": [{"url": "http://example.com",
              "title": "Example Com"},
             {"url": "http://example.org",
              "title": "Example Org",
              "snippet": "Example organization"}]}

And this would also be totally valid serialized representation. I.e. you can store fields both by names and by numbers. This is occasionally useful for ad-hoc requests or you may be willing to receive non-compact data from frontend, then validate and push data in more compact format for storage.

In Python serialization looks like:

from probor import struct

class Page(object):

    def __init__(self, url, title, snippet=None):
        # .. your constructor .. omitted for brevity

    probor_protocol = struct(
        required={(0, "url"): str, (1, "title"): str},
        optional={(2, "snippet"): str})

class SearchResults(object):
    def __init__(self, total_resutls, results):
        # .. your constructor .. omitted for brevity

    probor_protocol = struct(
        required={(0, "total_results"): int, (1, "results"): Page})

TODO: isn't syntax ugly? Should it be more imperative? Is setstate/getstate used?

Similarly in Rust it looks like:

#[macro_use] extern crate probor;

use probor::{Encoder, Encodable};
use probor::{Decoder, Config, decode};
use std::io::Cursor;

probor_struct!(
#[derive(PartialEq, Eq, Debug)]
struct Page {
    url: String => (#0),
    title: String => (#1),
    snippet: Option<String> => (#2 optional),
});

probor_struct!(
#[derive(PartialEq, Eq, Debug)]
struct SearchResults {
    total_results: u64 => (#0),
    results: Vec<Page> => (#1),
});


fn main() {
    let buf = Vec::new();
    let mut enc = Encoder::new(buf);
    SearchResults {
        total_results: 112,
        results: vec![Page {
            url: "http://url1.example.com".to_string(),
            title: "One example".to_string(),
            snippet: None,
        }, Page {
            url: "http://url2.example.com".to_string(),
            title: "Two example".to_string(),
            snippet: Some("Example Two".to_string()),
        }],
    }.encode(&mut enc).unwrap();
    let sr: SearchResults = decode(
        &mut Decoder::new(Config::default(), Cursor::new(enc.into_writer())))
        .unwrap();
    println!("Results {:?}", sr);
}

The Rust example is a bit longer which is bearable for rust. It's hugely based on macros, which may seem as similar to code generation. Still, we find it better, because you are in control of at least the following things:

1. The specific types used (e.g. u64 for int)
2. The structure definition (may use meta attributes including derive and repr and may use struct T(X, Y))
3. How objects are created (e.g. use VecDeque or BTreeMap instead of default Vec and HashMap)
4. How missing fields are handled (e.g. you can provide defaults for missing fields instead of using Option<T>)
5. You can include application-specific validation code

At the end of the day, writing a parser explicitly with few helper macros looks like a much better idea than adding all the data as the meta information to the schema file.

Type System

enum HtmlElement {
    Tag(String, Vec<HtmlElement>),
    Text(String),
}

from probor import enum

class HtmlElement:
    """Base class"""

class Tag(HtmlElement):
    def __init__(self, tag_name, children):
        # .. snip ..

    probor_protocol = ...

class Text(HtmlElement):

    def __init__(self, text)
        self.text = text

    probor_protocol = ...

HtmlElement.probor_protocol = enum({
    (0, 'Tag'): Tag,
    (1, 'Text'): Text,
})

Forward/Backward Compatibility

Comparing with protobuf, the probor serializer always considers all fields as optional. The required fields are only in IDL, so if your future type is smart enough to

Backwards compatibility is very similar to protobuf.

TBD: exact rules for backward compatibility

TBD: exact rules for forward compatibility

TBD: turning structure in algebraic type with compatibility

FAQ