#sdk #distributed #peer-to-peer #database #bluetooth

dittolive-ditto

Ditto is a peer to peer cross-platform database that allows mobile, web, IoT and server apps to sync with or without an internet connection

28 releases (18 stable)

Uses new Rust 2021

new 1.1.10-experimental.all-product-headers-yaml-issue.1 Jun 22, 2022
1.1.7 May 13, 2022
1.1.3 Mar 21, 2022
1.0.19 Dec 23, 2021
1.0.0-alpha9 Mar 24, 2021

#31 in Database interfaces

Download history 1/week @ 2022-03-06 60/week @ 2022-03-13 81/week @ 2022-03-20 56/week @ 2022-04-03 12/week @ 2022-04-10 5/week @ 2022-04-17 50/week @ 2022-04-24 286/week @ 2022-05-01 365/week @ 2022-05-08 602/week @ 2022-05-15 202/week @ 2022-05-22 293/week @ 2022-05-29 181/week @ 2022-06-05 71/week @ 2022-06-12 69/week @ 2022-06-19

647 downloads per month

Custom license

200KB
4.5K SLoC

Ditto Rust SDK

Overview

Describes the different layers of the Ditto Rust SDK.

Ditto's core codebase is written in Rust. Currently, however, Rust does not have a stable ABI suitable for directly linking. Therefore the Rust SDK, like the other Ditto SDKs, is exposed through an external interface that uses the C ABI calling conventions. This core library is compiled as a both a static and dynamic library for a variety of architectures. The dittolive-ditto-sys crate contains Rust bindings to this C ABI library. The build.rs script will also attempt to identify the proper library binary for the host environment, download it from Ditto, and link to it. The Rust SDK includes this -sys crate as a dependency and then exposes an idiomatic Rust interface on top of this library, along with documentation and example apps.

Building an App with the Rust SDK

The following outlines the general process of getting started with a new App based on the Ditto Rust SDK. Rust currently does not support a stable ABI. To work around this, the Ditto Rust SDK is distributed in two parts: an ergonomic, open-source crate and a closed-source pre-compiled library. It is essential to have the correct library for both your development system and ultimate production target.

There are also some key terms you will need to know, especially for cross-compiling the RustSDK.

  • HOST - The host system doing the compiling. This could be a developers MacBook Pro (x86_64-apple-darwin) for example.
  • DITTO_TARGET - The system where the final app is going to run. This will be a Raspberry Pi ZeroW with a CPU that supports ARMv6 32-bit instructions and hard floats: arm-unknown-linux-gnueabihf.
  • DITTO_ROOT - The absolute path for the root directory of the Ditto source tree (where available).
  • APP_ROOT - The absolute path for the root directory of your app.
  • LIBDITTO - The binary component of the Ditto SDK which exposes a C FFI wrapper.
  • RustSDK - The Rust library component of the Ditto SDK which links, via a "-sys crate" to LIBDITTO.

Using a pre-built binary for your RustSDK-based App

The easiest way to get started with the Ditto Rust SDK is to use a pre-built Ditto library for your target architecture, where available.

  1. Obtain a Ditto License.
  2. Install the nightly tool chain for your current development machine using [rustup]https://rustup.rs
  3. Create a new Rust repository using github or cargo new
  4. In the root of the newly created project directory, edit the Cargo.toml file. Add a dittolive-ditto as a dependency. This is the crate for the Ditto Rust SDK.
  5. Test building the automatically generated Hello World app by running cargo build. This will trigger linking against the Ditto SDK for the current host system. If Ditto is not present for the current host system, this step will report an error when run with --verbose logging. If curl is present on the host system and the binary component of the Ditto SDK is absent, the build.rs script will attempt to download the appropriate library for the compilation target automatically. This step will also create a TARGET_DIR target directory (ie. target/debug) in the project root. This is a good location for putting the binary component of the Ditto SDK, especially if the development and production hosts are different.
  6. For cross-compilation (ie. development on MacOS for a Linux target), the Rust build script and machinery such as pkg_config may not automatically find the correct library for your target platform. In this event the Ditto library should be downloaded manually. Manually prefetching this library may also be desirable for offline CI pipelines and other situations where network access is not desired.
  7. The URL for each target's Ditto SDK is https://software.ditto.live/rust/Ditto/<version>/<target-tripple>/<profile>/<library-name> For example, on an x86_64 MacOS developer machine, one would use https://software.ditto.live/rust/Ditto/1.0.3-alpha1/x86_64-apple-darwin/release/libdittoffi.dylib.
  8. The environment variable DITTOFFI_SEARCH_PATH may be used to manually set a directory to be used to find the binary component of the Ditto SDK. This is especially helpful when cross-compiling for different target architectures.
  9. With the binary component now in place, verify that cargo run executes successfully and doesn't throw a linker error.
  10. On some platforms, similar operations are required for other libraries such as various MacOS Frameworks (ie. CoreFoundation, Security). The cargo documentation provides guidance of various ways to provide these libraries to each platforms linker.

Compiling from libditto from source

If you have access to a copy of the full Ditto source code, you can compile the binary component of the Rust SDK, libdittoffi yourself. The compilation host machine will need 4GB of RAM (or swap file) for linking purposes. If your target machine doesn't have this, you'll need to cross-compile (see section below).

  1. First, clone the Ditto source tree. We will refer to the absolute path to the root of the Ditto source tree as DITTO_ROOT going forward.
  2. Select a target system. This can be the same as your host system, or another target triple (cross-compilation). The full list of supported target systems is available here. Note that you will need a valid C toolchain (eg. CC, ld) for both the host and target system. If you are compiling on Mac OS for Linux, this will require more set up (described in a following section).
export DITTO_TARGET=x86_64-unknown-linux-gnu
  1. Build libdittoffi for your target platform. This can take a while.
(cd ffi && cargo build --release --target $DITTO_TARGET)
  1. Confirm your build is successful. You should see libdittoffi.a and libdittoffi.so in $DITTO_ROOT/target/$DITTO_TARGET/release/. This path is known as your TARGET_DIR and is configured by a file .cargo/config.toml in DITTO_ROOT. Each target and profile (release, debug) will get its own sub-directory. Note that on Windows and Mac OS, the library file extensions will be different (.dll and .dylib, respectively).
  2. Optionally, you can now install libdittoffi.so into a cannonical search location for the target's linker to find. When compiling on the execution device, we recommend a symlink to /usr/local/lib or /opt/ditto/lib. This will make it easy for any apps outside of DITTO_ROOT to find libdittoffi.so.
sudo ln -sf "$DITTO_ROOT/target/$DITTO_TARGET/release/libdittoffi.so" /usr/local/bin
  1. Alternatively, you can specify your TARGET_DIR as the DITTOFFI_SEARCH_PATH that your app will search for this library.
export DITTOFFI_SEARCH_PATH="$DITTO_ROOT/target/$DITTO_TARGET/release"
  1. Create your new Ditto-powered App. We'll refer to the root of this app as APP_ROOT going forward.
cargo new myapp
  1. Add the ditto Rust SDK as a dependency by editing your app's Cargo.toml file. This should be the Path to the Rust SDK not libdittoffi.so.
[dependencies]
dittolive-ditto.path = "$DITTO_ROOT/rust"
  1. Test building your app from within the your apps root directory. Your app should build dittolive-ditto (the Rust SDK) which in turn will build dittolive-ditto-sys which links to libdittoffi.so or libdittoffi.a.
cargo build
  1. If you want to force static linking of your app to libdittoffi.a you can set the LIBDITTO_STATIC=1 env var. However, this may result in missing symbol errors on some platforms where shared system libraries are otherwise linked in by default. For example, on Mac OS Darwin you may see missing system framework symbols, because these symbols are only available as shared libraries. Your app will need to be configured to tell the linker how to source these symbols.
LIBDITTO_STATIC=1 cargo build
  1. Finally, to build a release of your app
cargo build --release --target $DITTO_TARGET
  1. Your final executable can be run as follows. This may be copied to another system, but be sure to also provide copies of libdittoffi.so and any other dynamically linked libraries.
$APP_ROOT/target/$DITTO_TARGET/release/myapp

How is the Ditto Library located

The build.rs script for dittolive-ditto-sys makes a best effort attempt to locate, and if absent download, libdittoffi in spite of each target OS having distinct linkers and conventions for managing shared libraries. The absolute location of these files depends on whether the SDK is built from source or downloaded from crates.io as a dependency. The search order is as follows:

  1. If DITTOFFI_SEARCH_PATH env var is set to a valid directory, use this. This takes priority over all other methods.
  2. Look in the CARGO_BUILD_TARGET_DIR (ie. target/$DITTO_TARGET/{debug,release}). This is automatic for Cargo.
  3. Look in the deps folder for the CARGO_BUILD_TARGET (ie. target/$DITTO_TARGET/{debug,release}/deps). This is automatic for Cargo.
  4. Look in the CARGO_MANIFEST_DIR.
  5. The current working directory of cargo.
  6. Common POSIX paths defined in the Filesystem Hierarchy Standard: /usr, /lib, /usr/local/lib, /opt.
  7. If on a linux system, try using pkg_config. Note that Raspberry Pi OS does not ship with pkg_config.
  8. If on a windows system, try using vcpkg.
  9. Download pre-built version from Ditto's S3 bucket

The end result of this search is to ensure that two key arguments are passed to the target linker when building your app:

  • -ldittoffi - Your app should be linked against the ditto library
  • -L DITTOFFI_SEARCH_PATH - The directory where the ditto library for the target system can be found You can see this process by building with cargo build -vv.

If desired, you can explicitly provide these two arguments using cargo rustc to build your app, or otherwise configuring your apps build system to provide these values when linking. We recommend explicitly specifying the search path for the ditto library, especially when cross-compiling, to ensure the correct DITTO_TARGET architecture is used.

Cross-compiling for IOT devices

Many IOT devices make poor compilation hosts for Rust, and thus it is better to cross-compile on a developer machine with more resources and then execute on the target IOT device.

Method 1 - Use a rustembedded/cross Docker and VS Studio Code "devContainer" extension to build from source

  1. Install docker on your HOST machine
  2. Install Visual Studio Code on your Host Machine
  3. Install the VS Code "Remote - Containers" extension.
  4. Configure your DITTO_TARGET so that you can ssh onto it and copy files using a utility such as scp, sftp, or rsync.
  5. Check out the Ditto source tree in VS Code, and then in the root of the project edit .devcontainer/devontainer.json's build.Dockerfile key to point to the docker image for your DITTO_TARGET. In this case Dockerfile.armv-unknown-linux-gnueabihf.
  6. Use the "Remote - Containers" plug in to "Reload in Container" (the green >< icon in the lower left corner). This will download the image, build it, mount the Ditto repo into /workspaces/ditto, which will be our effective DITTO_ROOT going forward.
  7. Open a terminal in the container and build libdittoffi. The machining target architecture will be automatically selected.
(cd ffi && cargo build --release)
  1. You may now copy the built artifact to your target device (on your host)
scp $DITTO_ROOT/target/$DITTO_TARGET/release/libdittoffi.so pi@raspberrypi:/home/pi
  1. SSH onto your Raspberry Pi and Symlink libdittoffi.so into a linker search directory
sudo ln -sf /home/pi/libdittoffi.so /usr/local/lib/libdittoffi.so
  1. Repeat steps 1 through 8 for your app. If your app lives outside of the DITTO_ROOT, you will need to set up a distinct VS Code project for it. Alternatively, you can compile your app in place on the Raspberry Pi device.
  2. You can verify your app on the target device with the following
ldd ./myapp

This should verify the target devices linker can resolve all the shared libraries (including libdittoffi.so) on the target device

Configuring the RUST SDK

The following env vars are commonly used to configure the Ditto SDK.

  • RUST_SDK_LOG_LEVEL - Sets the log level of the Ditto libraries internal logger. Values are error, warn, info, debug, and verbose. Default is "info".
  • DITTOFFI_SEARCH_PATH - Explicitly define where to look for libdittoffi for your platform.
  • DITTO_DB_PATH - The absolute path where Ditto should store local copies of documents and attachments. This can also be provided programatically.

Common config patterns

The following are common patterns used to configure test and example apps

  • DITTO_LICENSE - EnvVar containing a valid Ditto license token.
  • DITTO_APP_NAME - The full name of your app. Typically defined in reverse DNS format (ie. "live.ditto.carsapp").
  • DITTO_SITE_ID - A 64 bit unique identifier of this specific instance of your app. Often associated with a Security subject or Identity.
  • DITTO_BINDIP - The IP and/or Port (ie. 0.0.0.0:8080) where Ditto should listen for TCP or WebSocket Transport traffic.

Dependencies

~2.2–3.5MB
~68K SLoC