Sonicd

The problem

• Implementing data analytics APIs against multiple underlying data sources often results in code bloat and tight coupling.
• If you're using Akka Streams for stream processing, you are probably missing a data sourcing component (like Storm's Spouts).
• ODBC/JDBC are antiquated and not designed for streaming large datasets.

The solution

Sonicd is a data streaming gateway that abstracts over data source connectors and provides a modern protocol to stream data over WebSockets or over plain TCP.

Supported Sources

• PrestoSource: non-blocking streaming connector for Facebook's Presto.
• ElasticSearchSource: non-blocking streaming connector for ElasticSearch
• JdbcSource: JDBC connector for any database with a JDBC driver implementation (tested with Hive, Redshift, H2, MySQL, PostgreSQL).
• ZuoraObjectQueryLanguageSource: Zuora's SOAP API ZOQL streaming connector.
• LocalJsonStreamSource: stream changes in local JSON files.
• KafkaSource: Apache Kafka connector with simple DSL to select/filter streams efficiently.
• Composer: Combine in a single Sonic stream, any of the previous sources.

Deploy

Check server/src/main/resources/reference.conf for a config reference and docker run -d -v ${CONFIG_DIR}:/etc/sonicd:ro -p 9111:9111 -p 10001:10001 xarxa6/sonicd;. If intend to use the JDBC source, then you'll want to add JDBC drivers jars to the classpath:

docker run -d -p 9111:9111 -p 10001:10001 -v ${CONFIG_DIR}:/etc/sonicd/ -v ${JDBC_DRIVERS}:/var/lib/sonicd/ xarxa6/sonicd

Install CLI

Sonicd also provides a CLI to run ad hoc queries. If you have the rust toolchain installed already, then simply cargo install sonic, otherwise install rustup first with curl https://sh.rustup.rs -sSf | sh or check https://www.rustup.rs/.

Examples

Check examples folder. For an example in Rust check the cli.

Client libraries

• NodeJS
• Rust
• Scala/Java (Akka Streams)

Contribute

If you would like to contribute to the project, please fork the project, include your changes and submit a pull request back to the main repository.

License

MIT License