1 unstable release
0.0.1 | Mar 21, 2024 |
---|
#58 in #tooling
215KB
5K
SLoC
bolic-network: a modern network abstraction and tooling for building distributed systems.
Why not libp2p? (and others)
We believe our design is clutter-free, minimalist, and usable at different levels. It does what other libraries like libp2p can do, but with much better abstraction based on the authors' multi-year of cutting-edge research experience in distributed systems and consensus. It is not just a code base that "works", but also meets the demands of a wide spectrum of networked infrastructures, especially those requiring high volume of binary (short or moderately sized) messages. In fact, we build this library to be reused and serve well as the basis for any core infrastructure that exchanges messages in a networked manner.
Why not Tokio?
We do use tokio to manage some intertwined asynchronous logic in our code, but
our core data transmission is done by a carefully crafted engine directly based
on mio. Instead of throwing all connections into tokio, by giving them a better
abstraction (Endpoint
) and drive them through our well-engineered Driver
(which is a pool of Worker
s), we have established a good separation among
the following three aspects of a networking library: the notion of a peer-wise
data transmission channel, the actual underlying transportation protocol (could
even be SMS, if one implements Transport
trait) it utilizes, and the
execution logic that efficiently drives the asynchronous I/O induced by the
transmission. By learning from the past projects and with the help of Rust, we
believe our abstraction achieves clarity, simplicity and low overhead at the
same time. If not the best in terms of the absolute performance (one can always
handcraft a super adhoc solution, say in C, with optimal performance), it still
offers a solid basis for any distributed systems with great performance and
robustness.
Reusable at Different Levels
Our design deliberately divides our abstraction/implementation into different levels so engineers and researchers can easily use them for their own purpose with a lot of freedom of customization.
At the highest level, one can either use:
network
(TBD): will contain the code for self-organizing network, including node discovery, routing, etc.hub
: defines and implements a multi-transport "hub" that could automatically connect to other peers given their contacts. This could be useful for a relatively static network setup, such as a quorum-based (or "vote-based", "PoS") consensus protocol, or MPC projects.
In hub
, there are three important abstractions (and their implementations) could be customized:
-
Driver
: a pool ofWorker
s, where each worker has an OS-level thread and drives the I/O of multiple endpoints. -
Endpoint
: a user-friendly, bidirectional, virtual channel that transmits data to some remote recipient. An endpoint has a pluggableTransport
at its core that actually implements the network transmission. -
Transport
: a protocol-specific implementation that follows a unified interface to move data through the endpoint. The existence of this abstraction allowsEndpoint
to be connection-agnostic: the underlying transportation method could either be "connection-based" (such as TCP) or "connectionless" (such as UDP), but the user of endpoints does not need to worry about that. An endpoint may renew its underlying transport to be able to resume its data transmission once it is "clogged" (i.e., reconnection) -- all these, are invisible to the endpoint users. What's more, while doing all these, endpoints can be moved among the workers (for load-balancing purposes) on-the-fly, without losing any data.
Portability
The vast majority of our code base is also WASM-compatible. Which means one can turn web browsers and web-like environments (such as mobile apps) into a legit node that talks to others. This means one can achieve "true" P2P by running a lightweight node in the client to talk to "servers" (nodes with useful services), without having to invoke a cumbersome RPC protocol like JSON/RPC or web REST calls.
License
This project is licensed under Apache License 2.0. See NOTICE
file for more information.
Dependencies
~4–21MB
~380K SLoC