5 releases
0.2.1 | Apr 14, 2024 |
---|---|
0.2.0 | Apr 13, 2024 |
0.1.5 | Mar 27, 2024 |
#11 in #gossip
53 downloads per month
Used in 5 crates
(4 directly)
600KB
15K
SLoC
Memberlist
A highly customable, adaptable, runtime agnostic and WASM/WASI friendly Gossip protocol which helps manage cluster membership and member failure detection.
Port and improve HashiCorp's memberlist to Rust.
English | 简体中文
Introduction
memberlist is a rust crate that manages cluster membership and member failure detection using a gossip based protocol.
The use cases for such a library are far-reaching: all distributed systems require membership, and memberlist is a re-usable solution to managing cluster membership and node failure detection.
memberlist is eventually consistent but converges quickly on average. The speed at which it converges can be heavily tuned via various knobs on the protocol. Node failures are detected and network partitions are partially tolerated by attempting to communicate to potentially dead nodes through multiple routes.
memberlist is WASM/WASI friendly, all crates can be compiled to wasm-wasi
and wasm-unknown-unknown
(need to configure the crate features).
Design
Unlike the original Go implementation, Rust's memberlist use highly generic and layered architecture, users can easily implement a component by themselves and plug it to the memberlist. Users can even custom their own Id
and Address
.
Here are the layers:
-
Transport Layer
By default, Rust's memberlist provides two kinds of transport --
QuicTransport
andNetTransport
.-
Runtime Layer
Async runtime agnostic are provided by
agnostic
's Runtime trait,tokio
,async-std
andsmol
are supported by default. Users can implement their ownRuntime
and plug it into the memberlist. -
Address Resolver Layer
The address resolver layer is supported by
nodecraft
's AddressResolver trait. -
Serialize/Deserilize Layer
By default, Rust's memberlist is using
length-prefix encoding (Lpe)
to serialize/deserialize messages to bytes or visa-vise. The implemention ofLpe
tries the best to avoid reallocating when doing the serialize/deserialize.But, users can use any other serialize/deserialize framework by implementing
Wire
trait. -
Three kinds of different builtin stream layers for
NetTransport
:Tcp
: based on TCP and UDPTls
: based onrustls
and UDPNativeTls
: based onnative-tls
and UDP
-
QUIC transport is an experimental transport implementation, it is well tested but still experimental.
Two kinds of different builtin stream layers for
QuicTransport
:
Users can still implement their own stream layer for different kinds of transport implementations.
-
-
Delegate Layer
This layer is used as a reactor for different kinds of messages.
-
Delegate
Delegate is the trait that clients must implement if they want to hook into the gossip layer of Memberlist. All the methods must be thread-safe, as they can and generally will be called concurrently.
Here are the sub delegate traits:
-
AliveDelegate
Used to involve a client in processing a node "alive" message. When a node joins, either through a packet gossip or promised push/pull, we update the state of that node via an alive message. This can be used to filter a node out and prevent it from being considered a peer using application specific logic.
-
ConflictDelegate
Used to inform a client that a node has attempted to join which would result in a name conflict. This happens if two clients are configured with the same name but different addresses.
-
EventDelegate
A simpler delegate that is used only to receive notifications about members joining and leaving. The methods in this delegate may be called by multiple threads, but never concurrently. This allows you to reason about ordering.
-
MergeDelegate
Used to involve a client in a potential cluster merge operation. Namely, when a node does a promised push/pull (as part of a join), the delegate is involved and allowed to cancel the join based on custom logic. The merge delegate is NOT invoked as part of the push-pull anti-entropy.
-
NodeDelegate
Used to manage node related events. e.g. metadata
-
PingDelegate
Used to notify an observer how long it took for a ping message to complete a round trip. It can also be used for writing arbitrary byte slices into ack messages. Note that in order to be meaningful for RTT estimates, this delegate does not apply to indirect pings, nor fallback pings sent over promised connection.
-
-
CompositeDelegate
CompositeDelegate is a helpful struct to split the
Delegate
into multiple small delegates, so that users do not need to implement fullDelegate
when they only want to custom some methods in the Delegate.
-
Protocol
memberlist is based on "SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol". However, Hashicorp developers extends the protocol in a number of ways:
Several extensions are made to increase propagation speed and convergence rate. Another set of extensions, that Hashicorp developers call Lifeguard, are made to make memberlist more robust in the presence of slow message processing (due to factors such as CPU starvation, and network delay or loss). For details on all of these extensions, please read Hashicorp's paper "Lifeguard : SWIM-ing with Situational Awareness", along with the memberlist source.
Installation
[dependencies]
memberlist = "0.1"
Q & A
-
Does Rust's memberlist implemenetation compatible to Go's memberlist?
No but yes! By default, it is not compatible. But the secret is the serialize/deserilize layer, Go's memberlist use the msgpack as the serialization/deserialization framework, so in theory, if you can implement a
Wire
trait which compat to Go's memberlist, then it becomes compatible. -
If Go's memberlist adds more functionalities, will this project also support?
Yes! And this project may also add more functionalities whereas the Go's memberlist does not have. e.g. wasmer support, bindings to other languages and etc.
Related Projects
agnostic
: helps you to develop runtime agnostic cratesnodecraft
: crafting seamless node operations for distributed systems, which provides foundational traits for node identification and address resolution.transformable
: transform its representation between structured and byte form.peekable
: peekable reader and async reader
License
memberlist
is under the terms of the MPL-2.0 license.
See LICENSE for details.
Copyright (c) 2024 Al Liu.
Copyright (c) 2013 HashiCorp, Inc.
Dependencies
~12–26MB
~387K SLoC