Networking Machine

Purpose

The Networking Machine is responsible for message passing between engine instances, both locally (intra-node), and over the network (inter-node). The core functionality includes message routing and transport, upon which more complex peer-to-peer (P2P) protocols are built.

Scope

The Anoma v1 network is limited to the following.

• The network consists of multiple nodes that can establish direct connections with each other over QUIC/TLS transport protocols.

• Nodes know about each other's addresses by explicitly adding NodeAdverts to their configuration.

• Engines can send unicast and multicast (pub/sub) messages to both local and remote engines.

• Pub/sub is limited to a single publisher with directly connected subscribers.

The Anoma v2 network is going to support domains, dynamic P2P overlays, and P2P routing protocols. This is described in more detail in P2P Overlay Domains with Sovereignty (PODS). In PODS, the network architecture consists of a collection of nodes that are part of one or multiple heterogeneous domains. Each domain has its own overlay topology, and a distinct set of P2P intra-domain protocols, tailored to the characteristics and needs of the nodes in that domain.

Overview

Terminology

A node is the set of running engine instances that collectively participate in the network as a single entity. Each node has a Router engine instance responsible for intra- and inter-node message routing, and a Transport engine instance responsible for inter-node connectivity.

A peer is a connected neighbor of a node in the network.

The node identity in the networking context is the engine identity of the Router engine instance.

Message passing

Communication between engines follows the actor model with asynchronous message passing between engines.

Communicating engines can reside either on the same node or on different nodes connected over the network.

A message received by an engine may trigger reactions in the form of one or more messages sent by the receiver. A typical reaction is a response to a request in the often used request-response pattern, or a forwarding decision in a network protocol.

Message transmission, addressing, routing

Message transmission in the network is either one-to-one (unicast), few-to-many (multicast), or one-to-any (anycast).

A unicast message is sent between two engines by a single sender to a single recipient, and routed over the network either directly to the destination or over multiple hops, depending on the transport protocol used.

Multicast messaging follows the topic-based publish-subscribe (pub/sub) pattern, where a message (a.k.a. event) is sent by an authorized publisher to all subscribers of the topic, and routed over the network along multi-hop paths from a publisher towards subscribers.

Anycast messages are used when sending a request to any known member of a domain.

An EngineMessage is addressed from a source engine identity to a destination external identity, and the content of the message is authenticated by a signature of the source engine. The destination identity is either a NodeIdentity or EngineIdentity (for unicast messages), a pub/sub TopicIdentity (for multicast messages), or a DomainIdentity (for anycast messages).

The Router engine of each node is responsible for message routing between local engines, local and remote engines, and in some cases relaying between two remote engines. It makes routing decisions based on the destination identity in the message.

Message flow

Intra-node unicast & multicast messages

Unicast message from engine A_X to engine A_Y, along edges labeled Y. Multicast message from publisher engine A_Z to topic T, delivered to subscribed engines A_X and A_Y by the router A_R, along edges labeled T.

Inter-node unicast messages

Unicast message between engine A_X of node A and engine B_X of node B, via router engines A_R & B_R and transport engines A_T & B_T.

Inter-node multicast messages

Multicast (pub/sub) message with publisher A_X and subscribers B_X, C_X, D_X sent to topic T. Published by node A and forwarded to node B and C, then from node B to node D. Multicast messages are sent along edges labeled T, while B, C, D are unicast messages between PubSub engine instances.

Network architecture

The network consists of several sovereign domains with heterogenous protocols, where each domain maintains its own peer-to-peer overlay topology, manages its own membership, and determines the set of protocols that run inside the domain. A set of inter-domain protocols are responsible for clustering nodes based on domain membership and routing anycast messages to domains.

Intra-domain protocols

Inside each domain, a Domain membership and an overlay Topology maintenance protocol are responsible for responding to membership decisions, as well as keeping the overlay connected and keeping track of a partial view of online members.

The P2P PubSub (publish-subscribe) protocol is responsible for event dissemination from authorized publishers to subscribers, while the P2P Storage protocol offers block storage and retrieval.

Inter-domain protocols

Two inter-domain gossip protocol run in parallel: a Trust-Aware Peer Sampling (TAPS) and a Trust-Aware Clustering (TAC) protocol. The two protocols together construct a small world network, where TAPS provides continuously changing long-range routing links, while TAC discovers nodes with similar domain membership. Using trust information in these protocols aids in making these protocols more resilient against eclipse and hub attacks. TAPS is further enhanced by Uniform Peer Sampling (URPS) that uses statistical analysis to filter out peers over-represented in the peer sampling stream. Domain membership privacy in TAC is achieved by using the Bloom-and-Flip (BLIP) algorithm that calculates similarity between randomized Bloom filters (see BLIP), and then calculating the exact overlap using a Private Set Intersection (PSI) protocol. Participation in the inter-domain protocols is optional: nodes that do not wish to be discovered outside of their domains and have fixed domain membership may opt out of participating in inter-domain protocols, in order to decrease load, increase security, and thwart potential attacks coming from outsiders (e.g. DDoS). This allows domains to have internal-only and external-facing members.

There are two ways to interact with a Domain: either by sending external requests to any available domain member that may return a reply, or by joining the domain and participating in the intra-domain protocols. For both type of requests, a node needs to know about one or more domain members to send the request to, which can be discovered either via the Clustering protocol, or via a lookup request that is routed in the inter-domain overlay using a greedy routing algorithm by the Domain Routing protocol. Both methods use domain membership similarity as a distance metric.

Software architecture

Engines of the Networking Machine. Octagons are engines with a single instance per node. Double octagons are engines with multiple instances. Solid arrows mark unicast message flow directions. Dashed arrows mark multicast notifications. Dotted arrows mark messages sent via the Router.

Engines

Engines are grouped based on their scope in the network architecture.

Intra-node and inter-node protocols

This group of engines are responsible for facilitating communication between local engines (intra-node), and between engines of two directly connected remote nodes (inter-node).

These engines provide message routing and network transport functionality, and store information about network identities: advertisements received from the network and associated local metadata.

Router

The Router engine is responsible for message routing and handles both inter-node and intra-node messages. It authenticates received messages by verifying the signature of the source identity, and makes routing decisions based on the destination identity.

It also provides topic-based pub/sub functionality for local engines and performs local multicast message routing.

It can also relay messages between two remote nodes, in which case the EngineMessage is encrypted and wrapped in a RelayMessage.

The message routing algorithm is described in the EngineMessage section.

Note

An implementation may optimize intra-node messaging between local engine instances, such that they communicate directly instead of via the router.

Transport

The Transport engine is responsible for establishing and maintaining encrypted transport connections between peers. It supports various network transport protocols that are chosen according to transport preferences set by locally on a per-message or per-node basis, and defaults to the remote node's preferences specified in a NodeAdvert message.

Network Identity Store

The Network Identity Store engine maintains a data store with records that contain information associated with identities of engines, nodes, pub/sub topics, and domains.

The source of this information can be either local configuration or advertisements received from other nodes via P2P protocols.

For each EngineIdentity it stores its local engine address, if applicable to the implementation.

For each NodeIdentity, it stores transport addresses in order of preference, measurements, trust value, and reputation value.

For each TopicIdentity, it stores the PubSub TopicAdvert.

For each DomainIdentity, it stores the DomainAdvert.

Intra-domain protocols

This engine group implements intra-domain P2P protocols that run and offer services inside a domain.

Each domain has determines the set of protocols it runs and the authentication mechanism it uses.

PubSub

The PubSub engine implements a P2P topic-based pub/sub protocol and performs inter-node multicast message routing within a domain.

Storage

The Storage engine implements a P2P block storage protocol.