Intra-node & inter-node protocols¶
Purpose¶
This group of engines are responsible for facilitating communication between local engines (intra-node), and between engines of two directly connected remote nodes (inter-node).
These engines provide message routing and network transport functionality, and store information about network identities: advertisements received from the network and associated local metadata.
Overview¶
Router¶
Purpose¶
The Router is responsible for forwarding both intra-node and inter-node messages between engine instances. It forwards intra-node messages directly between local engines, and sends and receives inter-node messages via the Transport engine. It makes routing decisions based on the DestinationIdentity in EngineMessage headers, and retrieves routing information for identities from the Network Identity Store engine.
Addressing¶
Both the source and destination address of an EngineMessage is an ExternalIdentity. The source is always the EngineIdentity of an engine instance, while the DestinationIdentity can belong to either an engine instance, a domain, or a pub/sub topic, which corresponds to unicast, anycast, and multicast communication patterns, respectively.
A unicast message is sent between two (local or remote) engine instances, an anycast message is sent from an engine instance to any member of a domain, while a pub/sub message is sent from an engine instance to all subscribers of a topic. Pub/sub messages are sent between local engine instances via the Router that manages subscriptions and routes messages, whereas inter-node pub/sub messages are handled by the PubSub engine.
When an EngineMessage is sent between peers, it is wrapped in a P2PMessage and signed by the source NodeIdentity. The node identity is defined as the external identity of the Router engine instance of a node.
Routing¶
The Router makes routing decisions based on the DestinationIdentity of an EngineMessage, and forwards the message either to one or more local engines, or to the Transport engine for delivery to a remote node.
The message processing and routing process is described in the EngineMessage section.
Inter-node messages¶
When forwarding an EngineMessage to a remote node, the Router wraps it in P2PMessage, signs it with its NodeIdentity key, and forwards it to Transport for delivery over the network.
Transport¶
Purpose¶
The Transport engine is responsible for sending and receiving messages from other nodes over the network, establishing and maintaining authenticated and encrypted communication channels to other nodes via various transport protocols.
Connection pool¶
The Transport engine maintains a pool of connections to remote nodes, which can be either permanent or ephemeral. Ephemeral connections are disconnected after a period of inactivity, and lost in case of a network error. Permanent connections are maintained indefinitely and reconnected when the connection fails.
Transport protocols¶
The Transport engine delivers messages over the network via various transport protocols, either using protocols directly over IP (such as QUIC or Secure WebSocket), through an overlay network (such as onion routed or mix networks), or via asynchronous, delay-tolerant networks.
Transport protocols are responsible for the authentication and encryption of each sent and received TransportMessage, but not responsible for authenticating or verifying remote node identities, which is done by the Router instead, by signing and verifying each sent and received P2PMessage.
Transport preferences¶
Transport protocol choice is based on Transport preferences that can be specified in either an EngineMessage header, or a NodeIdentityRecord stored by the Network Identity Store engine, which contains both NodeAdverts received from the network and local preferences associated with nodes.
When multiple transport addresses of a peer are known for the transport(s) chosen according to transport preferences, then address choice is based on both the peer's own preferences in its NodeAdvert, and local measurements (e.g. latency) and preferences (e.g. prefer LAN over WAN IP addresses)
Connection establishement¶
To establish a connection, the Transport engine needs a TransportAddress to connect to, which contains the transport protocol and address for the network connection.
In case of transports over IP that use X.509 certificates (such as QUIC or Secure WebSocket), the TransportAddress contains the destination IP address & port, as well as the certificate issuer's public key, which is used for subsequent verification of the remote peer's transport certificate. Whereas for transports that use public key addressing, the destination public key address is given instead, possibly together with additional transport-specific dial information.
Once the connection is set up, Transport sends a PeerConnected notification, or in case of connection failure a PeerConnectFailed notification instead. While at the end of the connection, a PeerDisconnected notification is sent.
Network Identity Store¶
The Network Identity Store (NIS) engine stores information associated with known external identities (of peers, topics, domains, and engines), such as advertised addresses and local metadata. The information stored can come from various sources, such as local configuration, P2P protocols (e.g. peer sampling, clustering, pub/sub), other engines, or user input.
In case of peer identities, it stores known addresses along with local routing and transport preferences, latency measurements, trust metric and trust zone (e.g. local, remote).
Message flow¶
