github.com/aakash4dev/cometbft@v0.38.2/spec/p2p/implementation/transport.md (about)

     1  # Transport
     2  
     3  The transport establishes secure and authenticated connections with peers.
     4  
     5  The transport [`Dial`](#dial)s peer addresses to establish outbound connections,
     6  and [`Listen`](#listen)s in a configured network address
     7  to [`Accept`](#accept) inbound connections from peers.
     8  
     9  The transport establishes raw TCP connections with peers
    10  and [upgrade](#connection-upgrade) them into authenticated secret connections.
    11  The established secret connection is then wrapped into `Peer` instance, which
    12  is returned to the caller, typically the [switch](./switch.md).
    13  
    14  ## Dial
    15  
    16  The `Dial` method is used by the switch to establish an outbound connection with a peer.
    17  It is a synchronous method, which blocks until a connection is established or an error occurs.
    18  The method returns an outbound `Peer` instance wrapping the established connection.
    19  
    20  The transport first dials the provided peer's address to establish a raw TCP connection.
    21  The dialing maximum duration is determined by `dialTimeout`, hard-coded to 1 second.
    22  The established raw connection is then submitted to a set of [filters](#connection-filtering),
    23  which can reject it.
    24  If the connection is not rejected, it is recorded in the table of established connections.
    25  
    26  The established raw TCP connection is then [upgraded](#connection-upgrade) into
    27  an authenticated secret connection.
    28  This procedure should ensure, in particular, that the public key of the remote peer
    29  matches the ID of the dialed peer, which is part of peer address provided to this method.
    30  In the absence of errors,
    31  the established secret connection (`conn.SecretConnection` type)
    32  and the information about the peer (`NodeInfo` record) retrieved and verified
    33  during the version handshake,
    34  are wrapped into an outbound `Peer` instance and returned to the switch.
    35  
    36  ## Listen
    37  
    38  The `Listen` method produces a TCP listener instance for the provided network
    39  address, and spawns an `acceptPeers` routine to handle the raw connections
    40  accepted by the listener.
    41  The `NetAddress` method exports the listen address configured for the transport.
    42  
    43  The maximum number of simultaneous incoming connections accepted by the listener
    44  is bound to `MaxNumInboundPeer` plus the configured number of unconditional peers,
    45  using the `MultiplexTransportMaxIncomingConnections` option,
    46  in the node [initialization](https://github.com/aakash4dev/cometbft/blob/v0.34.x/node/node.go#L563).
    47  
    48  This method is called when a node is [started](https://github.com/aakash4dev/cometbft/blob/v0.34.x/node/node.go#L974).
    49  In case of errors, the `acceptPeers` routine is not started and the error is returned.
    50  
    51  ## Accept
    52  
    53  The `Accept` method returns to the switch inbound connections established with a peer.
    54  It is a synchronous method, which blocks until a connection is accepted or an error occurs.
    55  The method returns an inbound `Peer` instance wrapping the established connection.
    56  
    57  The transport handles incoming connections in the `acceptPeers` persistent routine.
    58  This routine is started by the [`Listen`](#listen) method
    59  and accepts raw connections from a TCP listener.
    60  A new routine is spawned for each accepted connection.
    61  The raw connection is submitted to a set of [filters](#connection-filtering),
    62  which can reject it.
    63  If the connection is not rejected, it is recorded in the table of established connections.
    64  
    65  The established raw TCP connection is then [upgraded](#connection-upgrade) into
    66  an authenticated secret connection.
    67  The established secret connection (`conn.SecretConnection` type),
    68  the information about the peer (`NodeInfo` record) retrieved and verified
    69  during the version handshake,
    70  as well any error returned in this process are added to a queue of accepted connections.
    71  This queue is consumed by the `Accept` method.
    72  
    73  > Handling accepted connection asynchronously was introduced due to this issue:
    74  > <https://github.com/tendermint/tendermint/issues/2047>
    75  
    76  ## Connection Filtering
    77  
    78  The `filterConn` method is invoked for every new raw connection established by the transport.
    79  Its main goal is avoid the transport to maintain duplicated connections with the same peer.
    80  It also runs a set of configured connection filters.
    81  
    82  The transports keeps a table `conns` of established connections.
    83  The table maps the remote address returned by a generic connection to a list of
    84  IP addresses, to which the connection remote address is resolved.
    85  If the remote address of the new connection is already present in the table,
    86  the connection is rejected.
    87  Otherwise, the connection's remote address is resolved into a list of IPs,
    88  which are recorded in the established connections table.
    89  
    90  The connection and the resolved IPs are then passed through a set of connection filters,
    91  configured via the `MultiplexTransportConnFilters` transport option.
    92  The maximum duration for the filters execution, which is performed in parallel,
    93  is determined by `filterTimeout`.
    94  Its default value is 5 seconds,
    95  which can be changed using the `MultiplexTransportFilterTimeout` transport option.
    96  
    97  If the connection and the resolved remote addresses are not filtered out,
    98  the transport registers them into the `conns` table and returns.
    99  
   100  In case of errors, the connection is removed from the table of established
   101  connections and closed.
   102  
   103  ### Errors
   104  
   105  If the address of the new connection is already present in the `conns` table,
   106  an `ErrRejected` error with the `isDuplicate` reason is returned.
   107  
   108  If the IP resolution of the connection's remote address fails,
   109  an `AddrError` or `DNSError` error is returned.
   110  
   111  If any of the filters reject the connection,
   112  an `ErrRejected` error with the `isRejected` reason is returned.
   113  
   114  If the filters execution times out,
   115  an `ErrFilterTimeout` error is returned.
   116  
   117  ## Connection Upgrade
   118  
   119  The `upgrade` method is invoked for every new raw connection established by the
   120  transport that was not [filtered out](#connection-filtering).
   121  It upgrades an established raw TCP connection into a secret authenticated
   122  connection, and validates the information provided by the peer.
   123  
   124  This is a complex procedure, that can be summarized by the following three
   125  message exchanges between the node and the new peer:
   126  
   127  1. Encryption: the nodes produce ephemeral key pairs and exchange ephemeral
   128     public keys, from which are derived: (i) a pair of secret keys used to
   129     encrypt the data exchanged between the nodes, and (ii) a challenge message.
   130  1. Authentication: the nodes exchange their persistent public keys and a
   131     signature of the challenge message produced with the their persistent
   132     private keys. This allows validating the peer's persistent public key,
   133     which plays the role of node ID.
   134  1. Version handshake: nodes exchange and validate each other `NodeInfo` records.
   135     This records contain, among other fields, their node IDs, the network/chain
   136     ID they are part of, and the list of supported channel IDs.
   137  
   138  Steps (1) and (2) are implemented in the `conn` package.
   139  In case of success, they produce the secret connection that is actually used by
   140  the node to communicate with the peer.
   141  An overview of this procedure, which implements the station-to-station (STS)
   142  [protocol][sts-paper] ([PDF][sts-paper-pdf]), can be found [here][peer-sts].
   143  The maximum duration for establishing a secret connection with the peer is
   144  defined by `handshakeTimeout`, hard-coded to 3 seconds.
   145  
   146  The established secret connection stores the persistent public key of the peer,
   147  which has been validated via the challenge authentication of step (2).
   148  If the connection being upgraded is an outbound connection, i.e., if the node has
   149  dialed the peer, the dialed peer's ID is compared to the peer's persistent public key:
   150  if they do not match, the connection is rejected.
   151  This verification is not performed in the case of inbound (accepted) connections,
   152  as the node does not know a priori the remote node's ID.
   153  
   154  Step (3), the version handshake, is performed by the transport.
   155  Its maximum duration is also defined by `handshakeTimeout`, hard-coded to 3 seconds.
   156  The version handshake retrieves the `NodeInfo` record of the new peer,
   157  which can be rejected for multiple reasons, listed [here][peer-handshake].
   158  
   159  If the connection upgrade succeeds, the method returns the established secret
   160  connection, an instance of `conn.SecretConnection` type,
   161  and the `NodeInfo` record of the peer.
   162  
   163  In case of errors, the connection is removed from the table of established
   164  connections and closed.
   165  
   166  ### Errors
   167  
   168  The timeouts for steps (1) and (2), and for step (3), are configured as the
   169  deadline for operations on the TCP connection that is being upgraded.
   170  If this deadline it is reached, the connection produces an
   171  `os.ErrDeadlineExceeded` error, returned by the corresponding step.
   172  
   173  Any error produced when establishing a secret connection with the peer (steps 1 and 2) or
   174  during the version handshake (step 3), including timeouts,
   175  is encapsulated into an `ErrRejected` error with reason `isAuthFailure` and returned.
   176  
   177  If the upgraded connection is an outbound connection, and the peer ID learned in step (2)
   178  does not match the dialed peer's ID,
   179  an `ErrRejected` error with reason `isAuthFailure` is returned.
   180  
   181  If the peer's `NodeInfo` record, retrieved in step (3), is invalid,
   182  or if reports a node ID that does not match peer ID learned in step (2),
   183  an `ErrRejected` error with reason `isAuthFailure` is returned.
   184  If it reports a node ID equals to the local node ID,
   185  an `ErrRejected` error with reason `isSelf` is returned.
   186  If it is not compatible with the local `NodeInfo`,
   187  an `ErrRejected` error with reason `isIncompatible` is returned.
   188  
   189  ## Close
   190  
   191  The `Close` method closes the TCP listener created by the `Listen` method,
   192  and sends a signal for interrupting the `acceptPeers` routine.
   193  
   194  This method is called when a node is [stopped](https://github.com/aakash4dev/cometbft/blob/v0.34.x/node/node.go#L1023).
   195  
   196  ## Cleanup
   197  
   198  The `Cleanup` method receives a `Peer` instance,
   199  and removes the connection established with a peer from the table of established connections.
   200  It also invokes the `Peer` interface method to close the connection associated with a peer.
   201  
   202  It is invoked when the connection with a peer is closed.
   203  
   204  ## Supported channels
   205  
   206  The `AddChannel` method registers a channel in the transport.
   207  
   208  The channel ID is added to the list of supported channel IDs,
   209  stored in the local `NodeInfo` record.
   210  
   211  The `NodeInfo` record is exchanged with peers in the version handshake.
   212  For this reason, this method is not invoked with a started transport.
   213  
   214  > The only call to this method is performed in the `CustomReactors` constructor
   215  > option of a node, i.e., before the node is started.
   216  > Note that the default list of supported channel IDs, including the default reactors,
   217  > is provided to the transport as its original `NodeInfo` record.
   218  
   219  [peer-sts]: ../legacy-docs/peer.md#authenticated-encryption-handshake
   220  [peer-handshake]: ../legacy-docs/peer.md#cometbft-version-handshake
   221  [sts-paper]: https://link.springer.com/article/10.1007/BF00124891
   222  [sts-paper-pdf]: https://github.com/tendermint/tendermint/blob/0.1/docs/sts-final.pdf