github.com/aakash4dev/cometbft@v0.38.2/spec/p2p/implementation/pex-protocol.md (about)

     1  # Peer Exchange Protocol
     2  
     3  The Peer Exchange (PEX) protocol enables nodes to exchange peer addresses, thus
     4  implementing a peer discovery mechanism.
     5  
     6  The PEX protocol uses two messages:
     7  
     8  - `PexRequest`: sent by a node to [request](#requesting-addresses) peer
     9    addresses to a peer
    10  - `PexAddrs`: a list of peer addresses [provided](#providing-addresses) to a
    11    peer as response to a `PexRequest` message
    12  
    13  While all nodes, with few exceptions, participate on the PEX protocol,
    14  a subset of nodes, configured as [seed nodes](#seed-nodes) have a particular
    15  role in the protocol.
    16  They crawl the network, connecting to random peers, in order to learn as many
    17  peer addresses as possible to provide to other nodes.
    18  
    19  ## Requesting Addresses
    20  
    21  A node requests peer addresses by sending a `PexRequest` message to a peer.
    22  
    23  For regular nodes, not operating in seed mode, a PEX request is sent when
    24  the node *needs* peers addresses, a condition checked:
    25  
    26  1. When an *outbound* peer is added, causing the node to request addresses from
    27     the new peer
    28  2. Periodically, by the `ensurePeersRoutine`, causing the node to request peer
    29     addresses to a randomly selected peer
    30  
    31  A node needs more peer addresses when its addresses book has
    32  [less than 1000 records](./addressbook.md#need-for-addresses).
    33  It is thus reasonable to assume that the common case is that a peer needs more
    34  peer addresses, so that PEX requests are sent whenever the above two situations happen.
    35  
    36  A PEX request is sent when a new *outbound* peer is added.
    37  The same does not happen with new inbound peers because the implementation
    38  considers outbound peers, that the node has chosen for dialing, more
    39  trustworthy than inbound peers, that the node has accepted.
    40  Moreover, when a node is short of peer addresses, it dials the configured seed nodes;
    41  since they are added as outbound peers, the node can immediately request peer addresses.
    42  
    43  The `ensurePeersRoutine` periodically checks, by default every 30 seconds (`ensurePeersPeriod`),
    44  whether the node has enough outbound peers.
    45  If it does not have, the node tries dialing some peer addresses stored in the address book.
    46  As part of this procedure, the node selects a peer at random,
    47  from the set of connected peers retrieved from the switch,
    48  and sends a PEX request to the selected peer.
    49  
    50  Sending a PEX request to a peer is implemented by the `RequestAddrs` method of
    51  the PEX reactor.
    52  
    53  ### Responses
    54  
    55  After a PEX request is sent to a peer, the node expects to receive,
    56  as a response, a `PexAddrs` message from the peer.
    57  This message encodes a list of peer addresses that are
    58  [added to address book](./addressbook.md#adding-addresses),
    59  having the peer from which the PEX response was received as their source.
    60  
    61  Received PEX responses are handled by the `ReceiveAddrs` method of the PEX reactor.
    62  In the case of a PEX response received from a peer which is configured as
    63  a seed node, the PEX reactor attempts immediately to dial the provided peer
    64  addresses, as detailed [here](./peer_manager.md#fast-dialing).
    65  
    66  ### Misbehavior
    67  
    68  Sending multiple PEX requests to a peer, before receiving a reply from it,
    69  is considered a misbehavior.
    70  To prevent it, the node maintains a `requestsSent` set of outstanding
    71  requests, indexed by destination peers.
    72  While a peer ID is present in the `requestsSent` set, the node does not send
    73  further PEX requests to that peer.
    74  A peer ID is removed from the `requestsSent` set when a PEX response is
    75  received from it.
    76  
    77  Sending a PEX response to a peer that has not requested peer addresses
    78  is also considered a misbehavior.
    79  So, if a PEX response is received from a peer that is not registered in
    80  the `requestsSent` set, a `ErrUnsolicitedList` error is produced.
    81  This leads the peer to be disconnected and [marked as a bad peer](./addressbook.md#bad-peers).
    82  
    83  ## Providing Addresses
    84  
    85  When a node receives a `PexRequest` message from a peer,
    86  it replies with a `PexAddrs` message.
    87  
    88  This message encodes a [random selection of peer addresses](./addressbook.md#random-selection)
    89  retrieved from the address book.
    90  
    91  Sending a PEX response to a peer is implemented by the `SendAddrs` method of
    92  the PEX reactor.
    93  
    94  ### Misbehavior
    95  
    96  Requesting peer addresses too often is considered a misbehavior.
    97  Since node are expected to send PEX requests every `ensurePeersPeriod`,
    98  the minimum accepted interval between requests from the same peer is set
    99  to `ensurePeersPeriod / 3`, 10 seconds by default.
   100  
   101  The `receiveRequest` method is responsible for verifying this condition.
   102  The node keeps a `lastReceivedRequests` map with the time of the last PEX
   103  request received from every peer.
   104  If the interval between successive requests is less than the minimum accepted
   105  one, the peer is disconnected and [marked as a bad peer](./addressbook.md#bad-peers).
   106  An exception is made for the first two PEX requests received from a peer.
   107  
   108  > The probably reason is that, when a new peer is added, the two conditions for
   109  > a node to request peer addresses can be triggered with an interval lower than
   110  > the minimum accepted interval.
   111  > Since this is a legit behavior, it should not be punished.
   112  
   113  ## Seed nodes
   114  
   115  A seed node is a node configured to operate in `SeedMode`.
   116  
   117  ### Crawling peers
   118  
   119  Seed nodes crawl the network, connecting to random peers and sending PEX
   120  requests to them, in order to learn as many peer addresses as possible.
   121  More specifically, a node operating in seed mode sends PEX requests in two cases:
   122  
   123  1. When an outbound peer is added, and the seed node needs more peer addresses,
   124     it requests peer addresses to the new peer
   125  2. Periodically, the `crawlPeersRoutine` sends PEX requests to a random set of
   126     peers, whose addresses are registered in the Address Book
   127  
   128  The first case also applies for nodes not operating in seed mode.
   129  The second case replaces the second for regular nodes, as seed nodes do not
   130  run the `ensurePeersRoutine`, as regular nodes,
   131  but run the `crawlPeersRoutine`, which is not run by regular nodes.
   132  
   133  The `crawlPeersRoutine` periodically, every 30 seconds (`crawlPeerPeriod`),
   134  starts a new peer discovery round.
   135  First, the seed node retrieves a random selection of peer addresses from its
   136  Address Book.
   137  This selection is produced in the same way as in the random selection of peer
   138  addresses that are [provided](#providing-addresses) to a requesting peer.
   139  Peers that the seed node has crawled recently,
   140  less than 2 minutes ago (`minTimeBetweenCrawls`), are removed from this selection.
   141  The remaining peer addresses are registered in the `crawlPeerInfos` table.
   142  
   143  The seed node is not necessarily connected to the peer whose address is
   144  selected for each round of crawling.
   145  So, the seed node dials the selected peer addresses.
   146  This is performed in foreground, one peer at a time.
   147  As a result, a round of crawling can take a substantial amount of time.
   148  For each selected peer it succeeds dialing to, this include already connected
   149  peers, the seed node sends a PEX request.
   150  
   151  Dialing a selected peer address can fail for multiple reasons.
   152  The seed node might have attempted to dial the peer too many times.
   153  In this case, the peer address is marked as [bad in the address book](./addressbook.md#bad-peers).
   154  The seed node might have attempted to dial the peer recently, without success,
   155  and the exponential `backoffDuration` has not yet passed.
   156  Or the current connection attempt might fail, which is registered in the address book.
   157  
   158  Failures to dial to a peer address produce an information that is important for
   159  a seed node.
   160  They indicate that a peer is unreachable, or is not operating correctly, and
   161  therefore its address should not be provided to other nodes.
   162  This occurs when, due to multiple failed connection attempts or authentication
   163  failures, the peer address ends up being removed from the address book.
   164  As a result, the periodically crawling of selected peers not only enables the
   165  discovery of new peers, but also allows the seed node to stop providing
   166  addresses of bad peers.
   167  
   168  ### Offering addresses
   169  
   170  Nodes operating in seed mode handle PEX requests differently than regular
   171  nodes, whose operation is described [here](#providing-addresses).
   172  
   173  This distinction exists because nodes dial a seed node with the main, if not
   174  exclusive goal of retrieving peer addresses.
   175  In other words, nodes do not dial a seed node because they intend to have it as
   176  a peer in the multiple CometBFT protocols, but because they believe that a
   177  seed node is a good source of addresses of nodes to which they can establish
   178  connections and interact in the multiple CometBFT protocols.
   179  
   180  So, when a seed node receives a `PexRequest` message from an inbound peer,
   181  it sends a `PexAddrs` message, containing a selection of peer
   182  addresses, back to the peer and *disconnects* from it.
   183  Seed nodes therefore treat inbound connections from peers as a short-term
   184  connections, exclusively intended to retrieve peer addresses.
   185  Once the requested peer addresses are sent, the connection with the peer is closed.
   186  
   187  Moreover, the selection of peer addresses provided to inbound peers by a seed
   188  node, although still essentially random, has a [bias toward old
   189  addresses](./addressbook.md#random-selection-with-bias).
   190  The selection bias is defined by `biasToSelectNewPeers`, hard-coded to `30%`,
   191  meaning that `70%` of the peer addresses provided by a seed node are expected
   192  to be old addresses.
   193  Although this nomenclature is not clear, *old* addresses are the addresses that
   194  survived the most in the address book, that is, are addresses that the seed
   195  node believes being from *good* peers (more details [here](./addressbook.md#good-peers)).
   196  
   197  Another distinction is on the handling of potential [misbehavior](#misbehavior-1)
   198  of peers requesting addresses.
   199  A seed node does not enforce, a priori, a minimal interval between PEX requests
   200  from inbound peers.
   201  Instead, it does not reply to more than one PEX request per peer inbound
   202  connection, and, as above mentioned, it disconnects from incoming peers after
   203  responding to them.
   204  If the same peer dials again to the seed node and requests peer addresses, the
   205  seed node will reply to this peer like it was the first time it has requested
   206  peer addresses.
   207  
   208  > This is more an implementation restriction than a desired behavior.
   209  > The `lastReceivedRequests` map stores the last time a PEX request was
   210  > received from a peer, and the entry relative to a peer is removed from this
   211  > map when the peer is disconnected.
   212  >
   213  > It is debatable whether this approach indeed prevents abuse against seed nodes.
   214  
   215  ### Disconnecting from peers
   216  
   217  Seed nodes treat connections with peers as short-term connections, which are
   218  mainly, if not exclusively, intended to exchange peer addresses.
   219  
   220  In the case of inbound peers, that have dialed the seed node, the intent of the
   221  connection is achieved once a PEX response is sent to the peer.
   222  The seed node thus disconnects from an inbound peer after sending a `PexAddrs`
   223  message to it.
   224  
   225  In the case of outbound peers, which the seed node has dialed for crawling peer
   226  addresses, the intent of the connection is essentially achieved when a PEX
   227  response is received from the peer.
   228  The seed node, however, does not disconnect from a peer after receiving a
   229  selection of peer addresses from it.
   230  As a result, after some rounds of crawling, a seed node will have established
   231  connections to a substantial amount of peers.
   232  
   233  To couple with the existence of multiple connections with peers that have no
   234  longer purpose for the seed node, the `crawlPeersRoutine` also invokes, after
   235  each round of crawling, the `attemptDisconnects` method.
   236  This method retrieves the list of connected peers from the switch, and
   237  disconnects from peers that are not persistent peers, and with which a
   238  connection is established for more than `SeedDisconnectWaitPeriod`.
   239  This period is a configuration parameter, set to 28 hours when the PEX reactor
   240  is created by the default node constructor.