github.com/vipernet-xyz/tm@v0.34.24/docs/introduction/architecture.md (about)

     1  ---
     2  order: false
     3  ---
     4  # Tendermint Architectural Overview
     5  
     6  
     7  > **November 2019**
     8  
     9  Over the next few weeks, @brapse, @marbar3778 and I (@tessr) are having a series of meetings to go over the architecture of Tendermint Core. These are my notes from these meetings, which will either serve as an artifact for onboarding future engineers; or will provide the basis for such a document.
    10  
    11  ## Communication
    12  
    13  There are three forms of communication (e.g., requests, responses, connections) that can happen in Tendermint Core: *internode communication*, *intranode communication*, and *client communication*.
    14  
    15  - Internode communication: Happens between a node and other peers. This kind of communication happens over TCP or HTTP. More on this below.
    16  - Intranode communication: Happens within the node itself (i.e., between reactors or other components). These are typically function or method calls, or occasionally happen through an event bus.
    17  
    18  - Client communication: Happens between a client (like a wallet or a browser) and a node on the network.
    19  
    20  ### Internode Communication
    21  
    22  Internode communication can happen in two ways:
    23  
    24  1. TCP connections through the p2p package
    25      - Most common form of internode communication
    26      - Connections between nodes are persisted and shared across reactors, facilitated by the switch. (More on the switch below.)
    27  2. RPC over HTTP
    28      - Reserved for short-lived, one-off requests
    29      - Example: reactor-specific state, like height
    30      - Also possible: web-sockets connected to channels for notifications (like new transactions)
    31  
    32  ### P2P Business (the Switch, the PEX, and the Address Book)
    33  
    34  When writing a p2p service, there are two primary responsibilities:
    35  
    36  1. Routing: Who gets which messages?
    37  2. Peer management: Who can you talk to? What is their state? And how can you do peer discovery?
    38  
    39  The first responsibility is handled by the Switch:
    40  
    41  - Responsible for routing connections between peers
    42  - Notably _only handles TCP connections_; RPC/HTTP is separate
    43  - Is a dependency for every reactor; all reactors expose a function `setSwitch`
    44  - Holds onto channels (channels on the TCP connection--NOT Go channels) and uses them to route
    45  - Is a global object, with a global namespace for messages
    46  - Similar functionality to libp2p
    47  
    48  TODO: More information (maybe) on the implementation of the Switch.
    49  
    50  The second responsibility is handled by a combination of the PEX and the Address Book.
    51  
    52   TODO: What is the PEX and the Address Book?
    53  
    54  #### The Nature of TCP, and Introduction to the `mconnection`
    55  
    56  Here are some relevant facts about TCP:
    57  
    58  1. All TCP connections have a "frame window size" which represents the packet size to the "confidence;" i.e., if you are sending packets along a new connection, you must start out with small packets. As the packets are received successfully, you can start to send larger and larger packets. (This curve is illustrated below.) This means that TCP connections are slow to spin up.
    59  2. The syn/ack process also means that there's a high overhead for small, frequent messages 
    60  3. Sockets are represented by file descriptors.
    61  
    62  ![tcp](../imgs/tcp-window.png)
    63  
    64  In order to have performant TCP connections under the conditions  created in Tendermint, we've created the `mconnection`, or the multiplexing connection. It is our own protocol built on top of TCP. It lets us reuse TCP connections to minimize overhead, and it keeps the window size high by sending auxiliary messages when necessary.
    65  
    66  The `mconnection` is represented by a struct, which contains a batch of messages, read and write buffers, and a map of channel IDs to reactors. It communicates with TCP via file descriptors, which it can write to. There is one `mconnection` per peer connection.
    67  
    68  The `mconnection` has two methods: `send`, which takes a raw handle to the socket and writes to it; and `trySend`, which writes to a different buffer. (TODO: which buffer?)
    69  
    70  The `mconnection` is owned by a peer, which is owned (potentially with many other peers) by a (global) transport, which is owned by the (global) switch:
    71  
    72  <!-- markdownlint-disable -->
    73  ```
    74  switch
    75   transport
    76    peer
    77     mconnection
    78    peer
    79     mconnection
    80    peer
    81     mconnection
    82  ```
    83  <!-- markdownlint-restore -->
    84  
    85  ## node.go
    86  
    87  node.go is the entrypoint for running a node. It sets up reactors, sets up the switch, and registers all the RPC endpoints for a node.
    88  
    89  ## Types of Nodes
    90  
    91  
    92  1. Validator Node:
    93  2. Full Node:
    94  3. Seed Node:
    95  
    96  TODO: Flesh out the differences between the types of nodes and how they're configured.
    97  
    98  ## Reactors
    99  
   100  Here are some Reactor Facts:
   101  
   102  - Every reactor holds a pointer to the global switch (set through `SetSwitch()`)
   103  - The switch holds a pointer to every reactor (`addReactor()`)
   104  - Every reactor gets set up in node.go (and if you are using custom reactors, this is where you specify that)
   105  - `addReactor` is called by the switch; `addReactor` calls `setSwitch` for that reactor
   106  - There's an assumption that all the reactors are added before
   107  - Sometimes reactors talk to each other by fetching references to one another via the switch (which maintains a pointer to each reactor). **Question: Can reactors talk to each other in any other way?**
   108  
   109  Furthermore, all reactors expose:
   110  
   111  1. A TCP channel
   112  2. A `receive` method
   113  3. An `addReactor` call
   114  
   115  The `receive` method can be called many times by the mconnection. It has the same signature across all reactors.
   116  
   117  The `addReactor` call does a for loop over all the channels on the reactor and creates a map of channel IDs->reactors. The switch holds onto this map, and passes it to the _transport_, a thin wrapper around TCP connections.
   118  
   119  The following is an exhaustive (?) list of reactors:
   120  
   121  - Blockchain Reactor
   122  - Consensus Reactor
   123  - Evidence Reactor
   124  - Mempool Reactor
   125  - PEX Reactor
   126  
   127  Each of these will be discussed in more detail later.
   128  
   129  
   130  ### Blockchain Reactor
   131  
   132  The blockchain reactor has two responsibilities:
   133  
   134  1. Serve blocks at the request of peers
   135  2. TODO: learn about the second responsibility of the blockchain reactor