trpc.group/trpc-go/trpc-go@v1.0.3/docs/user_guide/tnet.md (about)

     1  English | [中文](tnet.zh_CN.md)
     2  
     3  # Using high-performance networking framework tnet with tRPC-Go
     4  
     5  
     6  ## Introduction
     7  
     8  The Golang Net library provides a simple non-blocking interface, with a network model that employs `Goroutine-per-Connection`. In most scenarios, this model is straightforward and user-friendly. However, when dealing with thousands or even millions of connections, allocating a goroutine for each connection consumes a significant amount of memory, and managing a large number of goroutines becomes challenging.
     9  
    10  To support the capability of handling millions of connections, it is essential to break away from the `Goroutine-per-Connection` model. The high-performance networking library [tnet](https://github.com/trpc-group/tnet) is built on a `Reactor` network model, enabling the handling of millions of connections. The tRPC-Go framework integrates the tnet network library, thereby providing support for handling millions of connections. In addition to this, tnet also offers features such as batch packet transmission and reception, zero-copy buffering, and fine-grained memory management optimizations, making it outperform the native Golang Net library in terms of performance.
    11  
    12  ## Principle
    13  
    14  We use two diagrams to illustrate the basic principles of the `Goroutine-per-Connection` model and the `Reactor` model in Golang.
    15  
    16  ### Goroutine-per-Connection
    17  
    18  ![goroutine_per_connection](/.resources/user_guide/tnet/goroutine_per_connection.png)
    19  
    20  In the Goroutine-per-Connection model, when the server accepts a new connection, it creates a goroutine for that connection, and then reads data from the connection, processes the data, and sends data back to the connection in that goroutine.
    21  
    22  The scenario of millions of connections usually refers to long connection scenarios. Although the total number of connections is huge, only a small number of connections are active at any given time. Active connections refer to connections that have data to be read or written at a certain moment. Conversely, when there is no data to be read or written on a connection, it is called an idle connection. The idle connection goroutine will block on the Read call. Although the goroutine does not occupy scheduling resources, it still occupies memory resources, which ultimately leads to huge memory consumption. In this model, allocating a goroutine for each connection in a scenario of millions of connections is expensive.
    23  
    24  For example, as shown in the above diagram, the server accepts 5 connections and creates 5 goroutines. At this moment, the first 3 connections are active connections, and data can be read from them smoothly. After processing the data, the server sends data back to the connections to complete a data exchange, and then starts the second round of data reading. The last 2 connections are idle connections, and when reading data from them, the process will be blocked. Therefore, the subsequent process is not triggered. As we can see, although only 3 connections can successfully read data at this moment, 5 goroutines are allocated, resulting in a 40% waste of resources. The larger the proportion of idle connections, the more resources will be wasted.
    25  
    26  ### Reactor
    27  
    28  ![reactor](/.resources/user_guide/tnet/reactor.png)
    29  
    30  The Reactor model refers to using multiplexing (epoll/kqueue) to listen for events such as read and write on file descriptors (FDs), and then performing corresponding operations when events are triggered.
    31  
    32  In the diagram, the poller structure is responsible for listening to events on FDs. Each poller occupies a goroutine, and the number of pollers is usually equal to the number of CPUs. We use a separate poller to listen for read events on the listener port to accept new connections, and then listen for read events on each connection. When a connection becomes readable, a goroutine is allocated to read data from the connection, process the data, and send data back to the connection. At this point, there will be no idle connections occupying goroutines. In a scenario of millions of connections, only active connections are allocated goroutines, which can make full use of memory resources.
    33  
    34  For example, as shown in the above diagram, the server has 5 pollers, one of which is responsible for listening to the listener events and accepting new connections, and the other 4 pollers are responsible for listening to read events on connections. When 2 connections become readable at a certain moment, a goroutine is allocated for each connection to read data, process data, and send data back to the connection. Because it is already known that these two connections are readable, the Read process will not block, and the subsequent process can be executed smoothly. When writing data back to the connection, the goroutine registers a write event with the poller, and then exits. The poller listens for write events on the connection and sends data when the connection is writable, completing a round of data exchange.
    35  
    36  ## Quick start
    37  
    38  ### Enable tnet
    39  
    40  There are two ways to enable tnet in tRPC-Go. Choose one of them for configuration. It is recommended to use the first method.
    41  
    42  (1) Add tnet in the tRPC-Go framework configuration file.
    43  
    44  (2) Use the WithTransport() method in the code to enable tnet.
    45  
    46  #### Method 1: Configuration file (recommended)
    47  
    48  Add tnet to the transport field in the tRPC-Go configuration file. Since the plugin currently only supports TCP, please do not configure the tnet plugin for UDP services. The server and client can each independently activate tnet, and they do not interfere with each other.
    49  
    50  **Server**:
    51  
    52  ```yaml
    53  server:
    54    transport: tnet # Applies to all services
    55    service:
    56      - name: trpc.app.server.service
    57        network: tcp
    58        transport: tnet # Applies only to the current service
    59  ```
    60  
    61  After the server is started, the log indicates the successful activation of tnet:
    62  
    63  `INFO tnet/server_transport.go service:trpc.app.server.service is using tnet transport, current number of pollers: 1`
    64  
    65  **Client**:
    66  
    67  ```yaml
    68  client:
    69    transport: tnet # Applies to all services
    70    service:
    71      - name: trpc.app.server.service
    72        network: tcp
    73        transport: tnet # Applies only to the current service
    74        conn_type: multiplexed # Using multiplexed connection mode
    75        multiplexed:
    76          enable_metrics: true # Enable metrics for multiplexed pool
    77  ```
    78  
    79  It's recommended to use multiplexed connection mode with tnet to enhance performance, because it can fully leverage tnet's batch packet transmission capabilities.
    80  
    81  After the client is started, the log indicates the successful activation of tnet (Trace level):
    82  
    83  `Debug tnet/client_transport.go roundtrip to:127.0.0.1:8000 is using tnet transport, current number of pollers: 1`
    84  
    85  #### Method 2: Code configuration
    86  
    87  **Server**:
    88  
    89  Notics: This method will enable tnet for all services of the server.
    90  
    91  ```go
    92  import "trpc.group/trpc-go/trpc-go/transport/tnet"
    93  
    94  func main() {
    95      // Create a ServerTransport
    96      trans := tnet.NewServerTransport()
    97      // Create a trpc server
    98      s := trpc.NewServer(server.WithTransport(trans))
    99      pb.RegisterGreeterService(s, &greeterServiceImpl{})
   100      s.Serve()
   101  }
   102  ```
   103  
   104  **Client**:
   105  
   106  ```go
   107  import "trpc.group/trpc-go/trpc-go/transport/tnet"
   108  
   109  func main() {
   110      proxy := pb.NewGreeterClientProxy()
   111      trans := tnet.NewClientTransport()
   112      rsp, err := proxy.SayHello(trpc.BackgroundContext(), &pb.HelloRequest{Msg: "Hello"}, client.WithTransport(trans))
   113  }
   114  ```
   115  
   116  ## Use Cases
   117  
   118  According to the benchmark result, tnet transport outperforms gonet transport in specific scenarios. However, not all scenarios exhibit these advantages. Here, we summarize the scenarios in which tnet transport excels.
   119  
   120  **Advantageous Scenarios for tnet：**
   121  
   122  - When using tnet in server, if the client sends requests using multiplexed connection mode, it can fully utilize tnet's batch packet transmission capabilities, leading to increased QPS and reduced CPU usage.
   123  
   124  - When using tnet in server, if there are a large number of idle connections, it can reduce memory usage by lowering the number of goroutines.
   125  
   126  - When using tnet in client, if the multiplexed mode is enabled, it can fully leverage tnet's batch packet transmission capabilities, resulting in improved QPS.
   127  
   128  **Other Scenarios:**
   129  
   130  - When using tnet in server, if the client sends requests not using multiplexed connection mode, performance is similar to gonet.
   131  
   132  - When using tnet in client, if the multiplexed mode is disable, performance is similar to gonet.
   133  
   134  ## FAQ
   135  
   136  #### Q：Does tnet support HTTP？
   137  
   138  Tnet doesn't support HTTP. When tnet is used in HTTP server/client, it automatically falls back to using the golang net package.
   139  
   140  #### Q：Why doesn't performance improve after enabling tnet?
   141  
   142  Tnet is not a universal solution, and it can significantly boost service performance by fully utilizing Writev for batch packet transmission and reducing system calls in specific scenarios. If you find that the service performance is still not satisfactory in tnet's advantageous scenarios, you can consider optimizing your service using the following steps:
   143  
   144  Enable the client-side multiplexed connection mode with tnet and make full use of Writev for batch packet transmission whenever possible;
   145  
   146  Enable tnet and multiplexed connection mode for the entire service chain. If the upstream server utilizes multiplexed, the current server can also take advantage of Writev for batch packet transmission;
   147  
   148  If you have enabled the multiplexed connection mode, you can enable metrics to inspect the number of virtual connections on each connection. If there is substantial concurrency, causing an excessive number of virtual connections on a single connection, it can also impact performance. Configure and enable multiplexed metrics accordingly.
   149  
   150  ```yaml
   151  client:
   152    service:
   153      - name: trpc.test.helloworld.Greeter1
   154        transport: tnet
   155        conn_type: multiplexed
   156        multiplexed:
   157          enable_metrics: true # Enable metrics for multiplexed pool
   158  ```
   159  
   160  Every 3 seconds, logs containing the multiplexed status are printed. For example, you can see that the current number of active connections is 1, and the total number of virtual connections is 98.
   161  
   162  `DEBUG tnet multiplex status: network: tcp, address: 127.0.0.1:7002, connections number: 1, concurrent virtual connection number: 98`
   163  
   164  It also reports status to custom metrics, and the format of the metrics items is as follows:
   165  
   166  Active connections：`trpc.MuxConcurrentConnections.$network.$address`
   167  
   168  Virtual connections：`trpc.MuxConcurrentVirConns.$network.$address`
   169  
   170  Assuming you want to set the maximum concurrent virtual connections per connection to 25, you can add the following configuration:
   171  
   172  ```yaml
   173  client:
   174    service:
   175      - name: trpc.test.helloworld.Greeter1
   176        transport: tnet
   177        conn_type: multiplexed
   178        multiplexed:
   179          enable_metrics: true
   180          max_vir_conns_per_conn: 25 # maximum number of concurrent virtual connections per connection
   181  ```
   182  
   183  #### Q：Why does it log `switch to gonet default transport, tnet server transport doesn't support network type [udp]` after enabling tnet？
   184  
   185  The log indicates tnet transport does't support UDP. It will automatically falls back to using golang net package.