trpc.group/trpc-go/trpc-go@v1.0.3/restful/README.md

English | [中文](README.zh_CN.md)

# Introduction

The tRPC framework uses PB to define services, but it is still a common requirement to provide REST-style APIs based on
the HTTP protocol. Unifying RPC and REST is not an easy task, and the tRPC-Go framework's HTTP RPC protocol aims to 
define the same set of PB files that can be called through RPC (through the client's NewXXXClientProxy provided by 
the stub code) or through native HTTP requests. However, such HTTP calls do not comply with the RESTful specification,
for example, custom routes cannot be defined, wildcards are not supported, and the response body is empty when an 
error occurs (the error message can only be placed in the response header). Therefore, trpc additionally support the
RESTful protocol and no longer attempt to force RPC and REST together. If the service is specified as RESTful 
protocol, it does not support the use of stub code calls and only supports HTTP client calls. However, the benefit 
of this approach is that it can provide APIs that comply with RESTful specification through the protobuf annotation
in the same set of PB files and can use various tRPC framework plugins or filters.

# Principles
## Transcoder

Unlike other protocol plugins in the tRPC-Go framework, the RESTful protocol plugin implements a tRPC and HTTP/JSON 
transcoder based on the tRPC HttpRule at the Transport layer. This eliminates the need for Codec encoding and decoding 
processes as PB is directly obtained after transcoding and processed in the REST Stub generated by the trpc tool.

![restful-overall-design](/.resources/user_guide/server/restful/restful-overall-design.png)

## Transcoder Core: HttpRule
For a service defined using the same set of PB files, support for both RPC and REST calls requires a set of rules to 
indicate the mapping between RPC and REST, or more precisely, the transcoding between PB and HTTP/JSON. In the industry, 
Google has defined such rules, namely HttpRule, which tRPC's implementation also references. tRPC's HttpRule needs to 
be specified in the PB file as an option: option (trpc.api.http), which means that the same set of PB-defined services 
support both RPC and REST calls. 
Now, let's take an example of how to bind HttpRule to the `SayHello` method in a Greeter service:

```protobuf
service Greeter {
  rpc SayHello(HelloRequest) returns (HelloReply) {
    option (trpc.api.http) = {
      post: "/v1/foobar/{name}"
      body: "*"
      additional_bindings: {
        post: "/v1/foo/{name=/x/y/**}"
        body: "single_nested"
        response_body: "message"
      }
    };
  }
}
message HelloRequest {
  string name = 1;
  Nested single_nested = 2;
  oneof oneof_value {
    google.protobuf.Empty oneof_empty = 3;
    string oneof_string = 4;
  }
}
message Nested {
  string name = 1;
}
message HelloReply {
  string message = 1;
}
```

Through the above example, it can be seen that HttpRule has the following fields: 

> - The "body" field indicates which field of the PB request message is carried in the HTTP request body. 
> - The "response_body" field indicates which field of the PB response message is carried in the HTTP response body. 
> - The "additional_bindings" field represents additional HttpRule, meaning that an RPC method can be bound to multiple HttpRules.


Combining the specific rules of HttpRule, let's take a look at how the HTTP request/response are mapped to HelloRequest and 
HelloReply in the above example:

> When mapping, the "leaf fields" of the RPC request Proto Message (which refers to the fields that cannot be nested and 
> traversed further, in the above example HelloRequest.Name is a leaf field, while HelloRequest.SingleNested is not, 
> and only HelloRequest.SingleNested.Name is) are mapped in three ways:
>
> * The leaf fields are referenced by the URL Path of the HttpRule: If the URL Path of the HttpRule references one or 
> more fields in the RPC request message, then these fields are passed through the HTTP request URL Path. However, these
> fields must be non-array fields of native basic types, and do not support fields of message types or array fields. 
> In the above example, if the HttpRule selector field is defined as post: "/v1/foobar/{name}", then the value of the 
> HelloRequest.Name field is mapped to "xyz" when the HTTP request POST /v1/foobar/xyz is made.
> * The leaf fields are referenced by the Body of the HttpRule: If the field to be mapped is specified in the Body of 
> the HttpRule, then this field in the RPC request message is passed through the HTTP request Body. In the above example, 
> if the HttpRule body field is defined as body: "name", then the value of the HelloRequest.Name field is mapped to 
> "xyz" when the HTTP request Body is "xyz".
> * Other leaf fields: Other leaf fields are automatically turned into URL query parameters, and if they are repeated 
> fields, multiple queries of the same URL query parameter are supported. In the above example, if the selector in 
> the additional_bindings specifies post: "/v1/foo/{name=/x/y/*}", and the body is not specified as body: "", then 
> all fields in HelloRequest except HelloRequest.Name are passed through URL query parameters. For example, if the 
> HTTP request POST /v1/foo/x/y/z/xyz?single_nested.name=abc is made, the value of the HelloRequest.Name field is 
> mapped to "/x/y/z/xyz", and the value of the HelloRequest.SingleNested.Name field is mapped to "abc".
>
> Supplement:
> * If the field is not specified in the Body of the HttpRule and is defined as "", then each field of the request 
> message that is not bound by the URL path is passed through the Body of the HTTP request. That is, the URL query 
> parameters are invalid.
> * If the Body of the HttpRule is empty, then every field of the request message that is not bound by the URL path 
> becomes a URL query parameter. That is, the Body is invalid.
> * If the response_body of the HttpRule is empty, then the entire PB response message will be serialized into the 
> HTTP response Body. In the above example, if response_body is "", then the serialized HelloReply is the HTTP response Body.
> * HttpRule body and response_body fields can reference fields of the PB Message, which may or may not be leaf fields, 
> but must be first-level fields in the PB Message. For example, for HelloRequest, HttpRule body can be defined as 
> "name" or "single_nested", but not as "single_nested.name".

Now let's take a look at a few more examples to better understand how to use HttpRule.


**1. Take the content that matches "messages/\*" inside the URL Path as the value of the "name" field:**

```protobuf
service Messaging {
  rpc GetMessage(GetMessageRequest) returns (Message) {
    option (trpc.api.http) = {
        get: "/v1/{name=messages/*}"
    };
  }
}
message GetMessageRequest {
  string name = 1; // Mapped to URL path.
}
message Message {
  string text = 1; // The resource content.
}
```

The HttpRule above results in the following mapping:

| HTTP                    | tRPC                                |
| ----------------------- | ----------------------------------- |
| GET /v1/messages/123456 | GetMessage(name: "messages/123456") |



**2. A more complex nested message construction, using "123456" in the URL Path as the value of "message_id", and the 
value of "sub.subfield" in the URL Path as the value of the "subfield" field in the nested message:**

```protobuf
service Messaging {
  rpc GetMessage(GetMessageRequest) returns (Message) {
    option (trpc.api.http) = {
        get:"/v1/messages/{message_id}"
    };
  }
}
message GetMessageRequest {
  message SubMessage {
    string subfield = 1;
  }
  string message_id = 1; // Mapped to URL path.
  int64 revision = 2;    // Mapped to URL query parameter `revision`.
  SubMessage sub = 3;    // Mapped to URL query parameter `sub.subfield`.
}
```

The HttpRule above results in the following mapping:

| HTTP                                                | tRPC                                                                          |
| --------------------------------------------------- | ----------------------------------------------------------------------------- |
| GET /v1/messages/123456?revision=2&sub.subfield=foo | GetMessage(message_id: "123456" revision: 2 sub: SubMessage(subfield: "foo")) |



**3. Parse the entire HTTP Body as a Message type, i.e. use "Hi!" as the value of "message.text":**


```protobuf
service Messaging {
  rpc UpdateMessage(UpdateMessageRequest) returns (Message) {
    option (trpc.api.http) = {
      post: "/v1/messages/{message_id}"
      body: "message"
    };
  }
}
message UpdateMessageRequest {
  string message_id = 1; // mapped to the URL
  Message message = 2;   // mapped to the body
}
```

The HttpRule above results in the following mapping:

| HTTP                                       | tRPC                                                        |
| ------------------------------------------ | ----------------------------------------------------------- |
| POST /v1/messages/123456 { "text": "Hi!" } | UpdateMessage(message_id: "123456" message { text: "Hi!" }) |



**4. Parse the field in the HTTP Body as the "text" field of the Message:**

```protobuf
service Messaging {
  rpc UpdateMessage(Message) returns (Message) {
    option (trpc.api.http) = {
      post: "/v1/messages/{message_id}"
      body: "*"
    };
  }
}
message Message {
  string message_id = 1;
  string text = 2;
}
```

The HttpRule above results in the following mapping:

| HTTP                                      | tRPC                                            |
| ----------------------------------------- | ----------------------------------------------- |
| POST/v1/messages/123456 { "text": "Hi!" } | UpdateMessage(message_id: "123456" text: "Hi!") |

**5. Using additional_bindings to indicate APIs with additional bindings:**

```protobuf
service Messaging {
  rpc GetMessage(GetMessageRequest) returns (Message) {
    option (trpc.api.http) = {
      get: "/v1/messages/{message_id}"
      additional_bindings {
        get: "/v1/users/{user_id}/messages/{message_id}"
      }
    };
  }
}
message GetMessageRequest {
  string message_id = 1;
  string user_id = 2;
}
```

The HttpRule above results in the following mapping:

| HTTP                             | tRPC                                           |
| -------------------------------- | ---------------------------------------------- |
| GET /v1/messages/123456          | GetMessage(message_id: "123456")               |
| GET /v1/users/me/messages/123456 | GetMessage(user_id: "me" message_id: "123456") |

# Implementation

Please refer to the [trpc-go/restful](/restful).

# Examples

After understanding HttpRule, let's take a look at how to enable tRPC-Go's RESTful service.

**1. PB Definition**

First, update the `trpc-cmdline` tool to the latest version. To use the **trpc.api.http** annotation, you need 
to import a proto file:

```protobuf
import "trpc/api/annotations.proto";
```

Let's define a PB for a Greeter service:

```protobuf
...
import "trpc/api/annotations.proto";
service Greeter {
  rpc SayHello(HelloRequest) returns (HelloReply) {
    option (trpc.api.http) = {
      post: "/v1/foobar"
      body: "*"
      additional_bindings: {
        post: "/v1/foo/{name}"
      }
    };
  }
}
message HelloRequest {
  string name = 1;
  ...
}  
...
```

**2. Generating Stub Code**

Use the `trpc create` command to generate stub code directly.

**3. Configuration**

Just like configuring other protocols, set the protocol configuration of the service in `trpc_go.yaml` to `restful`.

```yaml
server: 
  ...
  service:                                         
    - name: trpc.test.helloworld.Greeter      
      ip: 127.0.0.1                            
      # nic: eth0
      port: 8080                
      network: tcp                             
      protocol: restful              
      timeout: 1000
```

A more common scenario is to configure a tRPC protocol service and add a RESTful protocol service, so that one set of 
PB files can simultaneously support providing both RPC services and RESTful services.

```yaml
server: 
  ...
  service:                                         
    - name: trpc.test.helloworld.Greeter1      
      ip: 127.0.0.1                            
      # nic: eth0
      port: 12345                
      network: tcp                             
      protocol: trpc              
      timeout: 1000
    - name: trpc.test.helloworld.Greeter2      
      ip: 127.0.0.1                            
      # nic: eth0
      port: 54321                
      network: tcp                             
      protocol: restful              
      timeout: 1000
```

**Note: Each service in tRPC must be configured with a different port number.**

**4. starting the Service**

Starting the service is the same as other protocols:

```go
package main
import (
    ...
    pb "trpc.group/trpc-go/trpc-go/examples/restful/helloworld"
)
func main() {
    s := trpc.NewServer()
    pb.RegisterGreeterService(s, &greeterServerImpl{})
    // Start
    if err := s.Serve(); err != nil {
        ...
    }
}
```

**5. Calling**

Since you are building a RESTful service, please use any REST client to make calls. It is not supported to use the RPC 
method of calling using NewXXXClientProxy.

```go
package main
import "net/http"
func main() {
    ...
    // native HTTP invocation
    req, err := http.NewRequest("POST", "http://127.0.0.1:8080/v1/foobar", bytes.Newbuffer([]byte(`{"name": "xyz"}`)))
    if err != nil {
        ...
    }
    cli := http.Client{}
    resp, err := cli.Do(req)
    if err != nil {
        ...
    }
    ...
}
```
Of course, if you have configured a tRPC protocol service in step 3 [Configuration], you can still call the tRPC 
protocol service using the RPC method of NewXXXClientProxy, but be sure to distinguish the port.


**6. Mapping Custom HTTP Headers to RPC Context**

HttpRule resolves the transcoding between tRPC message body and HTTP/JSON, but how can HTTP requests pass the RPC call 
context? This requires defining the mapping of HTTP headers to RPC context.

The HeaderMatcher for RESTful service is defined as follows:

```go
type HeaderMatcher func(
    ctx context.Context,
    w http.ResponseWriter,
    r *http.Request,
    serviceName, methodName string,
) (context.Context, error)
```

The default handling of HeaderMatcher is as follows:

```go
var defaultHeaderMatcher = func(
    ctx context.Context,
    w http.ResponseWriter,
    req *http.Request,
    serviceName, methodName string,
) (context.Context, error) {
    // It is recommended to customize and pass the codec.Msg in the ctx, and specify the target service and method name.
    ctx, msg := codec.WithNewMessage(ctx)
    msg.WithCalleeServiceName(service)
    msg.WithServerRPCName(method)
    msg.WithSerializationType(codec.SerializationTypePB)
    return ctx, nil
}
```

You can set the HeaderMatcher through the `WithOptions` method:

```go
service := server.New(server.WithRESTOptions(restful.WithHeaderMatcher(xxx)))
```

**7. Customize the Response Handling [Set the Return Code for Successful Request Handling]**

The "response_body" field in HttpRule specifies the RPC response, for example, in the above example, the "HelloReply" 
needs to be serialized into the HTTP Response Body as a whole or for a specific field. However, users may want to 
perform additional custom operations, such as setting the response code for successful requests.

The custom response handling function for RESTful services is defined as follows:

```go
type CustomResponseHandler func(
    ctx context.Context,
    w http.ResponseWriter,
    r *http.Request,
    resp proto.Message,
    body []byte,
) error
```

The "trpc-go/restful" package provides a function that allows users to set the response code for successful request handling:

```go
func SetStatusCodeOnSucceed(ctx context.Context, code int) {}
```

The default custom response handling function is as follows:

```go
var defaultResponseHandler = func(
    ctx context.Context,
    w http.ResponseWriter,
    r *http.Request,
    resp proto.Message,
    body []byte,
) error {
    // compress
    var writer io.Writer = w
    _, compressor := compressorForRequest(r)
    if compressor != nil {
        writeCloser, err := compressor.Compress(w)
        if err != nil {
            return fmt.Errorf("failed to compress resp body: %w", err)
        }
        defer writeCloser.Close()
        w.Header().Set(headerContentEncoding, compressor.ContentEncoding())
        writer = writeCloser
    }
    // Set StatusCode
    statusCode := GetStatusCodeOnSucceed(ctx)
    w.WriteHeader(statusCode)
    // Set body
    if statusCode != http.StatusNoContent && statusCode != http.StatusNotModified {
        writer.Write(body)
    }
    return nil
}
```

If using the default custom response handling function, users can set the return code in their own RPC handling functions
 (if not set, it will return 200 for success):

```go
func (s *greeterServerImpl) SayHello(ctx context.Context, req *pb.HelloRequest) (*pb.HelloReply, error) {   
    ...
    restful.SetStatusCodeOnSucceed(ctx, 200) // Set the return code for success.
    return rsp, nil
}
```

You can set the HeaderMatcher through the `WithOptions` method:

```go
var xxxResponseHandler = func(
    ctx context.Context,
    w http.ResponseWriter,
    r *http.Request,
    resp proto.Message,
    body []byte,
) error {
    reply, ok := resp.(*pb.HelloReply)
    if !ok {
        return errors.New("xxx")
    }
    ...
    w.Header().Set("x", "y")
    expiration := time.Now()
    expiration := expiration.AddDate(1, 0, 0)
    cookie := http.Cookie{Name: "abc", Value: "def", Expires: expiration}
    http.SetCookie(w, &cookie)
    w.Write(body)
    return nil
}
...
service := server.New(server.WithRESTOptions(restful.WithResponseHandler(xxxResponseHandler)))
```

**8. Custom Error Handling [Error Code]**

The definition of the error handling function for RESTful services is as follows:

```go
type ErrorHandler func(context.Context, http.ResponseWriter, *http.Request, error)
```

You can set the HeaderMatcher through the `WithOptions` method:

```go
var xxxErrorHandler = func(ctx context.Context, w http.ResponseWriter, r *http.Request, err error) {
    if err == errors.New("say hello failed") {
        w.WriteHeader(500)
    }
    ...
}
service := server.New(server.WithRESTOptions(restful.WithErrorHandler(xxxErrorHandler)))
```


**Recommend using the default error handling function of the trpc-go/restful package or referring to the implementation
 to create your own error handling function.**

Regarding **error codes:**

If an error of the type defined in the "trpc-go/errs" package is returned during RPC processing, the default error 
handling function of "trpc-go/restful" will map tRPC's error codes to HTTP error codes. If users want to decide 
what error code is used for a specific error, they can use `WithStatusCode` defined in the "trpc-go/restful" package.


```go
type WithStatusCode struct {
    StatusCode int
    Err        error
}
```

Wrap your own error in a function and return it, such as:

```go
func (s *greeterServerImpl) SayHello(ctx context.Context, req *hpb.HelloRequest, rsp *hpb.HelloReply) (err error) {
    if req.Name != "xyz" {
        return &restful.WithStatusCode{
            StatusCode: 400,
            Err:        errors.New("test error"),
        }
    }
    return nil
}
```

If the error type is not the `Error` type defined by "trpc-go/errs" and is not wrapped with `WithStatusCode` defined 
in the "trpc-go/restful" package, the default error code 500 will be returned.

**9. Body Serialization and Compression**

Like normal REST requests, it's specified through HTTP headers and supports several popular formats.

> **Supported Content-Type (or Accept) for serialization: application/json, application/x-www-form-urlencoded, 
> application/octet-stream. By default it is application/json.**

Serialization interface is defined as follows:

```go
type Serializer interface {
    // Marshal serializes the tRPC message or one of its fields into the HTTP body. 
    Marshal(v interface{}) ([]byte, error)
    // Unmarshal deserializes the HTTP body into the tRPC message or one of its fields. 
    Unmarshal(data []byte, v interface{}) error
    // Name Serializer Name
    Name() string
    // ContentType  is set when returning the HTTP response.
    ContentType() string
}
```

**Users can implement their own serializer and register it using the `restful.RegisterSerializer()` function.**

> **Compression is supported through Content-Encoding (or Accept-Encoding): gzip. By default, there is no compression.**

Compression interface is defined as follows:

```go
type Compressor interface {
    // Compress 
    Compress(w io.Writer) (io.WriteCloser, error)
    // Decompress 
    Decompress(r io.Reader) (io.Reader, error)
    // Name represents the name of the compressor.
    Name() string
    // ContentEncoding represents the Content-Encoding that is set when returning the HTTP response.
    ContentEncoding() string
}
```

**Users can implement their own serializer and register it using the `restful.RegisterSerializer()` function.**

**10. Cross-Origin Requests**

RESTful also supports [trpc-filter/cors](https://github.com/trpc-ecosystem/go-filter/tree/main/cors) cross-origin requests 
plugin. To use it, you need to add the HTTP OPTIONS method in pb by using `custom`, for example:

```protobuf
service HelloTrpcGo {
  rpc Hello(HelloReq) returns (HelloRsp) {
    option (trpc.api.http) = {
      post: "/hello"
      body: "*"
      additional_bindings: {
        get: "/hello/{name}"
      }
      additional_bindings: {
        custom: { // use custom verb
          kind: "OPTIONS"
          path: "/hello"
        }
      }
    };
  }
}
```

Next, regenerate the stub code using the trpc-cmdline command-line tool.
Finally, add the CORS plugin to the service interceptors.

If you do not want to modify the protobuf file, RESTful also provides a code-based custom method for cross-origin requests.

The RESTful protocol plugin will generate a corresponding http.Handler for each Service. You can retrieve it before 
starting to listen, and replace it with you own custom http.Handler:


```go
func allowCORS(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if origin := r.Header.Get("Origin"); origin != "" {
            w.Header().Set("Access-Control-Allow-Origin", origin)
            if r.Method == "OPTIONS" && r.Header.Get("Access-Control-Request-Method") != "" {
                preflightHandler(w, r)
                return
            }
        }
        h.ServeHTTP(w, r)
    })
}
func main() {
    // set custom header matcher
    s := trpc.NewServer()
    //  register service implementation
    pb.RegisterPingService(s, &pingServiceImpl{})
    // retrieve restful.Router
    router := restful.GetRouter(pb.PingServer_ServiceDesc.ServiceName)
    // wrap it up and re-register it again
    restful.RegisterRouter(pb.PingServer_ServiceDesc.ServiceName, allowCORS(router))
    // start
    if err := s.Serve(); err != nil {
        log.Fatal(err)
    }
}
```

# Performance

To improve performance, the RESTful protocol plugin also supports handling HTTP packets based on [fasthttp](https://github.com/valyala/fasthttp). 
The performance of the RESTful protocol plugin is related to the complexity of the registered URL path and the method of 
passing PB Message fields. Here is a comparison between the two modes in the simplest echo test scenario: 

Test PB:

```protobuf
service Greeter {
  rpc SayHello(HelloRequest) returns (HelloReply) {
    option (trpc.api.http) = {
      get: "/v1/foobar/{name}"
    };
  }
}
message HelloRequest {
  string name = 1;
}
message HelloReply {
  string message = 1;
}
```

Greeter implementation

```go
type greeterServiceImpl struct{}
func (s *greeterServiceImpl) SayHello(ctx context.Context, req *pb.HelloRequest) (*pb.HelloReply, error) {
    return &pb.HelloReply{Message: Name}, nil
}
```

Test machine: 8 cores 

| mode              | QPS when P99 < 10ms |
| ----------------- | ------------------- |
| based on net/http | 16w                 |
| base on fasthttp  | 25w                 |

- To enable fasthttp, add one line of code before `trpc.NewServer()` as follows:

```go
package main
import (
    "trpc.group/trpc-go/trpc-go/transport"
    thttp "trpc.group/trpc-go/trpc-go/http"
)
func main() {
    transport.RegisterServerTransport("restful", thttp.NewRESTServerTransport(true))
    s := trpc.NewServer()
    ...
}
```