github.com/night-codes/go-json@v0.9.15/README.md (about)

     1  # go-json
     2  
     3  ![Go](https://github.com/night-codes/go-json/workflows/Go/badge.svg) [![GoDoc](https://godoc.org/github.com/night-codes/go-json?status.svg)](https://pkg.go.dev/github.com/night-codes/go-json?tab=doc) [![codecov](https://codecov.io/gh/night-codes/go-json/branch/master/graph/badge.svg)](https://codecov.io/gh/night-codes/go-json)
     4  
     5  Fast JSON encoder/decoder compatible with encoding/json for Go
     6  
     7  <img width="400px" src="https://user-images.githubusercontent.com/209884/92572337-42b42900-f2bf-11ea-973a-c74a359553a5.png"></img>
     8  
     9  # Roadmap
    10  
    11  ```
    12  * version ( expected release date )
    13  
    14  * v0.9.0
    15   |
    16   | while maintaining compatibility with encoding/json, we will add convenient APIs
    17   |
    18   v
    19  * v1.0.0
    20  ```
    21  
    22  We are accepting requests for features that will be implemented between v0.9.0 and v.1.0.0. If you have the API you need, please submit your issue [here](https://github.com/night-codes/go-json/issues).
    23  
    24  # Features
    25  
    26  -   Drop-in replacement of `encoding/json`
    27  -   Fast ( See [Benchmark section](https://github.com/night-codes/go-json#benchmarks) )
    28  -   Flexible customization with options
    29  -   Coloring the encoded string
    30  -   Can propagate context.Context to `MarshalJSON` or `UnmarshalJSON`
    31  -   Can dynamically filter the fields of the structure type-safely
    32  
    33  # Installation
    34  
    35  ```
    36  go get github.com/night-codes/go-json
    37  ```
    38  
    39  # How to use
    40  
    41  Replace import statement from `encoding/json` to `github.com/night-codes/go-json`
    42  
    43  ```
    44  -import "encoding/json"
    45  +import "github.com/night-codes/go-json"
    46  ```
    47  
    48  # JSON library comparison
    49  
    50  |                                       name                                        | encoder | decoder | compatible with `encoding/json` |
    51  | :-------------------------------------------------------------------------------: | :-----: | :-----: | :-----------------------------: |
    52  |                                   encoding/json                                   |   yes   |   yes   |               N/A               |
    53  |              [json-iterator/go](https://github.com/json-iterator/go)              |   yes   |   yes   |             partial             |
    54  |                  [easyjson](https://github.com/mailru/easyjson)                   |   yes   |   yes   |               no                |
    55  |                   [gojay](https://github.com/francoispqt/gojay)                   |   yes   |   yes   |               no                |
    56  | [segmentio/encoding/json](https://github.com/segmentio/encoding/tree/master/json) |   yes   |   yes   |             partial             |
    57  |                   [jettison](https://github.com/wI2L/jettison)                    |   yes   |   no    |               no                |
    58  |                [simdjson-go](https://github.com/minio/simdjson-go)                |   no    |   yes   |               no                |
    59  |                                night-codes/go-json                                |   yes   |   yes   |               yes               |
    60  
    61  -   `json-iterator/go` isn't compatible with `encoding/json` in many ways (e.g. https://github.com/json-iterator/go/issues/229 ), but it hasn't been supported for a long time.
    62  -   `segmentio/encoding/json` is well supported for encoders, but some are not supported for decoder APIs such as `Token` ( streaming decode )
    63  
    64  ## Other libraries
    65  
    66  -   [jingo](https://github.com/bet365/jingo)
    67  
    68  I tried the benchmark but it didn't work. Also, it seems to panic when it receives an unexpected value because there is no error handling...
    69  
    70  -   [ffjson](https://github.com/pquerna/ffjson)
    71  
    72  Benchmarking gave very slow results. It seems that it is assumed that the user will use the buffer pool properly. Also, development seems to have already stopped
    73  
    74  # Benchmarks
    75  
    76  ```
    77  $ cd benchmarks
    78  $ go test -bench .
    79  ```
    80  
    81  ## Encode
    82  
    83  <img width="700px" src="https://user-images.githubusercontent.com/209884/107126758-0845cb00-68f5-11eb-8db7-086fcf9bcfaa.png"></img> <img width="700px" src="https://user-images.githubusercontent.com/209884/107126757-07ad3480-68f5-11eb-87aa-858cc5eacfcb.png"></img>
    84  
    85  ## Decode
    86  
    87  <img width="700" alt="" src="https://user-images.githubusercontent.com/209884/107979944-bd1d6d80-7002-11eb-944b-9d17b6674e3f.png">
    88  <img width="700" alt="" src="https://user-images.githubusercontent.com/209884/107979931-b989e680-7002-11eb-87a0-66fc22d90dd4.png">
    89  <img width="700" alt="" src="https://user-images.githubusercontent.com/209884/107979940-bc84d700-7002-11eb-9647-869bbc25c9d9.png">
    90  
    91  # Fuzzing
    92  
    93  [go-json-fuzz](https://github.com/night-codes/go-json-fuzz) is the repository for fuzzing tests. If you run the test in this repository and find a bug, please commit to corpus to go-json-fuzz and report the issue to [go-json](https://github.com/night-codes/go-json/issues).
    94  
    95  # How it works
    96  
    97  `go-json` is very fast in both encoding and decoding compared to other libraries. It's easier to implement by using automatic code generation for performance or by using a dedicated interface, but `go-json` dares to stick to compatibility with `encoding/json` and is the simple interface. Despite this, we are developing with the aim of being the fastest library.
    98  
    99  Here, we explain the various speed-up techniques implemented by `go-json`.
   100  
   101  ## Basic technique
   102  
   103  The techniques listed here are the ones used by most of the libraries listed above.
   104  
   105  ### Buffer reuse
   106  
   107  Since the only value required for the result of `json.Marshal(interface{}) ([]byte, error)` is `[]byte`, the only value that must be allocated during encoding is the return value `[]byte` .
   108  
   109  Also, as the number of allocations increases, the performance will be affected, so the number of allocations should be kept as low as possible when creating `[]byte`.
   110  
   111  Therefore, there is a technique to reduce the number of times a new buffer must be allocated by reusing the buffer used for the previous encoding by using `sync.Pool`.
   112  
   113  Finally, you allocate a buffer that is as long as the resulting buffer and copy the contents into it, you only need to allocate the buffer once in theory.
   114  
   115  ```go
   116  type buffer struct {
   117      data []byte
   118  }
   119  
   120  var bufPool = sync.Pool{
   121      New: func() interface{} {
   122          return &buffer{data: make([]byte, 0, 1024)}
   123      },
   124  }
   125  
   126  buf := bufPool.Get().(*buffer)
   127  data := encode(buf.data) // reuse buf.data
   128  
   129  newBuf := make([]byte, len(data))
   130  copy(newBuf, buf)
   131  
   132  buf.data = data
   133  bufPool.Put(buf)
   134  ```
   135  
   136  ### Elimination of reflection
   137  
   138  As you know, the reflection operation is very slow.
   139  
   140  Therefore, using the fact that the address position where the type information is stored is fixed for each binary ( we call this `typeptr` ), we can use the address in the type information to call a pre-built optimized process.
   141  
   142  For example, you can get the address to the type information from `interface{}` as follows and you can use that information to call a process that does not have reflection.
   143  
   144  To process without reflection, pass a pointer (`unsafe.Pointer`) to the value is stored.
   145  
   146  ```go
   147  
   148  type emptyInterface struct {
   149      typ unsafe.Pointer
   150      ptr unsafe.Pointer
   151  }
   152  
   153  var typeToEncoder = map[uintptr]func(unsafe.Pointer)([]byte, error){}
   154  
   155  func Marshal(v interface{}) ([]byte, error) {
   156      iface := (*emptyInterface)(unsafe.Pointer(&v)
   157      typeptr := uintptr(iface.typ)
   158      if enc, exists := typeToEncoder[typeptr]; exists {
   159          return enc(iface.ptr)
   160      }
   161      ...
   162  }
   163  ```
   164  
   165  ※ In reality, `typeToEncoder` can be referenced by multiple goroutines, so exclusive control is required.
   166  
   167  ## Unique speed-up technique
   168  
   169  ## Encoder
   170  
   171  ### Do not escape arguments of `Marshal`
   172  
   173  `json.Marshal` and `json.Unmarshal` receive `interface{}` value and they perform type determination dynamically to process. In normal case, you need to use the `reflect` library to determine the type dynamically, but since `reflect.Type` is defined as `interface`, when you call the method of `reflect.Type`, The reflect's argument is escaped.
   174  
   175  Therefore, the arguments for `Marshal` and `Unmarshal` are always escape to the heap. However, `go-json` can use the feature of `reflect.Type` while avoiding escaping.
   176  
   177  `reflect.Type` is defined as `interface`, but in reality `reflect.Type` is implemented only by the structure `rtype` defined in the `reflect` package. For this reason, to date `reflect.Type` is the same as `*reflect.rtype`.
   178  
   179  Therefore, by directly handling `*reflect.rtype`, which is an implementation of `reflect.Type`, it is possible to avoid escaping because it changes from `interface` to using `struct`.
   180  
   181  The technique for working with `*reflect.rtype` directly from `go-json` is implemented at [rtype.go](https://github.com/night-codes/go-json/blob/master/internal/runtime/rtype.go)
   182  
   183  Also, the same technique is cut out as a library ( https://github.com/goccy/go-reflect )
   184  
   185  Initially this feature was the default behavior of `go-json`. But after careful testing, I found that I passed a large value to `json.Marshal()` and if the argument could not be assigned to the stack, it could not be properly escaped to the heap (a bug in the Go compiler).
   186  
   187  Therefore, this feature will be provided as an **optional** until this issue is resolved.
   188  
   189  To use it, add `NoEscape` like `MarshalNoEscape()`
   190  
   191  ### Encoding using opcode sequence
   192  
   193  I explained that you can use `typeptr` to call a pre-built process from type information.
   194  
   195  In other libraries, this dedicated process is processed by making it an function calling like anonymous function, but function calls are inherently slow processes and should be avoided as much as possible.
   196  
   197  Therefore, `go-json` adopted the Instruction-based execution processing system, which is also used to implement virtual machines for programming language.
   198  
   199  If it is the first type to encode, create the opcode ( instruction ) sequence required for encoding. From the second time onward, use `typeptr` to get the cached pre-built opcode sequence and encode it based on it. An example of the opcode sequence is shown below.
   200  
   201  ```go
   202  json.Marshal(struct{
   203      X int `json:"x"`
   204      Y string `json:"y"`
   205  }{X: 1, Y: "hello"})
   206  ```
   207  
   208  When encoding a structure like the one above, create a sequence of opcodes like this:
   209  
   210  ```
   211  - opStructFieldHead ( `{` )
   212  - opStructFieldInt ( `"x": 1,` )
   213  - opStructFieldString ( `"y": "hello"` )
   214  - opStructEnd ( `}` )
   215  - opEnd
   216  ```
   217  
   218  ※ When processing each operation, write the letters on the right.
   219  
   220  In addition, each opcode is managed by the following structure ( Pseudo code ).
   221  
   222  ```go
   223  type opType int
   224  const (
   225      opStructFieldHead opType = iota
   226      opStructFieldInt
   227      opStructFieldStirng
   228      opStructEnd
   229      opEnd
   230  )
   231  type opcode struct {
   232      op opType
   233      key []byte
   234      next *opcode
   235  }
   236  ```
   237  
   238  The process of encoding using the opcode sequence is roughly implemented as follows.
   239  
   240  ```go
   241  func encode(code *opcode, b []byte, p unsafe.Pointer) ([]byte, error) {
   242      for {
   243          switch code.op {
   244          case opStructFieldHead:
   245              b = append(b, '{')
   246              code = code.next
   247          case opStructFieldInt:
   248              b = append(b, code.key...)
   249              b = appendInt((*int)(unsafe.Pointer(uintptr(p)+code.offset)))
   250              code = code.next
   251          case opStructFieldString:
   252              b = append(b, code.key...)
   253              b = appendString((*string)(unsafe.Pointer(uintptr(p)+code.offset)))
   254              code = code.next
   255          case opStructEnd:
   256              b = append(b, '}')
   257              code = code.next
   258          case opEnd:
   259              goto END
   260          }
   261      }
   262  END:
   263      return b, nil
   264  }
   265  ```
   266  
   267  In this way, the huge `switch-case` is used to encode by manipulating the linked list opcodes to avoid unnecessary function calls.
   268  
   269  ### Opcode sequence optimization
   270  
   271  One of the advantages of encoding using the opcode sequence is the ease of optimization. The opcode sequence mentioned above is actually converted into the following optimized operations and used.
   272  
   273  ```
   274  - opStructFieldHeadInt ( `{"x": 1,` )
   275  - opStructEndString ( `"y": "hello"}` )
   276  - opEnd
   277  ```
   278  
   279  It has been reduced from 5 opcodes to 3 opcodes ! Reducing the number of opcodees means reducing the number of branches with `switch-case`. In other words, the closer the number of operations is to 1, the faster the processing can be performed.
   280  
   281  In `go-json`, optimization to reduce the number of opcodes itself like the above and it speeds up by preparing opcodes with optimized paths.
   282  
   283  ### Change recursive call from CALL to JMP
   284  
   285  Recursive processing is required during encoding if the type is defined recursively as follows:
   286  
   287  ```go
   288  type T struct {
   289      X int
   290      U *U
   291  }
   292  
   293  type U struct {
   294      T *T
   295  }
   296  
   297  b, err := json.Marshal(&T{
   298      X: 1,
   299      U: &U{
   300          T: &T{
   301              X: 2,
   302          },
   303      },
   304  })
   305  fmt.Println(string(b)) // {"X":1,"U":{"T":{"X":2,"U":null}}}
   306  ```
   307  
   308  In `go-json`, recursive processing is processed by the operation type of `opStructFieldRecursive`.
   309  
   310  In this operation, after acquiring the opcode sequence used for recursive processing, the function is **not** called recursively as it is, but the necessary values ​​are saved by itself and implemented by moving to the next operation.
   311  
   312  The technique of implementing recursive processing with the `JMP` operation while avoiding the `CALL` operation is a famous technique for implementing a high-speed virtual machine.
   313  
   314  For more details, please refer to [the article](https://engineering.mercari.com/blog/entry/1599563768-081104c850) ( but Japanese only ).
   315  
   316  ### Dispatch by typeptr from map to slice
   317  
   318  When retrieving the data cached from the type information by `typeptr`, we usually use map. Map requires exclusive control, so use `sync.Map` for a naive implementation.
   319  
   320  However, this is slow, so it's a good idea to use the `atomic` package for exclusive control as implemented by `segmentio/encoding/json` ( https://github.com/segmentio/encoding/blob/master/json/codec.go#L41-L55 ).
   321  
   322  This implementation slows down the set instead of speeding up the get, but it works well because of the nature of the library, it encodes much more for the same type.
   323  
   324  However, as a result of profiling, I noticed that `runtime.mapaccess2` accounts for a significant percentage of the execution time. So I thought if I could change the lookup from map to slice.
   325  
   326  There is an API named `typelinks` defined in the `runtime` package that the `reflect` package uses internally. This allows you to get all the type information defined in the binary at runtime.
   327  
   328  The fact that all type information can be acquired means that by constructing slices in advance with the acquired total number of type information, it is possible to look up with the value of `typeptr` without worrying about out-of-range access.
   329  
   330  However, if there is too much type information, it will use a lot of memory, so by default we will only use this optimization if the slice size fits within **2Mib** .
   331  
   332  If this approach is not available, it will fall back to the `atomic` based process described above.
   333  
   334  If you want to know more, please refer to the implementation [here](https://github.com/night-codes/go-json/blob/master/internal/runtime/type.go#L36-L100)
   335  
   336  ## Decoder
   337  
   338  ### Dispatch by typeptr from map to slice
   339  
   340  Like the encoder, the decoder also uses typeptr to call the dedicated process.
   341  
   342  ### Faster termination character inspection using NUL character
   343  
   344  In order to decode, you have to traverse the input buffer character by position. At that time, if you check whether the buffer has reached the end, it will be very slow.
   345  
   346  `buf` : `[]byte` type variable. holds the string passed to the decoder `cursor` : `int64` type variable. holds the current read position
   347  
   348  ```go
   349  buflen := len(buf)
   350  for ; cursor < buflen; cursor++ { // compare cursor and buflen at all times, it is so slow.
   351      switch buf[cursor] {
   352      case ' ', '\n', '\r', '\t':
   353      }
   354  }
   355  ```
   356  
   357  Therefore, by adding the `NUL` (`\000`) character to the end of the read buffer as shown below, it is possible to check the termination character at the same time as other characters.
   358  
   359  ```go
   360  for {
   361      switch buf[cursor] {
   362      case ' ', '\n', '\r', '\t':
   363      case '\000':
   364          return nil
   365      }
   366      cursor++
   367  }
   368  ```
   369  
   370  ### Use Boundary Check Elimination
   371  
   372  Due to the `NUL` character optimization, the Go compiler does a boundary check every time, even though `buf[cursor]` does not cause out-of-range access.
   373  
   374  Therefore, `go-json` eliminates boundary check by fetching characters for hotspot by pointer operation. For example, the following code.
   375  
   376  ```go
   377  func char(ptr unsafe.Pointer, offset int64) byte {
   378  	return *(*byte)(unsafe.Pointer(uintptr(ptr) + uintptr(offset)))
   379  }
   380  
   381  p := (*sliceHeader)(&unsafe.Pointer(buf)).data
   382  for {
   383      switch char(p, cursor) {
   384      case ' ', '\n', '\r', '\t':
   385      case '\000':
   386          return nil
   387      }
   388      cursor++
   389  }
   390  ```
   391  
   392  ### Checking the existence of fields of struct using Bitmaps
   393  
   394  I found by the profiling result, in the struct decode, lookup process for field was taking a long time.
   395  
   396  For example, consider decoding a string like `{"a":1,"b":2,"c":3}` into the following structure:
   397  
   398  ```go
   399  type T struct {
   400      A int `json:"a"`
   401      B int `json:"b"`
   402      C int `json:"c"`
   403  }
   404  ```
   405  
   406  At this time, it was found that it takes a lot of time to acquire the decoding process corresponding to the field from the field name as shown below during the decoding process.
   407  
   408  ```go
   409  fieldName := decodeKey(buf, cursor) // "a" or "b" or "c"
   410  decoder, exists := fieldToDecoderMap[fieldName] // so slow
   411  if exists {
   412      decoder(buf, cursor)
   413  } else {
   414      skipValue(buf, cursor)
   415  }
   416  ```
   417  
   418  To improve this process, `json-iterator/go` is optimized so that it can be branched by switch-case when the number of fields in the structure is 10 or less (switch-case is faster than map). However, there is a risk of hash collision because the value hashed by the FNV algorithm is used for conditional branching. Also, `gojay` processes this part at high speed by letting the library user yourself write `switch-case`.
   419  
   420  `go-json` considers and implements a new approach that is different from these. I call this **bitmap field optimization**.
   421  
   422  The range of values ​​per character can be represented by `[256]byte`. Also, if the number of fields in the structure is 8 or less, `int8` type can represent the state of each field. In other words, it has the following structure.
   423  
   424  -   Base ( 8bit ): `00000000`
   425  -   Key "a": `00000001` ( assign key "a" to the first bit )
   426  -   Key "b": `00000010` ( assign key "b" to the second bit )
   427  -   Key "c": `00000100` ( assign key "c" to the third bit )
   428  
   429  Bitmap structure is the following
   430  
   431  ```
   432          | key index(0) |
   433  ------------------------
   434   0      | 00000000     |
   435   1      | 00000000     |
   436  ~~      |              |
   437  97 (a)  | 00000001     |
   438  98 (b)  | 00000010     |
   439  99 (c)  | 00000100     |
   440  ~~      |              |
   441  255     | 00000000     |
   442  ```
   443  
   444  You can think of this as a Bitmap with a height of `256` and a width of the maximum string length in the field name. In other words, it can be represented by the following type .
   445  
   446  ```go
   447  [maxFieldKeyLength][256]int8
   448  ```
   449  
   450  When decoding a field character, check whether the corresponding character exists by referring to the pre-built bitmap like the following.
   451  
   452  ```go
   453  var curBit int8 = math.MaxInt8 // 11111111
   454  
   455  c := char(buf, cursor)
   456  bit := bitmap[keyIdx][c]
   457  curBit &= bit
   458  if curBit == 0 {
   459      // not found field
   460  }
   461  ```
   462  
   463  If `curBit` is not `0` until the end of the field string, then the string is You may have hit one of the fields. But the possibility is that if the decoded string is shorter than the field string, you will get a false hit.
   464  
   465  -   input: `{"a":1}`
   466  
   467  ```go
   468  type T struct {
   469      X int `json:"abc"`
   470  }
   471  ```
   472  
   473  ※ Since `a` is shorter than `abc`, it can decode to the end of the field character without `curBit` being 0.
   474  
   475  Rest assured. In this case, it doesn't matter because you can tell if you hit by comparing the string length of `a` with the string length of `abc`.
   476  
   477  Finally, calculate the position of the bit where `1` is set and get the corresponding value, and you're done.
   478  
   479  Using this technique, field lookups are possible with only bitwise operations and access to slices.
   480  
   481  `go-json` uses a similar technique for fields with 9 or more and 16 or less fields. At this time, Bitmap is constructed as `[maxKeyLen][256]int16` type.
   482  
   483  Currently, this optimization is not performed when the maximum length of the field name is long (specifically, 64 bytes or more) in addition to the limitation of the number of fields from the viewpoint of saving memory usage.
   484  
   485  ### Others
   486  
   487  I have done a lot of other optimizations. I will find time to write about them. If you have any questions about what's written here or other optimizations, please visit the `#go-json` channel on `gophers.slack.com` .
   488  
   489  ## Reference
   490  
   491  Regarding the story of go-json, there are the following articles in Japanese only.
   492  
   493  -   https://speakerdeck.com/goccy/zui-su-falsejsonraiburariwoqiu-mete
   494  -   https://engineering.mercari.com/blog/entry/1599563768-081104c850/
   495  
   496  # Looking for Sponsors
   497  
   498  I'm looking for sponsors this library. This library is being developed as a personal project in my spare time. If you want a quick response or problem resolution when using this library in your project, please register as a [sponsor](https://github.com/sponsors/goccy). I will cooperate as much as possible. Of course, this library is developed as an MIT license, so you can use it freely for free.
   499  
   500  # License
   501  
   502  MIT