gitee.com/quant1x/pkg@v0.2.8/fastjson/README.md (about) 1 [](https://travis-ci.org/valyala/fastjson) 2 [](http://godoc.org/github.com/valyala/fastjson) 3 [](https://goreportcard.com/report/github.com/valyala/fastjson) 4 [](https://codecov.io/gh/valyala/fastjson) 5 6 # fastjson - fast JSON parser and validator for Go 7 8 9 ## Features 10 11 * Fast. As usual, up to 15x faster than the standard [encoding/json](https://golang.org/pkg/encoding/json/). 12 See [benchmarks](#benchmarks). 13 * Parses arbitrary JSON without schema, reflection, struct magic and code generation 14 contrary to [easyjson](https://github.com/mailru/easyjson). 15 * Provides simple [API](http://godoc.org/github.com/valyala/fastjson). 16 * Outperforms [jsonparser](https://github.com/buger/jsonparser) and [gjson](https://github.com/tidwall/gjson) 17 when accessing multiple unrelated fields, since `fastjson` parses the input JSON only once. 18 * Validates the parsed JSON unlike [jsonparser](https://github.com/buger/jsonparser) 19 and [gjson](https://github.com/tidwall/gjson). 20 * May quickly extract a part of the original JSON with `Value.Get(...).MarshalTo` and modify it 21 with [Del](https://godoc.org/github.com/valyala/fastjson#Value.Del) 22 and [Set](https://godoc.org/github.com/valyala/fastjson#Value.Set) functions. 23 * May parse array containing values with distinct types (aka non-homogenous types). 24 For instance, `fastjson` easily parses the following JSON array `[123, "foo", [456], {"k": "v"}, null]`. 25 * `fastjson` preserves the original order of object items when calling 26 [Object.Visit](https://godoc.org/github.com/valyala/fastjson#Object.Visit). 27 28 29 ## Known limitations 30 31 * Requies extra care to work with - references to certain objects recursively 32 returned by [Parser](https://godoc.org/github.com/valyala/fastjson#Parser) 33 must be released before the next call to [Parse](https://godoc.org/github.com/valyala/fastjson#Parser.Parse). 34 Otherwise the program may work improperly. The same applies to objects returned 35 by [Arena](https://godoc.org/github.com/valyala/fastjson#Arena). 36 Adhere recommendations from [docs](https://godoc.org/github.com/valyala/fastjson). 37 * Cannot parse JSON from `io.Reader`. There is [Scanner](https://godoc.org/github.com/valyala/fastjson#Scanner) 38 for parsing stream of JSON values from a string. 39 40 41 ## Usage 42 43 One-liner accessing a single field: 44 ```go 45 s := []byte(`{"foo": [123, "bar"]}`) 46 fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0")) 47 48 // Output: 49 // foo.0=123 50 ``` 51 52 Accessing multiple fields with error handling: 53 ```go 54 var p fastjson.Parser 55 v, err := p.Parse(`{ 56 "str": "bar", 57 "int": 123, 58 "float": 1.23, 59 "bool": true, 60 "arr": [1, "foo", {}] 61 }`) 62 if err != nil { 63 log.Fatal(err) 64 } 65 fmt.Printf("foo=%s\n", v.GetStringBytes("str")) 66 fmt.Printf("int=%d\n", v.GetInt("int")) 67 fmt.Printf("float=%f\n", v.GetFloat64("float")) 68 fmt.Printf("bool=%v\n", v.GetBool("bool")) 69 fmt.Printf("arr.1=%s\n", v.GetStringBytes("arr", "1")) 70 71 // Output: 72 // foo=bar 73 // int=123 74 // float=1.230000 75 // bool=true 76 // arr.1=foo 77 ``` 78 79 See also [examples](https://godoc.org/github.com/valyala/fastjson#pkg-examples). 80 81 82 ## Security 83 84 * `fastjson` shouldn't crash or panic when parsing input strings specially crafted 85 by an attacker. It must return error on invalid input JSON. 86 * `fastjson` requires up to `sizeof(Value) * len(inputJSON)` bytes of memory 87 for parsing `inputJSON` string. Limit the maximum size of the `inputJSON` 88 before parsing it in order to limit the maximum memory usage. 89 90 91 ## Performance optimization tips 92 93 * Re-use [Parser](https://godoc.org/github.com/valyala/fastjson#Parser) 94 and [Scanner](https://godoc.org/github.com/valyala/fastjson#Scanner) 95 for parsing many JSONs. This reduces memory allocations overhead. 96 [ParserPool](https://godoc.org/github.com/valyala/fastjson#ParserPool) may be useful in this case. 97 * Prefer calling `Value.Get*` on the value returned from [Parser](https://godoc.org/github.com/valyala/fastjson#Parser) 98 instead of calling `Get*` one-liners when multiple fields 99 must be obtained from JSON, since each `Get*` one-liner re-parses 100 the input JSON again. 101 * Prefer calling once [Value.Get](https://godoc.org/github.com/valyala/fastjson#Value.Get) 102 for common prefix paths and then calling `Value.Get*` on the returned value 103 for distinct suffix paths. 104 * Prefer iterating over array returned from [Value.GetArray](https://godoc.org/github.com/valyala/fastjson#Object.Visit) 105 with a range loop instead of calling `Value.Get*` for each array item. 106 107 ## Fuzzing 108 Install [go-fuzz](https://github.com/dvyukov/go-fuzz) & optionally the go-fuzz-corpus. 109 110 ```bash 111 go get -u github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build 112 ``` 113 114 Build using `go-fuzz-build` and run `go-fuzz` with an optional corpus. 115 116 ```bash 117 mkdir -p workdir/corpus 118 cp $GOPATH/src/github.com/dvyukov/go-fuzz-corpus/json/corpus/* workdir/corpus 119 go-fuzz-build github.com/valyala/fastjson 120 go-fuzz -bin=fastjson-fuzz.zip -workdir=workdir 121 ``` 122 123 ## Benchmarks 124 125 Go 1.12 has been used for benchmarking. 126 127 Legend: 128 129 * `small` - parse [small.json](testdata/small.json) (190 bytes). 130 * `medium` - parse [medium.json](testdata/medium.json) (2.3KB). 131 * `large` - parse [large.json](testdata/large.json) (28KB). 132 * `canada` - parse [canada.json](testdata/canada.json) (2.2MB). 133 * `citm` - parse [citm_catalog.json](testdata/citm_catalog.json) (1.7MB). 134 * `twitter` - parse [twitter.json](testdata/twitter.json) (617KB). 135 136 * `stdjson-map` - parse into a `map[string]interface{}` using `encoding/json`. 137 * `stdjson-struct` - parse into a struct containing 138 a subset of fields of the parsed JSON, using `encoding/json`. 139 * `stdjson-empty-struct` - parse into an empty struct using `encoding/json`. 140 This is the fastest possible solution for `encoding/json`, may be used 141 for json validation. See also benchmark results for json validation. 142 * `fastjson` - parse using `fastjson` without fields access. 143 * `fastjson-get` - parse using `fastjson` with fields access similar to `stdjson-struct`. 144 145 ``` 146 $ GOMAXPROCS=1 go test github.com/valyala/fastjson -bench='Parse$' 147 goos: linux 148 goarch: amd64 149 pkg: github.com/valyala/fastjson 150 BenchmarkParse/small/stdjson-map 200000 7305 ns/op 26.01 MB/s 960 B/op 51 allocs/op 151 BenchmarkParse/small/stdjson-struct 500000 3431 ns/op 55.37 MB/s 224 B/op 4 allocs/op 152 BenchmarkParse/small/stdjson-empty-struct 500000 2273 ns/op 83.58 MB/s 168 B/op 2 allocs/op 153 BenchmarkParse/small/fastjson 5000000 347 ns/op 547.53 MB/s 0 B/op 0 allocs/op 154 BenchmarkParse/small/fastjson-get 2000000 620 ns/op 306.39 MB/s 0 B/op 0 allocs/op 155 BenchmarkParse/medium/stdjson-map 30000 40672 ns/op 57.26 MB/s 10196 B/op 208 allocs/op 156 BenchmarkParse/medium/stdjson-struct 30000 47792 ns/op 48.73 MB/s 9174 B/op 258 allocs/op 157 BenchmarkParse/medium/stdjson-empty-struct 100000 22096 ns/op 105.40 MB/s 280 B/op 5 allocs/op 158 BenchmarkParse/medium/fastjson 500000 3025 ns/op 769.90 MB/s 0 B/op 0 allocs/op 159 BenchmarkParse/medium/fastjson-get 500000 3211 ns/op 725.20 MB/s 0 B/op 0 allocs/op 160 BenchmarkParse/large/stdjson-map 2000 614079 ns/op 45.79 MB/s 210734 B/op 2785 allocs/op 161 BenchmarkParse/large/stdjson-struct 5000 298554 ns/op 94.18 MB/s 15616 B/op 353 allocs/op 162 BenchmarkParse/large/stdjson-empty-struct 5000 268577 ns/op 104.69 MB/s 280 B/op 5 allocs/op 163 BenchmarkParse/large/fastjson 50000 35210 ns/op 798.56 MB/s 5 B/op 0 allocs/op 164 BenchmarkParse/large/fastjson-get 50000 35171 ns/op 799.46 MB/s 5 B/op 0 allocs/op 165 BenchmarkParse/canada/stdjson-map 20 68147307 ns/op 33.03 MB/s 12260502 B/op 392539 allocs/op 166 BenchmarkParse/canada/stdjson-struct 20 68044518 ns/op 33.08 MB/s 12260123 B/op 392534 allocs/op 167 BenchmarkParse/canada/stdjson-empty-struct 100 17709250 ns/op 127.11 MB/s 280 B/op 5 allocs/op 168 BenchmarkParse/canada/fastjson 300 4182404 ns/op 538.22 MB/s 254902 B/op 381 allocs/op 169 BenchmarkParse/canada/fastjson-get 300 4274744 ns/op 526.60 MB/s 254902 B/op 381 allocs/op 170 BenchmarkParse/citm/stdjson-map 50 27772612 ns/op 62.19 MB/s 5214163 B/op 95402 allocs/op 171 BenchmarkParse/citm/stdjson-struct 100 14936191 ns/op 115.64 MB/s 1989 B/op 75 allocs/op 172 BenchmarkParse/citm/stdjson-empty-struct 100 14946034 ns/op 115.56 MB/s 280 B/op 5 allocs/op 173 BenchmarkParse/citm/fastjson 1000 1879714 ns/op 918.87 MB/s 17628 B/op 30 allocs/op 174 BenchmarkParse/citm/fastjson-get 1000 1881598 ns/op 917.94 MB/s 17628 B/op 30 allocs/op 175 BenchmarkParse/twitter/stdjson-map 100 11289146 ns/op 55.94 MB/s 2187878 B/op 31266 allocs/op 176 BenchmarkParse/twitter/stdjson-struct 300 5779442 ns/op 109.27 MB/s 408 B/op 6 allocs/op 177 BenchmarkParse/twitter/stdjson-empty-struct 300 5738504 ns/op 110.05 MB/s 408 B/op 6 allocs/op 178 BenchmarkParse/twitter/fastjson 2000 774042 ns/op 815.86 MB/s 2541 B/op 2 allocs/op 179 BenchmarkParse/twitter/fastjson-get 2000 777833 ns/op 811.89 MB/s 2541 B/op 2 allocs/op 180 ``` 181 182 Benchmark results for json validation: 183 184 ``` 185 $ GOMAXPROCS=1 go test github.com/valyala/fastjson -bench='Validate$' 186 goos: linux 187 goarch: amd64 188 pkg: github.com/valyala/fastjson 189 BenchmarkValidate/small/stdjson 2000000 955 ns/op 198.83 MB/s 72 B/op 2 allocs/op 190 BenchmarkValidate/small/fastjson 5000000 384 ns/op 493.60 MB/s 0 B/op 0 allocs/op 191 BenchmarkValidate/medium/stdjson 200000 10799 ns/op 215.66 MB/s 184 B/op 5 allocs/op 192 BenchmarkValidate/medium/fastjson 300000 3809 ns/op 611.30 MB/s 0 B/op 0 allocs/op 193 BenchmarkValidate/large/stdjson 10000 133064 ns/op 211.31 MB/s 184 B/op 5 allocs/op 194 BenchmarkValidate/large/fastjson 30000 45268 ns/op 621.14 MB/s 0 B/op 0 allocs/op 195 BenchmarkValidate/canada/stdjson 200 8470904 ns/op 265.74 MB/s 184 B/op 5 allocs/op 196 BenchmarkValidate/canada/fastjson 500 2973377 ns/op 757.07 MB/s 0 B/op 0 allocs/op 197 BenchmarkValidate/citm/stdjson 200 7273172 ns/op 237.48 MB/s 184 B/op 5 allocs/op 198 BenchmarkValidate/citm/fastjson 1000 1684430 ns/op 1025.39 MB/s 0 B/op 0 allocs/op 199 BenchmarkValidate/twitter/stdjson 500 2849439 ns/op 221.63 MB/s 312 B/op 6 allocs/op 200 BenchmarkValidate/twitter/fastjson 2000 1036796 ns/op 609.10 MB/s 0 B/op 0 allocs/op 201 ``` 202 203 ## FAQ 204 205 * Q: _There are a ton of other high-perf packages for JSON parsing in Go. Why creating yet another package?_ 206 A: Because other packages require either rigid JSON schema via struct magic 207 and code generation or perform poorly when multiple unrelated fields 208 must be obtained from the parsed JSON. 209 Additionally, `fastjson` provides nicer [API](http://godoc.org/github.com/valyala/fastjson). 210 211 * Q: _What is the main purpose for `fastjson`?_ 212 A: High-perf JSON parsing 213 for [RTB](https://www.iab.com/wp-content/uploads/2015/05/OpenRTB_API_Specification_Version_2_3_1.pdf) 214 and other [JSON-RPC](https://en.wikipedia.org/wiki/JSON-RPC) services. 215 216 * Q: _Why fastjson doesn't provide fast marshaling (serialization)?_ 217 A: Actually it provides some sort of marshaling - 218 see [Value.MarshalTo](https://godoc.org/github.com/valyala/fastjson#Value.MarshalTo). 219 But I'd recommend using [quicktemplate](https://github.com/valyala/quicktemplate#use-cases) 220 for high-performance JSON marshaling :) 221 222 * Q: _`fastjson` crashes my program!_ 223 A: There is high probability of improper use. 224 * Make sure you don't hold references to objects recursively returned by `Parser` / `Scanner` 225 beyond the next `Parser.Parse` / `Scanner.Next` call 226 if such restriction is mentioned in [docs](https://github.com/valyala/fastjson/issues/new). 227 * Make sure you don't access `fastjson` objects from concurrently running goroutines 228 if such restriction is mentioned in [docs](https://github.com/valyala/fastjson/issues/new). 229 * Build and run your program with [-race](https://golang.org/doc/articles/race_detector.html) flag. 230 Make sure the race detector detects zero races. 231 * If your program continue crashing after fixing issues mentioned 232 above, [file a bug](https://github.com/valyala/fastjson/issues/new).