github.com/hamba/avro/v2@v2.22.1-0.20240518180522-aff3955acf7d/README.md (about) 1 <picture> 2 <source media="(prefers-color-scheme: dark)" srcset="http://svg.wiersma.co.za/hamba/project?title=avro&tag=A%20fast%20Go%20avro%20codec&mode=dark"> 3 <source media="(prefers-color-scheme: light)" srcset="http://svg.wiersma.co.za/hamba/project?title=avro&tag=A%20fast%20Go%20avro%20codec"> 4 <img alt="Logo" src="http://svg.wiersma.co.za/hamba/project?title=avro&tag=A%20fast%20Go%20avro%20codec"> 5 </picture> 6 7 [](https://goreportcard.com/report/github.com/hamba/avro/v2) 8 [](https://github.com/hamba/avro/actions) 9 [](https://coveralls.io/github/hamba/avro?branch=main) 10 [](https://pkg.go.dev/github.com/hamba/avro/v2) 11 [](https://github.com/hamba/avro/releases) 12 [](https://raw.githubusercontent.com/hamba/avro/master/LICENSE) 13 14 A fast Go avro codec 15 16 ## Overview 17 18 Install with: 19 20 ```shell 21 go get github.com/hamba/avro/v2 22 ``` 23 24 **Note:** This project has renamed the default branch from `master` to `main`. You will need to update your local environment. 25 26 ## Usage 27 28 ```go 29 type SimpleRecord struct { 30 A int64 `avro:"a"` 31 B string `avro:"b"` 32 } 33 34 schema, err := avro.Parse(`{ 35 "type": "record", 36 "name": "simple", 37 "namespace": "org.hamba.avro", 38 "fields" : [ 39 {"name": "a", "type": "long"}, 40 {"name": "b", "type": "string"} 41 ] 42 }`) 43 if err != nil { 44 log.Fatal(err) 45 } 46 47 in := SimpleRecord{A: 27, B: "foo"} 48 49 data, err := avro.Marshal(schema, in) 50 if err != nil { 51 log.Fatal(err) 52 } 53 54 fmt.Println(data) 55 // Outputs: [54 6 102 111 111] 56 57 out := SimpleRecord{} 58 err = avro.Unmarshal(schema, data, &out) 59 if err != nil { 60 log.Fatal(err) 61 } 62 63 fmt.Println(out) 64 // Outputs: {27 foo} 65 ``` 66 67 More examples in the [godoc](https://godoc.org/github.com/hamba/avro/v2). 68 69 #### Types Conversions 70 71 | Avro | Go Struct | Go Interface | 72 |-------------------------------|--------------------------------------------------------|--------------------------| 73 | `null` | `nil` | `nil` | 74 | `boolean` | `bool` | `bool` | 75 | `bytes` | `[]byte` | `[]byte` | 76 | `float` | `float32` | `float32` | 77 | `double` | `float64` | `float64` | 78 | `long` | `int64`, `uint32`\* | `int64`, `uint32` | 79 | `int` | `int`, `int32`, `int16`, `int8`, `uint8`\*, `uint16`\* | `int`, `uint8`, `uint16` | 80 | `fixed` | `uint64` | `uint64` | 81 | `string` | `string` | `string` | 82 | `array` | `[]T` | `[]any` | 83 | `enum` | `string` | `string` | 84 | `fixed` | `[n]byte` | `[n]byte` | 85 | `map` | `map[string]T{}` | `map[string]any` | 86 | `record` | `struct` | `map[string]any` | 87 | `union` | *see below* | *see below* | 88 | `int.date` | `time.Time` | `time.Time` | 89 | `int.time-millis` | `time.Duration` | `time.Duration` | 90 | `long.time-micros` | `time.Duration` | `time.Duration` | 91 | `long.timestamp-millis` | `time.Time` | `time.Time` | 92 | `long.timestamp-micros` | `time.Time` | `time.Time` | 93 | `long.local-timestamp-millis` | `time.Time` | `time.Time` | 94 | `long.local-timestamp-micros` | `time.Time` | `time.Time` | 95 | `bytes.decimal` | `*big.Rat` | `*big.Rat` | 96 | `fixed.decimal` | `*big.Rat` | `*big.Rat` | 97 | `string.uuid` | `string` | `string` | 98 99 \* Please note that when the Go type is an unsigned integer care must be taken to ensure that information is not lost 100 when converting between the Avro type and Go type. For example, storing a *negative* number in Avro of `int = -100` 101 would be interpreted as `uint16 = 65,436` in Go. Another example would be storing numbers in Avro `int = 256` that 102 are larger than the Go type `uint8 = 0`. 103 104 ##### Unions 105 106 The following union types are accepted: `map[string]any`, `*T` and `any`. 107 108 * **map[string]any:** If the union value is `nil`, a `nil` map will be en/decoded. 109 When a non-`nil` union value is encountered, a single key is en/decoded. The key is the avro 110 type name, or scheam full name in the case of a named schema (enum, fixed or record). 111 * ***T:** This is allowed in a "nullable" union. A nullable union is defined as a two schema union, 112 with one of the types being `null` (ie. `["null", "string"]` or `["string", "null"]`), in this case 113 a `*T` is allowed, with `T` matching the conversion table above. In the case of a slice, the slice can be used 114 directly. 115 * **any:** An `interface` can be provided and the type or name resolved. Primitive types 116 are pre-registered, but named types, maps and slices will need to be registered with the `Register` function. 117 In the case of arrays and maps the enclosed schema type or name is postfix to the type with a `:` separator, 118 e.g `"map:string"`. Behavior when a type cannot be resolved will depend on your chosen configuation options: 119 * !Config.UnionResolutionError && !Config.PartialUnionTypeResolution: the map type above is used 120 * Config.UnionResolutionError && !Config.PartialUnionTypeResolution: an error is returned 121 * !Config.UnionResolutionError && Config.PartialUnionTypeResolution: any registered type will get resolved while any unregistered type will fallback to the map type above. 122 * Config.UnionResolutionError && !Config.PartialUnionTypeResolution: any registered type will get resolved while any unregistered type will return an error. 123 124 ##### TextMarshaler and TextUnmarshaler 125 126 The interfaces `TextMarshaler` and `TextUnmarshaler` are supported for a `string` schema type. The object will 127 be tested first for implementation of these interfaces, in the case of a `string` schema, before trying regular 128 encoding and decoding. 129 130 Enums may also implement `TextMarshaler` and `TextUnmarshaler`, and must resolve to valid symbols in the given enum schema. 131 132 ##### Identical Underlying Types 133 134 One type can be [ConvertibleTo](https://go.dev/ref/spec#Conversions) another type if they have identical underlying types. 135 A non-native type is allowed be used if it can be convertible to *time.Time*, *big.Rat* or *avro.LogicalDuration* for the particular of *LogicalTypes*. 136 137 Ex.: `type Timestamp time.Time` 138 139 ##### Untrusted Input With Bytes and Strings 140 141 For security reasons, the configuration `Config.MaxByteSliceSize` restricts the maximum size of `bytes` and `string` types created 142 by the `Reader`. The default maximum size is `1MiB` and is configurable. This is required to stop untrusted input from consuming all memory and 143 crashing the application. Should this not be need, setting a negative number will disable the behaviour. 144 145 ### Recursive Structs 146 147 At this moment recursive structs are not supported. It is planned for the future. 148 149 ## Benchmark 150 151 Benchmark source code can be found at: [https://github.com/nrwiersma/avro-benchmarks](https://github.com/nrwiersma/avro-benchmarks) 152 153 ``` 154 BenchmarkGoAvroDecode-8 788455 1505 ns/op 418 B/op 27 allocs/op 155 BenchmarkGoAvroEncode-8 624343 1908 ns/op 806 B/op 63 allocs/op 156 BenchmarkGoGenAvroDecode-8 1360375 876.4 ns/op 320 B/op 11 allocs/op 157 BenchmarkGoGenAvroEncode-8 2801583 425.9 ns/op 240 B/op 3 allocs/op 158 BenchmarkHambaDecode-8 5046832 238.7 ns/op 47 B/op 0 allocs/op 159 BenchmarkHambaEncode-8 6017635 196.2 ns/op 112 B/op 1 allocs/op 160 BenchmarkLinkedinDecode-8 1000000 1003 ns/op 1688 B/op 35 allocs/op 161 BenchmarkLinkedinEncode-8 3170553 381.5 ns/op 248 B/op 5 allocs/op 162 ``` 163 164 Always benchmark with your own workload. The result depends heavily on the data input. 165 166 ## Go structs generation 167 168 Go structs can be generated for you from the schema. The types generated follow the same logic in [types conversions](#types-conversions) 169 170 Install the struct generator with: 171 172 ```shell 173 go install github.com/hamba/avro/v2/cmd/avrogen@<version> 174 ``` 175 176 Example usage assuming there's a valid schema in `in.avsc`: 177 178 ```shell 179 avrogen -pkg avro -o bla.go -tags json:snake,yaml:upper-camel in.avsc 180 ``` 181 182 **Tip:** Omit `-o FILE` to dump the generated Go structs to stdout instead of a file. 183 184 Check the options and usage with `-h`: 185 186 ```shell 187 avrogen -h 188 ``` 189 190 Or use it as a lib in internal commands, it's the `gen` package 191 192 ## Avro schema validation 193 194 A small Avro schema validation command-line utility is also available. This simple tool leverages the 195 schema parsing functionality of the library, showing validation errors or optionally dumping parsed 196 schemas to the console. It can be used in CI/CD pipelines to validate schema changes in a repository. 197 198 Install the Avro schema validator with: 199 200 ```shell 201 go install github.com/hamba/avro/v2/cmd/avrosv@<version> 202 ``` 203 204 Example usage assuming there's a valid schema in `in.avsc` (exit status code is `0`): 205 206 ```shell 207 avrosv in.avsc 208 ``` 209 210 An invalid schema will result in a diagnostic output and a non-zero exit status code: 211 212 ```shell 213 avrosv bad-default-schema.avsc; echo $? 214 Error: avro: invalid default for field someString. <nil> not a string 215 2 216 ``` 217 218 Schemas referencing other schemas can also be validated by providing all of them (schemas are parsed in order): 219 220 ```shell 221 avrosv base-schema.avsc schema-withref.avsc 222 ``` 223 224 Check the options and usage with `-h`: 225 226 ```shell 227 avrosv -h 228 ``` 229 230 ## Go Version Support 231 232 This library supports the last two versions of Go. While the minimum Go version is 233 not guarantee to increase along side Go, it may jump from time to time to support 234 additional features. This will be not be considered a breaking change.