github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/processors/parquet.md (about) 1 --- 2 title: parquet 3 type: processor 4 status: experimental 5 categories: ["Parsing"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/processor/parquet.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 :::caution EXPERIMENTAL 19 This component is experimental and therefore subject to change or removal outside of major version releases. 20 ::: 21 Converts batches of documents to or from [Parquet files](https://parquet.apache.org/documentation/latest/). 22 23 Introduced in version 3.62.0. 24 25 ```yaml 26 # Config fields, showing default values 27 label: "" 28 parquet: 29 operator: "" 30 compression: snappy 31 schema_file: "" 32 schema: "" 33 ``` 34 35 ### Troubleshooting 36 37 This processor is experimental and the error messages that it provides are often vague and unhelpful. An error message of the form `interface {} is nil, not <value type>` implies that a field of the given type was expected but not found in the processed message when writing parquet files. 38 39 Unfortunately the name of the field will sometimes be missing from the error, in which case it's worth double checking the schema you provided to make sure that there are no typos in the field names, and if that doesn't reveal the issue it can help to mark fields as OPTIONAL in the schema and gradually change them back to REQUIRED until the error returns. 40 41 ### Defining the Schema 42 43 The schema must be specified as a JSON string, containing an object that describes the fields expected at the root of each document. Each field can itself have more fields defined, allowing for nested structures: 44 45 ```json 46 { 47 "Tag": "name=root, repetitiontype=REQUIRED", 48 "Fields": [ 49 {"Tag": "name=name, inname=NameIn, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"}, 50 {"Tag": "name=age, inname=Age, type=INT32, repetitiontype=REQUIRED"}, 51 {"Tag": "name=id, inname=Id, type=INT64, repetitiontype=REQUIRED"}, 52 {"Tag": "name=weight, inname=Weight, type=FLOAT, repetitiontype=REQUIRED"}, 53 { 54 "Tag": "name=favPokemon, inname=FavPokemon, type=LIST, repetitiontype=OPTIONAL", 55 "Fields": [ 56 {"Tag": "name=name, inname=PokeName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"}, 57 {"Tag": "name=coolness, inname=Coolness, type=FLOAT, repetitiontype=REQUIRED"} 58 ] 59 } 60 ] 61 } 62 ``` 63 64 ## Fields 65 66 ### `operator` 67 68 Determines whether the processor converts messages into a parquet file or expands parquet files into messages. Converting into JSON allows subsequent processors and mappings to convert the data into any other format. 69 70 71 Type: `string` 72 73 | Option | Summary | 74 |---|---| 75 | `from_json` | Compress a batch of JSON documents into a file. | 76 | `to_json` | Expand a file into one or more JSON messages. | 77 78 79 ### `compression` 80 81 The type of compression to use when writing parquet files, this field is ignored when consuming parquet files. 82 83 84 Type: `string` 85 Default: `"snappy"` 86 Options: `uncompressed`, `snappy`, `gzip`, `lz4`, `zstd`. 87 88 ### `schema_file` 89 90 A file path containing a schema used to describe the parquet files being generated or consumed, the format of the schema is a JSON document detailing the tag and fields of documents. The schema can be found at: https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json. Either a `schema_file` or `schema` field must be specified. 91 92 93 Type: `string` 94 95 ```yaml 96 # Examples 97 98 schema_file: schemas/foo.json 99 ``` 100 101 ### `schema` 102 103 A schema used to describe the parquet files being generated or consumed, the format of the schema is a JSON document detailing the tag and fields of documents. The schema can be found at: https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json. Either a `schema_file` or `schema` field must be specified. 104 105 106 Type: `string` 107 108 ```yaml 109 # Examples 110 111 schema: |- 112 { 113 "Tag": "name=root, repetitiontype=REQUIRED", 114 "Fields": [ 115 {"Tag":"name=name,inname=NameIn,type=BYTE_ARRAY,convertedtype=UTF8, repetitiontype=REQUIRED"}, 116 {"Tag":"name=age,inname=Age,type=INT32,repetitiontype=REQUIRED"} 117 ] 118 } 119 ``` 120 121 ## Examples 122 123 <Tabs defaultValue="Batching Output Files" values={[ 124 { label: 'Batching Output Files', value: 'Batching Output Files', }, 125 ]}> 126 127 <TabItem value="Batching Output Files"> 128 129 Parquet is often used to write batches of documents to a file store. 130 131 ```yaml 132 output: 133 broker: 134 outputs: 135 - file: 136 path: ./stuff-${! uuid_v4() }.parquet 137 codec: all-bytes 138 batching: 139 count: 100 140 period: 30s 141 processors: 142 - parquet: 143 operator: from_json 144 schema: |- 145 { 146 "Tag": "name=root, repetitiontype=REQUIRED", 147 "Fields": [ 148 {"Tag":"name=name,inname=NameIn,type=BYTE_ARRAY,convertedtype=UTF8, repetitiontype=REQUIRED"}, 149 {"Tag":"name=age,inname=Age,type=INT32,repetitiontype=REQUIRED"} 150 ] 151 } 152 ``` 153 154 </TabItem> 155 </Tabs> 156 157