github.com/observiq/carbon@v0.9.11-0.20200820160507-1b872e368a5e/docs/types/timestamp.md (about) 1 ## `timestamp` parsing parameters 2 3 Parser operators can parse a timestamp and attach the resulting time value to a log entry. 4 5 | Field | Default | Description | 6 | --- | --- | --- | 7 | `parse_from` | required | A [field](/docs/types/field.md) that indicates the field to be parsed as JSON | 8 | `layout_type` | `strptime` | The type of timestamp. Valid values are `strptime`, `gotime`, and `epoch` | 9 | `layout` | required | The exact layout of the timestamp to be parsed | 10 | `preserve` | false | Preserve the unparsed value on the record | 11 12 13 ### How to specify timestamp parsing parameters 14 15 Most parser operators, such as [`regex_parser`](/docs/operators/regex_parser.md) support these fields inside of a `timestamp` block. 16 17 If a timestamp block is specified, the parser operator will perform the timestamp parsing _after_ performing its other parsing actions, but _before_ passing the entry to the specified output operator. 18 19 ```yaml 20 - type: regex_parser 21 regexp: '^Time=(?P<timestamp_field>\d{4}-\d{2}-\d{2}), Host=(?P<host>[^,]+)' 22 timestamp: 23 parse_from: timestamp_field 24 layout_type: strptime 25 layout: '%Y-%m-%d' 26 ``` 27 28 --- 29 30 As a special case, the [`time_parser`](/docs/operators/time_parser.md) operator supports these fields inline. This is because time parsing is the primary purpose of the operator. 31 ```yaml 32 - type: time_parser 33 parse_from: timestamp_field 34 layout_type: strptime 35 layout: '%Y-%m-%d' 36 ``` 37 38 ### Example Configurations 39 40 #### Parse a timestamp using a `strptime` layout 41 42 The default `layout_type` is `strptime`, which uses "directives" such as `%Y` (4-digit year) and `%H` (2-digit hour). A full list of supported directives is found [here](https://github.com/observiq/ctimefmt/blob/3e07deba22cf7a753f197ef33892023052f26614/ctimefmt.go#L63). 43 44 Configuration: 45 ```yaml 46 - type: time_parser 47 parse_from: timestamp_field 48 layout_type: strptime 49 layout: '%a %b %e %H:%M:%S %Z %Y' 50 ``` 51 52 <table> 53 <tr><td> Input entry </td> <td> Output entry </td></tr> 54 <tr> 55 <td> 56 57 ```json 58 { 59 "timestamp": "", 60 "record": { 61 "timestamp_field": "Jun 5 13:50:27 EST 2020" 62 } 63 } 64 ``` 65 66 </td> 67 <td> 68 69 ```json 70 { 71 "timestamp": "2020-06-05T13:50:27-05:00", 72 "record": {} 73 } 74 ``` 75 76 </td> 77 </tr> 78 </table> 79 80 #### Parse a timestamp using a `gotime` layout 81 82 The `gotime` layout type uses Golang's native time parsing capabilities. Golang takes an [unconventional approach](https://www.pauladamsmith.com/blog/2011/05/go_time.html) to time parsing. Finer details are well-documented [here](https://golang.org/src/time/format.go?s=25102:25148#L9). 83 84 Configuration: 85 ```yaml 86 - type: time_parser 87 parse_from: timestamp_field 88 layout_type: gotime 89 layout: Jan 2 15:04:05 MST 2006 90 ``` 91 92 <table> 93 <tr><td> Input entry </td> <td> Output entry </td></tr> 94 <tr> 95 <td> 96 97 ```json 98 { 99 "timestamp": "", 100 "record": { 101 "timestamp_field": "Jun 5 13:50:27 EST 2020" 102 } 103 } 104 ``` 105 106 </td> 107 <td> 108 109 ```json 110 { 111 "timestamp": "2020-06-05T13:50:27-05:00", 112 "record": {} 113 } 114 ``` 115 116 </td> 117 </tr> 118 </table> 119 120 #### Parse a timestamp using an `epoch` layout (and preserve the original value) 121 122 The `epoch` layout type uses can consume epoch-based timestamps. The following layouts are supported: 123 124 | Layout | Meaning | Example | `parse_from` data type support | 125 | --- | --- | --- | --- | 126 | `s` | Seconds since the epoch | 1136214245 | `string`, `int64`, `float64` | 127 | `ms` | Milliseconds since the epoch | 1136214245123 | `string`, `int64`, `float64` | 128 | `us` | Microseconds since the epoch | 1136214245123456 | `string`, `int64`, `float64` | 129 | `ns` | Nanoseconds since the epoch | 1136214245123456789 | `string`, `int64`, `float64`<sup>[2]</sup> | 130 | `s.ms` | Seconds plus milliseconds since the epoch | 1136214245.123 | `string`, `int64`<sup>[1]</sup>, `float64` | 131 | `s.us` | Seconds plus microseconds since the epoch | 1136214245.123456 | `string`, `int64`<sup>[1]</sup>, `float64` | 132 | `s.ns` | Seconds plus nanoseconds since the epoch | 1136214245.123456789 | `string`, `int64`<sup>[1]</sup>, `float64`<sup>[2]</sup> | 133 134 <sub>[1] Interpretted as seconds. Equivalent to using `s` layout.</sub><br/> 135 <sub>[2] Due to floating point precision limitations, loss of up to 100ns may be expected.</sub> 136 137 138 139 Configuration: 140 ```yaml 141 - type: time_parser 142 parse_from: timestamp_field 143 layout_type: epoch 144 layout: s 145 preserve: true 146 ``` 147 148 <table> 149 <tr><td> Input entry </td> <td> Output entry </td></tr> 150 <tr> 151 <td> 152 153 ```json 154 { 155 "timestamp": "", 156 "record": { 157 "timestamp_field": 1136214245 158 } 159 } 160 ``` 161 162 </td> 163 <td> 164 165 ```json 166 { 167 "timestamp": "2006-01-02T15:04:05-07:00", 168 "record": { 169 "timestamp_field": 1136214245 170 } 171 } 172 ``` 173 174 </td> 175 </tr> 176 </table>