github.com/netdata/go.d.plugin@v0.58.1/modules/weblog/metadata.yaml (about) 1 plugin_name: go.d.plugin 2 modules: 3 - meta: 4 id: collector-go.d.plugin-web_log 5 plugin_name: go.d.plugin 6 module_name: web_log 7 monitored_instance: 8 name: Web server log files 9 link: "" 10 categories: 11 - data-collection.web-servers-and-web-proxies 12 icon_filename: webservers.svg 13 keywords: 14 - webserver 15 - apache 16 - httpd 17 - nginx 18 - lighttpd 19 - logs 20 most_popular: false 21 info_provided_to_referring_integrations: 22 description: "" 23 related_resources: 24 integrations: 25 list: [] 26 overview: 27 data_collection: 28 metrics_description: | 29 This collector monitors web servers by parsing their log files. 30 method_description: "" 31 default_behavior: 32 auto_detection: 33 description: | 34 It automatically detects log files of web servers running on localhost. 35 limits: 36 description: "" 37 performance_impact: 38 description: "" 39 additional_permissions: 40 description: "" 41 multi_instance: true 42 supported_platforms: 43 include: [] 44 exclude: [] 45 setup: 46 prerequisites: 47 list: [] 48 configuration: 49 file: 50 name: go.d/web_log.conf 51 options: 52 description: | 53 Weblog is aware of how to parse and interpret the following fields (**known fields**): 54 55 > [nginx](https://nginx.org/en/docs/varindex.html) 56 > 57 > [apache](https://httpd.apache.org/docs/current/mod/mod_log_config.html) 58 59 | nginx | apache | description | 60 |-------------------------|----------|------------------------------------------------------------------------------------------| 61 | $host ($http_host) | %v | Name of the server which accepted a request. | 62 | $server_port | %p | Port of the server which accepted a request. | 63 | $scheme | - | Request scheme. "http" or "https". | 64 | $remote_addr | %a (%h) | Client address. | 65 | $request | %r | Full original request line. The line is "$request_method $request_uri $server_protocol". | 66 | $request_method | %m | Request method. Usually "GET" or "POST". | 67 | $request_uri | %U | Full original request URI. | 68 | $server_protocol | %H | Request protocol. Usually "HTTP/1.0", "HTTP/1.1", or "HTTP/2.0". | 69 | $status | %s (%>s) | Response status code. | 70 | $request_length | %I | Bytes received from a client, including request and headers. | 71 | $bytes_sent | %O | Bytes sent to a client, including request and headers. | 72 | $body_bytes_sent | %B (%b) | Bytes sent to a client, not counting the response header. | 73 | $request_time | %D | Request processing time. | 74 | $upstream_response_time | - | Time spent on receiving the response from the upstream server. | 75 | $ssl_protocol | - | Protocol of an established SSL connection. | 76 | $ssl_cipher | - | String of ciphers used for an established SSL connection. | 77 78 Notes: 79 80 - Apache `%h` logs the IP address if [HostnameLookups](https://httpd.apache.org/docs/2.4/mod/core.html#hostnamelookups) is Off. The web log collector counts hostnames as IPv4 addresses. We recommend either to disable HostnameLookups or use `%a` instead of `%h`. 81 - Since httpd 2.0, unlike 1.3, the `%b` and `%B` format strings do not represent the number of bytes sent to the client, but simply the size in bytes of the HTTP response. It will differ, for instance, if the connection is aborted, or if SSL is used. The `%O` format provided by [`mod_logio`](https://httpd.apache.org/docs/2.4/mod/mod_logio.html) will log the actual number of bytes sent over the network. 82 - To get `%I` and `%O` working you need to enable `mod_logio` on Apache. 83 - NGINX logs URI with query parameters, Apache doesnt. 84 - `$request` is parsed into `$request_method`, `$request_uri` and `$server_protocol`. If you have `$request` in your log format, there is no sense to have others. 85 - Don't use both `$bytes_sent` and `$body_bytes_sent` (`%O` and `%B` or `%b`). The module does not distinguish between these parameters. 86 folding: 87 title: Config options 88 enabled: true 89 list: 90 - name: update_every 91 description: Data collection frequency. 92 default_value: 1 93 required: false 94 - name: autodetection_retry 95 description: Recheck interval in seconds. Zero means no recheck will be scheduled. 96 default_value: 0 97 required: false 98 - name: path 99 description: Path to the web server log file. 100 default_value: "" 101 required: true 102 - name: exclude_path 103 description: Path to exclude. 104 default_value: "*.gz" 105 required: false 106 - name: url_patterns 107 description: List of URL patterns. 108 default_value: "[]" 109 required: false 110 detailed_description: | 111 "URL pattern" scope metrics will be collected for each URL pattern. 112 113 Option syntax: 114 115 ```yaml 116 url_patterns: 117 - name: name1 118 pattern: pattern1 119 - name: name2 120 pattern: pattern2 121 ``` 122 - name: url_patterns.name 123 description: Used as a dimension name. 124 default_value: "" 125 required: true 126 - name: url_patterns.pattern 127 description: Used to match against full original request URI. Pattern syntax in [matcher](https://github.com/netdata/go.d.plugin/tree/master/pkg/matcher#supported-format). 128 default_value: "" 129 required: true 130 - name: parser 131 description: Log parser configuration. 132 default_value: "" 133 required: false 134 - name: parser.log_type 135 description: Log parser type. 136 default_value: auto 137 required: false 138 detailed_description: | 139 Weblog supports 5 different log parsers: 140 141 | Parser type | Description | 142 |-------------|-------------------------------------------| 143 | auto | Use CSV and auto-detect format | 144 | csv | A comma-separated values | 145 | json | [JSON](https://www.json.org/json-en.html) | 146 | ltsv | [LTSV](http://ltsv.org/) | 147 | regexp | Regular expression with named groups | 148 149 Syntax: 150 151 ```yaml 152 parser: 153 log_type: auto 154 ``` 155 156 If `log_type` parameter set to `auto` (which is default), weblog will try to auto-detect appropriate log parser and log format using the last line of the log file. 157 158 - checks if format is `CSV` (using regexp). 159 - checks if format is `JSON` (using regexp). 160 - assumes format is `CSV` and tries to find appropriate `CSV` log format using predefined list of formats. It tries to parse the line using each of them in the following order (the first one matches is used later): 161 162 ```sh 163 $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time 164 $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time 165 $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time $upstream_response_time 166 $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time 167 $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent 168 $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time 169 $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time 170 $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time $upstream_response_time 171 $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time 172 $remote_addr - - [$time_local] "$request" $status $body_bytes_sent 173 ``` 174 175 If you're using the default Apache/NGINX log format, auto-detect will work for you. If it doesn't work you need to set the format manually. 176 - name: parser.csv_config 177 description: CSV log parser config. 178 default_value: "" 179 required: false 180 - name: parser.csv_config.delimiter 181 description: CSV field delimiter. 182 default_value: "," 183 required: false 184 - name: parser.csv_config.format 185 description: CSV log format. 186 default_value: "" 187 required: false 188 detailed_description: "" 189 - name: parser.ltsv_config 190 description: LTSV log parser config. 191 default_value: "" 192 required: false 193 - name: parser.ltsv_config.field_delimiter 194 description: LTSV field delimiter. 195 default_value: "\\t" 196 required: false 197 - name: parser.ltsv_config.value_delimiter 198 description: LTSV value delimiter. 199 default_value: ":" 200 required: false 201 - name: parser.ltsv_config.mapping 202 description: LTSV fields mapping to **known fields**. 203 default_value: "" 204 required: true 205 detailed_description: | 206 The mapping is a dictionary where the key is a field, as in logs, and the value is the corresponding **known field**. 207 208 > **Note**: don't use `$` and `%` prefixes for mapped field names. 209 210 ```yaml 211 parser: 212 log_type: ltsv 213 ltsv_config: 214 mapping: 215 label1: field1 216 label2: field2 217 ``` 218 - name: parser.json_config 219 description: JSON log parser config. 220 default_value: "" 221 required: false 222 - name: parser.json_config.mapping 223 description: JSON fields mapping to **known fields**. 224 default_value: "" 225 required: true 226 detailed_description: | 227 The mapping is a dictionary where the key is a field, as in logs, and the value is the corresponding **known field**. 228 229 > **Note**: don't use `$` and `%` prefixes for mapped field names. 230 231 ```yaml 232 parser: 233 log_type: json 234 json_config: 235 mapping: 236 label1: field1 237 label2: field2 238 ``` 239 - name: parser.regexp_config 240 description: RegExp log parser config. 241 default_value: "" 242 required: false 243 - name: parser.regexp_config.pattern 244 description: RegExp pattern with named groups. 245 default_value: "" 246 required: true 247 detailed_description: | 248 Use pattern with subexpressions names. These names should be **known fields**. 249 250 > **Note**: don't use `$` and `%` prefixes for mapped field names. 251 252 Syntax: 253 254 ```yaml 255 parser: 256 log_type: regexp 257 regexp_config: 258 pattern: PATTERN 259 ``` 260 examples: 261 folding: 262 title: Config 263 enabled: true 264 list: [] 265 troubleshooting: 266 problems: 267 list: [] 268 alerts: 269 - name: web_log_1m_unmatched 270 metric: web_log.excluded_requests 271 info: percentage of unparsed log lines over the last minute 272 link: https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf 273 - name: web_log_1m_requests 274 metric: web_log.type_requests 275 info: "ratio of successful HTTP requests over the last minute (1xx, 2xx, 304, 401)" 276 link: https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf 277 - name: web_log_1m_redirects 278 metric: web_log.type_requests 279 info: "ratio of redirection HTTP requests over the last minute (3xx except 304)" 280 link: https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf 281 - name: web_log_1m_bad_requests 282 metric: web_log.type_requests 283 info: "ratio of client error HTTP requests over the last minute (4xx except 401)" 284 link: https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf 285 - name: web_log_1m_internal_errors 286 metric: web_log.type_requests 287 info: "ratio of server error HTTP requests over the last minute (5xx)" 288 link: https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf 289 - name: web_log_web_slow 290 metric: web_log.request_processing_time 291 info: average HTTP response time over the last 1 minute 292 link: https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf 293 - name: web_log_5m_requests_ratio 294 metric: web_log.type_requests 295 info: ratio of successful HTTP requests over over the last 5 minutes, compared with the previous 5 minutes 296 link: https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf 297 metrics: 298 folding: 299 title: Metrics 300 enabled: false 301 description: "" 302 availability: [] 303 scopes: 304 - name: global 305 description: These metrics refer to the entire monitored application. 306 labels: [] 307 metrics: 308 - name: web_log.requests 309 description: Total Requests 310 unit: requests/s 311 chart_type: line 312 dimensions: 313 - name: requests 314 - name: web_log.excluded_requests 315 description: Excluded Requests 316 unit: requests/s 317 chart_type: stacked 318 dimensions: 319 - name: unmatched 320 - name: web_log.type_requests 321 description: Requests By Type 322 unit: requests/s 323 chart_type: stacked 324 dimensions: 325 - name: success 326 - name: bad 327 - name: redirect 328 - name: error 329 - name: web_log.status_code_class_responses 330 description: Responses By Status Code Class 331 unit: responses/s 332 chart_type: stacked 333 dimensions: 334 - name: 1xx 335 - name: 2xx 336 - name: 3xx 337 - name: 4xx 338 - name: 5xx 339 - name: web_log.status_code_class_1xx_responses 340 description: Informational Responses By Status Code 341 unit: responses/s 342 chart_type: stacked 343 dimensions: 344 - name: a dimension per 1xx code 345 - name: web_log.status_code_class_2xx_responses 346 description: Successful Responses By Status Code 347 unit: responses/s 348 chart_type: stacked 349 dimensions: 350 - name: a dimension per 2xx code 351 - name: web_log.status_code_class_3xx_responses 352 description: Redirects Responses By Status Code 353 unit: responses/s 354 chart_type: stacked 355 dimensions: 356 - name: a dimension per 3xx code 357 - name: web_log.status_code_class_4xx_responses 358 description: Client Errors Responses By Status Code 359 unit: responses/s 360 chart_type: stacked 361 dimensions: 362 - name: a dimension per 4xx code 363 - name: web_log.status_code_class_5xx_responses 364 description: Server Errors Responses By Status Code 365 unit: responses/s 366 chart_type: stacked 367 dimensions: 368 - name: a dimension per 5xx code 369 - name: web_log.bandwidth 370 description: Bandwidth 371 unit: kilobits/s 372 chart_type: area 373 dimensions: 374 - name: received 375 - name: sent 376 - name: web_log.request_processing_time 377 description: Request Processing Time 378 unit: milliseconds 379 chart_type: line 380 dimensions: 381 - name: min 382 - name: max 383 - name: avg 384 - name: web_log.requests_processing_time_histogram 385 description: Requests Processing Time Histogram 386 unit: requests/s 387 chart_type: line 388 dimensions: 389 - name: a dimension per bucket 390 - name: web_log.upstream_response_time 391 description: Upstream Response Time 392 unit: milliseconds 393 chart_type: line 394 dimensions: 395 - name: min 396 - name: max 397 - name: avg 398 - name: web_log.upstream_responses_time_histogram 399 description: Upstream Responses Time Histogram 400 unit: requests/s 401 chart_type: line 402 dimensions: 403 - name: a dimension per bucket 404 - name: web_log.current_poll_uniq_clients 405 description: Current Poll Unique Clients 406 unit: clients 407 chart_type: stacked 408 dimensions: 409 - name: ipv4 410 - name: ipv6 411 - name: web_log.vhost_requests 412 description: Requests By Vhost 413 unit: requests/s 414 chart_type: stacked 415 dimensions: 416 - name: a dimension per vhost 417 - name: web_log.port_requests 418 description: Requests By Port 419 unit: requests/s 420 chart_type: stacked 421 dimensions: 422 - name: a dimension per port 423 - name: web_log.scheme_requests 424 description: Requests By Scheme 425 unit: requests/s 426 chart_type: stacked 427 dimensions: 428 - name: http 429 - name: https 430 - name: web_log.http_method_requests 431 description: Requests By HTTP Method 432 unit: requests/s 433 chart_type: stacked 434 dimensions: 435 - name: a dimension per HTTP method 436 - name: web_log.http_version_requests 437 description: Requests By HTTP Version 438 unit: requests/s 439 chart_type: stacked 440 dimensions: 441 - name: a dimension per HTTP version 442 - name: web_log.ip_proto_requests 443 description: Requests By IP Protocol 444 unit: requests/s 445 chart_type: stacked 446 dimensions: 447 - name: ipv4 448 - name: ipv6 449 - name: web_log.ssl_proto_requests 450 description: Requests By SSL Connection Protocol 451 unit: requests/s 452 chart_type: stacked 453 dimensions: 454 - name: a dimension per SSL protocol 455 - name: web_log.ssl_cipher_suite_requests 456 description: Requests By SSL Connection Cipher Suite 457 unit: requests/s 458 chart_type: stacked 459 dimensions: 460 - name: a dimension per SSL cipher suite 461 - name: web_log.url_pattern_requests 462 description: URL Field Requests By Pattern 463 unit: requests/s 464 chart_type: stacked 465 dimensions: 466 - name: a dimension per URL pattern 467 - name: web_log.custom_field_pattern_requests 468 description: Custom Field Requests By Pattern 469 unit: requests/s 470 chart_type: stacked 471 dimensions: 472 - name: a dimension per custom field pattern 473 - name: custom time field 474 description: TBD 475 labels: [] 476 metrics: 477 - name: web_log.custom_time_field_summary 478 description: Custom Time Field Summary 479 unit: milliseconds 480 chart_type: line 481 dimensions: 482 - name: min 483 - name: max 484 - name: avg 485 - name: web_log.custom_time_field_histogram 486 description: Custom Time Field Histogram 487 unit: observations 488 chart_type: line 489 dimensions: 490 - name: a dimension per bucket 491 - name: custom numeric field 492 description: TBD 493 labels: [] 494 metrics: 495 - name: web_log.custom_numeric_field_{{field_name}}_summary 496 description: Custom Numeric Field Summary 497 unit: '{{units}}' 498 chart_type: line 499 dimensions: 500 - name: min 501 - name: max 502 - name: avg 503 - name: URL pattern 504 description: TBD 505 labels: [] 506 metrics: 507 - name: web_log.url_pattern_status_code_responses 508 description: Responses By Status Code 509 unit: responses/s 510 chart_type: line 511 dimensions: 512 - name: a dimension per pattern 513 - name: web_log.url_pattern_http_method_requests 514 description: Requests By HTTP Method 515 unit: requests/s 516 chart_type: line 517 dimensions: 518 - name: a dimension per HTTP method 519 - name: web_log.url_pattern_bandwidth 520 description: Bandwidth 521 unit: kilobits/s 522 chart_type: area 523 dimensions: 524 - name: received 525 - name: sent 526 - name: web_log.url_pattern_request_processing_time 527 description: Request Processing Time 528 unit: milliseconds 529 chart_type: line 530 dimensions: 531 - name: min 532 - name: max 533 - name: avg