github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/docs/sources/api/_index.md (about) 1 --- 2 title: HTTP API 3 menuTitle: "HTTP API" 4 description: "Loki exposes REST endpoints for operating on a Loki cluster. This section details the endpoints." 5 weight: 900 6 --- 7 8 # Grafana Loki HTTP API 9 10 Grafana Loki exposes an HTTP API for pushing, querying, and tailing log data. 11 Note that authenticating against the API is 12 out of scope for Loki. 13 14 ## Microservices mode 15 16 When deploying Loki in microservices mode, the set of endpoints exposed by each 17 component is different. 18 19 These endpoints are exposed by all components: 20 21 - [`GET /ready`](#identify-ready-loki-instance) 22 - [`GET /metrics`](#return-exposed-prometheus-metrics) 23 - [`GET /config`](#list-current-configuration) 24 - [`GET /services`](#list-running-services) 25 - [`GET /loki/api/v1/status/buildinfo`](#list-build-information) 26 27 These endpoints are exposed by the querier and the query frontend: 28 29 - [`GET /loki/api/v1/query`](#query-loki) 30 - [`GET /loki/api/v1/query_range`](#query-loki-over-a-range-of-time) 31 - [`GET /loki/api/v1/labels`](#list-labels-within-a-range-of-time) 32 - [`GET /loki/api/v1/label/<name>/values`](#list-label-values-within-a-range-of-time) 33 - [`GET /loki/api/v1/series`](#list-series) 34 - [`GET /loki/api/v1/index/stats`](#index-stats) 35 - [`GET /loki/api/v1/tail`](#stream-log-messages) 36 - [`POST /loki/api/v1/push`](#push-log-entries-to-loki) 37 - [`GET /ready`](#identify-ready-loki-instance) 38 - [`GET /metrics`](#return-exposed-prometheus-metrics) 39 - **Deprecated** [`GET /api/prom/tail`](#get-apipromtail) 40 - **Deprecated** [`GET /api/prom/query`](#get-apipromquery) 41 - **Deprecated** [`GET /api/prom/label`](#get-apipromlabel) 42 - **Deprecated** [`GET /api/prom/label/<name>/values`](#get-apipromlabelnamevalues) 43 - **Deprecated** [`POST /api/prom/push`](#post-apiprompush) 44 45 These endpoints are exposed by the distributor: 46 47 - [`POST /loki/api/v1/push`](#push-log-entries-to-loki) 48 - [`GET /distributor/ring`](#display-distributor-consistent-hash-ring-status) 49 50 These endpoints are exposed by the ingester: 51 52 - [`POST /flush`](#flush-in-memory-chunks-to-backing-store) 53 - [`POST /ingester/shutdown`](#flush-in-memory-chunks-and-shut-down) 54 - **Deprecated** [`POST /ingester/flush_shutdown`](#post-ingesterflush_shutdown) 55 56 The API endpoints starting with `/loki/` are [Prometheus API-compatible](https://prometheus.io/docs/prometheus/latest/querying/api/) and the result formats can be used interchangeably. 57 58 These endpoints are exposed by the ruler: 59 60 - [`GET /ruler/ring`](#ruler-ring-status) 61 - [`GET /loki/api/v1/rules`](#list-rule-groups) 62 - [`GET /loki/api/v1/rules/{namespace}`](#get-rule-groups-by-namespace) 63 - [`GET /loki/api/v1/rules/{namespace}/{groupName}`](#get-rule-group) 64 - [`POST /loki/api/v1/rules/{namespace}`](#set-rule-group) 65 - [`DELETE /loki/api/v1/rules/{namespace}/{groupName}`](#delete-rule-group) 66 - [`DELETE /loki/api/v1/rules/{namespace}`](#delete-namespace) 67 - [`GET /api/prom/rules`](#list-rule-groups) 68 - [`GET /api/prom/rules/{namespace}`](#get-rule-groups-by-namespace) 69 - [`GET /api/prom/rules/{namespace}/{groupName}`](#get-rule-group) 70 - [`POST /api/prom/rules/{namespace}`](#set-rule-group) 71 - [`DELETE /api/prom/rules/{namespace}/{groupName}`](#delete-rule-group) 72 - [`DELETE /api/prom/rules/{namespace}`](#delete-namespace) 73 - [`GET /prometheus/api/v1/rules`](#list-rules) 74 - [`GET /prometheus/api/v1/alerts`](#list-alerts) 75 76 These endpoints are exposed by the compactor: 77 - [`GET /compactor/ring`](#get-compactorring) 78 - [`POST /loki/api/v1/delete`](#post-lokiapiv1delete) 79 - [`GET /loki/api/v1/delete`](#get-lokiapiv1delete) 80 - [`DELETE /loki/api/v1/delete`](#delete-lokiapiv1delete) 81 82 A [list of clients](../clients) can be found in the clients documentation. 83 84 ## Matrix, vector, and streams 85 86 Some Loki API endpoints return a result of a matrix, a vector, or a stream: 87 88 - Matrix: a table of values where each row represents a different label set 89 and the columns are each sample value for that row over the queried time. 90 Matrix types are only returned when running a query that computes some value. 91 92 - Instant Vector: denoted in the type as just `vector`, an Instant Vector 93 represents the latest value of a calculation for a given labelset. Instant 94 Vectors are only returned when doing a query against a single point in 95 time. 96 97 - Stream: a Stream is a set of all values (logs) for a given label set over the 98 queried time range. Streams are the only type that will result in log lines 99 being returned. 100 101 ## Timestamp formats 102 103 The API accepts several formats for timestamps. An integer with ten or fewer digits is interpreted as a Unix timestamp in seconds. More than ten digits are interpreted as a Unix timestamp in nanoseconds. A floating point number is a Unix timestamp with fractions of a second. 104 105 The timestamps can also be written in `RFC3339` and `RFC3339Nano` format, as supported by Go's [time](https://pkg.go.dev/time) package. 106 107 ## Query Loki 108 109 ``` 110 GET /loki/api/v1/query 111 ``` 112 113 `/loki/api/v1/query` allows for doing queries against a single point in time. The URL 114 query parameters support the following values: 115 116 - `query`: The [LogQL](../logql/) query to perform 117 - `limit`: The max number of entries to return. It defaults to `100`. Only applies to query types which produce a stream(log lines) response. 118 - `time`: The evaluation time for the query as a nanosecond Unix epoch or another [supported format](#timestamp-formats). Defaults to now. 119 - `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward`. 120 121 In microservices mode, `/loki/api/v1/query` is exposed by the querier and the frontend. 122 123 Response format: 124 125 ``` 126 { 127 "status": "success", 128 "data": { 129 "resultType": "vector" | "streams", 130 "result": [<vector value>] | [<stream value>], 131 "stats" : [<statistics>] 132 } 133 } 134 ``` 135 136 where `<vector value>` is: 137 138 ``` 139 { 140 "metric": { 141 <label key-value pairs> 142 }, 143 "value": [ 144 <number: second unix epoch>, 145 <string: value> 146 ] 147 } 148 ``` 149 150 and `<stream value>` is: 151 152 ``` 153 { 154 "stream": { 155 <label key-value pairs> 156 }, 157 "values": [ 158 [ 159 <string: nanosecond unix epoch>, 160 <string: log line> 161 ], 162 ... 163 ] 164 } 165 ``` 166 167 The items in the `values` array are sorted by timestamp. 168 The most recent item is first when using `direction=backward`. 169 The oldest item is first when using `direction=forward`. 170 171 See [statistics](#statistics) for information about the statistics returned by Loki. 172 173 ### Examples 174 175 This example query 176 ```bash 177 curl -G -s "http://localhost:3100/loki/api/v1/query" \ 178 --data-urlencode \ 179 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq 180 ``` 181 182 gave this response: 183 184 ```json 185 { 186 "status": "success", 187 "data": { 188 "resultType": "vector", 189 "result": [ 190 { 191 "metric": {}, 192 "value": [ 193 1588889221, 194 "1267.1266666666666" 195 ] 196 }, 197 { 198 "metric": { 199 "level": "warn" 200 }, 201 "value": [ 202 1588889221, 203 "37.77166666666667" 204 ] 205 }, 206 { 207 "metric": { 208 "level": "info" 209 }, 210 "value": [ 211 1588889221, 212 "37.69" 213 ] 214 } 215 ], 216 "stats": { 217 ... 218 } 219 } 220 } 221 ``` 222 223 If your cluster has 224 [Grafana Loki Multi-Tenancy](../operations/multi-tenancy/) enabled, 225 set the `X-Scope-OrgID` header to identify the tenant you want to query. 226 Here is the same example query for the single tenant called `Tenant1`: 227 228 ```bash 229 curl -H 'X-Scope-OrgID:Tenant1' \ 230 -G -s "http://localhost:3100/loki/api/v1/query" \ 231 --data-urlencode \ 232 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq 233 ``` 234 235 To query against the three tenants `Tenant1`, `Tenant2`, and `Tenant3`, 236 specify the tenant names separated by the pipe (`|`) character: 237 238 ```bash 239 curl -H 'X-Scope-OrgID:Tenant1|Tenant2|Tenant3' \ 240 -G -s "http://localhost:3100/loki/api/v1/query" \ 241 --data-urlencode \ 242 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq 243 ``` 244 245 The same example query for Grafana Enterprise Logs 246 uses Basic Authentication and specifies the tenant names as a `user`. 247 The tenant names are separated by the pipe (`|`) character. 248 The password in this example is an access policy token that has been 249 defined in the `API_TOKEN` environment variable: 250 251 ```bash 252 curl -u "Tenant1|Tenant2|Tenant3:$API_TOKEN" \ 253 -G -s "http://localhost:3100/loki/api/v1/query" \ 254 --data-urlencode \ 255 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq 256 ``` 257 258 ## Query Loki over a range of time 259 260 ``` 261 GET /loki/api/v1/query_range 262 ``` 263 264 `/loki/api/v1/query_range` is used to do a query over a range of time and 265 accepts the following query parameters in the URL: 266 267 - `query`: The [LogQL](../logql/) query to perform 268 - `limit`: The max number of entries to return. It defaults to `100`. Only applies to query types which produce a stream(log lines) response. 269 - `start`: The start time for the query as a nanosecond Unix epoch or another [supported format](#timestamp-formats). Defaults to one hour ago. 270 - `end`: The end time for the query as a nanosecond Unix epoch or another [supported format](#timestamp-formats). Defaults to now. 271 - `step`: Query resolution step width in `duration` format or float number of seconds. `duration` refers to Prometheus duration strings of the form `[0-9]+[smhdwy]`. For example, 5m refers to a duration of 5 minutes. Defaults to a dynamic value based on `start` and `end`. Only applies to query types which produce a matrix response. 272 - `interval`: <span style="background-color:#f3f973;">This parameter is experimental; see the explanation under Step versus interval.</span> Only return entries at (or greater than) the specified interval, can be a `duration` format or float number of seconds. Only applies to queries which produce a stream response. 273 - `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward.` 274 275 In microservices mode, `/loki/api/v1/query_range` is exposed by the querier and the frontend. 276 277 ### Step versus interval 278 279 Use the `step` parameter when making metric queries to Loki, or queries which return a matrix response. It is evaluated in exactly the same way Prometheus evaluates `step`. First the query will be evaluated at `start` and then evaluated again at `start + step` and again at `start + step + step` until `end` is reached. The result will be a matrix of the query result evaluated at each step. 280 281 Use the `interval` parameter when making log queries to Loki, or queries which return a stream response. It is evaluated by returning a log entry at `start`, then the next entry will be returned an entry with timestampe >= `start + interval`, and again at `start + interval + interval` and so on until `end` is reached. It does not fill missing entries. 282 283 <span style="background-color:#f3f973;">Note about the experimental nature of the interval parameter:</span> This flag may be removed in the future, if so it will likely be in favor of a LogQL expression to perform similar behavior, however that is uncertain at this time. [Issue 1779](https://github.com/grafana/loki/issues/1779) was created to track the discussion, if you are using `interval` please go add your use case and thoughts to that issue. 284 285 286 287 Response: 288 289 ``` 290 { 291 "status": "success", 292 "data": { 293 "resultType": "matrix" | "streams", 294 "result": [<matrix value>] | [<stream value>] 295 "stats" : [<statistics>] 296 } 297 } 298 ``` 299 300 Where `<matrix value>` is: 301 302 ``` 303 { 304 "metric": { 305 <label key-value pairs> 306 }, 307 "values": [ 308 [ 309 <number: second unix epoch>, 310 <string: value> 311 ], 312 ... 313 ] 314 } 315 ``` 316 317 The items in the `values` array are sorted by timestamp, and the oldest item is first. 318 319 And `<stream value>` is: 320 321 ``` 322 { 323 "stream": { 324 <label key-value pairs> 325 }, 326 "values": [ 327 [ 328 <string: nanosecond unix epoch>, 329 <string: log line> 330 ], 331 ... 332 ] 333 } 334 ``` 335 336 The items in the `values` array are sorted by timestamp. 337 The most recent item is first when using `direction=backward`. 338 The oldest item is first when using `direction=forward`. 339 340 See [statistics](#statistics) for information about the statistics returned by Loki. 341 342 ### Examples 343 344 ```bash 345 $ curl -G -s "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' --data-urlencode 'step=300' | jq 346 { 347 "status": "success", 348 "data": { 349 "resultType": "matrix", 350 "result": [ 351 { 352 "metric": { 353 "level": "info" 354 }, 355 "values": [ 356 [ 357 1588889221, 358 "137.95" 359 ], 360 [ 361 1588889221, 362 "467.115" 363 ], 364 [ 365 1588889221, 366 "658.8516666666667" 367 ] 368 ] 369 }, 370 { 371 "metric": { 372 "level": "warn" 373 }, 374 "values": [ 375 [ 376 1588889221, 377 "137.27833333333334" 378 ], 379 [ 380 1588889221, 381 "467.69" 382 ], 383 [ 384 1588889221, 385 "660.6933333333334" 386 ] 387 ] 388 } 389 ], 390 "stats": { 391 ... 392 } 393 } 394 } 395 ``` 396 397 ```bash 398 $ curl -G -s "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="varlogs"}' | jq 399 { 400 "status": "success", 401 "data": { 402 "resultType": "streams", 403 "result": [ 404 { 405 "stream": { 406 "filename": "/var/log/myproject.log", 407 "job": "varlogs", 408 "level": "info" 409 }, 410 "values": [ 411 [ 412 "1569266497240578000", 413 "foo" 414 ], 415 [ 416 "1569266492548155000", 417 "bar" 418 ] 419 ] 420 } 421 ], 422 "stats": { 423 ... 424 } 425 } 426 } 427 ``` 428 429 ## List labels within a range of time 430 431 ``` 432 GET /loki/api/v1/labels 433 ``` 434 435 `/loki/api/v1/labels` retrieves the list of known labels within a given time span. 436 Loki may use a larger time span than the one specified. 437 It accepts the following query parameters in the URL: 438 439 - `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago. 440 - `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now. 441 442 In microservices mode, `/loki/api/v1/labels` is exposed by the querier. 443 444 Response: 445 446 ``` 447 { 448 "status": "success", 449 "data": [ 450 <label string>, 451 ... 452 ] 453 } 454 ``` 455 456 ### Examples 457 458 ```bash 459 $ curl -G -s "http://localhost:3100/loki/api/v1/labels" | jq 460 { 461 "status": "success", 462 "data": [ 463 "foo", 464 "bar", 465 "baz" 466 ] 467 } 468 ``` 469 470 ## List label values within a range of time 471 472 ``` 473 GET /loki/api/v1/label/<name>/values 474 ``` 475 476 `/loki/api/v1/label/<name>/values` retrieves the list of known values for a given 477 label within a given time span. Loki may use a larger time span than the one specified. 478 It accepts the following query parameters in the URL: 479 480 - `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago. 481 - `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now. 482 483 In microservices mode, `/loki/api/v1/label/<name>/values` is exposed by the querier. 484 485 Response: 486 487 ``` 488 { 489 "status": "success", 490 "data": [ 491 <label value>, 492 ... 493 ] 494 } 495 ``` 496 497 ### Examples 498 499 ```bash 500 $ curl -G -s "http://localhost:3100/loki/api/v1/label/foo/values" | jq 501 { 502 "status": "success", 503 "data": [ 504 "cat", 505 "dog", 506 "axolotl" 507 ] 508 } 509 ``` 510 511 ## Stream log messages 512 513 ``` 514 GET /loki/api/v1/tail 515 ``` 516 517 `/loki/api/v1/tail` is a WebSocket endpoint that will stream log messages based on 518 a query. It accepts the following query parameters in the URL: 519 520 - `query`: The [LogQL](../logql/) query to perform 521 - `delay_for`: The number of seconds to delay retrieving logs to let slow 522 loggers catch up. Defaults to 0 and cannot be larger than 5. 523 - `limit`: The max number of entries to return. It defaults to `100`. 524 - `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago. 525 526 In microservices mode, `/loki/api/v1/tail` is exposed by the querier. 527 528 Response (streamed): 529 530 ``` 531 { 532 "streams": [ 533 { 534 "stream": { 535 <label key-value pairs> 536 }, 537 "values": [ 538 [ 539 <string: nanosecond unix epoch>, 540 <string: log line> 541 ] 542 ] 543 } 544 ], 545 "dropped_entries": [ 546 { 547 "labels": { 548 <label key-value pairs> 549 }, 550 "timestamp": "<nanosecond unix epoch>" 551 } 552 ] 553 } 554 ``` 555 556 ## Push log entries to Loki 557 558 ``` 559 POST /loki/api/v1/push 560 ``` 561 562 `/loki/api/v1/push` is the endpoint used to send log entries to Loki. The default 563 behavior is for the POST body to be a snappy-compressed protobuf message: 564 565 - [Protobuf definition](https://github.com/grafana/loki/blob/main/pkg/logproto/logproto.proto) 566 - [Go client library](https://github.com/grafana/loki/blob/main/clients/pkg/promtail/client/client.go) 567 568 Alternatively, if the `Content-Type` header is set to `application/json`, a 569 JSON post body can be sent in the following format: 570 571 ``` 572 { 573 "streams": [ 574 { 575 "stream": { 576 "label": "value" 577 }, 578 "values": [ 579 [ "<unix epoch in nanoseconds>", "<log line>" ], 580 [ "<unix epoch in nanoseconds>", "<log line>" ] 581 ] 582 } 583 ] 584 } 585 ``` 586 587 You can set `Content-Encoding: gzip` request header and post gzipped JSON. 588 589 Loki can be configured to [accept out-of-order writes](../configuration/#accept-out-of-order-writes). 590 591 In microservices mode, `/loki/api/v1/push` is exposed by the distributor. 592 593 ### Examples 594 595 ```bash 596 $ curl -v -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw \ 597 '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1570818238000000000", "fizzbuzz" ] ] }]}' 598 ``` 599 600 601 ## Identify ready Loki instance 602 603 ``` 604 GET /ready 605 ``` 606 607 `/ready` returns HTTP 200 when the Loki instance is ready to accept traffic. If 608 running Loki on Kubernetes, `/ready` can be used as a readiness probe. 609 610 In microservices mode, the `/ready` endpoint is exposed by all components. 611 612 ## Flush in-memory chunks to backing store 613 614 ``` 615 POST /flush 616 ``` 617 618 `/flush` triggers a flush of all in-memory chunks held by the ingesters to the 619 backing store. Mainly used for local testing. 620 621 In microservices mode, the `/flush` endpoint is exposed by the ingester. 622 623 ## Flush in-memory chunks and shut down 624 625 ``` 626 POST /ingester/shutdown 627 ``` 628 629 `/ingester/shutdown` triggers a shutdown of the ingester and notably will _always_ flush any in memory chunks it holds. 630 This is helpful for scaling down WAL-enabled ingesters where we want to ensure old WAL directories are not orphaned, 631 but instead flushed to our chunk backend. 632 633 It accepts three URL query parameters `flush`, `delete_ring_tokens`, and `terminate`. 634 635 **URL query parameters:** 636 637 * `flush=<bool>`: 638 Flag to control whether to flush any in-memory chunks the ingester holds. Defaults to `true`. 639 * `delete_ring_tokens=<bool>`: 640 Flag to control whether to delete the file that contains the ingester ring tokens of the instance if the `-ingester.token-file-path` is specified. 641 * `terminate=<bool>`: 642 Flag to control whether to terminate the Loki process after service shutdown. Defaults to `true`. 643 644 This handler, in contrast to the deprecated `/ingester/flush_shutdown` handler, terminates the Loki process by default. 645 This behaviour can be changed by setting the `terminate` query parameter to `false`. 646 647 In microservices mode, the `/ingester/shutdown` endpoint is exposed by the ingester. 648 649 ## Display distributor consistent hash ring status 650 651 ``` 652 GET /distributor/ring 653 ``` 654 655 Displays a web page with the distributor hash ring status, including the state, healthy and last heartbeat time of each distributor. 656 657 ## Return exposed Prometheus metrics 658 659 ``` 660 GET /metrics 661 ``` 662 663 `/metrics` returns exposed Prometheus metrics. See 664 [Observing Loki](../operations/observability/) 665 for a list of exported metrics. 666 667 In microservices mode, the `/metrics` endpoint is exposed by all components. 668 669 ## List current configuration 670 671 ``` 672 GET /config 673 ``` 674 675 `/config` exposes the current configuration. The optional `mode` query parameter can be used to 676 modify the output. If it has the value `diff` only the differences between the default configuration 677 and the current are returned. A value of `defaults` returns the default configuration. 678 679 In microservices mode, the `/config` endpoint is exposed by all components. 680 681 ## List running services 682 683 ``` 684 GET /services 685 ``` 686 687 `/services` returns a list of all running services and their current states. 688 689 Services can have the following states: 690 691 - **New**: Service is new, not running yet (initial state) 692 - **Starting**: Service is starting; if starting succeeds, service enters **Running** state 693 - **Running**: Service is fully running now; when service stops running, it enters **Stopping** state 694 - **Stopping**: Service is shutting down 695 - **Terminated**: Service has stopped successfully (terminal state) 696 - **Failed**: Service has failed in **Starting**, **Running** or **Stopping** state (terminal state) 697 698 ## List build information 699 700 ``` 701 GET /loki/api/v1/status/buildinfo 702 ``` 703 704 `/loki/api/v1/status/buildinfo` exposes the build information in a JSON object. The fields are `version`, `revision`, `branch`, `buildDate`, `buildUser`, and `goVersion`. 705 706 ## List series 707 708 The Series API is available under the following: 709 - `GET /loki/api/v1/series` 710 - `POST /loki/api/v1/series` 711 - `GET /api/prom/series` 712 - `POST /api/prom/series` 713 714 This endpoint returns the list of time series that match a certain label set. 715 716 URL query parameters: 717 718 - `match[]=<series_selector>`: Repeated log stream selector argument that selects the streams to return. At least one `match[]` argument must be provided. 719 - `start=<nanosecond Unix epoch>`: Start timestamp. 720 - `end=<nanosecond Unix epoch>`: End timestamp. 721 722 You can URL-encode these parameters directly in the request body by using the POST method and `Content-Type: application/x-www-form-urlencoded` header. This is useful when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits. 723 724 In microservices mode, these endpoints are exposed by the querier. 725 726 ### Examples 727 728 ``` bash 729 $ curl -s "http://localhost:3100/loki/api/v1/series" --data-urlencode 'match[]={container_name=~"prometheus.*", component="server"}' --data-urlencode 'match[]={app="loki"}' | jq '.' 730 { 731 "status": "success", 732 "data": [ 733 { 734 "container_name": "loki", 735 "app": "loki", 736 "stream": "stderr", 737 "filename": "/var/log/pods/default_loki-stack-0_50835643-1df0-11ea-ba79-025000000001/loki/0.log", 738 "name": "loki", 739 "job": "default/loki", 740 "controller_revision_hash": "loki-stack-757479754d", 741 "statefulset_kubernetes_io_pod_name": "loki-stack-0", 742 "release": "loki-stack", 743 "namespace": "default", 744 "instance": "loki-stack-0" 745 }, 746 { 747 "chart": "prometheus-9.3.3", 748 "container_name": "prometheus-server-configmap-reload", 749 "filename": "/var/log/pods/default_loki-stack-prometheus-server-696cc9ddff-87lmq_507b1db4-1df0-11ea-ba79-025000000001/prometheus-server-configmap-reload/0.log", 750 "instance": "loki-stack-prometheus-server-696cc9ddff-87lmq", 751 "pod_template_hash": "696cc9ddff", 752 "app": "prometheus", 753 "component": "server", 754 "heritage": "Tiller", 755 "job": "default/prometheus", 756 "namespace": "default", 757 "release": "loki-stack", 758 "stream": "stderr" 759 }, 760 { 761 "app": "prometheus", 762 "component": "server", 763 "filename": "/var/log/pods/default_loki-stack-prometheus-server-696cc9ddff-87lmq_507b1db4-1df0-11ea-ba79-025000000001/prometheus-server/0.log", 764 "release": "loki-stack", 765 "namespace": "default", 766 "pod_template_hash": "696cc9ddff", 767 "stream": "stderr", 768 "chart": "prometheus-9.3.3", 769 "container_name": "prometheus-server", 770 "heritage": "Tiller", 771 "instance": "loki-stack-prometheus-server-696cc9ddff-87lmq", 772 "job": "default/prometheus" 773 } 774 ] 775 } 776 ``` 777 778 779 ## Index Stats 780 781 The `/loki/api/v1/index/stats` endpoint can be used to query the index for the number of `streams`, `chunks`, `entries`, and `bytes` that a query resolves to. 782 783 URL query parameters: 784 785 - `query`: The [LogQL](../logql/) matchers to check (i.e. `{job="foo", env!="dev"}`) 786 - `start=<nanosecond Unix epoch>`: Start timestamp. 787 - `end=<nanosecond Unix epoch>`: End timestamp. 788 789 You can URL-encode these parameters directly in the request body by using the POST method and `Content-Type: application/x-www-form-urlencoded` header. This is useful when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits. 790 791 Response: 792 ```json 793 { 794 "streams": 100, 795 "chunks": 1000, 796 "entries": 5000, 797 "bytes": 100000, 798 } 799 ``` 800 801 It is an approximation with the following caveats: 802 * It does not include data from the ingesters 803 * It is a probabilistic technique 804 * streams/chunks which span multiple period configurations may be counted twice. 805 806 These make it generally more helpful for larger queries. 807 It can be used for better understanding the throughput requirements and data topology for a list of matchers over a period of time. 808 809 810 ## Statistics 811 812 Query endpoints such as `/api/prom/query`, `/loki/api/v1/query` and `/loki/api/v1/query_range` return a set of statistics about the query execution. Those statistics allow users to understand the amount of data processed and at which speed. 813 814 The example belows show all possible statistics returned with their respective description. 815 816 ```json 817 { 818 "status": "success", 819 "data": { 820 "resultType": "streams", 821 "result": [], 822 "stats": { 823 "ingester" : { 824 "compressedBytes": 0, // Total bytes of compressed chunks (blocks) processed by ingesters 825 "decompressedBytes": 0, // Total bytes decompressed and processed by ingesters 826 "decompressedLines": 0, // Total lines decompressed and processed by ingesters 827 "headChunkBytes": 0, // Total bytes read from ingesters head chunks 828 "headChunkLines": 0, // Total lines read from ingesters head chunks 829 "totalBatches": 0, // Total batches sent by ingesters 830 "totalChunksMatched": 0, // Total chunks matched by ingesters 831 "totalDuplicates": 0, // Total of duplicates found by ingesters 832 "totalLinesSent": 0, // Total lines sent by ingesters 833 "totalReached": 0 // Amount of ingesters reached. 834 }, 835 "store": { 836 "compressedBytes": 0, // Total bytes of compressed chunks (blocks) processed by the store 837 "decompressedBytes": 0, // Total bytes decompressed and processed by the store 838 "decompressedLines": 0, // Total lines decompressed and processed by the store 839 "chunksDownloadTime": 0, // Total time spent downloading chunks in seconds (float) 840 "totalChunksRef": 0, // Total chunks found in the index for the current query 841 "totalChunksDownloaded": 0, // Total of chunks downloaded 842 "totalDuplicates": 0 // Total of duplicates removed from replication 843 }, 844 "summary": { 845 "bytesProcessedPerSecond": 0, // Total of bytes processed per second 846 "execTime": 0, // Total execution time in seconds (float) 847 "linesProcessedPerSecond": 0, // Total lines processed per second 848 "queueTime": 0, // Total queue time in seconds (float) 849 "totalBytesProcessed":0, // Total amount of bytes processed overall for this request 850 "totalLinesProcessed":0 // Total amount of lines processed overall for this request 851 } 852 } 853 } 854 } 855 ``` 856 857 ## Ruler 858 859 The ruler API endpoints require to configure a backend object storage to store the recording rules and alerts. The ruler API uses the concept of a "namespace" when creating rule groups. This is a stand-in for the name of the rule file in Prometheus. Rule groups must be named uniquely within a namespace. 860 861 ### Ruler ring status 862 863 ``` 864 GET /ruler/ring 865 ``` 866 867 Displays a web page with the ruler hash ring status, including the state, healthy and last heartbeat time of each ruler. 868 869 ### List rule groups 870 871 872 ``` 873 GET /loki/api/v1/rules 874 ``` 875 876 List all rules configured for the authenticated tenant. This endpoint returns a YAML dictionary with all the rule groups for each namespace and `200` status code on success. 877 878 #### Example response 879 880 ```yaml 881 --- 882 <namespace1>: 883 - name: <string> 884 interval: <duration;optional> 885 rules: 886 - alert: <string> 887 expr: <string> 888 for: <duration> 889 annotations: 890 <annotation_name>: <string> 891 labels: 892 <label_name>: <string> 893 - name: <string> 894 interval: <duration;optional> 895 rules: 896 - alert: <string> 897 expr: <string> 898 for: <duration> 899 annotations: 900 <annotation_name>: <string> 901 labels: 902 <label_name>: <string> 903 <namespace2>: 904 - name: <string> 905 interval: <duration;optional> 906 rules: 907 - alert: <string> 908 expr: <string> 909 for: <duration> 910 annotations: 911 <annotation_name>: <string> 912 labels: 913 <label_name>: <string> 914 ``` 915 916 ### Get rule groups by namespace 917 918 ``` 919 GET /loki/api/v1/rules/{namespace} 920 ``` 921 922 Returns the rule groups defined for a given namespace. 923 924 #### Example response 925 926 ```yaml 927 name: <string> 928 interval: <duration;optional> 929 rules: 930 - alert: <string> 931 expr: <string> 932 for: <duration> 933 annotations: 934 <annotation_name>: <string> 935 labels: 936 <label_name>: <string> 937 ``` 938 939 ### Get rule group 940 941 ``` 942 GET /loki/api/v1/rules/{namespace}/{groupName} 943 ``` 944 945 Returns the rule group matching the request namespace and group name. 946 947 ### Set rule group 948 949 ``` 950 POST /loki/api/v1/rules/{namespace} 951 ``` 952 953 Creates or updates a rule group. This endpoint expects a request with `Content-Type: application/yaml` header and the rules **YAML** definition in the request body, and returns `202` on success. 954 955 #### Example request 956 957 Request headers: 958 - `Content-Type: application/yaml` 959 960 Request body: 961 962 ```yaml 963 name: <string> 964 interval: <duration;optional> 965 rules: 966 - alert: <string> 967 expr: <string> 968 for: <duration> 969 annotations: 970 <annotation_name>: <string> 971 labels: 972 <label_name>: <string> 973 ``` 974 975 ### Delete rule group 976 977 ``` 978 DELETE /loki/api/v1/rules/{namespace}/{groupName} 979 980 ``` 981 982 Deletes a rule group by namespace and group name. This endpoints returns `202` on success. 983 984 ### Delete namespace 985 986 ``` 987 DELETE /loki/api/v1/rules/{namespace} 988 ``` 989 990 Deletes all the rule groups in a namespace (including the namespace itself). This endpoint returns `202` on success. 991 992 ### List rules 993 994 ``` 995 GET /prometheus/api/v1/rules 996 ``` 997 998 Prometheus-compatible rules endpoint to list alerting and recording rules that are currently loaded. 999 1000 For more information, refer to the [Prometheus rules](https://prometheus.io/docs/prometheus/latest/querying/api/#rules) documentation. 1001 1002 ### List alerts 1003 1004 ``` 1005 GET /prometheus/api/v1/alerts 1006 ``` 1007 1008 Prometheus-compatible rules endpoint to list all active alerts. 1009 1010 For more information, please check out the Prometheus [alerts](https://prometheus.io/docs/prometheus/latest/querying/api/#alerts) documentation. 1011 1012 ## Compactor 1013 1014 ### Compactor ring status 1015 1016 ``` 1017 GET /compactor/ring 1018 ``` 1019 1020 Displays a web page with the compactor hash ring status, including the state, health, and last heartbeat time of each compactor. 1021 1022 ### Request log deletion 1023 1024 ``` 1025 POST /loki/api/v1/delete 1026 PUT /loki/api/v1/delete 1027 ``` 1028 1029 Create a new delete request for the authenticated tenant. 1030 The [log entry deletion](../operations/storage/logs-deletion/) documentation has configuration details. 1031 1032 Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store. 1033 1034 Query parameters: 1035 1036 * `query=<series_selector>`: query argument that identifies the streams from which to delete with optional line filters. 1037 * `start=<rfc3339 | unix_seconds_timestamp>`: A timestamp that identifies the start of the time window within which entries will be deleted. This parameter is required. 1038 * `end=<rfc3339 | unix_seconds_timestamp>`: A timestamp that identifies the end of the time window within which entries will be deleted. If not specified, defaults to the current time. 1039 1040 A 204 response indicates success. 1041 1042 The query parameter can also include filter operations. For example `query={foo="bar"} |= "other"` will filter out lines that contain the string "other" for the streams matching the stream selector `{foo="bar"}`. 1043 1044 #### Examples 1045 1046 URL encode the `query` parameter. This sample form of a cURL command URL encodes `query={foo="bar"}`: 1047 1048 ```bash 1049 curl -g -X POST \ 1050 'http://127.0.0.1:3100/loki/api/v1/delete?query={foo="bar"}&start=1591616227&end=1591619692' \ 1051 -H 'X-Scope-OrgID: 1' 1052 ``` 1053 1054 The same example deletion request for Grafana Enterprise Logs uses Basic Authentication and specifies the tenant name as a user; `Tenant1` is the tenant name in this example. The password in this example is an access policy token that has been defined in the API_TOKEN environment variable. The token must be for an access policy with `logs:delete` scope for the tenant specified in the user field: 1055 1056 ```bash 1057 curl -u "Tenant1:$API_TOKEN" \ 1058 -g -X POST \ 1059 'http://127.0.0.1:3100/loki/api/v1/delete?query={foo="bar"}&start=1591616227&end=1591619692' 1060 ``` 1061 1062 ### List log deletion requests 1063 1064 ``` 1065 GET /loki/api/v1/delete 1066 ``` 1067 1068 List the existing delete requests for the authenticated tenant. 1069 The [log entry deletion](../operations/storage/logs-deletion/) documentation has configuration details. 1070 1071 Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store. 1072 1073 List the existing delete requests using the following API: 1074 1075 ``` 1076 GET /loki/api/v1/delete 1077 ``` 1078 1079 This endpoint returns both processed and unprocessed deletion requests. It does not list canceled requests, as those requests will have been removed from storage. 1080 1081 #### Examples 1082 1083 Example cURL command: 1084 1085 ``` 1086 curl -X GET \ 1087 <compactor_addr>/loki/api/v1/delete \ 1088 -H 'X-Scope-OrgID: <orgid>' 1089 ``` 1090 1091 The same example deletion request for Grafana Enterprise Logs uses Basic Authentication and specifies the tenant name as a user; `Tenant1` is the tenant name in this example. The password in this example is an access policy token that has been defined in the API_TOKEN environment variable. The token must be for an access policy with `logs:delete` scope for the tenant specified in the user field. 1092 1093 ```bash 1094 curl -u "Tenant1:$API_TOKEN" \ 1095 -X GET \ 1096 <compactor_addr>/loki/api/v1/delete 1097 ``` 1098 1099 ### Request cancellation of a delete request 1100 1101 ``` 1102 DELETE /loki/api/v1/delete 1103 ``` 1104 1105 Remove a delete request for the authenticated tenant. 1106 The [log entry deletion](../operations/storage/logs-deletion/) documentation has configuration details. 1107 1108 Loki allows cancellation of delete requests until the requests are picked up for processing. It is controlled by the `delete_request_cancel_period` YAML configuration or the equivalent command line option when invoking Loki. 1109 1110 Log entry deletion is supported _only_ when the BoltDB Shipper is configured for the index store. 1111 1112 Cancel a delete request using this compactor endpoint: 1113 1114 ``` 1115 DELETE /loki/api/v1/delete 1116 ``` 1117 1118 Query parameters: 1119 1120 * `request_id=<request_id>`: Identifies the delete request to cancel; IDs are found using the `delete` endpoint. 1121 1122 A 204 response indicates success. 1123 1124 #### Examples 1125 1126 Example cURL command: 1127 1128 ``` 1129 curl -X DELETE \ 1130 '<compactor_addr>/loki/api/v1/delete?request_id=<request_id>' \ 1131 -H 'X-Scope-OrgID: <tenant-id>' 1132 ``` 1133 1134 The same example deletion cancellation request for Grafana Enterprise Logs uses Basic Authentication and specifies the tenant name as a user; `Tenant1` is the tenant name in this example. The password in this example is an access policy token that has been defined in the API_TOKEN environment variable. The token must be for an access policy with `logs:delete` scope for the tenant specified in the user field. 1135 1136 ```bash 1137 curl -u "Tenant1:$API_TOKEN" \ 1138 -X DELETE \ 1139 '<compactor_addr>/loki/api/v1/delete?request_id=<request_id>' 1140 ``` 1141 1142 ## Deprecated endpoints 1143 1144 ### `GET /api/prom/tail` 1145 1146 > **DEPRECATED**: `/api/prom/tail` is deprecated. Use `/loki/api/v1/tail` 1147 > instead. 1148 1149 `/api/prom/tail` is a WebSocket endpoint that will stream log messages based on 1150 a query. It accepts the following query parameters in the URL: 1151 1152 - `query`: The [LogQL](../logql/) query to perform 1153 - `delay_for`: The number of seconds to delay retrieving logs to let slow 1154 loggers catch up. Defaults to 0 and cannot be larger than 5. 1155 - `limit`: The max number of entries to return 1156 - `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago. 1157 1158 In microservices mode, `/api/prom/tail` is exposed by the querier. 1159 1160 Response (streamed): 1161 1162 ```json 1163 { 1164 "streams": [ 1165 { 1166 "labels": "<LogQL label key-value pairs>", 1167 "entries": [ 1168 { 1169 "ts": "<RFC3339Nano timestamp>", 1170 "line": "<log line>" 1171 } 1172 ] 1173 } 1174 ], 1175 "dropped_entries": [ 1176 { 1177 "Timestamp": "<RFC3339Nano timestamp>", 1178 "Labels": "<LogQL label key-value pairs>" 1179 } 1180 ] 1181 } 1182 ``` 1183 1184 `dropped_entries` will be populated when the tailer could not keep up with the 1185 amount of traffic in Loki. When present, it indicates that the entries received 1186 in the streams is not the full amount of logs that are present in Loki. Note 1187 that the keys in `dropped_entries` will be sent as uppercase `Timestamp` 1188 and `Labels` instead of `labels` and `ts` like in the entries for the stream. 1189 1190 As the response is streamed, the object defined by the response format above 1191 will be sent over the WebSocket multiple times. 1192 1193 ### `GET /api/prom/query` 1194 1195 > **WARNING**: `/api/prom/query` is DEPRECATED; use `/loki/api/v1/query_range` 1196 > instead. 1197 1198 `/api/prom/query` supports doing general queries. The URL query parameters 1199 support the following values: 1200 1201 - `query`: The [LogQL](../logql/) query to perform 1202 - `limit`: The max number of entries to return 1203 - `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago. 1204 - `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now. 1205 - `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward.` 1206 - `regexp`: a regex to filter the returned results 1207 1208 In microservices mode, `/api/prom/query` is exposed by the querier and the frontend. 1209 1210 Note that the larger the time span between `start` and `end` will cause 1211 additional load on Loki and the index store, resulting in slower queries. 1212 1213 Response: 1214 1215 ``` 1216 { 1217 "streams": [ 1218 { 1219 "labels": "<LogQL label key-value pairs>", 1220 "entries": [ 1221 { 1222 "ts": "<RFC3339Nano string>", 1223 "line": "<log line>" 1224 }, 1225 ... 1226 ], 1227 }, 1228 ... 1229 ], 1230 "stats": [<statistics>] 1231 } 1232 ``` 1233 1234 See [statistics](#statistics) for information about the statistics returned by Loki. 1235 1236 #### Examples 1237 1238 ```bash 1239 $ curl -G -s "http://localhost:3100/api/prom/query" --data-urlencode 'query={foo="bar"}' | jq 1240 { 1241 "streams": [ 1242 { 1243 "labels": "{filename=\"/var/log/myproject.log\", job=\"varlogs\", level=\"info\"}", 1244 "entries": [ 1245 { 1246 "ts": "2019-06-06T19:25:41.972739Z", 1247 "line": "foo" 1248 }, 1249 { 1250 "ts": "2019-06-06T19:25:41.972722Z", 1251 "line": "bar" 1252 } 1253 ] 1254 } 1255 ], 1256 "stats": { 1257 ... 1258 } 1259 } 1260 ``` 1261 1262 ### `GET /api/prom/label/<name>/values` 1263 1264 > **WARNING**: `/api/prom/label/<name>/values` is DEPRECATED; use `/loki/api/v1/label/<name>/values` 1265 1266 `/api/prom/label/<name>/values` retrieves the list of known values for a given 1267 label within a given time span. It accepts the following query parameters in 1268 the URL: 1269 1270 - `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago. 1271 - `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now. 1272 1273 In microservices mode, `/api/prom/label/<name>/values` is exposed by the querier. 1274 1275 Response: 1276 1277 ``` 1278 { 1279 "values": [ 1280 <label value>, 1281 ... 1282 ] 1283 } 1284 ``` 1285 1286 #### Examples 1287 1288 ```bash 1289 $ curl -G -s "http://localhost:3100/api/prom/label/foo/values" | jq 1290 { 1291 "values": [ 1292 "cat", 1293 "dog", 1294 "axolotl" 1295 ] 1296 } 1297 ``` 1298 1299 ### `GET /api/prom/label` 1300 1301 > **WARNING**: `/api/prom/label` is DEPRECATED; use `/loki/api/v1/label` 1302 1303 `/api/prom/label` retrieves the list of known labels within a given time span. It 1304 accepts the following query parameters in the URL: 1305 1306 - `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago. 1307 - `end`: The end time for the query as a nanosecond Unix epoch. Defaults to now. 1308 1309 In microservices mode, `/api/prom/label` is exposed by the querier. 1310 1311 Response: 1312 1313 ``` 1314 { 1315 "values": [ 1316 <label string>, 1317 ... 1318 ] 1319 } 1320 ``` 1321 1322 #### Examples 1323 1324 ```bash 1325 $ curl -G -s "http://localhost:3100/api/prom/label" | jq 1326 { 1327 "values": [ 1328 "foo", 1329 "bar", 1330 "baz" 1331 ] 1332 } 1333 ``` 1334 1335 ### `POST /api/prom/push` 1336 1337 > **WARNING**: `/api/prom/push` is DEPRECATED; use `/loki/api/v1/push` 1338 > instead. 1339 1340 `/api/prom/push` is the endpoint used to send log entries to Loki. The default 1341 behavior is for the POST body to be a snappy-compressed protobuf message: 1342 1343 - [Protobuf definition](https://github.com/grafana/loki/tree/master/pkg/logproto/logproto.proto) 1344 - [Go client library](https://github.com/grafana/loki/tree/master/pkg/promtail/client/client.go) 1345 1346 Alternatively, if the `Content-Type` header is set to `application/json`, a 1347 JSON post body can be sent in the following format: 1348 1349 ``` 1350 { 1351 "streams": [ 1352 { 1353 "labels": "<LogQL label key-value pairs>", 1354 "entries": [ 1355 { 1356 "ts": "<RFC3339Nano string>", 1357 "line": "<log line>" 1358 } 1359 ] 1360 } 1361 ] 1362 } 1363 ``` 1364 1365 Loki can be configured to [accept out-of-order writes](../configuration/#accept-out-of-order-writes). 1366 1367 In microservices mode, `/api/prom/push` is exposed by the distributor. 1368 1369 #### Examples 1370 1371 ```bash 1372 $ curl -H "Content-Type: application/json" -XPOST -s "https://localhost:3100/api/prom/push" --data-raw \ 1373 '{"streams": [{ "labels": "{foo=\"bar\"}", "entries": [{ "ts": "2018-12-18T08:28:06.801064-04:00", "line": "fizzbuzz" }] }]}' 1374 ``` 1375 1376 ### `POST /ingester/flush_shutdown` 1377 1378 > **WARNING**: `/ingester/flush_shutdown` is DEPRECATED; use `/ingester/shutdown?flush=true` 1379 > instead. 1380 1381 `/ingester/flush_shutdown` triggers a shutdown of the ingester and notably will _always_ flush any in memory chunks it holds. 1382 This is helpful for scaling down WAL-enabled ingesters where we want to ensure old WAL directories are not orphaned, 1383 but instead flushed to our chunk backend. 1384 1385 In microservices mode, the `/ingester/flush_shutdown` endpoint is exposed by the ingester. 1386