github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/docs/sources/design-documents/2020-02-Promtail-Push-API.md (about)

     1  ---
     2  title: Promtail Push API
     3  weight: 20
     4  ---
     5  # Promtail Push API
     6  
     7  - Author: Robert Fratto (@rfratto)
     8  - Date: Feb 4 2020
     9  - Status: DRAFT
    10  
    11  Despite being an optional piece of software, Promtail provides half the power
    12  of Loki's story: log transformations, service discovery, metrics from logs,
    13  and context switching between your existing metrics and logs. Today, Promtail
    14  can only be operated to consume logs from very specific sources: files, journal,
    15  or syslog. If users wanted to write custom tooling to ship logs, the tooling
    16  has to bypass Promtail and push directly to Loki. This can lead users to
    17  reimplement functionality Promtail already provides, including its error retries
    18  and batching code.
    19  
    20  This document proposed a Push API for Promtail. The preferred implementation is
    21  by copying the existing Loki Push API and implementing it for Promtail. By
    22  being compatible with the Loki Push API, the Promtail Push API can allow batches
    23  of logs to be processed at once for optimizing performance. Matching the
    24  Promtail API also allows users to transparently switch their push URLs from
    25  their existing tooling. Finally, a series of alternative solutions will be detailed.
    26  
    27  ## Configuration
    28  
    29  Promtail will have a new target called HTTPTarget, configurable in the
    30  `scrape_config` array with the following schema:
    31  
    32  ```yaml
    33  # Defines an HTTP target, which exposes an endpoint against the Promtail
    34  # HTTP server to accept log traffic.
    35  http:
    36    # Defines the base URL for the push path, adding a prefix to the
    37    # exposed endpoint. The final endpoint path is
    38    # <base_url>loki/api/v1/push. If omitted, defaults to /.
    39    #
    40    # Multiple http targets with the same base_url must not exist.
    41    base_url: /
    42  
    43    # Map of labels to add to every log line passed through to the target.
    44    labels: {}
    45  ```
    46  
    47  ### Considerations
    48  
    49  Users will be able to define multiple `http` scrape configs, but the base URL
    50  value must be different for each instance. This allows to cleanly separate
    51  pipelines through different push endpoints.
    52  
    53  Users must also be aware about problems with running Promtail with an HTTP
    54  target behind a load balancer: if payloads are load balanced between multiple
    55  Promtail instances, ordering of logs in Loki will be disrupted leading to
    56  rejected pushes. Users are recommended to do one of the following:
    57  
    58  1. Have a dedicated Promtail instance for receiving pushes. This also applies to
    59     using the syslog target.
    60  1. Have a separated k8s service that always resolves to the same Promtail pod,
    61     bypassing the load balancing issue.
    62  
    63  ## Implementation
    64  
    65  As discussed in this document, this feature will be implemented by copying the
    66  existing [Loki Push API](https://grafana.com/docs/loki/latest/api/#post-lokiapiv1push)
    67  and exposing it via Promtail.
    68  
    69  ## Considered Alternatives
    70  
    71  Using the existing API was chosen for its simplicity and capabilities of being
    72  used for interesting configurations (e.g., chaining Promtails together). These
    73  other options were considered but rejected as not the best solution for the
    74  problem being solved.
    75  
    76  Note that Option 3 has value and may be implemented separately from this
    77  feature.
    78  
    79  ### Option 1: JSON / Protobuf Payload
    80  
    81  A new JSON and Protobuf payload format can be designed instead of the existing
    82  Loki push payload. Both formats would have to be exposed to support clients that
    83  either can't or won't use protobuf marshalling.
    84  
    85  The primary benefit of this approach is to allow us to tweak the payload schema
    86  independently of Loki's existing schema, but otherwise may not be very useful
    87  and is essentially just code duplication.
    88  
    89  ### Option 2: gRPC Service
    90  
    91  The
    92  [logproto.Pusher](https://github.com/grafana/loki/blob/f7ee1c753c76ef63338d53cfba782188a165144d/pkg/logproto/logproto.proto#L8-L10)
    93  service could be exposed through Promtail. This would enable clients stubs to be
    94  generated for languages that have gRPC support, and, for HTTP1 support, a
    95  [gRPC Gateway](https://github.com/grpc-ecosystem/grpc-gateway) would be embedded
    96  in Promtail itself.
    97  
    98  This implementation option is similar to the original proposed solution, but
    99  uses the gRPC gateway to handle HTTP1 traffic instead of the HTTP1 shim that
   100  Loki uses. There are some concerns with this approach:
   101  
   102  1. The gRPC Gateway reverse proxy will need to play nice with the existing HTTP
   103     mux used in Promtail.
   104  1. We couldn't control the HTTP and Protobuf formats separately as Loki can.
   105  1. Log lines will be double-encoded thanks to the reverse proxy.
   106  1. A small overhead of using a reverse proxy in-process will be introduced.
   107  1. This breaks our normal pattern of writing our own shim functions; may add
   108     some cognitive overhead of having to deal with the gRPC gateway as an outlier
   109     in the code.
   110  
   111  ### Option 3: Plaintext Payload
   112  
   113  Prometheus' [Push Gateway API](https://github.com/prometheus/pushgateway#command-line)
   114  is cleverly designed and we should consider implementing our API in the same
   115  format: users would push to `http://promtail-url/push/label1/value1?timestamp=now`
   116  with a plaintext POST body. For example:
   117  
   118  ```
   119  curl -X POST http://promtail.default/push/foo/bar/fizz/buzz -d “hello, world!”
   120  ```
   121  
   122  This approach may be slightly faster when compared to a non-plaintext payload as
   123  no unmarshaling needs to be performed. This URL path and timestamp still needs
   124  to be parsed, but this will generally be faster than the reflection requirements
   125  imposed by JSON.
   126  
   127  However, note that this API limits Promtail to accepting one line at a time and
   128  may cause performance issues when trying to handle large volumes of traffic. As
   129  an alternative, this API could also be implemented by external tooling and be
   130  built on top of any of the other implementation options.
   131  
   132  An [example implementation](https://github.com/grafana/loki/pull/1270) was
   133  created and has received positive support for its simplicity and ease of
   134  integration.
   135