github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/production/nomad/loki-distributed/README.md (about)

     1  # Microservices mode
     2  
     3  This Nomad job will deploy Loki in
     4  [microservices mode](https://grafana.com/docs/loki/latest/fundamentals/architecture/deployment-modes/#microservices-mode)
     5  using boltdb-shipper and S3 backend.
     6  
     7  ## Usage
     8  
     9  Have a look at the job file and Loki configuration file and change it to suite
    10  your environment.
    11  
    12  ### Run job
    13  
    14  Inside directory with job run:
    15  
    16  ```shell
    17  nomad run job.nomad.hcl
    18  ```
    19  
    20  To deploy a different version change `variable.version` default value or
    21  specify from command line:
    22  
    23  ```shell
    24  nomad job run -var="version=2.6.1" job.nomad.hcl
    25  ```
    26  
    27  ### Scale Loki
    28  
    29  Change `count` in job file in `group "loki"` and run:
    30  
    31  ```shell
    32  nomad run job.nomad.hcl
    33  ```
    34  
    35  or use Nomad CLI
    36  
    37  ```shell
    38  nomad job scale loki distributor <count>
    39  ```
    40  
    41  ## Recommendations for running in production
    42  
    43  ### Gather metrics
    44  
    45  To collect metrics from all components use this config:
    46  
    47  ```yaml
    48  - job_name: "loki"
    49    consul_sd_configs:
    50      - services:
    51          - "loki-compactor"
    52          - "loki-ruler"
    53          - "loki-distributor"
    54          - "loki-ingestor"
    55          - "loki-querier"
    56          - "loki-index-gateway"
    57          - "loki-query-frontend"
    58          - "loki-query-scheduler"
    59    relabel_configs:
    60      - source_labels: ["__meta_consul_service_metadata_alloc_id"]
    61        target_label: "instance"
    62      - source_labels: ["__meta_consul_service_metadata_component"]
    63        target_label: "component"
    64  ```
    65  
    66  ### Secure HTTP endpoints with Consul Connect
    67  
    68  Set network to `bridge` mode and add `health` port, that will be used by Consul
    69  healthcheck:
    70  
    71  ```hcl
    72      network {
    73        mode = "bridge"
    74  
    75        port "http" {}
    76        port "health" {}
    77        port "grpc" {}
    78      }
    79  ```
    80  
    81  ```hcl
    82      task "distibutor" {
    83        driver       = "docker"
    84        user         = "nobody"
    85        kill_timeout = "90s"
    86  
    87        config {
    88          image = "grafana/loki:${var.versions.loki}"
    89          ports = [
    90            "http",
    91            "health", # do not forget to publish health port
    92            "grpc",
    93          ]
    94  ```
    95  
    96  Bind HTTP endpoint to `127.0.0.1:80` so it is not accessible from outside:
    97  
    98  ```yml
    99  server:
   100    http_listen_address: 127.0.0.1
   101    http_listen_port: 80
   102  ```
   103  
   104  Add service registration with Consul Connect enabled, `/metrics` and `/ready`
   105  endpoint [exposed](https://www.nomadproject.io/docs/job-specification/expose)
   106  and API accessible with basicauth through Traefik with Consul Connect
   107  integration:
   108  
   109  ```hcl
   110      service {
   111        name = "loki-distributor"
   112        port = "http"
   113  
   114        meta {
   115          alloc_id  = NOMAD_ALLOC_ID
   116          component = "distributor"
   117        }
   118  
   119        tags = [
   120          "traefik.enable=true",
   121          "traefik.consulcatalog.connect=true",
   122  
   123          "traefik.http.routers.loki-distributor.entrypoints=https",
   124          "traefik.http.routers.loki-distributor.rule=Host(`loki-distributor.service.consul`)",
   125          "traefik.http.middlewares.loki-distributor.basicauth.users=promtail:$$apr1$$wnih40yf$$vcxJYiqcEQLknQAZcpy/I1",
   126          "traefik.http.routers.loki-distirbutor.middlewares=loki-distributor@consulcatalog",
   127  
   128          "traefik.http.routers.loki-distributor-ring.entrypoints=https",
   129          "traefik.http.routers.loki-distributor-ring.rule=Host(`loki-distributor.service.consul`) && Path(`/distributor/ring`)",
   130          "traefik.http.middlewares.loki-distributor-ring.basicauth.users=devops:$apr1$bNIZL02A$QrOgT3NAOx.koXWnqfXbo0",
   131          "traefik.http.routers.loki-distributor-ring.middlewares=loki-distributor-ring@consulcatalog",
   132        ]
   133  
   134        check {
   135          name     = "Loki distibutor"
   136          port     = "health"
   137          type     = "http"
   138          path     = "/ready"
   139          interval = "20s"
   140          timeout  = "1s"
   141        }
   142  
   143        connect {
   144          sidecar_service {
   145            proxy {
   146              local_service_port = 80
   147  
   148              expose {
   149                path {
   150                  path            = "/metrics"
   151                  protocol        = "http"
   152                  local_path_port = 80
   153                  listener_port   = "http"
   154                }
   155  
   156                path {
   157                  path            = "/ready"
   158                  protocol        = "http"
   159                  local_path_port = 80
   160                  listener_port   = "health"
   161                }
   162              }
   163            }
   164          }
   165        }
   166      }
   167  ```
   168  
   169  ## Secure GRPC endpoints with mTLS
   170  
   171  Unfortenately Consul Connect cannot be used to secure GRPC communication between
   172  Loki components, since some components should be able to connect to all
   173  instances of other components. We can secure components GRPC communication with
   174  Vault [PKI engine](https://www.vaultproject.io/docs/secrets/pki).
   175  
   176  Certificate generation can be made less verbose with the following HCL trick:
   177  
   178  1. Add the following to `locals`:
   179  
   180  ```hcl
   181  locals {
   182    certs = {
   183      "CA"   = "issuing_ca",
   184      "cert" = "certificate",
   185      "key"  = "private_key",
   186    }
   187  }
   188  ```
   189  
   190  2. Add dynamic template per service:
   191  
   192  ```hcl
   193        dynamic "template" {
   194          for_each = local.certs
   195          content {
   196            data = <<-EOH
   197            {{- with secret "pki/issue/internal" "ttl=10d" "common_name=loki-<component_name>.service.consul" (env "attr.unique.network.ip-address" | printf "ip_sans=%s") -}}
   198            {{ .Data.${template.value} }}
   199            {{- end -}}
   200            EOH
   201  
   202            destination = "secrets/certs/${template.key}.pem"
   203            change_mode = "restart"
   204            splay       = "5m"
   205          }
   206        }
   207  ```
   208  
   209  3. Update config to use generated certificates