github.com/pachyderm/pachyderm@v1.13.4/doc/docs/master/deploy-manage/deploy/loki.md (about)

     1  !!! note 
     2      To deploy and configure a Pachyderm cluster
     3      to ship logs to Loki, 
     4      a ***Pachyderm Enterprise License*** is required. 
     5  
     6  # Enabling Loki
     7  
     8  Enabling Loki logging will require to set the following [environment variables](https://docs.pachyderm.com/latest/deploy-manage/deploy/environment-variables/) on the pachd container:
     9  
    10  - `LOKI_LOGGING` to `true`  to ship the logs to Loki
    11  - `LOKI_SERVICE_HOST` and `LOKI_SERVICE_PORT` to fetch the logs from Loki
    12  
    13  ## Shipping logs to Loki
    14  
    15  Loki retrieves logs from pods in Kubernetes through
    16  an agent service called **Promtail**. 
    17  **Promtail** runs on each node and
    18  sends logs from Kubernetes pods to the Loki API Server,
    19  tagging each log entry with information
    20  about the pod that produced it. 
    21  
    22  You need to [configure Promtail](https://grafana.com/docs/loki/latest/clients/promtail/configuration/) for your environment
    23  to ship logs to your Loki instance. 
    24  If you are running **multiple nodes**, 
    25  then you will need to install and configure Promtail
    26  **for each node** shipping logs to Loki.
    27  
    28  
    29  ## Fetching logs
    30  
    31  While enabling Loki will enable the collection of logs, commands such as `pachctl logs` will not fetch logs directly from Loki until the `LOKI_SERVICE_HOST` and `LOKI_SERVICE_PORT` environment variables are set on the `pachd` container.
    32  
    33  For example, a `deployment.json` generated with 
    34  ```shell
    35      pachctl deploy local --dry-run > deployment.json`     
    36  ```
    37  can be modified to make logs available for Loki as follows:
    38  
    39  ```json
    40  {
    41      "containers": [{
    42          "name": "pachd",
    43          "image": "pachyderm/pachd:local",
    44          "command": ["/pachd"],
    45          "ports": [],
    46          "env": [
    47              {
    48                  "name": "LOKI_LOGGING",
    49                  "value": "true"
    50              },
    51              {
    52                  "name": "LOKI_SERVICE_HOST",
    53                  "value": "10.107.254.102"
    54              },
    55              {
    56                  "name": "LOKI_SERVICE_PORT",
    57                  "value": "3100"
    58              }
    59          ]
    60      }]
    61  }
    62  ```
    63  
    64  Pachyderm reads logs from the Loki API Server with a particular set of tags. The URI at which Pachyderm reads from the Loki API Server is set by the `LOKI_SERVICE_HOST` and `LOKI_SERVICE_PORT` values.
    65  
    66  !!! note 
    67      If you are not running Promtail on the node 
    68      where your Pachyderm pods are located, you
    69      will be unable to get logs for pipelines running
    70      on that node via `pachctl logs -p pipelineName`.
    71  
    72  ## References
    73  
    74  * Loki Documentation - https://grafana.com/docs/loki/latest/
    75  * Promtail Documentation - https://grafana.com/docs/loki/latest/clients/promtail/
    76  * Operating Loki - https://grafana.com/docs/loki/latest/operations/