github.com/pachyderm/pachyderm@v1.13.4/doc/docs/1.9.x/deploy-manage/deploy/tracing.md (about)

     1  # Configure Tracing with Jaeger
     2  
     3  Pachyderm has the ability to trace requests using Jaeger. This
     4  can be useful when diagnosing slow clusters.
     5  
     6  ![Successful Trace](../../assets/images/healthy.png)
     7  
     8  # Collecting Traces
     9  
    10  To use tracing in Pachyderm, complete the following steps:
    11  
    12  1. Run Jaeger in Kubernetes
    13  
    14      ```shell
    15      kubectl apply -f https://raw.githubusercontent.com/pachyderm/pachyderm/master/etc/deploy/tracing/jaeger-all-in-one.yaml
    16      ```
    17  
    18  2. Point Pachyderm at Jaeger
    19  
    20      ```shell
    21      # For pachctl
    22      $ export JAEGER_ENDPOINT=localhost:14268
    23      $ kubectl port-forward svc/jaeger-collector 14268 & # Collector service
    24  
    25  
    26      # For pachd
    27      $ kubectl delete po -l suite=pachyderm,app=pachd
    28      ```
    29      The port-forward command is necessary because `pachctl` sends traces to
    30      Jaeger (it actually initiates every trace), and reads the `JAEGER_ENDPOINT`
    31      environment variable for the address to which it will send the trace info.
    32  
    33      Restarting the `pachd` pod is necessary because `pachd` also sends trace
    34      information to Jaeger, but it reads the environment variables corresponding
    35      to the Jaeger service[1] on startup to find Jaeger (the Jaeger service is
    36      created by the `jaeger-all-in-one.yaml` manifest). Killing the pods
    37      restarts them, which causes them to connect to Jaeger.
    38  
    39  3. Send Pachyderm a traced request
    40  
    41      Just set the `PACH_TRACE` environment variable to "true" before
    42      running any `pachctl` command (note that `JAEGER_ENDPOINT` must also be
    43      set/exported):
    44      ```
    45      PACH_TRACE=true pachctl list job # for example
    46      ```
    47  
    48      We generally don't recommend exporting `PACH_TRACE` because
    49      tracing calls can slow them down somewhat and make interesting traces hard
    50      to find in Jaeger.  Therefore you may only want to set this variable for
    51      the specific calls you want to trace.
    52  
    53      However, Pachyderm's client library reads this variable and implements the
    54      relevant tracing, so any binary that uses Pachyderm's go client library can
    55      trace calls if these variables are set.
    56  
    57  ## View Traces
    58  
    59  To view traces, run:
    60  ```
    61  $ kubectl port-forward svc/jaeger-query 16686:80 & # UI service
    62  
    63  ```
    64  then connect to `localhost:16686` in your browser, and you should see all
    65  collected traces.
    66  
    67  !!! note "See Also:"
    68      [Kubernetes Service Environment Variables](https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables)
    69  
    70  ## Troubleshooting
    71  
    72  1. If you see `<trace-without-root-span>`, this likely means that `pachd` has
    73      connected to Jaeger, but `pachctl` has not. Make sure that the
    74      `JAEGER_ENDPOINT` environment variable is set on your local machine, and
    75      that `kubectl port-forward "po/${jaeger_pod}" 14268` is running.
    76  
    77  2. If you see a trace appear in Jaeger with no subtraces, like so:
    78  
    79      ![Trace with no children](../../assets/images/no-traces.png)
    80  
    81      ...this likely means that `pachd` has not connected to Jaeger, but
    82      `pachctl` has. Make sure to restart the `pachd` pods *after* creating the
    83      Jaeger service in Kubernetes