sigs.k8s.io/cluster-api-provider-azure@v1.14.3/hack/observability/opentelemetry/readme.md (about)

     1  # OpenTelemetry Tracing
     2  
     3  This directory contains resources to help deploy [opentelemetry-collector][1] and to export and view
     4  [traces][2] locally in the Jaeger UI or in the cloud with Azure Application Insights.
     5  
     6  When creating a management cluster for development with `make tilt-up`, OpenTelemetry tracing is
     7  automatically enabled.
     8  
     9  ## View traces in Jaeger
    10  
    11  `make tilt-up` installs a Jaeger "all-in-one" resource for viewing traces.
    12  
    13  **NOTE:** The Jaeger "all-in-one" component uses in-memory storage and is designed for testing.
    14  
    15  <!-- markdown-link-check-disable-next-line -->
    16  Visit http://localhost:16686/ or select the "traces: jaeger-all-in-one" resource in the Tilt UI and
    17  click on "View traces" near the top of the screen.
    18  
    19  ![Jaeger Trace UI](assets/jaeger_ui.png)
    20  
    21  ## View traces in Application Insights
    22  
    23  Traces can be exported to the App Insights resource that you specify
    24  with `AZURE_INSTRUMENTATION_KEY`.
    25  
    26  If you don't have an App Insights resource, create one through the Azure Portal
    27  at https://ms.portal.azure.com/#create/Microsoft.AppInsights.
    28  
    29  To the upper right of the "Overview" page for your App Insights resource is an entry
    30  for "Instrumentation Key".
    31  
    32  ![Instrumentation Key](assets/portal_instrumentation_key.png)
    33  
    34  Paste the Instrumentation Key from the portal as `AZURE_INSTRUMENTATION_KEY`
    35  in `tilt-settings.yaml`:
    36  
    37  ```yaml
    38  kustomize_substitutions:
    39    AZURE_INSTRUMENTATION_KEY: "12345678-1234-1234-1234-1234567890abc"
    40  ```
    41  
    42  Once your management cluster is up, search for recent traces on the "Transaction search" page.
    43  
    44  ![Transaction Search](assets/transaction_search.png)
    45  
    46  Click on "See all data in the last 24 hours" and choose a "Dependency" entry to follow the traces
    47  and timings of the CAPZ controller-controller-manager.
    48  
    49  ![Transaction Details](./assets/transaction_details.png)
    50  
    51  Traces may not show up immediately. Also, your management cluster needs to do work in order to
    52  produce traces, so click on a cluster flavor in the Tilt web interface to create a workload cluster.
    53  
    54  ## Add OpenTelemetry to an existing cluster
    55  
    56  **NOTE:** This tracing pipeline is evolving and should not be considered for production usage.
    57  
    58  To add these OpenTelemetry components and configuration to an existing management cluster, first
    59  install the Helm chart:
    60  
    61  ```shell
    62  helm install opentelemetry-collector ./hack/observability/opentelemetry/chart \
    63    --namespace capz-system --values ./hack/observability/opentelemetry/values.yaml \
    64    --set config.exporters.azuremonitor.instrumentation_key=$AZURE_INSTRUMENTATION_KEY
    65  ```
    66  
    67  Then enable tracing for the capz-controller-manager deployment:
    68  
    69  - Edit the deployment in the capz-system namespace
    70  - Under `spec.template.spec`, find the "manager" container
    71  - Add this line to `args`:
    72      ```yaml
    73      --enable-tracing
    74      ```
    75  - Save your changes
    76  
    77  Once the updated deployment creates a new capz-controller-manager pod, it will send traces to the
    78  "opentelemetry-collector" service on port 14268. The collector will then export the traces to the
    79  App Insights resource.
    80  
    81  ## Contents
    82  
    83  ```
    84  ./hack/observability/opentelemetry/
    85  ├── controller-manager-patch.yaml
    86  ├── fetch-otel-resources.yaml
    87  ├── readme.md
    88  ├── values.yaml
    89  └── chart
    90      ├── .helmignore
    91      ├── Chart.yaml
    92      ├── README.md
    93      ├── values.schema.json
    94      ├── values.yaml
    95      └── templates
    96          ├── _config.tpl
    97          ├── configmap-agent.yaml
    98          ├── deployment.yaml
    99          └── ... (and many others)
   100  ```
   101  
   102  ### CAPZ controller-manager patch
   103  
   104  `controller-manager-patch.yaml` enables tracing on the CAPZ controller-manager pod. It is applied
   105  automatically when doing development with `make tilt-up`.
   106  
   107  ### Helm chart values
   108  
   109  `values.yaml` has configuration values for the opetelemetry-collector Helm chart. It is applied
   110  automatically when doing development with `make tilt-up`.
   111  
   112  ### Helm chart
   113  
   114  The `chart/` directory contains the [opentelemetry-collector][3] Helm chart. We store a copy locally
   115  in order to use a known working version without needing to fetch it over the network.
   116  
   117  Together, the chart and values create a tracing pipeline from the management cluster to
   118  App Insights.
   119  
   120  ## Updating Resources
   121  
   122  To update the local Helm chart to a [newer release][4], such as "0.6.1", run
   123  `CHART_RELEASE=0.6.1 ./fetch-otel-resources.sh`. That script fetches the Helm chart and unpacks
   124  it to the `chart` directory.
   125  
   126  
   127  [1]: https://github.com/open-telemetry/opentelemetry-collector
   128  [2]: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#tracing-signal
   129  [3]: https://github.com/open-telemetry/opentelemetry-helm-charts
   130  [4]: https://github.com/open-telemetry/opentelemetry-helm-charts/releases