github.com/grafana/pyroscope@v1.18.0/examples/tracing/tempo/README.md (about) 1 # Span Profiles with Grafana Tempo and Pyroscope 2 3 The docker compose consists of: 4 - Ride share demo application 5 - Tempo 6 - Pyroscope 7 - Grafana 8 9 `rideshare` applications generate traces and profiling data that should be available in Grafana. 10 Pyroscope and Tempo datasources are provisioned automatically. 11 12 ### Build and run 13 14 The project can be run locally with the following commands: 15 16 ```shell 17 # Pull latest pyroscope and grafana images: 18 docker pull grafana/pyroscope:latest 19 docker pull grafana/grafana:latest 20 21 docker-compose up 22 ``` 23 24 Navigate to the [Explore page](http://localhost:3000/explore?schemaVersion=1&panes=%7B%22yM9%22:%7B%22datasource%22:%22tempo%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22tempo%22,%22uid%22:%22tempo%22%7D,%22queryType%22:%22traceqlSearch%22,%22limit%22:20,%22tableType%22:%22traces%22,%22filters%22:%5B%7B%22id%22:%22e73a615e%22,%22operator%22:%22%3D%22,%22scope%22:%22span%22%7D,%7B%22id%22:%22service-name%22,%22tag%22:%22service.name%22,%22operator%22:%22%3D%22,%22scope%22:%22resource%22,%22value%22:%5B%22ride-sharing-app%22%5D,%22valueType%22:%22string%22%7D%5D%7D%5D,%22range%22:%7B%22from%22:%22now-6h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1), select a trace and click on one of its spans 25 that have a linked profile: 26 27  28 29 By default, only the root span gets labeled (the first span created locally): such spans are marked with the _link_ icon 30 and have `pyroscope.profile.id` attribute set to the corresponding span ID. 31 Please note that presence of the attribute does not necessarily 32 indicate that the span has a profile: stack trace samples might not be collected, if the utilized CPU time is 33 less than the sample interval (10ms). 34 35 ### Instrumentation 36 37 - `rideshare` demo application instrumented with OpenTelemetry: 38 - Go [OTel integration](https://github.com/grafana/otel-profiling-go) 39 - Java [OTel integration](https://github.com/grafana/otel-profiling-java) 40 - `pyroscope` itself is instrumented with `opentracing-go` SDK and [`spanprofiler`](https://github.com/grafana/dskit/tree/main/spanprofiler) for profiling integration. 41 42 ### Grafana Tempo configuration 43 44 In order to correlate trace spans with profiling data, Tempo datasource should be configured: 45 - Data source of the profiling data. 46 - Tags to use in the query. 47 - Profile type: as of now, only CPU time profile is fully supported. 48 - Query override. 49 50  51 52 While tags are optional, configuring them is highly recommended for optimizing query performance. 53 In our example, we configured the `host.name` tag for use in Pyroscope queries as the `hostname` label. 54 This configuration restricts the data set for lookup to a specific host, ensuring that queries remain 55 consistently fast. Note that the tags you configure must be present in the spans attributes or resources 56 for a trace to profiles span link to appear. 57 58 Please refer to our [documentation](https://grafana.com/docs/grafana/next/datasources/tempo/configure-tempo-data-source/#trace-to-profiles) for more details.