This section describes how to use Grafana Tempo for WizTelemetry (referred to as Tempo) to visualize the distributed tracing data from WizTelemetry Tracing in the Grafana console, i.e., to visualize the complete flow of a user request through the distributed system.

Prerequisites

A Grafana console has been deployed via the Grafana for WizTelemetry extension.

Steps

  1. Install the Grafana Tempo for WizTelemetry extension into the cluster.

  2. Enter the cluster where Tempo is installed. On the Application Workloads > Services page, find the service tempo-distributor, click its name to enter the details page. Click Operations > Edit External Access, enable NodePort. Then, on this page, obtain the NodePort corresponding to service port 4317.

    tempo distributor nodeport

    Note

    Depending on your network environment, you may need to configure traffic forwarding rules and allow this NodePort port in the firewall.

  3. Install the WizTelemetry Tracing extension, and modify the extension configuration to send trace data to Tempo.

    collector:
  collector:
    config: |
      exporters:
        otlp:
          # tempo-distributor.wiz-telemetry-tracing:4317
          endpoint: < tempo distributor grpc endpoint >
          headers:
            x-scope-orgid: wiztelemetry-tracing-ks
          tls:
            insecure: true
      service:
        pipelines:
          traces:
            exporters:
              - otlp
              - otlphttp

    Note: Replace < tempo distributor grpc endpoint > with <cluster-node-ip>:<grpc nodeport>. For example, based on the NodePort obtained in the previous step, it should be modified to <cluster-node-ip>:31459.

  4. Configure the Tempo data source in the Grafana console.

    If the Tempo and Grafana for WizTelemetry extensions are deployed in the same cluster, after Tempo installation is complete, the Tempo data source will be automatically added to the Grafana console.

    tempo data source

    If the Tempo and Grafana for WizTelemetry extensions are deployed in different clusters, you need to expose the service via NodePort or LoadBalancer tempo-query-frontend to obtain the server address for the Tempo data source, and then manually add the Tempo data source in the Grafana console. Detailed steps are as follows:

    • Expose the tempo-query-frontend service

      The following uses the NodePort method to expose the tempo-query-frontend service.

      1. Enter the cluster where Tempo is installed. On the Application Workloads > Services page, find the service tempo-query-frontend, click its name to enter the details page.

      2. Click Operations > Edit External Access, enable NodePort. Then, on this page, obtain the NodePort corresponding to service port 3100.

      tempo query frontend nodeport

    • Manually add the Tempo data source

      1. After logging into the Grafana console, click Connections in the left navigation pane.

      2. Search for a data source, such as Tempo.

      3. Click the data source name to enter the data source overview page.

      4. Click Add new data source in the upper right corner to enter the data source configuration page.

      5. Enter the server address for the Tempo data source: <cluster-node-ip>:<tempo-query-frontend-nodeport>. For example, based on the NodePort obtained in the previous step, you should enter <cluster-node-ip>:30835.

        url

      6. Add a header. x-scope-orgid: wiztelemetry-tracing-ks

        http header

      7. Set the Prometheus data source for Service graph.

        prometheus

      8. Click Save & test at the bottom to complete adding the data source.

  5. In the Grafana console, click Explore in the left navigation pane to enter the page, and select the Tempo data source.

  6. Click the TraceQL tab, then click the refresh-light in the upper right corner to run the query and list the latest trace data. Click one of them to view the trace graph.

    trace tab

  7. Click the Service Graph tab, then click the refresh-light in the upper right corner to run the query and view the service topology graph generated by Tempo.

    graph

    Note

    For more information about Tempo, please refer to the Grafana Tempo official documentation.