github.com/grafana/synthetic-monitoring-agent@v0.64.2/docs/architecture/index.md (about)

     1  ---
     2  component: index
     3  source_paths:
     4    - .
     5  last_reviewed_commit: f77bec5e
     6  generated: 2026-05-23
     7  ---
     8  
     9  # Synthetic Monitoring agent — architecture documentation
    10  
    11  This directory describes how the Synthetic Monitoring agent is put together.
    12  Read it once to build a mental model; come back to specific component docs
    13  when you change code in that area.
    14  
    15  ## What the agent does
    16  
    17  The agent is a distributed probe. It connects to a central API server,
    18  receives a list of checks to run (HTTP, DNS, TCP, ICMP, gRPC, Traceroute,
    19  and k6-based scripted/browser/multihttp checks), executes them on schedule,
    20  and publishes results as Prometheus metrics and Loki log lines.
    21  
    22  ## Top-level data flow
    23  
    24  ```mermaid
    25  flowchart LR
    26      API["API server<br/>(gRPC)"]
    27      Updater["Updater<br/>internal/checks"]
    28      Adhoc["Adhoc handler<br/>internal/adhoc"]
    29      Scrapers["Scrapers<br/>internal/scraper"]
    30      Probers["Probers<br/>internal/prober"]
    31      K6["k6 runner<br/>internal/k6runner"]
    32      Publisher["Publisher<br/>internal/pusher"]
    33      Prom[("Prometheus<br/>(remote_write)")]
    34      Loki[("Loki<br/>(push)")]
    35  
    36      API -->|GetChanges stream| Updater
    37      API -->|GetAdHocChecks stream| Adhoc
    38      Updater -->|spawn / tear down| Scrapers
    39      Scrapers -->|Probe| Probers
    40      Probers -.->|scripted / browser / multihttp| K6
    41      Scrapers -->|metrics + logs| Publisher
    42      Adhoc -->|results as logs| Publisher
    43      Publisher --> Prom
    44      Publisher --> Loki
    45  ```
    46  
    47  ## Reader's guide
    48  
    49  | If you want to…                                          | Start with                                  |
    50  | -------------------------------------------------------- | ------------------------------------------- |
    51  | Understand startup, flags, signals, dependency wiring    | [cmd.md](cmd.md)  |
    52  | Trace how a check arrives from the API and starts running | [updater.md](updater.md) |
    53  | Understand how a single check gets executed on schedule  | [scraper.md](scraper.md) |
    54  | Add a new check type or modify an existing prober        | [prober.md](prober.md) |
    55  | Touch scripted / browser / multihttp execution           | [k6runner.md](k6runner.md) |
    56  | Modify how metrics and logs are shipped out              | [publisher.md](publisher.md) |
    57  | Work on the "test this check now" UI flow                | [adhoc.md](adhoc.md) |
    58  
    59  ## Component index
    60  
    61  - **[Glue / bootstrap](cmd.md)** — `cmd/synthetic-monitoring-agent/`. Wires every other component together; owns flag parsing, the HTTP server, the gRPC connection, signal handling.
    62  - **[Updater](updater.md)** — `internal/checks`. Holds the long-lived `GetChanges()` stream; owns the lifecycle of every scraper.
    63  - **[Scraper](scraper.md)** — `internal/scraper`. One per active check; runs the prober on schedule, decorates output, manages metric lifecycle.
    64  - **[Prober](prober.md)** — `internal/prober`. Per-check-type implementations (HTTP, DNS, TCP, ICMP, gRPC, Traceroute, plus k6-backed scripted/browser/multihttp).
    65  - **[k6 runner](k6runner.md)** — `internal/k6runner`. Runs k6 scripts either as a local subprocess or via a remote HTTP runner.
    66  - **[Publisher](publisher.md)** — `internal/pusher`. Per-tenant push handlers; batches and ships to Prometheus and Loki.
    67  - **[Adhoc handler](adhoc.md)** — `internal/adhoc`. Separate gRPC stream for on-demand "test this check now" runs.
    68  
    69  ## Supporting components (follow-up docs)
    70  
    71  These packages are real architectural components but are not documented in
    72  this initial pass. They each warrant a dedicated doc later.
    73  
    74  - `internal/tenants` — Tenant metadata cache (TTL with jitter; per-tenant locks). Provides auth context to the publisher. *TODO: dedicated doc.*
    75  - `internal/secrets` — Remote secret store integration; fetches credentials for checks. *TODO: dedicated doc.*
    76  - `internal/limits` — Per-tenant quota and label-cardinality enforcement. *TODO: dedicated doc.*
    77  - `internal/cache` — Pluggable cache (memcached / local / no-op) used to shed load from upstream APIs. *TODO: dedicated doc.*
    78  - `internal/telemetry` — Region-level telemetry pushed to an internal backend. *TODO: dedicated doc.*
    79  - `internal/usage` — Anonymous usage reporting to `stats.grafana.com`. *TODO: dedicated doc.*
    80  - `internal/metamonitoring` — Publishes the agent's own internal metrics as a synthetic check. *TODO: dedicated doc.*
    81  - `internal/k6version` — k6 version resolution and caching, used by k6runner. *TODO: dedicated doc (or fold into k6runner).*
    82  - `internal/feature` — Feature-flag plumbing. *Likely too small for a dedicated doc; revisit.*
    83  - `internal/cals` — Cost-attribution labels for Loki streams. *Likely too small for a dedicated doc; revisit.*
    84  
    85  ## Other binaries
    86  
    87  - `cmd/synthetic-monitoring-proto` — Small utility for manipulating serialized check protobufs. Not part of the running agent.
    88  - `cmd/test-api` — Mock gRPC server used during development. Not part of the running agent.
    89  
    90  ## Cross-cutting concerns
    91  
    92  ### Error classification
    93  
    94  Two custom error types in `internal/error_types` control how the agent
    95  reacts to failures:
    96  
    97  - `FatalError` — stop the affected component (e.g. unauthorised probe, incompatible API).
    98  - `TransientError` — retry with back-off (e.g. transport closing, gRPC `Unavailable`).
    99  
   100  Wrapping is done with `fmt.Errorf("...: %w", err)` and unwrapping with
   101  `errors.As` / `errors.Is`. The Updater's `handleError`
   102  (`internal/checks/checks.go`) is the canonical switch — copy that shape
   103  when adding a new long-lived loop.
   104  
   105  ### Tenant isolation
   106  
   107  Every payload is tagged with a tenant ID. The publisher maintains separate
   108  push handlers per tenant. Secrets are stored remotely, not in the agent.
   109  See `internal/tenants` and `internal/secrets`.
   110  
   111  ### Metric lifecycle
   112  
   113  The agent goes out of its way to give Prometheus a clean view of check
   114  results:
   115  
   116  - **Stale markers** — when a scraper shuts down, it emits NaN values so Prometheus knows the series ended.
   117  - **Republishing** — for checks with an interval longer than ~2 minutes, the scraper resends the most recent values with updated timestamps so the series doesn't go stale between probes.
   118  - **`sm_check_info`** — a special metric carrying check metadata (geohash, frequency, alert sensitivity) for use in joins.
   119  
   120  See [scraper.md](scraper.md) for the details.
   121  
   122  ### Signals
   123  
   124  - `SIGINT` / `SIGTERM` — graceful shutdown.
   125  - `SIGUSR1` (or `POST /disconnect`) — disconnect from the API but keep running checks; allows another agent to take over for zero-downtime upgrades. If no other agent connects within ~1 minute, the disconnected agent reconnects.
   126  
   127  Implemented in `cmd/synthetic-monitoring-agent/main.go` and
   128  `cmd/synthetic-monitoring-agent/http.go`. See
   129  [cmd.md](cmd.md).
   130  
   131  ### Testing
   132  
   133  Each component doc carries a **Testing strategy** section. A few
   134  cross-cutting points worth knowing up front:
   135  
   136  - **Run everything**: `make test` (Docker, race detector enabled, `CGO_ENABLED=1`, coverage).
   137  - **Fast subset**: `make test-fast` (adds `-short`, skips slower integration / timer-based tests).
   138  - **One package**: `make test-go GO_TEST_ARGS=./internal/<pkg>/...`. Hand it any `go test` flags via `GO_TEST_ARGS`.
   139  - **Mocks are rare**. The conventions are dependency injection with real types, in-process `httptest.Server` for HTTP endpoints, and an in-process gRPC server when one is needed. Where mocks exist they're named `Mock*` or `Fake*`.
   140  - **Test helpers** live in `internal/testhelper` — `Context`, `Logger`, `K6Paths`, mock secret stores, etc. Use these instead of rolling your own.
   141  - **Golden files**: `internal/scraper/testdata/*.txt` holds end-to-end Prometheus output expectations per check type (regenerate with `-update-golden`); `internal/k6runner/testdata/` holds a fake k6 binary plus golden script/output/log pairs.
   142  - **Code-generation discrepancy tests**: `pkg/pb/synthetic_monitoring/multihttp_string_test.go` pins the protobuf enum ↔ string mapping, and `pkg/accounting/accounting_test.go` ensures every check type has an accounting entry.
   143  - **Linting**: `make lint` runs `golangci-lint` v2 in Docker, plus `go vet` and `shellcheck`. Custom rules live in `internal/rules/rules.go` (ruleguard).
   144  
   145  ## Keeping these docs current
   146  
   147  Each component doc ends with a **When to update this doc** checklist. Use
   148  that as the source of truth when you change code. The table below is a
   149  quick index from "I changed code under X" to "I should update doc Y".
   150  
   151  | If you change…                                          | Update                              |
   152  | ------------------------------------------------------- | ----------------------------------- |
   153  | `cmd/synthetic-monitoring-agent/*`                      | `cmd.md`               |
   154  | `internal/checks/*`                                     | `updater.md`           |
   155  | `internal/scraper/*`                                    | `scraper.md`           |
   156  | `internal/prober/*` (any subdir)                        | `prober.md`            |
   157  | `internal/k6runner/*`, `internal/k6version/*`           | `k6runner.md`          |
   158  | `internal/pusher/*`, `internal/pkg/{prom,loki}/*`       | `publisher.md`         |
   159  | `internal/adhoc/*`                                      | `adhoc.md`             |
   160  | Top-level data flow or any new cross-component pathway  | this file                           |
   161  | Add a new gRPC RPC                                      | the doc for the component that owns it |
   162  | Add a new check type                                    | `prober.md` (+ `k6runner.md` if k6-backed) |
   163  
   164  If you find yourself changing code that isn't well-represented in any doc,
   165  either (a) extend the relevant doc, or (b) move one of the *follow-up*
   166  components above into its own dedicated doc.