github.com/grafana/synthetic-monitoring-agent@v0.64.2/docs/architecture/adhoc.md (about)

     1  ---
     2  component: Ad-hoc handler
     3  source_paths:
     4    - internal/adhoc/
     5  last_reviewed_commit: f77bec5e
     6  generated: 2026-05-23
     7  ---
     8  
     9  # Ad-hoc handler — `internal/adhoc`
    10  
    11  ## Purpose
    12  
    13  The ad-hoc handler powers the **"test this check now"** flow from the
    14  UI. A separate gRPC stream (`GetAdHocChecks`) carries one-shot probe
    15  requests; the handler runs each request once, captures the output as a
    16  single Loki log entry, and forwards it to the publisher. No
    17  scheduling, no retry, no scraper, no metric publishing.
    18  
    19  Where the [Updater](updater.md) is about *managing
    20  recurring checks*, the ad-hoc handler is about *executing single
    21  checks on demand*.
    22  
    23  ## Where it lives
    24  
    25  `internal/adhoc/`
    26  
    27  | File             | Responsibility                                          |
    28  | ---------------- | ------------------------------------------------------- |
    29  | `adhoc.go`       | `Handler`, `runner`, gRPC loop, `defaultRunnerFactory`. |
    30  | `adhoc_test.go`  | Mocked-stream unit tests.                               |
    31  
    32  ## How it fits in
    33  
    34  ```mermaid
    35  flowchart LR
    36      API["API server"]
    37      Handler["adhoc.Handler"]
    38      Prober
    39      K6Runner
    40      Publisher
    41  
    42      API -->|GetAdHocChecks stream| Handler
    43      API -->|RegisterProbe| Handler
    44      Handler -->|ProberFactory.New| Prober
    45      Prober -.->|scripted/browser/multihttp| K6Runner
    46      Handler -->|results as one log entry| Publisher
    47      Handler -->|tenants embedded in request| TenantCh[(tenantCh)]
    48  ```
    49  
    50  ## Lifecycle
    51  
    52  ### Construction
    53  
    54  `NewHandler(HandlerOpts)` is called from
    55  `cmd/synthetic-monitoring-agent/main.go`. The options are deliberately
    56  a subset of the Updater's: gRPC connection, logger, back-off,
    57  publisher, tenant channel, Prometheus registerer, features, k6 runner,
    58  secret provider, `SupportsProtocolSecrets`.
    59  
    60  The handler builds its own `prober.ProberFactory` with **`probeId = 0`**,
    61  intentionally — ad-hoc runs must not inject the `x-sm-id` request
    62  header, which the factory only adds when `probeId != 0`. The
    63  `ProberFactory` is built once in `NewHandler` and never updated, even
    64  after `RegisterProbe` returns a probe identity.
    65  
    66  The `runnerFactory` and `grpcAdhocChecksClientFactory` option fields
    67  are *unexported* — they exist purely to let tests inject fakes
    68  without exposing the seam in the public API.
    69  
    70  ### Run loop
    71  
    72  `Handler.Run(ctx)`:
    73  
    74  ```mermaid
    75  sequenceDiagram
    76      participant Run as Handler.Run
    77      participant Loop as Handler.loop
    78      participant API as API server
    79      participant Proc as processAdHocChecks
    80      participant Runner as runner.Run
    81  
    82      Run->>Loop: loop()
    83      Loop->>API: RegisterProbe (ad-hoc)
    84      Loop->>API: GetAdHocChecks (stream)
    85      Loop->>Proc: processAdHocChecks
    86      loop request received
    87          Proc->>Runner: go runner.Run (one-shot)
    88          Runner->>Publisher: Publish(adhocData{logs only})
    89      end
    90      Note over Loop: stream ends / error
    91      Loop-->>Run: classified error
    92      alt errIdleTimeout
    93          Run->>Run: reset backoff, sleep, retry
    94      else errTransportClosing
    95          Run->>Run: reset backoff, retry
    96      else errProbeUnregistered
    97          Run->>Run: sleep 1 minute, retry
    98      else fatal (auth / incompatible)
    99          Run-->>Run: return err
   100      end
   101  ```
   102  
   103  `Run`'s outer loop classifies errors and decides whether to retry,
   104  exit, or sleep. Notable differences from the Updater's error handling:
   105  
   106  - `codes.Unavailable` → `errIdleTimeout` rather than a generic transient error. Ad-hoc streams are expected to idle when no one is using the UI; the handler reconnects without alarm.
   107  - `errProbeUnregistered` triggers a fixed **1-minute sleep** before retrying — there is no scenario in which retrying faster helps.
   108  - Fatal codes (`PermissionDenied`, `Unimplemented`) cause `Run` to exit, which under the `errgroup` in `cmd/` brings the whole agent down (intentional: a misconfigured ad-hoc handler is a misconfigured agent).
   109  
   110  The default back-off is a `constantBackoff(60 * time.Second)` — set
   111  inside `NewHandler` when no `Backoff` is supplied. In practice
   112  `cmd/` supplies the same exponential back-off as the Updater.
   113  
   114  ### Per-request handling
   115  
   116  `processAdHocChecks` consumes the stream. Each `AdHocRequest` becomes:
   117  
   118  1. `defaultRunnerFactory(ctx, req)` builds a `runner` — constructs a one-off `model.Check` from the request, calls the prober factory, and computes a per-check timeout. For k6-backed types the timeout adds **`k6AdhocGraceTime` (20 s)** to keep the client and remote runner from timing out at the same moment.
   119  2. `go runner.Run(ctx, tenantID, publisher)` runs the probe in a *new* goroutine and returns immediately so the stream consumer doesn't block.
   120  3. If the request carried a tenant snapshot (`ahReq.Tenant != nil`), it's forwarded to `tenantCh` so the shared tenant manager learns about it.
   121  
   122  `runner.Run`:
   123  
   124  - Creates a fresh `prometheus.Registry` and registers two gauges (`probe_success`, `probe_duration_seconds`).
   125  - Captures the probe's logger output via the in-package `jsonLogger` (`adhoc.go`) — a `logger.Logger`-shaped sink that collects `[]map[string]string`.
   126  - Calls `prober.Probe(ctxWithTimeout, target, registry, jsonLogger)`.
   127  - Gathers metric families and serialises a **single** zerolog warn-level JSON message containing `id`, `target`, `probe`, `check_name`, the collected `logs`, and the gathered `timeseries`.
   128  - Wraps that JSON line in an `adhocData{streams: [...]}` and calls `publisher.Publish`. `adhocData.Metrics()` always returns `nil` — ad-hoc results never go through remote-write, only Loki push.
   129  
   130  The fixed Loki label set is:
   131  `{probe="<name>", source="synthetic-monitoring", type="adhoc"}`
   132  
   133  Operators use the `type="adhoc"` label to filter ad-hoc results out of
   134  normal alerting.
   135  
   136  ### Shutdown
   137  
   138  Parent context cancellation unwinds the stream consumer and the run
   139  loop. Outstanding `runner.Run` goroutines hold no scraper state and
   140  end naturally when their context is cancelled. There is no graceful
   141  "drain" — ad-hoc results are best-effort.
   142  
   143  ## Differences from the Updater
   144  
   145  | Aspect                       | Updater                                | Adhoc                                              |
   146  | ---------------------------- | -------------------------------------- | -------------------------------------------------- |
   147  | Stream RPC                   | `GetChanges`                           | `GetAdHocChecks`                                   |
   148  | Scheduling                   | Per-check `Scraper.Run` ticker         | None — single probe call per request.              |
   149  | Retry                        | Reconnect-only; checks rerun on schedule | None — failed probes are reported as failed.    |
   150  | Result destination           | Prometheus + Loki via Publisher        | Loki only (one JSON log line per run).             |
   151  | Probe identity               | `probeId` injected into HTTP headers   | `probeId = 0`; no header injection.                |
   152  | State held between runs      | Scrapers, payload memory               | None — every request is fresh.                    |
   153  | Failure of one request       | Logged, scraper continues              | Logged via `processAdHocChecks`; stream continues. |
   154  | Tenant changes from API      | Forwarded via `tenantCh`               | Same — embedded in request when present.          |
   155  
   156  The shared resources are the gRPC connection, the publisher, the
   157  prober factory, the secret provider, and the k6 runner. None of
   158  those are duplicated.
   159  
   160  ## Key types and entry points
   161  
   162  | Type / function                       | File         | Notes                                                 |
   163  | ------------------------------------- | ------------ | ----------------------------------------------------- |
   164  | `Handler`                             | `adhoc.go`   | The struct.                                           |
   165  | `HandlerOpts`                         | `adhoc.go`   | Construction inputs from `cmd/`.                      |
   166  | `NewHandler(opts)`                    | `adhoc.go`   | Registers `sm_adhoc_ops_total`, wires defaults.       |
   167  | `(*Handler).Run(ctx)`                 | `adhoc.go`   | Reconnect/back-off loop, error classification.        |
   168  | `(*Handler).loop(ctx)`                | `adhoc.go`   | One full connected session.                           |
   169  | `processAdHocChecks(ctx, client)`     | `adhoc.go`   | Stream consumer.                                      |
   170  | `(*Handler).handleAdHocCheck(...)`    | `adhoc.go`   | One request → one goroutine.                          |
   171  | `defaultRunnerFactory(...)`           | `adhoc.go`   | Builds the per-request `runner`.                      |
   172  | `runner.Run(ctx, tenantID, publisher)`| `adhoc.go`   | Executes the probe; emits one log entry.              |
   173  | `jsonLogger`                          | `adhoc.go`   | In-memory `logger.Logger` for capturing probe logs.   |
   174  | `adhocData`                           | `adhoc.go`   | `pusher.Payload` impl: streams only, no time series.  |
   175  
   176  The single Prometheus metric the handler owns is
   177  `sm_adhoc_ops_total{type}`, incremented when an `AdHocRequest`
   178  arrives.
   179  
   180  ## Testing strategy
   181  
   182  `adhoc_test.go` exercises:
   183  
   184  - Stream wiring and reconnect classification (using injected `grpcAdhocChecksClientFactory`).
   185  - Per-request handling via injected `runnerFactory`.
   186  - Tenant forwarding (`tenantCh`).
   187  - Error mapping (`errNotAuthorized`, `errIncompatibleApi`, `errIdleTimeout`, `errProbeUnregistered`).
   188  
   189  The pattern matches the Updater's tests: mock gRPC client and probes,
   190  drive the handler, assert observable side-effects. No real network is
   191  required.
   192  
   193  Run only this package:
   194  
   195  ```bash
   196  make test-go GO_TEST_ARGS=./internal/adhoc/...
   197  ```
   198  
   199  ## When to update this doc
   200  
   201  Update this document when you:
   202  
   203  - Change the `AdHocChecks` gRPC contract (RPCs, request/response shape).
   204  - Change how `defaultRunnerFactory` builds the synthetic `model.Check` (timeout selection, grace time, settings copy).
   205  - Change the Loki label set on ad-hoc payloads (`adhocData.Streams` shape) — operators filter on `type="adhoc"`.
   206  - Add a new error class to `Run`'s switch — including the `errIdleTimeout` handling that differs from the Updater.
   207  - Change the `probeId = 0` assumption in the prober factory wiring.
   208  - Add a new top-level field to `HandlerOpts`.
   209  - Change the contents of the JSON log emitted by `runner.Run` (downstream UI parses it).