github.com/secure-build/gitlab-runner@v12.5.0+incompatible/docs/monitoring/README.md (about)

     1  # GitLab Runner monitoring
     2  
     3  GitLab Runner can be monitored using [Prometheus].
     4  
     5  ## Embedded Prometheus metrics
     6  
     7  > The embedded HTTP Statistics Server with Prometheus metrics was
     8  introduced in GitLab Runner 1.8.0.
     9  
    10  The GitLab Runner is instrumented with native Prometheus
    11  metrics, which can be exposed via an embedded HTTP server on the `/metrics`
    12  path. The server - if enabled - can be scraped by the Prometheus monitoring
    13  system or accessed with any other HTTP client.
    14  
    15  The exposed information includes:
    16  
    17  - Runner business logic metrics (e.g., the number of currently running jobs)
    18  - Go-specific process metrics (garbage collection stats, goroutines, memstats, etc.)
    19  - general process metrics (memory usage, CPU usage, file descriptor usage, etc.)
    20  - build version information
    21  
    22  The metrics format is documented in Prometheus'
    23  [Exposition formats](https://prometheus.io/docs/instrumenting/exposition_formats/)
    24  specification.
    25  
    26  These metrics are meant as a way for operators to monitor and gain insight into
    27  GitLab Runners. For example, you may be interested if the load average increase
    28  on your runner's host is related to an increase of processed jobs or not. Or
    29  you are running a cluster of machines to be used for the jobs and you want to
    30  track build trends to plan changes in your infrastructure.
    31  
    32  ### Learning more about Prometheus
    33  
    34  To learn how to set up a Prometheus server to scrape this HTTP endpoint and
    35  make use of the collected metrics, see Prometheus's [Getting
    36  started](https://prometheus.io/docs/prometheus/latest/getting_started/) guide. Also
    37  see the [Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)
    38  section for more details on how to configure Prometheus, as well as the section
    39  on [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) and setting up
    40  an [Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) to
    41  dispatch alert notifications.
    42  
    43  ## `pprof` HTTP endpoints
    44  
    45  > `pprof` integration was introduced in GitLab Runner 1.9.0.
    46  
    47  While having metrics about internal state of Runner process is useful
    48  we've found that in some cases it would be good to check what is happening
    49  inside of the Running process in real time. That's why we've introduced
    50  the `pprof` HTTP endpoints.
    51  
    52  `pprof` endpoints will be available via an embedded HTTP server on `/debug/pprof/`
    53  path.
    54  
    55  You can read more about using `pprof` in its [documentation][go-pprof].
    56  
    57  ## Configuration of the metrics HTTP server
    58  
    59  > **Note:**
    60  The metrics server exports data about the internal state of the
    61  GitLab Runner process and should not be publicly available!
    62  
    63  The metrics HTTP server can be configured in two ways:
    64  
    65  - with a `listen_address` global configuration option in `config.toml` file,
    66  - with a `--listen-address` command line option for the `run` command.
    67  
    68  In both cases the option accepts a string with the format `[host]:<port>`,
    69  where:
    70  
    71  - `host` can be an IP address or a host name,
    72  - `port` is a valid TCP port or symbolic service name (like `http`). We recommend to use port `9252` which is already [allocated in Prometheus](https://github.com/prometheus/prometheus/wiki/Default-port-allocations).
    73  
    74  If the listen address does not contain a port, it will default to `9252`.
    75  
    76  Examples of addresses:
    77  
    78  - `:9252` - will listen on all IPs of all interfaces on port `9252`
    79  - `localhost:9252` - will only listen on the loopback interface on port `9252`
    80  - `[2001:db8::1]:http` - will listen on IPv6 address `[2001:db8::1]` on the HTTP port `80`
    81  
    82  Remember that for listening on ports below `1024` - at least on Linux/Unix
    83  systems - you need to have root/administrator rights.
    84  
    85  Also please notice, that HTTP server is opened on selected `host:port`
    86  **without any authorization**. If you plan to bind the metrics server
    87  to a public interface then you should consider to use your firewall to
    88  limit access to this server or add a HTTP proxy which will add the
    89  authorization and access control layer.
    90  
    91  [go-pprof]: https://golang.org/pkg/net/http/pprof/
    92  [prometheus]: https://prometheus.io