github.com/1aal/kubeblocks@v0.0.0-20231107070852-e1c03e598921/docs/user_docs/observability/monitor-database.md (about)

     1  ---
     2  title: Monitor database
     3  description: How to monitor your database
     4  keywords: [monitor database, monitor a cluster, monitor]
     5  sidebar_position: 1
     6  ---
     7  
     8  # Monitor a database
     9  
    10  With the built-in database observability, you can observe the database health status and track and measure your database in real-time to optimize database performance. This section shows you how database monitoring tools work with KubeBlocks and how to use the function.
    11  
    12  ## For Playground/test
    13  
    14  KubeBlocks integrates open-source monitoring components, such as Prometheus, AlertManager, and Grafana, by add-ons and adopts the custom `apecloud-otel-collector` to collect the monitoring indicators of databases and host machines. All monitoring add-ons are enabled when KubeBlocks Playground is deployed.
    15  
    16  KubeBlock Playground supports the following built-in monitoring add-ons:
    17  
    18  * `prometheus`: it includes Prometheus and AlertManager add-ons.
    19  * `grafana`: it includes Grafana monitoring add-ons.
    20  * `alertmanager-webhook-adaptor`: it includes the notification extension add-on and is used to extend the notification capability of AlertManager. Currently, the custom bots of Feishu, DingTalk, and Wechat Enterprise are supported.
    21  * `apecloud-otel-collector`: it is used to collect the indicators of databases and the host machine.
    22  
    23  1. View all built-in add-ons and make sure the monitoring add-ons are enabled.
    24  
    25     ```bash
    26     # View all add-ons supported
    27     kbcli addon list
    28     ...
    29     grafana                        Helm   Enabled                   true                                                                                    
    30     alertmanager-webhook-adaptor   Helm   Enabled                   true                                                                                    
    31     prometheus                     Helm   Enabled    alertmanager   true 
    32     ...
    33     ```
    34  
    35  2. View the dashboard list.
    36  
    37     ```bash
    38     kbcli dashboard list
    39     >
    40     NAME                                 NAMESPACE   PORT    CREATED-TIME
    41     kubeblocks-grafana                   kb-system   13000   Jul 24,2023 11:38 UTC+0800
    42     kubeblocks-prometheus-alertmanager   kb-system   19093   Jul 24,2023 11:38 UTC+0800
    43     kubeblocks-prometheus-server         kb-system   19090   Jul 24,2023 11:38 UTC+0800
    44     ```
    45  
    46  3. Open and view the web console of a monitoring dashboard. For example,
    47  
    48     ```bash
    49     kbcli dashboard open kubeblocks-grafana
    50     ```
    51  
    52  ## For production environment
    53  
    54  In the production environment, it is highly recommended to build your monitoring system or purchase a third-party monitoring service.
    55  
    56  ### Enable monitoring function
    57  
    58  KubeBlocks provides an add-on, `victoria-metrics-agent`, to push the monitoring data to a third-party monitoring system compatible with the Prometheus Remote Write protocol. Compared with the native Prometheus, [vmgent](https://docs.victoriametrics.com/vmagent.html) is lighter and supports the horizontal extension.
    59  
    60  1. Enable data push.
    61  
    62     You just need to provide the endpoint which supports the Prometheus Remote Write protocol and multiple endpoints can be supported. Refer to the tutorials of your third-party monitoring system for how to get an endpoint.
    63  
    64     The following examples show how to enable data push by different options.
    65  
    66     ```bash
    67     # The default option. You only need to provide an endpoint with no verification.
    68     # Endpoint example: http://localhost:8428/api/v1/write
    69     kbcli addon enable victoria-metrics-agent --set remoteWriteUrls={http://<remoteWriteUrl>:<port>/<remote write path>}
    70     ```
    71  
    72     ```bash
    73     # Basic Auth
    74     kbcli addon enable victoria-metrics-agent --set "extraArgs.remoteWrite.basicAuth.username=<your username>,remoteWrite.basicAuth.password=<your password>,remoteWriteUrls={http://<remoteWriteUrl>:<port>/<remote write path>}"
    75     ```
    76  
    77     ```bash
    78     # TLS
    79     kbcli addon enable victoria-metrics-agent --set "extraArgs.tls=true,extraArgs.tlsCertFile=<path to certifle>,extraArgs.tlsKeyFile=<path to keyfile>,remoteWriteUrls={http://<remoteWriteUrl>:<port>/<remote write path>}"
    80     ```
    81  
    82     ```bash
    83     # AWS SigV4
    84     kbcli addon enable victoria-metrics-agent --set "extraArgs.remoteWrite.aws.region=<your AMP region>,extraArgs.remoteWrite.aws.accessKey=<your accessKey>,extraArgs.remoteWrite.aws.secretKey=<your secretKey>,remoteWriteUrls={http://<remoteWriteUrl>:<port>/<remote write path>}"
    85     ```
    86  
    87  2. (Optional) Horizontally scale the `victoria-metrics-agent` add-on.
    88  
    89     When the amount of database instances continues to increase, a single-node vmagent becomes the bottleneck. This problem can be solved by scaling vmagent. The multiple-node vmagent automatically divides the task of data collection according to the Hash strategy.
    90  
    91     ```bash
    92     kbcli addon enable victoria-metrics-agent --replicas <replica count> --set remoteWriteUrls={http://<remoteWriteUrl>:<port>/<remote write path>}
    93     ```
    94  
    95  3. (Optional) Disable the `victoria-metrics-agent` add-on.
    96  
    97     ```bash
    98     kbcli addon disable victoria-metrics-agent
    99     ```
   100  
   101  ### Integrate Dashboard and Alert Rules
   102  
   103  Kubeblocks provides Grafana Dashboards and Prometheus AlertRules for mainstream engines, which you can obtain from [the repository](https://github.com/apecloud/kubeblocks-mixin), or convert and customize according to your needs.
   104  
   105  For the importing method, refer to the tutorials of your third-party monitoring service.
   106  
   107  ## Enable the monitoring function for a database
   108  
   109  The monitoring function is enabled by default when a database is created. The open-source or customized Exporter is injected after the monitoring function is enabled. This Exporter can be found by Prometheus server automatically and scrape monitoring indicators at regular intervals. You can change mysql as `postgresql`, `mongodb`, `redis` to create a cluster of other database engines.
   110  
   111  * For a new cluster, run the command below to create a database cluster.
   112  
   113      ```bash
   114      # Search the cluster definition
   115      kbcli clusterdefinition list 
   116  
   117      # Create a cluster
   118      kbcli cluster create mysql <clustername> 
   119      ```
   120  
   121  :::note
   122  
   123  You can change `--monitoring-interval` as `0` to disable the monitoring function but it is not recommended.
   124  
   125  ```bash
   126  kbcli cluster create mysql mycluster --monitoring-interval=0
   127  ```
   128  
   129  :::
   130  
   131  * For the existing cluster with the monitoring function disabled, you can update it to enable the monitor function by the `update` command.
   132  
   133      ```bash
   134      kbcli cluster update mycluster --monitoring-interval=15s
   135      ```
   136  
   137  You can view the dashboard of the corresponding cluster via Grafana Web Console. For more detailed information, see the [Grafana dashboard documentation](https://grafana.com/docs/grafana/latest/dashboards/).