github.com/greenboxal/deis@v1.12.1/docs/managing_deis/platform_monitoring.rst (about)

     1  :title: Platform monitoring
     2  :description: Configuring platform monitoring.
     3  
     4  .. _platform_monitoring:
     5  
     6  Platform monitoring
     7  ===================
     8  
     9  While Deis itself doesn't have a built-in monitoring system, Deis components and deployed
    10  applications alike run entirely within Docker containers. This means that monitoring tools and
    11  services which support Docker containers should work with Deis. A few tools and monitoring services
    12  which support Docker integrations are detailed below.
    13  
    14  Tools
    15  -----
    16  
    17  cadvisor
    18  ~~~~~~~~
    19  
    20  Google's Container Advisor (`cadvisor`_) runs inside a Docker container and shows memory and CPU
    21  usage for all containers running on the host. To run cAdvisor:
    22  
    23  .. code-block:: console
    24  
    25      sudo docker run \
    26      --volume=/:/rootfs:ro \
    27      --volume=/var/run:/var/run:rw \
    28      --volume=/sys:/sys:ro \
    29      --volume=/var/lib/docker/:/var/lib/docker:ro \
    30      --publish=8080:8080 \
    31      --detach=true \
    32      --name=cadvisor \
    33      google/cadvisor:latest
    34  
    35  To run cAdvisor on all hosts in the cluster, you can submit and start a fleet service:
    36  
    37  .. code-block:: console
    38  
    39      [Unit]
    40      Description=Google Container Advisor
    41      Requires=docker.socket
    42      After=docker.socket
    43  
    44      [Service]
    45      ExecStartPre=/bin/sh -c "docker history google/cadvisor:latest >/dev/null 2>&1 || docker pull google/cadvisor:latest"
    46      ExecStartPre=/bin/sh -c "docker inspect cadvisor >/dev/null 2>&1 && docker rm -f cadvisor || true"
    47      ExecStart=/usr/bin/docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --name=cadvisor google/cadvisor:latest
    48      ExecStopPost=-/usr/bin/docker rm -f cadvisor
    49      Restart=on-failure
    50      RestartSec=5
    51  
    52      [Install]
    53      WantedBy=multi-user.target
    54  
    55      [X-Fleet]
    56      Global=true
    57  
    58  Save the file as ``cadvisor.service``. Load and start the service with
    59  ``fleetctl load cadvisor.service && fleetctl start cadvisor.service``.
    60  
    61  The web interface will be accessible at port 8080 on each host.
    62  
    63  In addition to starting a cAdvisor instance on each CoreOS host, there's also a project called
    64  `heapster`_ from the Google Cloud Platform team, which seems to be a cluster-aware cAdvisor.
    65  
    66  Monitoring services
    67  -------------------
    68  
    69  These are a few monitoring services which are known to provide Docker integrations.
    70  Additions to this reference guide are much appreciated!
    71  
    72  Datadog
    73  ~~~~~~~
    74  
    75  The `Datadog`_ cloud monitoring service provides a monitor agent which runs on the host and provides
    76  metrics for all Docker containers (which is functionally similar to cAdvisor's implementation).
    77  See `this blog post`_ for details. The `Datadog agent`_ for Docker can be run on a single host as
    78  follows:
    79  
    80  .. code-block:: console
    81  
    82      docker run -d --privileged --name dd-agent -h `hostname` -v /var/run/docker.sock:/var/run/docker.sock -v /proc/mounts:/host/proc/mounts:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY=YOUR_REAL_API_KEY datadog/docker-dd-agent
    83  
    84  Be sure to substitute ``YOUR_REAL_API_KEY`` for your Datadog API key.
    85  
    86  To run Datadog for the entire cluster, you can submit and start a fleet service (again, substitute ``YOUR_REAL_API_KEY``):
    87  
    88  .. code-block:: console
    89  
    90      [Unit]
    91      Description=Datadog
    92      Requires=docker.socket
    93      After=docker.socket
    94  
    95      [Service]
    96      ExecStartPre=/bin/sh -c "docker history datadog/docker-dd-agent:latest >/dev/null || docker pull datadog/docker-dd-agent:latest"
    97      ExecStart=/usr/bin/docker run --privileged --name dd-agent -h %H -v /var/run/docker.sock:/var/run/docker.sock -v /proc/mounts:/host/proc/mounts:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY=YOUR_REAL_API_KEY datadog/docker-dd-agent
    98  
    99      [Install]
   100      WantedBy=multi-user.target
   101  
   102      [X-Fleet]
   103      Global=true
   104  
   105  Save the file as ``datadog.service``. Load and start the service with
   106  ``fleetctl load datadog.service && fleetctl start datadog.service``.
   107  
   108  Shortly thereafter, you should start to see metrics from your Deis cluster appear in your Datadog dashboard.
   109  
   110  New Relic
   111  ~~~~~~~~~
   112  
   113  The `New Relic`_ monitoring service's agent will run on the CoreOS host and report metrics to New Relic.
   114  
   115  The Deis community's own Johannes Würbach has developed a fleet service for New Relic in his
   116  `newrelic-sysmond`_ repository.
   117  
   118  SPM Performance Monitoring
   119  ~~~~~~~~~~~~~~~~~~~~~~~~~~
   120  
   121  `SPM for Docker`_ provides Monitoring, Anomaly Detection and Alerting. Detailed metrics and events are collected for the host machine and running containers (e.g. cpu, memory, IO, network, limits). The SPM agent runs in a Docker container.
   122  
   123  `Create a new SPM App`_ of type "Docker" and copy the Application Token 
   124  
   125  Set the Application Token (SPM_TOKEN) via etcd to make it available in the cluster.
   126  
   127  .. code-block:: console
   128  
   129      etcdctl set /SPM_TOKEN YOUR_SPM_APP_TOKEN
   130  
   131  SPM Docker Agent can be run using Docker as follows (assuming the SPM_TOKEN is set via etcdctl):
   132  
   133  .. code-block:: console
   134  
   135      docker run -d --name spm-agent -e SPM_TOKEN=`etcdctl get SPM_TOKEN` -e HOSTNAME=$HOSTNAME -v /var/run/docker.sock:/var/run/docker.sock sematext/spm-agent-docker
   136      
   137  
   138  To activate SPM Docker Agent for the entire cluster submit this unit file to fleet
   139  
   140  .. code-block:: console
   141  
   142      [Unit]
   143      Description=SPM Docker Agent
   144      After=docker.service
   145      Requires=docker.service
   146  
   147      [Service]
   148      TimeoutStartSec=0
   149      EnvironmentFile=/etc/environment
   150      Restart=always
   151      RestartSec=30s
   152      ExecStartPre=-/usr/bin/docker kill spm-agent
   153      ExecStartPre=-/usr/bin/docker rm spm-agent
   154      ExecStartPre=/usr/bin/docker pull sematext/spm-agent-docker:latest
   155      ExecStart=/bin/sh -c 'set -ex; /usr/bin/docker run --name spm-agent -e SPM_TOKEN=$(etcdctl get SPM_TOKEN) -e HOSTNAME=$HOSTNAME -v /var/run/docker.sock:/var/run/docker.sock sematext/spm-agent-docker'
   156      ExecStop=/usr/bin/docker stop spm-agent
   157  
   158      [Install]
   159      WantedBy=multi-user.target
   160      
   161      [X-Fleet]
   162      Global=true
   163  
   164  Save the file as ``spm-agent.service``. 
   165  
   166  .. code-block:: console
   167  
   168      wget https://raw.githubusercontent.com/sematext/spm-agent-docker/master/coreos/spm-agent.service
   169  
   170  Load and start the service with
   171  
   172  .. code-block:: console
   173  
   174      fleetctl load spm-agent.service && fleetctl start spm-agent.service 
   175  
   176  After one minute, you should see metrics in SPM. 
   177  
   178  Documentation, source code and support information is available here:
   179  `https://github.com/sematext/spm-agent-docker`_. 
   180  
   181  
   182  
   183  .. _`cadvisor`: https://github.com/google/cadvisor
   184  .. _`Datadog`: https://www.datadoghq.com
   185  .. _`Datadog agent`: https://github.com/DataDog/docker-dd-agent
   186  .. _`heapster`: https://github.com/GoogleCloudPlatform/heapster/blob/master/clusters/coreos/README.md
   187  .. _`this blog post`: https://www.datadoghq.com/2014/06/monitor-docker-datadog/
   188  .. _`New Relic`: http://newrelic.com/
   189  .. _`newrelic-sysmond`: https://github.com/johanneswuerbach/newrelic-sysmond-service
   190  .. _`SPM for Docker`: http://sematext.com/spm/integrations/docker-monitoring.html
   191  .. _`Create a new SPM App`: https://apps.sematext.com/spm-reports/registerApplication.do
   192  .. _`https://github.com/sematext/spm-agent-docker`: https://github.com/sematext/spm-agent-docker