github.com/smintz/nomad@v0.8.3/website/source/docs/job-specification/check_restart.html.md (about)

     1  ---
     2  layout: "docs"
     3  page_title: "check_restart Stanza - Job Specification"
     4  sidebar_current: "docs-job-specification-check_restart"
     5  description: |-
     6    The "check_restart" stanza instructs Nomad when to restart tasks with
     7    unhealthy service checks.
     8  ---
     9  
    10  # `check_restart` Stanza
    11  
    12  <table class="table table-bordered table-striped">
    13    <tr>
    14      <th width="120">Placement</th>
    15      <td>
    16        <code>job -> group -> task -> service -> **check_restart**</code>
    17        <br>
    18        <code>job -> group -> task -> service -> check -> **check_restart**</code>
    19      </td>
    20    </tr>
    21  </table>
    22  
    23  As of Nomad 0.7 the `check_restart` stanza instructs Nomad when to restart
    24  tasks with unhealthy service checks.  When a health check in Consul has been
    25  unhealthy for the `limit` specified in a `check_restart` stanza, it is
    26  restarted according to the task group's [`restart` policy][restart_stanza]. The
    27  `check_restart` settings apply to [`check`s][check_stanza], but may also be
    28  placed on [`service`s][service_stanza] to apply to all checks on a service.
    29  If `check_restart` is set on both the check and service, the stanzas are
    30  merged with the check values taking precedence.
    31  
    32  ```hcl
    33  job "mysql" {
    34    group "mysqld" {
    35  
    36      restart {
    37        attempts = 3
    38        delay    = "10s"
    39        interval = "10m"
    40        mode     = "fail"
    41      }
    42  
    43      task "server" {
    44        service {
    45          tags = ["leader", "mysql"]
    46  
    47          port = "db"
    48  
    49          check {
    50            type     = "tcp"
    51            port     = "db"
    52            interval = "10s"
    53            timeout  = "2s"
    54          }
    55  
    56          check {
    57            type     = "script"
    58            name     = "check_table"
    59            command  = "/usr/local/bin/check_mysql_table_status"
    60            args     = ["--verbose"]
    61            interval = "60s"
    62            timeout  = "5s"
    63  
    64            check_restart {
    65              limit = 3
    66              grace = "90s"
    67              ignore_warnings = false
    68            }
    69          }
    70        }
    71      }
    72    }
    73  }
    74  ```
    75  
    76  - `limit` `(int: 0)` - Restart task when a health check has failed `limit`
    77    times.  For example 1 causes a restart on the first failure. The default,
    78    `0`, disables health check based restarts. Failures must be consecutive. A
    79    single passing check will reset the count, so flapping services may not be
    80    restarted.
    81  
    82  - `grace` `(string: "1s")` - Duration to wait after a task starts or restarts
    83    before checking its health.
    84  
    85  - `ignore_warnings` `(bool: false)` - By default checks with both `critical`
    86    and `warning` statuses are considered unhealthy. Setting `ignore_warnings =
    87    true` treats a `warning` status like `passing` and will not trigger a restart.
    88  
    89  ## Example Behavior
    90  
    91  Using the example `mysql` above would have the following behavior:
    92  
    93  ```hcl
    94  check_restart {
    95    # ...
    96    grace = "90s"
    97    # ...
    98  }
    99  ```
   100  
   101  When the `server` task first starts and is registered in Consul, its health
   102  will not be checked for 90 seconds. This gives the server time to startup.
   103  
   104  ```hcl
   105  check_restart {
   106    limit = 3
   107    # ...
   108  }
   109  ```
   110  
   111  After the grace period if the script check fails, it has 180 seconds (`60s
   112  interval * 3 limit`) to pass before a restart is triggered. Once a restart is
   113  triggered the task group's [`restart` policy][restart_stanza] takes control:
   114  
   115  ```hcl
   116  restart {
   117    # ...
   118    delay    = "10s"
   119    # ...
   120  }
   121  ```
   122  
   123  The [`restart` stanza][restart_stanza] controls the restart behavior of the
   124  task. In this case it will stop the task and then wait 10 seconds before
   125  starting it again.
   126  
   127  Once the task restarts Nomad waits the `grace` period again before starting to
   128  check the task's health.
   129  
   130  
   131  ```hcl
   132  restart {
   133    attempts = 3
   134    # ...
   135    interval = "10m"
   136    mode     = "fail"
   137  }
   138  ```
   139  
   140  If the check continues to fail, the task will be restarted up to `attempts`
   141  times within an `interval`. If the `restart` attempts are reached within the
   142  `limit` then the `mode` controls the behavior. In this case the task would fail
   143  and not be restarted again. See the [`restart` stanza][restart_stanza] for
   144  details.
   145  
   146  [check_stanza]:  /docs/job-specification/service.html#check-parameters "check stanza"
   147  [restart_stanza]: /docs/job-specification/restart.html "restart stanza"
   148  [service_stanza]: /docs/job-specification/service.html "service stanza"