github.com/smintz/nomad@v0.8.3/website/source/guides/operating-a-job/failure-handling-strategies/reschedule.html.md (about)

     1  ---
     2  layout: "guides"
     3  page_title: "Reschedule Stanza - Operating a Job"
     4  sidebar_current: "guides-operating-a-job-failure-handling-strategies-reschedule"
     5  description: |-
     6    Nomad can reschedule failing tasks after any local restart attempts have been
     7    exhausted. This is useful to recover from failures stemming from problems in the node
     8    running the task.
     9  ---
    10  
    11  # Reschedule Stanza
    12  
    13  Tasks can sometimes fail due to network, CPU or memory issues on the node running the task. In such situations,
    14  Nomad can reschedule the task on another node. The [`reschedule` stanza][reschedule] can be used to configure how
    15  Nomad should try placing failed tasks on another node in the cluster. Reschedule attempts have a delay between
    16  each attempt, and the delay can be configured to increase between each rescheduling attempt according to a configurable
    17  `delay_function`. See the [`reschedule` stanza][reschedule] for more information.
    18  
    19  Service jobs are configured by default to have unlimited reschedule attempts. We recommend using the reschedule
    20  stanza to ensure that failed tasks are automatically reattempted on another node without needing operator intervention.
    21  
    22  # Example
    23  The following CLI example shows job and allocation statuses for a task being rescheduled by Nomad.
    24  The CLI shows the number of previous attempts if there is a limit on the number of reschedule attempts.
    25  The CLI also shows when the next reschedule will be attempted.
    26  
    27  ```text
    28  $nomad job status demo
    29  ID            = demo
    30  Name          = demo
    31  Submit Date   = 2018-04-12T15:48:37-05:00
    32  Type          = service
    33  Priority      = 50
    34  Datacenters   = dc1
    35  Status        = pending
    36  Periodic      = false
    37  Parameterized = false
    38  
    39  Summary
    40  Task Group  Queued  Starting  Running  Failed  Complete  Lost
    41  demo        0       0         0        2       0         0
    42  
    43  Future Rescheduling Attempts
    44  Task Group  Eval ID   Eval Time
    45  demo        ee3de93f  5s from now
    46  
    47  Allocations
    48  ID        Node ID   Task Group  Version  Desired  Status  Created  Modified
    49  39d7823d  f2c2eaa6  demo        0        run      failed  5s ago   5s ago
    50  fafb011b  f2c2eaa6  demo        0        run      failed  11s ago  10s ago
    51  
    52  ```
    53  
    54  ```text
    55  $nomad alloc status 3d0b
    56  ID                     = 3d0bbdb1
    57  Eval ID                = 79b846a9
    58  Name                   = demo.demo[0]
    59  Node ID                = 8a184f31
    60  Job ID                 = demo
    61  Job Version            = 0
    62  Client Status          = failed
    63  Client Description     = <none>
    64  Desired Status         = run
    65  Desired Description    = <none>
    66  Created                = 15s ago
    67  Modified               = 15s ago
    68  Reschedule Attempts    = 3/5
    69  Reschedule Eligibility = 25s from now
    70  
    71  Task "demo" is "dead"
    72  Task Resources
    73  CPU      Memory   Disk     IOPS  Addresses
    74  100 MHz  300 MiB  300 MiB  0     p1: 127.0.0.1:27646
    75  
    76  Task Events:
    77  Started At     = 2018-04-12T20:44:25Z
    78  Finished At    = 2018-04-12T20:44:25Z
    79  Total Restarts = 0
    80  Last Restart   = N/A
    81  
    82  Recent Events:
    83  Time                       Type            Description
    84  2018-04-12T15:44:25-05:00  Not Restarting  Policy allows no restarts
    85  2018-04-12T15:44:25-05:00  Terminated      Exit Code: 127
    86  2018-04-12T15:44:25-05:00  Started         Task started by client
    87  2018-04-12T15:44:25-05:00  Task Setup      Building Task Directory
    88  2018-04-12T15:44:25-05:00  Received        Task received by client
    89  
    90  ```
    91  
    92  [reschedule]: /docs/job-specification/reschedule.html "Nomad reschedule Stanza"