github.com/smintz/nomad@v0.8.3/website/source/guides/operating-a-job/failure-handling-strategies/reschedule.html.md (about) 1 --- 2 layout: "guides" 3 page_title: "Reschedule Stanza - Operating a Job" 4 sidebar_current: "guides-operating-a-job-failure-handling-strategies-reschedule" 5 description: |- 6 Nomad can reschedule failing tasks after any local restart attempts have been 7 exhausted. This is useful to recover from failures stemming from problems in the node 8 running the task. 9 --- 10 11 # Reschedule Stanza 12 13 Tasks can sometimes fail due to network, CPU or memory issues on the node running the task. In such situations, 14 Nomad can reschedule the task on another node. The [`reschedule` stanza][reschedule] can be used to configure how 15 Nomad should try placing failed tasks on another node in the cluster. Reschedule attempts have a delay between 16 each attempt, and the delay can be configured to increase between each rescheduling attempt according to a configurable 17 `delay_function`. See the [`reschedule` stanza][reschedule] for more information. 18 19 Service jobs are configured by default to have unlimited reschedule attempts. We recommend using the reschedule 20 stanza to ensure that failed tasks are automatically reattempted on another node without needing operator intervention. 21 22 # Example 23 The following CLI example shows job and allocation statuses for a task being rescheduled by Nomad. 24 The CLI shows the number of previous attempts if there is a limit on the number of reschedule attempts. 25 The CLI also shows when the next reschedule will be attempted. 26 27 ```text 28 $nomad job status demo 29 ID = demo 30 Name = demo 31 Submit Date = 2018-04-12T15:48:37-05:00 32 Type = service 33 Priority = 50 34 Datacenters = dc1 35 Status = pending 36 Periodic = false 37 Parameterized = false 38 39 Summary 40 Task Group Queued Starting Running Failed Complete Lost 41 demo 0 0 0 2 0 0 42 43 Future Rescheduling Attempts 44 Task Group Eval ID Eval Time 45 demo ee3de93f 5s from now 46 47 Allocations 48 ID Node ID Task Group Version Desired Status Created Modified 49 39d7823d f2c2eaa6 demo 0 run failed 5s ago 5s ago 50 fafb011b f2c2eaa6 demo 0 run failed 11s ago 10s ago 51 52 ``` 53 54 ```text 55 $nomad alloc status 3d0b 56 ID = 3d0bbdb1 57 Eval ID = 79b846a9 58 Name = demo.demo[0] 59 Node ID = 8a184f31 60 Job ID = demo 61 Job Version = 0 62 Client Status = failed 63 Client Description = <none> 64 Desired Status = run 65 Desired Description = <none> 66 Created = 15s ago 67 Modified = 15s ago 68 Reschedule Attempts = 3/5 69 Reschedule Eligibility = 25s from now 70 71 Task "demo" is "dead" 72 Task Resources 73 CPU Memory Disk IOPS Addresses 74 100 MHz 300 MiB 300 MiB 0 p1: 127.0.0.1:27646 75 76 Task Events: 77 Started At = 2018-04-12T20:44:25Z 78 Finished At = 2018-04-12T20:44:25Z 79 Total Restarts = 0 80 Last Restart = N/A 81 82 Recent Events: 83 Time Type Description 84 2018-04-12T15:44:25-05:00 Not Restarting Policy allows no restarts 85 2018-04-12T15:44:25-05:00 Terminated Exit Code: 127 86 2018-04-12T15:44:25-05:00 Started Task started by client 87 2018-04-12T15:44:25-05:00 Task Setup Building Task Directory 88 2018-04-12T15:44:25-05:00 Received Task received by client 89 90 ``` 91 92 [reschedule]: /docs/job-specification/reschedule.html "Nomad reschedule Stanza"