github.com/smintz/nomad@v0.8.3/website/source/guides/operating-a-job/accessing-logs.html.md (about) 1 --- 2 layout: "guides" 3 page_title: "Accessing Logs - Operating a Job" 4 sidebar_current: "guides-operating-a-job-accessing-logs" 5 description: |- 6 Nomad provides a top-level mechanism for viewing application logs and data 7 files via the command line interface. This section discusses the nomad alloc 8 logs command and API interface. 9 --- 10 11 # Accessing Logs 12 13 Viewing application logs is critical for debugging issues, examining performance 14 problems, or even just verifying the application started correctly. To make this 15 as simple as possible, Nomad provides: 16 17 - Job specification for [log rotation](/docs/job-specification/logs.html) 18 - CLI command for [log viewing](/docs/commands/alloc/logs.html) 19 - API for programatic [log access](/api/client.html#stream-logs) 20 21 This section will utilize the job named "docs" from the [previous 22 sections](/guides/operating-a-job/submitting-jobs.html), but these operations 23 and command largely apply to all jobs in Nomad. 24 25 As a reminder, here is the output of the run command from the previous example: 26 27 ```text 28 $ nomad job run docs.nomad 29 ==> Monitoring evaluation "42d788a3" 30 Evaluation triggered by job "docs" 31 Allocation "04d9627d" created: node "a1f934c9", group "example" 32 Allocation "e7b8d4f5" created: node "012ea79b", group "example" 33 Allocation "5cbf23a1" modified: node "1e1aa1e0", group "example" 34 Evaluation status changed: "pending" -> "complete" 35 ==> Evaluation "42d788a3" finished with status "complete" 36 ``` 37 38 The provided allocation ID (which is also available via the `nomad status` 39 command) is required to access the application's logs. To access the logs of our 40 application, we issue the following command: 41 42 ```shell 43 $ nomad alloc logs 04d9627d 44 ``` 45 46 The output will look something like this: 47 48 ```text 49 <timestamp> 10.1.1.196:5678 10.1.1.196:33407 "GET / HTTP/1.1" 200 12 "curl/7.35.0" 21.809µs 50 <timestamp> 10.1.1.196:5678 10.1.1.196:33408 "GET / HTTP/1.1" 200 12 "curl/7.35.0" 20.241µs 51 <timestamp> 10.1.1.196:5678 10.1.1.196:33409 "GET / HTTP/1.1" 200 12 "curl/7.35.0" 13.629µs 52 ``` 53 54 By default, this will return the logs of the task. If more than one task is 55 defined in the job file, the name of the task is a required argument: 56 57 ```shell 58 $ nomad alloc logs 04d9627d server 59 ``` 60 61 The logs command supports both displaying the logs as well as following logs, 62 blocking for more output, similar to `tail -f`. To follow the logs, use the 63 appropriately named `-f` flag: 64 65 ```shell 66 $ nomad alloc logs -f 04d9627d 67 ``` 68 69 This will stream logs to our console. 70 71 If you wish to see only the tail of a log, use the `-tail` and `-n` flags: 72 73 ```shell 74 $ nomad alloc logs -tail -n 25 04d9627d 75 ``` 76 This will show the last 25 lines. If you omit the `-n` flag, `-tail` will 77 default to 10 lines. 78 79 By default, only the logs on stdout are displayed. To show the log output from 80 stderr, use the `-stderr` flag: 81 82 ```shell 83 $ nomad alloc logs -stderr 04d9627d 84 ``` 85 86 ## Log Shipper Pattern 87 88 While the logs command works well for quickly accessing application logs, it 89 generally does not scale to large systems or systems that produce a lot of log 90 output, especially for the long-term storage of logs. Nomad's retention of log 91 files is best effort, so chatty applications should use a better log retention 92 strategy. 93 94 Since applications log to the `alloc/` directory, all tasks within the same task 95 group have access to each others logs. Thus it is possible to have a task group 96 as follows: 97 98 ```hcl 99 group "my-group" { 100 task "server" { 101 # ... 102 103 # Setting the server task as the leader of the task group allows us to 104 # signal the log shipper task to gracefully shutdown when the server exits. 105 leader = true 106 } 107 108 task "log-shipper" { 109 # ... 110 } 111 } 112 ``` 113 114 In the above example, the `server` task is the application that should be run 115 and will be producing the logs. The `log-shipper` reads those logs from the 116 `alloc/logs/` directory and sends them to a longer-term storage solution such as 117 Amazon S3 or an internal log aggregation system. 118 119 When using the log shipper pattern, especially for batch jobs, the main task 120 should be marked as the [leader task](/docs/job-specification/task.html#leader). 121 By marking the main task as a leader, when the task completes all other tasks 122 within the group will be gracefully shutdown. This allows the log shipper to 123 finish sending any logs and then exiting itself. The log shipper should set a 124 high enough [`kill_timeout`](/docs/job-specification/task.html#kill_timeout) 125 such that it can ship any remaining logs before exiting.