github.com/smintz/nomad@v0.8.3/website/source/guides/operating-a-job/accessing-logs.html.md (about)

     1  ---
     2  layout: "guides"
     3  page_title: "Accessing Logs - Operating a Job"
     4  sidebar_current: "guides-operating-a-job-accessing-logs"
     5  description: |-
     6    Nomad provides a top-level mechanism for viewing application logs and data
     7    files via the command line interface. This section discusses the nomad alloc
     8    logs command and API interface.
     9  ---
    10  
    11  # Accessing Logs
    12  
    13  Viewing application logs is critical for debugging issues, examining performance
    14  problems, or even just verifying the application started correctly. To make this
    15  as simple as possible, Nomad provides:
    16  
    17  - Job specification for [log rotation](/docs/job-specification/logs.html)
    18  - CLI command for [log viewing](/docs/commands/alloc/logs.html)
    19  - API for programatic [log access](/api/client.html#stream-logs)
    20  
    21  This section will utilize the job named "docs" from the [previous
    22  sections](/guides/operating-a-job/submitting-jobs.html), but these operations
    23  and command largely apply to all jobs in Nomad.
    24  
    25  As a reminder, here is the output of the run command from the previous example:
    26  
    27  ```text
    28  $ nomad job run docs.nomad
    29  ==> Monitoring evaluation "42d788a3"
    30      Evaluation triggered by job "docs"
    31      Allocation "04d9627d" created: node "a1f934c9", group "example"
    32      Allocation "e7b8d4f5" created: node "012ea79b", group "example"
    33      Allocation "5cbf23a1" modified: node "1e1aa1e0", group "example"
    34      Evaluation status changed: "pending" -> "complete"
    35  ==> Evaluation "42d788a3" finished with status "complete"
    36  ```
    37  
    38  The provided allocation ID (which is also available via the `nomad status`
    39  command) is required to access the application's logs. To access the logs of our
    40  application, we issue the following command:
    41  
    42  ```shell
    43  $ nomad alloc logs 04d9627d
    44  ```
    45  
    46  The output will look something like this:
    47  
    48  ```text
    49  <timestamp> 10.1.1.196:5678 10.1.1.196:33407 "GET / HTTP/1.1" 200 12 "curl/7.35.0" 21.809µs
    50  <timestamp> 10.1.1.196:5678 10.1.1.196:33408 "GET / HTTP/1.1" 200 12 "curl/7.35.0" 20.241µs
    51  <timestamp> 10.1.1.196:5678 10.1.1.196:33409 "GET / HTTP/1.1" 200 12 "curl/7.35.0" 13.629µs
    52  ```
    53  
    54  By default, this will return the logs of the task. If more than one task is
    55  defined in the job file, the name of the task is a required argument:
    56  
    57  ```shell
    58  $ nomad alloc logs 04d9627d server
    59  ```
    60  
    61  The logs command supports both displaying the logs as well as following logs,
    62  blocking for more output, similar to `tail -f`. To follow the logs, use the
    63  appropriately named `-f` flag:
    64  
    65  ```shell
    66  $ nomad alloc logs -f 04d9627d
    67  ```
    68  
    69  This will stream logs to our console.
    70  
    71  If you wish to see only the tail of a log, use the `-tail` and `-n` flags:
    72  
    73  ```shell
    74  $ nomad alloc logs -tail -n 25 04d9627d
    75  ```
    76  This will show the last 25 lines. If you omit the `-n` flag, `-tail` will
    77  default to 10 lines.
    78  
    79  By default, only the logs on stdout are displayed. To show the log output from
    80  stderr, use the `-stderr` flag:
    81  
    82  ```shell
    83  $ nomad alloc logs -stderr 04d9627d
    84  ```
    85  
    86  ## Log Shipper Pattern
    87  
    88  While the logs command works well for quickly accessing application logs, it
    89  generally does not scale to large systems or systems that produce a lot of log
    90  output, especially for the long-term storage of logs. Nomad's retention of log
    91  files is best effort, so chatty applications should use a better log retention
    92  strategy.
    93  
    94  Since applications log to the `alloc/` directory, all tasks within the same task
    95  group have access to each others logs. Thus it is possible to have a task group
    96  as follows:
    97  
    98  ```hcl
    99  group "my-group" {
   100    task "server" {
   101      # ...
   102  
   103      # Setting the server task as the leader of the task group allows us to
   104      # signal the log shipper task to gracefully shutdown when the server exits.
   105      leader = true
   106    }
   107  
   108    task "log-shipper" {
   109      # ...
   110    }
   111  }
   112  ```
   113  
   114  In the above example, the `server` task is the application that should be run
   115  and will be producing the logs. The `log-shipper` reads those logs from the
   116  `alloc/logs/` directory and sends them to a longer-term storage solution such as
   117  Amazon S3 or an internal log aggregation system.
   118  
   119  When using the log shipper pattern, especially for batch jobs, the main task
   120  should be marked as the [leader task](/docs/job-specification/task.html#leader).
   121  By marking the main task as a leader, when the task completes all other tasks
   122  within the group will be gracefully shutdown. This allows the log shipper to
   123  finish sending any logs and then exiting itself. The log shipper should set a
   124  high enough [`kill_timeout`](/docs/job-specification/task.html#kill_timeout)
   125  such that it can ship any remaining logs before exiting.