github.com/smintz/nomad@v0.8.3/website/source/docs/agent/index.html.md (about) 1 --- 2 layout: "docs" 3 page_title: "Nomad Agent" 4 sidebar_current: "docs-agent" 5 description: |- 6 The Nomad agent is a long running process which can be used either in 7 a client or server mode. 8 --- 9 10 # Nomad Agent 11 12 The Nomad agent is a long running process which runs on every machine that 13 is part of the Nomad cluster. The behavior of the agent depends on if it is 14 running in client or server mode. Clients are responsible for running tasks, 15 while servers are responsible for managing the cluster. 16 17 Client mode agents are relatively simple. They make use of fingerprinting 18 to determine the capabilities and resources of the host machine, as well as 19 determining what [drivers](/docs/drivers/index.html) are available. Clients 20 register with servers to provide the node information, heartbeat to provide 21 liveness, and run any tasks assigned to them. 22 23 Servers take on the responsibility of being part of the 24 [consensus protocol](/docs/internals/consensus.html) and [gossip protocol](/docs/internals/gossip.html). 25 The consensus protocol, powered by Raft, allows the servers to perform 26 leader election and state replication. The gossip protocol allows for simple 27 clustering of servers and multi-region federation. The higher burden on the 28 server nodes means that usually they should be run on dedicated instances -- 29 they are more resource intensive than a client node. 30 31 Client nodes make up the majority of the cluster, and are very lightweight as 32 they interface with the server nodes and maintain very little state of their 33 own. Each cluster has usually 3 or 5 server mode agents and potentially 34 thousands of clients. 35 36 ## Running an Agent 37 38 The agent is started with the [`nomad agent` command](/docs/commands/agent.html). This 39 command blocks, running forever or until told to quit. The agent command takes a variety 40 of configuration options, but most have sane defaults. 41 42 When running `nomad agent`, you should see output similar to this: 43 44 ```text 45 $ nomad agent -dev 46 ==> Starting Nomad agent... 47 ==> Nomad agent configuration: 48 49 Client: true 50 Log Level: INFO 51 Region: global (DC: dc1) 52 Server: true 53 54 ==> Nomad agent started! Log data will stream in below: 55 56 [INFO] serf: EventMemberJoin: server-1.node.global 127.0.0.1 57 [INFO] nomad: starting 4 scheduling worker(s) for [service batch _core] 58 ... 59 ``` 60 61 There are several important messages that `nomad agent` outputs: 62 63 - **Client**: This indicates whether the agent has enabled client mode. 64 Client nodes fingerprint their host environment, register with servers, 65 and run tasks. 66 67 - **Log Level**: This indicates the configured log level. Only messages with 68 an equal or higher severity will be logged. This can be tuned to increase 69 verbosity for debugging, or reduced to avoid noisy logging. 70 71 - **Region**: This is the region and datacenter in which the agent is configured 72 to run. Nomad has first-class support for multi-datacenter and multi-region 73 configurations. The `-region` and `-dc` flags can be used to set the region 74 and datacenter. The default is the `global` region in `dc1`. 75 76 - **Server**: This indicates whether the agent has enabled server mode. 77 Server nodes have the extra burden of participating in the consensus protocol, 78 storing cluster state, and making scheduling decisions. 79 80 ## Stopping an Agent 81 82 An agent can be stopped in two ways: gracefully or forcefully. By default, 83 any signal to an agent (interrupt, terminate, kill) will cause the agent 84 to forcefully stop. Graceful termination can be configured by either 85 setting `leave_on_interrupt` or `leave_on_terminate` to respond to the 86 respective signals. 87 88 When gracefully exiting, clients will update their status to terminal on 89 the servers so that tasks can be migrated to healthy agents. Servers 90 will notify their intention to leave the cluster which allows them to 91 leave the [consensus](/docs/internals/consensus.html) peer set. 92 93 It is especially important that a server node be allowed to leave gracefully 94 so that there will be a minimal impact on availability as the server leaves 95 the consensus peer set. If a server does not gracefully leave, and will not 96 return into service, the [`server force-leave` command](/docs/commands/server/force-leave.html) 97 should be used to eject it from the consensus peer set. 98 99 ## Lifecycle 100 101 Every agent in the Nomad cluster goes through a lifecycle. Understanding 102 this lifecycle is useful for building a mental model of an agent's interactions 103 with a cluster and how the cluster treats a node. 104 105 When a client agent is first started, it fingerprints the host machine to 106 identify its attributes, capabilities, and [task drivers](/docs/drivers/index.html). 107 These are reported to the servers during an initial registration. The addresses 108 of known servers are provided to the agent via configuration, potentially using 109 DNS for resolution. Using [Consul](https://www.consul.io) provides a way to avoid hard 110 coding addresses and resolving them on demand. 111 112 While a client is running, it is performing heartbeating with servers to 113 maintain liveness. If the heartbeats fail, the servers assume the client node 114 has failed, and stop assigning new tasks while migrating existing tasks. 115 It is impossible to distinguish between a network failure and an agent crash, 116 so both cases are handled the same. Once the network recovers or a crashed agent 117 restarts the node status will be updated and normal operation resumed. 118 119 To prevent an accumulation of nodes in a terminal state, Nomad does periodic 120 garbage collection of nodes. By default, if a node is in a failed or 'down' 121 state for over 24 hours it will be garbage collected from the system. 122 123 Servers are slightly more complex as they perform additional functions. They 124 participate in a [gossip protocol](/docs/internals/gossip.html) both to cluster 125 within a region and to support multi-region configurations. When a server is 126 first started, it does not know the address of other servers in the cluster. 127 To discover its peers, it must _join_ the cluster. This is done with the 128 [`server join` command](/docs/commands/server/join.html) or by providing the 129 proper configuration on start. Once a node joins, this information is gossiped 130 to the entire cluster, meaning all nodes will eventually be aware of each other. 131 132 When a server _leaves_, it specifies its intent to do so, and the cluster marks that 133 node as having _left_. If the server has _left_, replication to it will stop and it 134 is removed from the consensus peer set. If the server has _failed_, replication 135 will attempt to make progress to recover from a software or network failure. 136 137 ## Permissions 138 139 Nomad servers should be run with the lowest possible permissions. Nomad clients 140 must be run as root due to the OS isolation mechanisms that require root 141 privileges. In all cases, it is recommended you create a `nomad` user with the 142 minimal set of required privileges.