github.com/anth0d/nomad@v0.0.0-20221214183521-ae3a0a2cad06/website/content/docs/concepts/architecture.mdx (about)

     1  ---
     2  layout: docs
     3  page_title: Architecture
     4  description: Learn about the internal architecture of Nomad.
     5  ---
     6  
     7  # Architecture
     8  
     9  Nomad is a complex system that has many different pieces. To help both users and developers of Nomad
    10  build a mental model of how it works, this page documents the system architecture.
    11  
    12  ~> **Advanced Topic!** This page covers technical details
    13  of Nomad. You do not need to understand these details to
    14  effectively use Nomad. The details are documented here for
    15  those who wish to learn about them without having to go
    16  spelunking through the source code.
    17  
    18  # Glossary
    19  
    20  Before describing the architecture, we provide a glossary of terms to help
    21  clarify what is being discussed:
    22  
    23  #### Job
    24  
    25  A Job is a specification provided by users that declares a workload for
    26  Nomad. A Job is a form of _desired state_; the user is expressing that the
    27  job should be running, but not where it should be run. The responsibility of
    28  Nomad is to make sure the _actual state_ matches the user desired state. A
    29  Job is composed of one or more task groups.
    30  
    31  #### Task Group
    32  
    33  A Task Group is a set of tasks that must be run together. For example, a web
    34  server may require that a log shipping co-process is always running as well. A
    35  task group is the unit of scheduling, meaning the entire group must run on the
    36  same client node and cannot be split.
    37  
    38  #### Driver
    39  
    40  A Driver represents the basic means of executing your **Tasks**.  Example
    41  Drivers include Docker, QEMU, Java, and static binaries.
    42  
    43  #### Task
    44  
    45  A Task is the smallest unit of work in Nomad. Tasks are executed by drivers,
    46  which allow Nomad to be flexible in the types of tasks it supports. Tasks
    47  specify their driver, configuration for the driver, constraints, and resources
    48  required.
    49  
    50  #### Client
    51  
    52  A Nomad client is an agent configured to run and manage tasks using available
    53  compute resources on a machine. The agent is responsible for registering with
    54  the servers, watching for any work to be assigned and executing tasks. The
    55  Nomad agent is a long lived process which interfaces with the servers.
    56  
    57  #### Allocation
    58  
    59  An Allocation is a mapping between a task group in a job and a client node. A
    60  single job may have hundreds or thousands of task groups, meaning an
    61  equivalent number of allocations must exist to map the work to client
    62  machines. Allocations are created by the Nomad servers as part of scheduling
    63  decisions made during an evaluation.
    64  
    65  #### Evaluation
    66  
    67  Evaluations are the mechanism by which Nomad makes scheduling decisions.  When
    68  either the _desired state_ (jobs) or _actual state_ (clients) changes, Nomad
    69  creates a new evaluation to determine if any actions must be taken. An
    70  evaluation may result in changes to allocations if necessary.
    71  
    72  #### Deployment
    73  
    74  Deployments are the mechanism by which Nomad rolls out changes to cluster state
    75  in a step-by-step fashion. Deployments are only available for Jobs with the type
    76  `service`. When an Evaluation is processed, the scheduler creates only the
    77  number of Allocations permitted by the [`update`][] block and the current state
    78  of the cluster. The Deployment is used to monitor the health of those
    79  Allocations and emit a new Evaluation for the next step of the update.
    80  
    81  #### Server
    82  
    83  Nomad servers are the brains of the cluster. There is a cluster of servers per
    84  region and they manage all jobs and clients, run evaluations, and create task
    85  allocations.  The servers replicate data between each other and perform leader
    86  election to ensure high availability. More information about latency
    87  requirements for servers can be found in [Network
    88  Topology](/docs/install/production/requirements#network-topology).
    89  
    90  #### Regions
    91  
    92  Nomad models infrastructure as regions and datacenters. A region will contain
    93  one or more datacenters. A set of servers joined together will represent a
    94  single region. Servers federate across regions to make Nomad globally aware.
    95  
    96  #### Datacenters
    97  
    98  Nomad models a datacenter as an abstract grouping of clients within a
    99  region. Nomad clients are not required to be in the same datacenter as the
   100  servers they are joined with, but do need to be in the same
   101  region. Datacenters provide a way to express fault tolerance among jobs as
   102  well as isolation of infrastructure.
   103  
   104  #### Bin Packing
   105  
   106  Bin Packing is the process of filling bins with items in a way that maximizes
   107  the utilization of bins. This extends to Nomad, where the clients are "bins"
   108  and the items are task groups. Nomad optimizes resources by efficiently bin
   109  packing tasks onto client machines.
   110  
   111  # High-Level Overview
   112  
   113  Looking at only a single region, at a high level Nomad looks like this:
   114  
   115  [![Regional Architecture](/img/nomad-architecture-region.png)](/img/nomad-architecture-region.png)
   116  
   117  Within each region, we have both clients and servers. Servers are responsible for
   118  accepting jobs from users, managing clients, and [computing task placements](/docs/concepts/scheduling/scheduling).
   119  Each region may have clients from multiple datacenters, allowing a small number of servers
   120  to handle very large clusters.
   121  
   122  In some cases, for either availability or scalability, you may need to run multiple
   123  regions. Nomad supports federating multiple regions together into a single cluster.
   124  At a high level, this setup looks like this:
   125  
   126  [![Global Architecture](/img/nomad-architecture-global.png)](/img/nomad-architecture-global.png)
   127  
   128  Regions are fully independent from each other, and do not share jobs, clients, or
   129  state. They are loosely-coupled using a gossip protocol, which allows users to
   130  submit jobs to any region or query the state of any region transparently. Requests
   131  are forwarded to the appropriate server to be processed and the results returned.
   132  Data is _not_ replicated between regions.
   133  
   134  The servers in each region are all part of a single consensus group. This means
   135  that they work together to elect a single leader which has extra duties. The leader
   136  is responsible for processing all queries and transactions. Nomad is optimistically
   137  concurrent, meaning all servers participate in making scheduling decisions in parallel.
   138  The leader provides the additional coordination necessary to do this safely and
   139  to ensure clients are not oversubscribed.
   140  
   141  Each region is expected to have either three or five servers. This strikes a balance
   142  between availability in the case of failure and performance, as consensus gets
   143  progressively slower as more servers are added. However, there is no limit to the number
   144  of clients per region.
   145  
   146  Clients are configured to communicate with their regional servers and communicate
   147  using remote procedure calls (RPC) to register themselves, send heartbeats for liveness,
   148  wait for new allocations, and update the status of allocations. A client registers
   149  with the servers to provide the resources available, attributes, and installed drivers.
   150  Servers use this information for scheduling decisions and create allocations to assign
   151  work to clients.
   152  
   153  Users make use of the Nomad CLI or API to submit jobs to the servers. A job represents
   154  a desired state and provides the set of tasks that should be run. The servers are
   155  responsible for scheduling the tasks, which is done by finding an optimal placement for
   156  each task such that resource utilization is maximized while satisfying all constraints
   157  specified by the job. Resource utilization is maximized by bin packing, in which
   158  the scheduling tries to make use of all the resources of a machine without
   159  exhausting any dimension. Job constraints can be used to ensure an application is
   160  running in an appropriate environment. Constraints can be technical requirements based
   161  on hardware features such as architecture and availability of GPUs, or software features
   162  like operating system and kernel version, or they can be business constraints like
   163  ensuring PCI compliant workloads run on appropriate servers.
   164  
   165  # Getting in Depth
   166  
   167  This has been a brief high-level overview of the architecture of Nomad. There
   168  are more details available for each of the sub-systems. The [consensus protocol](/docs/concepts/consensus),
   169  [gossip protocol](/docs/concepts/gossip), and [scheduler design](/docs/concepts/scheduling/scheduling)
   170  are all documented in more detail.
   171  
   172  For other details, either consult the code, ask in IRC or reach out to the mailing list.
   173  
   174  
   175  [`update`]: /docs/job-specification/update