github.com/demonoid81/containerd@v1.3.4/docs/ops.md (about)

     1  # containerd for Ops and Admins
     2  
     3  containerd is meant to be a simple daemon to run on any system.
     4  It provides a minimal config with knobs to configure the daemon and what plugins are used when necessary.
     5  
     6  ```
     7  NAME:
     8     containerd -
     9                      __        _                     __
    10    _________  ____  / /_____ _(_)___  ___  _________/ /
    11   / ___/ __ \/ __ \/ __/ __ `/ / __ \/ _ \/ ___/ __  /
    12  / /__/ /_/ / / / / /_/ /_/ / / / / /  __/ /  / /_/ /
    13  \___/\____/_/ /_/\__/\__,_/_/_/ /_/\___/_/   \__,_/
    14  
    15  high performance container runtime
    16  
    17  
    18  USAGE:
    19     containerd [global options] command [command options] [arguments...]
    20  
    21  VERSION:
    22     v1.0.0-alpha3-36-ge9b86af
    23  
    24  COMMANDS:
    25       config   information on the containerd config
    26       help, h  Shows a list of commands or help for one command
    27  
    28  GLOBAL OPTIONS:
    29     --config value, -c value     path to the configuration file (default: "/etc/containerd/config.toml")
    30     --log-level value, -l value  set the logging level [debug, info, warn, error, fatal, panic]
    31     --address value, -a value    address for containerd's GRPC server
    32     --root value                 containerd root directory
    33     --help, -h                   show help
    34     --version, -v                print the version
    35  ```
    36  
    37  While a few daemon level options can be set from CLI flags the majority of containerd's configuration is kept in the configuration file.
    38  The default path for the config file is located at `/etc/containerd/config.toml`.
    39  You can change this path via the `--config,-c` flags when booting the daemon.
    40  
    41  ## systemd
    42  
    43  If you are using systemd as your init system, which most modern linux OSs are, the service file requires a few modifications.
    44  
    45  ```systemd
    46  [Unit]
    47  Description=containerd container runtime
    48  Documentation=https://containerd.io
    49  After=network.target
    50  
    51  [Service]
    52  ExecStartPre=-/sbin/modprobe overlay
    53  ExecStart=/usr/local/bin/containerd
    54  Delegate=yes
    55  KillMode=process
    56  
    57  [Install]
    58  WantedBy=multi-user.target
    59  ```
    60  
    61  `Delegate=yes` and `KillMode=process` are the two most important changes you need to make in the `[Service]` section.
    62  
    63  `Delegate` allows containerd and its runtimes to manage the cgroups of the containers that it creates.
    64  Without setting this option, systemd will try to move the processes into its own cgroups, causing problems for containerd and its runtimes to properly account for resource usage with the containers.
    65  
    66  `KillMode` handles when containerd is being shut down.
    67  By default, systemd will look in its named cgroup and kill every process that it knows about for the service.
    68  This is not what we want.
    69  As ops, we want to be able to upgrade containerd and allow existing containers to keep running without interruption.
    70  Setting `KillMode` to `process` ensures that systemd only kills the containerd daemon and not any child processes such as the shims and containers.
    71  
    72  The following `systemd-run` command starts containerd in a similar way:
    73  ```
    74  sudo systemd-run -p Delegate=yes -p KillMode=process /usr/local/bin/containerd
    75  ```
    76  
    77  ## Base Configuration
    78  
    79  In the containerd config file you will find settings for persistent and runtime storage locations as well as grpc, debug, and metrics addresses for the various APIs.
    80  
    81  There are a few settings that are important for ops.
    82  The first setting is the `oom_score`.  Because containerd will be managing multiple containers, we need to ensure that containers are killed before the containerd daemon in an out of memory condition.
    83  We also do not want to make containerd unkillable, but we want to lower its score to the level of other system daemons.
    84  
    85  The `subreaper` setting is also important on linux systems.
    86  This allows containerd to reap any re-parented processes from the shims or containers.
    87  
    88  containerd also exports its own metrics as well as container level metrics via the prometheus metrics format.
    89  Currently, prometheus only supports TCP endpoints, therefore, the metrics address should be a TCP address that your prometheus infrastructure can scrape metrics from.
    90  
    91  containerd also has two different storage locations on a host system.
    92  One is for persistent data and the other is for runtime state.
    93  
    94  `root` will be used to store any type of persistent data for containerd.
    95  Snapshots, content, metadata for containers and image, as well as any plugin data will be kept in this location.
    96  The root is also namespaced for plugins that containerd loads.
    97  Each plugin will have its own directory where it stores data.
    98  containerd itself does not actually have any persistent data that it needs to store, its functionality comes from the plugins that are loaded.
    99  
   100  
   101  ```
   102  /var/lib/containerd/
   103  ├── io.containerd.content.v1.content
   104  │   ├── blobs
   105  │   └── ingest
   106  ├── io.containerd.metadata.v1.bolt
   107  │   └── meta.db
   108  ├── io.containerd.runtime.v1.linux
   109  │   ├── default
   110  │   └── example
   111  ├── io.containerd.snapshotter.v1.btrfs
   112  └── io.containerd.snapshotter.v1.overlayfs
   113      ├── metadata.db
   114      └── snapshots
   115  ```
   116  
   117  `state` will be used to store any type of ephemeral data.
   118  Sockets, pids, runtime state, mount points, and other plugin data that must not persist between reboots are stored in this location.
   119  
   120  ```
   121  /run/containerd
   122  ├── containerd.sock
   123  ├── debug.sock
   124  ├── io.containerd.runtime.v1.linux
   125  │   └── default
   126  │       └── redis
   127  │           ├── config.json
   128  │           ├── init.pid
   129  │           ├── log.json
   130  │           └── rootfs
   131  │               ├── bin
   132  │               ├── data
   133  │               ├── dev
   134  │               ├── etc
   135  │               ├── home
   136  │               ├── lib
   137  │               ├── media
   138  │               ├── mnt
   139  │               ├── proc
   140  │               ├── root
   141  │               ├── run
   142  │               ├── sbin
   143  │               ├── srv
   144  │               ├── sys
   145  │               ├── tmp
   146  │               ├── usr
   147  │               └── var
   148  └── runc
   149      └── default
   150          └── redis
   151              └── state.json
   152  ```
   153  
   154  Both the `root` and `state` directories are namespaced for plugins.
   155  Both directories are an implementation detail of containerd and its plugins.
   156  They should not be tampered with as corruption and bugs can and will happen.
   157  External apps reading or watching changes in these directories have been known to cause `EBUSY` and stale file handles when containerd and/or its plugins try to cleanup resources.
   158  
   159  ```toml
   160  # persistent data location
   161  root = "/var/lib/containerd"
   162  # runtime state information
   163  state = "/run/containerd"
   164  # set containerd as a subreaper on linux when it is not running as PID 1
   165  subreaper = true
   166  # set containerd's OOM score
   167  oom_score = -999
   168  
   169  # grpc configuration
   170  [grpc]
   171    address = "/run/containerd/containerd.sock"
   172    # socket uid
   173    uid = 0
   174    # socket gid
   175    gid = 0
   176  
   177  # debug configuration
   178  [debug]
   179    address = "/run/containerd/debug.sock"
   180    # socket uid
   181    uid = 0
   182    # socket gid
   183    gid = 0
   184    # debug level
   185    level = "info"
   186  
   187  # metrics configuration
   188  [metrics]
   189    # tcp address!
   190    address = "127.0.0.1:1234"
   191  ```
   192  
   193  ## Plugin Configuration
   194  
   195  At the end of the day, containerd's core is very small.
   196  The real functionality comes from plugins.
   197  Everything from snapshotters, runtimes, and content are all plugins that are registered at runtime.
   198  Because these various plugins are so different we need a way to provide type safe configuration to the plugins.
   199  The only way we can do this is via the config file and not CLI flags.
   200  
   201  In the config file you can specify plugin level options for the set of plugins that you use via the `[plugins.<name>]` sections.
   202  You will have to read the plugin specific docs to find the options that your plugin accepts.
   203  
   204  ### Linux Runtime Plugin
   205  
   206  The linux runtime allows a few options to be set to configure the shim and the runtime that you are using.
   207  
   208  ```toml
   209  [plugins.linux]
   210  	# shim binary name/path
   211  	shim = ""
   212  	# runtime binary name/path
   213  	runtime = "runc"
   214  	# do not use a shim when starting containers, saves on memory but
   215  	# live restore is not supported
   216  	no_shim = false
   217  	# display shim logs in the containerd daemon's log output
   218  	shim_debug = true
   219  ```
   220  
   221  ### Bolt Metadata Plugin
   222  
   223  The bolt metadata plugin allows configuration of the content sharing policy between namespaces.
   224  
   225  The default mode "shared" will make blobs available in all namespaces once it is pulled into any namespace.
   226  The blob will be pulled into the namespace if a writer is opened with the "Expected" digest that is already present in the backend.
   227  
   228  The alternative mode, "isolated" requires that clients prove they have access to the content by providing all of the content to the ingest before the blob is added to the namespace.
   229  
   230  Both modes share backing data, while "shared" will reduce total bandwidth across namespaces, at the cost of allowing access to any blob just by knowing its digest.
   231  
   232  The default is "shared". While this is largely the most desired policy, one can change to "isolated" mode with the following configuration:
   233  
   234  ```toml
   235  [plugins.bolt]
   236  	content_sharing_policy = "isolated"
   237  ```