github.com/brandond/cni@v0.8.1/SPEC.md (about)

     1  # Container Network Interface Specification
     2  
     3  - [Container Network Interface Specification](#container-network-interface-specification)
     4    - [Version](#version)
     5        - [Released versions](#released-versions)
     6    - [Overview](#overview)
     7    - [General considerations](#general-considerations)
     8    - [CNI Plugin](#cni-plugin)
     9      - [Overview](#overview-1)
    10      - [Parameters](#parameters)
    11      - [Result](#result)
    12      - [Network Configuration](#network-configuration)
    13      - [Example configurations](#example-configurations)
    14        - [Example bridge configuration](#example-bridge-configuration)
    15        - [Example ovs configuration](#example-ovs-configuration)
    16        - [Example macvlan configuration](#example-macvlan-configuration)
    17      - [Network Configuration Lists](#network-configuration-lists)
    18        - [Network Configuration List Error Handling](#network-configuration-list-error-handling)
    19        - [Example network configuration lists](#example-network-configuration-lists)
    20        - [Network configuration list runtime examples](#network-configuration-list-runtime-examples)
    21      - [IP Allocation](#ip-allocation)
    22        - [IP Address Management (IPAM) Interface](#ip-address-management-ipam-interface)
    23        - [Notes](#notes)
    24      - [Well-known Structures](#well-known-structures)
    25        - [IPs](#ips)
    26        - [Routes](#routes)
    27        - [DNS](#dns)
    28    - [Well-known Error Codes](#well-known-error-codes)
    29  
    30  ## Version
    31  
    32  This is CNI **spec** version **0.4.0**.
    33  
    34  Note that this is **independent from the version of the CNI library and plugins** in this repository (e.g. the versions of [releases](https://github.com/containernetworking/cni/releases)).
    35  
    36  #### Released versions
    37  
    38  Released versions of the spec are available as Git tags.
    39  
    40  | tag                                                                                  | spec permalink                                                                        | major changes                     |
    41  | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | --------------------------------- |
    42  | [`spec-v0.4.0`](https://github.com/containernetworking/cni/releases/tag/spec-v0.4.0) | [spec at v0.4.0](https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md) | Introduce the CHECK command and passing prevResult on DEL |
    43  | [`spec-v0.3.1`](https://github.com/containernetworking/cni/releases/tag/spec-v0.3.1) | [spec at v0.3.1](https://github.com/containernetworking/cni/blob/spec-v0.3.1/SPEC.md) | none (typo fix only)              |
    44  | [`spec-v0.3.0`](https://github.com/containernetworking/cni/releases/tag/spec-v0.3.0) | [spec at v0.3.0](https://github.com/containernetworking/cni/blob/spec-v0.3.0/SPEC.md) | rich result type, plugin chaining |
    45  | [`spec-v0.2.0`](https://github.com/containernetworking/cni/releases/tag/spec-v0.2.0) | [spec at v0.2.0](https://github.com/containernetworking/cni/blob/spec-v0.2.0/SPEC.md) | VERSION command                   |
    46  | [`spec-v0.1.0`](https://github.com/containernetworking/cni/releases/tag/spec-v0.1.0) | [spec at v0.1.0](https://github.com/containernetworking/cni/blob/spec-v0.1.0/SPEC.md) | initial version                   |
    47  
    48  *Do not rely on these tags being stable.  In the future, we may change our mind about which particular commit is the right marker for a given historical spec version.*
    49  
    50  
    51  ## Overview
    52  
    53  This document proposes a generic plugin-based networking solution for application containers on Linux, the _Container Networking Interface_, or _CNI_.
    54  It is derived from the rkt Networking Proposal, which aimed to satisfy many of the design considerations for networking in [rkt][rkt-github].
    55  
    56  For the purposes of this proposal, we define two terms very specifically:
    57  - _container_ can be considered synonymous with a [Linux _network namespace_][namespaces]. What unit this corresponds to depends on a particular container runtime implementation: for example, in implementations of the [App Container Spec][appc-github] like rkt, each _pod_ runs in a unique network namespace. In [Docker][docker], on the other hand, network namespaces generally exist for each separate Docker container.
    58  - _network_ refers to a group of entities that are uniquely addressable that can communicate amongst each other. This could be either an individual container (as specified above), a machine, or some other network device (e.g. a router). Containers can be conceptually _added to_ or _removed from_ one or more networks.
    59  
    60  This document aims to specify the interface between "runtimes" and "plugins". Whilst there are certain well known fields, runtimes may wish to pass additional information to plugins. These extensions are not part of this specification but are documented as [conventions](CONVENTIONS.md). The key words "must", "must not", "required", "shall", "shall not", "should", "should not", "recommended", "may" and "optional" are used as specified in [RFC 2119][rfc-2119].
    61  
    62  [rkt-github]: https://github.com/coreos/rkt
    63  [namespaces]: http://man7.org/linux/man-pages/man7/namespaces.7.html 
    64  [appc-github]: https://github.com/appc/spec
    65  [docker]: https://docker.com
    66  [rfc-2119]: https://www.ietf.org/rfc/rfc2119.txt
    67  
    68  ## General considerations
    69  
    70  - The container runtime must create a new network namespace for the container before invoking any plugins.
    71  - The runtime must then determine which networks this container should belong to, and for each network, which plugins must be executed.
    72  - The network configuration is in JSON format and can easily be stored in a file. The network configuration includes mandatory fields such as "name" and "type" as well as plugin (type) specific ones. The network configuration allows for fields to change values between invocations. For this purpose there is an optional field "args" which must contain the varying information.
    73  - The container runtime must add the container to each network by executing the corresponding plugins for each network sequentially.
    74  - Upon completion of the container lifecycle, the runtime must execute the plugins in reverse order (relative to the order in which they were executed to add the container) to disconnect the container from the networks.
    75  - The container runtime must not invoke parallel operations for the same container, but is allowed to invoke parallel operations for different containers.
    76  - The container runtime must order ADD and DEL operations for a container, such that ADD is always eventually followed by a corresponding DEL. DEL may be followed by additional DELs but plugins should handle multiple DELs permissively (i.e. plugin DEL should be idempotent).
    77  - A container must be uniquely identified by a ContainerID. Plugins that store state should do so using a primary key of `(network name, CNI_CONTAINERID, CNI_IFNAME)`.
    78  - A runtime must not call ADD twice (without a corresponding DEL) for the same `(network name, container id, name of the interface inside the container)`. This implies that a given container ID may be added to a specific network more than once only if each addition is done with a different interface name.
    79  - Fields in CNI structures (like [Network Configuration](#network-configuration) and [CNI Plugin Result](#result)) are required unless specifically marked optional.
    80  
    81  ## CNI Plugin
    82  
    83  ### Overview
    84  
    85  Each CNI plugin must be implemented as an executable that is invoked by the container management system (e.g. rkt or Kubernetes).
    86  
    87  A CNI plugin is responsible for inserting a network interface into the container network namespace (e.g. one end of a veth pair) and making any necessary changes on the host (e.g. attaching the other end of the veth into a bridge).
    88  It should then assign the IP to the interface and setup the routes consistent with the IP Address Management section by invoking appropriate IPAM plugin.
    89  
    90  ### Parameters
    91  
    92  The operations that CNI plugins must support are:
    93  
    94  - `ADD`: Add container to network
    95    - Parameters:
    96      - **Container ID**. A unique plaintext identifier for a container, allocated by the runtime. Must not be empty.  Must start with a alphanumeric character, optionally followed by any combination of one or more alphanumeric characters, underscore (_), dot (.) or hyphen (-).
    97      - **Network namespace path**. This represents the path to the network namespace to be added, i.e. /proc/[pid]/ns/net or a bind-mount/link to it.
    98      - **Network configuration**. This is a JSON document describing a network to which a container can be joined. The schema is described below.
    99      - **Extra arguments**. This provides an alternative mechanism to allow simple configuration of CNI plugins on a per-container basis.
   100      - **Name of the interface inside the container**. This is the name that should be assigned to the interface created inside the container (network namespace); consequently it must comply with the standard Linux restrictions on interface names, must not be empty, must not be "." or "..", must be less than 16 characters and must not contain / or : or any whitespace characters.
   101    - Result:
   102      - **Interfaces list**. Depending on the plugin, this can include the sandbox (eg, container or hypervisor) interface name and/or the host interface name, the hardware addresses of each interface, and details about the sandbox (if any) the interface is in.
   103      - **IP configuration assigned to each interface**. The IPv4 and/or IPv6 addresses, gateways, and routes assigned to sandbox and/or host interfaces.
   104      - **DNS information**. Dictionary that includes DNS information for nameservers, domain, search domains and options.
   105  
   106  - `DEL`: Delete container from network
   107    - Parameters:
   108      - **Container ID**, as defined above.
   109      - **Network namespace path**, as defined above.
   110      - **Network configuration**, as defined above.
   111      - **Extra arguments**, as defined above.
   112      - **Name of the interface inside the container**, as defined above.
   113    - All parameters should be the same as those passed to the corresponding add operation.
   114    - A delete operation should release all resources held by the supplied containerid in the configured network.
   115    - If there was a known previous `ADD` action for the container, the runtime MUST add a `prevResult` field to the configuration JSON of the plugin (or all plugins in a chain), which MUST be the `Result` of the immediately previous `ADD` action in JSON format ([see below](#network-configuration-list-runtime-examples)).  The runtime may wish to use libcni's support for caching `Result`s.
   116    - When `CNI_NETNS` and/or `prevResult` are not provided, the plugin should clean up as many resources as possible (e.g. releasing IPAM allocations) and return a successful response.
   117    - If the runtime cached the `Result` of a previous `ADD` response for a given container, it must delete that cached response on a successful `DEL` for that container.
   118  
   119  Plugins should generally complete a `DEL` action without error even if some resources are missing.  For example, an IPAM plugin should generally release an IP allocation and return success even if the container network namespace no longer exists, unless that network namespace is critical for IPAM management. While DHCP may usually send a 'release' message on the container network interface, since DHCP leases have a lifetime this release action would not be considered critical and no error should be returned. For another example, the `bridge` plugin should delegate the DEL action to the IPAM plugin and clean up its own resources (if present) even if the container network namespace and/or container network interface no longer exist.
   120  
   121  - `CHECK`: Check container's networking is as expected
   122    - Parameters:
   123      - **Container ID**, as defined for `ADD`.
   124      - **Network namespace path**, as defined for `ADD`.
   125      - **Network configuration** as defined for `ADD`, which must include a `prevResult` field containing the `Result` of the immediately preceding `ADD` for the container.
   126      - **Extra arguments**, as defined for `ADD`.
   127      - **Name of the interface inside the container**, as defined for `ADD`.
   128    - Result:
   129      - The plugin must return either nothing or an error.
   130    - The plugin must consult the `prevResult` to determine the expected interfaces and addresses.
   131    - The plugin must allow for a later chained plugin to have modified networking resources, e.g. routes.
   132    - The plugin should return an error if a resource included in the CNI Result type (interface, address or route) was created by the plugin, and is listed in `prevResult`, but is missing or in an invalid state.
   133    - The plugin should return an error if other resources not tracked in the Result type such as the following are missing or are in an invalid state:
   134      - Firewall rules
   135      - Traffic shaping controls
   136      - IP reservations
   137      - External dependencies such as a daemon required for connectivity
   138      - etc.
   139    - The plugin should return an error if it is aware of a condition where the container is generally unreachable.
   140    - The plugin must handle `CHECK` being called immediately after an `ADD`, and therefore should allow a reasonable convergence delay for any asynchronous resources.
   141    - The plugin should call `CHECK` on any delegated (e.g. IPAM) plugins and pass any errors on to its caller.
   142    - A runtime must not call `CHECK` for a container that has not been `ADD`ed, or has been `DEL`eted after its last `ADD`.
   143    - A runtime must not call `CHECK` if `disableCheck` is set to `true` in the [configuration list](#network-configuration-lists).
   144    - A runtime must include a `prevResult` field in the network configuration containing the `Result` of the immediately preceding `ADD` for the container. The runtime may wish to use libcni's support for caching `Result`s.
   145    - A runtime may choose to stop executing `CHECK` for a chain when a plugin returns an error.
   146    - A runtime may execute `CHECK` from immediately after a successful `ADD`, up until the container is `DEL`eted from the network.
   147    - A runtime may assume that a failed `CHECK` means the container is permanently in a misconfigured state.
   148  
   149  - `VERSION`: Report version
   150    - Parameters: NONE.
   151    - Result: information about the CNI spec versions supported by the plugin
   152  
   153        ```
   154        {
   155          "cniVersion": "0.4.0", // the version of the CNI spec in use for this output
   156          "supportedVersions": [ "0.1.0", "0.2.0", "0.3.0", "0.3.1", "0.4.0" ] // the list of CNI spec versions that this plugin supports
   157        }
   158        ```
   159  
   160  Runtimes must use the type of network (see [Network Configuration](#network-configuration) below) as the name of the executable to invoke.
   161  Runtimes should then look for this executable in a list of predefined directories (the list of directories is not prescribed by this specification). Once found, it must invoke the executable using the following environment variables for argument passing:
   162  
   163  - `CNI_COMMAND`: indicates the desired operation; `ADD`, `DEL`, `CHECK`, or `VERSION`.
   164  - `CNI_CONTAINERID`: Container ID
   165  - `CNI_NETNS`: Path to network namespace file
   166  - `CNI_IFNAME`: Interface name to set up; if the plugin is unable to use this interface name it must return an error
   167  - `CNI_ARGS`: Extra arguments passed in by the user at invocation time. Alphanumeric key-value pairs separated by semicolons; for example, "FOO=BAR;ABC=123"
   168  - `CNI_PATH`: List of paths to search for CNI plugin executables. Paths are separated by an OS-specific list separator; for example ':' on Linux and ';' on Windows
   169  
   170  Network configuration in JSON format must be streamed to the plugin through stdin. This means it is not tied to a particular file on disk and may contain information which changes between invocations.
   171  
   172  
   173  ### Result
   174  
   175  Note that IPAM plugins should return an abbreviated `Result` structure as described in [IP Allocation](#ip-allocation).
   176  
   177  Plugins must indicate success with a return code of zero and the following JSON printed to stdout in the case of the ADD command. The `ips` and `dns` items should be the same output as was returned by the IPAM plugin (see [IP Allocation](#ip-allocation) for details) except that the plugin should fill in the `interface` indexes appropriately, which are missing from IPAM plugin output since IPAM plugins should be unaware of interfaces.
   178  
   179  ```
   180  {
   181    "cniVersion": "0.4.0",
   182    "interfaces": [                                            (this key omitted by IPAM plugins)
   183        {
   184            "name": "<name>",
   185            "mac": "<MAC address>",                            (required if L2 addresses are meaningful)
   186            "sandbox": "<netns path or hypervisor identifier>" (required for container/hypervisor interfaces, empty/omitted for host interfaces)
   187        }
   188    ],
   189    "ips": [
   190        {
   191            "version": "<4-or-6>",
   192            "address": "<ip-and-prefix-in-CIDR>",
   193            "gateway": "<ip-address-of-the-gateway>",          (optional)
   194            "interface": <numeric index into 'interfaces' list>
   195        },
   196        ...
   197    ],
   198    "routes": [                                                (optional)
   199        {
   200            "dst": "<ip-and-prefix-in-cidr>",
   201            "gw": "<ip-of-next-hop>"                           (optional)
   202        },
   203        ...
   204    ],
   205    "dns": {                                                   (optional)
   206      "nameservers": <list-of-nameservers>                     (optional)
   207      "domain": <name-of-local-domain>                         (optional)
   208      "search": <list-of-additional-search-domains>            (optional)
   209      "options": <list-of-options>                             (optional)
   210    }
   211  }
   212  ```
   213  
   214  `cniVersion` specifies a [Semantic Version 2.0](https://semver.org) of CNI specification used by the plugin. A plugin may support multiple CNI spec versions (as it reports via the `VERSION` command), here the `cniVersion` returned by the plugin in the result must be consistent with the `cniVersion` specified in [Network Configuration](#network-configuration). If the `cniVersion` in the network configuration is not supported by the plugin, the plugin should return an error code 1 (see [Well-known Error Codes](#well-known-error-codes) for details).
   215  
   216  `interfaces` describes specific network interfaces the plugin created.
   217  If the `CNI_IFNAME` variable exists the plugin must use that name for the sandbox/hypervisor interface or return an error if it cannot.
   218  - `mac` (string): the hardware address of the interface.
   219     If L2 addresses are not meaningful for the plugin then this field is optional.
   220  - `sandbox` (string): container/namespace-based environments should return the full filesystem path to the network namespace of that sandbox.
   221     Hypervisor/VM-based plugins should return an ID unique to the virtualized sandbox the interface was created in.
   222     This item must be provided for interfaces created or moved into a sandbox like a network namespace or a hypervisor/VM.
   223  
   224  The `ips` field is a list of IP configuration information.
   225  See the [IP well-known structure](#ips) section for more information.
   226  
   227  The `routes` field is a list of route configuration information.
   228  See the [Routes well-known structure](#routes) section for more information.
   229  
   230  The `dns` field contains a dictionary consisting of common DNS information.
   231  See the [DNS well-known structure](#dns) section for more information.
   232  
   233  The specification does not declare how this information must be processed by CNI consumers.
   234  Examples include generating an `/etc/resolv.conf` file to be injected into the container filesystem or running a DNS forwarder on the host.
   235  
   236  Errors must be indicated by a non-zero return code and the following JSON being printed to stdout:
   237  ```
   238  {
   239    "cniVersion": "0.4.0",
   240    "code": <numeric-error-code>,
   241    "msg": <short-error-message>,
   242    "details": <long-error-message> (optional)
   243  }
   244  ```
   245  
   246  `cniVersion` specifies a [Semantic Version 2.0](https://semver.org) of CNI specification used by the plugin.
   247  Error codes 0-99 are reserved for well-known errors (see [Well-known Error Codes](#well-known-error-codes) section).
   248  Values of 100+ can be freely used for plugin specific errors. 
   249  
   250  In addition, stderr can be used for unstructured output such as logs.
   251  
   252  ### Network Configuration
   253  
   254  The network configuration is described in JSON form. The configuration may be stored on disk or generated from other sources by the container runtime. The following fields are well-known and have the following meaning:
   255  - `cniVersion` (string): [Semantic Version 2.0](https://semver.org) of CNI specification to which this configuration conforms.
   256  - `name` (string): Network name. This should be unique across all containers on the host (or other administrative domain).  Must start with a alphanumeric character, optionally followed by any combination of one or more alphanumeric characters, underscore (_), dot (.) or hyphen (-).
   257  - `type` (string): Refers to the filename of the CNI plugin executable.
   258  - `args` (dictionary, optional): Additional arguments provided by the container runtime. For example a dictionary of labels could be passed to CNI plugins by adding them to a labels field under `args`.
   259  - `ipMasq` (boolean, optional): If supported by the plugin, sets up an IP masquerade on the host for this network. This is necessary if the host will act as a gateway to subnets that are not able to route to the IP assigned to the container.
   260  - `ipam` (dictionary, optional): Dictionary with IPAM specific values:
   261    - `type` (string): Refers to the filename of the IPAM plugin executable.
   262  - `dns` (dictionary, optional): Dictionary with DNS specific values:
   263    - `nameservers` (list of strings, optional): list of a priority-ordered list of DNS nameservers that this network is aware of. Each entry in the list is a string containing either an IPv4 or an IPv6 address.
   264    - `domain` (string, optional): the local domain used for short hostname lookups.
   265    - `search` (list of strings, optional): list of priority ordered search domains for short hostname lookups. Will be preferred over `domain` by most resolvers.
   266    - `options` (list of strings, optional): list of options that can be passed to the resolver
   267  
   268  Plugins may define additional fields that they accept and may generate an error if called with unknown fields. The exception to this is the `args` field may be used to pass arbitrary data which should be ignored by plugins if not understood.
   269  
   270  ### Example configurations
   271  
   272  #### Example bridge configuration
   273  
   274  ```jsonc
   275  {
   276    "cniVersion": "0.4.0",
   277    "name": "dbnet",
   278    "type": "bridge",
   279    // type (plugin) specific
   280    "bridge": "cni0",
   281    "ipam": {
   282      "type": "host-local",
   283      // ipam specific
   284      "subnet": "10.1.0.0/16",
   285      "gateway": "10.1.0.1"
   286    },
   287    "dns": {
   288      "nameservers": [ "10.1.0.1" ]
   289    }
   290  }
   291  ```
   292  
   293  #### Example ovs configuration
   294  
   295  ```jsonc
   296  {
   297    "cniVersion": "0.4.0",
   298    "name": "pci",
   299    "type": "ovs",
   300    // type (plugin) specific
   301    "bridge": "ovs0",
   302    "vxlanID": 42,
   303    "ipam": {
   304      "type": "dhcp",
   305      "routes": [ { "dst": "10.3.0.0/16" }, { "dst": "10.4.0.0/16" } ]
   306    },
   307    // args may be ignored by plugins
   308    "args": {
   309      "labels" : {
   310          "appVersion" : "1.0"
   311      }
   312    }
   313  }
   314  ```
   315  
   316  #### Example macvlan configuration
   317  
   318  ```jsonc
   319  {
   320    "cniVersion": "0.4.0",
   321    "name": "wan",
   322    "type": "macvlan",
   323    // ipam specific
   324    "ipam": {
   325      "type": "dhcp",
   326      "routes": [ { "dst": "10.0.0.0/8", "gw": "10.0.0.1" } ]
   327    },
   328    "dns": {
   329      "nameservers": [ "10.0.0.1" ]
   330    }
   331  }
   332  ```
   333  
   334  ### Network Configuration Lists
   335  
   336  Network configuration lists provide a mechanism to run multiple CNI plugins for a single container in a defined order, passing the result of each plugin to the next plugin.
   337  The list is composed of well-known fields and list of one or more standard CNI network configurations (see above).
   338  
   339  The list is described in JSON form, and can be stored on disk or generated from other sources by the container runtime. The following fields are well-known and have the following meaning:
   340  
   341  - `cniVersion` (string): [Semantic Version 2.0](https://semver.org) of CNI specification to which this configuration list and all the individual configurations conform.
   342  - `name` (string): Network name. This should be unique across all containers on the host (or other administrative domain).  Must start with a alphanumeric character, optionally followed by any combination of one or more alphanumeric characters, underscore (_), dot (.) or hyphen (-).
   343  - `disableCheck` (string): Either `true` or `false`.  If `disableCheck` is `true`, runtimes must not call `CHECK` for this network configuration list.  This allows an administrator to prevent `CHECK`ing where a combination of plugins is known to return spurious errors.
   344  - `plugins` (list): A list of standard CNI network configuration dictionaries (see above).
   345  
   346  When executing a plugin list, the runtime MUST replace the `name` and `cniVersion` fields in each individual network configuration in the list with the `name` and `cniVersion` field of the list itself. This ensures that the name and CNI version is the same for all plugin executions in the list, preventing versioning conflicts between plugins.
   347  The runtime may also pass capability-based keys as a map in the top-level `runtimeConfig` key of the plugin's config JSON if a plugin advertises it supports a specific capability via the `capabilities` key of its network configuration.  The key passed in `runtimeConfig` MUST match the name of the specific capability from the `capabilities` key of the plugins network configuration. See CONVENTIONS.md for more information on capabilities and how they are sent to plugins via the `runtimeConfig` key.
   348  
   349  For the `ADD` action, the runtime MUST also add a `prevResult` field to the configuration JSON of any plugin after the first one, which MUST be the `Result` of the previous plugin (if any) in JSON format ([see below](#network-configuration-list-runtime-examples)).
   350  For the `CHECK` and `DEL` actions, the runtime MUST (except that it may be omitted for `DEL` if not available) add a `prevResult` field to the configuration JSON of each plugin, which MUST be the `Result` of the immediately previous `ADD` action in JSON format ([see below](#network-configuration-list-runtime-examples)).
   351  For the `ADD` action, plugins SHOULD echo the contents of the `prevResult` field to their stdout to allow subsequent plugins (and the runtime) to receive the result, unless they wish to modify or suppress a previous result.
   352  Plugins are allowed to modify or suppress all or part of a `prevResult`.
   353  However, plugins that support a version of the CNI specification that includes the `prevResult` field MUST handle `prevResult` by either passing it through, modifying it, or suppressing it explicitly.
   354  It is a violation of this specification to be unaware of the `prevResult` field.
   355  
   356  The runtime MUST also execute each plugin in the list with the same environment.
   357  
   358  For the `DEL` action, the runtime MUST execute the plugins in reverse-order.
   359  
   360  #### Network Configuration List Error Handling
   361  
   362  When an error occurs while executing an action on a plugin list (eg, either `ADD` or `DEL`) the runtime MUST stop execution of the list.
   363  
   364  If an `ADD` action fails, when the runtime decides to handle the failure it should execute the `DEL` action (in reverse order from the `ADD` as specified above) for all plugins in the list, even if some were not called during the `ADD` action.
   365  
   366  #### Example network configuration lists
   367  
   368  ```jsonc
   369  {
   370    "cniVersion": "0.4.0",
   371    "name": "dbnet",
   372    "plugins": [
   373      {
   374        "type": "bridge",
   375        // type (plugin) specific
   376        "bridge": "cni0",
   377        // args may be ignored by plugins
   378        "args": {
   379          "labels" : {
   380              "appVersion" : "1.0"
   381          }
   382        },
   383        "ipam": {
   384          "type": "host-local",
   385          // ipam specific
   386          "subnet": "10.1.0.0/16",
   387          "gateway": "10.1.0.1"
   388        },
   389        "dns": {
   390          "nameservers": [ "10.1.0.1" ]
   391        }
   392      },
   393      {
   394        "type": "tuning",
   395        "sysctl": {
   396          "net.core.somaxconn": "500"
   397        }
   398      }
   399    ]
   400  }
   401  ```
   402  
   403  #### Network configuration list runtime examples
   404  
   405  Given the network configuration list JSON [shown above](#example-network-configuration-lists) the container runtime would perform the following steps for the `ADD` action.
   406  Note that the runtime adds the `cniVersion` and `name` fields from configuration list to the configuration JSON passed to each plugin, to ensure consistent versioning and names for all plugins in the list.
   407  
   408  1) first call the `bridge` plugin with the following JSON:
   409  
   410  ```jsonc
   411  {
   412    "cniVersion": "0.4.0",
   413    "name": "dbnet",
   414    "type": "bridge",
   415    "bridge": "cni0",
   416    "args": {
   417      "labels" : {
   418          "appVersion" : "1.0"
   419      }
   420    },
   421    "ipam": {
   422      "type": "host-local",
   423      // ipam specific
   424      "subnet": "10.1.0.0/16",
   425      "gateway": "10.1.0.1"
   426    },
   427    "dns": {
   428      "nameservers": [ "10.1.0.1" ]
   429    }
   430  }
   431  ```
   432  
   433  2) next call the `tuning` plugin with the following JSON, including the `prevResult` field containing the JSON response from the `bridge` plugin:
   434  
   435  ```json
   436  {
   437    "cniVersion": "0.4.0",
   438    "name": "dbnet",
   439    "type": "tuning",
   440    "sysctl": {
   441      "net.core.somaxconn": "500"
   442    },
   443    "prevResult": {
   444      "ips": [
   445          {
   446            "version": "4",
   447            "address": "10.0.0.5/32",
   448            "interface": 2
   449          }
   450      ],
   451      "interfaces": [
   452          {
   453              "name": "cni0",
   454              "mac": "00:11:22:33:44:55"
   455          },
   456          {
   457              "name": "veth3243",
   458              "mac": "55:44:33:22:11:11"
   459          },
   460          {
   461              "name": "eth0",
   462              "mac": "99:88:77:66:55:44",
   463              "sandbox": "/var/run/netns/blue"
   464          }
   465      ],
   466      "dns": {
   467        "nameservers": [ "10.1.0.1" ]
   468      }
   469    }
   470  }
   471  ```
   472  
   473  Given the same network configuration JSON list, the container runtime would perform the following steps for the `CHECK` action.
   474  
   475  1) first call the `bridge` plugin with the following JSON, including the `prevResult` field containing the JSON response from the `ADD` operation:
   476  
   477  ```jsonc
   478  {
   479    "cniVersion": "0.4.0",
   480    "name": "dbnet",
   481    "type": "bridge",
   482    "bridge": "cni0",
   483    "args": {
   484      "labels" : {
   485          "appVersion" : "1.0"
   486      }
   487    },
   488    "ipam": {
   489      "type": "host-local",
   490      // ipam specific
   491      "subnet": "10.1.0.0/16",
   492      "gateway": "10.1.0.1"
   493    },
   494    "dns": {
   495      "nameservers": [ "10.1.0.1" ]
   496    },
   497    "prevResult": {
   498      "ips": [
   499          {
   500            "version": "4",
   501            "address": "10.0.0.5/32",
   502            "interface": 2
   503          }
   504      ],
   505      "interfaces": [
   506          {
   507              "name": "cni0",
   508              "mac": "00:11:22:33:44:55"
   509          },
   510          {
   511              "name": "veth3243",
   512              "mac": "55:44:33:22:11:11"
   513          },
   514          {
   515              "name": "eth0",
   516              "mac": "99:88:77:66:55:44",
   517              "sandbox": "/var/run/netns/blue"
   518          }
   519      ],
   520      "dns": {
   521        "nameservers": [ "10.1.0.1" ]
   522      }
   523    }
   524  }
   525  ```
   526  
   527  2) next call the `tuning` plugin with the following JSON, including the `prevResult` field containing the JSON response from the `ADD` operation:
   528  
   529  ```json
   530  {
   531    "cniVersion": "0.4.0",
   532    "name": "dbnet",
   533    "type": "tuning",
   534    "sysctl": {
   535      "net.core.somaxconn": "500"
   536    },
   537    "prevResult": {
   538      "ips": [
   539          {
   540            "version": "4",
   541            "address": "10.0.0.5/32",
   542            "interface": 2
   543          }
   544      ],
   545      "interfaces": [
   546          {
   547              "name": "cni0",
   548              "mac": "00:11:22:33:44:55"
   549          },
   550          {
   551              "name": "veth3243",
   552              "mac": "55:44:33:22:11:11"
   553          },
   554          {
   555              "name": "eth0",
   556              "mac": "99:88:77:66:55:44",
   557              "sandbox": "/var/run/netns/blue"
   558          }
   559      ],
   560      "dns": {
   561        "nameservers": [ "10.1.0.1" ]
   562      }
   563    }
   564  }
   565  ```
   566  
   567  Given the same network configuration JSON list, the container runtime would perform the following steps for the `DEL` action.
   568  Note that plugins are executed in reverse order from the `ADD` and `CHECK` actions.
   569  
   570  1) first call the `tuning` plugin with the following JSON, including the `prevResult` field containing the JSON response from the `ADD` action:
   571  
   572  ```json
   573  {
   574    "cniVersion": "0.4.0",
   575    "name": "dbnet",
   576    "type": "tuning",
   577    "sysctl": {
   578      "net.core.somaxconn": "500"
   579    },
   580    "prevResult": {
   581      "ips": [
   582          {
   583            "version": "4",
   584            "address": "10.0.0.5/32",
   585            "interface": 2
   586          }
   587      ],
   588      "interfaces": [
   589          {
   590              "name": "cni0",
   591              "mac": "00:11:22:33:44:55"
   592          },
   593          {
   594              "name": "veth3243",
   595              "mac": "55:44:33:22:11:11"
   596          },
   597          {
   598              "name": "eth0",
   599              "mac": "99:88:77:66:55:44",
   600              "sandbox": "/var/run/netns/blue"
   601          }
   602      ],
   603      "dns": {
   604        "nameservers": [ "10.1.0.1" ]
   605      }
   606    }
   607  }
   608  ```
   609  
   610  2) next call the `bridge` plugin with the following JSON, including the `prevResult` field containing the JSON response from the `ADD` action:
   611  
   612  ```jsonc
   613  {
   614    "cniVersion": "0.4.0",
   615    "name": "dbnet",
   616    "type": "bridge",
   617    "bridge": "cni0",
   618    "args": {
   619      "labels" : {
   620          "appVersion" : "1.0"
   621      }
   622    },
   623    "ipam": {
   624      "type": "host-local",
   625      // ipam specific
   626      "subnet": "10.1.0.0/16",
   627      "gateway": "10.1.0.1"
   628    },
   629    "dns": {
   630      "nameservers": [ "10.1.0.1" ]
   631    },
   632    "prevResult": {
   633      "ips": [
   634          {
   635            "version": "4",
   636            "address": "10.0.0.5/32",
   637            "interface": 2
   638          }
   639      ],
   640      "interfaces": [
   641          {
   642              "name": "cni0",
   643              "mac": "00:11:22:33:44:55"
   644          },
   645          {
   646              "name": "veth3243",
   647              "mac": "55:44:33:22:11:11"
   648          },
   649          {
   650              "name": "eth0",
   651              "mac": "99:88:77:66:55:44",
   652              "sandbox": "/var/run/netns/blue"
   653          }
   654      ],
   655      "dns": {
   656        "nameservers": [ "10.1.0.1" ]
   657      }
   658    }
   659  }
   660  ```
   661  
   662  ### IP Allocation
   663  
   664  As part of its operation, a CNI plugin is expected to assign (and maintain) an IP address to the interface and install any necessary routes relevant for that interface. This gives the CNI plugin great flexibility but also places a large burden on it. Many CNI plugins would need to have the same code to support several IP management schemes that users may desire (e.g. dhcp, host-local). 
   665  
   666  To lessen the burden and make IP management strategy be orthogonal to the type of CNI plugin, we define a second type of plugin -- IP Address Management Plugin (IPAM plugin). It is however the responsibility of the CNI plugin to invoke the IPAM plugin at the proper moment in its execution. The IPAM plugin must determine the interface IP/subnet, Gateway and Routes and return this information to the "main" plugin to apply. The IPAM plugin may obtain the information via a protocol (e.g. dhcp), data stored on a local filesystem, the "ipam" section of the Network Configuration file or a combination of the above.
   667  
   668  #### IP Address Management (IPAM) Interface
   669  
   670  Like CNI plugins, the IPAM plugins are invoked by running an executable. The executable is searched for in a predefined list of paths, indicated to the CNI plugin via `CNI_PATH`. The IPAM Plugin must receive all the same environment variables that were passed in to the CNI plugin. Just like the CNI plugin, IPAM plugins receive the network configuration via stdin.
   671  
   672  Success must be indicated by a zero return code and the following JSON being printed to stdout (in the case of the ADD command):
   673  
   674  ```
   675  {
   676    "cniVersion": "0.4.0",
   677    "ips": [
   678        {
   679            "version": "<4-or-6>",
   680            "address": "<ip-and-prefix-in-CIDR>",
   681            "gateway": "<ip-address-of-the-gateway>"  (optional)
   682        },
   683        ...
   684    ],
   685    "routes": [                                       (optional)
   686        {
   687            "dst": "<ip-and-prefix-in-cidr>",
   688            "gw": "<ip-of-next-hop>"                  (optional)
   689        },
   690        ...
   691    ]
   692    "dns": {                                          (optional)
   693      "nameservers": <list-of-nameservers>            (optional)
   694      "domain": <name-of-local-domain>                (optional)
   695      "search": <list-of-search-domains>              (optional)
   696      "options": <list-of-options>                    (optional)
   697    }
   698  }
   699  ```
   700  
   701  Note that unlike regular CNI plugins, IPAM plugins should return an abbreviated `Result` structure that does not include the `interfaces` key, since IPAM plugins should be unaware of interfaces configured by their parent plugin except those specifically required for IPAM (eg, like the `dhcp` IPAM plugin).
   702  
   703  `cniVersion` specifies a [Semantic Version 2.0](https://semver.org) of CNI specification used by the IPAM plugin. An IPAM plugin may support multiple CNI spec versions (as it reports via the `VERSION` command), here the `cniVersion` returned by the IPAM plugin in the result must be consistent with the `cniVersion` specified in [Network Configuration](#network-configuration). If the `cniVersion` in the network configuration is not supported by the IPAM plugin, the plugin should return an error code 1 (see [Well-known Error Codes](#well-known-error-codes) for details).
   704  
   705  The `ips` field is a list of IP configuration information.
   706  See the [IP well-known structure](#ips) section for more information.
   707  
   708  The `routes` field is a list of route configuration information.
   709  See the [Routes well-known structure](#routes) section for more information.
   710  
   711  The `dns` field contains a dictionary consisting of common DNS information.
   712  See the [DNS well-known structure](#dns) section for more information.
   713  
   714  Errors and logs are communicated in the same way as the CNI plugin. See [CNI Plugin Result](#result) section for details.
   715  
   716  IPAM plugin examples:
   717   - **host-local**: Select an unused (by other containers on the same host) IP within the specified range.
   718   - **dhcp**: Use DHCP protocol to acquire and maintain a lease. The DHCP requests will be sent via the created container interface; therefore, the associated network must support broadcast.
   719  
   720  #### Notes
   721   - Routes are expected to be added with a 0 metric.
   722   - A default route may be specified via "0.0.0.0/0". Since another network might have already configured the default route, the CNI plugin should be prepared to skip over its default route definition.
   723  
   724  ### Well-known Structures
   725  
   726  #### IPs
   727  
   728  ```
   729    "ips": [
   730        {
   731            "version": "<4-or-6>",
   732            "address": "<ip-and-prefix-in-CIDR>",
   733            "gateway": "<ip-address-of-the-gateway>",      (optional)
   734            "interface": <numeric index into 'interfaces' list> (not required for IPAM plugins)
   735        },
   736        ...
   737    ]
   738  ```
   739  
   740  The `ips` field is a list of IP configuration information determined by the plugin. Each item is a dictionary describing of IP configuration for a network interface.
   741  IP configuration for multiple network interfaces and multiple IP configurations for a single interface may be returned as separate items in the `ips` list.
   742  All properties known to the plugin should be provided, even if not strictly required.
   743  - `version` (string): either "4" or "6" and corresponds to the IP version of the addresses in the entry.
   744     All IP addresses and gateways provided must be valid for the given `version`.
   745  - `address` (string): an IP address in CIDR notation (eg "192.168.1.3/24").
   746  - `gateway` (string): the default gateway for this subnet, if one exists.
   747     It does not instruct the CNI plugin to add any routes with this gateway: routes to add are specified separately via the `routes` field.
   748     An example use of this value is for the CNI `bridge` plugin to add this IP address to the Linux bridge to make it a gateway.
   749  - `interface` (uint): the index into the `interfaces` list for a [CNI Plugin Result](#result) indicating which interface this IP configuration should be applied to.
   750     IPAM plugins should not return this key since they have no information about network interfaces.
   751  
   752  #### Routes
   753  
   754  ```
   755    "routes": [
   756        {
   757            "dst": "<ip-and-prefix-in-cidr>",
   758            "gw": "<ip-of-next-hop>"               (optional)
   759        },
   760        ...
   761    ]
   762  ```
   763  
   764  Each `routes` entry is a dictionary with the following fields.  All IP addresses in the `routes` entry must be the same IP version, either 4 or 6.
   765    - `dst` (string): destination subnet specified in CIDR notation.
   766    - `gw` (string): IP of the gateway. If omitted, a default gateway is assumed (as determined by the CNI plugin).
   767  
   768  Each `routes` entry must be relevant for the sandbox interface specified by CNI_IFNAME.
   769  
   770  #### DNS
   771  
   772  ```
   773    "dns": {
   774      "nameservers": <list-of-nameservers>                 (optional)
   775      "domain": <name-of-local-domain>                     (optional)
   776      "search": <list-of-additional-search-domains>        (optional)
   777      "options": <list-of-options>                         (optional)
   778    }
   779  ```
   780  
   781  The `dns` field contains a dictionary consisting of common DNS information.
   782  - `nameservers` (list of strings): list of a priority-ordered list of DNS nameservers that this network is aware of. Each entry in the list is a string containing either an IPv4 or an IPv6 address.
   783  - `domain` (string): the local domain used for short hostname lookups.
   784  - `search` (list of strings): list of priority ordered search domains for short hostname lookups. Will be preferred over `domain` by most resolvers.
   785  - `options` (list of strings): list of options that can be passed to the resolver.
   786    See [CNI Plugin Result](#result) section for more information.
   787    
   788  ## Well-known Error Codes
   789  
   790  Error codes 1-99 must not be used other than as specified here.
   791  
   792  Error Code|Error Description
   793  ---|---
   794  `-1`|Incompatible CNI version
   795  `-2`|Unsupported field in network configuration. The error message must contain the key and value of the unsupported field.
   796  `-3`|Container unknown or does not exist. This error implies the runtime does not need to perform any container network cleanup (for example, calling the `DEL` action on the container).
   797  `-4`|Invalid necessary environment variables, like CNI_COMMAND, CNI_CONTAINERID, etc. The error message must contain the names of invalid variables.
   798  `-5`|I/O failure. For example, failed to read network config bytes from stdin.
   799  `-6`|Failed to decode content. For example, failed to unmarshal network config from bytes or failed to decode version info from string.
   800  `-7`|Invalid network config. If some validations on network configs do not pass, this error will be raised.
   801  `-11`|Try again later. If the plugin detects some transient condition that should clear up, it can use this code to notify the runtime it should re-try the operation later.