github.com/containerd/containerd@v22.0.0-20200918172823-438c87b8e050+incompatible/design/data-flow.md (about)

     1  # Data Flow
     2  
     3  In the past, container systems have hidden the complexity of pulling container
     4  images, hiding many details and complexity. This document intends to shed light
     5  on that complexity and detail how a "pull" operation will look from the
     6  perspective of a containerd user. We use the _bundle_ as the target object in
     7  this workflow, and walk back from there to describe the full process. In this
     8  context, we describe both pulling an image and creating a bundle from that
     9  image.
    10  
    11  With containerd, we redefine the "pull" to comprise the same set of steps
    12  encompassed in prior container engines. In this model, an image defines a
    13  collection of resources that can be used to create a _bundle_. There is no
    14  specific format or object called an image. The goal of the pull is to produce a
    15  set of steps is to resolve the resources that comprise an image, with the
    16  separation providing lifecycle points in the process. 
    17  
    18  A reference implementation of the complete "pull", performed client-side, will
    19  be provided as part of containerd, but there may not be a single "pull" API
    20  call.
    21  
    22  A rough diagram of the dataflow, along with the relevant components, is below.
    23  
    24  ![Data Flow](data-flow.png)
    25  
    26  While the process proceeds left to right in the diagram, this document is
    27  written right to left. By working through this process backwards, we can best
    28  understand the approach employed by containerd.
    29  
    30  ## Running a Container
    31  
    32  For containerd, we'd generally like to retrieve a _bundle_. This is the
    33  runtime, on-disk container layout, which includes the filesystem and
    34  configuration required to run the container.
    35  
    36  Generically, speaking, we can say we have the following directory:
    37  
    38  ```
    39  config.json
    40  rootfs/
    41  ```
    42  
    43  The contents of `config.json` isn't interesting in this context, but for
    44  clarity, it may be the runc config or a containerd specific configuration file
    45  for setting up a running container. The `rootfs` is a directory where
    46  containerd will setup the runtime container's filesystem.
    47  
    48  While containerd doesn't have the concept of an image, we can effectively build
    49  this structure from an image, as projected into containerd. Given this, we can
    50  say that are requirements for running a container are to do the following:
    51  
    52  1. Convert the configuration from the container image into the target format
    53     for the containerd runtime.
    54  2. Reproduce the root filesystem from the container image. While we could
    55     unpack this into `rootfs` in the bundle, we can also just pass this as a set
    56     of mounts to the container configuration.
    57  
    58  The above defines the framework in which we will operate. Put differently, we
    59  can say that we want to create a bundle by creating these two components of a
    60  bundle.
    61  
    62  ## Creating a Bundle
    63  
    64  Now that we've defined what is required to run a container, a _bundle_, we need
    65  to create one.
    66  
    67  Let's say we have the following:
    68  
    69  ```
    70  ctr run ubuntu
    71  ```
    72  
    73  This does no pulling of images. It only takes the name and creates a _bundle_.
    74  Broken down into steps, the process looks as follows:
    75  
    76  1. Lookup the digest of the image in metadata store.
    77  2. Resolve the manifest in the content store.
    78  3. Resolve the layer snapshots in the snapshot subsystem.
    79  4. Transform the config into the target bundle format.
    80  5. Create a runtime snapshot for the rootfs of the container, including resolution of mounts.
    81  6. Run the container.
    82  
    83  From this, we can understand the required resources to _pull_ an image:
    84  
    85  1. An entry in the metadata store a name pointing at a particular digest.
    86  2. The manifest must be available in the content store.
    87  3. The result of successively applied layers must be available as a snapshot.
    88  
    89  ## Unpacking Layers
    90  
    91  While this process may be pull or run driven, the idea is quite simple. For
    92  each layer, apply the result to a snapshot of the previous layer. The result
    93  should be stored under the chain id (as defined by OCI) of the resulting
    94  application.
    95  
    96  ## Pulling an Image
    97  
    98  With all the above defined, pulling an image simply becomes the following:
    99  
   100  1. Fetch the manifest for the image, verify and store it.
   101  2. Fetch each layer of the image manifest, verify and store them.
   102  3. Store the manifest digest under the provided name.
   103  
   104  Note that we leave off using the name to resolve a particular location. We'll
   105  leave that for another doc!