github.com/Cloud-Foundations/Dominator@v0.3.4/user-guide/image-manifest.md (about)

     1  # Manfest-driven Image Building
     2  The *[imaginator](../cmd/imaginator/README.md)* may be used to build complete
     3  images using *image manifsts* which describe the contents of images.
     4  
     5  ## Image Building Concepts
     6  *Image streams* are a progression of images (i.e. v0, v1, v2 and so on) over
     7  time. Every *image stream* has an associated *image manifest* which describes
     8  the components desired. In general, *image manifests* are stored in a Git
     9  repository.
    10  
    11  Images are built by taking a *SourceImage*, adding files and packages and
    12  running scripts at various stages during the process. Any image that is built by
    13  the *[imaginator](../cmd/imaginator/README.md)* can be used as the *SourceImage*
    14  for another image, leading to a tree of images. The root of the tree is a
    15  *bootstrap* image which is generated by the
    16  *[imaginator](../cmd/imaginator/README.md)*. These *bootstrap* images are built
    17  using vendor-supplied scripts such as `debootstrap` and `yumbootstrap`.
    18  
    19  If a *SourceImage* is missing (or too old (default: 1 hour)), it is
    20  automatically built. In a typical configuration, some commonly used images are
    21  automatically and periodically rebuilt. This supports fast building of
    22  application images which automatically have the latest security patches from the
    23  OS vendor.
    24  
    25  Built images are stored in an *[imageserver](../cmd/imageserver/README.md)*
    26  which de-duplicates common files and efficiently replicates the data to multiple
    27  regions if desired. Machines may be live-patched with these images using the
    28  *[dominator](../cmd/dominator/README.md)*, or tools such as the
    29  *[ami-publisher](http://bit.ly/2BgUW1C)* may be used to publish image artefacts
    30  across multiple AWS accounts and regions in a couple of minutes to create new
    31  virtual machines.
    32  
    33  ## Manifest Components
    34  The image manifest has multiple components. These are processed and combined in
    35  the following order to build the image contents.
    36  
    37  ### `manifest` file
    38  This is a required file, containing a JSON encoded object with the following
    39  fields:
    40  - `FilterLines`: an array of regular expressions matching files which should not
    41                   be included in the image
    42  - `MtimesCopyAddFilterLines`: add to the common mtime copy filter
    43  - `MtimesCopyFilterLines`: replace the common mtime copy filter
    44  - `SourceImage` (required): the name of the image stream that will be used as
    45                              the starting basis for the image to be built. The
    46  			    most recent image in the specified stream will be
    47  			    used as the base
    48  - `SourceImageBuildVariables`: key:value variables to be used if a `SourceImage`
    49                                 must be built
    50  - `SourceImageGitCommitId`: the Git Commit ID to use when searching for a source
    51                              image
    52  - `SourceImageTagsToMatch`: key:[value] tag matches to use when searching for a
    53                              source image
    54  
    55  Other fields may be present and they will be ignored by the
    56  *[imaginator](../cmd/imaginator/README.md)*. This is typically used to store
    57  other image metadata such as the tags to apply when creating an AMI (Amazon
    58  Machine Image).
    59  
    60  ### `files` directory tree
    61  If present, any files and symbolic links in this directory tree will be
    62  copied verbatim into the image (prior to installing packages), preserving the
    63  directory structure.
    64  
    65  ### `pre-install-scripts` directory
    66  An optional directory containing scripts to run prior to installing packages.
    67  These are processed in lexical order. The scripts are run in a contained
    68  environment where the root directory is the root directory of the image being
    69  built.
    70  
    71  ### `package-list` file
    72  An optional file containing a newline-separated list of packages to install. The
    73  contents of the `files` directory tree should contain any package repositories.
    74  
    75  ### `post-install-files` directory tree
    76  If present, any files and symbolic links in this directory tree will be
    77  copied verbatim into the image, preserving the directory structure.
    78  
    79  ### `scripts` directory
    80  An optional directory containing scripts to run. These are processed in lexical
    81  order. The scripts are run in a contained environment where the root directory
    82  is the root directory of the image being built.
    83  
    84  ### `post-scripts-files` directory tree
    85  If present, any files and symbolic links in this directory tree will be
    86  copied verbatim into the image (after the `scripts` are run), preserving the
    87  directory structure.
    88  
    89  ### `post-cleanup-scripts` directory
    90  An optional directory containing scripts to run after cleanup operations (such
    91  as after cleaning up downloaded packages and bind mounts). These are processed
    92  in lexical order. The scripts are run in a contained environment where the root
    93  directory is the root directory of the image being built.
    94  
    95  ### `computed-files` file
    96  An optional file listing the *computed files*. This is relevant only for images
    97  which will be lived patched onto machines with the
    98  *[dominator](../cmd/dominator/README.md)*. Each line of the file must contain
    99  the following fields:
   100  - `Filename`: the name of the file in the image
   101  - `Source`: the network address of the *file generator* that will provide the
   102              file data
   103  
   104  An alternative format file `computed-files.json` is supported, where the format
   105  is JSON data which must contain an array of objects with the above fields.
   106  
   107  ### `computed-files.add` file
   108  This is similar to the `computed-files` file, except that the list of computed
   109  files are *added* to the list of computed files in the *SourceImage*, thus
   110  inheriting and (if not empty) extending the list of computed files. This must
   111  not be present if the `computed-files` file is present.
   112  
   113  An alternative format file `computed-files.add.json` is supported, which must
   114  contain JSON data.
   115  
   116  ### `filter` file
   117  An optional file containing a newline-separated list of regular expressions
   118  matching files which should not be updated on target machines. This is relevant
   119  only for images which will be lived patched onto machines with the
   120  *[dominator](../cmd/dominator/README.md)*. This must not be present if the
   121  `filter.add` file is present.
   122  
   123  ### `filter.add` file
   124  This is similar to the `filter` file, except that the filter expressions are
   125  *added* to the filter of the *SourceImage*, thus inheriting and (if not empty)
   126  extending the filter. This must not be present if the `filter` file is present.
   127  
   128  ### `tags.json` file
   129  An optional JSON encoded file containing key:value tags to add to the image.
   130  Variable expansion is performed on the values.
   131  
   132  ### `triggers` file
   133  An optional JSON encoded file listing the *triggers* (services which should be
   134  restarted when certain files change when live patching machines). This is
   135  relevant only for images which will be lived patched onto machines with the
   136  *[dominator](../cmd/dominator/README.md)*. The JSON data must contain an array
   137  of objects with the following fields:
   138  - `MatchLines`: an array of regular expressions
   139  - `Service`: the service to restart if a file is changed which matches one of
   140    	     the regular expressions
   141  - `DoReboot`: if true, reboot the machine after restarting all services that
   142                require restarting, provided those restarts succeed
   143  - `HighImpact`: if true, restarting the service will have a high impact on the
   144    		machine (i.e. a reboot)
   145  
   146  This must not be present if the `triggers.add` file is present.
   147  
   148  ### `triggers.add` file
   149  This is similar to the `triggers` file, except that the triggers are *added*
   150  (merged) to the triggers of the *SourceImage*, thus inheriting and (if not
   151  `[]`) extending the triggers. This must not be present if the `triggers` file is
   152  present.
   153  
   154  ### `tests` directory
   155  An optional directory containing test scripts to run. These are copied into the
   156  `/tests` directory tree in the image, merging with tests from the *SourceImage*.
   157  The tests are run concurrently after the image content is built. If any test
   158  fails or exceeds the 10 second timeout, the image is not uploaded and the build
   159  fails. The scripts are run in a contained environment where the root directory
   160  is the root directory of the image that was built.