github.com/cptmikhailov/conmon@v2.0.20+incompatible/contrib/cirrus/README.md (about)

     1  # Cirrus-CI
     2  
     3  Similar to other integrated github CI/CD services, Cirrus utilizes a simple
     4  YAML-based configuration/description file: ``.cirrus.yml``.  Ref: https://cirrus-ci.org/
     5  
     6  
     7  ## Workflow
     8  
     9  All tasks execute in parallel, unless there are conditions or dependencies
    10  which alter this behavior.  Within each task, each script executes in sequence,
    11  so long as any previous script exited successfully.  The overall state of each
    12  task (pass or fail) is set based on the exit status of the last script to execute.
    13  
    14  
    15  ### ``integration`` Task
    16  
    17  1. After `gating` passes, spin up one VM per
    18     `matrix: image_name` item. Once accessible, ``ssh``
    19     into each VM as the `root` user.
    20  
    21  2. ``setup_environment.sh``: Configure root's `.bash_profile`
    22      for all subsequent scripts (each run in a new shell).  Any
    23      distribution-specific environment variables are also defined
    24      here.  For example, setting tags/flags to use compiling.
    25  
    26  5. ``integration_test.sh``: Execute integration-testing.  This is
    27     much more involved, and relies on access to external
    28     resources like container images and code from other repositories.
    29     Total execution time is capped at 2-hours (includes all the above)
    30     but this script normally completes in less than an hour.
    31  
    32  
    33  ### ``cache_images`` Task
    34  
    35  1. When a PR is merged (``$CIRRUS_BRANCH`` == ``master``), run another
    36     round of the ``integration`` task (above).
    37  
    38  2. After confirming the tests all pass post-merge, cirrus will
    39     spin up a special VM capable of communicating with the GCE API.
    40     Once accessible, it will ``ssh`` into the special VM and run
    41     the following scripts.
    42  
    43  3. ``setup_environment.sh``: Same as the ``integration``
    44     Task (above).
    45  
    46  4. ``build_vm_images.sh``: Examine the merged PR's description on github.
    47     If it contains the magic string ``***CIRRUS: REBUILD IMAGES***``, then
    48     continue, otherwise display a message, take no further action, and
    49     exit successfully.  This prevents production of new VM images unless
    50     they are called for, thereby saving the cost of needlessly storing them.
    51  
    52  5. If the magic string was found, utilize [the packer tool](http://packer.io/docs/)
    53     to produce new VM images.  Create new VMs starting from *base-images* (see below),
    54     connect to them with ``ssh``, and perform the steps defined by the
    55     ``conmon_images.yml`` file.
    56  
    57      1. Copy the current state of the repository into ``/tmp/conmon``.
    58      2. Execute distribution-specific scripts to prepare the image for
    59         use by the ``integration`` task (above).  For example,
    60         ``fedora_setup.sh``.
    61      3. If successful, shut down each VM and create a new GCE Image
    62         named with the base image, and the commit sha of the merge.
    63  
    64  ***Note:*** The ``.cirrus.yml`` file must be manually updated with the new
    65  images names, then the change sent in via a secondary pull-request.  This
    66  ensures that all the ``integration`` tasks can pass with the new images,
    67  before subjecting all future PRs to them.  A workflow to automate this
    68  process is described in comments at the end of the ``.cirrus.yml`` file.
    69  
    70  
    71  ### Base-images
    72  
    73  Base-images are VM disk-images specially prepared for executing as GCE VMs.
    74  In particular, they run services on startup similar in purpose/function
    75  as the standard 'cloud-init' services.
    76  
    77  *  The google services are required for full support of ssh-key management
    78     and GCE OAuth capabilities.  Google provides native images in GCE
    79     with services pre-installed, for many platforms. For example,
    80     RHEL, CentOS, and Ubuntu.
    81  
    82  *  Google does ***not*** provide any images for Fedora or Fedora Atomic
    83     Host (as of 11/2018), nor do they provide a base-image prepared to
    84     run packer for creating other images in the ``build_vm_images`` Task
    85     (above).
    86  
    87  *  Base images do not need to be produced often, but doing so completely
    88     manually would be time-consuming and error-prone.  Therefor a special
    89     semi-automatic *Makefile* target is provided to assist with producing
    90     all the base-images: ``conmon_base_images``
    91  
    92  To produce new base-images, including an `image-builder-image` (used by
    93  the ``cache_images`` Task) some input parameters are required:
    94  
    95      *  ``GCP_PROJECT_ID``: The complete GCP project ID string e.g. foobar-12345
    96         identifying where the images will be stored.
    97  
    98      *  ``GOOGLE_APPLICATION_CREDENTIALS``: A *JSON* file containing
    99         credentials for a GCE service account.  This can be [a service
   100         account](https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually)
   101         or [end-user
   102         credentials](https://cloud.google.com/docs/authentication/end-user#creating_your_client_credentials]
   103  
   104      *  CSV's of builders to use must be specified to ``PACKER_BUILDS``
   105         to limit the base-images produced.  For example,
   106         ``PACKER_BUILDS=fedora,image-builder-image``.
   107  
   108  The following process should be performed on a bare-metal CentOS 7 machine
   109  with network access to GCE.  Software dependencies can be obtained from
   110  the ``packer/image-builder-image_base_setup.sh`` script.
   111  
   112  Alternatively, an existing image-builder-image may be used from within GCE.
   113  However it must be created with elevated cloud privileges.  For example,
   114  
   115  ```
   116  $ alias pgcloud='sudo podman run -it --rm -e AS_ID=$UID
   117      -e AS_USER=$USER -v /home/$USER:/home/$USER:z cevich/gcloud_centos:latest'
   118  
   119  $ URL=https://www.googleapis.com/auth
   120  $ SCOPES=$URL/userinfo.email,$URL/compute,$URL/devstorage.full_control
   121  
   122  $ pgcloud compute instances create $USER-making-images \
   123      --image-family image-builder-image \
   124      --boot-disk-size "200GB" \
   125      --min-cpu-platform "Intel Haswell" \
   126      --machine-type n1-standard-2 \
   127      --scopes $SCOPES
   128  
   129  $ pgcloud compute ssh centos@$USER-making-images
   130  ...
   131  ```
   132  
   133  When ready, change to the ``packer`` sub-directory, and run:
   134  
   135  ```
   136  $ make conmon_base_images GCP_PROJECT_ID=<VALUE> \
   137      GOOGLE_APPLICATION_CREDENTIALS=<VALUE> \
   138      PACKER_BUILDS=<OPTIONAL>
   139  ```
   140  
   141  Assuming this is successful (hence the semi-automatic part), packer will
   142  produce a ``packer-manifest.json`` output file.  This contains the base-image
   143  names suitable for updating in ``.cirrus.yml``, `env` keys ``*_BASE_IMAGE``.
   144  
   145  On failure, it should be possible to determine the problem from the packer
   146  output.  The only exception is for the Fedora and FAH builds, which utilize
   147  local qemu-kvm virtualisation.  To observe the serial-port output from those
   148  builds, set the ``TTYDEV`` parameter to your current device.  For example:
   149  
   150  ```
   151  $ make conmon_base_images ... TTYDEV=$(tty)
   152    ...
   153  ```