github.com/coreos/mantle@v0.13.0/platforms.md (about)

     1  # Mantle platforms
     2  
     3  Platforms are an API interface to different environments to run clusters,
     4  create images, collect logging information, etc.
     5  
     6  ## Authentication
     7  
     8  Authentication differs based on the platform. Some platforms like `aws` utilize the
     9  configuration files from their command-line tooling while others define their
    10  own custom configuration format and default locations (like [DigitalOcean](https://github.com/coreos/mantle/tree/master/auth/do.go)).
    11  Generally if any extensions / custom configurations are needed a new file is
    12  created inside of the `auth` package which will define the default location,
    13  the structure of the configuration, and a function to parse the configuration
    14  file (usually named Read<platform>Config and emits a
    15  `map[string]<platform>Config` object).
    16  
    17  ## API
    18  
    19  Platform APIs wrap each cloud provider's golang SDK and live inside of
    20  `platform/api/<platform>/`. There is no direct requirement for what
    21  functionality is present in the API.
    22  
    23  ## Cluster & Machine
    24  
    25  Clusters must implement the `Cluster` [interface](https://github.com/coreos/mantle/tree/master/platform/platform.go#L75-L97).
    26  Machines must implement the `Machine` [interface](https://github.com/coreos/mantle/tree/master/platform/platform.go#L40-L73).
    27  
    28  ## Adding a new platform to the kola runner
    29  
    30  To add a new platform to the `kola` runner the following things must be added:
    31   1. A platform specific struct inside of `cmd/kola/kola.go` which contains the
    32   fields that should be logged to give more information about a test run.
    33   Generally this will contain things like `version` or `ami` and `region`. [This](https://github.com/coreos/mantle/tree/master/cmd/kola/kola.go#L138-L142)
    34   is an example of the struct and [this](https://github.com/coreos/mantle/tree/master/cmd/kola/kola.go#L179-L183) shows the data
    35   being added to the output (which can be found in
    36   `_kola_temp/<platform>-latest/properties.json`).
    37   2. The platform specific options inside of `cmd/kola/options.go`
    38   ([for example DigitalOcean](https://github.com/coreos/mantle/tree/master/cmd/kola/kola.go#L179-L183)). The flags will
    39   generally contain an override for the configuration file location, region,
    40   type/size, and the image.
    41   3. The platform needs to be added to the `kolaPlatforms` list inside of
    42   `cmd/kola/options.go` [here](https://github.com/coreos/mantle/tree/master/cmd/kola/options.go#L32)
    43   4. The platform options & new cluster inside of `kola/harness.go`. The platform
    44   options variables are defined [here](https://github.com/coreos/mantle/tree/master/kola/harness.go#L54-L60) and the
    45   `NewCluster` method is defined [here](https://github.com/coreos/mantle/tree/master/kola/harness.go#L143-L161).
    46  
    47  ## Other things to consider adding
    48  
    49  It is generally preferred that new platforms add garbage collection methods via
    50  the `ore <platform> gc` command.
    51  
    52  For platforms which support adding custom images `ore` commands to upload &
    53  create images from the image file.
    54  
    55  # Existing Platforms
    56  
    57  ## AWS
    58  
    59   - The AWS platform wraps [aws-sdk-go](https://github.com/aws/aws-sdk-go).
    60   - By default SSH keys will be passed via both the AWS metadata AND the userdata.
    61   - UserData is passed to the instances via the AWS metadata service.
    62   - Instances are tagged with `Name:<generated name>` and `CreatedBy:mantle`. The `CreatedBy` tag is used by `GC` when searching for instances to terminate.
    63   - Serial Console data on AWS is only saved by the cloud during boot sequences (initial boot and all subsequent reboot / shutdowns). This means that sometimes the serial console will not be complete as only the [most recent 64KB is stored](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-console.html).
    64   - If a security group matching the name in `aws-sg` (default: `kola`) is not found then one will be created, along with a VPC, internet gateway, route table, and subnets.
    65   - Both AMI's as well as the channel names are accepted via the `aws-ami` parameter, if a channel is given it will be resolved via the release bucket's `coreos_production_ami_all.json` file.
    66  
    67  ## Azure
    68  
    69   - The Azure platform wraps [azure-sdk-for-go](https://github.com/Azure/azure-sdk-for-go).
    70   - By default SSH keys will be passed via both the Azure metadata AND the userdata.
    71   - UserData is passed to the instances via the Azure metadata service.
    72   - For each cluster to be created a Resource Group is created containing the necessary networking resources and a storage account (which will be used to store boot diagnostics for any machines in the cluster).
    73   - There are two types of images in Azure (published images and custom images). For using published images the channel can be passed via the `azure-sku` parameter and the version can be passed via the `azure-version` parameter. To specify a custom image you can pass the `azure-disk-uri` parameter.
    74   - `kola` works entirely on ARM based authentication, `ore` has methods for both ASM or ARM credentials.
    75   - `GC` in Azure searches for Resource Groups with a prefix of `kola-cluster` in the name.
    76  
    77  ## DigitalOcean
    78  
    79   - The DO platform wraps [godo](https://github.com/digitalocean/godo).
    80   - By default SSH keys will be passed via both the DO metadata AND the userdata.
    81   - UserData is passed to the instances via the DO metadata service.
    82   - DigitalOcean has no method for uploading custom images, as a result the `ore do create-image` command does a [~~terrifying~~ special workaround](https://github.com/coreos/mantle/blob/master/cmd/ore/do/create-image.go#L117-L173) which specifies custom userdata that does the following (after which the machine is snapshotted):
    83     - configure networking in the initramfs
    84     - Download a custom image
    85     - replaces `/root/initramfs/shutdown` with a script to:
    86        - unmount the root filesystem
    87        - dd the custom image over the disk
    88        - reboot
    89   - Instances are given the tag `mantle` which is used by `GC`.
    90   - The `do-image` parameter accepts snapshot ID's as well as the channel names, if a channel is given the relevant slug `coreos-<channel>` will be passed to DO.
    91  
    92  ## ESX
    93  
    94   - The ESX platform wraps [govmomi](https://github.com/vmware/govmomi).
    95   - ESX is actually a bit of a misnomer, it actually requires that the target is a vSphere instance (as it uses features not present in the free ESXi hypervisor).
    96   - SSH keys will be passed via userdata.
    97   - Userdata is passed to the instances via the `ovfenv`. When a machine is created the API updates the `config.vAppConfig.property.guestinfo.coreos.config.data` field to be a base64 encoded userdata config.
    98   - The general workflow for running tests on ESX is to run the `ore esx create-base` to upload an OVF image and then the resulting image name is specified to `kola` via the `esx-base-vm` parameter. This is done to only require kola to perform a clone operation rather than a full re-upload of the image.
    99  
   100  ## GCE
   101  
   102   - The GCE/gcloud platform wraps [this SDK](google.golang.org/api).
   103   - By default SSH keys will be passed via both the GCE metadata AND the userdata.
   104   - UserData is passed to the instances via the GCE metadata service.
   105   - Instances are tagged with `created-by:mantle` which is used when filtering instances for `GC`.
   106  
   107  ## OpenStack
   108  
   109   - The OpenStack platform wraps [gophercloud](https://github.com/gophercloud/gophercloud).
   110   - By default SSH keys will be passed via both the OpenStack metadata AND the userdata.
   111   - UserData is passed to the instances via the OpenStack metadata service.
   112   - Instances are tagged with `CreatedBy: mantle` which is used when filtering instances for `GC`.
   113  
   114  ## Packet
   115  
   116   - The Packet platform wraps [packngo](https://github.com/packethost/packngo).
   117   - SSH keys will be passed via userdata.
   118   - Custom images do not use the Packet Custom Images API but rather the machine creation actually writes a custom iPXE script (which is uploaded to Google Storage) that sets `coreos.config.url` on the kernel command-line to point at a userdata file (which is also uploaded to Google Storage). This userdata file contains multiple systemd units & file definitions -- the actual metadata is written to `/userdata`. The systemd units will run `coreos-install` to install the custom image on the machine (and pass the config file).
   119   - Packet provides a URL for accessing the serial console, an SSH client is created to this endpoint and the stdout is fed to the `Console` object.
   120   - Devices are tagged with `mantle` which is used by `GC`.