github.com/munnerz/test-infra@v0.0.0-20190108210205-ce3d181dc989/boskos/mason/README.md (about)

     1  # mason
     2  
     3  
     4  ## Introduction
     5  
     6  Mason creates and update virtual resources from existing physical resources. As an example, in order to create
     7  a GKE cluster you need a GCP Project. Mason will construct virtual resources based on the configuration of a
     8  given resource type. Each configuration defines its physical resources requirement. Mason will acquire those
     9  resources from Boskos and pass them to the specific implementation that will turn them into a virtual resource.
    10  Mason only handles lease and release of physical resources.
    11  
    12  ## Configuration
    13  
    14  Each configuration is represented by a name. The name of configuration maps to the virtual resource type in Boskos.
    15  
    16  Here is an example configuration:
    17  
    18  ```yaml
    19  
    20  configs:
    21  - name: type2
    22    needs:
    23      type1: 1
    24    config:
    25      type: GCPResourceConfig
    26      content: |
    27        projectconfigs:
    28        - type: type1
    29          clusters:
    30          - machinetype: n1-standard-2
    31            numnodes: 4
    32            version: 1.7
    33            zone: us-central-1f
    34          vms:
    35          - machinetype: n1-standard-4
    36            sourceimage: projects/debian-cloud/global/images/debian-9-stretch-v20180105
    37            zone: us-central-1f
    38            tags:
    39            - http-server
    40            - https-server
    41            scopes:
    42            - https://www.googleapis.com/auth/cloud-platform
    43  
    44  ```
    45  
    46  As you can see in this example. In order to create a virtual resource of type2 we need one resource
    47  of type1. Once we have acquired the right resources, we need Masonable interface of type GCPResourceConfig
    48  that will know how to parse this configuration.
    49  
    50  ## Operation
    51  
    52  Mason is only operating on a given set of resource types.
    53  In order to use Mason, one needs to register a Masonable interface using the RegisterConfigConverter function.
    54  
    55  A simple binary would like that:
    56  
    57  ```golang
    58  
    59  func main() {
    60    flag.Parse()
    61    logrus.SetFormatter(&logrus.JSONFormatter{})
    62  
    63    if *configPath == "" {
    64      logrus.Panic("--config must be set")
    65    }
    66  
    67    mason := mason.NewMasonFromFlags()
    68  
    69    // Registering Masonable Converters
    70    if err := mason.RegisterConfigConverter(gcp.ResourceConfigType, gcp.ConfigConverter); err != nil {
    71      logrus.WithError(err).Panicf("unable tp register config converter")
    72    }
    73    if err := mason.UpdateConfigs(*configPath); err != nil {
    74      logrus.WithError(err).Panicf("failed to update mason config")
    75    }
    76    go func() {
    77      for range time.NewTicker(defaultUpdatePeriod).C {
    78        if err := mason.UpdateConfigs(*configPath); err != nil {
    79          logrus.WithError(err).Warning("failed to update mason config")
    80        }
    81      }
    82    }()
    83  
    84    mason.Start()
    85    defer mason.Stop()
    86    stop := make(chan os.Signal, 1)
    87    signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
    88    <-stop
    89  ```
    90  
    91  ### Recycling Thread
    92  
    93  The recycling thread is acquiring dirty virtual resources and releasing the associated physical resources as
    94  dirty such Janitor can clean them up as an example. It then put the virtual resource to the Fulfilling queue.
    95  
    96  ### Fulfilling Thread
    97  
    98  The fulfilling thread will look at the config resource needs, and will acquire the necessary resources.
    99  Once a resource is acquired it will update the virtual resource LEASED_RESOURCES user data. Once all resources
   100  are acquired, it will be put on the cleaning queue.
   101  
   102  ### Cleaning Thread
   103  
   104  The cleaning thread will construct the virtual resources from the leased physical resources and update the user
   105  data with the user data provided by the Masonable implementation
   106  
   107  ### Freeing Thread
   108  
   109  The freeing thread will release all resources. It will release the leased physical resources with a state that is
   110  equal to the name of virtual resource and release the virtual resource as free.
   111  
   112  ### Mason Client
   113  Mason comes with its own client to ease usage. The mason client takes care of
   114  acquiring and release all the right resources from the User Data information.
   115  
   116  ## References
   117  
   118  [Istio Deployment](https://github.com/istio/test-infra/tree/master/boskos)
   119  
   120  [Design Doc](https://goo.gl/vHNfww)
   121