github.com/criteo/command-launcher@v0.0.0-20230407142452-fb616f546e98/gh-pages/content/en/docs/overview/provider-guide.md (about)

     1  ---
     2  title: "CLI provider guide"
     3  description: "Complete guide to integrate your CLI to command launcher"
     4  lead: "Complete guide to integrate your CLI to command launcher"
     5  date: 2022-10-02T18:45:11+02:00
     6  lastmod: 2022-10-02T18:45:11+02:00
     7  draft: false
     8  images: []
     9  menu:
    10    docs:
    11      parent: "overview"
    12      identifier: "provider-guide-e4586ef1fd15acd1ef9a2ca69711c418"
    13  weight: 230
    14  toc: true
    15  ---
    16  
    17  
    18  > NOTE: in this page, we use `cola` as the command launcher's binary name, you can build your own command launcher with a different name. See: [build from source](../../quickstart/build-from-source)
    19  
    20  Command launcher synchronizes commands from the remote command repository. Commands are packaged into a `package`, then uploaded to remote command repository. The following diagram shows this architecture.
    21  
    22  ```text
    23                 ┌─────────────────────────────┐
    24                 │  Remote Command Repository  │
    25                 └──────────────┬──────────────┘
    26                    ┌───────────┴────────────┐
    27                    ▼                        ▼
    28              ┌────────────┐           ┌───────────┐
    29              │  pacakge 1 │           │ pakcage 2 │
    30              └────────────┘           └───────────┘
    31                    │                        │
    32            ┌───────┤─────────┐          ┌───┴────┐
    33            ▼       ▼         ▼          ▼        ▼
    34        ┌───────┐┌───────┐┌───────┐  ┌───────┐┌───────┐
    35        │ cmd A ││ cmd B ││ cmd C │  │ cmd D ││ cmd E │
    36        └───────┘└───────┘└───────┘  └───────┘└───────┘
    37  ```
    38  
    39  ## Remote command repository
    40  
    41  A remote command repository is a simple http server with following endpoints:
    42  
    43  - `/index.json`, which returns the list of packages available.
    44  
    45  It is up-to-you to implement such an http server. You can configure command launcher to point to your remote repository with following command:
    46  
    47  ```shell
    48  cola config command_repository_base_url https://my-remote-repository/root/url
    49  ```
    50  
    51  > NOTE: the command launcher will search for [command_repository_base_url]/index.json for the remote registry. You can also use a local folder as the "base url", for example, `/tmp/cola-remote-repository/`. In this case, you need to create the `index.json` registry file in the folder. This is useful for test purpose.
    52  
    53  You need to config an endpoint to auto update command launcher itself as well:
    54  
    55  ```shell
    56  cola config self_update_latest_version_url https://my-remote-repository/cola/root/url/version
    57  cola config self_update_base_url https://my-remote-repository/cola/root/url
    58  ```
    59  
    60  The `self_update_latest_version_url` configuration defines the url to download the metadata of command launchers's latest version in YAML or JSON format, see: [Command Launcher version metadata](#command-launcher-version-metadata)
    61  
    62  The `self_update_base_url` configuratin defines the base url to download command launcher binary. It follows the following pattern: `[SELF_UPDATE_BASE_URL]/{version}/{binaryName}_{OS}_{ARCH}_{version}{extension}`. If you build your own binary, you should make it available following the same convention.
    63  
    64  ### Remote repository registry /index.json
    65  
    66  Remote repository registry is a json file, which contains all available packages:
    67  
    68  The following example demonstrates a registry, which has three packages. Note that the package "hotfix" has two different versions, and the version `1.0.0-45149` targets to 30% of the user (partition 6, 7, and 8). More details about the partition see [Progressive Rollout](#progressive-rollout)
    69  
    70  ```json
    71  [
    72    {
    73      "name": "hotfix",
    74      "version": "1.0.0-44733",
    75      "checksum": "5f5f47e4966b984a4c7d33003dd2bbe8fff5d31bf2bee0c6db3add099e4542b3",
    76      "url": "https://the-url-of-the-env-package/any-name.zip",
    77      "startPartition": 0,
    78      "endPartition": 9
    79    },
    80    {
    81      "name": "hotfix",
    82      "version": "1.0.0-45149",
    83      "checksum": "773a919429e50346a7a002eb3ecbf2b48d058bae014df112119a67fc7d9a3598",
    84      "url": "https://the-url-of-the-env-package/hotfix-1.0.0-45149.zip",
    85      "startPartition": 6,
    86      "endPartition": 8
    87    },
    88    {
    89      "name": "env",
    90      "version": "0.0.1",
    91      "checksum": "c87a417cce3d26777bcc6b8b0dea2ec43a0d78486438b1bf3f3fbd2cafc2c7cc",
    92      "url": "https://the-url-of-the-env-package/package.zip",
    93      "startPartition": 0,
    94      "endPartition": 9
    95    }
    96  ]
    97  ```
    98  
    99  ### Command launcher version metadata
   100  
   101  Command launcher update itself by checking an endpoint defined in config `self_update_latest_version_url`. This endpoint returns the command version metadata:
   102  
   103  In JSON
   104  
   105  ```json
   106  {
   107      "version": "45861",
   108      "releaseNotes": "- feature 1\n-feature 2\nfeature 3",
   109      "startPartition": 0,
   110      "endPartition": 9
   111  }
   112  ```
   113  
   114  Or in YAML
   115  
   116  ```YAML
   117  version: "45861"
   118  releaseNotes: |
   119    * feature 1
   120    * feature 2
   121    * feature 3
   122  startPartition: 0
   123  endPartition: 9
   124  ```
   125  
   126  > The version metadata endpoint supports both YAML and JSON format. It is recommandded to use YAML for this endpoint because of the multiple line string support in YAML
   127  
   128  You can also target a small portion of your command user by specifying the partition. More details see: [Progressive Rollout](#progressive-rollout)
   129  
   130  ## Integrate your command into command launcher
   131  
   132  ### Package your command into a command package
   133  
   134  > A command package = zip(your commands, manifest file)
   135  
   136  A command package is simply a zip of your command and a manifest file that tell command launcher how to run your command. It is up-to-you to organize the structure of the package, the only requirement here is to keep the `manifest.mf` file in the root of the package.
   137  
   138  For example, the following structure keeps the binary in different folder according to the os.
   139  
   140  ```text
   141  my-package.pkg
   142  ├─linux/
   143  ├─windows/
   144  ├─macosx/
   145  └─manifest.mf
   146  ```
   147  
   148  ### Package manifest file, manifest.mf
   149  
   150  See [manifest.mf specification](../manifest)
   151  
   152  ## Upload your package, and update package registry
   153  
   154  Once you have your command package ready, you can upload it to a remote server that command launcher have access. Depends on how you implement the http server of your remote command repository, the upload process could be different, The only requirement here is to ensure your package is accessible from command launcher.
   155  
   156  You also need to update the [index.json](#remote-repository-registry-indexjson) endpoint to include your package in it and specify the package url in the `url` property.
   157  
   158  ## Progressive Rollout
   159  
   160  Command launcher will assign each machine a unique partition ID from 0 to 9. When you roll out your package, you can specify the partition that you want to target to. For example, you just developed a new version, packaged into package `my-pkg 1.1.0`, uploaded it to remote repository. You can edit the `/index.json` registry, add following entry to target 40% of your audience (partition 4, 5, 6, and 7):
   161  
   162  ```json
   163  {
   164    "name": "my-pkg",
   165    "version": "1.1.0"
   166    "checksum": "",
   167    "startPartition": 4,
   168    "endPartition": 7
   169  }
   170  ```
   171  
   172  You will have different monitoring vectors for each partition, which will help you making A/B tests.
   173  
   174  ## Monitoring
   175  
   176  Command launcher current implements a built-in graphite exporter. It reports the following metrics to graphite:
   177  
   178  1. success command execution count: `devtools.cdt.[package name].[group].[name].ok.count`
   179  2. success command duration: `devtools.cdt.[package name].[group].[name].ok.duration`
   180  3. fail command execution count: `devtools.cdt.[package name].[group].[name].ko.count`
   181  4. fail command duration: `devtools.cdt.[package name].[group].[name].ko.duration`
   182  
   183  You can add your custom metrics exporter by a `__metrics__` command hook in a system package, see [system package](../system-package)
   184  
   185  ## Credential Management
   186  
   187  Command launcher has a built-in login command, which will prompt the user to enter his user name and password. The default implementation will store the user name and password securely in the system credential manager. Each command can request the access to such credential in its manifest. Command launcher will ensure user consent on accessing these credentials and pass them to the underlying command through environment variables. More detail see [Manage resources](../resources)