github.com/ouraigua/jenkins-library@v0.0.0-20231028010029-fbeaf2f3aa9b/documentation/docs/infrastructure/vault.md (about)

     1  # Vault for Pipeline Secrets
     2  
     3  Project "Piper" supports fetching your pipeline secrets directly from [Vault](https://www.hashicorp.com/products/vault).
     4  Currently, Vault's key value engine is supported in version 1 and 2, although we recommend version 2 since it supports
     5  the versioning of secrets
     6  
     7  Parameters that support being fetched from Vault are marked with the Vault Label in the Step Documentation.
     8  
     9  ![Vault Label](../images/parameter-with-vault-support.png)
    10  
    11  ## Authenticating Piper to Vault
    12  
    13  Piper currently supports Vault's `AppRole` and `Token` authentication. However, `AppRole` authentication is recommended
    14  since Piper is able to regularly rotate the SecretID, which is not possible with a Token.
    15  
    16  ### AppRole Authentication
    17  
    18  To authenticate against Vault, using [AppRole](https://www.vaultproject.io/docs/auth/approle) authentication you need to
    19  do the following things
    20  
    21  - Enable AppRole authentication in your Vault instance.
    22  - After that you have
    23    to [create an AppRole Role](https://www.vaultproject.io/api-docs/auth/approle#create-update-approle) for Piper
    24  - Assign the necessary policies to your newly created AppRole.
    25  - Take the **AppRole ID** and create a Jenkins `Secret Text` credential.
    26  - Take the **AppRole Secret ID** and create a Jenkins `Secret Text` credential.
    27  
    28  ![Create two jenkins secret text credentials](../images/jenkins-vault-credential.png)
    29  
    30  ### Token Authentication
    31  
    32  First step to use Token authentication is
    33  to [Create a Vault Token](https://www.vaultproject.io/api/auth/token#create-token)
    34  In order to use a Vault token for authentication you need to store the Vault token inside your Jenkins instance as shown
    35  below.
    36  
    37  ![Create a Jenkins secret text credential](../images/jenkins-vault-token-credential.png)
    38  
    39  ## Setup a Secret Store in Vault
    40  
    41  The first step to store your pipeline secrets in Vault, is to enable a the
    42  [Key-Value Engine](https://www.vaultproject.io/docs/secrets/kv/kv-v2). Then create a policy which grants read access to
    43  the key value engine.
    44  
    45  ![Enable a new secret engine in vault](../images/vault-secret-engine-enable.png)
    46  
    47  ## Pipeline Configuration
    48  
    49  For pipelines to actually use the secrets stored in Vault you need to adjust your `config.yml`:
    50  
    51  ```yml
    52  general:
    53    ...
    54    vaultServerUrl: '<YOUR_VAULT_SERVER_URL>'
    55    vaultNamespace: '<YOUR_NAMESPACE_NAME>' # if you are not using vault's namespace feature you can remove this line
    56    vaultPath: 'kv/my-pipeline' # the path under which your jenkins secrets are stored
    57    ...
    58  ```
    59  
    60  To authenticate you need to provide `PIPER_vaultAppRoleID` and `PIPER_vaultAppRoleSecretID` if you use app role authentication or `PIPER_vaultToken` if you use token authentication.
    61  
    62  !!! note "Jenkins"
    63      When running a step via the Jenkins library you can use Jenkins credentials for pass this values. Use `vaultAppRoleTokenCredentialsId` and `vaultAppRoleSecretTokenCredentialsId` or `vaultTokenCredentialsId` in your `config.yml`.
    64  
    65  ## Configuring the Secret Lookup
    66  
    67  When Piper is configured to lookup secrets in Vault, there are some aspects that need to be considered.
    68  
    69  ### Overwriting of Parameters
    70  
    71  Whenever a parameter is provided via `config.yml` or passed to the CLI it gets overwritten when a secret is found in
    72  Vault. To disable overriding parameters put a `vaultDisableOverwrite: false` on `Step` `Stage` or `General` Section in
    73  your config.
    74  
    75  ```yaml
    76  general:
    77    ...
    78    vaultDisableOverwrite: true
    79    ...
    80  steps:
    81    executeBuild:
    82      vaultDisableOverwrite: false
    83      ...
    84  ```
    85  
    86  ### Skipping Vault Secret Lookup
    87  
    88  It is also possible to skip Vault for `Steps`, `Stages` or in `General` by using the `skipVault` config parameter as
    89  shown below.
    90  
    91  ```yaml
    92  ...
    93  steps:
    94    executeBuild:
    95      skipVault: true   # Skip Vault Secret Lookup for this step
    96  ```
    97  
    98  ## Using Vault for general purpose and test credentials
    99  
   100  Vault can be used with piper to fetch any credentials, e.g. when they need to be appended to custom piper extensions or when they need to be appended to test command. The configuration for Vault general purpose credentials can be added to **any** piper golang-based step. The configuration has to be done as follows:
   101  
   102  ```yaml
   103  general:
   104    < your Vault configuration > # see above
   105  ...
   106  steps:
   107    < piper go step >:
   108      vaultCredentialPath: 'myStepCredentials'
   109      vaultCredentialKeys: ['myAppId', 'myAppSecret']
   110  ```
   111  
   112  In case if you want to retrieve secrets from multiple vault folders, pass several paths with keys:
   113  
   114  ```yaml
   115  general:
   116    < your Vault configuration > # see above
   117  ...
   118  steps:
   119    < piper go step >:
   120      vaultCredentialPath: ['myStepCredentials1', 'myStepCredentials2']
   121      vaultCredentialKeys: [['myAppId1', 'myAppSecret1'], ['myAppId2', 'myAppSecret2']]
   122  ```
   123  
   124  The `vaultCredentialPath` parameter is the endpoint of your credential path in Vault. Depending on your _general_ config, the lookup for the credential IDs will be done in the following order respectively locations. The first path with found general purpose credentials will be used.
   125  
   126  1. `<vaultPath>/<vaultCredentialPath>`
   127  2. `<vaultBasePath>/<vaultPipelineName>/<vaultCredentialPath>`
   128  3. `<vaultBasePath>/GROUP-SECRETS/<vaultCredentialPath>`
   129  
   130  The `vaultCredentialKeys`parameter is a list of credential IDs. The secret value of the credential will be exposed as an environment variable prefixed by "PIPER_VAULTCREDENTIAL_" and transformed to a valid variable name. For a credential ID named `myAppId` the forwarded environment variable to the step will be `PIPER_VAULTCREDENTIAL_MYAPPID` containing the secret. The Base64 encoded secret value will be exposed as environment variable to the step as  `PIPER_VAULTCREDENTIAL_MYAPPID_BASE64`. Hyphens will be replaced by underscores and other non-alphanumeric characters will be removed.
   131  
   132  !!! hint "Using a custom prefix for test credentials"
   133      By default the prefix for test credentials is `PIPER_VAULTCREDENTIAL_`.
   134  
   135      It is possible to use a custom prefix by setting for example `vaultCredentialEnvPrefix: MY_CUSTOM_PREFIX` in your configuration.
   136      With this above credential ID named `myAppId` will be populated into an environment variable with the name `MY_CUSTOM_PREFIX_MYAPPID`.
   137  
   138  Extended logging for Vault secret fetching (e.g. found credentials and environment variable names) can be activated via `verbose: true` configuration.
   139  
   140  ## Using Vault for test credentials (Deprecated : use general purpose and test credentials as above)
   141  
   142  Vault can be used with piper to fetch any credentials, e.g. when they need to be appended to test command. The configuration for Vault test credentials can be added to **any** piper golang-based step. The configuration has to be done as follows:
   143  
   144  ```yaml
   145  general:
   146    < your Vault configuration > # see above
   147  ...
   148  steps:
   149    < piper go step >:
   150      vaultTestCredentialPath: 'myTestStepCrecetials'
   151      vaultTestCredentialKeys: ['myAppId', 'myAppSecret']
   152  ```
   153  
   154  The `vaultTestCredentialPath` parameter is the endpoint of your credential path in vault. Depending on your _general_ config, the lookup for the credential IDs will be done in the following order respectively locations. The first path with found test credentials will be used.
   155  
   156  1. `<vaultPath>/<vaultTestCredentialPath>`
   157  2. `<vaultBasePath>/<vaultPipelineName>/<vaultTestCredentialPath>`
   158  3. `<vaultBasePath>/GROUP-SECRETS/<vaultTestCredentialPath>`
   159  
   160  The `vaultTestCredentialKeys`parameter is a list of credential IDs. The secret value of the credential will be exposed as an environment variable prefixed by "PIPER_TESTCREDENTIAL_" and transformed to a valid variable name. For a credential ID named `myAppId` the forwarded environment variable to the step will be `PIPER_TESTCREDENTIAL_MYAPPID` containing the secret. Hyphens will be replaced by underscores and other non-alphanumeric characters will be removed.
   161  
   162  !!! hint "Using a custom prefix for test credentials"
   163      By default the prefix for test credentials is `PIPER_TESTCREDENTIAL_`.
   164  
   165      It is possible to use a custom prefix by setting for example `vaultTestCredentialEnvPrefix: MY_CUSTOM_PREFIX_` in your configuration.
   166      With this above credential ID named `myAppId` will be populated into an environment variable with the name `MY_CUSTOM_PREFIX_MYAPPID`.
   167  
   168  Extended logging for Vault secret fetching (e.g. found credentials and environment variable names) can be activated via `verbose: true` configuration.