github.com/esnet/gdg@v0.6.1-0.20240412190737-6b6eba9c14d8/website/content/docs/gdg/configuration.md (about)

     1  ---
     2  title: "Configuration"
     3  weight: 14
     4  ---
     5  
     6  ## Configuration
     7  
     8  make a copy of [config/importer-example.yml](https://github.com/esnet/gdg/blob/master/config/importer-example.yml) and
     9  name it `config/importer.yml` or simply run `gdg tools ctx new <name>` which will walk you through setting up a new
    10  context to use with your grafana installation.
    11  
    12  ## Authentication
    13  
    14  ### Authentication Token
    15  
    16  You can use an Authentication Token / API Key to authenticate with the Grafana API, which can be generated in your
    17  Grafana Web UI => Configuration => API Keys. You can then use it in your configuration file (eg. `importer.yml`).
    18  
    19  {{< callout context="caution" title="Caution" icon="alert-triangle" >}}
    20  gdg is currently using viper to read in the config. Since viper makes all keys lowercase, we also have the same
    21  limitation. Camel case will be read in but be aware that a section named fooBar == Foobar == foobar etc.
    22  {{< /callout >}}
    23  
    24  ### Service Accounts
    25  
    26  Service Accounts are supported and interchangeable for tokens. If you wish to use a service account, simply put the
    27  token value from the service account for `token:`. Please make sure you've granted the account the needed permissions
    28  for the operations you are trying to perform.
    29  
    30  ```yaml
    31  context_name: main
    32  
    33  contexts:
    34    main:
    35      url: https://grafana.example.org
    36      token: "<<Grafana API Token>>"
    37      output_path: "myFolder"
    38      ignore_filters: False  # When set to true all Watched filtered folders will be ignored and ALL folders will be acted on
    39      watched:
    40        - Example
    41      connections:
    42        credential_rules:
    43          - rules:
    44              - field: "name"
    45                regex: "misc"
    46              - field: "url"
    47                value: ".*esproxy2*"
    48            secure_data: "misc_auth.json"
    49          - rules:
    50              - field: "url"
    51                regex: ".*esproxy2*"
    52            secure_data: "proxy.json"
    53          - rules:
    54              - field: "name"
    55                regex: ".*"
    56            secure_data: "default.json"
    57  global:
    58    debug: true
    59    ignore_ssl_errors: false
    60  ```
    61  
    62  ### Username / Password
    63  
    64  You can also use username/password credentials of an admin user to authenticate with the Grafana API. You can specify
    65  them in your configuration file (eg. `importer.yml`).
    66  
    67  ```
    68  context_name: main
    69  
    70  contexts:
    71    main:
    72      url: https://grafana.example.org
    73      user_name: <your username>
    74      password: <your password>
    75      output_path: "myFolder"
    76      watched:
    77        - Example
    78  
    79  global:
    80    debug: true
    81    ignore_ssl_errors: false
    82  ```
    83  ## Cloud Configuration
    84  
    85  Several S3 compatible cloud providers are supported.  Please see this [section](https://software.es.net/gdg/docs/gdg/cloud-configuration/) for more detailed instructions.
    86  
    87  ## Connection
    88  
    89  ### Connection Credentials
    90  
    91  #### Current Behavior (Version +0.5.2)
    92  
    93  If the connection has BasicAuth enabled, then we'll attempt to set the auth with the following rules.
    94  
    95  We will try to find a match given the rules specified:
    96  
    97  - `field`: matches any valid json path and retrieves its value. ie.  `jsonData.maxConcurrentShardRequests` and validates
    98    it against a golang supported [Regex](https://github.com/google/re2/wiki/Syntax).
    99  - It goes down the list of rules and returns the auth for the first matching one. The rules should be listed from more
   100    specific to more broad. The default rule ideally should be at the end.
   101  
   102  ```yaml
   103    testing:
   104      output_path: testing_data
   105      connections:
   106        credential_rules:
   107          - rules:
   108              - field: "name"
   109                regex: "misc"
   110              - field: "url"
   111                regex: ".*esproxy2*"
   112            secure_data: "default.json"
   113      url: http://localhost:3000
   114      user_name: admin
   115      password: admin
   116      ignore_filters: False  # When set to true all Watched filtered folders will be ignored and ALL folders will be acted on
   117      watched:
   118        - General
   119        - Other
   120  
   121   ```
   122  
   123  the secure_data will read the file from {output_path}/secure/. It will then use that
   124  information to construct the datasource.
   125  
   126  Default setting if you use basic auth is shown below.
   127  
   128  ```json
   129  {
   130    "basicAuthPassword": "password",
   131    "user": "user"
   132  }
   133  ```
   134  
   135  #### Version v0.4.2-v0.5.1
   136  
   137  {{< details "Legacy Behavior " >}}
   138  Preview behavior did not support the use of a secure/secureData.json pattern, instead an auth: codeblock was used.
   139  
   140  Please note that only basicAuth worked prior to version v0.5.2
   141  
   142  Example can be seen below:
   143  
   144  ```yaml
   145    testing:
   146      output_path: testing_data
   147      connections:
   148        credential_rules:
   149          - rules:
   150              - field: name
   151                regex: .*
   152            auth:
   153              user: user
   154              password: secret
   155      url: http://localhost:3000
   156      user_name: admin
   157      password: admin
   158      ignore_filters: False  # When set to true all Watched filtered folders will be ignored and ALL folders will be acted on
   159      watched:
   160        - General
   161        - Other
   162  
   163   ```
   164  
   165  {{< /details >}}
   166  
   167  #### Version Prior to v0.4.2
   168  
   169  {{< details "Legacy Behavior " >}}
   170  
   171  If the connection has BasicAuth enabled, then we'll attempt to set the auth with the following precedence on matches:
   172  
   173  1. Match of DS credentials based on DS name.
   174  2. Match URL regex for the DS if regex specified.
   175  3. Use Default Credentials if the above two both failed.
   176  
   177  An example of a configuration can be seen below
   178  
   179  ```yaml
   180    testing:
   181      output_path: testing_data
   182      connections:
   183        credentials:
   184          default:
   185            user: user
   186            password: password
   187          misc:
   188            user: admin
   189            password: secret
   190            url_regex: .*esproxy2*
   191      url: http://localhost:3000
   192      user_name: admin
   193      password: admin
   194      ignore_filters: False  # When set to true all Watched filtered folders will be ignored and ALL folders will be acted on
   195      watched:
   196        - General
   197        - Other
   198  
   199   ```
   200  
   201  {{< /details >}}
   202  
   203  ### Connection Filters
   204  
   205  #### Current Behavior (+v0.4.2)
   206  
   207  You can filter based on any field and have it be an exclusive (default) or inclusive (ie Only allow values that match)
   208  to be listed/imported etc.
   209  
   210  Pattern matching is the same as the Credentials mapping.
   211  
   212  - field represents any valid JSON Path
   213  - regex: any valid [regex](https://github.com/google/re2/wiki/Syntax) supported by golang
   214  
   215  The example below will exclude any connections named "Google Sheets". It will also only include connections with the
   216  type elasticsearch or mysql
   217  
   218  ```yaml
   219  contexts:
   220    testing:
   221      output_path: test/data
   222      connections:
   223        exclude_filters:
   224          - field: "name"
   225            regex: "Google Sheets"
   226          - field: "type"
   227            regex: "elasticsearch|mysql"
   228            inclusive: true
   229  ```
   230  
   231  #### Legacy Behavior
   232  
   233  {{< details "Legacy Behavior " >}}
   234  This feature allows you to exclude connection by name or include them by type. Please note that the logic switches based
   235  on the data type.
   236  
   237  **name filter:**
   238  
   239  ```yaml
   240  ...
   241  datasources:
   242    filters:
   243      name_exclusions: "DEV-*|-Dev-*"
   244  ```
   245  
   246  Will exclude any connection that matches the name regex.
   247  
   248  **Type Filter**
   249  
   250  Will ONLY include connection that are listed.
   251  
   252  ```yaml
   253  datasources:
   254    filters:
   255      valid_types:
   256        - elasticsearch
   257  ```
   258  
   259  The snippet above will ONLY import connections for elasticsearch
   260  
   261  {{< /details >}}
   262  
   263  {{< callout context="note" title="Note" icon="info-circle" >}}
   264  If you configure both, an auth `Token` and `BasicAuth`, then the Token is given priority.
   265  Watched folders under grafana is a white list of folders that are being managed by gdg. By default, only "General" is managed.
   266  
   267  {{< /callout >}}
   268  
   269  ## Organization
   270  
   271  The organization is set for a given context via the `orgnization_name`.  If the org is not set, gdg will fallback on the default value that grafana starts out with `Main Org.`
   272  
   273  Additionally, if there is an organization specific behavior, it can be added to a context by adding the following config to your config:
   274  
   275  ```yaml
   276      watched_folders_override:
   277        - organization_name: "Special Org"
   278          folders:
   279            - General
   280            - SpecialFolder
   281  ```
   282  
   283  In this case, watched_folder is ignored in favor of the newly provided list.
   284  
   285  
   286  ## Users
   287  
   288  Users can be imported/exported but the behavior is a bit limited.  We cannot retrieve the credentials of the given user.  If the users are uploaded, then any user uploaded will have a new password set.  The default behavior is set password to the sha256 of the login.json
   289  
   290  Example:
   291  
   292  ```sh
   293  echo -n admin.json  | openssl sha256
   294  > SHA2-256(stdin)= f172318957c89be30c2c54abcebb778a86246bbad2325d7133c4dc605319f72b
   295  ```
   296  
   297  As this can be a security risk if an intruder knows the algorithm, an option to generate random passwords is also available.  This can be configured for any `context`
   298  
   299  ```yaml
   300      user:
   301        random_password: true
   302        min_length: 8
   303        max_length: 20
   304  ```
   305  
   306  The downside, is naturally this is a one time operation.  Once the password is set it once again can no longer be retrieved.  The only time the password is printed is after the successful upload of all users.
   307  
   308  ## Global Flags
   309  
   310  These are flags that apply across all contexts. They are top level configuration and are used to drive gdg's application
   311  behavior.
   312  
   313  Here are the currently supported flags you may configure.
   314  
   315  ```yaml
   316  global:
   317    debug: true
   318    ignore_ssl_errors: false ##when set to true will ignore invalid SSL errors
   319    retry_count: 3 ## Will retry any failed API request up to 3 times.
   320    retry_delay: 5s  ## will wait for specified duration before trying again.
   321  
   322  ```
   323  
   324  - `debug`: when set will print a more verbose output (Development In Progress). Setting the env flag of DEBUG=1 will
   325    also generate verbose output for all HTTP calls.
   326  - `ignore_ssl_errors`: when set will disregard any SSL errors and proceed as expected
   327  - `retry_count`: will try N number of times before giving up
   328  - `retry_delay`: a duration to wait before trying again. Be careful with this setting, if it's too short, the retry
   329    won't matter, if too long the app will be very slow.
   330  
   331  ## Environment Overrides
   332  
   333  {{< callout context="caution" title="Caution" icon="alert-triangle" >}}
   334  Complex data type is not supported. If the value is an array it can't be currently set, via ENV overrides.
   335  
   336  {{< /callout >}}
   337  
   338  If you wish to override certain value via the environment, like credentials and such you can do so.
   339  
   340  The pattern for GDG's is as follows:  `GDG_SECTION__SECTION__keyname`
   341  
   342  For example if I want to set the context name to a different value I can use:
   343  
   344  ```sh
   345  GDG_CONTEXT_NAME="testing" gdg tools ctx show ## Which will override the value from the context file.
   346  GDG_CONTEXTS__TESTING__URL="www.google.com" Will override the URL with the one provided.
   347   ```
   348  
   349  
   350  
   351