github.com/onflow/flow-go@v0.35.7-crescendo-preview.23-atree-inlining/config/README.md (about)

     1  ## config 
     2  config is a package to hold all configuration values for each Flow component. This package centralizes configuration management providing access 
     3  to the entire FlowConfig and utilities to add a new config value, corresponding CLI flag, and validation.
     4  
     5  ### Package structure
     6  The root config package contains the FlowConfig struct and the default config file [default-config.yml](https://github.com/onflow/flow-go/blob/master/config/default-config.yml). The `default-config.yml` file is the default configuration that is loaded when the config package is initialized.
     7  The `default-config.yml` is a snapshot of all the configuration values defined for Flow.
     8  Each subpackage contains configuration structs and utilities for components and their related subcomponents. These packages also contain the CLI flags for each configuration value. The [netconf](https://github.com/onflow/flow-go/tree/master/network/netconf) package
     9  is a good example of this pattern. The network component is a large component made of many other large components and subcomponents. Each configuration 
    10  struct is defined for all of these network related components in the netconf subpackage and CLI flags. 
    11  
    12  ### Overriding default values
    13  The entire default config can be overridden using the `--config-file` CLI flag. When set the config package will attempt to parse the specified config file and override all the values 
    14  defined. A single default value can be overridden by setting the CLI flag for that specific config. For example `--networking-connection-pruning=false` will override the default network connection pruning 
    15  config to false.
    16  Override entire config file.
    17  ```shell
    18  go build -o flow-access-node ./cmd/access
    19  ./flow-access-node --config-file=config/config.yml
    20  ```
    21  Override a single configuration value.
    22  ```shell
    23  go build -o flow-access-node ./cmd/access
    24  ./flow-access-node --networking-connection-pruning=false
    25  ```
    26  ### Adding a new config value
    27  Adding a new config to the FlowConfig can be done in a few easy steps.
    28  
    29  The network package can be used as a good example of how to add CLI flags and will be used in the steps below.
    30  
    31  1. Create a new configuration package for the new configuration structs and CLI flags. Although it is encouraged to define all package configuration structs and CLI flags as a subpackage of the [config package](https://github.com/onflow/flow-go/tree/master/config),
    32  the configuration package can live anywhere. This configuration package will define the configuration structs and CLI flags for overriding.
    33      ```shell
    34      mkdir example_config 
    35      ```
    36      For the network package we have a configuration subpackage created in [network/netconf](https://github.com/onflow/flow-go/tree/master/network/netconf). 
    37  
    38  2. Add a new CLI flag for the config value. 
    39      ```go
    40      const workersCLIFlag = "app-workers"
    41      flags.String(workersCLIFlag, 1, "number of app workers")
    42      ```
    43  
    44     All network package CLI flags are defined in [network/netconf/flags.go](https://github.com/onflow/flow-go/blob/master/network/netconf/flags.go) in:
    45      - `const` block
    46      - `AllFlagNames` function
    47      - `InitializeNetworkFlags` function
    48  
    49      `InitializeNetworkFlags` is used during initialization of all flags 
    50      in the `InitializePFlagSet` function in the [config/base_flags.go](https://github.com/onflow/flow-go/blob/master/config/base_flags.go).
    51  
    52  3. Add the configuration as a new field to an existing configuration struct or create a new configuration struct. Each configuration struct must be a field on the FlowConfig struct so that it is unmarshalled during configuration initialization.
    53      Each field on a configuration struct must contain the following field tags.
    54     1. `validate` - validate tag is used to perform validation on field structs using the [validator](https://github.com/go-playground/validator) package. In the example below you will notice 
    55     the `validate:"gt=0"` tag, this will ensure that the value of `AppWorkers` is greater than 0. The top level `FlowConfig` struct has a Validate method that performs struct validation. This 
    56     validation is done with the validator package, each validate tag on every struct field and sub struct field will be validated and validation errors are returned.
    57     2. `mapstructure` - [mapstructure](https://github.com/mitchellh/mapstructure) tag is used for unmarshalling and must match the CLI flag name defined in step or else the field will not be set when the config is unmarshalled.
    58     ```go
    59          type MyComponentConfig struct {
    60              AppWorkers int `validate:"gt=0" mapstructure:"app-workers"`
    61          }
    62      ```
    63     It's important to make sure that each mapstructure field tag matches the CLI flag name in [config/default-config.yml](https://github.com/onflow/flow-go/blob/master/config/default-config.yml) to avoid parsing errors.
    64     
    65     All network package configuration structs are defined in [network/netconf/config.go](https://github.com/onflow/flow-go/blob/master/network/netconf/config.go)
    66  
    67  4. Add the new configuration properties and default values to [config/default-config.yml](https://github.com/onflow/flow-go/blob/master/config/default-config.yml). Ensure that each new property added matches the mapstructure value of the configuration struct field.
    68      ```yaml
    69        config-file: "./default-config.yml"
    70        network-config:
    71        ...
    72        my-component:
    73          app-workers: 1
    74      ```
    75     
    76      All network package configuration values are defined under `network-config` in `default-config.yml`
    77  
    78  5. If a new struct was created in step 3, add it as a new field to `FlowConfig` struct in [config/config.go](https://github.com/onflow/flow-go/blob/master/config/config.go). In the previous steps we added a new config struct and added a new property to [config/default-config.yml](https://github.com/onflow/flow-go/blob/master/config/default-config.yml) for the `my-component` struct. This property name
    79      must match the mapstructure field tag for the struct when added to `FlowConfig`.
    80      ```go
    81      // FlowConfig Flow configuration.
    82      type FlowConfig struct {
    83          ConfigFile    string          `validate:"filepath" mapstructure:"config-file"`
    84          NetworkConfig *network.Config `mapstructure:"network-config"`
    85          MyComponentConfig *mypackage.MyComponentConfig `mapstructure:"my-component"`
    86      }
    87      ```
    88     
    89      The network package configuration struct, `NetworkConfig`, is already embedded as a field in `FlowConfig` struct.
    90      This means that new fields can be added to the network package configuration struct without having to update the `FlowConfig` struct.
    91  
    92  ### Nested structs
    93  In an effort to keep the configuration yaml structure readable some configuration properties will be defined in nested structs. When this is the case the mapstructure [squash](https://pkg.go.dev/github.com/mitchellh/mapstructure#example-Decode-EmbeddedStruct) tag must be used in the parent struct so that the corresponding nested struct will be 
    94  flattened before the configuration is unmarshalled. This is used in the network package which is a collection of configuration structs nested in the `network.Config` struct:
    95  ```go
    96  type Config struct {
    97      UnicastConfig                 `mapstructure:",squash"`
    98      p2pconf.ResourceManagerConfig `mapstructure:",squash"`
    99      ConnectionManagerConfig       `mapstructure:",squash"`
   100      
   101      p2pconf.GossipSubConfig `mapstructure:",squash"`
   102      AlspConfig              `mapstructure:",squash"`
   103      ...
   104  }
   105  ```
   106  Each nested struct must have the mapstructure squash tag so that the nested struct will be flattened before unmarshalling.
   107  The nesting can be as deep as needed. For example, the `UnicastConfig` struct in the `Config` struct has a nested `UnicastRateLimitersConfig` struct that also uses the `squash` tag:
   108  
   109  ```go
   110  type UnicastConfig struct {
   111      // UnicastRateLimitersConfig configuration for all unicast rate limiters.
   112      UnicastRateLimitersConfig `mapstructure:",squash"`
   113      ...
   114  }
   115  ```
   116  
   117  ### Setting Aliases
   118  Most configs will not be defined on the top layer FlowConfig but instead be defined on nested structs and in nested properties of the configuration yaml. When the default config is initially loaded the underlying config [viper](https://github.com/spf13/viper) store will store 
   119  each configuration with a key that is prefixed with each parent property. For example, because `networking-connection-pruning` is found on the `network-config` property of the configuration yaml, the key used by the config store to 
   120  store this config value will be prefixed with `network-config` e.g.
   121  ```network-config.networking-connection-pruning```
   122  
   123  Later in the config process we bind the underlying config store with our pflag set, this allows us to override default values using CLI flags.
   124  At this time the underlying config store would have 2 separate keys `networking-connection-pruning` and `network-config.networking-connection-pruning` for the same configuration value. This is because we don't use the network prefix for the CLI flags
   125  used to override network configs. As a result, an alias must be set from `network-config.networking-connection-pruning` -> `networking-connection-pruning` so that they both point to the value loaded from the CLI flag. See `SetAliases` in [network/netconf/flags.go](https://github.com/onflow/flow-go/blob/master/config/network/netconf/flags.go) in the network package for a reference.