vitess.io/vitess@v0.16.2/doc/design-docs/TabletServerParamsAsYAML.md (about)

     1  # VTTablet YAML configuration
     2  
     3  ## Background
     4  This issue is an expansion of #5791, and covers the details of how we’ll use yaml to implement a hierarchy of parameters for initializing and customizing TabletServer components within a process.
     5  
     6  Using the yaml approach, we intend to solve the following problems:
     7  
     8  * Multiple tablet servers need to exist within a process.
     9  * Badly named command line parameters: The new yaml specifications will have simpler and easier to remember parameter names.
    10  * Sections for improved readability.
    11  * Unreasonable default values: The new specs will introduce a simplified approach that will save the user from tweaking too many variables without knowing their consequences.
    12  * Repetition: If multiple TabletServers have to share the same parameter values, they should not be repeated.
    13  * Backward compatibility: The system must be backward compatible.
    14  
    15  ## Proposed Approach
    16  We will introduce two new command-line options that will be specified like this:
    17  ```
    18  -defaults=defaults.yaml -tablets=tablets.yaml
    19  ```
    20  
    21  The defaults option allows the user to specify the default settings for all tablets. The “tablets” option allows the user to specify tablet specific options.
    22  
    23  We’ll provide predefined “defaults” files that people can use as starting points. We’ll start with something simple like small.yaml, medium.yaml, and large.yaml.
    24  
    25  ### Protobuf
    26  We explored the use of protobufs for converting to and from yaml. However, the protobuf JSON implementation, which was required to correctly parse enums, was unable to preserve the original values for sub-objects. This was a showstopper for making defaults work.
    27  
    28  The existing `tabletenv.TabletConfig` data structure will be converted into a struct with the new names and appropriate JSON tags.
    29  
    30  ## Detailed specification
    31  The following section lists all the properties that can be specified in a yaml file along with the existing defaults.
    32  
    33  During implementation, we’ll need to decide between creating a universal protobuf that represents this structure vs converting the original one into something more usable.
    34  
    35  Convention used: same as the one followed by kubernetes, and we’ll be making use of https://github.com/kubernetes-sigs/yaml for the implementation.
    36  
    37  Note that certain properties (like tabletID) are only applicable to tablet-specific files.
    38  
    39  ```
    40  tabletID: zone-1234
    41  
    42  init:
    43    dbName:            # init_db_name_override
    44    keyspace:          # init_keyspace
    45    shard:             # init_shard
    46    tabletType:        # init_tablet_type
    47    timeoutSeconds: 60 # init_timeout
    48  
    49  db:
    50    socket:     # db_socket
    51    host:       # db_host
    52    port: 0     # db_port
    53    charSet:    # db_charset
    54    flags: 0    # db_flags
    55    flavor:     # db_flavor
    56    sslCa:      # db_ssl_ca
    57    sslCaPath:  # db_ssl_ca_path
    58    sslCert:    # db_ssl_cert
    59    sslKey:     # db_ssl_key
    60    serverName: # db_server_name
    61    connectTimeoutMilliseconds: 0 # db_connect_timeout_ms
    62    app:
    63      user: vt_app      # db_app_user
    64      password:         # db_app_password
    65      useSsl: true      # db_app_use_ssl
    66      preferSocket: true
    67    dba:
    68      user: vt_dba      # db_dba_user
    69      password:         # db_dba_password
    70      useSsl: true      # db_dba_use_ssl
    71      preferSocket: true
    72    filtered:
    73      user: vt_filtered # db_filtered_user
    74      password:         # db_filtered_password
    75      useSsl: true      # db_filtered_use_ssl
    76      preferSocket: true
    77    repl:
    78      user: vt_repl     # db_repl_user
    79      password:         # db_repl_password
    80      useSsl: true      # db_repl_use_ssl
    81      preferSocket: true
    82    appdebug:
    83      user: vt_appdebug # db_appdebug_user
    84      password:         # db_appdebug_password
    85      useSsl: true      # db_appdebug_use_ssl
    86      preferSocket: true
    87    allprivs:
    88      user: vt_allprivs # db_allprivs_user
    89      password:         # db_allprivs_password
    90      useSsl: true      # db_allprivs_use_ssl
    91      preferSocket: true
    92  
    93  oltpReadPool:
    94    size: 16                 # queryserver-config-pool-size
    95    timeoutSeconds: 0        # queryserver-config-query-pool-timeout
    96    idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout
    97    prefillParallelism: 0    # queryserver-config-pool-prefill-parallelism
    98    maxWaiters: 50000        # queryserver-config-query-pool-waiter-cap
    99  
   100  olapReadPool:
   101    size: 200                # queryserver-config-stream-pool-size
   102    timeoutSeconds: 0        # queryserver-config-query-pool-timeout
   103    idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout
   104    prefillParallelism: 0    # queryserver-config-stream-pool-prefill-parallelism
   105    maxWaiters: 0
   106  
   107  txPool:
   108    size: 20                 # queryserver-config-transaction-cap
   109    timeoutSeconds: 1        # queryserver-config-txpool-timeout
   110    idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout
   111    prefillParallelism: 0    # queryserver-config-transaction-prefill-parallelism
   112    maxWaiters: 50000        # queryserver-config-txpool-waiter-cap
   113  
   114  oltp:
   115    queryTimeoutSeconds: 30 # queryserver-config-query-timeout
   116    txTimeoutSeconds: 30    # queryserver-config-transaction-timeout
   117    maxRows: 10000          # queryserver-config-max-result-size
   118    warnRows: 0             # queryserver-config-warn-result-size
   119  
   120  hotRowProtection:
   121    mode: disable|dryRun|enable # enable_hot_row_protection, enable_hot_row_protection_dry_run
   122    # Default value is same as txPool.size.
   123    maxQueueSize: 20            # hot_row_protection_max_queue_size
   124    maxGlobalQueueSize: 1000    # hot_row_protection_max_global_queue_size
   125    maxConcurrency: 5           # hot_row_protection_concurrent_transactions
   126  
   127  consolidator: enable|disable|notOnPrimary # enable-consolidator, enable-consolidator-replicas
   128  heartbeatIntervalMilliseconds: 0         # heartbeat_enable, heartbeat_interval
   129  shutdownGracePeriodSeconds: 0            # transaction_shutdown_grace_period
   130  passthroughDML: false                    # queryserver-config-passthrough-dmls
   131  streamBufferSize: 32768                  # queryserver-config-stream-buffer-size
   132  queryCacheSize: 5000                     # queryserver-config-query-cache-size
   133  schemaReloadIntervalSeconds: 1800        # queryserver-config-schema-reload-time
   134  watchReplication: false                  # watch_replication_stream
   135  terseErrors: false                       # queryserver-config-terse-errors
   136  messagePostponeParallelism: 4            # queryserver-config-message-postpone-cap
   137  cacheResultFields: true                  # enable-query-plan-field-caching
   138  sanitizeLogMessages: false               # sanitize_log_messages
   139  
   140  
   141  # The following flags are currently not supported.
   142  # enforce_strict_trans_tables
   143  # queryserver-config-strict-table-acl
   144  # queryserver-config-enable-table-acl-dry-run
   145  # queryserver-config-acl-exempt-acl
   146  # enable-tx-throttler
   147  # tx-throttler-config
   148  # tx-throttler-healthcheck-cells
   149  # enable_transaction_limit
   150  # enable_transaction_limit_dry_run
   151  # transaction_limit_per_user
   152  # transaction_limit_by_username
   153  # transaction_limit_by_principal
   154  # transaction_limit_by_component
   155  # transaction_limit_by_subcomponent
   156  ```
   157  
   158  There are also other global parameters. VTTablet has a total of 405 flags. We may have to later move some of them into these yamls.
   159  
   160  ## Implementation
   161  We'll use the unified tabletenv.TabletConfig data structure to load defaults as well as tablet-specific values. The following changes will be made:
   162  
   163  * TabletConfig will be changed to match the above specifications.
   164  * The existing flags will be changed to update the values in a global TabletConfig.
   165  * In case of type mismatch (like time.Duration vs a "Seconds" variable), an extra step will be performed after parsing to convert the flag variables into TabletConfig members.
   166  * The code will be changed to honor the new TabletConfig members.
   167  * After the flag.Parse step, a copy of the global Config will be used as input to load the defaults file. This means that any command line flags that are not overridden in the yaml files will be preserved. This behavior is needed to support backward compatibility in case we decide to move more flags into the yaml.
   168  * For each tablet entry, we create a copy of the result TabletConfig and read the tablet yaml into those, which will set the tablet-specific values. This will then be used to instantiate a TabletServer.
   169  
   170  The exact format of the tablets.yaml file is not fully finalized. Our options are to allow a list of files where each is for a single tablet, or, to require only one file containing a dictionary of tablet-specific overrides.