github.com/square/finch@v0.0.0-20240412205204-6530c03e2b96/docs/content/data/limits.md (about)

     1  ---
     2  weight: 4
     3  ---
     4  
     5  _Data limits_ limit how much or how fast Finch reads or writes data.
     6  By default, Finch has no limits.
     7  But most benchmarks use at least two limits: row count and [`runtime`]({{< relref "syntax/stage-file#runtime" >}}).
     8  Finch has these limits and others.
     9  
    10  Multiple limits can be specified.
    11  Execution is stopped (or delayed for throughput limits) when any limit is reached.
    12  
    13  {{< toc >}}
    14  
    15  ## Count
    16  
    17  The [`-- rows`]({{< relref "syntax/trx-file#rows" >}}) statement modifier will stop the client after inserting the configured number of rows.
    18  Combine with multiple clients and [CSV expansion]({{< relref "syntax/trx-file#csv" >}}) to bulk insert rows.
    19  
    20  {{< hint type=warning >}}
    21  Multi-client, multi-row inserts will exceed the configured row count by one set of multi-rows because the count tracking is async.
    22  If you need the exact number of `-- rows`, use a single client, or submit a PR to improve this feature.
    23  {{< /hint >}}
    24  
    25  You can indirectly limit data access with limited iterations:
    26  
    27  * [`stage.workload[].iter`]({{< relref "syntax/stage-file#iter" >}})
    28  * [`stage.workload[].iter-clients`]({{< relref "syntax/stage-file#iter-clients" >}})
    29  * [`stage.workload[].iter-exec-group`]({{< relref "syntax/stage-file#iter-exec-group" >}})
    30  
    31  For example, `iter = 100` for a single-row `UPDATE` will limit the client to 100 updates.
    32  Or to update a lot of rows quickly, use multiple clients and `iter-clients` to apply a shared limit to all clients.
    33  
    34  ## Size
    35  
    36  Row counts are common but arbitrary.
    37  A thousand rows of a huge table with many secondary indexes and blob columns is significantly different than one million rows of table with a few integer columns.
    38  How much RAM a system has (and MySQL is configured to use) is another factor: even 10 million rows might fit in RAM.
    39  
    40  Depending on the benchmark, it might be better to generate certain data sizes, rather than row counts:
    41  
    42  * [`database-size`]({{< relref "syntax/trx-file#database-size" >}})
    43  * [`table-size`]({{< relref "syntax/trx-file#table-size" >}})
    44  
    45  These statement modifiers are usually used in [DDL stages]({{< relref "benchmark/overview#ddl" >}}).
    46  Combine with a [parallel load]({{< relref "benchmark/workload#parallel-load" >}}) and you can load terabytes of data relatively quickly.
    47  (For benchmarking, "relatively quickly" means hours and days for terabytes of data.)
    48  
    49  There are currently no size-based data limits built into any [data generators]({{< relref "data/generators" >}}), but it would be possible to implement for both reading and writing data.
    50  
    51  ## Throughput
    52  
    53  Finch has QPS (queries per second) and TPS (transactions per second) throughput limits.
    54  
    55  Use [`stage.qps`]({{< relref "syntax/stage-file#qps" >}}) to ensure that Finch does not exceed a certain QPS no matter the workload or other limits.
    56  This is a top-level limit, and since limits are combined with logical OR, even a higher limit specified in the workload will be limited to `stage.qps`.
    57  
    58  The workload-specific QPS limits are:
    59  
    60  * [`stage.workload[].qps`]({{< relref "syntax/stage-file#qps-1" >}})
    61  * [`stage.workload[].qps-clients`]({{< relref "syntax/stage-file#qps-clients" >}})
    62  * [`stage.workload[].qps-exec-group`]({{< relref "syntax/stage-file#qps-exec-group" >}})
    63  
    64  Finch checks every QPS limit before executing each query.
    65  Delay for QPS throttling is _not_ measured or reported.
    66  
    67  The TPS limits are the same, just "tps" instead of "qps":
    68  
    69  * [`stage.tps`]({{< relref "syntax/stage-file#tps" >}})
    70  * [`stage.workload[].tps`]({{< relref "syntax/stage-file#tps-1" >}})
    71  * [`stage.workload[].tps-clients`]({{< relref "syntax/stage-file#tps-clients" >}})
    72  * [`stage.workload[].tps-exec-group`]({{< relref "syntax/stage-file#tps-exec-group" >}})
    73  
    74  Finch checks all TPS limits on explicit `BEGIN` statements.
    75  And the TPS [statistic]({{< relref "benchmark/statistics" >}}) is measured on explicit `COMMIT` statements.
    76  
    77  ## Automatic
    78  
    79  Finch automatically sets `iter = 1` for a client group with any DDL in any assigned trx.
    80  See [Benchmark / Workload / Auto-allocation]({{< relref "benchmark/workload#auto-allocation" >}}).