github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/docs/sources/clients/k6/write-scenario.md (about)

     1  ---
     2  title: Write path testing
     3  weight: 20
     4  ---
     5  # Write path load testing
     6  
     7  There are multiple considerations when
     8  load testing a Loki cluster's write path.
     9  
    10  The most important consideration is the setup of the target cluster.
    11  Keep these items in mind when writing your load test.
    12  
    13  - Deployment mode. The cluster might be deployed as
    14  a single-binary, as a simple scalable deployment, or as microservices
    15  
    16  - Quantity of component instances. This aids in predicting the need
    17  to horizontally scale the quantity of component instances.
    18  
    19  - Resource allocation such as CPU, memory, disk, and network.
    20  This aids in predicting the need to vertically scale the
    21  underlying hardware.
    22  
    23  These parameters can be adjusted in the load test:
    24  
    25  * The quantity of distinct labels and their cardinality
    26  
    27      This will define how many active streams your load test will generate.
    28      Start with only a few label values,
    29      to keep the quantity of streams small enough,
    30      such that it does not overwhelm your cluster.
    31  
    32  * The batch size the client sends
    33  
    34      The batch size indirectly controls how many log lines per push request are
    35      sent. The smaller the batch size, and the larger the quantity
    36      of active streams you have,
    37      the longer it takes before chunks are flushed.
    38      Keeping lots of chunks
    39      in the ingester increases memory consumption.
    40  
    41  * The number of virtual users (VUs)
    42  
    43      VUs can be used to control the amount of parallelism with which logs should
    44      be pushed. Every VU runs it's own loop of iterations.
    45      Therfore, the number of VUs has the most impact on
    46      the generated log throughput.
    47      Since generating logs is CPU-intensive, there is a threshold above which
    48      increasing the number VUs does not result in a higher amount of log data.
    49      A rule of thumb is that the
    50      most data can be generated when the number of VUs are set to 1-1.5 times
    51      the quantity of CPU cores available on the k6 worker machine.
    52      For example,
    53      set the value in the range of 8 to 12 for an 8-core machine.
    54  
    55  * The way to run k6
    56  
    57      k6 can be run locally, self-managed in a distributed way,
    58      or it can be run highly-scalable within the k6 cloud.
    59      Whereas running your k6 load test from a single (local
    60      or remote) machine is easy to set up and fine for smaller Loki clusters,
    61      the single machine does not load test large Loki installations,
    62      because it cannot create the data to saturate the write path.
    63      Therefore, it makes sense to run
    64      the tests in the [k6 Cloud](https://k6.io/cloud/).
    65  
    66  ## Metrics
    67  
    68  The extension collects two metrics that are printed in the
    69  [end-of-test summary](https://k6.io/docs/results-visualization/end-of-test-summary/) in addition to the built-in metrics.
    70  
    71  | name | description |
    72  | ---- | ----------- |
    73  | `loki_client_uncompressed_bytes` | the quantity of uncompressed log data pushed to Loki, in bytes |
    74  | `loki_client_lines` | the number of log lines pushed to Loki |
    75  
    76  ## k6 value checks
    77  
    78  An HTTP request that successfully pushes logs to Loki
    79  responds with status `204 No Content`.
    80  The status code should be checked explicitly with a [k6 check](https://k6.io/docs/javascript-api/k6/check-val-sets-tags/).
    81  
    82  
    83  ## Javascript example
    84  
    85  ```javascript
    86  import { check, fail } from 'k6';
    87  import loki from 'k6/x/loki';
    88  
    89  /*
    90   * Host name with port
    91   * @constant {string}
    92   */
    93  const HOST = "localhost:3100";
    94  
    95  /**
    96   * Name of the Loki tenant
    97   * passed as X-Scope-OrgID header to requests.
    98   * If tenant is omitted, xk6-loki runs in multi-tenant mode,
    99   * and every VU will use its own ID.
   100   * @constant {string}
   101   */
   102  const TENANT_ID = "my_org_id"
   103  
   104  /**
   105   * URL used for push and query requests
   106   * Path is automatically appended by the client
   107   * @constant {string}
   108   */
   109  const BASE_URL = `${TENANT_ID}@${HOST}`;
   110  
   111  /**
   112   * Minimum amount of virtual users (VUs)
   113   * @constant {number}
   114   */
   115  const MIN_VUS = 1
   116  
   117  /**
   118   * Maximum amount of virtual users (VUs)
   119   * @constant {number}
   120   */
   121  const MAX_VUS = 10;
   122  
   123  /**
   124   * Constants for byte values
   125   * @constant {number}
   126   */
   127  const KB = 1024;
   128  const MB = KB * KB;
   129  
   130  /**
   131   * Definition of test scenario
   132   */
   133  export const options = {
   134    thresholds: {
   135      'http_req_failed': [{ threshold: 'rate<=0.01', abortOnFail: true }],
   136    },
   137    scenarios: {
   138      write: {
   139        executor: 'ramping-vus',
   140        exec: 'write',
   141        startVUs: MIN_VUS,
   142        stages: [
   143          { duration: '5m', target: MAX_VUS },
   144          { duration: '30m', target: MAX_VUS },
   145        ],
   146        gracefulRampDown: '1m',
   147      },
   148    },
   149  };
   150  
   151  const labelCardinality = {
   152    "app": 5,
   153    "namespace": 1,
   154  };
   155  const timeout = 10000; // 10s
   156  const ratio = 0.9; // 90% Protobuf
   157  const conf = new loki.Config(BASE_URL, timeout, ratio, labelCardinality);
   158  const client = new loki.Client(conf);
   159  
   160  /**
   161   * Entrypoint for write scenario
   162   */
   163  export function write() {
   164    let streams = randomInt(4, 8);
   165    let res = client.pushParameterized(streams, 800 * KB, 1 * MB);
   166    check(res,
   167      {
   168        'successful write': (res) => {
   169          let success = res.status === 204;
   170          if (!success) console.log(res.status, res.body);
   171          return success;
   172        },
   173      }
   174    );
   175  }
   176  
   177  /**
   178   * Return a random integer between min and max including min and max
   179   */
   180  function randomInt(min, max) {
   181    return Math.floor(Math.random() * (max - min + 1) + min);
   182  }
   183  ```