github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/docs/sources/clients/k6/write-scenario.md (about) 1 --- 2 title: Write path testing 3 weight: 20 4 --- 5 # Write path load testing 6 7 There are multiple considerations when 8 load testing a Loki cluster's write path. 9 10 The most important consideration is the setup of the target cluster. 11 Keep these items in mind when writing your load test. 12 13 - Deployment mode. The cluster might be deployed as 14 a single-binary, as a simple scalable deployment, or as microservices 15 16 - Quantity of component instances. This aids in predicting the need 17 to horizontally scale the quantity of component instances. 18 19 - Resource allocation such as CPU, memory, disk, and network. 20 This aids in predicting the need to vertically scale the 21 underlying hardware. 22 23 These parameters can be adjusted in the load test: 24 25 * The quantity of distinct labels and their cardinality 26 27 This will define how many active streams your load test will generate. 28 Start with only a few label values, 29 to keep the quantity of streams small enough, 30 such that it does not overwhelm your cluster. 31 32 * The batch size the client sends 33 34 The batch size indirectly controls how many log lines per push request are 35 sent. The smaller the batch size, and the larger the quantity 36 of active streams you have, 37 the longer it takes before chunks are flushed. 38 Keeping lots of chunks 39 in the ingester increases memory consumption. 40 41 * The number of virtual users (VUs) 42 43 VUs can be used to control the amount of parallelism with which logs should 44 be pushed. Every VU runs it's own loop of iterations. 45 Therfore, the number of VUs has the most impact on 46 the generated log throughput. 47 Since generating logs is CPU-intensive, there is a threshold above which 48 increasing the number VUs does not result in a higher amount of log data. 49 A rule of thumb is that the 50 most data can be generated when the number of VUs are set to 1-1.5 times 51 the quantity of CPU cores available on the k6 worker machine. 52 For example, 53 set the value in the range of 8 to 12 for an 8-core machine. 54 55 * The way to run k6 56 57 k6 can be run locally, self-managed in a distributed way, 58 or it can be run highly-scalable within the k6 cloud. 59 Whereas running your k6 load test from a single (local 60 or remote) machine is easy to set up and fine for smaller Loki clusters, 61 the single machine does not load test large Loki installations, 62 because it cannot create the data to saturate the write path. 63 Therefore, it makes sense to run 64 the tests in the [k6 Cloud](https://k6.io/cloud/). 65 66 ## Metrics 67 68 The extension collects two metrics that are printed in the 69 [end-of-test summary](https://k6.io/docs/results-visualization/end-of-test-summary/) in addition to the built-in metrics. 70 71 | name | description | 72 | ---- | ----------- | 73 | `loki_client_uncompressed_bytes` | the quantity of uncompressed log data pushed to Loki, in bytes | 74 | `loki_client_lines` | the number of log lines pushed to Loki | 75 76 ## k6 value checks 77 78 An HTTP request that successfully pushes logs to Loki 79 responds with status `204 No Content`. 80 The status code should be checked explicitly with a [k6 check](https://k6.io/docs/javascript-api/k6/check-val-sets-tags/). 81 82 83 ## Javascript example 84 85 ```javascript 86 import { check, fail } from 'k6'; 87 import loki from 'k6/x/loki'; 88 89 /* 90 * Host name with port 91 * @constant {string} 92 */ 93 const HOST = "localhost:3100"; 94 95 /** 96 * Name of the Loki tenant 97 * passed as X-Scope-OrgID header to requests. 98 * If tenant is omitted, xk6-loki runs in multi-tenant mode, 99 * and every VU will use its own ID. 100 * @constant {string} 101 */ 102 const TENANT_ID = "my_org_id" 103 104 /** 105 * URL used for push and query requests 106 * Path is automatically appended by the client 107 * @constant {string} 108 */ 109 const BASE_URL = `${TENANT_ID}@${HOST}`; 110 111 /** 112 * Minimum amount of virtual users (VUs) 113 * @constant {number} 114 */ 115 const MIN_VUS = 1 116 117 /** 118 * Maximum amount of virtual users (VUs) 119 * @constant {number} 120 */ 121 const MAX_VUS = 10; 122 123 /** 124 * Constants for byte values 125 * @constant {number} 126 */ 127 const KB = 1024; 128 const MB = KB * KB; 129 130 /** 131 * Definition of test scenario 132 */ 133 export const options = { 134 thresholds: { 135 'http_req_failed': [{ threshold: 'rate<=0.01', abortOnFail: true }], 136 }, 137 scenarios: { 138 write: { 139 executor: 'ramping-vus', 140 exec: 'write', 141 startVUs: MIN_VUS, 142 stages: [ 143 { duration: '5m', target: MAX_VUS }, 144 { duration: '30m', target: MAX_VUS }, 145 ], 146 gracefulRampDown: '1m', 147 }, 148 }, 149 }; 150 151 const labelCardinality = { 152 "app": 5, 153 "namespace": 1, 154 }; 155 const timeout = 10000; // 10s 156 const ratio = 0.9; // 90% Protobuf 157 const conf = new loki.Config(BASE_URL, timeout, ratio, labelCardinality); 158 const client = new loki.Client(conf); 159 160 /** 161 * Entrypoint for write scenario 162 */ 163 export function write() { 164 let streams = randomInt(4, 8); 165 let res = client.pushParameterized(streams, 800 * KB, 1 * MB); 166 check(res, 167 { 168 'successful write': (res) => { 169 let success = res.status === 204; 170 if (!success) console.log(res.status, res.body); 171 return success; 172 }, 173 } 174 ); 175 } 176 177 /** 178 * Return a random integer between min and max including min and max 179 */ 180 function randomInt(min, max) { 181 return Math.floor(Math.random() * (max - min + 1) + min); 182 } 183 ```