github.com/pyroscope-io/pyroscope@v0.37.3-0.20230725203016-5f6947968bd0/benchmark/README.md (about)

     1  ## What is this?
     2  
     3  This is a benchmark suite. It is used to benchmark pyroscope server against various workloads, particularly data ingestion from clients in push mode and data retrieval from clients in pull mode.
     4  
     5  You can use it locally to test hypothetical scenarios, or you can use it to compare the performance of different versions of pyroscope.
     6  
     7  It uses a docker-compose file to run the whole benchmark suite. It uses multiple `docker-compose.yml` files (`*.dev.yml`, `*.ci.yml`) to augment the suite for development or ci environments.
     8  
     9  Here's an overview diagram of the whole system:
    10  [![architecture](./architecture.svg)](./architecture.svg)
    11  
    12  ## Prerequisites
    13  * You'll need `docker-compose` installed.
    14  * Ideally you want to be running this on a large enough machine (> 4 cores, 16 GB of RAM), otherwise services might run out of memory. See Troubleshooting guide below for more info.
    15  
    16  ## Usage
    17  
    18  To start benchmark locally run:
    19  ```
    20  ./start.sh
    21  ```
    22  
    23  ## Configuration
    24  
    25  All of configuration is done with `config.env` file.
    26  
    27  To configure the benchmarking parameters, edit `config.env` file. It contains variables that are used by pyroscope and pyrobench.
    28  
    29  For example, the following config can be used to only test pull mode:
    30  ```
    31  PYROBENCH_APPS=0
    32  PYROBENCH_NO_EXIT_WHEN_DONE=true
    33  ```
    34  
    35  You can find documentation for all of the possible flags [here](https://github.com/pyroscope-io/pyroscope/blob/3fab8a8949b3538e98af051598edf514b14eaafe/benchmark/internal/config/config.go#L14-L39)
    36  
    37  ## Browsing results
    38  To view results open http://localhost:8080/d/tsWRL6ReZQkirFirmyvnWX1akHXJeHT8I8emjGJo/main?orgId=1.
    39  
    40  ## Configuration
    41  Edit `run-parameters.env` file to change the parameters of the benchmark run.
    42  
    43  ### Use cases
    44  
    45  #### Running indefinitely
    46  Maybe you want to leave the load generator running for an indefine amount of time.
    47  
    48  For that, just pick a big enough value for `PYROBENCH_REQUESTS`, like `100000` (the default)
    49  
    50  
    51  ## Troubleshooting
    52  
    53  Make sure you have enough memory allocated for docker, e.g on a mac:
    54  
    55  ![image](https://user-images.githubusercontent.com/662636/128406795-f4a50e4b-03d7-4eed-a637-45f0c638a16b.png)
    56  
    57  
    58  ## Design Goals
    59  
    60  This benchmark suite attempts to be as flexible as possible, meaning you can use it to test different scenarios with minimum changes to the benchmark suite. We attempt to achieve this by keeping the codebase as small as possible and thus easily editable.
    61  
    62  ## PR Benchmark
    63  
    64  This benchmark is also used to benchmark each pull request against the `main` branch.
    65  
    66  * it runs 2 instances of pyroscope (the one in the PR and the main one) in docker-compose.
    67  * generates test load against both instances
    68  * takes a screenshot of the dashboard panes
    69  * posts a PR comment using [dangerjs](https://danger.systems/js/)
    70  
    71  ## Enhancing PR comment content
    72  
    73  * To add more panes edit `monitoring/benchmark-pr.jsonnet` file
    74  * To add more things to the report update the `queries.yml` file
    75  
    76  ## TODO
    77  
    78  Future improvements:
    79  * [ ] better options for mock data
    80