github.com/eth-easl/loader@v0.0.0-20230908084258-8a37e1d94279/README.md (about)

     1  # Loader
     2  
     3  A load generator for benchmarking serverless systems.
     4  
     5  ## Prerequisites
     6  
     7  The experiments require a server-grade node running Linux (tested on Ubuntu 20, Intel Xeon). On CloudLab, one
     8  can choose the APT cluster `d430` node.
     9  
    10  ## Create a cluster
    11  
    12  First, configure `script/setup/setup.cfg`. You can specify there which vHive branch to use, loader branch, operation mode
    13  (sandbox type), maximum number of pods per node, and the Github token. All these configurations are mandatory.
    14  We currently support the following modes: containerd (`container`), Firecracker (`firecracker`), and Firecracker with
    15  snapshots (`firecracker_snapshots`).
    16  The token needs `repo` and `admin:public_key` permissions and will be used for adding SSH key to the user's account for
    17  purpose of cloning Github private repositories.
    18  Loader will be cloned on every node specified as argument of the cluster create script. The same holds for Kubernetes
    19  API server certificate.
    20  
    21  * To create a multi-node cluster, specify the node addresses as the arguments and run the following command:
    22  
    23  ```bash
    24  $ bash ./scripts/setup/create_multinode.sh <master_node@IP> <loader_node@IP> <worker_node@IP> ...
    25  ```
    26  
    27  This command will create the following setup: control plane is placed on master node, loader node is used for running
    28  loader and monitoring pods (mostly, Prometheus, if enabled in setup config), workers are used purely for working pods. In
    29  this setup, neither control plane nor workers are affected by loader and monitoring, creating more reliable measurements
    30  of performance.
    31  
    32  * Single-node cluster (experimental)
    33  
    34  ```bash
    35  $ bash ./scripts/setup/create_singlenode_container.sh <node@IP>
    36  ```
    37  
    38  This mode is only for debugging purposes, and there is no guarantees of isolation between the loader and the master-node
    39  components.
    40  
    41  ### Check cluster health (on the master node)
    42  
    43  Once the setup scripts are finished, we need to check if they have completed their jobs fully, because, e.g., there
    44  might be race conditions where a few nodes are unavailable.
    45  
    46  * First, log out the status of the control plane components on the master node and monitoring deployments by running the
    47    following command:
    48  
    49  ```bash
    50  $ bash ./scripts/util/log_kn_status.sh
    51  ```
    52  
    53  * If you see everything is `Running`, check if the cluster capacity is stretched to the desired capacity by running the
    54    following script:
    55  
    56  ```bash
    57  $ bash ./scripts/util/check_node_capacity.sh
    58  ```
    59  
    60  If you want to check
    61  the [pod CIDR range](https://www.ibm.com/docs/en/cloud-private/3.1.2?topic=networking-kubernetes-network-model), run the
    62  following
    63  
    64  ```bash
    65  $ bash ./scripts/util/get_pod_cidr.sh
    66  ```
    67  
    68  * Next, try deploying a function (`myfunc`) with the desired `scale` (i.e., the number of instances that must be active
    69    at all times).
    70  
    71  ```bash
    72  $ bash ./scripts/util/set_function_scale.sh <scale>
    73  ```
    74  
    75  * One should verify that the system was able to start the requested number of function instances, by using the following
    76    command.
    77  
    78  ```bash
    79  $ kubectl -n default get podautoscalers
    80  ```
    81  
    82  ## Tune the timing for the benchmark function
    83  
    84  Before start any experiments, the timing of the benchmark function should be tuned so that it consumes the required
    85  service time more precisely.
    86  
    87  First, run the following command to deploy the timing benchmark that yields the number of execution units* the function
    88  needs to run given a required service time.
    89  
    90  * The execution unit is approximately 100 `SQRTSD` x86 instructions.
    91  
    92  ```bash
    93  $ kubectl apply -f server/benchmark/timing.yaml
    94  ```
    95  
    96  Then, monitor and collect the `ITERATIONS_MULTIPLIER` from the job logs as follows:
    97  
    98  ```bash
    99  $ watch kubectl logs timing
   100  ```
   101  
   102  Finally, set the `ITERATIONS_MULTIPLIER` in the function template `workloads/$SANDBOX_TYPE/trace_func_go.yaml` to the
   103  value previously collected.
   104  
   105  To account for difference in CPU performance set `ITERATIONS_MULTIPLIER=102` if using
   106  Cloudlab `xl170` or `d430` machines. (Date of measurement: 18-Oct-2022)
   107  
   108  ## Single execution
   109  
   110  To run load generator use the following command:
   111  
   112  ```bash
   113  $ go run cmd/loader.go --config cmd/config.json
   114  ```
   115  
   116  Additionally, one can specify log verbosity argument as `--verbosity [info, debug, trace]`. The default value is `info`.
   117  
   118  For to configure the workload for load generator, please refer to `docs/configuration.md`.
   119  
   120  There are a couple of constants that should not be exposed to the users. They can be examined and changed
   121  in `pkg/common/constants.go`.
   122  
   123  ## Build the image for a synthetic function
   124  
   125  The reason for existence of Firecracker and container version is because of different ports for gRPC server. Firecracker
   126  uses port 50051, whereas containers are accessible via port 80.
   127  
   128  Synthetic function runs for a given time and allocates the required amount of memory:
   129  
   130  * `trace-firecracker` - for Firecracker
   131  * `trace-container` - for regular containers
   132  
   133  For testing cold start performance:
   134  
   135  * `empty-firecracker` - for Firecracker - returns immediately
   136  * `empty-container` - for regular containers - returns immediately
   137  
   138  ```bash
   139  $ make build <trace-firecracker|trace-container|empty-firecracker|empty-container>
   140  ```
   141  
   142  ## Clean up between runs
   143  
   144  ```bash
   145  $ make clean
   146  ```
   147  
   148  ## Running the Experiment Driver
   149  
   150  Within the tools/driver folder, Configure the driverConfig.json file based on your username on Cloudlab, 
   151  the address of the loader node that you are running the experiment on and any experiment parameters you'd like to set.
   152  Note that you need to have the relevant trace files on the machine running the experiment driver, they will then get
   153  transferred to the loader node.  
   154  Then run the following command to launch an experiment:
   155  ```bash
   156  $ go run experiment_driver.go -c driverConfig.json
   157  ```
   158  For more details take a look at the README in the tools/driver folder.
   159  
   160  ---
   161  
   162  For more options, please see the `Makefile`.
   163  
   164  ## Using OpenWhisk
   165  
   166  For instructions on how to use the loader with OpenWhisk go to `openwhisk_setup/README.md`.
   167