github.com/google/syzkaller@v0.0.0-20251211124644-a066d2bc4b02/docs/syz_testbed.md (about)

     1  # syz-testbed
     2  
     3  `syz-testbed` is the tool that simplifies the process of evaluating the
     4  performance of different syzkaller versions (or configurations) against each
     5  other. The tool automates checking out syzkaller repos, building them, running
     6  `syz-manager`s and collecting/summarizing their results.
     7  
     8  ## Configuring syz-testbed
     9  
    10  `syz-testbed` requires a JSON config file. An example of such a file:
    11  
    12  ```json
    13  {
    14    "workdir": "/tmp/syz-testbed-workdir/",
    15    "corpus": "/tmp/corpus.db",
    16    "target": "syz-manager",
    17    "max_instances": 5,
    18    "run_time": "24h",
    19    "http": "0.0.0.0:50000",
    20    "checkouts": [
    21      {
    22        "name": "first",
    23        "repo": "https://github.com/google/syzkaller.git",
    24      },
    25      {
    26        "name": "second",
    27        "repo": "https://github.com/google/syzkaller.git",
    28        "branch": "some-dev-branch",
    29      }
    30    ],
    31    "manager_config": {
    32  	  "target": "linux/amd64",
    33  	  "kernel_obj": "/tmp/linux-stable",
    34  	  "image": "/tmp/kernel-image/trixie.img",
    35  	  "sshkey": "/tmp/kernel-image/trixie.id_rsa",
    36  	  "procs": 8,
    37  	  "type": "qemu",
    38  	  "vm": {
    39            "count": 2,
    40            "kernel": "/tmp/linux-stable/arch/x86/boot/bzImage",
    41            "cpu": 2,
    42            "mem": 2048
    43  	  }
    44    }
    45  }
    46  ```
    47  
    48  Given such a configuration file, `syz-testbed` will do the following:
    49  1. Check out the `master` branch of `https://github.com/google/syzkaller.git`
    50     into `/tmp/syz-testbed-workdir/checkouts/first/` and build it.
    51  2. Check out the `some-dev-branch` of `https://github.com/google/syzkaller.git`
    52     into `/tmp/syz-testbed-workdir/checkouts/second/` and build it.
    53  3. Set up and run 3 instances of `first` and 2 instances of `second`
    54  (`max_instances = 5`).
    55  
    56  The directory structure looks as follows:
    57  ```
    58  /tmp/syz-testbed-workdir/
    59  └── checkouts
    60      ├── first
    61      │   ├── run-first-0
    62      │   │   ├── log.txt
    63      │   │   ├── manager.cfg
    64      │   │   └── workdir
    65      │   ├── run-first-1
    66      │   │   ├── log.txt
    67      │   │   ├── manager.cfg
    68      │   │   └── workdir
    69      │   └── run-first-4
    70      │   │   ├── log.txt
    71      │   │   ├── manager.cfg
    72      │   │   └── workdir
    73      └── second
    74          ├── run-second-2
    75          │   ├── log.txt
    76          │   ├── manager.cfg
    77          │   └── workdir
    78          └── run-second-3
    79              ├── log.txt
    80              ├── manager.cfg
    81              └── workdir
    82  ```
    83  4. After 24 hours (as `run_hours` is 24), stop those 5 instances.
    84  5. Create and run 2 instances of `first` and 3 instances of `second`.
    85  6. <Repeat those steps over and over>
    86  
    87  The tool stops after receiving a SIGINT (e.g. after Ctrl+C) or a SIGTERM
    88  signal. Also, if one of the instances has exited due to some error, this also
    89  stops the whole experiment.
    90  
    91  ## Web interface
    92  
    93  The tool has a simple web interface that displays the current information about
    94  the experiment (the number of active and finished instances, the time until
    95  instances are stopped, etc.) and the latest statistics collected from the
    96  `syz-manager`s.
    97  
    98  If the `benchmp` parameter points to the `syz-benchcmp` executable, then the web
    99  interface can also generate graphs of various parameters over time or the number
   100  of executions.
   101  
   102  In order to enable the interface, set the `http` parameter to the IP address and
   103  port to which `syz-testbed` should bind. E.g. `"http": "0.0.0.0:50000"`.
   104  
   105  ## Statistics
   106  
   107  `syz-testbed` provides two "views" of the statistics:
   108  1. `complete` - only includes data from the finished instances (i.e. those that
   109     have been running for `run_hours`).
   110  2. `all` - also includes the data from the currently active instances. The
   111     statistics from the finished instances is winded back to match the current
   112     uptime of the active instances.
   113  
   114  Therefore, the statistics is laid out the following way.
   115  
   116  ```bash
   117  $ tree -L 2 /tmp/syz-testbed-workdir/
   118  /tmp/syz-testbed-workdir/
   119  ├── stats_all
   120  │   ├── benches
   121  │   │   ├── avg_first.txt
   122  │   │   ├── avg_second.txt
   123  │   ├── bugs.csv
   124  │   ├── checkout_stats.csv
   125  │   └── instance_stats.csv
   126  ├── stats_completed
   127  │   ├── benches
   128  │   │   ├── avg_first.txt
   129  │   │   ├── avg_second.txt
   130  │   ├── bugs.csv
   131  │   ├── checkout_stats.csv
   132  │   └── instance_stats.csv
   133  └── testbed.csv
   134  ```
   135  
   136  1. `bugs.csv` contains all the bugs found by the running instances. If a single
   137     checkout has several instances (i.e. `count` > 1), `syz-testbed` takes a
   138     union of bugs found by them. The purpose is ultimately to collect all bugs
   139     that could be found by that version of syzkaller.
   140  2. Statistics that is generated by individual `syz-manager`s is saved into
   141     `instance_stats.csv`. The same data is also averaged among instances that
   142     belong to the same checkouts and saved into the `checkout_stats.csv` file.
   143  3. Bench files (see `tools/syz-benchcmp`) of all `syz-manager`s belonging to a
   144     single checkout are averaged and saved into the corresponding files in the
   145     `benches` folder.
   146  
   147  The statics is updated once every 90 seconds.
   148  
   149  ## Running syz-testbed
   150  
   151  First, checkout the most recent version of syzkaller itself:
   152  
   153  ```
   154  $ git clone https://github.com/google/syzkaller.git
   155  ```
   156  
   157  Then, build `syz-testbed`:
   158  
   159  ```
   160  $ cd syzkaller/tools/syz-testbed/
   161  $ go build
   162  ```
   163  
   164  Write and save the configuration file (e.g. into the `config.json` file). Then,
   165  `syz-testbed` can be run using the following command:
   166  
   167  ```
   168  $ ./syz-testbed -config config.json
   169  ```
   170  
   171  Stopping the `syz-testbed` process results in stopping all the syzkaller instances.
   172  
   173  ## Testing syz-repro
   174  
   175  `syz-testbed` can also be used to test syzkaller's ability to reproduce bugs. To do
   176  so, set the `target` property in the `syz-testbed`'s config file to `syz-repro`.
   177  
   178  One can also specify the source of the crash log files. This is either just a folder,
   179  whose files will be treated accordingly or it can be a syzkaller's workdir.
   180  `input_logs` must point to the folde with crash logs - `syz-testbed` will traverse
   181  it and treat each file as an input. Otherwise, `input_workdir` must be used.
   182  
   183  For example:
   184  ```json
   185    "repro_config": {
   186      "input_workdir": "/tmp/some-syzkaller-workdir",
   187      "crashes_per_bug": 2,
   188      "skip_bugs": ["SYZFAIL", "no output", "corrupted", "lost connection"]
   189    },
   190  ```
   191  
   192  In this case, `syz-testbed` will traverse all bugs found by the syzkaller, skip
   193  those that match "SYZFAIL", "no output", "corrupted" or "lost connection", then
   194  pick 2 random crash logs for each such bug for later processing.
   195  
   196  `syz-testbed` will check out and compile the specified syzkaller instances and will
   197  go on executing their `syz-repro`s on each picked up crash log file, as long as
   198  the tool is not stopped.