github.com/google/syzkaller@v0.0.0-20240517125934-c0f1611a36d6/docs/syz_verifier.md (about)

     1  # syz-verifier
     2  
     3  Many bugs are easy to detect: they might cause assertions failures, crash our
     4  system, or cause other forms of undefined behaviour detectable by various
     5  dynamic analysis tools. However, certain classes of bugs, referred to as
     6  *semantic bugs*, cause none of these while still resulting in a misbehaving
     7  faulty system.
     8  
     9  To find semantic bugs, one needs to establish a specification of the system's
    10  *intended behaviour*. Depending on the complexity of the system, creating and
    11  centralising such specifications can be difficult. For example, the
    12  "specification" of the Linux kernel is not found in one place, but is rather a
    13  collection of documentation, man pages, and the implied expectations of a vast
    14  collection of user space programs. As such, detecting semantic bugs in the
    15  Linux kernel is significantly harder than other classes of bugs. Indeed, many
    16  test suites are meant to detect regressions, but creating and maintaining test
    17  cases, as well as covering new features requires significant amounts of
    18  engineering effort.
    19  
    20  *Differential fuzzing* is a way to automate detection of semantic bugs by
    21  providing the same input to different implementations of the same systems and
    22  then cross-comparing the resulting behaviour to determine whether it is
    23  identical. In case the systems disagree, at least one of them is assumed to be
    24  wrong.
    25  
    26  `syz-verifier` is a differential fuzzing tool that cross-compares the execution
    27  of programs on different versions of the Linux kernel to detect semantic bugs.
    28  
    29  The architecture of `syz-verifier` is shown in the following diagram.
    30  
    31  ![Architecture overview](syz_verifier_structure.png)
    32  
    33  The `syz-verifier` process starts and manages VM instances with the kernels to
    34  be cross-compared. It also starts the `syz-runner` process on the VMs.
    35  Communication between the host and the guest is done via RPCs.
    36  
    37  `syz-verifier` generates and sends a continuous stream of programs to
    38  `syz-runner` via RPCs while `syz-runner` is responsible for starting
    39  `syz-executor` processes and turning the program into input for those.
    40  `syz-executor` processes the input, which triggers a sequence of syscalls in
    41  the kernel. Then, `syz-runner` collects the results and sends them back to the
    42  host.
    43  
    44  At the moment, the results contain the errnos returned by each system call.
    45  When `syz-verifier` has received results from all the kernels for a specific
    46  program, it verifies them to ensure they are identical. If a mismatch is found,
    47  the program is rerun on all the kernels to ensure the mismatch is not flaky
    48  (i.e. it didn't occur because of some background activity or external state).
    49  If the mismatch occurs in all reruns, `syz-verifier` creates a report for the
    50  program and write it to persistent storage.
    51  
    52  # How to use `syz-verifier`
    53  
    54  After cloning the repository (see how
    55  [here](/docs/linux/setup.md#go-and-syzkaller)), build the tool as:
    56  
    57  ```
    58  make verifier runner executor
    59  ```
    60  
    61  To start using the tool, separate configuration files need to be created for
    62  each kernel you want to include in the verification. An example of Linux
    63  configs can be found [here](/docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md#syzkaller). The configuration files
    64  are identical to those used by `syz-manager`.
    65  
    66  If you want to generate programs from a specific set of system calls, these can
    67  be listed in the kernel config files using the `enable_syscalls` option. If you
    68  want to disable some system calls, use the `disable_syscalls` option.
    69  
    70  Start `syz-verifier` as:
    71  ```
    72  ./bin/syz-verifier -configs=kernel0.cfg,kernel1.cfg
    73  ```
    74  
    75  `syz-verifier` will also gather statistics throughout execution. They will be
    76  printed to `stdout` by default, but an alternative file can be specified using
    77  the `stat` flag.
    78  
    79  # How to interpret the results
    80  
    81  Results can be found in `workdir/results`.
    82  
    83  When `syz-verifier` finds a mismatch in a program, it will create a report for
    84  that program. The report lists the results returned for each system call, by
    85  each of the cross-compared kernels, highlighting the ones were a mismatch was
    86  found. The system calls are listed in the order they appear in the program.
    87  
    88  An extract of such a report is shown below:
    89  
    90  ```
    91  ERRNO mismatches found for program:
    92  
    93  [=] io_uring_register$IORING_REGISTER_PERSONALITY(0xffffffffffffffff, 0x9, 0x0, 0x0)
    94          ↳ Pool: 0, Flags: 3, Errno: 9 (bad file descriptor)
    95          ↳ Pool: 1, Flags: 3, Errno: 9 (bad file descriptor)
    96  
    97  [=] syz_genetlink_get_family_id$devlink(&(0x7f0000000000), 0xffffffffffffffff)
    98          ↳ Pool: 0, Flags: 3, Errno: 2 (no such file or directory)
    99          ↳ Pool: 1, Flags: 3, Errno: 2 (no such file or directory)
   100  
   101  [!] r1 = io_uring_setup(0x238e, &(0x7f0000000240)={0x0, 0xf39a, 0x20, 0x0, 0x146})
   102          ↳ Pool: 0, Flags: 3, Errno: 6 (no such device or address)
   103          ↳ Pool: 1, Flags: 3, Errno: 9 (bad file descriptor)
   104  ...
   105  ```
   106  
   107  The order of the results is given by the order in which configuration files
   108  were passed so `Pool: 0 ` reports results for the kernel created using
   109  `kernel0.cfg` and so on.
   110  
   111  The [Flags](/pkg/ipc/ipc.go#L82) can be used to determine the state reached by
   112  the system call:
   113  * `0` = syscall not even started
   114  * `1` = syscall started
   115  * `3` = syscall finished executing
   116  * `7` = syscall blocked