github.com/google/syzkaller@v0.0.0-20240517125934-c0f1611a36d6/docs/setup_syzbot.md (about) 1 # How to set up syzbot 2 3 This doc will be useful to you: 4 - should you wish to hack on user interface bits like the dashboard / mailing list integration or 5 - should you wish to continuously run a separate syzbot dashboard for your own kernels 6 7 Note: For most development purposes you don't need a full syzbot setup. The meat of syzkaller is really located in syz-manager, syz-fuzzer and syz-executor. You can run syz-manager directly which is usually what you will want to do during fuzzer development. [See this documentation for syz-manager setup instructions](setup.md). 8 9 This doc assumes that you: 10 - have a GCP account and billing setup 11 - created a GCP project for running syzbot in 12 - are running a reasonably modern linux distro 13 - locally installed `gcloud`, `ssh`, `go` and `build-essential` 14 - may need to install `google-cloud-sdk-app-engine-go` for the GAE deployment to work 15 - ran `gcloud auth login` to run authenticated gcloud commands 16 - read [go/syzbot-setup](https://goto.google.com/syzbot-setup) if you are a Googler 17 18 While most syzkaller bits happily run on various operating systems, the syzbot dashboard does not. The dashboard is a Google App Engine or GAE project. GAE allows developers to develop web applications without needing to worry about the underlying servers. Instead developers just push their code and GAE takes care of web servers, load balancers and more. Hence this document is more Google Cloud focused than the rest of our documentation. 19 20 We will also deploy a syz-ci instance. syz-ci keeps track of the syzkaller and kernel repositories and continuously rebuilds the kernel under test, itself and other syzkaller components when new commits land in the upstream repositories. syz-ci also takes care of (re)starting syz-manager instances, which in turn (re)start VMs fuzzing the target kernel. For simplicity we will run everything in this doc on GCP even though syz-ci could run elsewhere. 21 22  23 24 25 ## Deploying Syz-ci 26 27 [local] First prepare an initial syz-ci build locally (later syz-ci rebuilds itself) and a rootfs: 28 29 ```sh 30 # Most syzkaller components can be build even outside of the GOPATH, however 31 # the syzbot app engine deployment only works from the GOPATH right now.. 32 export GOOGLE_GO=$HOME/gopath/src/github.com/google/ 33 mkdir -p $GOOGLE_GO 34 git clone https://github.com/google/syzkaller.git 35 mv syzkaller $GOOGLE_GO/ 36 cd $GOOGLE_GO/syzkaller 37 make ci 38 39 cd ~/repos 40 git clone git://git.buildroot.net/buildroot 41 cd buildroot 42 $GOOGLE_GO/syzkaller/tools/create-buildroot-image.sh 43 ``` 44 45 [local] Enable various services in the project, create a VM, storage bucket, scp assets and login: 46 47 ```sh 48 export PROJECT='your-gcp-project' 49 export CI_HOSTNAME='ci-linux' 50 export GOOGLE_GO=$HOME/gopath/src/github.com/google/ 51 52 gcloud services enable compute.googleapis.com --project="$PROJECT" 53 gcloud compute instances create "$CI_HOSTNAME" --image-family=debian-11 --image-project=debian-cloud --machine-type=e2-standard-16 --zone=us-central1-a --boot-disk-size=250 --scopes=cloud-platform --project="$PROJECT" 54 55 # Enabling compute.googleapis.com created a service account. We allow the syz-ci VM 56 # to assume the permissions of that service account. As syz-ci needs query / create / delete 57 # other VMs in the project, we need to give the new service account various permissions 58 gcloud services enable iam.googleapis.com --project $PROJECT 59 SERVICE_ACCOUNT=`gcloud iam service-accounts list --filter 'displayName:Compute Engine default service account' --format='value(email)' --project $PROJECT` 60 gcloud projects add-iam-policy-binding "$PROJECT" --role="roles/editor" --member="serviceAccount:$SERVICE_ACCOUNT" --quiet 61 62 gcloud services enable storage-api.googleapis.com --project="$PROJECT" 63 gsutil mb -p "$PROJECT" "gs://$PROJECT-bucket" 64 65 gcloud services enable cloudbuild.googleapis.com --project="$PROJECT" 66 67 # We need to wait a bit for the VM to become accessible. Let's just… 68 sleep 10 69 70 # Copy in buildroot 71 gcloud compute scp --zone us-central1-a --project="$PROJECT" ~/repos/buildroot/output/images/disk.img "$CI_HOSTNAME":~/ 72 73 # Copy in syz-ci binary 74 gcloud compute scp --zone us-central1-a --project="$PROJECT" $GOOGLE_GO/syzkaller/bin/syz-ci "$CI_HOSTNAME":~/ 75 76 # Prepare syz-ci config 77 cat <<EOF > /tmp/config.ci 78 { 79 "name": "$CI_HOSTNAME", 80 "http": ":80", 81 "manager_port_start": 50010, 82 "syzkaller_repo": "https://github.com/google/syzkaller.git", 83 "managers": [ 84 { 85 "repo": "git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git", 86 "repo_alias": "upstream", 87 "userspace": "disk.img", 88 "kernel_config": "config/linux/upstream-apparmor-kasan.config", 89 "manager_config": { 90 "name": "ci-upstream-kasan-gce", 91 "target": "linux/amd64", 92 "procs": 6, 93 "type": "gce", 94 "vm": { 95 "count": 5, 96 "machine_type": "e2-standard-2", 97 "gcs_path": "$PROJECT-bucket/disks" 98 }, 99 "disable_syscalls": [ "perf_event_open*" ] 100 } 101 } 102 ] 103 } 104 EOF 105 gcloud compute scp --zone us-central1-a --project="$PROJECT" /tmp/config.ci "$CI_HOSTNAME":~/ 106 107 # ssh into the syz-ci machine. Will be required in the next step. 108 gcloud compute ssh "$CI_HOSTNAME" --zone us-central1-a --project="$PROJECT" 109 ``` 110 111 [syz-ci] Let's install and configure the syz-ci service on our syz-ci VM: 112 113 ```sh 114 sudo apt install -y wget git docker.io build-essential 115 116 # We need a recent go version, not yet available in debian 11 117 wget 'https://go.dev/dl/go1.18.linux-amd64.tar.gz' 118 sudo tar -zxvf go1.18.linux-amd64.tar.gz -C /usr/local/ 119 echo "export PATH=/usr/local/go/bin:${PATH}" | sudo tee /etc/profile.d/go.sh 120 source /etc/profile.d/go.sh 121 122 sudo mkdir /syzkaller 123 sudo mv ~/syz-ci /syzkaller/ 124 sudo mv ~/disk.img /syzkaller/ 125 sudo mv ~/config.ci /syzkaller/ 126 sudo ln -s /syzkaller/gopath/src/github.com/google/syzkaller/dashboard/config /syzkaller/config 127 128 # Pull docker container used by syz-ci for building the linux kernel 129 # We also do this on systemd start, but the first pull might take a long time, 130 # resulting in startup timeouts if we don't pull here once first. 131 sudo /usr/bin/docker pull gcr.io/syzkaller/syzbot 132 133 cat <<EOF > /tmp/syz-ci.service 134 [Unit] 135 Description=syz-ci 136 Requires=docker.service 137 After=docker.service 138 139 [Service] 140 Type=simple 141 User=root 142 ExecStartPre=-/usr/bin/docker rm --force syz-ci 143 ExecStartPre=/usr/bin/docker pull gcr.io/syzkaller/syzbot 144 ExecStartPre=/usr/bin/docker image prune --filter="dangling=true" -f 145 # --privileged is required for pkg/osutil sandboxing, 146 # otherwise unshare syscall fails with EPERM. 147 # Consider giving it finer-grained permissions, 148 # or maybe running an unpriv container is better than 149 # our sandboxing (?) then we could instead add 150 # --env SYZ_DISABLE_SANDBOXING=yes. 151 # However, we will also need to build GCE images, 152 # which requires access to loop devices, mount, etc. 153 # Proxying /dev is required for image build, 154 # otherwise partition devices (/dev/loop0p1) 155 # don't appear inside of the container. 156 # Host network is required because syz-manager inside 157 # of the container will create GCE VMs which will 158 # connect back to the syz-manager using this VM's IP 159 # and syz-manager port generated inside of the container. 160 # Without host networking the port is not open on the machine. 161 ExecStart=/usr/bin/docker run --rm --name syz-ci \ 162 --privileged \ 163 --network host \ 164 --volume /var/run/docker.sock:/var/run/docker.sock \ 165 --volume /syzkaller:/syzkaller \ 166 --volume /dev:/dev \ 167 --workdir /syzkaller \ 168 --env HOME=/syzkaller \ 169 gcr.io/syzkaller/syzbot \ 170 /syzkaller/syz-ci -config config.ci 171 ExecStop=/usr/bin/docker stop -t 600 syz-ci 172 Restart=always 173 RestartSec=10 174 KillMode=mixed 175 176 [Install] 177 WantedBy=multi-user.target 178 EOF 179 sudo mv /tmp/syz-ci.service /etc/systemd/system/ 180 sudo systemctl daemon-reload 181 sudo systemctl restart syz-ci 182 sudo systemctl enable syz-ci 183 sudo journalctl -fu syz-ci 184 ``` 185 186 Check the syc-ci journal logs at this point to see if the service comes up fine. Now syz-ci needs to do a bunch of time consuming stuff like building the kernel under test, so be patient. 187 188 If you want to hack on syz-ci you can stop here. Otherwise the next section builds on the syz-ci instructions and extends the setup with a dashboard deployment. 189 190 ## Deploying Syzbot dashboard 191 192 [locally] deploy the dashboard to Google App Engine: 193 194 ```sh 195 export PROJECT='your-gcp-project' 196 export CI_HOSTNAME='ci-linux' 197 # A random string used by the syz-ci to authenticate against the dashboard 198 export CI_KEY='fill-with-random-ci-key-string' 199 # A random string used by the syz-manager to authenticate against the dashboard 200 export MANAGER_KEY='fill-with-random-manager-key-string' 201 # A random string used for hashing, can be anything, but once fixed it can't 202 # be changed as it becomes a part of persistent bug identifiers. 203 export KEY='fill-with-random-key-string' 204 # This email will receive all of the crashes found by your instance. 205 export EMAIL='syzkaller@example.com' 206 207 gcloud app create --region us-central --project $PROJECT --quiet 208 209 # Grant the app engine service account access to Datastore 210 SERVICE_ACCOUNT=`gcloud iam service-accounts list --filter 'displayName:App Engine default service account' --format='value(email)' --project $PROJECT` 211 gcloud projects add-iam-policy-binding "$PROJECT" \ 212 --member="serviceAccount:$SERVICE_ACCOUNT" \ 213 --role="roles/editor" 214 gcloud projects add-iam-policy-binding "$PROJECT" \ 215 --member="serviceAccount:$SERVICE_ACCOUNT" \ 216 --role="roles/datastore.owner" 217 218 GOOGLE_GO=$HOME/gopath/src/github.com/google/ 219 cd $GOOGLE_GO/syzkaller 220 221 # Enable some crons for sending emails and such 222 gcloud services enable cloudscheduler.googleapis.com --project $PROJECT 223 gcloud app deploy ./dashboard/app/cron.yaml --project $PROJECT --quiet 224 225 # Create required Datastore indexes. Requires a few minutes to 226 # generate before they (and hence syzbot) become usable 227 gcloud datastore indexes create ./dashboard/app/index.yaml --project $PROJECT --quiet 228 229 cat <<EOF > ./dashboard/app/config_not_prod.go 230 package main 231 import ( 232 "time" 233 "github.com/google/syzkaller/dashboard/dashapi" 234 ) 235 const ( 236 reportingUpstream = "upstream" 237 moderationDailyLimit = 30 238 internalDailyLimit = 30 239 reportingDelay = 0 240 domainLinux = "linux" 241 ) 242 func init() { 243 checkConfig(prodConfig) 244 mainConfig = prodConfig 245 } 246 var prodConfig = &GlobalConfig{ 247 AccessLevel: AccessPublic, 248 AuthDomain: "@google.com", 249 CoverPath: "https://storage.googleapis.com/syzkaller/cover/", 250 Clients: map[string]string{ 251 "$CI_HOSTNAME": "$CI_KEY", 252 }, 253 Obsoleting: ObsoletingConfig{ 254 MinPeriod: 90 * 24 * time.Hour, 255 MaxPeriod: 120 * 24 * time.Hour, 256 NonFinalMinPeriod: 60 * 24 * time.Hour, 257 NonFinalMaxPeriod: 90 * 24 * time.Hour, 258 }, 259 DefaultNamespace: "upstream", 260 Namespaces: map[string]*Config{ 261 "upstream": { 262 AccessLevel: AccessPublic, 263 DisplayTitle: "Linux", 264 SimilarityDomain: domainLinux, 265 Key: "$KEY", 266 Clients: map[string]string{ 267 "ci-upstream-kasan-gce": "$MANAGER_KEY", 268 }, 269 Repos: []KernelRepo{ 270 { 271 URL: "git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git", 272 Branch: "master", 273 Alias: "upstream", 274 ReportingPriority: 9, 275 }, 276 }, 277 MailWithoutReport: true, 278 ReportingDelay: reportingDelay, 279 WaitForRepro: 0, 280 Managers: map[string]ConfigManager{}, 281 Reporting: []Reporting{ 282 { 283 AccessLevel: AccessPublic, 284 Name: reportingUpstream, 285 DailyLimit: 30, 286 Config: &EmailConfig{ 287 Email: "$EMAIL", 288 SubjectPrefix: "[syzbot-test]", 289 MailMaintainers: false, 290 }, 291 }, 292 }, 293 TransformCrash: func(build *Build, crash *dashapi.Crash) bool { 294 return true 295 }, 296 NeedRepro: func(bug *Bug) bool { 297 return true 298 }, 299 }, 300 }, 301 } 302 EOF 303 304 # Deploy the actual dashboard GAE application 305 GOPATH=~/gopath GO111MODULE=off gcloud beta app deploy ./dashboard/app/app.yaml --project "$PROJECT" --quiet 306 ``` 307 308 ### Integrating Syz-ci with syzbot 309 310 [locally] Prepare config and login to syz-ci VM: 311 312 ```sh 313 export PROJECT='your-gcp-project' 314 export CI_HOSTNAME='ci-linux' 315 export CI_KEY='fill-with-random-ci-key-string' 316 export MANAGER_KEY='fill-with-random-manager-key-string' 317 export DASHBOARD_FQDN=`gcloud app describe --project $PROJECT --format 'value(defaultHostname)'` 318 319 cat <<EOF > /tmp/config.ci 320 { 321 "name": "$CI_HOSTNAME", 322 "http": ":80", 323 "manager_port_start": 50010, 324 "dashboard_addr": "https://$DASHBOARD_FQDN", 325 "dashboard_client": "$CI_HOSTNAME", 326 "dashboard_key": "$CI_KEY", 327 "syzkaller_repo": "https://github.com/google/syzkaller.git", 328 "managers": [ 329 { 330 "repo": "git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git", 331 "repo_alias": "upstream", 332 "dashboard_client": "ci-upstream-kasan-gce", 333 "dashboard_key": "$MANAGER_KEY", 334 "userspace": "disk.img", 335 "kernel_config": "config/linux/upstream-apparmor-kasan.config", 336 "manager_config": { 337 "name": "ci-upstream", 338 "target": "linux/amd64", 339 "procs": 6, 340 "type": "gce", 341 "vm": { 342 "count": 5, 343 "machine_type": "e2-standard-2", 344 "gcs_path": "$PROJECT-bucket/disks" 345 }, 346 "disable_syscalls": [ "perf_event_open*" ] 347 } 348 } 349 ] 350 } 351 EOF 352 gcloud compute scp --zone us-central1-a --project="$PROJECT" /tmp/config.ci "$CI_HOSTNAME":~/ 353 354 gcloud compute ssh "$CI_HOSTNAME" --zone us-central1-a --project="$PROJECT" 355 ``` 356 357 [syz-ci] Reconfigure syz-ci to start sending results to the dashboard: 358 359 ```sh 360 sudo mv ~/config.ci /syzkaller/ 361 sudo systemctl restart syz-ci 362 sudo journalctl -fu syz-ci 363 ``` 364 365 [locally] Open the dashboard in your browser: 366 ``` 367 gcloud app browse --project=$PROJECT 368 ``` 369 Once syzkaller finds the first crashes they should show up here. This might take a while.