github.com/containers/podman/v2@v2.2.2-0.20210501105131-c1e07d070c4c/troubleshooting.md (about) 1 ![PODMAN logo](logo/podman-logo-source.svg) 2 3 # Troubleshooting 4 5 ## A list of common issues and solutions for Podman 6 7 --- 8 ### 1) Variety of issues - Validate Version 9 10 A large number of issues reported against Podman are often found to already be fixed 11 in more current versions of the project. Before reporting an issue, please verify the 12 version you are running with `podman version` and compare it to the latest release 13 documented on the top of Podman's [README.md](README.md). 14 15 If they differ, please update your version of PODMAN to the latest possible 16 and retry your command before reporting the issue. 17 18 --- 19 ### 2) Can't use volume mount, get permission denied 20 21 $ podman run -v ~/mycontent:/content fedora touch /content/file 22 touch: cannot touch '/content/file': Permission denied 23 24 #### Solution 25 26 This is usually caused by SELinux. 27 28 Labeling systems like SELinux require that proper labels are placed on volume 29 content mounted into a container. Without a label, the security system might 30 prevent the processes running inside the container from using the content. By 31 default, Podman does not change the labels set by the OS. 32 33 To change a label in the container context, you can add either of two suffixes 34 **:z** or **:Z** to the volume mount. These suffixes tell Podman to relabel file 35 objects on the shared volumes. The **z** option tells Podman that two containers 36 share the volume content. As a result, Podman labels the content with a shared 37 content label. Shared volume labels allow all containers to read/write content. 38 The **Z** option tells Podman to label the content with a private unshared label. 39 Only the current container can use a private volume. 40 41 $ podman run -v ~/mycontent:/content:Z fedora touch /content/file 42 43 Make sure the content is private for the container. Do not relabel system directories and content. 44 Relabeling system content might cause other confined services on your machine to fail. For these 45 types of containers we recommend that disable SELinux separation. The option `--security-opt label=disable` 46 will disable SELinux separation for the container. 47 48 $ podman run --security-opt label=disable -v ~:/home/user fedora touch /home/user/file 49 50 --- 51 ### 3) No such image or Bare keys cannot contain ':' 52 53 When doing a `podman pull` or `podman build` command and a "common" image cannot be pulled, 54 it is likely that the `/etc/containers/registries.conf` file is either not installed or possibly 55 misconfigured. 56 57 #### Symptom 58 59 ```console 60 $ sudo podman build -f Dockerfile 61 STEP 1: FROM alpine 62 error building: error creating build container: no such image "alpine" in registry: image not known 63 ``` 64 65 or 66 67 ```console 68 $ sudo podman pull fedora 69 error pulling image "fedora": unable to pull fedora: error getting default registries to try: Near line 9 (last key parsed ''): Bare keys cannot contain ':'. 70 ``` 71 72 #### Solution 73 74 * Verify that the `/etc/containers/registries.conf` file exists. If not, verify that the containers-common package is installed. 75 * Verify that the entries in the `[registries.search]` section of the /etc/containers/registries.conf file are valid and reachable. 76 * i.e. `registries = ['registry.fedoraproject.org', 'quay.io', 'registry.access.redhat.com']` 77 78 --- 79 ### 4) http: server gave HTTP response to HTTPS client 80 81 When doing a Podman command such as `build`, `commit`, `pull`, or `push` to a registry, 82 tls verification is turned on by default. If authentication is not used with 83 those commands, this error can occur. 84 85 #### Symptom 86 87 ```console 88 $ sudo podman push alpine docker://localhost:5000/myalpine:latest 89 Getting image source signatures 90 Get https://localhost:5000/v2/: http: server gave HTTP response to HTTPS client 91 ``` 92 93 #### Solution 94 95 By default tls verification is turned on when communicating to registries from 96 Podman. If the registry does not require authentication the Podman commands 97 such as `build`, `commit`, `pull` and `push` will fail unless tls verification is turned 98 off using the `--tls-verify` option. **NOTE:** It is not at all recommended to 99 communicate with a registry and not use tls verification. 100 101 * Turn off tls verification by passing false to the tls-verification option. 102 * I.e. `podman push --tls-verify=false alpine docker://localhost:5000/myalpine:latest` 103 104 --- 105 ### 5) rootless containers cannot ping hosts 106 107 When using the ping command from a non-root container, the command may 108 fail because of a lack of privileges. 109 110 #### Symptom 111 112 ```console 113 $ podman run --rm fedora ping -W10 -c1 redhat.com 114 PING redhat.com (209.132.183.105): 56 data bytes 115 116 --- redhat.com ping statistics --- 117 1 packets transmitted, 0 packets received, 100% packet loss 118 ``` 119 120 #### Solution 121 122 It is most likely necessary to enable unprivileged pings on the host. 123 Be sure the UID of the user is part of the range in the 124 `/proc/sys/net/ipv4/ping_group_range` file. 125 126 To change its value you can use something like: `sysctl -w 127 "net.ipv4.ping_group_range=0 2000000"`. 128 129 To make the change persistent, you'll need to add a file in 130 `/etc/sysctl.d` that contains `net.ipv4.ping_group_range=0 $MAX_UID`. 131 132 --- 133 ### 6) Build hangs when the Dockerfile contains the useradd command 134 135 When the Dockerfile contains a command like `RUN useradd -u 99999000 -g users newuser` the build can hang. 136 137 #### Symptom 138 139 If you are using a useradd command within a Dockerfile with a large UID/GID, it will create a large sparse file `/var/log/lastlog`. This can cause the build to hang forever. Go language does not support sparse files correctly, which can lead to some huge files being created in your container image. 140 141 #### Solution 142 143 If the entry in the Dockerfile looked like: RUN useradd -u 99999000 -g users newuser then add the `--no-log-init` parameter to change it to: `RUN useradd --no-log-init -u 99999000 -g users newuser`. This option tells useradd to stop creating the lastlog file. 144 145 ### 7) Permission denied when running Podman commands 146 147 When rootless Podman attempts to execute a container on a non exec home directory a permission error will be raised. 148 149 #### Symptom 150 151 If you are running Podman or buildah on a home directory that is mounted noexec, 152 then they will fail. With a message like: 153 154 ``` 155 podman run centos:7 156 standard_init_linux.go:203: exec user process caused "permission denied" 157 ``` 158 159 #### Solution 160 161 Since the administrator of the system setup your home directory to be noexec, you will not be allowed to execute containers from storage in your home directory. It is possible to work around this by manually specifying a container storage path that is not on a noexec mount. Simply copy the file /etc/containers/storage.conf to ~/.config/containers/ (creating the directory if necessary). Specify a graphroot directory which is not on a noexec mount point and to which you have read/write privileges. You will need to modify other fields to writable directories as well. 162 163 For example 164 165 ``` 166 cat ~/.config/containers/storage.conf 167 [storage] 168 driver = "overlay" 169 runroot = "/run/user/1000" 170 graphroot = "/execdir/myuser/storage" 171 [storage.options] 172 mount_program = "/bin/fuse-overlayfs" 173 ``` 174 175 ### 8) Permission denied when running systemd within a Podman container 176 177 When running systemd as PID 1 inside of a container on an SELinux 178 separated machine, it needs to write to the cgroup file system. 179 180 #### Symptom 181 182 Systemd gets permission denied when attempting to write to the cgroup file 183 system, and AVC messages start to show up in the audit.log file or journal on 184 the system. 185 186 #### Solution 187 188 Newer versions of Podman (2.0 or greater) support running init based containers 189 with a different SELinux labels, which allow the container process access to the 190 cgroup file system. This feature requires container-selinux-2.132 or newer 191 versions. 192 193 Prior to Podman 2.0, the SELinux boolean `container_manage_cgroup` allows 194 container processes to write to the cgroup file system. Turn on this boolean, 195 on SELinux separated systems, to allow systemd to run properly in the container. 196 Only do this on systems running older versions of Podman. 197 198 `setsebool -P container_manage_cgroup true` 199 200 ### 9) Newuidmap missing when running rootless Podman commands 201 202 Rootless Podman requires the newuidmap and newgidmap programs to be installed. 203 204 #### Symptom 205 206 If you are running Podman or buildah as a not root user, you get an error complaining about 207 a missing newuidmap executable. 208 209 ``` 210 podman run -ti fedora sh 211 cannot find newuidmap: exec: "newuidmap": executable file not found in $PATH 212 ``` 213 214 #### Solution 215 216 Install a version of shadow-utils that includes these executables. Note that for RHEL and CentOS 7, at least the 7.7 release must be installed for support to be available. 217 218 ### 10) rootless setup user: invalid argument 219 220 Rootless Podman requires the user running it to have a range of UIDs listed in /etc/subuid and /etc/subgid. 221 222 #### Symptom 223 224 An user, either via --user or through the default configured for the image, is not mapped inside the namespace. 225 226 ``` 227 podman run --rm -ti --user 1000000 alpine echo hi 228 Error: container create failed: container_linux.go:344: starting container process caused "setup user: invalid argument" 229 ``` 230 231 #### Solution 232 233 Update the /etc/subuid and /etc/subgid with fields for users that look like: 234 235 ``` 236 cat /etc/subuid 237 johndoe:100000:65536 238 test:165536:65536 239 ``` 240 241 The format of this file is USERNAME:UID:RANGE 242 243 * username as listed in /etc/passwd or getpwent. 244 * The initial uid allocated for the user. 245 * The size of the range of UIDs allocated for the user. 246 247 This means johndoe is allocated UIDS 100000-165535 as well as his standard UID in the 248 /etc/passwd file. 249 250 You should ensure that each user has a unique range of uids, because overlapping UIDs, 251 would potentially allow one user to attack another user. 252 253 You could also use the usermod program to assign UIDs to a user. 254 255 If you update either the /etc/subuid or /etc/subgid file, you need to 256 stop all running containers and kill the pause process. This is done 257 automatically by the `system migrate` command, which can also be used 258 to stop all the containers and kill the pause process. 259 260 ``` 261 usermod --add-subuids 200000-201000 --add-subgids 200000-201000 johndoe 262 grep johndoe /etc/subuid /etc/subgid 263 /etc/subuid:johndoe:200000:1001 264 /etc/subgid:johndoe:200000:1001 265 ``` 266 267 ### 11) Changing the location of the Graphroot leads to permission denied 268 269 When I change the graphroot storage location in storage.conf, the next time I 270 run Podman I get an error like: 271 272 ``` 273 # podman run -p 5000:5000 -it centos bash 274 275 bash: error while loading shared libraries: /lib64/libc.so.6: cannot apply additional memory protection after relocation: Permission denied 276 ``` 277 278 For example, the admin sets up a spare disk to be mounted at `/src/containers`, 279 and points storage.conf at this directory. 280 281 282 #### Symptom 283 284 SELinux blocks containers from using random locations for overlay storage. 285 These directories need to be labeled with the same labels as if the content was 286 under /var/lib/containers/storage. 287 288 #### Solution 289 290 Tell SELinux about the new containers storage by setting up an equivalence record. 291 This tells SELinux to label content under the new path, as if it was stored 292 under `/var/lib/containers/storage`. 293 294 ``` 295 semanage fcontext -a -e /var/lib/containers /srv/containers 296 restorecon -R -v /srv/containers 297 ``` 298 299 The semanage command above tells SELinux to setup the default labeling of 300 `/srv/containers` to match `/var/lib/containers`. The `restorecon` command 301 tells SELinux to apply the labels to the actual content. 302 303 Now all new content created in these directories will automatically be created 304 with the correct label. 305 306 ### 12) Anonymous image pull fails with 'invalid username/password' 307 308 Pulling an anonymous image that doesn't require authentication can result in an 309 `invalid username/password` error. 310 311 #### Symptom 312 313 If you pull an anonymous image, one that should not require credentials, you can receive 314 and `invalid username/password` error if you have credentials established in the 315 authentication file for the target container registry that are no longer valid. 316 317 ``` 318 podman run -it --rm docker://docker.io/library/alpine:latest ls 319 Trying to pull docker://docker.io/library/alpine:latest...ERRO[0000] Error pulling image ref //alpine:latest: Error determining manifest MIME type for docker://alpine:latest: unable to retrieve auth token: invalid username/password 320 Failed 321 Error: unable to pull docker://docker.io/library/alpine:latest: unable to pull image: Error determining manifest MIME type for docker://alpine:latest: unable to retrieve auth token: invalid username/password 322 ``` 323 324 This can happen if the authentication file is modified 'by hand' or if the credentials 325 are established locally and then the password is updated later in the container registry. 326 327 #### Solution 328 329 Depending upon which container tool was used to establish the credentials, use `podman logout` 330 or `docker logout` to remove the credentials from the authentication file. 331 332 ### 13) Running Podman inside a container causes container crashes and inconsistent states 333 334 Running Podman in a container and forwarding some, but not all, of the required host directories can cause inconsistent container behavior. 335 336 #### Symptom 337 338 After creating a container with Podman's storage directories mounted in from the host and running Podman inside a container, all containers show their state as "configured" or "created", even if they were running or stopped. 339 340 #### Solution 341 342 When running Podman inside a container, it is recommended to mount at a minimum `/var/lib/containers/storage/` as a volume. 343 Typically, you will not mount in the host version of the directory, but if you wish to share containers with the host, you can do so. 344 If you do mount in the host's `/var/lib/containers/storage`, however, you must also mount in the host's `/var/run/libpod` and `/var/run/containers/storage` directories. 345 Not doing this will cause Podman in the container to detect that temporary files have been cleared, leading it to assume a system restart has taken place. 346 This can cause Podman to reset container states and lose track of running containers. 347 348 For running containers on the host from inside a container, we also recommend the [Podman remote client](remote_client.md), which only requires a single socket to be mounted into the container. 349 350 ### 14) Rootless 'podman build' fails EPERM on NFS: 351 352 NFS enforces file creation on different UIDs on the server side and does not understand user namespace, which rootless Podman requires. 353 When a container root process like YUM attempts to create a file owned by a different UID, NFS Server denies the creation. 354 NFS is also a problem for the file locks when the storage is on it. Other distributed file systems (for example: Lustre, Spectrum Scale, the General Parallel File System (GPFS)) are also not supported when running in rootless mode as these file systems do not understand user namespace. 355 356 #### Symptom 357 ```console 358 $ podman build . 359 ERRO[0014] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: open /root/.bash_logout: permission denied 360 error creating build container: Error committing the finished image: error adding layer with blob "sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17": ApplyLayer exit status 1 stdout: stderr: open /root/.bash_logout: permission denied 361 ``` 362 363 #### Solution 364 Choose one of the following: 365 * Setup containers/storage in a different directory, not on an NFS share. 366 * Create a directory on a local file system. 367 * Edit `~/.config/containers/containers.conf` and point the `volume_path` option to that local directory. (Copy /usr/share/containers/containers.conf if ~/.config/containers/containers.conf does not exist) 368 * Otherwise just run Podman as root, via `sudo podman` 369 370 ### 15) Rootless 'podman build' fails when using OverlayFS: 371 372 The Overlay file system (OverlayFS) requires the ability to call the `mknod` command when creating whiteout files 373 when extracting an image. However, a rootless user does not have the privileges to use `mknod` in this capacity. 374 375 #### Symptom 376 ```console 377 podman build --storage-driver overlay . 378 STEP 1: FROM docker.io/ubuntu:xenial 379 Getting image source signatures 380 Copying blob edf72af6d627 done 381 Copying blob 3e4f86211d23 done 382 Copying blob 8d3eac894db4 done 383 Copying blob f7277927d38a done 384 Copying config 5e13f8dd4c done 385 Writing manifest to image destination 386 Storing signatures 387 Error: error creating build container: Error committing the finished image: error adding layer with blob "sha256:8d3eac894db4dc4154377ad28643dfe6625ff0e54bcfa63e0d04921f1a8ef7f8": Error processing tar file(exit status 1): operation not permitted 388 $ podman build . 389 ERRO[0014] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: open /root/.bash_logout: permission denied 390 error creating build container: Error committing the finished image: error adding layer with blob "sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17": ApplyLayer exit status 1 stdout: stderr: open /root/.bash_logout: permission denied 391 ``` 392 393 #### Solution 394 Choose one of the following: 395 * Complete the build operation as a privileged user. 396 * Install and configure fuse-overlayfs. 397 * Install the fuse-overlayfs package for your Linux Distribution. 398 * Add `mount_program = "/usr/bin/fuse-overlayfs"` under `[storage.options]` in your `~/.config/containers/storage.conf` file. 399 400 ### 16) RHEL 7 and CentOS 7 based `init` images don't work with cgroup v2 401 402 The systemd version shipped in RHEL 7 and CentOS 7 doesn't have support for cgroup v2. Support for cgroup V2 requires version 230 of systemd or newer, which 403 was never shipped or supported on RHEL 7 or CentOS 7. 404 405 #### Symptom 406 ```console 407 408 sh# podman run --name test -d registry.access.redhat.com/rhel7-init:latest && sleep 10 && podman exec test systemctl status 409 c8567461948439bce72fad3076a91ececfb7b14d469bfa5fbc32c6403185beff 410 Failed to get D-Bus connection: Operation not permitted 411 Error: non zero exit code: 1: OCI runtime error 412 ``` 413 414 #### Solution 415 You'll need to either: 416 417 * configure the host to use cgroup v1 418 419 ``` 420 On Fedora you can do: 421 # dnf install -y grubby 422 # grubby --update-kernel=ALL --args=âsystemd.unified_cgroup_hierarchy=0" 423 # reboot 424 ``` 425 426 * update the image to use an updated version of systemd. 427 428 ### 17) rootless containers exit once the user session exits 429 430 You need to set lingering mode through loginctl to prevent user processes to be killed once 431 the user session completed. 432 433 #### Symptom 434 435 Once the user logs out all the containers exit. 436 437 #### Solution 438 You'll need to either: 439 440 * loginctl enable-linger $UID 441 442 or as root if your user has not enough privileges. 443 444 * sudo loginctl enable-linger $UID 445 446 ### 18) `podman run` fails with "bpf create: permission denied error" 447 448 The Kernel Lockdown patches deny eBPF programs when Secure Boot is enabled in the BIOS. [Matthew Garrett's post](https://mjg59.dreamwidth.org/50577.html) describes the relationship between Lockdown and Secure Boot and [Jan-Philip Gehrcke's](https://gehrcke.de/2019/09/running-an-ebpf-program-may-require-lifting-the-kernel-lockdown/) connects this with eBPF. [RH bug 1768125](https://bugzilla.redhat.com/show_bug.cgi?id=1768125) contains some additional details. 449 450 #### Symptom 451 452 Attempts to run podman result in 453 454 ```Error: bpf create : Operation not permitted: OCI runtime permission denied error``` 455 456 #### Solution 457 458 One workaround is to disable Secure Boot in your BIOS. 459 460 ### 19) error creating libpod runtime: there might not be enough IDs available in the namespace 461 462 Unable to pull images 463 464 #### Symptom 465 466 ```console 467 $ podman unshare cat /proc/self/uid_map 468 0 1000 1 469 ``` 470 471 #### Solution 472 473 ```console 474 $ podman system migrate 475 ``` 476 477 Original command now returns 478 479 ``` 480 $ podman unshare cat /proc/self/uid_map 481 0 1000 1 482 1 100000 65536 483 ``` 484 485 Reference [subuid](http://man7.org/linux/man-pages/man5/subuid.5.html) and [subgid](http://man7.org/linux/man-pages/man5/subgid.5.html) man pages for more detail. 486 487 ### 20) Passed-in device can't be accessed in rootless container 488 489 As a non-root user you have group access rights to a device that you want to 490 pass into a rootless container with `--device=...`. 491 492 #### Symptom 493 494 Any access inside the container is rejected with "Permission denied". 495 496 #### Solution 497 498 The runtime uses `setgroups(2)` hence the process looses all additional groups 499 the non-root user has. If you use the `crun` runtime, 0.10.4 or newer, 500 then you can enable a workaround by adding `--annotation io.crun.keep_original_groups=1` 501 to the `podman` command line. 502 503 ### 21) A rootless container running in detached mode is closed at logout 504 505 When running a container with a command like `podman run --detach httpd` as 506 a rootless user, the container is closed upon logout and is not kept running. 507 508 #### Symptom 509 510 When logging out of a rootless user session, all containers that were started 511 in detached mode are stopped and are not kept running. As the root user, these 512 same containers would survive the logout and continue running. 513 514 #### Solution 515 516 When systemd notes that a session that started a Podman container has exited, 517 it will also stop any containers that has been associated with it. To avoid 518 this, use the following command before logging out: `loginctl enable-linger`. 519 To later revert the linger functionality, use `loginctl disable-linger`. 520 521 LOGINCTL(1), SYSTEMD(1) 522 523 ### 22) Containers default detach keys conflict with shell history navigation 524 525 Podman defaults to `ctrl-p,ctrl-q` to detach from a running containers. The 526 bash and zsh shells default to ctrl-p for the displaying of the previous 527 command. This causes issues when running a shell inside of a container. 528 529 #### Symptom 530 531 With the default detach key combo ctrl-p,ctrl-q, shell history navigation 532 (tested in bash and zsh) using ctrl-p to access the previous command will not 533 display this previous command. Or anything else. Conmon is waiting for an 534 additional character to see if the user wants to detach from the container. 535 Adding additional characters to the command will cause it to be displayed along 536 with the additional character. If the user types ctrl-p a second time the shell 537 display the 2nd to last command. 538 539 #### Solution 540 541 The solution to this is to change the default detach_keys. For example in order 542 to change the defaults to `ctrl-q,ctrl-q` use the `--detach-keys` option. 543 544 ``` 545 podman run -ti --detach-keys ctrl-q,ctrl-q fedora sh 546 ``` 547 548 To make this change the default for all containers, users can modify the 549 containers.conf file. This can be done simply in your home directory, but adding the 550 following lines to users containers.conf 551 552 ``` 553 $ cat >> ~/.config/containers/containers.conf < _eof 554 [engine] 555 detach_keys="ctrl-q,ctrl-q" 556 _eof 557 ``` 558 559 In order to effect root running containers and all users, modify the system 560 wide defaults in /etc/containers/containers.conf 561 562 563 ### 23) Container with exposed ports won't run in a pod 564 565 A container with ports that have been published with the `--publish` or `-p` option 566 can not be run within a pod. 567 568 #### Symptom 569 570 ``` 571 $ podman pod create --name srcview -p 127.0.0.1:3434:3434 -p 127.0.0.1:7080:7080 -p 127.0.0.1:3370:3370 4b2f4611fa2cbd60b3899b936368c2b3f4f0f68bc8e6593416e0ab8ecb0a3f1d 572 573 $ podman run --pod srcview --name src-expose -p 3434:3434 -v "${PWD}:/var/opt/localrepo":Z,ro sourcegraph/src-expose:latest serve /var/opt/localrepo 574 Error: cannot set port bindings on an existing container network namespace 575 ``` 576 577 #### Solution 578 579 This is a known limitation. If a container will be run within a pod, it is not necessary 580 to publish the port for the containers in the pod. The port must only be published by the 581 pod itself. Pod network stacks act like the network stack on the host - you have a 582 variety of containers in the pod, and programs in the container, all sharing a single 583 interface and IP address, and associated ports. If one container binds to a port, no other 584 container can use that port within the pod while it is in use. Containers in the pod can 585 also communicate over localhost by having one container bind to localhost in the pod, and 586 another connect to that port. 587 588 In the example from the symptom section, dropping the `-p 3434:3434` would allow the 589 `podman run` command to complete, and the container as part of the pod would still have 590 access to that port. For example: 591 592 ``` 593 $ podman run --pod srcview --name src-expose -v "${PWD}:/var/opt/localrepo":Z,ro sourcegraph/src-expose:latest serve /var/opt/localrepo 594 ``` 595 596 ### 24) Podman container images fail with `fuse: device not found` when run 597 598 Some container images require that the fuse kernel module is loaded in the kernel 599 before they will run with the fuse filesystem in play. 600 601 #### Symptom 602 603 When trying to run the container images found at quay.io/podman, quay.io/containers 604 registry.access.redhat.com/ubi8 or other locations, an error will sometimes be returned: 605 606 ``` 607 ERRO error unmounting /var/lib/containers/storage/overlay/30c058cdadc888177361dd14a7ed7edab441c58525b341df321f07bc11440e68/merged: invalid argument 608 error mounting container "1ae176ca72b3da7c70af31db7434bcf6f94b07dbc0328bc7e4e8fc9579d0dc2e": error mounting build container "1ae176ca72b3da7c70af31db7434bcf6f94b07dbc0328bc7e4e8fc9579d0dc2e": error creating overlay mount to /var/lib/containers/storage/overlay/30c058cdadc888177361dd14a7ed7edab441c58525b341df321f07bc11440e68/merged: using mount program /usr/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first 609 fuse-overlayfs: cannot mount: No such device 610 : exit status 1 611 ERRO exit status 1 612 ``` 613 614 #### Solution 615 616 If you encounter a `fuse: device not found` error when running the container image, it is likely that 617 the fuse kernel module has not been loaded on your host system. Use the command `modprobe fuse` to load the 618 module and then run the container image afterwards. To enable this automatically at boot time, you can add a configuration 619 file to `/etc/modules.load.d`. See `man modules-load.d` for more details. 620 621 ### 25) podman run --rootfs link/to//read/only/dir does not work 622 623 An error such as "OCI runtime error" on a read-only filesystem or the error "{image} is not an absolute path or is a symlink" are often times indicators for this issue. For more details, review this [issue]( 624 https://github.com/containers/podman/issues/5895). 625 626 #### Symptom 627 628 Rootless Podman requires certain files to exist in a file system in order to run. 629 Podman will create /etc/resolv.conf, /etc/hosts and other file descriptors on the rootfs in order 630 to mount volumes on them. 631 632 #### Solution 633 634 Run the container once in read/write mode, Podman will generate all of the FDs on the rootfs, and 635 from that point forward you can run with a read-only rootfs. 636 637 $ podman run --rm --rootfs /path/to/rootfs true 638 639 The command above will create all the missing directories needed to run the container. 640 641 After that, it can be used in read only mode, by multiple containers at the same time: 642 643 $ podman run --read-only --rootfs /path/to/rootfs .... 644 645 Another option would be to create an overlay file system on the directory as a lower and then 646 then allow podman to create the files on the upper. 647 648 ### 26) Running containers with CPU limits fails with a permissions error 649 650 On some systemd-based systems, non-root users do not have CPU limit delegation 651 permissions. This causes setting CPU limits to fail. 652 653 #### Symptom 654 655 Running a container with a CPU limit options such as `--cpus`, `--cpu-period`, 656 or `--cpu-quota` will fail with an error similar to the following: 657 658 Error: opening file `cpu.max` for writing: Permission denied: OCI runtime permission denied error 659 660 This means that CPU limit delegation is not enabled for the current user. 661 662 #### Solution 663 664 You can verify whether CPU limit delegation is enabled by running the following command: 665 666 cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers" 667 668 Example output might be: 669 670 memory pids 671 672 In the above example, `cpu` is not listed, which means the current user does 673 not have permission to set CPU limits. 674 675 If you want to enable CPU limit delegation for all users, you can create the 676 file `/etc/systemd/system/user@.service.d/delegate.conf` with the contents: 677 678 [Service] 679 Delegate=memory pids cpu io 680 681 After logging out and loggin back in, you should have permission to set CPU 682 limits. 683 684 ### 26) `exec container process '/bin/sh': Exec format error` (or another binary than `bin/sh`) 685 686 This can happen when running a container from an image for another architecture than the one you are running on. 687 688 For example, if a remote repository only has, and thus send you, a `linux/arm64` _OS/ARCH_ but you run on `linux/amd64` (as happened in https://github.com/openMF/community-app/issues/3323 due to https://github.com/timbru31/docker-ruby-node/issues/564).