github.com/containers/libpod@v1.9.4-0.20220419124438-4284fd425507/troubleshooting.md (about) 1 ![PODMAN logo](logo/podman-logo-source.svg) 2 3 # Troubleshooting 4 5 ## A list of common issues and solutions for Podman 6 7 --- 8 ### 1) Variety of issues - Validate Version 9 10 A large number of issues reported against Podman are often found to already be fixed 11 in more current versions of the project. Before reporting an issue, please verify the 12 version you are running with `podman version` and compare it to the latest release 13 documented on the top of Podman's [README.md](README.md). 14 15 If they differ, please update your version of PODMAN to the latest possible 16 and retry your command before reporting the issue. 17 18 --- 19 ### 2) No such image or Bare keys cannot contain ':' 20 21 When doing a `podman pull` or `podman build` command and a "common" image cannot be pulled, 22 it is likely that the `/etc/containers/registries.conf` file is either not installed or possibly 23 misconfigured. 24 25 #### Symptom 26 27 ```console 28 $ sudo podman build -f Dockerfile 29 STEP 1: FROM alpine 30 error building: error creating build container: no such image "alpine" in registry: image not known 31 ``` 32 33 or 34 35 ```console 36 $ sudo podman pull fedora 37 error pulling image "fedora": unable to pull fedora: error getting default registries to try: Near line 9 (last key parsed ''): Bare keys cannot contain ':'. 38 ``` 39 40 #### Solution 41 42 * Verify that the `/etc/containers/registries.conf` file exists. If not, verify that the containers-common package is installed. 43 * Verify that the entries in the `[registries.search]` section of the /etc/containers/registries.conf file are valid and reachable. 44 * i.e. `registries = ['registry.fedoraproject.org', 'quay.io', 'registry.access.redhat.com']` 45 46 --- 47 ### 3) http: server gave HTTP response to HTTPS client 48 49 When doing a Podman command such as `build`, `commit`, `pull`, or `push` to a registry, 50 tls verification is turned on by default. If authentication is not used with 51 those commands, this error can occur. 52 53 #### Symptom 54 55 ```console 56 $ sudo podman push alpine docker://localhost:5000/myalpine:latest 57 Getting image source signatures 58 Get https://localhost:5000/v2/: http: server gave HTTP response to HTTPS client 59 ``` 60 61 #### Solution 62 63 By default tls verification is turned on when communicating to registries from 64 Podman. If the registry does not require authentication the Podman commands 65 such as `build`, `commit`, `pull` and `push` will fail unless tls verification is turned 66 off using the `--tls-verify` option. **NOTE:** It is not at all recommended to 67 communicate with a registry and not use tls verification. 68 69 * Turn off tls verification by passing false to the tls-verification option. 70 * I.e. `podman push --tls-verify=false alpine docker://localhost:5000/myalpine:latest` 71 72 --- 73 ### 4) Rootless: could not get runtime - database configuration mismatch 74 75 In Podman release 0.11.1, a default path for rootless containers was changed, 76 potentially causing rootless Podman to be unable to function. The new default 77 path is not a problem for new installations, but existing installations will 78 need to work around it with the following fix. 79 80 #### Symptom 81 82 ```console 83 $ podman info 84 could not get runtime: database run root /run/user/1000/run does not match our run root /run/user/1000: database configuration mismatch 85 ``` 86 87 #### Solution 88 89 This problem has been fixed in Podman release 0.12.1 and it is recommended 90 to upgrade to that version. If that is not possible use the following procedure. 91 92 To work around the new default path, we can manually set the path Podman is 93 expecting in a configuration file. 94 95 First, we need to make a new local configuration file for rootless Podman. 96 * `mkdir -p ~/.config/containers` 97 * `cp /usr/share/containers/libpod.conf ~/.config/containers` 98 99 Next, edit the new local configuration file 100 (`~/.config/containers/libpod.conf`) with your favorite editor. Comment out the 101 line starting with `cgroup_manager` by adding a `#` character at the beginning 102 of the line, and change the path in the line starting with `tmp_dir` to point to 103 the first path in the error message Podman gave (in this case, 104 `/run/user/1000/tmp`). 105 106 --- 107 ### 5) rootless containers cannot ping hosts 108 109 When using the ping command from a non-root container, the command may 110 fail because of a lack of privileges. 111 112 #### Symptom 113 114 ```console 115 $ podman run --rm fedora ping -W10 -c1 redhat.com 116 PING redhat.com (209.132.183.105): 56 data bytes 117 118 --- redhat.com ping statistics --- 119 1 packets transmitted, 0 packets received, 100% packet loss 120 ``` 121 122 #### Solution 123 124 It is most likely necessary to enable unprivileged pings on the host. 125 Be sure the UID of the user is part of the range in the 126 `/proc/sys/net/ipv4/ping_group_range` file. 127 128 To change its value you can use something like: `sysctl -w 129 "net.ipv4.ping_group_range=0 2000000"`. 130 131 To make the change persistent, you'll need to add a file in 132 `/etc/sysctl.d` that contains `net.ipv4.ping_group_range=0 $MAX_UID`. 133 134 --- 135 ### 6) Build hangs when the Dockerfile contains the useradd command 136 137 When the Dockerfile contains a command like `RUN useradd -u 99999000 -g users newuser` the build can hang. 138 139 #### Symptom 140 141 If you are using a useradd command within a Dockerfile with a large UID/GID, it will create a large sparse file `/var/log/lastlog`. This can cause the build to hang forever. Go language does not support sparse files correctly, which can lead to some huge files being created in your container image. 142 143 #### Solution 144 145 If the entry in the Dockerfile looked like: RUN useradd -u 99999000 -g users newuser then add the `--no-log-init` parameter to change it to: `RUN useradd --no-log-init -u 99999000 -g users newuser`. This option tells useradd to stop creating the lastlog file. 146 147 ### 7) Permission denied when running Podman commands 148 149 When rootless Podman attempts to execute a container on a non exec home directory a permission error will be raised. 150 151 #### Symptom 152 153 If you are running Podman or buildah on a home directory that is mounted noexec, 154 then they will fail. With a message like: 155 156 ``` 157 podman run centos:7 158 standard_init_linux.go:203: exec user process caused "permission denied" 159 ``` 160 161 #### Solution 162 163 Since the administrator of the system setup your home directory to be noexec, you will not be allowed to execute containers from storage in your home directory. It is possible to work around this by manually specifying a container storage path that is not on a noexec mount. Simply copy the file /etc/containers/storage.conf to ~/.config/containers/ (creating the directory if necessary). Specify a graphroot directory which is not on a noexec mount point and to which you have read/write privileges. You will need to modify other fields to writable directories as well. 164 165 For example 166 167 ``` 168 cat ~/.config/containers/storage.conf 169 [storage] 170 driver = "overlay" 171 runroot = "/run/user/1000" 172 graphroot = "/execdir/myuser/storage" 173 [storage.options] 174 mount_program = "/bin/fuse-overlayfs" 175 ``` 176 177 ### 8) Permission denied when running systemd within a Podman container 178 179 When running systemd as PID 1 inside of a container on an SELinux 180 separated machine, it needs to write to the cgroup file system. 181 182 #### Symptom 183 184 Systemd gets permission denied when attempting to write to the cgroup file 185 system, and AVC messages start to show up in the audit.log file or journal on 186 the system. 187 188 #### Solution 189 190 SELinux provides a boolean `container_manage_cgroup`, which allows container 191 processes to write to the cgroup file system. Turn on this boolean, on SELinux separated systems, to allow systemd to run properly in the container. 192 193 `setsebool -P container_manage_cgroup true` 194 195 ### 9) Newuidmap missing when running rootless Podman commands 196 197 Rootless Podman requires the newuidmap and newgidmap programs to be installed. 198 199 #### Symptom 200 201 If you are running Podman or buildah as a not root user, you get an error complaining about 202 a missing newuidmap executable. 203 204 ``` 205 podman run -ti fedora sh 206 cannot find newuidmap: exec: "newuidmap": executable file not found in $PATH 207 ``` 208 209 #### Solution 210 211 Install a version of shadow-utils that includes these executables. Note RHEL7 and Centos 7 will not have support for this until RHEL7.7 is released. 212 213 ### 10) rootless setup user: invalid argument 214 215 Rootless Podman requires the user running it to have a range of UIDs listed in /etc/subuid and /etc/subgid. 216 217 #### Symptom 218 219 An user, either via --user or through the default configured for the image, is not mapped inside the namespace. 220 221 ``` 222 podman run --rm -ti --user 1000000 alpine echo hi 223 Error: container create failed: container_linux.go:344: starting container process caused "setup user: invalid argument" 224 ``` 225 226 #### Solution 227 228 Update the /etc/subuid and /etc/subgid with fields for users that look like: 229 230 ``` 231 cat /etc/subuid 232 johndoe:100000:65536 233 test:165536:65536 234 ``` 235 236 The format of this file is USERNAME:UID:RANGE 237 238 * username as listed in /etc/passwd or getpwent. 239 * The initial uid allocated for the user. 240 * The size of the range of UIDs allocated for the user. 241 242 This means johndoe is allocated UIDS 100000-165535 as well as his standard UID in the 243 /etc/passwd file. 244 245 You should ensure that each user has a unique range of uids, because overlapping UIDs, 246 would potentially allow one user to attack another user. 247 248 You could also use the usermod program to assign UIDs to a user. 249 250 If you update either the /etc/subuid or /etc/subgid file, you need to 251 stop all running containers and kill the pause process. This is done 252 automatically by the `system migrate` command, which can also be used 253 to stop all the containers and kill the pause process. 254 255 ``` 256 usermod --add-subuids 200000-201000 --add-subgids 200000-201000 johndoe 257 grep johndoe /etc/subuid /etc/subgid 258 /etc/subuid:johndoe:200000:1001 259 /etc/subgid:johndoe:200000:1001 260 ``` 261 262 ### 11) Changing the location of the Graphroot leads to permission denied 263 264 When I change the graphroot storage location in storage.conf, the next time I 265 run Podman I get an error like: 266 267 ``` 268 # podman run -p 5000:5000 -it centos bash 269 270 bash: error while loading shared libraries: /lib64/libc.so.6: cannot apply additional memory protection after relocation: Permission denied 271 ``` 272 273 For example, the admin sets up a spare disk to be mounted at `/src/containers`, 274 and points storage.conf at this directory. 275 276 277 #### Symptom 278 279 SELinux blocks containers from using random locations for overlay storage. 280 These directories need to be labeled with the same labels as if the content was 281 under /var/lib/containers/storage. 282 283 #### Solution 284 285 Tell SELinux about the new containers storage by setting up an equivalence record. 286 This tells SELinux to label content under the new path, as if it was stored 287 under `/var/lib/containers/storage`. 288 289 ``` 290 semanage fcontext -a -e /var/lib/containers /srv/containers 291 restorecon -R -v /src/containers 292 ``` 293 294 The semanage command above tells SELinux to setup the default labeling of 295 `/srv/containers` to match `/var/lib/containers`. The `restorecon` command 296 tells SELinux to apply the labels to the actual content. 297 298 Now all new content created in these directories will automatically be created 299 with the correct label. 300 301 ### 12) Anonymous image pull fails with 'invalid username/password' 302 303 Pulling an anonymous image that doesn't require authentication can result in an 304 `invalid username/password` error. 305 306 #### Symptom 307 308 If you pull an anonymous image, one that should not require credentials, you can receive 309 and `invalid username/password` error if you have credentials established in the 310 authentication file for the target container registry that are no longer valid. 311 312 ``` 313 podman run -it --rm docker://docker.io/library/alpine:latest ls 314 Trying to pull docker://docker.io/library/alpine:latest...ERRO[0000] Error pulling image ref //alpine:latest: Error determining manifest MIME type for docker://alpine:latest: unable to retrieve auth token: invalid username/password 315 Failed 316 Error: unable to pull docker://docker.io/library/alpine:latest: unable to pull image: Error determining manifest MIME type for docker://alpine:latest: unable to retrieve auth token: invalid username/password 317 ``` 318 319 This can happen if the authentication file is modified 'by hand' or if the credentials 320 are established locally and then the password is updated later in the container registry. 321 322 #### Solution 323 324 Depending upon which container tool was used to establish the credentials, use `podman logout` 325 or `docker logout` to remove the credentials from the authentication file. 326 327 ### 13) Running Podman inside a container causes container crashes and inconsistent states 328 329 Running Podman in a container and forwarding some, but not all, of the required host directories can cause inconsistent container behavior. 330 331 #### Symptom 332 333 After creating a container with Podman's storage directories mounted in from the host and running Podman inside a container, all containers show their state as "configured" or "created", even if they were running or stopped. 334 335 #### Solution 336 337 When running Podman inside a container, it is recommended to mount at a minimum `/var/lib/containers/storage/` as a volume. 338 Typically, you will not mount in the host version of the directory, but if you wish to share containers with the host, you can do so. 339 If you do mount in the host's `/var/lib/containers/storage`, however, you must also mount in the host's `/var/run/libpod` and `/var/run/containers/storage` directories. 340 Not doing this will cause Podman in the container to detect that temporary files have been cleared, leading it to assume a system restart has taken place. 341 This can cause Podman to reset container states and lose track of running containers. 342 343 For running containers on the host from inside a container, we also recommend the [Podman remote client](remote_client.md), which only requires a single socket to be mounted into the container. 344 345 ### 14) Rootless 'podman build' fails EPERM on NFS: 346 347 NFS enforces file creation on different UIDs on the server side and does not understand user namespace, which rootless Podman requires. 348 When a container root process like YUM attempts to create a file owned by a different UID, NFS Server denies the creation. 349 NFS is also a problem for the file locks when the storage is on it. Other distributed file systems (for example: Lustre, Spectrum Scale, the General Parallel File System (GPFS)) are also not supported when running in rootless mode as these file systems do not understand user namespace. 350 351 #### Symptom 352 ```console 353 $ podman build . 354 ERRO[0014] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: open /root/.bash_logout: permission denied 355 error creating build container: Error committing the finished image: error adding layer with blob "sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17": ApplyLayer exit status 1 stdout: stderr: open /root/.bash_logout: permission denied 356 ``` 357 358 #### Solution 359 Choose one of the following: 360 * Setup containers/storage in a different directory, not on an NFS share. 361 * Create a directory on a local file system. 362 * Edit `~/.config/containers/libpod.conf` and point the `volume_path` option to that local directory. 363 * Otherwise just run Podman as root, via `sudo podman` 364 365 ### 15) Rootless 'podman build' fails when using OverlayFS: 366 367 The Overlay file system (OverlayFS) requires the ability to call the `mknod` command when creating whiteout files 368 when extracting an image. However, a rootless user does not have the privileges to use `mknod` in this capacity. 369 370 #### Symptom 371 ```console 372 podman build --storage-driver overlay . 373 STEP 1: FROM docker.io/ubuntu:xenial 374 Getting image source signatures 375 Copying blob edf72af6d627 done 376 Copying blob 3e4f86211d23 done 377 Copying blob 8d3eac894db4 done 378 Copying blob f7277927d38a done 379 Copying config 5e13f8dd4c done 380 Writing manifest to image destination 381 Storing signatures 382 Error: error creating build container: Error committing the finished image: error adding layer with blob "sha256:8d3eac894db4dc4154377ad28643dfe6625ff0e54bcfa63e0d04921f1a8ef7f8": Error processing tar file(exit status 1): operation not permitted 383 $ podman build . 384 ERRO[0014] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: open /root/.bash_logout: permission denied 385 error creating build container: Error committing the finished image: error adding layer with blob "sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17": ApplyLayer exit status 1 stdout: stderr: open /root/.bash_logout: permission denied 386 ``` 387 388 #### Solution 389 Choose one of the following: 390 * Complete the build operation as a privileged user. 391 * Install and configure fuse-overlayfs. 392 * Install the fuse-overlayfs package for your Linux Distribution. 393 * Add `mount_program = "/usr/bin/fuse-overlayfs"` under `[storage.options]` in your `~/.config/containers/storage.conf` file. 394 395 ### 16) rhel7-init based images don't work with cgroups v2 396 397 The systemd version shipped in rhel7-init doesn't have support for cgroups v2. You'll need at least systemd 230. 398 399 #### Symptom 400 ```console 401 402 sh# podman run --name test -d registry.access.redhat.com/rhel7-init:latest && sleep 10 && podman exec test systemctl status 403 c8567461948439bce72fad3076a91ececfb7b14d469bfa5fbc32c6403185beff 404 Failed to get D-Bus connection: Operation not permitted 405 Error: non zero exit code: 1: OCI runtime error 406 ``` 407 408 #### Solution 409 You'll need to either: 410 411 * configure the host to use cgroups v1 412 * update the image to use an updated version of systemd. 413 414 ### 17) rootless containers exit once the user session exits 415 416 You need to set lingering mode through loginctl to prevent user processes to be killed once 417 the user session completed. 418 419 #### Symptom 420 421 Once the user logs out all the containers exit. 422 423 #### Solution 424 You'll need to either: 425 426 * loginctl enable-linger $UID 427 428 or as root if your user has not enough privileges. 429 430 * sudo loginctl enable-linger $UID 431 432 ### 18) `podman run` fails with "bpf create: permission denied error" 433 434 The Kernel Lockdown patches deny eBPF programs when Secure Boot is enabled in the BIOS. [Matthew Garrett's post](https://mjg59.dreamwidth.org/50577.html) describes the relationship between Lockdown and Secure Boot and [Jan-Philip Gehrcke's](https://gehrcke.de/2019/09/running-an-ebpf-program-may-require-lifting-the-kernel-lockdown/) connects this with eBPF. [RH bug 1768125](https://bugzilla.redhat.com/show_bug.cgi?id=1768125) contains some additional details. 435 436 #### Symptom 437 438 Attempts to run podman result in 439 440 ```Error: bpf create : Operation not permitted: OCI runtime permission denied error``` 441 442 #### Solution 443 444 One workaround is to disable Secure Boot in your BIOS. 445 446 ### 19) error creating libpod runtime: there might not be enough IDs available in the namespace 447 448 Unable to pull images 449 450 #### Symptom 451 452 ```console 453 $ podman unshare cat /proc/self/uid_map 454 0 1000 1 455 ``` 456 457 #### Solution 458 459 ```console 460 $ podman system migrate 461 ``` 462 463 Original command now returns 464 465 ``` 466 $ podman unshare cat /proc/self/uid_map 467 0 1000 1 468 1 100000 65536 469 ``` 470 471 Reference [subuid](http://man7.org/linux/man-pages/man5/subuid.5.html) and [subgid](http://man7.org/linux/man-pages/man5/subgid.5.html) man pages for more detail. 472 473 ### 20) Passed-in device can't be accessed in rootless container 474 475 As a non-root user you have group access rights to a device that you want to 476 pass into a rootless container with `--device=...`. 477 478 #### Symptom 479 480 Any access inside the container is rejected with "Permission denied". 481 482 #### Solution 483 484 The runtime uses `setgroups(2)` hence the process looses all additional groups 485 the non-root user has. If you use the `crun` runtime, 0.10.4 or newer, 486 then you can enable a workaround by adding `--annotation io.crun.keep_original_groups=1` 487 to the `podman` command line.