github.com/rkt/rkt@v1.30.1-0.20200224141603-171c416fac02/Documentation/troubleshooting.md (about)

     1  # Troubleshooting
     2  
     3  This document lists common rkt problems and how to fix or work around them.
     4  
     5  ## Missing container logs
     6  
     7  When checking the logs of a container, they might be missing with an error like this:
     8  
     9  ```
    10  $ journalctl -M rkt-3f045be0-1632-42f1-ba15-df984a82636f
    11  Journal file /var/lib/rkt/pods/run/3f045be0-1632-42f1-ba15-df984a82636f/stage1/rootfs/var/log/journal/3f045be0163242f1ba15df984a82636f/system.journal uses an unsupported feature, ignoring file.
    12  -- No entries --
    13  ```
    14  
    15  This is because rkt's journald integration is only supported if systemd is compiled with `lz4` compression enabled.
    16  
    17  You can check if it is enabled by making sure you see `+LZ4` when running `systemctl --version`:
    18  
    19  ```
    20  $ systemctl --version
    21  systemd 235
    22  [...] +LZ4 [...]
    23  ```
    24  
    25  ## Bad system call
    26  
    27  During rkt execution, you might encounter the message `Bad system call` followed by rkt terminating.
    28  It's most likely a result of a too restrictive seccomp profile.
    29  
    30  As a workaround, you can disable seccomp with `--insecure-options=seccomp`.
    31  
    32  As a proper fix, you can [tweak the seccomp profile][seccomp-guide].
    33  
    34  ## Operation not permitted errors
    35  
    36  During rkt execution, you might encounter a `Operation not permitted` message followed by rkt exiting.
    37  Your image probably uses more capabilities than allowed in rkt's default list.
    38  
    39  As a workaround, you can disable capabilities enforcement with `--insecure-options=capabilities`.
    40  
    41  As a proper fix, you can [create your own list][capabilities-guide].
    42  
    43  ## BTRFS + overlay
    44  
    45  ```
    46  prepare-app@opt-stage2-alpine\x2dsh-rootfs.service: Job prepare-app@opt-stage2-alpine\x2dsh-rootfs.service/start failed with result 'dependency'.
    47  systemd-journald.service: Unit entered failed state.
    48  systemd-journald.service: Failed with result 'signal'.
    49  systemd-journald.service: Service has no hold-off time, scheduling restart.
    50  ```
    51  
    52  To solve this update to Linux 4.5.2 or newer (see [#2175](https://github.com/rkt/rkt/issues/2175)).
    53  
    54  ## SELinux + overlay
    55  
    56  You might se an error like this one when starting a rkt pod:
    57  
    58  ```
    59  /usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied
    60  ```
    61  
    62  The overlay filesystem doesn't work with SELinux in kernels older than 4.9 (see [1727](https://github.com/rkt/rkt/issues/1727)).
    63  Please update your kernel to a newer version.
    64  
    65  ## Garbage collect not working in old kernels
    66  
    67  You might see messages like these when running `rkt gc`:
    68  
    69  ```
    70  Unable to remove pod "42e78965-c60b-4f4f-b412-484cd381fe90": remove /var/lib/rkt/pods/exited-garbage/42e78965-c60b-4f4f-b412-484cd381fe90/stage1/rootfs: device or resource busy
    71  ```
    72  
    73  This might be due to using a kernel older than 3.18 (see [lazy umounts on unlinked files and directories](https://github.com/torvalds/linux/commit/8ed936b) and [#1922](https://github.com/rkt/rkt/issues/1922)).
    74  Please update your kernel to a newer version.
    75  
    76  ## Running rkt on top of an overlay filesystem
    77  
    78  Due to limitations in the Linux kernel, using rkt's overlay support on top of an overlay filesystem requires the upperdir and workdir to support the creation of trusted.* extended attributes and valid d_type in readdir responses (see [kernel/Documentation/filesystems/overlayfs.txt](https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt)).
    79  
    80  The symptom is an error message like this:
    81  
    82  ```
    83  stage0: error setting up stage1
    84    └─error rendering overlay filesystem
    85      └─problem mounting overlay filesystem
    86        └─error mounting overlay with options 'lowerdir=/var/lib/rkt/cas/tree/deps-sha512-f3d5f69d7faba1be7067d610f33131c18ac59eb43b1495016ade65bd13912578/rootfs,upperdir=/var/lib/rkt/pods/run/307bd207-7eab-4028-8d12-2d525e5b8ed9/overlay/deps-sha512-f3d5f69d7faba1be7067d610f33131c18ac59eb43b1495016ade65bd13912578/upper,workdir=/var/lib/rkt/pods/run/307bd207-7eab-4028-8d12-2d525e5b8ed9/overlay/deps-sha512-f3d5f69d7faba1be7067d610f33131c18ac59eb43b1495016ade65bd13912578/work' and dest '/var/lib/rkt/pods/run/307bd207-7eab-4028-8d12-2d525e5b8ed9/stage1/rootfs'
    87          └─invalid argument
    88  ```
    89  
    90  This problem typically happens when trying to run rkt inside rkt.
    91  To successfully run rkt inside rkt, use one of the following workarounds:
    92  - set up `/var/lib/rkt` in the outer rkt as a host volume
    93  - use `--no-overlay` for either the outer or the inner rkt
    94  
    95  [capabilities-guide]: capabilities-guide.md
    96  [seccomp-guide]: seccomp-guide.md