github.com/containerd/nerdctl/v2@v2.0.0-beta.5.0.20240520001846-b5758f54fa28/examples/nerdctl-ipfs-registry-kubernetes/ipfs-stargz-snapshotter/README.md (about)

     1  # Example: Node-to-Node image sharing on Kubernetes with lazy pulling using `nerdctl ipfs registry` with eStargz and Stargz Snapshotter
     2  
     3  This directory contains an example Kubernetes setup for node-to-node image sharing with lazy pulling (eStargz).
     4  
     5  Usage:
     6  - Generate `bootstrap.yaml` by executing `bootstrap.yaml.sh` (e.g. `./bootstrap.yaml.sh > ${DIR_LOCATION}/bootstrap.yaml`)
     7    - [`ipfs-swarm-key-gen`](https://github.com/Kubuxu/go-ipfs-swarm-key-gen) is required (see https://github.com/ipfs/kubo/blob/v0.15.0/docs/experimental-features.md#private-networks)
     8  - Deploy `bootstrap.yaml` and `nerdctl-ipfs-registry.yaml` (e.g. using `kubectl apply`)
     9  - Make sure nodes contain containerd >= v1.5.8 and stargz-snapshotter.
    10    - Here we use `ghcr.io/containerd/stargz-snapshotter:0.12.1-kind` that contains both of them. (This image requires kind >= 0.16.0)
    11  - You might want to change some configuration written in `nerdctl-ipfs-registry.yaml` (e.g. [chaning profile based on your node's resouce requirements](https://docs.ipfs.tech/how-to/default-profile/#available-profiles))
    12  
    13  ## About eStargz and Stargz Snapshotter
    14  
    15  eStargz is an OCI-compatible image format to enable to startup a container before the entire image contents become locally available.
    16  This allows fast cold startup of containers.
    17  Necessary chunks of image contents are fetched to the node on-demand.
    18  This technique is called *lazy pulling*.
    19  [Stargz Snapshotter](https://github.com/containerd/stargz-snapshotter) is the plugin of containerd that enables lazy pulling of eStargz on containerd.
    20  
    21  This example runs stargz snapshotter on each node as a systemd service and plugs it into containerd on the same node.
    22  containerd on each node performs lazy pulling of eStargz images from IPFS via `nerdct ipfs registry`.
    23  Thus, the eStargz image starts up without waiting for the entire contents becoming locally available so faster cold start can be expected.
    24  
    25  For more details about eStargz and lazy pulling, please refer to [Stargz Snapshotter](https://github.com/containerd/stargz-snapshotter) repository.
    26  
    27  ## Example on kind
    28  
    29  Prepare cluster (make sure kind nodes contain containerd >= v1.5.8).
    30  
    31  ```console
    32  $ cat <<EOF > /tmp/kindconfig.yaml
    33  kind: Cluster
    34  apiVersion: kind.x-k8s.io/v1alpha4
    35  nodes:
    36  - role: control-plane
    37  - role: worker
    38  - role: worker
    39  EOF
    40  $ kind create cluster --image=ghcr.io/containerd/stargz-snapshotter:0.12.1-kind --config=/tmp/kindconfig.yaml
    41  $ ./bootstrap.yaml.sh > ./bootstrap.yaml
    42  $ kubectl apply -f .
    43  ```
    44  
    45  Prepare `kind-worker` (1st node) for importing an image to IPFS
    46  
    47  (in `kind-worker`)
    48  
    49  ```console
    50  $ docker exec -it kind-worker /bin/bash
    51  (kind-worker)# NERDCTL_VERSION=0.23.0
    52  (kind-worker)# curl -sSL --output /tmp/nerdctl.tgz https://github.com/containerd/nerdctl/releases/download/v${NERDCTL_VERSION}/nerdctl-${NERDCTL_VERSION}-linux-amd64.tar.gz
    53  (kind-worker)# tar zxvf /tmp/nerdctl.tgz -C /usr/local/bin/
    54  ```
    55  
    56  Add an image to `kind-worker`.
    57  
    58  ```console
    59  $ docker exec -it kind-worker /bin/bash
    60  (kind-worker)# mkdir -p /tmp/ipfsapi ; echo -n /ip4/127.0.0.1/tcp/5001 >  /tmp/ipfsapi/api
    61  (kind-worker)# export IPFS_PATH=/tmp/ipfsapi
    62  (kind-worker)# nerdctl pull ghcr.io/stargz-containers/jenkins:2.60.3-esgz
    63  (kind-worker)# nerdctl push ipfs://ghcr.io/stargz-containers/jenkins:2.60.3-esgz
    64  (kind-worker)# nerdctl rmi ghcr.io/stargz-containers/jenkins:2.60.3-esgz
    65  ```
    66  
    67  > NOTE: This example copies a pre-converted eStargz image (`ghcr.io/stargz-containers/jenkins:2.60.3-esgz`) from the registry to IPFS but you can push non-eStargz image to IPFS with converting it to eStargz using `--estargz` flag of `nerdctl push`. This flag automatically performs convertion of the image to eStargz.
    68  
    69  The eStargz image added to `kind-worker` is shared to `kind-worker2` via IPFS.
    70  You can perform lazy pulling of this eStargz image among nodes using the following manifest.
    71  CID of the pushed image is printed when `nerdctl push` is succeeded (we assume that the image is added to IPFS as CID `bafkreidqrxutnnuc3oilje27px5o3gggzrfyomumrprcavr7nquoy3cdje`).
    72  
    73  
    74  ```console
    75  $ cat <<EOF | kubectl apply -f -
    76  apiVersion: apps/v1
    77  kind: Deployment
    78  metadata:
    79    name: jenkins
    80  spec:
    81    replicas: 2
    82    selector:
    83      matchLabels:
    84        app: jenkins
    85    template:
    86      metadata:
    87        labels:
    88          app: jenkins
    89      spec:
    90        containers:
    91        - name: jenkins
    92          image: localhost:5050/ipfs/bafkreidqrxutnnuc3oilje27px5o3gggzrfyomumrprcavr7nquoy3cdje
    93          resources:
    94            requests:
    95              cpu: 1
    96  EOF
    97  ```
    98  
    99  > NOTE1: Kubernetes doesn't support `ipfs://CID` URL on YAML as of now so we need to use `localhost:5050/ipfs/CID` form instead. In the future, this limitation should be eliminated.
   100  
   101  > NOTE2: stargz-snapshotter currently perfoms lazy pulling via `nerdctl ipfs registry` running on localhost instead of leveraging its [native support for fetching contents via ipfs daemon](https://github.com/containerd/stargz-snapshotter/blob/v0.12.0/docs/ipfs.md). This is because of the limitation described in NOTE1 and expected to be fixed once NOTE1 is solved.
   102  
   103  The image runs on all nodes.
   104  You may observe faster pulling of the image by eStargz.
   105  
   106  ```console
   107  $ kubectl get pods -owide | grep jenkins
   108  jenkins-959bc9548-6hcwc           1/1     Running   0          8s     10.244.2.4   kind-worker    <none>           <none>
   109  jenkins-959bc9548-rfsxm           1/1     Running   0          8s     10.244.1.3   kind-worker2   <none>           <none>
   110  $ kubectl get pods -o name | grep jenkins | xargs -I{} kubectl describe {} | grep Pulled
   111    Normal  Pulled     2m13s  kubelet            Successfully pulled image "localhost:5050/ipfs/bafkreidqrxutnnuc3oilje27px5o3gggzrfyomumrprcavr7nquoy3cdje" in 1.339830318s
   112    Normal  Pulled     2m13s  kubelet            Successfully pulled image "localhost:5050/ipfs/bafkreidqrxutnnuc3oilje27px5o3gggzrfyomumrprcavr7nquoy3cdje" in 1.585882041s
   113  ```
   114  
   115  eStargz filesystem is used as the rootfs of the container.
   116  The file contents are lazily downloaded to the node.
   117  
   118  ```console
   119  $ docker exec -it kind-worker2 mount | grep "stargz on"
   120  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/37/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   121  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/38/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   122  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/39/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   123  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/40/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   124  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/41/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   125  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/42/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   126  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/43/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   127  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/44/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   128  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/45/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   129  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/46/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   130  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/47/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   131  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/48/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   132  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/49/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   133  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/50/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   134  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/51/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   135  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/52/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   136  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/53/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   137  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/54/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   138  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/55/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   139  stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/56/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
   140  ```