github.com/containerd/nerdctl/v2@v2.0.0-beta.5.0.20240520001846-b5758f54fa28/examples/nerdctl-ipfs-registry-kubernetes/ipfs-cluster/README.md (about)

     1  # Example: Node-to-Node image sharing on Kubernetes with content replication using `nerdctl ipfs registry` with ipfs-cluster
     2  
     3  This directory contains an example Kubernetes setup for node-to-node image sharing with content replication (ipfs-cluster).
     4  
     5  Usage:
     6  - Generate `bootstrap.yaml` by executing `bootstrap.yaml.sh` (e.g. `./bootstrap.yaml.sh > ${DIR_LOCATION}/bootstrap.yaml`)
     7    - [`ipfs-swarm-key-gen`](https://github.com/Kubuxu/go-ipfs-swarm-key-gen) is required (see https://github.com/ipfs/kubo/blob/v0.15.0/docs/experimental-features.md#private-networks)
     8    - [`ipfs-key`](https://github.com/whyrusleeping/ipfs-key) is required (see https://ipfscluster.io/documentation/guides/k8s/)
     9  - Deploy `bootstrap.yaml` and `nerdctl-ipfs-registry.yaml` (e.g. using `kubectl apply`)
    10  - Make sure nodes contain containerd >= v1.5.8
    11  - You might want to change some configuration written in `nerdctl-ipfs-registry.yaml` (e.g. [chaning profile based on your node's resouce requirements](https://docs.ipfs.tech/how-to/default-profile/#available-profiles))
    12  
    13  ## Example on kind
    14  
    15  Prepare cluster (make sure kind nodes contain containerd >= v1.5.8).
    16  
    17  ```console
    18  $ cat <<EOF > /tmp/kindconfig.yaml
    19  kind: Cluster
    20  apiVersion: kind.x-k8s.io/v1alpha4
    21  nodes:
    22  - role: control-plane
    23  - role: worker
    24  - role: worker
    25  - role: worker
    26  EOF
    27  $ kind create cluster --image=kindest/node:v1.25.2 --config=/tmp/kindconfig.yaml
    28  $ ./bootstrap.yaml.sh > ./bootstrap.yaml
    29  $ kubectl apply -f .
    30  ```
    31  
    32  Prepare `kind-worker` (1st node) for importing an image to IPFS
    33  
    34  (in `kind-worker`)
    35  
    36  ```console
    37  $ docker exec -it kind-worker /bin/bash
    38  (kind-worker)# NERDCTL_VERSION=0.23.0
    39  (kind-worker)# curl -sSL --output /tmp/nerdctl.tgz https://github.com/containerd/nerdctl/releases/download/v${NERDCTL_VERSION}/nerdctl-${NERDCTL_VERSION}-linux-amd64.tar.gz
    40  (kind-worker)# tar zxvf /tmp/nerdctl.tgz -C /usr/local/bin/
    41  ```
    42  
    43  Add an image to `kind-worker`.
    44  
    45  > NOTE: port 9095 needs to be used as the IPFS API port instead of 5001 (see also https://cluster.ipfs.io/documentation/reference/proxy/)
    46  
    47  ```console
    48  $ docker exec -it kind-worker /bin/bash
    49  (kind-worker)# mkdir -p /tmp/ipfsapi ; echo -n /ip4/127.0.0.1/tcp/9095 >  /tmp/ipfsapi/api
    50  (kind-worker)# export IPFS_PATH=/tmp/ipfsapi
    51  (kind-worker)# nerdctl pull ghcr.io/stargz-containers/jenkins:2.60.3-org
    52  (kind-worker)# nerdctl push ipfs://ghcr.io/stargz-containers/jenkins:2.60.3-org
    53  (kind-worker)# nerdctl rmi ghcr.io/stargz-containers/jenkins:2.60.3-org
    54  ```
    55  
    56  The image added to `kind-worker` is shared to other nodes via IPFS.
    57  You can run this image on the nodes using the following manifest.
    58  CID of the pushed image is printed when `nerdctl push` is succeeded (we assume that the image is added to IPFS as CID `bafkreictyyoysj56v772xbfhyfrcvmgmfpa4vodmqaroz53ytvai7nof6u`).
    59  
    60  ```console
    61  $ cat <<EOF | kubectl apply -f -
    62  apiVersion: apps/v1
    63  kind: Deployment
    64  metadata:
    65    name: jenkins
    66  spec:
    67    replicas: 2
    68    selector:
    69      matchLabels:
    70        app: jenkins
    71    template:
    72      metadata:
    73        labels:
    74          app: jenkins
    75      spec:
    76        containers:
    77        - name: jenkins
    78          image: localhost:5050/ipfs/bafkreictyyoysj56v772xbfhyfrcvmgmfpa4vodmqaroz53ytvai7nof6u
    79          resources:
    80            requests:
    81              cpu: 1
    82  EOF
    83  ```
    84  
    85  > NOTE: Kubernetes doesn't support `ipfs://CID` URL on YAML as of now so we need to use `localhost:5050/ipfs/CID` form instead. In the future, this limitation should be eliminated.
    86  
    87  The image runs on the nodes.
    88  
    89  ```console
    90  $ kubectl get pods -owide | grep jenkins
    91  jenkins-7fcb4687c4-9gmrr          1/1     Running   0          22s     10.244.1.3   kind-worker    <none>           <none>
    92  jenkins-7fcb4687c4-kvdbl          1/1     Running   0          22s     10.244.3.3   kind-worker3   <none>           <none>
    93  ```
    94  
    95  You can see that ipfs-cluster is activated on the cluster.
    96  
    97  ```console
    98  $ kubectl exec -it ipfs-n59wf -c ipfs-cluster -- /bin/sh
    99  / # ipfs-cluster-ctl status bafkreictyyoysj56v772xbfhyfrcvmgmfpa4vodmqaroz53ytvai7nof6u
   100  bafkreictyyoysj56v772xbfhyfrcvmgmfpa4vodmqaroz53ytvai7nof6u:
   101      > ipfs-6cmlq           : PINNED | 2022-10-18T06:28:46Z | Attempts: 0 | Priority: false
   102      > ipfs-rvcb2           : PINNED | 2022-10-18T06:28:46Z | Attempts: 0 | Priority: false
   103      > ipfs-2m4tm           : REMOTE | 2022-10-18T06:30:16.23637825Z | Attempts: 0 | Priority: false
   104      > ipfs-bootstrap-67c54bc878-mgtg4 : REMOTE | 2022-10-18T06:30:16.23637825Z | Attempts: 0 | Priority: false
   105  ```