github.com/anth0d/nomad@v0.0.0-20221214183521-ae3a0a2cad06/demo/csi/kadalu-csi/README.md (about) 1 # Kadalu CSI Plugin 2 3 Author: @leelavg and the [Kadalu][kadalu_org] team. 4 5 The configuration here is for using external [Gluster] volumes as persistent 6 storage in Nomad using [Kadalu CSI][kadalu]. 7 8 Refer to the actual job files before proceeding with this demo and change the 9 configuration as required. Follow along with the commands according to your 10 configuration. 11 12 Locally tested against Nomad v1.3.1 13 14 ## Local Development 15 16 This section can be skipped if you already have a Nomad cluster setup. 17 18 ```console 19 # Clone configuration repository used to create local Nomad cluster in Docker 20 $ git clone https://github.com/leelavg/kadalu-nomad && cd kadalu-nomad 21 22 # Install Shipyard following the instructions in https://shipyard.run 23 # Create local cluster 24 $ shipyard run 25 [...] 26 $ eval $(shipyard env) 27 $ export job_dir="$(pwd)/kadalu" 28 ``` 29 30 ## Demo 31 32 ### Pre-requisites 33 - Configure varisables mentioned in `cluster.vars` to reflect your external 34 Gluster details. 35 - For convenience the necessary variables are set from the CLI when running the 36 job. 37 38 ```console 39 $ export volname="sample-pool" gluster_hosts="10.x.x.x" gluster_volname="sample-vol" job_dir="${job_dir:-$(pwd)}" 40 41 # Make sure external gluster volume is started and quota is set 42 $ ssh $gluster_hosts "gluster volume info $gluster_volname | grep Status" 43 Status: Started 44 45 $ ssh $gluster_hosts "gluster volume quota $gluster_volname enable" 46 volume quota : success 47 ``` 48 49 ### CSI Deployment 50 51 Deploy the CSI plugin controller. 52 53 ```console 54 $ nomad run -var="volname=$volname" -var="gluster_hosts=$gluster_hosts" -var="gluster_volname=$gluster_volname" $job_dir/controller.nomad 55 ==> 2021-09-20T18:23:07+05:30: Monitoring evaluation "19317b74" 56 2021-09-20T18:23:07+05:30: Evaluation triggered by job "kadalu-csi-controller" 57 ==> 2021-09-20T18:23:08+05:30: Monitoring evaluation "19317b74" 58 2021-09-20T18:23:08+05:30: Evaluation within deployment: "d9ee4dd7" 59 2021-09-20T18:23:08+05:30: Allocation "d55e314d" created: node "4e105698", group "controller" 60 2021-09-20T18:23:08+05:30: Evaluation status changed: "pending" -> "complete" 61 ==> 2021-09-20T18:23:08+05:30: Evaluation "19317b74" finished with status "complete" 62 ==> 2021-09-20T18:23:08+05:30: Monitoring deployment "d9ee4dd7" 63 ✓ Deployment "d9ee4dd7" successful 64 65 2021-09-20T18:23:28+05:30 66 ID = d9ee4dd7 67 Job ID = kadalu-csi-controller 68 Job Version = 0 69 Status = successful 70 Description = Deployment completed successfully 71 72 Deployed 73 Task Group Desired Placed Healthy Unhealthy Progress Deadline 74 controller 1 1 1 0 2021-09-20T13:03:27Z 75 ``` 76 77 Deploy the CSI node plugin. 78 79 ```console 80 $ nomad run -var="volname=$volname" -var="gluster_hosts=$gluster_hosts" -var="gluster_volname=$gluster_volname" $job_dir/nodeplugin.nomad 81 ==> 2021-09-20T18:23:53+05:30: Monitoring evaluation "bd4d95d1" 82 2021-09-20T18:23:53+05:30: Evaluation triggered by job "kadalu-csi-nodeplugin" 83 ==> 2021-09-20T18:23:54+05:30: Monitoring evaluation "bd4d95d1" 84 2021-09-20T18:23:54+05:30: Allocation "4c05ab5a" created: node "4e105698", group "nodeplugin" 85 2021-09-20T18:23:54+05:30: Evaluation status changed: "pending" -> "complete" 86 ==> 2021-09-20T18:23:54+05:30: Evaluation "bd4d95d1" finished with status "complete" 87 ``` 88 89 Verify the CSI plugin status. 90 91 ```console 92 $ nomad plugin status kadalu-csi 93 ID = kadalu-csi 94 Provider = kadalu 95 Version = 0.8.15 96 Controllers Healthy = 1 97 Controllers Expected = 1 98 Nodes Healthy = 1 99 Nodes Expected = 1 100 101 Allocations 102 ID Node ID Task Group Version Desired Status Created Modified 103 d55e314d 4e105698 controller 0 run running 1m20s ago 1m ago 104 4c05ab5a 4e105698 nodeplugin 0 run running 35s ago 20s ago 105 ``` 106 107 ### Volume Management 108 109 Next, you will go through volume creation, attachment and deletion operations, 110 covering a typical volume life-cycle. 111 112 #### Creating a Volume 113 114 ```console 115 # Create Nomad volume 116 $ sed -e "s/POOL/$volname/" -e "s/GHOST/$gluster_hosts/" -e "s/GVOL/$gluster_volname/" $job_dir/volume.hcl | nomad volume create - 117 Created external volume csi-test with ID csi-test 118 ``` 119 120 #### Attaching and Using a Volume 121 122 ```console 123 # Attach the volume to a sample app 124 $ nomad run $job_dir/app.nomad 125 ==> 2021-09-20T18:28:28+05:30: Monitoring evaluation "e6dd3129" 126 2021-09-20T18:28:28+05:30: Evaluation triggered by job "sample-pv-check" 127 ==> 2021-09-20T18:28:29+05:30: Monitoring evaluation "e6dd3129" 128 2021-09-20T18:28:29+05:30: Evaluation within deployment: "814e328c" 129 2021-09-20T18:28:29+05:30: Allocation "64745b25" created: node "4e105698", group "apps" 130 2021-09-20T18:28:29+05:30: Evaluation status changed: "pending" -> "complete" 131 ==> 2021-09-20T18:28:29+05:30: Evaluation "e6dd3129" finished with status "complete" 132 ==> 2021-09-20T18:28:29+05:30: Monitoring deployment "814e328c" 133 ✓ Deployment "814e328c" successful 134 135 2021-09-20T18:28:58+05:30 136 ID = 814e328c 137 Job ID = sample-pv-check 138 Job Version = 0 139 Status = successful 140 Description = Deployment completed successfully 141 142 Deployed 143 Task Group Desired Placed Healthy Unhealthy Progress Deadline 144 apps 1 1 1 0 2021-09-20T13:08:56Z 145 146 # Export allocation ID (64745b25) from the previous command output 147 $ export app=64745b25 148 149 # Verify that the CSI Volume is accessible 150 $ nomad alloc exec $app bash /kadalu/script.sh 151 This is a sample application 152 153 # df -h 154 Filesystem Size Used Available Use% Mounted on 155 <gluster_hosts>:<gluster_volname> 181.2M 0 181.2M 0% /mnt/pv 156 157 # mount 158 Write/Read test on PV mount Mon 159 Sep 20 12:59:34 UTC 2021 160 SUCCESS 161 162 # Write some data on the volume 163 $ nomad alloc exec $app bash -c 'cd /mnt/pv; for i in {1..10}; do cat /dev/urandom | tr -dc [:space:][:print:] | head -c 1m > file$i; done;' 164 165 # Checksum the written data 166 $ nomad alloc exec $app bash -c 'ls /mnt/pv; find /mnt/pv -type f -exec md5sum {} + | cut -f1 -d" " | sort | md5sum' 167 file1 file2 file4 file6 file8 168 file10 file3 file5 file7 file9 169 6776dd355c0f2ba5a1781b9831e5c174 - 170 171 # Stop sample app and run it again to check data persistence 172 $ nomad status 173 ID Type Priority Status Submit Date 174 kadalu-csi-controller service 50 running 2021-09-20T18:23:07+05:30 175 kadalu-csi-nodeplugin system 50 running 2021-09-20T18:23:53+05:30 176 sample-pv-check service 50 running 2021-09-20T18:28:28+05:30 177 178 $ nomad stop sample-pv-check 179 ==> 2021-09-20T18:36:47+05:30: Monitoring evaluation "eecc0c00" 180 2021-09-20T18:36:47+05:30: Evaluation triggered by job "sample-pv-check" 181 ==> 2021-09-20T18:36:48+05:30: Monitoring evaluation "eecc0c00" 182 2021-09-20T18:36:48+05:30: Evaluation within deployment: "814e328c" 183 2021-09-20T18:36:48+05:30: Evaluation status changed: "pending" -> "complete" 184 ==> 2021-09-20T18:36:48+05:30: Evaluation "eecc0c00" finished with status "complete" 185 ==> 2021-09-20T18:36:48+05:30: Monitoring deployment "814e328c" 186 ✓ Deployment "814e328c" successful 187 188 2021-09-20T18:36:48+05:30 189 ID = 814e328c 190 Job ID = sample-pv-check 191 Job Version = 0 192 Status = successful 193 Description = Deployment completed successfully 194 195 Deployed 196 Task Group Desired Placed Healthy Unhealthy Progress Deadline 197 apps 1 1 1 0 2021-09-20T13:08:56Z 198 199 $ nomad run $job_dir/app.nomad 200 ==> 2021-09-20T18:37:49+05:30: Monitoring evaluation "e04b4549" 201 2021-09-20T18:37:49+05:30: Evaluation triggered by job "sample-pv-check" 202 ==> 2021-09-20T18:37:50+05:30: Monitoring evaluation "e04b4549" 203 2021-09-20T18:37:50+05:30: Evaluation within deployment: "66d246ee" 204 2021-09-20T18:37:50+05:30: Allocation "526d5543" created: node "4e105698", group "apps" 205 2021-09-20T18:37:50+05:30: Evaluation status changed: "pending" -> "complete" 206 ==> 2021-09-20T18:37:50+05:30: Evaluation "e04b4549" finished with status "complete" 207 ==> 2021-09-20T18:37:50+05:30: Monitoring deployment "66d246ee" 208 ✓ Deployment "66d246ee" successful 209 210 2021-09-20T18:38:10+05:30 211 ID = 66d246ee 212 Job ID = sample-pv-check 213 Job Version = 2 214 Status = successful 215 Description = Deployment completed successfully 216 217 Deployed 218 Task Group Desired Placed Healthy Unhealthy Progress Deadline 219 apps 1 1 1 0 2021-09-20T13:18:08Z 220 221 # Export the new allocation ID and verify that md5sum matches after stopping and 222 # running the same job 223 $ export app=526d5543 224 $ nomad alloc exec $app bash -c 'ls /mnt/pv; find /mnt/pv -type f -exec md5sum {} + | cut -f1 -d" " | sort | md5sum' 225 file1 file10 file2 file3 file4 file5 file6 file7 file8 file9 226 6776dd355c0f2ba5a1781b9831e5c174 - 227 ``` 228 229 #### Cleanup 230 ```console 231 # Stop sample app, delete the volume and stop the CSI plugin components 232 $ nomad stop sample-pv-check 233 $ nomad volume delete csi-test 234 $ nomad stop kadalu-csi-nodeplugin 235 $ nomad stop kadalu-csi-controller 236 237 # Destroy local Shipyard cluster 238 $ shipyard destroy 239 ``` 240 241 ## Contact 242 243 - For any extra information/feature with regards to the Kadalu CSI plugin, 244 please raise an issue against the [`kadalu` repo][kadalu]. 245 - For any extra information with regards to the local Nomad dev setup for CSI, 246 please raise an issue against the [`kadalu-nomad` repo][kadalu_nomad]. 247 - Based on ask/feature request, we may work on supporting internal Gluster 248 deployed and managed by Nomad itself (feature parity with current Kubernetes 249 deployments). 250 - If this folder isn't updated frequently you can find updated jobs at the 251 [`nomad` folder][nomad_folder] in the `kadalu` repository. 252 253 [Gluster]: https://www.gluster.org/ 254 [kadalu]: https://github.com/kadalu/kadalu 255 [kadalu_org]: https://github.com/kadalu 256 [kadalu_nomad]: https://github.com/leelavg/kadalu-nomad 257 [nomad_folder]: https://github.com/kadalu/kadalu/tree/devel/nomad