github.com/qsunny/k8s@v0.0.0-20220101153623-e6dca256d5bf/examples-master/staging/storm/README.md (about) 1 # Storm example 2 3 Following this example, you will create a functional [Apache 4 Storm](http://storm.apache.org/) cluster using Kubernetes and 5 [Docker](http://docker.io). 6 7 You will setup an [Apache ZooKeeper](http://zookeeper.apache.org/) 8 service, a Storm master service (a.k.a. Nimbus server), and a set of 9 Storm workers (a.k.a. supervisors). 10 11 For the impatient expert, jump straight to the [tl;dr](#tldr) 12 section. 13 14 ### Sources 15 16 Source is freely available at: 17 * Docker image - https://github.com/mattf/docker-storm 18 * Docker Trusted Build - https://registry.hub.docker.com/search?q=mattf/storm 19 20 ## Step Zero: Prerequisites 21 22 This example assumes you have a Kubernetes cluster installed and 23 running, and that you have installed the ```kubectl``` command line 24 tool somewhere in your path. Please see the [getting 25 started](https://kubernetes.io/docs/user-journeys/users/application-developer/foundational/#section-1) for installation 26 instructions for your platform. 27 28 ## Step One: Start your ZooKeeper service 29 30 ZooKeeper is a distributed coordination [service](https://kubernetes.io/docs/concepts/services-networking/service/) that Storm uses as a 31 bootstrap and for state storage. 32 33 Use the [`examples/storm/zookeeper.json`](zookeeper.json) file to create a [pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/) running 34 the ZooKeeper service. 35 36 ```sh 37 $ kubectl create -f examples/storm/zookeeper.json 38 ``` 39 40 Then, use the [`examples/storm/zookeeper-service.json`](zookeeper-service.json) file to create a 41 logical service endpoint that Storm can use to access the ZooKeeper 42 pod. 43 44 ```sh 45 $ kubectl create -f examples/storm/zookeeper-service.json 46 ``` 47 48 You should make sure the ZooKeeper pod is Running and accessible 49 before proceeding. 50 51 ### Check to see if ZooKeeper is running 52 53 ```sh 54 $ kubectl get pods 55 NAME READY STATUS RESTARTS AGE 56 zookeeper 1/1 Running 0 43s 57 ``` 58 59 ### Check to see if ZooKeeper is accessible 60 61 ```console 62 $ kubectl get services 63 NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE 64 zookeeper 10.254.139.141 <none> 2181/TCP name=zookeeper 10m 65 kubernetes 10.0.0.2 <none> 443/TCP <none> 1d 66 67 $ echo ruok | nc 10.254.139.141 2181; echo 68 imok 69 ``` 70 71 ## Step Two: Start your Nimbus service 72 73 The Nimbus service is the master (or head) service for a Storm 74 cluster. It depends on a functional ZooKeeper service. 75 76 Use the [`examples/storm/storm-nimbus.json`](storm-nimbus.json) file to create a pod running 77 the Nimbus service. 78 79 ```sh 80 $ kubectl create -f examples/storm/storm-nimbus.json 81 ``` 82 83 Then, use the [`examples/storm/storm-nimbus-service.json`](storm-nimbus-service.json) file to 84 create a logical service endpoint that Storm workers can use to access 85 the Nimbus pod. 86 87 ```sh 88 $ kubectl create -f examples/storm/storm-nimbus-service.json 89 ``` 90 91 Ensure that the Nimbus service is running and functional. 92 93 ### Check to see if Nimbus is running and accessible 94 95 ```sh 96 $ kubectl get services 97 NAME LABELS SELECTOR IP(S) PORT(S) 98 kubernetes component=apiserver,provider=kubernetes <none> 10.254.0.2 443 99 zookeeper name=zookeeper name=zookeeper 10.254.139.141 2181 100 nimbus name=nimbus name=nimbus 10.254.115.208 6627 101 102 $ sudo docker run -it -w /opt/apache-storm mattf/storm-base sh -c '/configure.sh 10.254.139.141 10.254.115.208; ./bin/storm list' 103 ... 104 No topologies running. 105 ``` 106 107 ## Step Three: Start your Storm workers 108 109 The Storm workers (or supervisors) do the heavy lifting in a Storm 110 cluster. They run your stream processing topologies and are managed by 111 the Nimbus service. 112 113 The Storm workers need both the ZooKeeper and Nimbus services to be 114 running. 115 116 Use the [`examples/storm/storm-worker-controller.yaml`](storm-worker-controller.yaml) file to create a 117 [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) that manages the worker pods. 118 119 ```sh 120 $ kubectl create -f examples/storm/storm-worker-controller.yaml 121 ``` 122 123 ### Check to see if the workers are running 124 125 One way to check on the workers is to get information from the 126 ZooKeeper service about how many clients it has. 127 128 ```sh 129 $ echo stat | nc 10.254.139.141 2181; echo 130 Zookeeper version: 3.4.6--1, built on 10/23/2014 14:18 GMT 131 Clients: 132 /192.168.48.0:44187[0](queued=0,recved=1,sent=0) 133 /192.168.45.0:39568[1](queued=0,recved=14072,sent=14072) 134 /192.168.86.1:57591[1](queued=0,recved=34,sent=34) 135 /192.168.8.0:50375[1](queued=0,recved=34,sent=34) 136 137 Latency min/avg/max: 0/2/2570 138 Received: 23199 139 Sent: 23198 140 Connections: 4 141 Outstanding: 0 142 Zxid: 0xa39 143 Mode: standalone 144 Node count: 13 145 ``` 146 147 There should be one client from the Nimbus service and one per 148 worker. Ideally, you should get ```stat``` output from ZooKeeper 149 before and after creating the replication controller. 150 151 (Pull requests welcome for alternative ways to validate the workers) 152 153 ## tl;dr 154 155 ```kubectl create -f zookeeper.json``` 156 157 ```kubectl create -f zookeeper-service.json``` 158 159 Make sure the ZooKeeper Pod is running (use: ```kubectl get pods```). 160 161 ```kubectl create -f storm-nimbus.json``` 162 163 ```kubectl create -f storm-nimbus-service.json``` 164 165 Make sure the Nimbus Pod is running. 166 167 ```kubectl create -f storm-worker-controller.yaml``` 168 169 170 <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> 171 []() 172 <!-- END MUNGE: GENERATED_ANALYTICS -->