github.com/deis/deis@v1.13.5-0.20170519182049-1d9e59fbdbfc/docs/contributing/hacking.rst (about) 1 :description: How to hack on Deis including setup instructions 2 3 .. _hacking: 4 5 Development Environment 6 ======================= 7 8 DigitalOcean_ is the recommended development environment for Deis project 9 contributors. :ref:`Provision a new DO cluster <deis_on_digitalocean>` and then 10 continue to follow the instructions below to get started hacking. 11 12 .. _digitalocean_credit: 13 14 .. important:: 15 16 Are you a new contributor to Deis? Your first `Pull Request`_ could earn you 17 credit at DigitalOcean_! Submit your changes and then email 18 deis@engineyard.com. When your PR is merged, the maintainer team will 19 send you a DigitalOcean credit based on the value of your contribution. 20 21 This document is for developers who are interested in working directly on the 22 Deis codebase. In this guide, we walk you through the process of setting up 23 a local development environment. While there are many ways to set up your 24 Deis environment, this document covers a specific setup: 25 26 - Developing on **Mac OSX** or **Linux** 27 - Managing virtualization with **Vagrant/Virtualbox** 28 - Hosting a docker registry with **docker-machine** (Mac) 29 30 We try to make it simple to hack on Deis. However, there are necessarily several moving 31 pieces and some setup required. We welcome any suggestions for automating or simplifying 32 this process. 33 34 If you're just getting into the Deis codebase, look for GitHub issues with the label 35 `easy-fix`_. These are more straightforward or low-risk issues and are a great way to 36 become more familiar with Deis. 37 38 Prerequisites 39 ------------- 40 41 You can develop on any supported platform including your laptop, cloud providers or 42 on bare metal. We strongly recommend a minimum 3-node cluster. We strongly 43 suggest using Vagrant and VirtualBox for your virtualization layer during 44 development. 45 46 At a glance, you will need: 47 48 - Python 2.7 or later (with ``pip``) 49 - virtualenv (``sudo pip install virtualenv``) 50 - Go 1.5 or later, with support for compiling to ``linux/amd64`` 51 - Godep (https://github.com/tools/godep) 52 - VirtualBox latest 53 - Vagrant 1.5 or later 54 - On Mac, you will also want 55 - Docker Machine (http://docs.docker.com/machine/install-machine/) 56 57 Additionally, you may need: 58 - shellcheck (https://github.com/koalaman/shellcheck) 59 - golint (https://github.com/golang/lint) 60 61 In most cases, you should simply install according to the instructions. There 62 are a few special cases, though. We cover these below. 63 64 Configuring Go 65 `````````````` 66 67 If your local workstation does not support the linux/amd64 target environment, you will 68 have to install Go from source with cross-compile support for that environment. This is 69 because some of the components are built on your local machine and then injected into a 70 docker container. 71 72 Homebrew users can just install with cross compiling support: 73 74 .. code-block:: console 75 76 $ brew install go --with-cc-common 77 78 It is also straightforward to build Go from source: 79 80 .. code-block:: console 81 82 $ sudo su 83 $ curl -sSL https://golang.org/dl/go1.5.src.tar.gz | tar -v -C /usr/local -xz 84 $ cd /usr/local/go/src 85 $ # compile Go for our default platform first, then add cross-compile support 86 $ ./make.bash --no-clean 87 $ GOOS=linux GOARCH=amd64 ./make.bash --no-clean 88 89 Once you can compile to ``linux/amd64``, you should be able to compile Deis' 90 components as normal. 91 92 Configuring Docker Machine (Mac) 93 ```````````````````````````````` 94 95 Deis needs a Docker registry running independently of the Deis cluster. On 96 OS X, you will need Docker Machine (http://docs.docker.com/machine/install-machine/) 97 to run the registry inside of a VirtualBox image. 98 99 .. note:: 100 101 Previously, Deis used boot2docker to run the registry. However, Docker has 102 deprecated boot2docker in favor of Docker Machine. 103 104 Install Docker Machine according to the normal installation instructions. Then 105 create a new image for hosting your Deis Docker registry: 106 107 .. code-block:: console 108 109 $ docker-machine create --driver virtualbox --virtualbox-disk-size=100000 \ 110 --engine-insecure-registry=192.168.0.0/16 deis-registry 111 112 This will create a new virtual machine named `deis-registry` that will take 113 up as much as 100,000 MB of disk space. Registries tend to be large, so 114 allocating a big disk is a good idea. 115 116 Once the deis-registry machine exists, source its values into your environment 117 so your docker client knows how to use the new machine. 118 119 .. code-block:: console 120 121 $ eval "$(docker-machine env deis-registry)" 122 123 .. note:: 124 125 Because the registry that we create will not have a valid SSL certificate, 126 we run the local registry as an insecure (HTTP, not HTTPS) registry. Each 127 time Docker Machine reboots, the registry will get a new IP address 128 somewhere in the 192.168.0.0/16 range. We must declare that explicitly when 129 configuring Docker Machine. 130 131 At this point, our `deis-registry` VM can now serve as a registry for Deis' 132 Docker images. Later we will return to this. 133 134 Fork the Deis Repository 135 ------------------------ 136 Once the prerequisites have been met, we can begin to work with Deis. 137 138 To get Deis running for development, first `fork the Deis repository`_, 139 then clone your fork of the repository. Since Deis is predominantly written 140 in Go, the best place to put it is in ``$GOPATH/src/github.com/deis/`` 141 142 .. code-block:: console 143 144 $ mkdir -p $GOPATH/src/github.com/deis 145 $ cd $GOPATH/src/github.com/deis 146 $ git clone git@github.com:<username>/deis.git 147 $ cd deis 148 149 .. note:: 150 151 By checking out the forked copy into the namespace ``github.com/deis/deis``, 152 we are tricking the Go toolchain into seeing our fork as the "official" 153 Deis tree. 154 155 If you are going to be issuing pull requests and working with official Deis 156 repository, we suggest configuring Git accordingly. There are various strategies 157 for doing this, but the `most common`_ is to add an ``upstream`` remote: 158 159 .. code-block:: console 160 161 $ git remote add upstream https://github.com/deis/deis.git 162 163 For the sake of simplicity, you may want to point an environment variable to 164 your Deis code: 165 166 .. code-block:: console 167 168 export DEIS=$GOPATH/src/github.com/deis/deis 169 170 Throughout the rest of this document, ``$DEIS`` refers to that location. 171 172 Alternative: Forking with a Pushurl 173 ``````````````````````````````````` 174 A number of Deis developers prefer to pull directly from ``deis/deis``, but 175 push to ``<username>/deis``. If that workflow suits you better, you can set it 176 up this way: 177 178 .. code-block:: console 179 180 $ git clone git@github.com:deis/deis.git 181 $ cd deis 182 $ git config remote.origin.pushurl git@github.com:<username>/deis.git 183 184 In this setup, fetching and pulling code will work directly with the upstream 185 repository, while pushing code will send changes to your fork. This makes it 186 easy to stay up to date, but also make changes and then issue pull requests. 187 188 Build deisctl 189 ------------- 190 191 ``deisctl`` is used for interacting with the Deis cluster. While you can use an 192 existing ``deisctl`` binary, we recommend that developers build it from source. 193 194 .. code-block:: console 195 196 $ cd $DEIS/deisctl 197 $ make build 198 $ make install # optionally 199 200 This will build just the ``deisctl`` portion of Deis. Running ``make install`` will 201 install the ``deisctl`` command in ``$GOPATH/bin/deisctl``. 202 203 You can verify that ``deisctl`` is correctly built and installed by running 204 ``deisctl -h``. That should print the help text and exit. 205 206 Configure SSH Tunneling for Deisctl 207 ----------------------------------- 208 209 To connect to the cluster using ``deisctl``, you must add the private key to ``ssh-agent``. 210 For example, when using Vagrant: 211 212 .. code-block:: console 213 214 $ ssh-add ~/.vagrant.d/insecure_private_key 215 216 Set ``DEISCTL_TUNNEL`` so the ``deisctl`` client on your workstation can connect to 217 one of the hosts in your cluster: 218 219 .. code-block:: console 220 221 $ export DEISCTL_TUNNEL=172.17.8.100 222 223 .. note:: 224 225 A number of times during this setup, tools will suggest that you export various 226 environment variables. You may find it convenient to store these in your shell's 227 RC file (`~/.bashrc` or `~/.zshrc`). 228 229 Install the Deis Client 230 ----------------------- 231 232 The ``deis`` client is also written in Go. Your Deis client should match your server's 233 version. Like ``deisctl``, we recommend that developers build ``deis`` from source: 234 235 .. code-block:: console 236 237 $ cd $DEIS/client 238 $ make build 239 $ make install # optionally 240 $ ./deis 241 Usage: deis <command> [<args>...] 242 243 244 Start Up a Development Cluster 245 ------------------------------ 246 247 Our host system is now configured for controlling a Deis cluster. The next 248 thing to do is begin standing up a development cluster. 249 250 When developing locally, we want deisctl to check our local unit files so that 251 any changes are reflected in our Deis cluster. The easiest way to do this is 252 to set an environment variable telling deisctl where to look. Assuming 253 the variable ``$DEIS`` points to the location if the deis source code, we want 254 something like this: 255 256 .. code-block:: console 257 258 export DEISCTL_UNITS=$DEIS/deisctl/units 259 260 To start up and configure a local vagrant cluster for development, you can use 261 the ``dev-cluster`` target. 262 263 .. code-block:: console 264 265 $ make dev-cluster 266 267 This may take a while to run the first time. At the end of the process, you 268 will be prompted to run ``deis start platform``. Hold off on that task for now. 269 We will come back to it later. 270 271 To verify that the cluster is running, you should be able to connect 272 to the nodes on your Deis cluster: 273 274 .. code-block:: console 275 276 $ vagrant status 277 Current machine states: 278 279 deis-01 running (virtualbox) 280 deis-02 running (virtualbox) 281 deis-03 running (virtualbox) 282 283 $ vagrant ssh deis-01 284 Last login: Tue Jun 2 18:26:30 2015 from 10.0.2.2 285 * * * ***** ddddd eeeeeee iiiiiii ssss 286 * * * * * * d d e e i s s 287 * * ***** ***** d d e i s 288 ***** * * * d d e i s 289 * * * * * * d d eee i sss 290 ***** * * ***** d d e i s 291 * ***** * * d d e i s 292 * * * * * * d d e e i s s 293 ***** ***** * * ddddd eeeeeee iiiiiii ssss 294 295 Welcome to Deis Powered by CoreOS 296 297 With a dev cluster now running, we are ready to set up a local Docker registry. 298 299 Configure a Docker Registry 300 --------------------------- 301 302 The development workflow requires Docker Registry set at the ``DEV_REGISTRY`` 303 environment variable. If you're developing locally you can use the ``dev-registry`` 304 target to spin up a quick, disposable registry inside a Docker container. 305 The target ``dev-registry`` prints the registry's address and port when using ``docker-machine``; 306 otherwise, use your host's IP address as returned by ``ifconfig`` with port 5000 for ``DEV_REGISTRY``. 307 308 .. code-block:: console 309 310 $ make dev-registry 311 312 To configure the registry for local Deis development: 313 export DEV_REGISTRY=192.168.59.103:5000 314 315 It is important that you export the ``DEV_REGISTRY`` variable as instructed. 316 317 If you are developing elsewhere, you must set up a registry yourself. 318 Make sure it meets the following requirements: 319 320 #. You can push Docker images from your workstation 321 #. Hosts in the cluster can pull images with the same URL 322 323 .. note:: 324 325 If the development registry is insecure and has an IP address in a range other than ``10.0.0.0/8``, 326 ``172.16.0.0/12``, or ``192.168.0.0/16``, you'll have to modify ``contrib/coreos/user-data.example`` 327 and whitelist your development registry so the daemons can pull your custom components. 328 329 Initial Platform Build 330 ---------------------- 331 332 The full environment is prepared. You can now build Deis from source code and 333 then run the platform. 334 335 We'll do three steps together: 336 337 - Build the source (``make build``) 338 - Update our local cluster with a dev release (``make dev-release``) 339 - Start the platform (``deisctl start platform``) 340 341 Conveniently, we can accomplish all three in one step: 342 343 .. code-block:: console 344 345 $ make deploy 346 347 348 Running ``deisctl list`` should display all of the services that your Deis 349 cluster is currently running. 350 351 You can now use your Deis cluster in all of the usual ways. 352 353 At this point, you are running Deis from the code in your Git clone. But since 354 rebuilding like this is time consuming, Deis has a simplified developer 355 workflow more suited to daily development. 356 357 Development Workflow 358 -------------------- 359 360 Deis includes ``Makefile`` targets designed to simplify the development workflow. 361 362 This workflow is typically: 363 364 #. Update source code and commit your changes using ``git`` 365 #. Use ``make -C <component> build`` to build a new Docker image 366 #. Use ``make -C <component> dev-release`` to push a snapshot release 367 #. Use ``make -C <component> restart`` to restart the component 368 369 This can be shortened to a one-liner using the ``deploy`` target: 370 371 .. code-block:: console 372 373 $ make -C controller deploy 374 375 You can also use the same tasks on the root ``Makefile`` to operate on all 376 components at once. For example, ``make deploy`` will build, dev-release, 377 and restart all components on the cluster. 378 379 .. note:: 380 381 You can export the ``DEIS_STATELESS=True`` environment variable to skip all 382 store components when using the root ``Makefile``. Useful when working 383 on a stateless platform (:ref:`running-deis-without-ceph`). 384 385 .. important:: 386 387 In order to cut a dev-release, you must commit changes using ``git`` to increment 388 the SHA used when tagging Docker images 389 390 Test Your Changes 391 ----------------- 392 393 Deis ships with a comprehensive suite of automated tests, most written in Go. 394 See :ref:`testing` for instructions on running the tests. 395 396 Useful Commands 397 --------------- 398 399 Once your controller is running, here are some helpful commands. 400 401 Tail Logs 402 ````````` 403 404 .. code-block:: console 405 406 $ deisctl journal controller 407 408 Rebuild Services from Source 409 ```````````````````````````` 410 411 .. code-block:: console 412 413 $ make -C controller build push restart 414 415 Restart Services 416 ```````````````` 417 418 .. code-block:: console 419 420 $ make -C controller restart 421 422 Django Shell 423 ```````````` 424 425 .. code-block:: console 426 427 $ deisctl list # determine which host runs the controller 428 $ ssh core@<host> # SSH into the controller host 429 $ nse deis-controller # inject yourself into the container 430 $ cd /app # change into the django project root 431 $ ./manage.py shell # get a django shell 432 433 Have commands other Deis developers might find useful? Send us a PR! 434 435 Pull Requests 436 ------------- 437 438 Please read :ref:`standards`. It contains a checklist of things you should do 439 when proposing a change to Deis. 440 441 .. _DigitalOcean: https://www.digitalocean.com/ 442 .. _`Pull Request`: https://github.com/deis/deis/pulls 443 .. _`easy-fix`: https://github.com/deis/deis/issues?labels=easy-fix&state=open 444 .. _`deisctl`: https://github.com/deis/deis/tree/master/deisctl 445 .. _`fork the Deis repository`: https://github.com/deis/deis/fork 446 .. _`Python 2.7`: https://www.python.org/downloads/release/python-279/ 447 .. _`running the tests`: https://github.com/deis/deis/tree/master/tests#readme 448 .. _`pull request`: https://github.com/deis/deis/pulls 449 .. _`most common`: https://help.github.com/articles/fork-a-repo/