github.com/deis/deis@v1.13.5-0.20170519182049-1d9e59fbdbfc/docs/contributing/hacking.rst (about)

     1  :description: How to hack on Deis including setup instructions
     2  
     3  .. _hacking:
     4  
     5  Development Environment
     6  =======================
     7  
     8  DigitalOcean_ is the recommended development environment for Deis project
     9  contributors. :ref:`Provision a new DO cluster <deis_on_digitalocean>` and then
    10  continue to follow the instructions below to get started hacking.
    11  
    12  .. _digitalocean_credit:
    13  
    14  .. important::
    15  
    16      Are you a new contributor to Deis? Your first `Pull Request`_ could earn you
    17      credit at DigitalOcean_! Submit your changes and then email
    18      deis@engineyard.com. When your PR is merged, the maintainer team will
    19      send you a DigitalOcean credit based on the value of your contribution.
    20  
    21  This document is for developers who are interested in working directly on the
    22  Deis codebase. In this guide, we walk you through the process of setting up
    23  a local development environment. While there are many ways to set up your
    24  Deis environment, this document covers a specific setup:
    25  
    26  - Developing on **Mac OSX** or **Linux**
    27  - Managing virtualization with **Vagrant/Virtualbox**
    28  - Hosting a docker registry with **docker-machine** (Mac)
    29  
    30  We try to make it simple to hack on Deis. However, there are necessarily several moving
    31  pieces and some setup required. We welcome any suggestions for automating or simplifying
    32  this process.
    33  
    34  If you're just getting into the Deis codebase, look for GitHub issues with the label
    35  `easy-fix`_. These are more straightforward or low-risk issues and are a great way to
    36  become more familiar with Deis.
    37  
    38  Prerequisites
    39  -------------
    40  
    41  You can develop on any supported platform including your laptop, cloud providers or
    42  on bare metal.  We strongly recommend a minimum 3-node cluster. We strongly
    43  suggest using Vagrant and VirtualBox for your virtualization layer during
    44  development.
    45  
    46  At a glance, you will need:
    47  
    48  - Python 2.7 or later (with ``pip``)
    49  - virtualenv (``sudo pip install virtualenv``)
    50  - Go 1.5 or later, with support for compiling to ``linux/amd64``
    51  - Godep (https://github.com/tools/godep)
    52  - VirtualBox latest
    53  - Vagrant 1.5 or later
    54  - On Mac, you will also want
    55    - Docker Machine (http://docs.docker.com/machine/install-machine/)
    56  
    57  Additionally, you may need:
    58  - shellcheck (https://github.com/koalaman/shellcheck)
    59  - golint (https://github.com/golang/lint)
    60  
    61  In most cases, you should simply install according to the instructions. There
    62  are a few special cases, though. We cover these below.
    63  
    64  Configuring Go
    65  ``````````````
    66  
    67  If your local workstation does not support the linux/amd64 target environment, you will
    68  have to install Go from source with cross-compile support for that environment. This is
    69  because some of the components are built on your local machine and then injected into a
    70  docker container.
    71  
    72  Homebrew users can just install with cross compiling support:
    73  
    74  .. code-block:: console
    75  
    76      $ brew install go --with-cc-common
    77  
    78  It is also straightforward to build Go from source:
    79  
    80  .. code-block:: console
    81  
    82      $ sudo su
    83      $ curl -sSL https://golang.org/dl/go1.5.src.tar.gz | tar -v -C /usr/local -xz
    84      $ cd /usr/local/go/src
    85      $ # compile Go for our default platform first, then add cross-compile support
    86      $ ./make.bash --no-clean
    87      $ GOOS=linux GOARCH=amd64 ./make.bash --no-clean
    88  
    89  Once you can compile to ``linux/amd64``, you should be able to compile Deis'
    90  components as normal.
    91  
    92  Configuring Docker Machine (Mac)
    93  ````````````````````````````````
    94  
    95  Deis needs a Docker registry running independently of the Deis cluster. On
    96  OS X, you will need Docker Machine (http://docs.docker.com/machine/install-machine/)
    97  to run the registry inside of a VirtualBox image.
    98  
    99  .. note::
   100  
   101      Previously, Deis used boot2docker to run the registry. However, Docker has
   102      deprecated boot2docker in favor of Docker Machine.
   103  
   104  Install Docker Machine according to the normal installation instructions. Then
   105  create a new image for hosting your Deis Docker registry:
   106  
   107  .. code-block:: console
   108  
   109      $ docker-machine create --driver virtualbox --virtualbox-disk-size=100000 \
   110      --engine-insecure-registry=192.168.0.0/16 deis-registry
   111  
   112  This will create a new virtual machine named `deis-registry` that will take
   113  up as much as 100,000 MB of disk space. Registries tend to be large, so
   114  allocating a big disk is a good idea.
   115  
   116  Once the deis-registry machine exists, source its values into your environment
   117  so your docker client knows how to use the new machine.
   118  
   119  .. code-block:: console
   120  
   121      $ eval "$(docker-machine env deis-registry)"
   122  
   123  .. note::
   124  
   125      Because the registry that we create will not have a valid SSL certificate,
   126      we run the local registry as an insecure (HTTP, not HTTPS) registry. Each
   127      time Docker Machine reboots, the registry will get a new IP address
   128      somewhere in the 192.168.0.0/16 range. We must declare that explicitly when
   129      configuring Docker Machine.
   130  
   131  At this point, our `deis-registry` VM can now serve as a registry for Deis'
   132  Docker images. Later we will return to this.
   133  
   134  Fork the Deis Repository
   135  ------------------------
   136  Once the prerequisites have been met, we can begin to work with Deis.
   137  
   138  To get Deis running for development, first `fork the Deis repository`_,
   139  then clone your fork of the repository. Since Deis is predominantly written
   140  in Go, the best place to put it is in ``$GOPATH/src/github.com/deis/``
   141  
   142  .. code-block:: console
   143  
   144      $ mkdir -p  $GOPATH/src/github.com/deis
   145      $ cd $GOPATH/src/github.com/deis
   146      $ git clone git@github.com:<username>/deis.git
   147      $ cd deis
   148  
   149  .. note::
   150  
   151      By checking out the forked copy into the namespace ``github.com/deis/deis``,
   152      we are tricking the Go toolchain into seeing our fork as the "official"
   153      Deis tree.
   154  
   155  If you are going to be issuing pull requests and working with official Deis
   156  repository, we suggest configuring Git accordingly. There are various strategies
   157  for doing this, but the `most common`_ is to add an ``upstream`` remote:
   158  
   159  .. code-block:: console
   160  
   161      $ git remote add upstream https://github.com/deis/deis.git
   162  
   163  For the sake of simplicity, you may want to point an environment variable to
   164  your Deis code:
   165  
   166  .. code-block:: console
   167  
   168      export DEIS=$GOPATH/src/github.com/deis/deis
   169  
   170  Throughout the rest of this document, ``$DEIS`` refers to that location.
   171  
   172  Alternative: Forking with a Pushurl
   173  ```````````````````````````````````
   174  A number of Deis developers prefer to pull directly from ``deis/deis``, but
   175  push to ``<username>/deis``. If that workflow suits you better, you can set it
   176  up this way:
   177  
   178  .. code-block:: console
   179  
   180      $ git clone git@github.com:deis/deis.git
   181      $ cd deis
   182      $ git config remote.origin.pushurl git@github.com:<username>/deis.git
   183  
   184  In this setup, fetching and pulling code will work directly with the upstream
   185  repository, while pushing code will send changes to your fork. This makes it
   186  easy to stay up to date, but also make changes and then issue pull requests.
   187  
   188  Build deisctl
   189  -------------
   190  
   191  ``deisctl`` is used for interacting with the Deis cluster. While you can use an
   192  existing ``deisctl`` binary, we recommend that developers build it from source.
   193  
   194  .. code-block:: console
   195  
   196    $ cd $DEIS/deisctl
   197    $ make build
   198    $ make install  # optionally
   199  
   200  This will build just the ``deisctl`` portion of Deis. Running ``make install`` will
   201  install the ``deisctl`` command in ``$GOPATH/bin/deisctl``.
   202  
   203  You can verify that ``deisctl`` is correctly built and installed by running
   204  ``deisctl -h``. That should print the help text and exit.
   205  
   206  Configure SSH Tunneling for Deisctl
   207  -----------------------------------
   208  
   209  To connect to the cluster using ``deisctl``, you must add the private key to ``ssh-agent``.
   210  For example, when using Vagrant:
   211  
   212  .. code-block:: console
   213  
   214      $ ssh-add ~/.vagrant.d/insecure_private_key
   215  
   216  Set ``DEISCTL_TUNNEL`` so the ``deisctl`` client on your workstation can connect to
   217  one of the hosts in your cluster:
   218  
   219  .. code-block:: console
   220  
   221      $ export DEISCTL_TUNNEL=172.17.8.100
   222  
   223  .. note::
   224  
   225    A number of times during this setup, tools will suggest that you export various
   226    environment variables. You may find it convenient to store these in your shell's
   227    RC file (`~/.bashrc` or `~/.zshrc`).
   228  
   229  Install the Deis Client
   230  -----------------------
   231  
   232  The ``deis`` client is also written in Go. Your Deis client should match your server's
   233  version. Like ``deisctl``, we recommend that developers build ``deis`` from source:
   234  
   235  .. code-block:: console
   236  
   237      $ cd $DEIS/client
   238      $ make build
   239      $ make install  # optionally
   240      $ ./deis
   241      Usage: deis <command> [<args>...]
   242  
   243  
   244  Start Up a Development Cluster
   245  ------------------------------
   246  
   247  Our host system is now configured for controlling a Deis cluster. The next
   248  thing to do is begin standing up a development cluster.
   249  
   250  When developing locally, we want deisctl to check our local unit files so that
   251  any changes are reflected in our Deis cluster. The easiest way to do this is
   252  to set an environment variable telling deisctl where to look. Assuming
   253  the variable ``$DEIS`` points to the location if the deis source code, we want
   254  something like this:
   255  
   256  .. code-block:: console
   257  
   258      export DEISCTL_UNITS=$DEIS/deisctl/units
   259  
   260  To start up and configure a local vagrant cluster for development, you can use
   261  the ``dev-cluster`` target.
   262  
   263  .. code-block:: console
   264  
   265      $ make dev-cluster
   266  
   267  This may take a while to run the first time. At the end of the process, you
   268  will be prompted to run ``deis start platform``. Hold off on that task for now.
   269  We will come back to it later.
   270  
   271  To verify that the cluster is running, you should be able to connect
   272  to the nodes on your Deis cluster:
   273  
   274  .. code-block:: console
   275  
   276      $ vagrant status
   277      Current machine states:
   278  
   279      deis-01               running (virtualbox)
   280      deis-02               running (virtualbox)
   281      deis-03               running (virtualbox)
   282  
   283      $ vagrant ssh deis-01
   284      Last login: Tue Jun  2 18:26:30 2015 from 10.0.2.2
   285       * *    *   *****    ddddd   eeeeeee iiiiiii   ssss
   286      *   *  * *  *   *     d   d   e    e    i     s    s
   287       * *  ***** *****     d    d  e         i    s
   288      *****  * *    *       d     d e         i     s
   289      *   * *   *  * *      d     d eee       i      sss
   290      *****  * *  *****     d     d e         i         s
   291        *   *****  * *      d    d  e         i          s
   292       * *  *   * *   *     d   d   e    e    i    s    s
   293      ***** *****  * *     ddddd   eeeeeee iiiiiii  ssss
   294  
   295      Welcome to Deis			Powered by CoreOS
   296  
   297  With a dev cluster now running, we are ready to set up a local Docker registry.
   298  
   299  Configure a Docker Registry
   300  ---------------------------
   301  
   302  The development workflow requires Docker Registry set at the ``DEV_REGISTRY``
   303  environment variable.  If you're developing locally you can use the ``dev-registry``
   304  target to spin up a quick, disposable registry inside a Docker container.
   305  The target ``dev-registry`` prints the registry's address and port when using ``docker-machine``;
   306  otherwise, use your host's IP address as returned by ``ifconfig`` with port 5000 for ``DEV_REGISTRY``.
   307  
   308  .. code-block:: console
   309  
   310      $ make dev-registry
   311  
   312      To configure the registry for local Deis development:
   313          export DEV_REGISTRY=192.168.59.103:5000
   314  
   315  It is important that you export the ``DEV_REGISTRY`` variable as instructed.
   316  
   317  If you are developing elsewhere, you must set up a registry yourself.
   318  Make sure it meets the following requirements:
   319  
   320   #. You can push Docker images from your workstation
   321   #. Hosts in the cluster can pull images with the same URL
   322  
   323  .. note::
   324  
   325      If the development registry is insecure and has an IP address in a range other than ``10.0.0.0/8``,
   326      ``172.16.0.0/12``, or ``192.168.0.0/16``, you'll have to modify ``contrib/coreos/user-data.example``
   327      and whitelist your development registry so the daemons can pull your custom components.
   328  
   329  Initial Platform Build
   330  ----------------------
   331  
   332  The full environment is prepared. You can now build Deis from source code and
   333  then run the platform.
   334  
   335  We'll do three steps together:
   336  
   337  - Build the source (``make build``)
   338  - Update our local cluster with a dev release (``make dev-release``)
   339  - Start the platform (``deisctl start platform``)
   340  
   341  Conveniently, we can accomplish all three in one step:
   342  
   343  .. code-block:: console
   344  
   345      $ make deploy
   346  
   347  
   348  Running ``deisctl list`` should display all of the services that your Deis
   349  cluster is currently running.
   350  
   351  You can now use your Deis cluster in all of the usual ways.
   352  
   353  At this point, you are running Deis from the code in your Git clone. But since
   354  rebuilding like this is time consuming, Deis has a simplified developer
   355  workflow more suited to daily development.
   356  
   357  Development Workflow
   358  --------------------
   359  
   360  Deis includes ``Makefile`` targets designed to simplify the development workflow.
   361  
   362  This workflow is typically:
   363  
   364    #. Update source code and commit your changes using ``git``
   365    #. Use ``make -C <component> build`` to build a new Docker image
   366    #. Use ``make -C <component> dev-release`` to push a snapshot release
   367    #. Use ``make -C <component> restart`` to restart the component
   368  
   369  This can be shortened to a one-liner using the ``deploy`` target:
   370  
   371  .. code-block:: console
   372  
   373      $ make -C controller deploy
   374  
   375  You can also use the same tasks on the root ``Makefile`` to operate on all
   376  components at once.  For example, ``make deploy`` will build, dev-release,
   377  and restart all components on the cluster.
   378  
   379  .. note::
   380  
   381     You can export the ``DEIS_STATELESS=True`` environment variable to skip all
   382     store components when using the root ``Makefile``. Useful when working
   383     on a stateless platform (:ref:`running-deis-without-ceph`).
   384  
   385  .. important::
   386  
   387     In order to cut a dev-release, you must commit changes using ``git`` to increment
   388     the SHA used when tagging Docker images
   389  
   390  Test Your Changes
   391  -----------------
   392  
   393  Deis ships with a comprehensive suite of automated tests, most written in Go.
   394  See :ref:`testing` for instructions on running the tests.
   395  
   396  Useful Commands
   397  ---------------
   398  
   399  Once your controller is running, here are some helpful commands.
   400  
   401  Tail Logs
   402  `````````
   403  
   404  .. code-block:: console
   405  
   406      $ deisctl journal controller
   407  
   408  Rebuild Services from Source
   409  ````````````````````````````
   410  
   411  .. code-block:: console
   412  
   413      $ make -C controller build push restart
   414  
   415  Restart Services
   416  ````````````````
   417  
   418  .. code-block:: console
   419  
   420      $ make -C controller restart
   421  
   422  Django Shell
   423  ````````````
   424  
   425  .. code-block:: console
   426  
   427      $ deisctl list             # determine which host runs the controller
   428      $ ssh core@<host>          # SSH into the controller host
   429      $ nse deis-controller      # inject yourself into the container
   430      $ cd /app                  # change into the django project root
   431      $ ./manage.py shell        # get a django shell
   432  
   433  Have commands other Deis developers might find useful? Send us a PR!
   434  
   435  Pull Requests
   436  -------------
   437  
   438  Please read :ref:`standards`. It contains a checklist of things you should do
   439  when proposing a change to Deis.
   440  
   441  .. _DigitalOcean: https://www.digitalocean.com/
   442  .. _`Pull Request`: https://github.com/deis/deis/pulls
   443  .. _`easy-fix`: https://github.com/deis/deis/issues?labels=easy-fix&state=open
   444  .. _`deisctl`: https://github.com/deis/deis/tree/master/deisctl
   445  .. _`fork the Deis repository`: https://github.com/deis/deis/fork
   446  .. _`Python 2.7`: https://www.python.org/downloads/release/python-279/
   447  .. _`running the tests`: https://github.com/deis/deis/tree/master/tests#readme
   448  .. _`pull request`: https://github.com/deis/deis/pulls
   449  .. _`most common`: https://help.github.com/articles/fork-a-repo/