github.com/argoproj/argo-cd/v2@v2.10.9/docs/developer-guide/toolchain-guide.md (about)

     1  # Development toolchain
     2  
     3  ## Preface
     4  
     5  !!!note "Before you start"
     6      The Argo CD project continuously grows, both in terms of features and community size. It gets adopted by more and more organisations which entrust Argo CD to handle their critical production workloads. Thus, we need to take great care with any changes that affect compatibility, performance, scalability, stability and security of Argo CD. For this reason, every new feature or larger enhancement must be properly designed and discussed before it gets accepted into the code base.
     7  
     8      We do welcome and encourage everyone to participate in the Argo CD project, but please understand that we can't accept each and every contribution from the community, for various reasons. If you want to submit code for a great new feature or enhancement, we kindly ask you to take a look at the
     9      [code contribution guide](code-contributions.md#) before you start to write code or submit a PR.
    10  
    11  We want to make contributing to Argo CD as simple and smooth as possible.
    12  
    13  This guide shall help you in setting up your build & test environment, so that you can start developing and testing bug fixes and feature enhancements without having to make too much effort in setting up a local toolchain.
    14  
    15  If you want to submit a PR, please read this document carefully, as it contains important information guiding you through our PR quality gates.
    16  
    17  As is the case with the development process, this document is under constant change. If you notice any error, or if you think this document is out-of-date, or if you think it is missing something: Feel free to submit a PR or submit a bug to our GitHub issue tracker.
    18  
    19  If you need guidance with submitting a PR, or have any other questions regarding development of Argo CD, do not hesitate to [join our Slack](https://argoproj.github.io/community/join-slack) and get in touch with us in the `#argo-contributors` channel!
    20  
    21  ## Before you start
    22  
    23  You will need at least the following things in your toolchain in order to develop and test Argo CD locally:
    24  
    25  * A Kubernetes cluster. You won't need a fully blown multi-master, multi-node cluster, but you will need something like K3S, Minikube or microk8s. You will also need a working Kubernetes client (`kubectl`) configuration in your development environment. The configuration must reside in `~/.kube/config` and the API server URL must point to the IP address of your local machine (or VM), and **not** to `localhost` or `127.0.0.1` if you are using the virtualized development toolchain (see below)
    26  
    27  * You will also need a working Docker runtime environment, to be able to build and run images. The Docker version must be 17.05.0 or higher, to support multi-stage builds. 
    28  
    29  * Obviously, you will need a `git` client for pulling source code and pushing back your changes.
    30  
    31  * Last but not least, you will need a Go SDK and related tools (such as GNU `make`) installed and working on your development environment. The minimum required Go version for building and testing Argo CD is **v1.17**.
    32  
    33  * We will assume that your Go workspace is at `~/go`.
    34  
    35  !!! note
    36      **Attention minikube users**: By default, minikube will create Kubernetes client configuration that uses authentication data from files. This is incompatible with the virtualized toolchain. So if you intend to use the virtualized toolchain, you have to embed this authentication data into the client configuration. To do so, start minikube using `minikube start --embed-certs`. Please also note that minikube using the Docker driver is currently not supported with the virtualized toolchain, because the Docker driver exposes the API server on 127.0.0.1 hard-coded. If in doubt, run `make verify-kube-connect` to find out.
    37  
    38  ## Submitting PRs
    39  
    40  ### Continuous Integration process
    41  
    42  When you submit a PR against Argo CD's GitHub repository, a couple of CI checks will be run automatically to ensure your changes will build fine and meet certain quality standards. Your contribution needs to pass those checks in order to be merged into the repository.
    43  
    44  !!!note
    45  
    46      Please make sure that you always create PRs from a branch that is up-to-date with the latest changes from Argo CD's master branch. Depending on how long it takes for the maintainers to review and merge your PR, it might be necessary to pull in latest changes into your branch again.
    47  
    48  Please understand that we, as an Open Source project, have limited capacities for reviewing and merging PRs to Argo CD. We will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected.
    49  
    50  The following read will help you to submit a PR that meets the standards of our CI tests:
    51  
    52  ### Title of the PR
    53  
    54  Please use a meaningful and concise title for your PR. This will help us to pick PRs for review quickly, and the PR title will also end up in the Changelog.
    55  
    56  We use [PR title checker](https://github.com/marketplace/actions/pr-title-checker) to categorize your PR into one of the following categories:
    57  
    58  * `fix` - Your PR contains one or more code bug fixes
    59  * `feat` - Your PR contains a new feature
    60  * `docs` - Your PR improves the documentation
    61  * `chore` - Your PR improves any internals of Argo CD, such as the build process, unit tests, etc
    62  
    63  Please prefix the title of your PR with one of the valid categories. For example, if you chose the title your PR `Add documentation for GitHub SSO integration`, please use `docs: Add documentation for GitHub SSO integration` instead.
    64  
    65  ### PR template checklist
    66  
    67  Upon opening a PR, the details will contain a checklist from a template. Please read the checklist, and tick those marks that apply to you.
    68  
    69  ### Automated builds & tests
    70  
    71  After you have submitted your PR, and whenever you push new commits to that branch, GitHub will run a number of Continuous Integration checks against your code. It will execute the following actions, and each of them has to pass:
    72  
    73  * Build the Go code (`make build`)
    74  * Generate API glue code and manifests (`make codegen`)
    75  * Run a Go linter on the code (`make lint`)
    76  * Run the unit tests (`make test`)
    77  * Run the End-to-End tests (`make test-e2e`)
    78  * Build and lint the UI code (`make lint-ui`)
    79  * Build the `argocd` CLI (`make cli`)
    80  
    81  If any of these tests in the CI pipeline fail, it means that some of your contribution is considered faulty (or a test might be flaky, see below).
    82  
    83  ### Code test coverage
    84  
    85  We use [CodeCov](https://codecov.io) in our CI pipeline to check for test coverage, and once you submit your PR, it will run and report on the coverage difference as a comment within your PR. If the difference is too high in the negative, i.e. your submission introduced a significant drop in code coverage, the CI check will fail.
    86  
    87  Whenever you develop a new feature or submit a bug fix, please also write appropriate unit tests for it. If you write a completely new module, please aim for at least 80% of coverage.
    88  If you want to see how much coverage just a specific module (i.e. your new one) has, you can set the `TEST_MODULE` to the (fully qualified) name of that module with `make test`, i.e.:
    89  
    90  ```bash
    91   make test TEST_MODULE=github.com/argoproj/argo-cd/server/cache
    92  ...
    93  ok      github.com/argoproj/argo-cd/server/cache        0.029s  coverage: 89.3% of statements
    94  ```
    95  
    96  ## Local vs Virtualized toolchain
    97  
    98  Argo CD provides a fully virtualized development and testing toolchain using Docker images. It is recommended to use those images, as they provide the same runtime environment as the final product and it is much easier to keep up-to-date with changes to the toolchain and dependencies. But as using Docker comes with a slight performance penalty, you might want to setup a local toolchain.
    99  
   100  Most relevant targets for the build & test cycles in the `Makefile` provide two variants, one of them suffixed with `-local`. For example, `make test` will run unit tests in the Docker container, `make test-local` will run it natively on your local system.
   101  
   102  If you are going to use the virtualized toolchain, please bear in mind the following things:
   103  
   104  * Your Kubernetes API server must listen on the interface of your local machine or VM, and not on `127.0.0.1` only.
   105  * Your Kubernetes client configuration (`~/.kube/config`) must not use an API URL that points to `localhost` or `127.0.0.1`.
   106  
   107  You can test whether the virtualized toolchain has access to your Kubernetes cluster by running `make verify-kube-connect` (*after* you have setup your development environment, as described below), which will run `kubectl version` inside the Docker container used for running all tests.
   108  
   109  The Docker container for the virtualized toolchain will use the following local mounts from your workstation, and possibly modify its contents:
   110  
   111  * `~/go/src` - Your Go workspace's source directory (modifications expected)
   112  * `~/.cache/go-build` - Your Go build cache (modifications expected)
   113  * `~/.kube` - Your Kubernetes client configuration (no modifications)
   114  * `/tmp` - Your system's temp directory (modifications expected)
   115  
   116  ## Setting up your development environment
   117  
   118  The following steps are required no matter whether you chose to use a virtualized or a local toolchain.
   119  
   120  !!!note "Docker privileges"
   121      If you opt in to use the virtualized toolchain, you will need to have the
   122      appropriate privileges to interact with the Docker daemon. It is not
   123      recommended to work as the root user, and if your user does not have the
   124      permissions to talk to the Docker user, but you have `sudo` setup on your
   125      system, you can set the environment variable `SUDO` to `sudo` in order to
   126      have the build scripts make any calls to the `docker` CLI using sudo,
   127      without affecting the other parts of the build scripts (which should be
   128      executed with your normal user privileges).
   129  
   130      You can either set this before calling `make`, like so for example:
   131  
   132      ```
   133      SUDO=sudo make sometarget
   134      ```
   135  
   136      Or you can opt to export this permanently to your environment, for example
   137      ```
   138      export SUDO=sudo
   139      ```
   140  
   141  ### Clone the Argo CD repository from your personal fork on GitHub
   142  
   143  * `mkdir -p ~/go/src/github.com/argoproj`
   144  * `cd ~/go/src/github.com/argoproj`
   145  * `git clone https://github.com/yourghuser/argo-cd`
   146  * `cd argo-cd`
   147  
   148  ### Optional: Setup an additional Git remote
   149  
   150  While everyone has their own Git workflow, the author of this document recommends to create a remote called `upstream` in your local copy pointing to the original Argo CD repository. This way, you can easily keep your local branches up-to-date by merging in latest changes from the Argo CD repository, i.e. by doing a `git pull upstream master` in your locally checked out branch. To create the remote, run `git remote add upstream https://github.com/argoproj/argo-cd`
   151  
   152  ### Install the must-have requirements
   153  
   154  Make sure you fulfill the pre-requisites above and run some preliminary tests. Neither of them should report an error.
   155  
   156  * Run `kubectl version`
   157  * Run `docker version`
   158  * Run `go version`
   159  
   160  ### Build the required Docker image
   161  
   162  Build the required Docker image by running `make test-tools-image`. This image offers the environment of the virtualized toolchain.
   163  
   164  The `Dockerfile` used to build these images can be found at `test/container/Dockerfile`.
   165  
   166  ### Test connection from build container to your K8s cluster
   167  
   168  Run `make verify-kube-connect`, it should execute without error.
   169  
   170  If you receive an error similar to the following:
   171  
   172  ```
   173  The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
   174  make: *** [Makefile:386: verify-kube-connect] Error 1
   175  ```
   176  
   177  you should edit your `~/.kube/config` and modify the `server` option to point to your correct K8s API (as described above).
   178  
   179  ### Using k3d
   180  
   181  [k3d](https://github.com/rancher/k3d) is a lightweight wrapper to run [k3s](https://github.com/rancher/k3s), a minimal Kubernetes distribution, in docker. Because it's running in a docker container, you're dealing with docker's internal networking rules when using k3d. A typical Kubernetes cluster running on your local machine is part of the same network that you're on, so you can access it using **kubectl**. However, a Kubernetes cluster running within a docker container (in this case, the one launched by make) cannot access 0.0.0.0 from inside the container itself, when 0.0.0.0 is a network resource outside the container itself (and/or the container's network). This is the cost of a fully self-contained, disposable Kubernetes cluster. The following steps should help with a successful `make verify-kube-connect` execution.
   182  
   183  1. Find your host IP by executing `ifconfig` on Mac/Linux and `ipconfig` on Windows. For most users, the following command works to find the IP address.
   184  
   185      * For Mac:
   186  
   187      ```
   188      IP=`ifconfig en0 | grep inet | grep -v inet6 | awk '{print $2}'`
   189      echo $IP
   190      ```
   191  
   192      * For Linux:
   193  
   194      ```
   195      IP=`ifconfig eth0 | grep inet | grep -v inet6 | awk '{print $2}'`
   196      echo $IP
   197      ```
   198  
   199      Keep in mind that this IP is dynamically assigned by the router so if your router restarts for any reason, your IP might change.
   200  
   201  2. Edit your ~/.kube/config and replace 0.0.0.0 with the above IP address.
   202  
   203  3. Execute a `kubectl version` to make sure you can still connect to the Kubernetes API server via this new IP. Run `make verify-kube-connect` and check if it works.
   204  
   205  4. Finally, so that you don't have to keep updating your kube-config whenever you spin up a new k3d cluster, add `--api-port $IP:6550` to your **k3d cluster create** command, where $IP is the value from step 1. An example command is provided here:
   206  
   207  ```
   208  k3d cluster create my-cluster --wait --k3s-arg '--disable=traefik@server:*' --api-port $IP:6550 -p 443:443@loadbalancer
   209  ```
   210  
   211  !!!note
   212  For k3d versions less than v5.0.0, the example command flags `--k3s-arg` and `'--disable=traefik@server:*'` should change to `--k3s-server-arg` and `'--disable=traefik'`, respectively.
   213  
   214  ## The development cycle
   215  
   216  When you have developed and possibly manually tested the code you want to contribute, you should ensure that everything will build correctly. Commit your changes to the local copy of your Git branch and perform the following steps:
   217  
   218  ### Pull in all build dependencies
   219  
   220  As build dependencies change over time, you have to synchronize your development environment with the current specification. In order to pull in all required dependencies, issue:
   221  
   222  * `make dep-ui`
   223  
   224  Argo CD recently migrated to Go modules. Usually, dependencies will be downloaded on build time, but the Makefile provides two targets to download and vendor all dependencies:
   225  
   226  * `make mod-download` will download all required Go modules and
   227  * `make mod-vendor` will vendor those dependencies into the Argo CD source tree
   228  
   229  ### Generate API glue code and other assets
   230  
   231  Argo CD relies on Google's [Protocol Buffers](https://developers.google.com/protocol-buffers) for its API, and this makes heavy use of auto-generated glue code and stubs. Whenever you touched parts of the API code, you must re-generate the auto generated code.
   232  
   233  * Run `make codegen`, this might take a while
   234  * Check if something has changed by running `git status` or `git diff`
   235  * Commit any possible changes to your local Git branch, an appropriate commit message would be `Changes from codegen`, for example.
   236  
   237  !!!note
   238      There are a few non-obvious assets that are auto-generated. You should not change the autogenerated assets, as they will be overwritten by a subsequent run of `make codegen`. Instead, change their source files. Prominent examples of non-obvious auto-generated code are `swagger.json` or the installation manifest YAMLs.
   239  
   240  ### Build your code and run unit tests
   241  
   242  After the code glue has been generated, your code should build and the unit tests should run without any errors. Execute the following statements:
   243  
   244  * `make build`
   245  * `make test`
   246  
   247  These steps are non-modifying, so there's no need to check for changes afterwards.
   248  
   249  ### Lint your code base
   250  
   251  In order to keep a consistent code style in our source tree, your code must be well-formed in accordance to some widely accepted rules, which are applied by a Linter.
   252  
   253  The Linter might make some automatic changes to your code, such as indentation fixes. Some other errors reported by the Linter have to be fixed manually.
   254  
   255  * Run `make lint` and observe any errors reported by the Linter
   256  * Fix any of the errors reported and commit to your local branch
   257  * Finally, after the Linter reports no errors anymore, run `git status` or `git diff` to check for any changes made automatically by Lint
   258  * If there were automatic changes, commit them to your local branch
   259  
   260  If you touched UI code, you should also run the Yarn linter on it:
   261  
   262  * Run `make lint-ui`
   263  * Fix any of the errors reported by it
   264  
   265  ## Contributing to Argo CD UI
   266  
   267  Argo CD, along with Argo Workflows, uses shared React components from [Argo UI](https://github.com/argoproj/argo-ui). Examples of some of these components include buttons, containers, form controls, 
   268  and others. Although you can make changes to these files and run them locally, in order to have these changes added to the Argo CD repo, you will need to follow these steps. 
   269  
   270  1. Fork and clone the [Argo UI repository](https://github.com/argoproj/argo-ui).
   271  
   272  2. `cd` into your `argo-ui` directory, and then run `yarn install`. 
   273  
   274  3. Make your file changes.
   275  
   276  4. Run `yarn start` to start a [storybook](https://storybook.js.org/) dev server and view the components in your browser. Make sure all your changes work as expected. 
   277  
   278  5. Use [yarn link](https://classic.yarnpkg.com/en/docs/cli/link/) to link Argo UI package to your Argo CD repository. (Commands below assume that `argo-ui` and `argo-cd` are both located within the same parent folder)
   279  
   280      * `cd argo-ui`
   281      * `yarn link`
   282      * `cd ../argo-cd/ui`
   283      * `yarn link argo-ui`
   284  
   285      Once `argo-ui` package has been successfully linked, test out changes in your local development environment. 
   286  
   287  6. Commit changes and open a PR to [Argo UI](https://github.com/argoproj/argo-ui). 
   288  
   289  7. Once your PR has been merged in Argo UI, `cd` into your `argo-cd/ui` folder and run `yarn add git+https://github.com/argoproj/argo-ui.git`. This will update the commit SHA in the `ui/yarn.lock` file to use the lastest master commit for argo-ui. 
   290  
   291  8. Submit changes to `ui/yarn.lock`in a PR to Argo CD. 
   292  
   293  ## Setting up a local toolchain
   294  
   295  For development, you can either use the fully virtualized toolchain provided as Docker images, or you can set up the toolchain on your local development machine. Due to the dynamic nature of requirements, you might want to stay with the virtualized environment.
   296  
   297  ### Install required dependencies and build-tools
   298  
   299  !!!note
   300      The installations instructions are valid for Linux hosts only. Mac instructions will follow shortly.
   301  
   302  For installing the tools required to build and test Argo CD on your local system, we provide convenient installer scripts. By default, they will install binaries to `/usr/local/bin` on your system, which might require `root` privileges.
   303  
   304  You can change the target location by setting the `BIN` environment before running the installer scripts. For example, you can install the binaries into `~/go/bin` (which should then be the first component in your `PATH` environment, i.e. `export PATH=~/go/bin:$PATH`):
   305  
   306  ```shell
   307  make BIN=~/go/bin install-tools-local
   308  ```
   309  
   310  Additionally, you have to install at least the following tools via your OS's package manager (this list might not be always up-to-date):
   311  
   312  * Git LFS plugin
   313  * GnuPG version 2
   314  
   315  ### Install Go dependencies
   316  
   317  You need to pull in all required Go dependencies. To do so, run
   318  
   319  * `make mod-download-local`
   320  * `make mod-vendor-local`
   321  
   322  ### Test your build toolchain
   323  
   324  The first thing you can do to test whether your build toolchain is setup correctly is by generating the glue code for the API and after that, run a normal build:
   325  
   326  * `make codegen-local`
   327  * `make build-local`
   328  
   329  This should return without any error.
   330  
   331  ### Run unit-tests
   332  
   333  The next thing is to make sure that unit tests are running correctly on your system. These will require that all dependencies, such as Helm, Kustomize, Git, GnuPG, etc are correctly installed and fully functioning:
   334  
   335  * `make test-local`
   336  
   337  ### Run end-to-end tests
   338  
   339  The final step is running the End-to-End testsuite, which makes sure that your Kubernetes dependencies are working properly. This will involve starting all the Argo CD components locally on your computer. The end-to-end tests consists of two parts: a server component, and a client component.
   340  
   341  * First, start the End-to-End server: `make start-e2e-local`. This will spawn a number of processes and services on your system.
   342  * When all components have started, run `make test-e2e-local` to run the end-to-end tests against your local services.
   343  
   344  For more information about End-to-End tests, refer to the [End-to-End test documentation](test-e2e.md).