github.com/abayer/test-infra@v0.0.5/mungegithub/README.md (about)

     1  # Deprecation warning
     2  
     3  *Mungegithub is a deprecated system. Please do not build any new infrastructure that needs to run on it. Any new GitHub interactions should be implemented as prow plugins, and any new cron jobs should be implemented as Kubernetes CronJobs.*
     4  
     5  ## Overview
     6  
     7  Mungegithub provides a number of tools intended to automate github processes. While mainly for the kubernetes community, some thought is put into making it generic. Mungegithub is built as a single binary, but is run in 2 different ways for 2 different purposes.
     8  
     9  1. submit-queue: This looks at open PRs and attempts to help automate the process getting PRs from open to merged.
    10  1. cherrypick: This looks at open and closed PRs with the `cherry-pick-candidate` label and attempts to help the branch managers deal with the cherry-pick process.
    11  
    12  One can see the specifics of how the `submit-queue` and `cherrypick` options are executed by looking at the deployment definition in their respective subdirectories.
    13  
    14  One may also look in the `example-one-off` directory for a small skeleton program which prints the number of all open PRs. It is an excellent place to start if you need to write a 'one-off' automation across a large number of PRs.
    15  
    16  ## Building and running
    17  
    18  Executing `make help` inside the `mungegithub` directory should inform you about the functions provided by the Makefile. A common pattern when developing is to run something like:
    19  ```sh
    20  make mungegithub && ./mungegithub --dry-run --token-file=/path/to/github/oauth/token --www=submit-queue/www --pr-mungers=submit-queue --min-pr-number=25000 --max-pr-number=25500 --organization=kubernetes --project=kubernetes --repo-dir=/tmp --stderrthreshold=0  --once=true
    21  ```
    22  
    23  A Github oauth token is required, even in readonly/test mode. For production, we use a token with write access to the repo in question. https://help.github.com/articles/creating-an-access-token-for-command-line-use/ discusses the procedure to get a personal oauth token. These tokens will need to be loaded into kubernetes `secret`s for use by the submit and/or cherry-pick queue. It is extremely easy to use up the 5,000 API calls per hour so the production token should not be re-used for tests.
    24  
    25  After successfully running the local binary one may build, test, and deploy in readonly mode to a real kube cluster. To do so, one must make sure that one's kubeconfig file is set up for the test/read-only cluster by running any necessary `kubectl config` commands by hand. One may also need a container repository with read & write access. It is just as easy to create a new dockerhub account, and a public repository named `submit-queue` within that. We will refer to the repository as `docker.io/$USERNAME` where `$USERNAME` is a placeholder. The steps required to deploy on a real cluster for the submit-queue application are as follows. The instructions to run the cherrypick application are along the same lines. Below, we explain the steps to run on the kubernetes main repository. Running on other repositories is similar, except that the corresponding YAML files are in a directory for that repository.
    26  
    27  First store your personal access token in a plain text file named (token) in the mungegithub directory. If the app you are deploying uses github webhooks, store the github secret key in file named (hook-secret) in the mungegithub directory just like the access token.
    28  
    29  Also create a persistent disk named `<reponame>-cache` on the mungegithub cluster for every repo that a submit-queue will target.
    30  
    31  If you are deploying for the first time to a new cluster run `APP=submit-queue TARGET=<reponame> make first_time_deploy` to create and upload a persistent volume and persistent volume claim, a new service, a new `<reponame>-github-token` secret, a configmap, and the actual deployment. If you are updating a deployment or if setup is partially complete, see the step-by-step instructions for deployment below.
    32  
    33  - Run `APP=submit-queue TARGET=<reponame> make push_secret` to generate a local.secret.yaml and load it to the cluster.
    34  - Run `APP=submit-queue TARGET=<reponame> make volume` to create a persistent volume and persistent volume claim. (If you are running a local cluster, and not on GCP, apply `mungegithub/submit-queue/pv-local.yaml` to create a persistent volume on your host then create the persistent volume claim from `mungegithub/submit-queue/deployment/<reponame>/pvc.yaml`. The pv-local.yaml file may need to be modified to match the expected name of the persistent volume by the deployment).
    35  - Check that the persistent volume claim is bound by checking `kubectl --kubeconfig=... get pvc`.
    36  
    37  After these steps, we may need to push a configmap, in case any of the commandline arguments were changed. Pushing a new configmap for the kubernetes repository looks like the following:
    38  ```sh
    39  TARGET=kubernetes APP=submit-queue KUBECONFIG=/path/to/kubeconfig make push_config
    40  ```
    41  
    42  It can finally deployed as:
    43  ```sh
    44  TARGET=kubernetes REPO=docker.io/$USERNAME APP=submit-queue KUBECONFIG=/path/to/kubeconfig make deploy
    45  ```
    46  
    47  **Note** The submit-queue should always be deployed from master so that looking at
    48  the master branch (configmap, etc) reflects what is currently running. Make sure you have appropriate version of all dependencies by running `godep restore`. Refer [this](https://github.com/tools/godep#restore) for more information.
    49  
    50  After this has successfully deployed to the test cluster in read-only mode, running in production involves running any required `kubectl config` commands to point to the production cluster, pushing a configmap if necessary, and then running:
    51  ```sh
    52  TARGET=kubernetes REPO=docker.io/$USERNAME APP=submit-queue KUBECONFIG=/path/to/kubeconfig READONLY=false make deploy
    53  ```
    54  
    55  ## About the mungers
    56  
    57  A small amount of information about some of the individual mungers inside each of the 3 varieties are listed below:
    58  
    59  ### submit-queue
    60  * block-paths - add `do-not-merge` label to PRs which change files which should not be changed (mainly old docs moved to [kubernetes/website](https://github.com/kubernetes/website))
    61  * blunderbuss - assigned PRs to individuals based on the contents of OWNERS files in the main repo
    62  * cherrypick-auto-approve - adds `cherrypick-approved` to PRs in a release branch if the 'parent' pr in master was approved
    63  * cherrypick-label-unapproved - adds `do-not-merge` label to PRs against a release-\* branch which do not have `cherrypick-approved`
    64  * comment-deleter - deletes comments created by the k8s-merge-robot which are no longer relevant. Such as comments about a rebase being required if it has been rebased.
    65  * comment-deleter-jenkins - deleted comments create by the k8s-bot jenkins bot which are no longer relevant. Such as old test results.
    66  * issue-triager - takes the title and body of an issue and asks another web
    67    service to guess the appropriate routing label
    68  * lgtm-after-commit - removes `lgtm` label if a PR is changed after the label was added
    69  * needs-rebase - adds and removes a `needs-rebase` label if a PR needs to be rebased before it can be applied.
    70  * path-label - adds labels, such as `kind/new-api` based on if ANY file which matches changed
    71  * size - Adds the xs/s/m/l/xl labels and comments to PRs
    72  * stale-green-ci - Reruns the CI tests every X hours (96?) for PRs which passed. So PRs which sit around for a long time will notice failures sooner.
    73  * stale-pending-ci - Reruns the CI tests if they have been 'in progress'/'pending' for 24 hours.
    74  * submit-queue - This is the brains that actually tracks and merges PRs. It also provides the web site interface.
    75  
    76  ### cherrypick
    77  * cherrypick-clear-after-merge - This watches for PRs against release branches which merged and removes the `cherrypick-candidate` label from the PR on master.
    78  * cherrypick-must-have-milestone - This complains on any PR against a release branch which does not have a vX.Y milestone.
    79  * cherrypick-queue - This is the web display of all PRs with the `cherrypick-candidate` label which a branch owner is likely to want to pay attention to.
    80  
    81  ### Instructions on running mungegithub locally with your own repository		
    82  
    83  Sometimes we may want to run QA tests locally using the mungegithub binary. The steps to do this are as follows.		
    84  
    85  * `cd` to the contrib/mungegithub directory.		
    86  * Run `go build` to compile the mungegithub binary.		
    87  * Running the binary is as simple as running `./mungegithub` and supplying the appropriate flags.		
    88  * The flags that are essential are as follows:		
    89      * `--pr-mungers`, `--organization`, `--project` are required flags. Based on the mungers specified in pr-mungers, other flags may be required.
    90      * `--token` or `--token-file` are needed. It is highly recommended that you provide a GitHub access token without write access to the repositories you are running on, as an extra measure of safety.
    91      * The `--dry-run=true` flag must be specified to ensure you're not posting comments accidentally.		
    92      * The `--repo-dir` should be pointed to /tmp if required.		
    93      * The `--www=submit-queue/www/` will start up the http server if specified with the submit-queue munger, and serve on localhost:8080.
    94  
    95  ### Instructions on turning up a new submit-queue instance.
    96  
    97  The steps below make use of the mungegithub cluster which runs the existing submit-queues.
    98  
    99  * Create a new directory for the repo on which you want to run the submit-queue instance. For example, if we want to call it `<TARGET>`, we create `contrib/submit-queue/deployments/<TARGET>`.
   100  * Add a service.yaml, pv.yaml, pvc.yaml, secret.yaml, configmap.yaml to the directory and configure them appropriately.
   101       * The configmap’s name must be `<TARGET>-sq-flags`.
   102       * The target-repo must be changed to `<TARGET>`.
   103       * The PV and PVC must be named `<TARGET>-cache`.
   104       * The secret must be named `<TARGET>-github-token`.
   105       * The service must be named `<TARGET>-sq-status`.
   106  * Create a persistent disk named `<TARGET>-cache` on the mungegithub cluster. It is typically 10G in size.
   107  * Switch context with kubectl to point to the mungegithub cluster.
   108  * Create the PV and PVC resources. After creation, the PV and PVC should be bound.
   109  * Create and load a new token secret using the below command, which uses an API token stored in `./token`, and generates a `local.secret.yaml` file.
   110  ```
   111  APP=submit-queue TARGET=<TARGET> make push_secret
   112  ```
   113  * A kubernetes secret named `<TARGET>-github-secret` may be required if the submit-queue is configured to accept github webhooks.  If this is the case then create it by putting the webhook secret into the file `./hook-secret` before using the `push_secret` make target.
   114  * Create the service which is of type NodePort using `APP=submit-queue TARGET=<TARGET> make push_service`.
   115  * Finally, update the ingress.yaml with the new URL and the new service to point to.
   116  * Apply changes to the running ingress instance.