github.com/abayer/test-infra@v0.0.5/prow/cmd/jenkins-operator/README.md (about) 1 # Jenkins operator 2 3 `jenkins-operator` is a controller that enables Prow to use Jenkins 4 as a backend for running jobs. 5 6 ## Jenkins configuration 7 8 A Jenkins master needs to be provided via `--jenkins-url` in order for 9 the operator to make requests to. By default, `--dry-run` is set to `true` 10 so the operator will not make any mutating requests to Jenkins, Github, 11 and Kubernetes, but you most probably want to set it to `false`. 12 The Jenkins operator expects to read the Prow configuration by default 13 in `/etc/config/config.yaml` which can be configured with `--config-path`. 14 15 The following stanza is config that can be optionally set in the Prow config file: 16 17 ```yaml 18 jenkins_operators: 19 - max_concurrency: 150 20 max_goroutines: 20 21 allow_cancellations: true 22 job_url_template: 'https://storage-for-logs/{{if eq .Spec.Type "presubmit"}}pr-logs/pull{{else if eq .Spec.Type "batch"}}pr-logs/pull{{else}}logs{{end}}{{if ne .Spec.Refs.Repo "origin"}}/{{.Spec.Refs.Org}}_{{.Spec.Refs.Repo}}{{end}}{{if eq .Spec.Type "presubmit"}}/{{with index .Spec.Refs.Pulls 0}}{{.Number}}{{end}}{{else if eq .Spec.Type "batch"}}/batch{{end}}/{{.Spec.Job}}/{{.Status.BuildID}}/' 23 report_template: '[Full PR test history](https://pr-history/{{if ne .Spec.Refs.Repo "origin"}}{{.Spec.Refs.Org}}_{{.Spec.Refs.Repo}}/{{end}}{{with index .Spec.Refs.Pulls 0}}{{.Number}}{{end}}).' 24 ``` 25 26 * `max_concurrency` is the maximum number of Jenkins builds that can 27 run in parallel, otherwise the operator is not going to start new builds. 28 Defaults to 0, which means no limit. 29 * `max_goroutines` is the maximum number of goroutines that the operator 30 will spin up to handle all Jenkins builds. Defaulted to 20. 31 * `allow_cancellations` allows canceling Jenkins builds for presubmit 32 jobs that have been superseded by jobs for newer commits. By default, 33 this is set to `false`. 34 * `job_url_template` is a Golang-templated URL that shows up in the Details 35 button next to the Github job status context. A ProwJob is provided as input 36 to the template. 37 * `report_template` is a Golang-templated message that shows up in Github in 38 case of a job failure. A ProwJob is provided as input to the template. 39 40 ### Security 41 42 Various flavors of authentication are supported: 43 * basic auth, using `--jenkins-user` and `--jenkins-token-file`. 44 * OpenShift bearer token auth, using `--jenkins-bearer-token-file`. 45 * certificate-based auth, using `--cert-file`, `--key-file`, and 46 optionally `--ca-cert-file`. 47 48 Basic auth and bearer token are mutually exclusive options whereas 49 cert-based auth is complementary to both of them. 50 51 If [CSRF protection](https://wiki.jenkins.io/display/JENKINS/CSRF+Protection) is enabled in Jenkins, `--csrf-protect=true` 52 needs to be used on the operator's side to allow Prow to work correctly. 53 54 ### Logs 55 56 Apart from a controller, the Jenkins operator also runs a http server 57 to serve Jenkins logs. You can configure the Prow frontend to show 58 Jenkins logs with the following Prow config: 59 ```yaml 60 deck: 61 external_agent_logs: 62 - agent: jenkins 63 url_template: 'http://jenkins-operator/job/{{.Spec.Job}}/{{.Status.BuildID}}/consoleText' 64 ``` 65 66 Deck uses `url_template` to contact jenkins-operator when a user 67 clicks the `Build log` button of a Jenkins job (`agent: jenkins`). 68 `jenkins-operator` forwards the request to Jenkins and serves back 69 the response. 70 71 ## Job configuration 72 73 Below follows the Prow configuration for a Jenkins job: 74 ```yaml 75 presubmits: 76 org/repo: 77 - name: pull-request-unit 78 agent: jenkins 79 always_run: true 80 context: ci/prow/unit 81 rerun_command: "/test unit" 82 trigger: "((?m)^/test( all| unit),?(\\s+|$))" 83 ``` 84 85 You can read more about the different types of Prow jobs [elsewhere](https://github.com/kubernetes/test-infra/tree/master/prow#how-to-add-new-jobs). 86 What is interesting for us here is the `agent` field which needs to 87 be set to `jenkins` in order for jobs to be dispatched to Jenkins and 88 `name` which is the name of the job inside Jenkins. 89 90 ## Sharding 91 92 Sharding of Jenkins jobs is supported via Kubernetes labels and label 93 selectors. This enables Prow to work with multiple Jenkins masters. 94 Three places need to be configured in order to use sharding: 95 * `--label-selector` in the Jenkins operator. 96 * `label_selector` in `jenkins_operators` in the Prow config. 97 * `labels` in the job config. 98 99 For example, one would set the following options: 100 * `--label-selector=master=jenkins-master` in a Jenkins operator. 101 102 This option forces the operator to list all ProwJobs with `master=jenkins-master`. 103 104 * `label_selector: master=jenkins-master` in the Prow config. 105 ```yaml 106 jenkins_operators: 107 - label_selector: master=jenkins-master 108 max_concurrency: 150 109 max_goroutines: 20 110 allow_cancellations: true 111 ``` 112 113 `jenkins_operators` in the Prow config can be read by multiple running operators 114 and based on `label_selector`, each operator knows which config stanza does it 115 need to use. Thus, `--label-selector` and `label_selector` need to match exactly. 116 117 * `labels: jenkins-master` in the job config. 118 119 ```yaml 120 presubmits: 121 org/repo: 122 - name: pull-request-unit 123 agent: jenkins 124 labels: 125 master: jenkins-master 126 always_run: true 127 context: ci/prow/unit 128 rerun_command: "/test unit" 129 trigger: "((?m)^/test( all| unit),?(\\s+|$))" 130 ``` 131 132 Labels in the job config are set in ProwJobs during their creation. 133 134 ## Kubernetes client 135 136 The Jenkins operator acts as a Kubernetes client since it manages ProwJobs 137 backed by Jenkins builds. It is expected to run as a pod inside a Kubernetes 138 cluster and so it uses the in-cluster client config. 139 140 ## Github integration 141 142 The operator needs to talk to Github for updating commit statuses and 143 adding comments about failed tests. Note that this functionality may 144 potentially move into its own service, then the Jenkins operator will 145 not need to contact the Github API. The required options are already 146 defaulted: 147 * `github-token-file` set to `/etc/github/oauth`. This is the Github bot 148 oauth token that is used for updating job statuses and adding comments 149 in Github. 150 * `github-endpoint` set to `https://api.github.com`. 151 152 153 ## Prometheus support 154 155 The following Prometheus metrics are exposed by the operator: 156 157 * `jenkins_requests` is the number of Jenkins requests made. 158 - `verb` is the type of request (`GET`, `POST`) 159 - `handler` is the path of the request, usually containing a 160 job name (eg. `job/test-pull-request-unit`). 161 - `code` is the status code of the request (`200`, `404`, etc.). 162 * `jenkins_request_retries` is the number of Jenkins request 163 retries made. 164 * `jenkins_request_latency` is the time for a request to roundtrip 165 between the operator and Jenkins. 166 * `resync_period_seconds` is the time the operator takes to complete 167 one reconciliation loop. 168 * `prowjobs` is the number of Jenkins prowjobs in the system. 169 - `job_name` is the name of the job. 170 - `type` is the type of the prowjob: presubmit, postsubmit, periodic, batch 171 - `state` is the state of the prowjob: triggered, pending, success, failure, aborted, error 172 173 If a push gateway needs to be used it can be configured in the Prow config: 174 ```yaml 175 push_gateway: 176 endpoint: http://prometheus-push-gateway 177 interval: 1m 178 ```