github.com/abayer/test-infra@v0.0.5/prow/cmd/jenkins-operator/README.md (about)

     1  # Jenkins operator
     2  
     3  `jenkins-operator` is a controller that enables Prow to use Jenkins
     4  as a backend for running jobs. 
     5  
     6  ## Jenkins configuration
     7  
     8  A Jenkins master needs to be provided via `--jenkins-url` in order for
     9  the operator to make requests to. By default, `--dry-run` is set to `true`
    10  so the operator will not make any mutating requests to Jenkins, Github,
    11  and Kubernetes, but you most probably want to set it to `false`.
    12  The Jenkins operator expects to read the Prow configuration by default
    13  in `/etc/config/config.yaml` which can be configured with `--config-path`.
    14  
    15  The following stanza is config that can be optionally set in the Prow config file:
    16  
    17  ```yaml
    18  jenkins_operators:
    19  - max_concurrency: 150
    20    max_goroutines: 20
    21    allow_cancellations: true
    22    job_url_template: 'https://storage-for-logs/{{if eq .Spec.Type "presubmit"}}pr-logs/pull{{else if eq .Spec.Type "batch"}}pr-logs/pull{{else}}logs{{end}}{{if ne .Spec.Refs.Repo "origin"}}/{{.Spec.Refs.Org}}_{{.Spec.Refs.Repo}}{{end}}{{if eq .Spec.Type "presubmit"}}/{{with index .Spec.Refs.Pulls 0}}{{.Number}}{{end}}{{else if eq .Spec.Type "batch"}}/batch{{end}}/{{.Spec.Job}}/{{.Status.BuildID}}/'
    23    report_template: '[Full PR test history](https://pr-history/{{if ne .Spec.Refs.Repo "origin"}}{{.Spec.Refs.Org}}_{{.Spec.Refs.Repo}}/{{end}}{{with index .Spec.Refs.Pulls 0}}{{.Number}}{{end}}).'
    24  ```
    25  
    26  * `max_concurrency` is the maximum number of Jenkins builds that can
    27  run in parallel, otherwise the operator is not going to start new builds.
    28  Defaults to 0, which means no limit.
    29  * `max_goroutines` is the maximum number of goroutines that the operator
    30  will spin up to handle all Jenkins builds. Defaulted to 20.
    31  * `allow_cancellations` allows canceling Jenkins builds for presubmit
    32  jobs that have been superseded by jobs for newer commits. By default,
    33  this is set to `false`.
    34  * `job_url_template` is a Golang-templated URL that shows up in the Details
    35  button next to the Github job status context. A ProwJob is provided as input
    36  to the template.
    37  * `report_template` is a Golang-templated message that shows up in Github in
    38  case of a job failure. A ProwJob is provided as input to the template.
    39  
    40  ### Security
    41  
    42  Various flavors of authentication are supported:
    43  * basic auth, using `--jenkins-user` and `--jenkins-token-file`.
    44  * OpenShift bearer token auth, using `--jenkins-bearer-token-file`.
    45  * certificate-based auth, using `--cert-file`, `--key-file`, and
    46  optionally `--ca-cert-file`.
    47  
    48  Basic auth and bearer token are mutually exclusive options whereas
    49  cert-based auth is complementary to both of them.
    50  
    51  If [CSRF protection](https://wiki.jenkins.io/display/JENKINS/CSRF+Protection) is enabled in Jenkins, `--csrf-protect=true`
    52  needs to be used on the operator's side to allow Prow to work correctly.
    53  
    54  ### Logs
    55  
    56  Apart from a controller, the Jenkins operator also runs a http server
    57  to serve Jenkins logs. You can configure the Prow frontend to show
    58  Jenkins logs with the following Prow config:
    59  ```yaml
    60  deck:
    61    external_agent_logs:
    62    - agent: jenkins
    63      url_template: 'http://jenkins-operator/job/{{.Spec.Job}}/{{.Status.BuildID}}/consoleText'
    64  ```
    65  
    66  Deck uses `url_template` to contact jenkins-operator when a user
    67  clicks the `Build log` button of a Jenkins job (`agent: jenkins`).
    68  `jenkins-operator` forwards the request to Jenkins and serves back
    69  the response.
    70  
    71  ## Job configuration
    72  
    73  Below follows the Prow configuration for a Jenkins job:
    74  ```yaml
    75  presubmits:
    76    org/repo:
    77    - name: pull-request-unit
    78      agent: jenkins
    79      always_run: true
    80      context: ci/prow/unit
    81      rerun_command: "/test unit"
    82      trigger: "((?m)^/test( all| unit),?(\\s+|$))"
    83  ```
    84  
    85  You can read more about the different types of Prow jobs [elsewhere](https://github.com/kubernetes/test-infra/tree/master/prow#how-to-add-new-jobs).
    86  What is interesting for us here is the `agent` field which needs to
    87  be set to `jenkins` in order for jobs to be dispatched to Jenkins and
    88  `name` which is the name of the job inside Jenkins.
    89  
    90  ## Sharding
    91  
    92  Sharding of Jenkins jobs is supported via Kubernetes labels and label
    93  selectors. This enables Prow to work with multiple Jenkins masters.
    94  Three places need to be configured in order to use sharding:
    95  * `--label-selector` in the Jenkins operator.
    96  * `label_selector` in `jenkins_operators` in the Prow config.
    97  * `labels` in the job config. 
    98  
    99  For example, one would set the following options:
   100  * `--label-selector=master=jenkins-master` in a Jenkins operator.
   101  
   102  This option forces the operator to list all ProwJobs with `master=jenkins-master`.
   103  
   104  * `label_selector: master=jenkins-master` in the Prow config.
   105  ```yaml
   106  jenkins_operators:
   107  - label_selector: master=jenkins-master
   108    max_concurrency: 150
   109    max_goroutines: 20
   110    allow_cancellations: true
   111  ```
   112  
   113  `jenkins_operators` in the Prow config can be read by multiple running operators
   114  and based on `label_selector`, each operator knows which config stanza does it
   115  need to use. Thus, `--label-selector` and `label_selector` need to match exactly.
   116  
   117  * `labels: jenkins-master` in the job config.
   118  
   119  ```yaml
   120  presubmits:
   121    org/repo:
   122    - name: pull-request-unit
   123      agent: jenkins
   124      labels:
   125        master: jenkins-master
   126      always_run: true
   127      context: ci/prow/unit
   128      rerun_command: "/test unit"
   129      trigger: "((?m)^/test( all| unit),?(\\s+|$))"
   130  ```
   131  
   132  Labels in the job config are set in ProwJobs during their creation.
   133  
   134  ## Kubernetes client
   135  
   136  The Jenkins operator acts as a Kubernetes client since it manages ProwJobs
   137  backed by Jenkins builds. It is expected to run as a pod inside a Kubernetes
   138  cluster and so it uses the in-cluster client config.
   139  
   140  ## Github integration
   141  
   142  The operator needs to talk to Github for updating commit statuses and
   143  adding comments about failed tests. Note that this functionality may
   144  potentially move into its own service, then the Jenkins operator will
   145  not need to contact the Github API. The required options are already
   146  defaulted:
   147  * `github-token-file` set to `/etc/github/oauth`. This is the Github bot
   148  oauth token that is used for updating job statuses and adding comments
   149  in Github.
   150  * `github-endpoint` set to `https://api.github.com`.
   151  
   152  
   153  ## Prometheus support
   154  
   155  The following Prometheus metrics are exposed by the operator:
   156  
   157  * `jenkins_requests` is the number of Jenkins requests made.
   158    - `verb` is the type of request (`GET`, `POST`)
   159    - `handler` is the path of the request, usually containing a
   160     job name (eg. `job/test-pull-request-unit`).
   161    - `code` is the status code of the request (`200`, `404`, etc.).
   162  * `jenkins_request_retries` is the number of Jenkins request
   163  retries made.
   164  * `jenkins_request_latency` is the time for a request to roundtrip
   165  between the operator and Jenkins.
   166  * `resync_period_seconds` is the time the operator takes to complete
   167  one reconciliation loop.
   168  * `prowjobs` is the number of Jenkins prowjobs in the system.
   169    - `job_name` is the name of the job.
   170    - `type` is the type of the prowjob: presubmit, postsubmit, periodic, batch
   171    - `state` is the state of the prowjob: triggered, pending, success, failure, aborted, error
   172  
   173  If a push gateway needs to be used it can be configured in the Prow config:
   174  ```yaml
   175  push_gateway:
   176    endpoint: http://prometheus-push-gateway
   177    interval: 1m
   178  ```