github.com/adecaro/fabric-ca@v2.0.0-alpha+incompatible/docs/source/ca-ci.md (about) 1 # Continuous Integration Process 2 3 This document explains the fabric-ca Jenkins pipeline flow and FAQ's on the build process 4 to help developer to get more familiarize with the process flow. 5 6 We use Jenkins as a CI tool and to manage jobs, we use [JJB](https://docs.openstack.org/infra/jenkins-job-builder). 7 Please see the pipeline job configuration template here https://ci-docs.readthedocs.io/en/latest/source/pipeline_jobs.html#job-templates. 8 9 ## CI Pipeline flow 10 11 - Every Gerrit patchset triggers a verify job with the Gerrit Refspec on the parent commit 12 of the patchset and run the below tests from the `Jenkinsfile`. Note: When you are ready 13 to merge a patchset, it's always a best practice to rebase the patchset on the latest commit. 14 15 - Basic Checks (make checks) 16 - Documentation build (tox -edocs) 17 - Unit tests (make unit-tests) 18 - FVT tests (make fvt-tests) 19 - E2E tests (Only after patchset is merged) 20 21 All the above tests run on the Hyperledger infarstructure x86_64 build nodes. All these nodes 22 uses the packer with pre-configured software packages. This helps us to run the tests in much 23 faster than installing required packages everytime. 24 25 Below steps shows what each stage does in the Jenkins pipeline verify and merge flow. 26 Every Gerrit patchset triggers the fabric-ca-verify-x86_64 job and runs the below tests on 27 x86_64 platform. Before execute the below tests, it clean the environment (Deletes the left 28 over build artifiacts) and clone the repository with the Gerrit Refspec. Based on the file 29 extenstions, Jenkins pipeline script triggers the stages. If the patchset contains specific 30 doc extension, it only triggers **Docs Build** Stage otherwise it triggers all stages. 31 32 ![](images/pipeline_flow.png) 33 34 #### Basic Checks 35 36 - We run `make checks` target to run the basic checks before kickoff the actual tests. 37 - It's run against every Patchset. 38 - You can run basic checks locally: 39 - make checks (Runs all check conditions (license, format, imports, lint and vet) 40 41 #### Docs Build 42 43 - We run `tox -edocs` from the root directory. 44 - Displays the output in the form of HTML Publisher on the `fabric-ca-verify-x86_64` job. 45 Click on **Docs Output** link on the Jenkins console. 46 47 #### Unit Tests 48 49 - We run `make unit-test` target to run the go based unit-tests and publish the coverage 50 report on the Jenkins console. Click on **Coverage Report** link on the Jenkins console 51 to see the code coverage. 52 53 #### FVT Tests 54 55 - We run `make fvt-tests` target to run fvt tests, which includes tests that performs end-to-end 56 test scenarios with PosgreSQL and MySQL databases. These tests include database migration, 57 backwards compatibility, and LDAP integration. https://github.com/hyperledger/fabric-ca/blob/master/scripts/fvt/README.md 58 59 #### E2E tests 60 61 - We run **e2e tests** in the **merge job** and it performs the following tests. The intention 62 of running e2e tests as part of the merge job is to test the dependent tests of fabric-ca. 63 - fabcar 64 - fabric-sdk-node - We run **gulp run-end-to-end** target which executes most of the end to end 65 tests of fabric-sdk-node which are depend on fabric-ca. 66 - fabric-sdk-java - We run **ci_run.sh** script which is pre-baked in fabric-sdk-java repository. 67 68 As we trigger `fabric-ca-verify-x86_64` and `fabric-ca-merge-x86_64` pipeline jobs for every gerrit patchset, 69 we execute these tests in the below order. 70 71 After the DocsBuild stage is passed, Jenkins Pipeline triggers the Unit and FVT Tests parallel 72 on two different nodes. After the tests are executed successfully it posts a Gerrit voting 73 on the patchset. If DocsBuild stage fails, it send the result back to Gerrit patchset and it 74 won't trigger further builds. 75 76 See below **FAQ's** for more information on the pipeline process. 77 78 ## FAQ's 79 80 #### What happens on the merge job? 81 82 After the patchset got merged in the fabric-ca repository, it follows the above pipeline flow and 83 executes the e2e tests in parallel to the Unit and FVT Tests. 84 85 **Merge Pipeline Flow** 86 87 ``` 88 CleanEnvironment -- OutputEnvironment -- CloneRefSpec -- BasicChecks -- DocsBuild - Tests (E2E, Unit, FVT Tests) 89 ``` 90 91 Jenkins clones the latest merged commit and executes the below steps 92 93 - Build fabric, fabric-ca images & binaries 94 - Pull Thirdparty Images from DockerHub (Couchdb, zookeeper, kafka) 95 - Pull javaenv, nodeenv images from nexus3 (latest stable images published after successful merge job of each repo) 96 - Tests 97 - fabcar tests 98 - fabric-sdk-node (npm install, gulp run-end-to-end) 99 - fabric-sdk-java (Run ci_run.sh) 100 101 #### What happens if one of the build stage fails? 102 103 As we are running these tests in `fastFailure: true` (if any build stage fails in the parallel 104 process, it will terminate/abort the current running tests and sends the result back to the 105 Gerrit Patchset. This way, CI will avoid runnning tests when there is a failure in one of the 106 parallel build stage. 107 108 It shows `aborted` on the aborted stage on pipeline staged view. 109 110 #### How to re-trigger failed tests? 111 112 With this pipeline flow, you can NOT re-trigger the specific stage, but you can post comments 113 `reverify` or `reverify-x` on the gerrit patchset to trigger the `fabric-ca-verify-x86_64` 114 job which triggers the pipeline flow as mentioned above. Also, we provided `remerge` or `remerge-x` 115 comment phrases to re-trigger the failed merge jobs. 116 117 #### Where should I see the output of the stages? 118 119 Piepline supports two views (staged and blueocean). **Staged views** shows on the Jenkins job main 120 page and it shows each stage in order and the status. For better view, we suggest you to access 121 BlueOcean plugin. Click on the JOB Number and click on the **Open Blue Ocean** link that shows 122 the build stages in pipeline view. 123 124 #### How to add more stages to this pipeline flow? 125 126 We use scripted pipeline syntax with groovy and shell scripts. Also, we use global shared library 127 scripts which are placed in https://github.com/hyperledger/ci-management/tree/master/vars. 128 Try to leverage these common functions in your code. All you have to do is, undestand the pipeline 129 flow of the tests and conditions, add more stages as mentioned in the existing Jenkinsfile. 130 131 #### How will I get build failure notifications? 132 133 On every merge failure, we send build failure email notications to the submitter of the patchset 134 and send build details to the Rocket Chat **jenkins-robot** channel. Check the result here 135 https://chat.hyperledger.org/channel/jenkins-robot 136 137 #### What steps I have to modify when I create a new branch from master? 138 139 As the Jenkinsfile is completely parametrzed, you no need to modify anything in the Jenkinsfile 140 but you may endup modifying **ci.properties** file with the appropirate Base Versions, 141 Baseimage versions etc... in the new branch. We suggest you to modify this file immediately 142 after you create a new branch to avoid running tests on old versions. 143 144 #### What are the supported platforms 145 146 - x86_64 (Run the tests on verify and merge job) 147 - s390x (Run the same tests as part of the daily job) 148 149 #### Where can I see the Build Scripts. 150 151 We use global shared library scripts and Jenkinsfile along with the build file. 152 153 Global Shared Library - https://github.com/hyperledger/ci-management/tree/master/vars 154 155 Jenkinsfile - https://github.com/hyperledger/fabric-ca/tree/master/Jenkinsfile 156 157 ci.properties - https://github.com/hyperledger/fabric-ca/tree/master/ci.properties 158 (ci.properties is the only file you have to modify with the values requried for the specific branch.) 159 160 Packer Scripts - https://github.com/hyperledger/ci-management/blob/master/packer/provision/docker.sh 161 (Packer is a tool for automatically creating VM and container images, configuring them and 162 post-processing them into standard output formats. We build Hyperledger's CI images via Packer 163 and attach them to x86_64 build nodes. On s390x, we install manually. See the packages we 164 install as a pre-requisite in the CI x86 build nodes.) 165 166 #### How to reach out to CI team? 167 168 Post your questions or feedback in https://chat.hyperledger.org/channel/ci-pipeline or https://chat.hyperledger.org/channel/fabric-ci Rocket Chat channels. 169 You can also create a JIRA task or bug in FABCI project. https://jira.hyperledger.org/projects/FABCI