github.com/blockchain-gm/fabric-ca@v0.0.0-20200423072702-b2c40c7ac69c/docs/source/ca-ci.md (about)

     1  # Continuous Integration Process
     2  
     3  This document explains the fabric-ca Jenkins pipeline flow and FAQ's on the build process
     4  to help developer to get more femilarize with the process flow.
     5  
     6  To manage CI jobs, we use [JJB](https://docs.openstack.org/infra/jenkins-job-builder).
     7  Please see the pipeline job configuration template here https://ci-docs.readthedocs.io/en/latest/source/pipeline_jobs.html#job-templates.
     8  
     9  ## CI Pipeline flow
    10  
    11  - Every Gerrit patchset triggers a verify job and run the below tests from the `Jenkinsfile`
    12  
    13      - Basic Checks (make checks)
    14      - Documentation build (tox -edocs)
    15      - Unit tests (make unit-tests)
    16      - FVT tests (make fvt-tests)
    17      - E2E tests
    18  
    19  All the above tests run on the Hyperledger infarstructure x86_64 build nodes. All these nodes
    20  uses the packer with pre-configured software packages. This helps us to run the tests in much
    21  faster than installing required packages everytime.
    22  
    23  Below steps shows what each stage does in the Jenkins pipeline verify and merge flow. Every
    24  Gerrit patchset triggers **fabric-ca-verify-x86_64** job and executes the below tests on x86_64
    25  platform. Before execute the below tests, it clean the environment (Deletes the left over build artifiacts)
    26  and clone the repository with the Gerrit Refspec. Based on the file extenstions, Jenkins
    27  pipeline script triggers the stages. If the patchset contains specific doc extension, it only
    28  triggers **Docs Build** stage otherwise it triggers all the stages.
    29  
    30  ![](images/pipeline_flow.png)
    31  
    32  #### Basic Checks
    33  
    34  - We run `make checks` target to run the basic checks before kickoff the actual tests.
    35  - It runs against every Patchset.
    36  - You can run basic checks locally:
    37      - make checks (Runs all check conditions (license, format, imports, lint and vet)
    38  
    39  #### Docs Build
    40  
    41  - This stage gets triggered only when a patchset contains .md, .rst etc doc related file exetensions.
    42  - We run `tox -edocs` from the root directory.
    43  - Displays the output in the form of HTML Publisher on the `fabric-ca-verify-x86_64` job.
    44    Click on **Docs Output** link on the Jenkins console.
    45  
    46  #### Unit Tests
    47  
    48  - We run `make unit-test` target to run the go based unit-tests and publish the coverage
    49    report on the Jenkins console. Click on **Coverage Report** link on the Jenkins console
    50    to see the code coverage.
    51  
    52  #### FVT Tests
    53  
    54  - We run `make fvt-tests` target to fun fvt tests, which includes tests that perform end-to-end
    55    test scenarios with PosgreSQL and MySQL databases. These tests include database migration,
    56    backwards compatibility, and LDAP integration. https://github.com/hyperledger/fabric-ca/blob/release-1.4/scripts/fvt/README.md
    57  
    58  #### E2E tests
    59  
    60  - We run **e2e tests** in the **merge job** and it performs the following tests. The intention of
    61    running e2e tests as part of the merge job is to test the dependent tests of fabric-ca.
    62          - fabcar
    63          - fabric-sdk-node - We run **gulp test** target which executes most of the end to end tests
    64            of fabric-sdk-node which are depend on fabric-ca.
    65          - fabric-sdk-java - We run **ci_run.sh** script which is pre-baked in fabric-sdk-java
    66            repository.
    67  
    68  As we trigger `fabric-ca-verify-x86_64` and `fabric-ca-merge-x86_64` pipeline jobs for every gerrit
    69  patchset, we execute these tests in the below order.
    70  
    71  After the DocsBuild stage is passed, Jenkins Pipeline triggers Unit and FVT Tests parallel on two different
    72  nodes. After the tests are executed successfully it posts a Gerrit voting on the patchset.
    73  If DocsBuild stage fails, it send the result back to Gerrit patchset and it won't trigger the further builds.
    74  
    75  See below **FAQ's** to contribute to CI changes.
    76  
    77  ## FAQ's
    78  
    79  #### What happens on the merge job?
    80  
    81  After the patchset got merged in the fabric-ca repository, it follows the above pipeline flow and
    82  executes the e2e tests in parallel to the Unit and FVT Tests.
    83  
    84  **Merge Pipeline Flow**
    85  
    86  ```
    87  CleanEnvironment -- OutputEnvironment -- CloneRefSpec -- BasicChecks -- DocsBuild - Tests (E2E, Unit, FVT Tests)
    88  ```
    89  
    90  Jenkins clones the latest merged commit and executes the below steps
    91  
    92  - Build fabric, fabric-ca Images & Binaries
    93  - Pull Thirdparty Images from DockerHub (Couchdb, zookeeper, kafka)
    94  - Pull javaenv, nodeenv images from nexus3 (latest stable images published after successful merge job of each repo)
    95  - Tests
    96    - fabcar tests
    97    - fabric-sdk-node (npm install, gulp test)
    98    - fabric-sdk-java (Run ci_run.sh)
    99  
   100  #### What happens if one of the build stage fails?
   101  
   102  As we are running these tests in `fastFailure: true` (if any build stage fails in the parallel
   103  process, it will terminate/abort the current running tests and sends the result back to the
   104  Gerrit Patchset. This way, CI will avoid runnning tests when there is a failure in one of the
   105  parallel build stage.
   106  
   107  It shows `aborted` on the aborted stage on pipeline staged view.
   108  
   109  #### How to re-trigger failed tests?
   110  
   111  With this pipeline flow, you can **NOT** re-trigger a specific stage, but you can post comments
   112  `reverify` or `reverify-x` on the gerrit patchset to trigger the `fabric-ca-verify-x86_64`
   113  job which triggers the pipeline flow as mentioned above. Also, we provided `remerge` or `remerge-x`
   114  comment phrases to re-trigger the failed merge jobs.
   115  
   116  #### Where to see the output of the stages?
   117  
   118  Piepline supports two views (staged and blueocean). **Staged views** shows on the Jenkins job
   119  main page and it shows each stage in order and the status. For better view, we suggest you
   120  to access BlueOcean plugin. Click on the JOB Number and click on the **Open Blue Ocean** link
   121  that shows the build stages in pipeline view.
   122  
   123  #### How to add more stages to this pipeline flow?
   124  
   125  We use scripted pipeline syntax with groovy and shell scripts. Also, we use global shared library
   126  scripts which are placed in https://github.com/hyperledger/ci-management/tree/master/vars.
   127  Try to leverage these common functions in your code. All you have to do is, undestand the pipeline
   128  flow of the tests and conditions, add more stages as mentioned in the existing Jenkinsfile.
   129  
   130  #### How will I get build failure notifications.
   131  
   132  On every merge failure, we send build failure email notications to the submitter of the patchset
   133  and send build details to the Rocket Chat **jenkins-robot** channel. Check the result here
   134  https://chat.hyperledger.org/channel/jenkins-robot.
   135  
   136  #### What steps I have to modify when I create a branch from master?
   137  
   138  As the Jenkinsfile is completely parametrzed, you no need to modify anything in the Jenkinsfile
   139  but you may endup modifying **ci.properties** file with the appropirate Base Versions,
   140  Baseimage versions etc... in the new branch. We suggest you to modify this file immediately
   141  after you create a new branch to avoid running tests on old versions.
   142  
   143  #### What are the supported platforms
   144  
   145  - x86_64 (Run the tests on verify and merge job)
   146  - s390x (Run the tests as part of daily job)
   147  
   148  #### Where can I see the Build Scripts.
   149  
   150  We use global shared library scripts and Jenkinsfile along with the build file.
   151  
   152  Global Shared Library - https://github.com/hyperledger/ci-management/tree/master/vars
   153  
   154  Jenkinsfile           - https://github.com/hyperledger/fabric-ca/tree/release-1.4/Jenkinsfile
   155  
   156  ci.properties         - https://github.com/hyperledger/fabric-ca/tree/release-1.4/ci.properties
   157  (ci.properties is the only file you have to modify with the values requried for the specific branch.)
   158  
   159  Packer Scripts        - https://github.com/hyperledger/ci-management/blob/master/packer/provision/docker.sh
   160  (Packer is a tool for automatically creating VM and container images, configuring them and
   161  post-processing them into standard output formats. We build Hyperledger's CI images via Packer
   162  and attach them to x86_64 build nodes. On s390x, we install manually. See the packages we install
   163  as a pre-requisite in the CI x86 build nodes.)
   164  
   165  #### How to reach out to CI team?
   166  
   167  Post your questions or feedback in https://chat.hyperledger.org/channel/ci-pipeline or https://chat.hyperledger.org/channel/fabric-ci Rocket Chat channels.
   168  You can also create a JIRA task or bug in FABCI project. https://jira.hyperledger.org/projects/FABCI