github.com/adecaro/fabric-ca@v2.0.0-alpha+incompatible/docs/source/ca-ci.md (about)

     1  # Continuous Integration Process
     2  
     3  This document explains the fabric-ca Jenkins pipeline flow and FAQ's on the build process
     4  to help developer to get more familiarize with the process flow.
     5  
     6  We use Jenkins as a CI tool and to manage jobs, we use [JJB](https://docs.openstack.org/infra/jenkins-job-builder).
     7  Please see the pipeline job configuration template here https://ci-docs.readthedocs.io/en/latest/source/pipeline_jobs.html#job-templates.
     8  
     9  ## CI Pipeline flow
    10  
    11  - Every Gerrit patchset triggers a verify job with the Gerrit Refspec on the parent commit
    12    of the patchset and run the below tests from the `Jenkinsfile`. Note: When you are ready
    13    to merge a patchset, it's always a best practice to rebase the patchset on the latest commit.
    14  
    15      - Basic Checks (make checks)
    16      - Documentation build (tox -edocs)
    17      - Unit tests (make unit-tests)
    18      - FVT tests (make fvt-tests)
    19      - E2E tests (Only after patchset is merged)
    20  
    21  All the above tests run on the Hyperledger infarstructure x86_64 build nodes. All these nodes
    22  uses the packer with pre-configured software packages. This helps us to run the tests in much
    23  faster than installing required packages everytime.
    24  
    25  Below steps shows what each stage does in the Jenkins pipeline verify and merge flow.
    26  Every Gerrit patchset triggers the fabric-ca-verify-x86_64 job and runs the below tests on
    27  x86_64 platform. Before execute the below tests, it clean the environment (Deletes the left
    28  over build artifiacts) and clone the repository with the Gerrit Refspec. Based on the file
    29  extenstions, Jenkins pipeline script triggers the stages. If the patchset contains specific
    30  doc extension, it only triggers **Docs Build** Stage otherwise it triggers all stages.
    31  
    32  ![](images/pipeline_flow.png)
    33  
    34  #### Basic Checks
    35  
    36  - We run `make checks` target to run the basic checks before kickoff the actual tests.
    37  - It's run against every Patchset.
    38  - You can run basic checks locally:
    39      - make checks (Runs all check conditions (license, format, imports, lint and vet)
    40  
    41  #### Docs Build
    42  
    43  - We run `tox -edocs` from the root directory.
    44  - Displays the output in the form of HTML Publisher on the `fabric-ca-verify-x86_64` job.
    45    Click on **Docs Output** link on the Jenkins console.
    46  
    47  #### Unit Tests
    48  
    49  - We run `make unit-test` target to run the go based unit-tests and publish the coverage
    50    report on the Jenkins console. Click on **Coverage Report** link on the Jenkins console
    51    to see the code coverage.
    52  
    53  #### FVT Tests
    54  
    55  - We run `make fvt-tests` target to run fvt tests, which includes tests that performs end-to-end
    56    test scenarios with PosgreSQL and MySQL databases. These tests include database migration,
    57    backwards compatibility, and LDAP integration. https://github.com/hyperledger/fabric-ca/blob/master/scripts/fvt/README.md
    58  
    59  #### E2E tests
    60  
    61  - We run **e2e tests** in the **merge job** and it performs the following tests. The intention
    62    of running e2e tests as part of the merge job is to test the dependent tests of fabric-ca.
    63      - fabcar
    64      - fabric-sdk-node - We run **gulp run-end-to-end** target which executes most of the end to end
    65        tests of fabric-sdk-node which are depend on fabric-ca.
    66      - fabric-sdk-java - We run **ci_run.sh** script which is pre-baked in fabric-sdk-java repository.
    67  
    68  As we trigger `fabric-ca-verify-x86_64` and `fabric-ca-merge-x86_64` pipeline jobs for every gerrit patchset,
    69  we execute these tests in the below order.
    70  
    71  After the DocsBuild stage is passed, Jenkins Pipeline triggers the Unit and FVT Tests parallel
    72  on two different nodes. After the tests are executed successfully it posts a Gerrit voting
    73  on the patchset. If DocsBuild stage fails, it send the result back to Gerrit patchset and it
    74  won't trigger further builds.
    75  
    76  See below **FAQ's** for more information on the pipeline process.
    77  
    78  ## FAQ's
    79  
    80  #### What happens on the merge job?
    81  
    82  After the patchset got merged in the fabric-ca repository, it follows the above pipeline flow and
    83  executes the e2e tests in parallel to the Unit and FVT Tests.
    84  
    85  **Merge Pipeline Flow**
    86  
    87  ```
    88  CleanEnvironment -- OutputEnvironment -- CloneRefSpec -- BasicChecks -- DocsBuild - Tests (E2E, Unit, FVT Tests)
    89  ```
    90  
    91  Jenkins clones the latest merged commit and executes the below steps
    92  
    93  - Build fabric, fabric-ca images & binaries
    94  - Pull Thirdparty Images from DockerHub (Couchdb, zookeeper, kafka)
    95  - Pull javaenv, nodeenv images from nexus3 (latest stable images published after successful merge job of each repo)
    96  - Tests
    97    - fabcar tests
    98    - fabric-sdk-node (npm install, gulp run-end-to-end)
    99    - fabric-sdk-java (Run ci_run.sh)
   100  
   101  #### What happens if one of the build stage fails?
   102  
   103  As we are running these tests in `fastFailure: true` (if any build stage fails in the parallel
   104  process, it will terminate/abort the current running tests and sends the result back to the
   105  Gerrit Patchset. This way, CI will avoid runnning tests when there is a failure in one of the
   106  parallel build stage.
   107  
   108  It shows `aborted` on the aborted stage on pipeline staged view.
   109  
   110  #### How to re-trigger failed tests?
   111  
   112  With this pipeline flow, you can NOT re-trigger the specific stage, but you can post comments
   113  `reverify` or `reverify-x` on the gerrit patchset to trigger the `fabric-ca-verify-x86_64`
   114  job which triggers the pipeline flow as mentioned above. Also, we provided `remerge` or `remerge-x`
   115  comment phrases to re-trigger the failed merge jobs.
   116  
   117  #### Where should I see the output of the stages?
   118  
   119  Piepline supports two views (staged and blueocean). **Staged views** shows on the Jenkins job main
   120  page and it shows each stage in order and the status. For better view, we suggest you to access
   121  BlueOcean plugin. Click on the JOB Number and click on the **Open Blue Ocean** link that shows
   122  the build stages in pipeline view.
   123  
   124  #### How to add more stages to this pipeline flow?
   125  
   126  We use scripted pipeline syntax with groovy and shell scripts. Also, we use global shared library
   127  scripts which are placed in https://github.com/hyperledger/ci-management/tree/master/vars.
   128  Try to leverage these common functions in your code. All you have to do is, undestand the pipeline
   129  flow of the tests and conditions, add more stages as mentioned in the existing Jenkinsfile.
   130  
   131  #### How will I get build failure notifications?
   132  
   133  On every merge failure, we send build failure email notications to the submitter of the patchset
   134  and send build details to the Rocket Chat **jenkins-robot** channel. Check the result here
   135  https://chat.hyperledger.org/channel/jenkins-robot
   136  
   137  #### What steps I have to modify when I create a new branch from master?
   138  
   139  As the Jenkinsfile is completely parametrzed, you no need to modify anything in the Jenkinsfile
   140  but you may endup modifying **ci.properties** file with the appropirate Base Versions,
   141  Baseimage versions etc... in the new branch. We suggest you to modify this file immediately
   142  after you create a new branch to avoid running tests on old versions.
   143  
   144  #### What are the supported platforms
   145  
   146  - x86_64 (Run the tests on verify and merge job)
   147  - s390x (Run the same tests as part of the daily job)
   148  
   149  #### Where can I see the Build Scripts.
   150  
   151  We use global shared library scripts and Jenkinsfile along with the build file.
   152  
   153  Global Shared Library - https://github.com/hyperledger/ci-management/tree/master/vars
   154  
   155  Jenkinsfile           - https://github.com/hyperledger/fabric-ca/tree/master/Jenkinsfile
   156  
   157  ci.properties         - https://github.com/hyperledger/fabric-ca/tree/master/ci.properties
   158  (ci.properties is the only file you have to modify with the values requried for the specific branch.)
   159  
   160  Packer Scripts        - https://github.com/hyperledger/ci-management/blob/master/packer/provision/docker.sh
   161  (Packer is a tool for automatically creating VM and container images, configuring them and
   162  post-processing them into standard output formats. We build Hyperledger's CI images via Packer
   163  and attach them to x86_64 build nodes. On s390x, we install manually. See the packages we
   164  install as a pre-requisite in the CI x86 build nodes.)
   165  
   166  #### How to reach out to CI team?
   167  
   168  Post your questions or feedback in https://chat.hyperledger.org/channel/ci-pipeline or https://chat.hyperledger.org/channel/fabric-ci Rocket Chat channels.
   169  You can also create a JIRA task or bug in FABCI project. https://jira.hyperledger.org/projects/FABCI