github.com/amrnt/deis@v1.3.1/docs/contributing/test_plan.rst (about)

     1  :title: Test Plan
     2  :description: Details the strategy used to verify the Deis platform
     3  
     4  .. _test_plan:
     5  
     6  Test Plan
     7  =========
     8  
     9  Identifier
    10  ----------
    11  
    12  This document's identifier is **OPD-MTP-1.0.1**. The identifier changes with
    13  any significant revision to the document.
    14  
    15  
    16  References
    17  ----------
    18  
    19  - "Testing Deis": http://docs.deis.io/en/v1.0.1/contributing/testing/
    20  - "Changes Must Include Tests": http://docs.deis.io/en/v1.0.1/contributing/standards/#include-tests
    21  - "Release Schedule": http://docs.deis.io/en/v1.0.1/contributing/schedule/
    22  
    23  
    24  Introduction
    25  ------------
    26  
    27  This document describes the master test plan for the Deis open source PaaS
    28  software platform. It is a *living document*, continually updated to reflect the
    29  current practice and goals of software quality assurance for Deis.
    30  
    31  Deis has unit_, functional_, integration_, and acceptance_ tests that cover
    32  essential product functionality. The Deis project relies on contributors to be
    33  responsible by exercising these tests themselves, and by including tests when
    34  proposing changes. Deis is tested and validated by a 24/7
    35  `continuous integration`_ platform, supplemented by intelligent manual testing.
    36  
    37  
    38  Test items
    39  ----------
    40  
    41  Within the scope of this master test plan are these items:
    42  
    43  - The Deis project codebase at https://github.com/deis/deis
    44  - The Docker containers that constitute Deis
    45  - The assembled Deis platform on a CoreOS cluster
    46  - The HTML documentation set for Deis
    47  - Binary installers for the ``deis`` CLI hosted at AWS S3
    48  - Binary installers for the ``deisctl`` CLI hosted at AWS S3
    49  - Hosted HTML documentation updates at http://docs.deis.io/
    50  - Docker images hosted at https://registry.hub.docker.com/repos/deis/
    51  
    52  
    53  .. _features_to_be_tested:
    54  
    55  Features to be Tested
    56  ---------------------
    57  
    58  At a high level, the overall features of the platform that are tested are:
    59  
    60  - The Deis platform can be installed on a new CoreOS vagrant cluster
    61  - The Deis platform can be upgraded from a recent release to the current one
    62  - Users can register with Deis and create and deploy applications
    63  - Deis can build and scale a variety of Heroku-style and Dockerfile-based apps
    64  - Users can grant and revoke application access to other users
    65  
    66  
    67  Features not to be Tested
    68  -------------------------
    69  
    70  While these features are effectively covered in ad-hoc testing and by existing
    71  customer usage, they are not specifically tested as part of the test plan yet.
    72  
    73  - The Deis platform can be installed on an existing CoreOS cluster
    74  - The Deis platform survives the loss of one or more nodes
    75  - The Deis platform can run for several weeks without interruption
    76  - The Deis platform handles stressful multi-user workloads
    77  
    78  These features are not included in the test plan currently due to resource
    79  limitations. Future test automation will move these features into the
    80  "to be tested" section.
    81  
    82  
    83  Approach
    84  --------
    85  
    86  Deis' test plan relies on extensive test automation, supplemented by spot
    87  testing by responsible developers. Continuous integration tests ensure that
    88  the platform functions and regressions are not introduced, and focused manual
    89  testing is also relied upon for acceptance testing a product release.
    90  
    91  Developers are expected to have run the same tests locally that will be run
    92  for them by continuous integration, specifically the test-integration.sh_
    93  script. This will execute documentation tests, unit tests, functional tests,
    94  and then an overall acceptance test against a Vagrant cluster.
    95  
    96  As changes are incorporated into Deis and the team plans a product release,
    97  maintainers begin acceptance tests against other cloud providers, following the
    98  instructions exactly as provided to users. When these platform-specific tests
    99  have passed, a final validation test occurs in continuous integration against
   100  a tagged codebase. If it succeeds, the release has passed.
   101  
   102  Nightly jobs run a subset of test-integration tests against the released CLI
   103  installers and the current codebase to detect regressions in product behavior.
   104  
   105  
   106  Item Pass/Fail Criteria
   107  -----------------------
   108  
   109  Integration testing has passed when no failures occurred in the
   110  test-integration.sh_ script as run by continuous integration. (Each proposed
   111  change also requires review and approval by two project maintainers before it
   112  is actually merged into the codebase, see :ref:`merge_approval`.)
   113  
   114  Acceptance testing has passed when no failures occurred in the "test-latest.sh"
   115  script as run by continuous integration in the "test-latest" job, and when
   116  maintainers have completed spot testing successfully, as defined by local
   117  release criteria.
   118  
   119  
   120  Suspension Criteria and Resumption Requirements
   121  -----------------------------------------------
   122  
   123  Suspension of automated testing occurs when any failure arises in any part of
   124  the test suite. There are no optional or weighted failures: everything
   125  must pass. Suspension of manual testing occurs when a failure arises that
   126  makes further testing unpredictable or of limited value.
   127  
   128  Resumption of testing occurs when a test failure has been addressed and fixed
   129  such that it is reasonable to assume tests may pass again.
   130  
   131  
   132  Test Deliverables
   133  -----------------
   134  
   135  - Detailed testing logs as generated by CI jobs.
   136    See https://ci.deis.io/job/test-master/85/console for an example.
   137  
   138  
   139  Remaining Test Tasks
   140  --------------------
   141  
   142  For acceptance testing, test deliverables are generated by a maintainer starting
   143  the CLI and test-master jobs when appropriate.
   144  
   145  
   146  Environmental needs
   147  -------------------
   148  
   149  Testing requires a Linux or Mac OS X host capable of running VirtualBox and
   150  Vagrant with good network connectivity. Specific environmental needs are
   151  outlined in the setup-node.sh_ script, which should be kept up-to-date with
   152  current needs.
   153  
   154  
   155  Staffing and training needs
   156  ---------------------------
   157  
   158  N/A
   159  
   160  
   161  Responsibilities
   162  ----------------
   163  
   164  A maintainer designated as "QA Lead" for an acceptance test process has the
   165  responsibility to execute the test task of starting appropriate CI jobs. The
   166  QA Lead is also tasked with overseeing manual testing activities executed by
   167  others.
   168  
   169  For a patch or minor release, the QA Lead may decide not to execute all aspects
   170  of acceptance testing.
   171  
   172  The QA Lead may also execute clerical tasks associated with a release as
   173  described in the :ref:`releases` documentation.
   174  
   175  
   176  Schedule
   177  --------
   178  
   179  As we describe an ongoing, evolving test plan here, there is no fixed project
   180  schedule to address, just a repeatable process.
   181  
   182  Deis releases early and often. The consequences of a failure in the test process
   183  described here are a delay to an expected release date and the restart of the
   184  test process once the failure has been addressed.
   185  
   186  
   187  Risks and Contingencies
   188  -----------------------
   189  
   190  Automated tests do not yet extend to all cloud providers, and it is possible
   191  that manual testing could miss something. We will address this by adding EC2_
   192  and other testing flavors soon.
   193  
   194  Resources are limited, and contention between development needs and testing
   195  needs has the potential to slow down the quality assurance process.
   196  
   197  
   198  Approvals
   199  ---------
   200  
   201  The Deis maintainer team as a whole approves this document through our normal
   202  pull request and merge approval process. Comments and additions will be made as
   203  pull requests against this documentation.
   204  
   205  
   206  .. _unit: http://en.wikipedia.org/wiki/Unit_testing
   207  .. _functional: http://en.wikipedia.org/wiki/Functional_testing
   208  .. _integration: http://en.wikipedia.org/wiki/Integration_testing
   209  .. _acceptance: http://en.wikipedia.org/wiki/Acceptance_testing
   210  .. _`continuous integration`: http://en.wikipedia.org/wiki/Continuous_integration
   211  .. _`source code`: https://github.com/deis/deis
   212  .. _test-integration.sh: https://github.com/deis/deis/blob/master/tests/bin/test-integration.sh
   213  .. _setup-node.sh: https://github.com/deis/deis/blob/master/tess/bin/setup-node.sh
   214  .. _EC2: http://aws.amazon.com/