github.com/inflatablewoman/deis@v1.0.1-0.20141111034523-a4511c46a6ce/docs/contributing/test_plan.rst (about)

     1  :title: Test Plan
     2  :description: Details the strategy used to verify the Deis platform
     3  
     4  .. _test_plan:
     5  
     6  Test Plan
     7  =========
     8  
     9  Identifier
    10  ----------
    11  
    12  This document's identifier is **OPD-MTP-0.16.0**. The identifier changes with
    13  any significant revision to the document.
    14  
    15  
    16  References
    17  ----------
    18  
    19  - "Testing Deis": http://docs.deis.io/en/v1.0.0/contributing/testing/
    20  - "Changes Must Include Tests": http://docs.deis.io/en/v1.0.0/contributing/standards/#include-tests
    21  
    22  
    23  Introduction
    24  ------------
    25  
    26  This document describes the master test plan for the Deis open source PaaS
    27  software platform. It is a *living document*, continually updated to reflect the
    28  current practice and goals of software quality assurance for Deis.
    29  
    30  Deis has unit_, functional_, integration_, and acceptance_ tests that cover
    31  essential product functionality. The Deis project relies on contributors to be
    32  responsible by exercising these tests themselves, and by including tests when
    33  proposing changes. Deis is tested and validated by a 24/7
    34  `continuous integration`_ platform, supplemented by intelligent manual testing.
    35  
    36  
    37  Test items
    38  ----------
    39  
    40  Within the scope of this master test plan are these items:
    41  
    42  - The Deis project codebase at https://github.com/deis/deis
    43  - The Docker containers that constitute Deis
    44  - The assembled Deis platform on a CoreOS cluster
    45  - The HTML documentation set for Deis
    46  - Binary installers for the ``deis`` CLI hosted at AWS S3
    47  - Binary installers for the ``deisctl`` CLI hosted at AWS S3
    48  - Hosted HTML documentation updates at http://docs.deis.io/
    49  - Docker images hosted at https://registry.hub.docker.com/repos/deis/
    50  
    51  
    52  Features to be Tested
    53  ---------------------
    54  
    55  At a high level, the overall features of the platform that are tested are:
    56  
    57  - The Deis platform can be installed on a new CoreOS vagrant cluster
    58  - The Deis platform can be upgraded from a recent release to the current one
    59  - Users can register with Deis and create and deploy applications
    60  - Deis can build and scale a variety of Heroku-style and Dockerfile-based apps
    61  - Users can grant and revoke application access to other users
    62  
    63  
    64  Features not to be Tested
    65  -------------------------
    66  
    67  While these features are effectively covered in ad-hoc testing and by existing
    68  customer usage, they are not specifically tested as part of the test plan yet.
    69  
    70  - The Deis platform can be installed on an existing CoreOS cluster
    71  - The Deis platform survives the loss of one or more nodes
    72  - The Deis platform can run for several weeks without interruption
    73  - The Deis platform handles stressful multi-user workloads
    74  
    75  These features are not included in the test plan currently due to resource
    76  limitations. Future test automation will move these features into the
    77  "to be tested" section.
    78  
    79  
    80  Approach
    81  --------
    82  
    83  Deis' test plan relies on extensive test automation, supplemented by spot
    84  testing by responsible developers. Continuous integration tests ensure that
    85  the platform functions and regressions are not introduced, and focused manual
    86  testing is also relied upon for acceptance testing a product release.
    87  
    88  Developers are expected to have run the same tests locally that will be run
    89  for them by continuous integration, specifically the test-integration.sh_
    90  script. This will execute documentation tests, unit tests, functional tests,
    91  and then an overall acceptance test against a Vagrant cluster.
    92  
    93  As changes are incorporated into Deis and the team plans a product release,
    94  maintainers begin acceptance tests against other cloud providers, following the
    95  instructions exactly as provided to users. When these platform-specific tests
    96  have passed, a final validation test occurs in continuous integration against
    97  a tagged codebase. If it succeeds, the release has passed.
    98  
    99  Nightly jobs run a subset of test-integration tests against the released CLI
   100  installers and the current codebase to detect regressions in product behavior.
   101  
   102  
   103  Item Pass/Fail Criteria
   104  -----------------------
   105  
   106  Integration testing has passed when no failures occurred in the
   107  test-integration.sh_ script as run by continuous integration. (Each proposed
   108  change also requires review and approval by two project maintainers before it
   109  is actually merged into the codebase, see :ref:`merge_approval`.)
   110  
   111  Acceptance testing has passed when no failures occurred in the "test-latest.sh"
   112  script as run by continuous integration in the "test-latest" job, and when
   113  maintainers have completed spot testing successfully, as defined by local
   114  release criteria.
   115  
   116  
   117  Suspension Criteria and Resumption Requirements
   118  -----------------------------------------------
   119  
   120  Suspension of automated testing occurs when any failure arises in any part of
   121  the test suite. There are no optional or weighted failures: everything
   122  must pass. Suspension of manual testing occurs when a failure arises that
   123  makes further testing unpredictable or of limited value.
   124  
   125  Resumption of testing occurs when a test failure has been addressed and fixed
   126  such that it is reasonable to assume tests may pass again.
   127  
   128  
   129  Test Deliverables
   130  -----------------
   131  
   132  - Detailed testing logs as generated by CI jobs.
   133    See http://ci.deis.io/job/test-master/85/console for an example.
   134  
   135  
   136  Remaining Test Tasks
   137  --------------------
   138  
   139  For acceptance testing, test deliverables are generated by a maintainer starting
   140  the CLI and test-master jobs when appropriate.
   141  
   142  
   143  Environmental needs
   144  -------------------
   145  
   146  Testing requires a Linux or Mac OS X host capable of running VirtualBox and
   147  Vagrant with good network connectivity. Specific environmental needs are
   148  outlined in the setup-node.sh_ script, which should be kept up-to-date with
   149  current needs.
   150  
   151  
   152  Staffing and training needs
   153  ---------------------------
   154  
   155  N/A
   156  
   157  
   158  Responsibilities
   159  ----------------
   160  
   161  A maintainer designated as "QA Lead" for an acceptance test process has the
   162  responsibility to execute the test task of starting appropriate CI jobs. The
   163  QA Lead is also tasked with overseeing manual testing activities executed by
   164  others.
   165  
   166  For a patch or minor release, the QA Lead may decide not to execute all aspects
   167  of acceptance testing.
   168  
   169  The QA Lead may also execute clerical tasks associated with a release as
   170  described in the :ref:`releases` documentation.
   171  
   172  
   173  Schedule
   174  --------
   175  
   176  As we describe an ongoing, evolving test plan here, there is no fixed project
   177  schedule to address, just a repeatable process.
   178  
   179  Deis releases early and often. The consequences of a failure in the test process
   180  described here are a delay to an expected release date and the restart of the
   181  test process once the failure has been addressed.
   182  
   183  
   184  Risks and Contingencies
   185  -----------------------
   186  
   187  Automated tests do not yet extend to all cloud providers, and it is possible
   188  that manual testing could miss something. We will address this by adding EC2_
   189  and other testing flavors soon.
   190  
   191  Resources are limited, and contention between development needs and testing
   192  needs has the potential to slow down the quality assurance process.
   193  
   194  
   195  Approvals
   196  ---------
   197  
   198  The Deis maintainer team as a whole approves this document through our normal
   199  pull request and merge approval process. Comments and additions will be made as
   200  pull requests against this documentation.
   201  
   202  
   203  .. _unit: http://en.wikipedia.org/wiki/Unit_testing
   204  .. _functional: http://en.wikipedia.org/wiki/Functional_testing
   205  .. _integration: http://en.wikipedia.org/wiki/Integration_testing
   206  .. _acceptance: http://en.wikipedia.org/wiki/Acceptance_testing
   207  .. _`continuous integration`: http://en.wikipedia.org/wiki/Continuous_integration
   208  .. _`source code`: https://github.com/deis/deis
   209  .. _test-integration.sh: https://github.com/deis/deis/blob/master/tests/bin/test-integration.sh
   210  .. _setup-node.sh: https://github.com/deis/deis/blob/master/tess/bin/setup-node.sh
   211  .. _EC2: http://aws.amazon.com/