github.com/cloudfoundry/cli@v7.1.0+incompatible/doc/adr/0007-integration-vs-command-test.md (about)

     1  # ADR 7: Test Coverage: Integration vs. Command
     2  
     3  ## Status
     4  
     5  Proposed
     6  
     7  ## Context
     8  
     9  The integration tests' coverage (e.g.
    10  `integration/v7/isolated/create_org_command_test.go`) overlaps with the command
    11  tests' (`command/v7/create_org_command_test.go`), and this causes confusion
    12  for developers—does a test belong in integration, command, or both? This ADR
    13  gives guidelines in order to assist developers making that decision.
    14  
    15  ## Decision
    16  
    17  We want to separate [black
    18  box](https://en.wikipedia.org/wiki/Black-box_testing) and [white
    19  box](https://en.wikipedia.org/wiki/White-box_testing) testing, where
    20  black box testing (functionality of an application) is done at the
    21  integration level, and white box testing (internal structures or
    22  workings of an application) is done at the unit (command) level.
    23  
    24  #### Integration
    25  Integration tests should continue to test things from a user's perspective:
    26  
    27  - Build contexts around use cases
    28  - Test output based on story parameters and the style guides.
    29  
    30  For example, the following behavior (from the `unbind-security-group`
    31  story) should be tested at the integration level because it describes
    32  output/functionality:
    33  
    34  > When `cf unbind-security-group` attempts to unbind a running security group
    35  > that is not bound to a space, it should return the following:
    36  >
    37  > ```
    38  > Security group my-group not bound to space my-space for lifecycle phase 'running'.
    39  > OK
    40  >
    41  > TIP: Changes require an app restart (for running) or restage (for staging) to apply to existing applications.
    42  > ```
    43  
    44  To do so, we would have to make a `BeforeEach()` that creates the scenario
    45  laid out in the story and write expectations on each line of output in
    46  the order they are listed. For example, from
    47  `integration/v7/global/unbind_security_group_command_test.go` (edited
    48  for readability):
    49  
    50  ```golang
    51  BeforeEach(func() {
    52  	port := "8443"
    53  	description := "some-description"
    54  	someSecurityGroup := helpers.NewSecurityGroup(securityGroupName, "tcp", "127.0.0.1", &port, &description)
    55  	helpers.CreateSecurityGroup(someSecurityGroup)
    56  	helpers.CreateSpace(spaceName)
    57  })
    58  
    59  When("the space isn't bound to the security group in any lifecycle", func() {
    60  	It("successfully runs the command", func() {
    61  		session := helpers.CF("unbind-security-group", securityGroupName, orgName, spaceName)
    62  		Eventually(session).Should(Say(`Unbinding security group %s from org %s / space %s as %s\.\.\.`, securityGroupName, orgName, spaceName, username))
    63  		Eventually(session.Err).Should(Say(`Security group %s not bound to space %s for lifecycle phase 'running'\.`, securityGroupName, spaceName))
    64  		Eventually(session).Should(Say("OK"))
    65  		Eventually(session).Should(Say(`TIP: Changes require an app restart \(for running\) or restage \(for staging\) to apply to existing applications\.`))
    66  		Eventually(session).Should(Exit(0))
    67  	})
    68  })
    69  ```
    70  
    71  #### Command (Unit)
    72  In unit tests we want to break away from this perspective. We
    73  want to organize our tests by the [line of
    74  sight](https://engineering.pivotal.io/post/go-flow-tests-like-code/)
    75  method, where we cover code paths as we see them.
    76  
    77  <!-- Brian is really uncomfortable linking to something he wrote -->
    78  
    79  In general, every `if err != nil {` should have a corresponding `When("actor returns an
    80  error", func() {`, but in most cases we do not need to test different
    81  possible errors that could be returned.
    82  
    83  Here's an example from `bind-security-group` code and unit test:
    84  
    85  ```golang
    86  securityGroup, warnings, err := cmd.Actor.GetSecurityGroup(cmd.RequiredArgs.SecurityGroupName)
    87  cmd.UI.DisplayWarnings(warnings)
    88  if err != nil {
    89  	return err
    90  }
    91  ```
    92  
    93  ```golang
    94  It("Retrieves the security group information", func() {
    95  	Expect(fakeActor.GetSecurityGroupCallCount).To(Equal(1))
    96  	securityGroupName := fakeActor.GetSecurityGroupArgsForCall(0)
    97  	Expect(securityGroupName).To(Equal(cmd.RequiredArgs.SecurityGroupName))
    98  })
    99  
   100  It("prints the warnings", func() {
   101  	Expect(testUI.Err).To(Say(getSecurityGroupWarning[0]))
   102  })
   103  
   104  When("an error is encountered getting the provided security group", func() {
   105  	var expectedErr error
   106  
   107  	BeforeEach(func() {
   108  		expectedErr = errors.New("get security group error")
   109  		fakeActor.GetSecurityGroupReturns(
   110  			resources.SecurityGroup{},
   111  			v7action.Warnings{"get security group warning"},
   112  			expectedErr)
   113  	})
   114  
   115  	It("returns the error and displays all warnings", func() {
   116  		Expect(executeErr).To(MatchError(expectedErr))
   117  	})
   118  })
   119  ```
   120  
   121  ## Consequences
   122  
   123  - Since this is not a very big change, we are not expecting to lose any
   124    converage or increase run times. This is mostly for organization
   125    purposes.
   126  
   127  - There are no plans to retrofit existing tests to adhere to this ADR.
   128  
   129  - These recommendations should not have any noticeable impact on the
   130    time it takes our unit and integration tests to complete; our
   131    integration are much lighter weight than other codebases.
   132  
   133  ## References
   134  
   135  <https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html>
   136  
   137  <https://kentcdodds.com/blog/write-tests>