github.com/redhat-appstudio/e2e-tests@v0.0.0-20240520140907-9709f6f59323/docs/Guidelines.md (about) 1 # Overview 2 The purpose of *this* document is to serve as a primer for developers/qe who are looking for best practices or adding new tests to RHTAP E2E framework. 3 4 ## General tips 5 6 * Make sure you've implemented any required controller functionality that is required for your tests within the following files 7 * `pkg/clients/<new controller directory>` - logic to interact with kube controllers via API 8 * `pkg/framework/framework.go` - import the new controller and update the `Framework` struct to be able to initialize the new controller 9 * Every test package should be imported to [cmd/e2e_test.go](https://github.com/redhat-appstudio/e2e-tests/blob/main/cmd/e2e_test.go#L15). 10 * Every new test should have correct [labels](docs/LabelsNaming.md). 11 * Every test should have meaningful description with JIRA/GitHub issue key. 12 * (Recommended) Use JIRA integration for linking issues and commits (just add JIRA issue key in the commit message). 13 * When running via mage you can filter the suites run by specifying the 14 `E2E_TEST_SUITE_LABEL` environment variable. For example: 15 `E2E_TEST_SUITE_LABEL=ec ./mage runE2ETests` 16 * `klog` level can be controlled via `KLOG_VERBOSITY` environment variable. For 17 example: `KLOG_VERBOSITY=9 ./mage runE2ETests` would output http requests 18 issued via Kubernetes client from sigs.k8s.io/controller-runtime 19 * To quickly debug a test, you can run only the desired suite. Example: `./bin/e2e-appstudio --ginkgo.focus="e2e-demos-suite"` 20 * Split tests in multiple scenarios. It's better to debug a small scenario than a very big one 21 22 ## Debuggability 23 24 If your test fails, it should provide as detailed as possible reasons for the failure in its failure message. The failure message is the string that gets passed (directly or indirectly) to ginkgo.Fail[f]. 25 26 A good failure message: 27 * identifies the test failure 28 * has enough details to provide some initial understanding of what went wrong 29 30 It is good practice to include details like the object that failed some assertion in the failure message because then a) the information is available when analyzing a failure that occurred in the CI and b) it only gets logged when some assertion fails. Always dumping objects via log messages can make the test output very large and may distract from the relevant information. 31 32 Dumping structs with `format.Object` is recommended. Starting with Kubernetes 1.26, format.Object will pretty-print Kubernetes API objects or structs as YAML and omit unset fields, which is more readable than other alternatives like `fmt.Sprintf("%+v")`. 33 34 ```golang 35 import ( 36 "fmt" 37 "k8s.io/api/core/v1" 38 "k8s.io/kubernetes/test/utils/format" 39 ) 40 41 var pod v1.Pod 42 fmt.Printf("format.Object:\n%s", format.Object(pod, 1 /* indent one level */)) 43 44 format.Object: 45 <v1.Pod>: 46 metadata: 47 creationTimestamp: null 48 spec: 49 containers: null 50 status: {} 51 ``` 52 ## Polling and timeouts 53 54 When waiting for something to happen, use a reasonable timeout. Without it, a test might keep running until the entire test suite gets killed by the CI. **Beware that the CI under load may take a lot longer to complete some operation compared to running the same test locally**. On the other hand, a too long timeout also has drawbacks: 55 56 * When a feature is broken so that the expected state doesn’t get reached, a test waiting for that state first needs to time out before the test fails. 57 * If a state is expected to be reached within a certain time frame, then a timeout that is much higher will cause test runs to be considered successful although the feature was too slow. A dedicated performance test in a well-known environment may be a better solution for testing such performance expectations. 58 59 Good code that waits for something to happen meets the following criteria: 60 * accepts a context for test timeouts 61 * full explanation when it fails: when it observes some state and then encounters errors reading the state, then dumping both the latest observed state and the latest error is useful 62 * early abort when condition cannot be reached anymore 63 64 ### Tips for writing and debugging long-running tests 65 66 * Use `ginkgo.By` to record individual steps. Ginkgo will use that information when describing where a test timed out. 67 * Use `gomega.Eventually` to wait for some condition. When it times out or gets stuck, the last failed assertion will be included in the report automatically. A good way to invoke it is: 68 ```go 69 70 Eventually(func() error { 71 _, err := s.GetSPIAccessToken(linkedAccessTokenName, namespace) 72 return err 73 }, 1*time.Minute, 100*time.Millisecond).Should(Succeed(), "SPI controller didn't create the SPIAccessToken") 74 ``` 75 * Use `gomega.Consistently` to ensure that some condition is true for a while. As with `gomega.Eventually`, make assertions about the value instead of checking the value with Go code and then asserting that the code returns true. 76 * Both `gomega.Consistently` and `gomega.Eventually` can be aborted early via `gomega.StopPolling`. 77 * Avoid polling with functions that don’t take a context (`wait.Poll`, `wait.PollImmediate`, `wait.Until`, …) and replace with their counterparts that do (`wait.PollWithContext`, `wait.PollImmediateWithContext`, `wait.UntilWithContext`, …) or even better, with `gomega.Eventually`. 78 79 ## E2E directory structure 80 81 This is a basic layout for RHTAP E2E framework project. It is a set of common directories for all teams in RHTAP. 82 83 * `/cmd`: Is the main for all e2e tests. Don't put a lot of code in the application directory. If you think the code can be imported and used in other projects, then it should live in the `/pkg` directory. 84 * `docs`: Documentation about RHTAP e2e world. 85 * `/magefiles`: The code definition about installing and running the e2e tests 86 * `/pipelines`: Tekton pipelines utilities for QE team like IC. 87 * `/pkg`: All used and imported packages for e2e tests. 88 * `/pkg/clients`: Definition of different clients connection providers (like Slack, GitHub, and Kubernetes Server) and all API interaction with different RHTAP controllers. 89 * `/pkg/constants`: Global constants of the e2e tests. 90 * `/pkg/framework`: In the framework folder are all controllers initialization, tests reports and the interaction with Report Portal. 91 * `/pkg/logs`: Tests logging utilities. 92 * `/pkg/sandbox`: Initialize Sandbox controller to make authenticated requests to a Kubernetes server. 93 * `/pkg/utils`: Util folders with all auxiliary functions used in the different RHTAP controllers. Futhermore, it also contains some tests utils that can be found in `/pkg/utils/util.go`. 94 * `/scripts`: Scripts to perform operations which cannot be do it at magefiles level. 95 * `/tests`: Folder where all RHTAP tests are defined and documented.