github.com/rohankumardubey/nomad@v0.11.8/e2e/README.md (about) 1 # End to End Tests 2 3 This package contains integration tests. 4 5 The `terraform` folder has provisioning code to spin up a Nomad cluster on AWS. 6 The tests work with the `NOMAD_ADDR` environment variable which can be set 7 either to a local dev Nomad agent or a Nomad client on AWS. 8 9 ## Local Development 10 11 The workflow when developing end to end tests locally is to run the 12 provisioning step described below once, and then run the tests as described 13 below. 14 15 When making local changes, use `./bin/update $(which nomad) /usr/local/bin/nomad` 16 and `./bin/run sudo systemctl restart nomad` to destructively modify the 17 provisioned cluster. 18 19 ## Provisioning Test Infrastructure on AWS 20 21 You'll need Terraform and AWS credentials (`AWS_ACCESS_KEY_ID` and 22 `AWS_SECRET_ACCESS_KEY`) to setup AWS instances on which e2e tests 23 will run. See the [README](https://github.com/hashicorp/nomad/blob/master/e2e/terraform/README.md) 24 for details. The number of servers and clients is configurable, as is 25 the configuration file for each client and server. 26 27 ## Provisioning e2e Framework Nomad Cluster 28 29 You can use the Terraform output from the previous step to generate a 30 provisioning configuration file for the e2e framework. 31 32 ```sh 33 # from the ./e2e/terraform directory 34 terraform output provisioning | jq . > ../provisioning.json 35 ``` 36 37 By default the `provisioning.json` will not include the Nomad version 38 that will be deployed to each node. You can pass the following flags 39 to `go test` to set the version for all nodes: 40 41 - `-nomad.local_file=string`: provision this specific local binary of 42 Nomad. This is a path to a Nomad binary on your own 43 host. Ex. `-nomad.local_file=/home/me/nomad` 44 - `-nomad.sha=string`: provision this specific sha from S3. This is a 45 Nomad binary identified by its full commit SHA that's stored in a 46 shared s3 bucket that Nomad team developers can access. That commit 47 SHA can be from any branch that's pushed to 48 remote. Ex. `-nomad.sha=0b6b475e7da77fed25727ea9f01f155a58481b6c` 49 - `-nomad.version=string`: provision this version from 50 [releases.hashicorp.com](https://releases.hashicorp.com/nomad). Ex. `-nomad.version=0.10.2` 51 52 Then deploy Nomad to the cluster by passing `-provision.terraform` 53 without a Nomad version flag: 54 55 ```sh 56 NOMAD_E2E=1 go test -v . -nomad.version=0.10.2 -provision.terraform ./provisioning.json -skipTests 57 ``` 58 59 Because it can take a little while for the cluster to settle, it's 60 recommended to run this provisioning step (with `-skipTests`) first, 61 and then run tests as separate step. 62 63 ## Running 64 65 After completing the provisioning step above, you can set the client 66 environment for `NOMAD_ADDR` and run the tests as shown below: 67 68 ```sh 69 # from the ./e2e/terraform directory, set your client environment 70 # if you haven't already 71 $(terraform output environment) 72 73 cd .. 74 go test -v . 75 ``` 76 77 If you want to run a specific suite, you can specify the `-suite` flag as 78 shown below. Only the suite with a matching `Framework.TestSuite.Component` 79 will be run, and all others will be skipped. 80 81 ```sh 82 go test -v -suite=Consul . 83 ``` 84 85 If you want to run a specific test, you'll need to regex-escape some of the 86 test's name so that the test runner doesn't skip over framework struct method 87 names in the full name of the tests: 88 89 ```sh 90 go test -v . -run 'TestE2E/Consul/\*consul\.ScriptChecksE2ETest/TestGroup' 91 ``` 92 93 ## I Want To... 94 95 ### ...SSH Into One Of The Test Machines 96 97 You can use the Terraform output to find the IP address. The keys will 98 in the `./terraform/keys/` directory. 99 100 ```sh 101 ssh -i keys/nomad-e2e-*.pem ubuntu@${EC2_IP_ADDR} 102 ``` 103 104 ### ...Deploy a Cluster of Mixed Nomad Versions 105 106 The `provisioning.json` file output by Terraform has a blank field for 107 `nomad_sha` for each node of the cluster (server and client). You can 108 manually edit the file to replace this value with a `nomad_sha`, 109 `nomad_local_binary`, or `nomad_version` for each node to create a 110 cluster of mixed versions. The provisioning framework accepts any of 111 the following options for those fields: 112 113 - `nomad_sha`: This is a Nomad binary identified by its full commit 114 SHA that's stored in a shared s3 bucket that Nomad team developers 115 can access. That commit SHA can be from any branch that's pushed to 116 remote. (Ex. `"nomad_sha": 117 "0b6b475e7da77fed25727ea9f01f155a58481b6c"`) 118 - `nomad_local_binary`: This is a path to a Nomad binary on your own 119 host. (Ex. `"nomad_local_binary": "/home/me/nomad"`) 120 - `nomad_version`: This is a version number of Nomad that's been 121 released to HashiCorp. (Ex. `"nomad_version": "0.10.2"`) 122 123 Then deploy Nomad to the cluster by passing `-provision.terraform` 124 without a Nomad version flag: 125 126 ```sh 127 go test -v . -provision.terraform ./provisioning.json -skipTests 128 ``` 129 130 ### ...Deploy Custom Configuration Files 131 132 The `provisioning.json` file includes a `bundles` section for each 133 node of the cluster (server and client). You can manually edit this 134 file to add, remove, or replace 135 136 ```json 137 "bundles": [ 138 { 139 "destination": "/ops/shared/nomad/base.hcl", 140 "source": "/home/me/custom.hcl" 141 } 142 ] 143 ``` 144 145 ### ...Deploy More Than 4 Linux Clients 146 147 Right now the framework doesn't support this out-of-the-box because of 148 the way the provisioning script adds specific client configurations to 149 each client node (for constraint testing). You'll need to add 150 additional configuration files to 151 `./e2e/terraform/shared/nomad/indexed`.