github.com/mweagle/Sparta@v1.15.0/docs_source/content/reference/step/fargate.md (about) 1 --- 2 date: 2018-12-16 07:02:28 3 title: Fargate 4 weight: 20 5 --- 6 7 {{< tweet 1073347546553217024 >}} 8 9 While Serverless and FaaS are often used interchangeably, there are types of 10 workloads that are more challenging to move to FaaS. Perhaps due to third 11 party libraries, latency, or storage requirements, the FaaS model 12 isn't an ideal fit. An example that is commonly provided is the need 13 to run [ffmpeg](https://www.ffmpeg.org/). 14 15 To benefit from the serverless model in these cases, Sparta provides 16 the ability to leverage the [Fargate](https://aws.amazon.com/fargate) service 17 to run Containers without needing to manage servers. 18 19 There are several steps to _Fargate-ifying_ your application and Sparta exposes 20 functions and hooks to make that operation scoped to a `provision` operation. 21 22 Those steps include: 23 24 1. Make the application Task-aware 25 1. Package your application in a Docker image 26 1. Push the Docker image to [Amazon ECR](https://aws.amazon.com/ecr/) 27 1. Reference the ECR URL in a Fargate Task 28 1. Provision an ECS cluster that hosts the Task 29 30 This overview is based on the [SpartaStepServicefull](https://github.com/mweagle/SpartaStepServicefull) 31 project. The implementation uses a combination of [ServiceDecoratorHookHandlers](https://godoc.org/github.com/mweagle/Sparta#ServiceDecoratorHookHandler) 32 to achieve the end result. 33 34 Please see [servicefull_build.go](https://github.com/mweagle/SpartaStepServicefull/blob/master/bootstrap/servicefull_build.go) 35 for the most up-to-date version of code samples. 36 37 ## Task Awareness 38 39 The first step is to provide an opportunity for our application to behave 40 differently when run as a Fargate task. To do this we add a new 41 application subcommand option that augments the standard `Main` behavior: 42 43 ```go 44 // Add a hook to do something 45 fargateTask := &cobra.Command{ 46 Use: "fargateTask", 47 Short: "Sample Fargate task", 48 Long: `Sample Fargate task that simply logs a message"`, 49 RunE: func(cmd *cobra.Command, args []string) error { 50 fmt.Printf("Insert your Fargate code here! 🎉") 51 return nil 52 }, 53 } 54 // Register the command with the Sparta root dispatcher. This 55 // command `fargateTask` matches the command line option in the 56 // Dockerfile that is used to build the image. 57 sparta.CommandLineOptions.Root.AddCommand(fargateTask) 58 ``` 59 60 This subcommand is defined in the [servicefull_task](https://github.com/mweagle/SpartaStepServicefull/blob/master/bootstrap/servicefull_task.go) 61 file. Note that the file uses `go` [build tags](https://dave.cheney.net/2013/10/12/how-to-use-conditional-compilation-with-the-go-build-tool) 62 so that the new **fargateTask** subcommand is only available when the 63 build target includes the _lambdaBinary_ flag: 64 65 ```go 66 // +build lambdabinary 67 68 package bootstrap 69 ``` 70 71 We can now package our Task-aware executable and deploy it to the cloud. 72 73 ## Package 74 75 The first step is to create a version of your application that 76 can support a Fargate task. This is done in the `ecrImageBuilderDecorator` 77 function which delegates the compiling and image creation to Sparta: 78 79 ```go 80 // Always build the image 81 buildErr := spartaDocker.BuildDockerImage(serviceName, 82 "", 83 dockerTags, 84 logger) 85 ``` 86 87 The second empty argument above is an optional _Dockerfile_ path. The sample 88 project uses the default _Dockerfile_ filename and defines that at the root 89 of the repository. The full _Dockerfile_ is: 90 91 ```docker 92 FROM alpine:3.8 93 RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/* 94 95 # Sparta provides the SPARTA_DOCKER_BINARY argument to the builder 96 # in order to embed the binary. 97 # Ref: https://docs.docker.com/engine/reference/builder/ 98 ARG SPARTA_DOCKER_BINARY 99 ADD $SPARTA_DOCKER_BINARY /SpartaServicefull 100 CMD ["/SpartaServicefull", "fargateTask"] 101 ``` 102 103 The `BuildDockerImage` function supplies the transient binary executable 104 path to docker via the **SPARTA_DOCKER_BINARY** [ARG](https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg) 105 value. 106 107 108 The `CMD` instruction includes our previously registered **fargateTask** 109 subcommand name to invoke the Task-appropriate codepath at runtime. 110 111 The log output includes the docker build info: 112 ``` 113 INFO[0002] Calling WorkflowHook 114 ServiceDecoratorHook=github.com/mweagle/SpartaStepServicefull/bootstrap.ecrImageBuilderDecorator.func1 115 WorkflowHookContext="map[]" 116 INFO[0002] Docker version 18.09.0, build 4d60db4 117 INFO[0002] Running `go generate` 118 INFO[0002] Compiling binary 119 Name=ServicefulStepFunction-1544976454011339000-docker.lambda.amd64 120 INFO[0003] Creating Docker image 121 Tags="map[servicefulstepfunction:adc67a77aef22b6dab9c6156d13853e2cfe06488.1544976453]" 122 NFO[0004] Sending build context to Docker daemon 35.43MB 123 INFO[0004] Step 1/5 : FROM alpine:3.8 124 INFO[0004] ---> 196d12cf6ab1 125 INFO[0004] Step 2/5 : RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/* 126 INFO[0004] ---> Using cache 127 INFO[0004] ---> 99402375b7f2 128 INFO[0004] Step 3/5 : ARG SPARTA_DOCKER_BINARY 129 INFO[0004] ---> Using cache 130 INFO[0004] ---> a44d27522c40 131 INFO[0004] Step 4/5 : ADD $SPARTA_DOCKER_BINARY /SpartaServicefull 132 INFO[0005] ---> 87ffd10e9901 133 INFO[0005] Step 5/5 : CMD ["/SpartaServicefull", "fargateTask"] 134 INFO[0005] ---> Running in 0a3b503201c7 135 INFO[0005] Removing intermediate container 0a3b503201c7 136 INFO[0005] ---> 7cb1b2261a92 137 INFO[0005] Successfully built 7cb1b2261a92 138 INFO[0005] Successfully tagged 139 servicefulstepfunction:adc67a77aef22b6dab9c6156d13853e2cfe06488.1544976453 140 ``` 141 142 ## Push to ECR 143 144 The next step is to push the locally built image to the Elastic 145 Container Registry. The push will return either the ECR URL 146 which will be used as Fargate Task [image](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-image) 147 property or an error: 148 149 ```go 150 // Push the image to ECR & store the URL s.t. we can properly annotate 151 // the CloudFormation template 152 ecrURLPush, pushImageErr := spartaDocker.PushDockerImageToECR(buildTag, 153 ecrRepositoryName, 154 awsSession, 155 logger) 156 ``` 157 158 The ECR push URL is stored in the `context` variable so that a downstream 159 Fargate cluster builder knows the image to use: 160 161 ```go 162 context[contextKeyImageURL] = ecrURLPush 163 ``` 164 165 ## State Machine 166 167 The Step Function definition indirectly references the Fargate 168 Task via task specific [parameters](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-ecs.html): 169 170 ```go 171 fargateParams := spartaStep.FargateTaskParameters{ 172 LaunchType: "FARGATE", 173 Cluster: gocf.Ref(resourceNames.ECSCluster).String(), 174 TaskDefinition: gocf.Ref(resourceNames.ECSTaskDefinition).String(), 175 NetworkConfiguration: &spartaStep.FargateNetworkConfiguration{ 176 AWSVPCConfiguration: &gocf.ECSServiceAwsVPCConfiguration{ 177 Subnets: gocf.StringList( 178 gocf.Ref(resourceNames.PublicSubnetAzs[0]).String(), 179 gocf.Ref(resourceNames.PublicSubnetAzs[1]).String(), 180 ), 181 AssignPublicIP: gocf.String("ENABLED"), 182 }, 183 }, 184 } 185 fargateState := spartaStep.NewFargateTaskState("Run Fargate Task", fargateParams) 186 ``` 187 188 The **ECSCluster** and **ECSTaskDefinition** are resources that are provisioned 189 by the `fargateClusterDecorator` decorator function. 190 191 ## Fargate Cluster 192 193 The final step is to provision the ECS cluster that supports the Fargate 194 task. This is encapsulated in the `fargateClusterDecorator` which creates 195 the required set of CloudFormation resources. The set of CloudFormation 196 resource names is represented in the `stackResourceNames` struct: 197 198 ```go 199 type stackResourceNames struct { 200 StepFunction string 201 SNSTopic string 202 ECSCluster string 203 ECSRunTaskRole string 204 ECSTaskDefinition string 205 ECSTaskDefinitionLogGroup string 206 ECSTaskDefinitionRole string 207 VPC string 208 InternetGateway string 209 AttachGateway string 210 RouteViaIgw string 211 PublicRouteViaIgw string 212 ECSSecurityGroup string 213 PublicSubnetAzs []string 214 } 215 ``` 216 217 The [ECS Task Definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html) 218 is of particular interest and is where the inline created **ECR_URL** is used to 219 define a FARGATE task. 220 221 ### ECS Task Definition 222 223 ```go 224 imageURL, _ := context[contextKeyImageURL].(string) 225 if imageURL == "" { 226 return errors.Errorf("Failed to get image URL from context with key %s", 227 contextKeyImageURL) 228 } 229 ... 230 // Create the ECS task definition 231 ecsTaskDefinition := &gocf.ECSTaskDefinition{ 232 ExecutionRoleArn: gocf.GetAtt(resourceNames.ECSTaskDefinitionRole, "Arn"), 233 RequiresCompatibilities: gocf.StringList(gocf.String("FARGATE")), 234 CPU: gocf.String("256"), 235 Memory: gocf.String("512"), 236 NetworkMode: gocf.String("awsvpc"), 237 ContainerDefinitions: &gocf.ECSTaskDefinitionContainerDefinitionList{ 238 gocf.ECSTaskDefinitionContainerDefinition{ 239 Image: gocf.String(imageURL), 240 Name: gocf.String("sparta-servicefull"), 241 Essential: gocf.Bool(true), 242 LogConfiguration: &gocf.ECSTaskDefinitionLogConfiguration{ 243 LogDriver: gocf.String("awslogs"), 244 // Options Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html 245 Options: map[string]interface{}{ 246 "awslogs-region": gocf.Ref("AWS::Region"), 247 "awslogs-group": strings.Join([]string{"", 248 sparta.ProperName, 249 serviceName}, "/"), 250 "awslogs-stream-prefix": serviceName, 251 "awslogs-create-group": "true", 252 }, 253 }, 254 }, 255 }, 256 } 257 ``` 258 259 ## Configuration 260 261 The final step is to provide the three decorators to the 262 [WorkflowHooks](https://godoc.org/github.com/mweagle/Sparta#WorkflowHooks) structure: 263 264 ```go 265 workflowHooks := &sparta.WorkflowHooks{ 266 ServiceDecorators: []sparta.ServiceDecoratorHookHandler{ 267 ecrImageBuilderDecorator("spartadocker"), 268 // Then build the state machine 269 stateMachine.StateMachineDecorator(), 270 // Then the ECS cluster that supports the Fargate task 271 fargateClusterDecorator(resourceNames), 272 }, 273 } 274 ``` 275 276 ## Provisioning 277 278 The provisioning workflow for this service is the same as a Lambda-based one: 279 280 ```shell 281 $ go run main.provision --s3Bucket $MY_S3_BUCKET 282 ``` 283 284 Output: 285 286 287 ``` 288 INFO[0000] ════════════════════════════════════════════════ 289 INFO[0000] ╔═╗╔═╗╔═╗╦═╗╔╦╗╔═╗ Version : 1.8.0 290 INFO[0000] ╚═╗╠═╝╠═╣╠╦╝ ║ ╠═╣ SHA : 597d3ba 291 INFO[0000] ╚═╝╩ ╩ ╩╩╚═ ╩ ╩ ╩ Go : go1.11.1 292 INFO[0000] ════════════════════════════════════════════════ 293 INFO[0000] Service: ServicefulStepFunction 294 LinkFlags= Option=provision UTC="2018-12-16T16:07:31Z" 295 INFO[0000] ════════════════════════════════════════════════ 296 INFO[0000] Using `git` SHA for StampedBuildID 297 Command="git rev-parse HEAD" SHA=adc67a77aef22b6dab9c6156d13853e2cfe06488 298 INFO[0000] Provisioning service 299 BuildID=adc67a77aef22b6dab9c6156d13853e2cfe06488 300 CodePipelineTrigger= 301 InPlaceUpdates=false 302 NOOP=false Tags= 303 WARN[0000] No lambda functions provided to Sparta.Provision() 304 INFO[0000] Verifying IAM Lambda execution roles 305 INFO[0000] IAM roles verified Count=0 306 ``` 307 308 309 ## Result 310 311 The end result is a Step function that uses our `go` binary, Step functions, 312 and SNS rather than Lambda functions: 313 314 