github.com/mweagle/Sparta@v1.15.0/docs_source/static/presentations/overview.html (about) 1 <!doctype html> 2 <html lang="en"> 3 4 <head> 5 <meta charset="utf-8"> 6 7 <title>Sparta - Go for AWS Lambda</title> 8 9 <meta name="description" content="A framework for transforming Go applications to AWS Serverless applications"> 10 <meta name="author" content="Matt Weagle"> 11 12 <meta name="apple-mobile-web-app-capable" content="yes"> 13 <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> 14 15 <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> 16 17 <link rel="stylesheet" href="/presentations/reveal.js-3.9.2/css/reveal.css"> 18 <link rel="stylesheet" href="/css/reveal-theme.css" id="theme"> 19 20 <!-- Theme used for syntax highlighting of code --> 21 <link rel="stylesheet" href="/presentations/reveal.js-3.9.2/lib/css/zenburn.css"> 22 23 <!-- Printing and PDF exports --> 24 <script> 25 var link = document.createElement( 'link' ); 26 link.rel = 'stylesheet'; 27 link.type = 'text/css'; 28 link.href = window.location.search.match( /print-pdf/gi ) ? 29 '/presentations/reveal.js-3.9.2/css/print/pdf.css' : 30 '/presentations/reveal.js-3.9.2/css/print/paper.css'; 31 document.getElementsByTagName( 'head' )[0].appendChild( link ); 32 </script> 33 34 <!-- Mermaid .JS --> 35 <script src="/external/jquery/jquery-2.2.0.min.js"></script> 36 <script src="/mermaid/mermaid.js/mermaid.js"></script> 37 <link rel="stylesheet" href="/css/mermaid-overrides.css"> 38 <script>mermaid.initialize({startOnLoad:true});</script> 39 40 <!--[if lt IE 9]> 41 <script src="lib/js/html5shiv.js"></script> 42 <![endif]--> 43 </head> 44 45 <body> 46 47 <div class="reveal"> 48 49 <!-- Any section element inside of this container is displayed as a slide --> 50 <div class="slides"> 51 52 <section data-markdown> 53 <textarea data-template> 54  55 #### A Go Framework for AWS Lambda 56 </textarea> 57 </section> 58 <!-- Introduction --> 59 60 <section data-markdown> 61 <textarea data-template> 62 ### Overview 63 * Concepts 64 * Hello World 65 * AWS Event Sources 66 * API-Gateway 67 * Static Web Sites 68 * Performance 69 * Visualization/testing/developer experience 70 * Observability/monitoring 71 * Alternative topologies 72 * Conclusion: pros/cons 73 </textarea> 74 </section> 75 76 <section> 77 <h3>Opinions</h3> 78 <p class="fragment fade-in">Serverless is the PaaS endgame</p> 79 <i><p class="fragment fade-in">...but does not suit all workloads</p></i> 80 <p class="fragment fade-up">Repo is the unit of collaboration</p> 81 <p class="fragment fade-up">Repo commit is the unit of promotion</p> 82 <p class="fragment fade-up">AWS Lambda is a viable execution platform</p> 83 <p class="fragment fade-up">Responsibilities rather than Roles</p> 84 <p class="fragment fade-up">Go is fantastic</p> 85 86 <aside class="notes"> 87 PaaS: Abstraction over infrastructure, service as first glass 88 Workloads: long running, resource intensive, stable load 89 Responsibilities over roles: Serverless leads to microservices http://philcalcado.com/2017/06/11/calcados_microservices_prerequisites.html and the hierarchy of needs is independent of compute layer: https://thenewstack.io/introducing-microservices-hierarchy-needs/ 90 91 Go: 'nough said. [Ben Kehoe on what's missing](https://www.youtube.com/watch?v=AqhDFbL6Nb4) 92 </aside> 93 </section> 94 95 <section data-markdown> 96 <textarea data-template> 97 <img src="./overview/GOPHER_INCLUSION.png" alt="Drawing" style="width: 50%"/> 98 99 <h5>Image courtesy of <a href="https://github.com/ashleymcnamara/gophers">Ashley McNamara<a/></h5> 100 </textarea> 101 </section> 102 103 <section data-markdown> 104 <textarea data-template> 105 ### Concepts 106 * **ServiceName**: Stable, logical service identifier 107 * **AWS Go Function**: `func(...)` 108 * **Privileges**: Lambda IAM Role Privilege 109 * **Permissions**: Configure AWS event publishers (eg: S3, Dynamo, Kinesis) 110 * **Dynamic Resources**: Other AWS infrastructure provisioned by a Sparta Application 111 </textarea> 112 </section> 113 114 <!-- Setup --> 115 <section data-markdown> 116 <textarea data-template> 117 ### Machine Setup 118 119 * AWS Credentials (`Administrator` privileges) 120 * Writable S3 bucket for code artifacts 121 * Environment variables: 122 * `AWS_REGION` 123 * `AWS_ACCESS_KEY_ID` 124 * `AWS_SECRET_ACCESS_KEY` 125 </textarea> 126 </section> 127 128 <!-- Hello World --> 129 <section data-markdown> 130 <textarea data-template> 131 ## Hello World 132 133 ### Sparta Style 134 </textarea> 135 </section> 136 137 <section data-markdown> 138 <textarea data-template> 139 ```bash 140 $ go get -u -v github.com/mweagle/SpartaHelloWorld 141 $ cd $GOPATH/src/github.com/mweagle/SpartaHelloWorld 142 $ go get -u -v ./... 143 $ go run main.go help 144 ``` 145 </textarea> 146 </section> 147 148 <section data-markdown> 149 <textarea data-template> 150 #### Options 151 ```shell 152 $ go run main.go --help 153 Simple Sparta application that demonstrates 154 core functionality 155 156 Usage: 157 main [command] 158 159 Available Commands: 160 delete Delete service 161 describe Describe service 162 execute Execute 163 explore Interactively explore service 164 help Help about any command 165 provision Provision service 166 version Sparta framework version 167 168 Flags: 169 -f, --format string Log format [text, json] (default "text") 170 -h, --help help for main 171 --ldflags string Go linker string definition flags (https://golang.org/cmd/link/) 172 -l, --level string Log level [panic, fatal, error, warn, info, debug] (default "info") 173 -n, --noop Dry-run behavior only (do not perform mutations) 174 -t, --tags string Optional build tags for conditional compilation 175 176 Use "main [command] --help" for more information about a command. 177 ``` 178 </textarea> 179 </section> 180 181 <section data-markdown data-separator-notes="^Note:"> 182 <textarea data-template> 183 ## Trust 184 <div style="font-size: 96px">位</div> 185 <small> 186 `go run main.go provision --level info --s3Bucket $S3BUCKET --noop` 187 </small> 188 <p /> 189 <small class="fragment fade-up"> 190 `go run main.go provision --level info --s3Bucket $S3BUCKET` 191 </small> 192 193 Note: 194 First run with the --noop flag 195 go run main.go provision --level info --s3Bucket weagle --noop 196 197 </textarea> 198 </section> 199 200 <section data-markdown> 201 <textarea data-template> 202 ### Verify 203 204  205 206 * Login to AWS Console 207 * Navigate to Lambda view, test function 208 * View CloudFormation Log output 209 </textarea> 210 </section> 211 212 <!-- Behind the scenes --> 213 <section data-markdown> 214 <textarea data-template> 215 ### Workflow 216 <a href="https://gosparta.io/sample_service/step2/" target="_blank">go run flow</a> 217 </textarea> 218 </section> 219 220 <!-- S3 Event Sources --> 221 <section data-markdown> 222 <textarea data-template> 223 ### S3 Event Sources 224 </textarea> 225 </section> 226 227 <section data-markdown> 228 <textarea data-template> 229 ```bash 230 $ go get -u -v github.com/mweagle/SpartaImager 231 $ cd $GOPATH/src/github.com/mweagle/SpartaImager 232 $ go run main.go provision --level info --s3Bucket $S3_BUCKET 233 ``` 234 </textarea> 235 </section> 236 237 <section data-markdown> 238 <textarea data-template> 239 #### IAM Role 240 ```golang 241 // Provision an IAM::Role as part of this application 242 var iamRole = sparta.IAMRoleDefinition{} 243 // Setup the ARN that includes all child keys 244 resourceArn := fmt.Sprintf("%s/*", s3EventBroadcasterBucket) 245 iamRole.Privileges = append(iamRole.Privileges, 246 sparta.IAMRolePrivilege{ 247 Actions: []string{"s3:GetObject", "s3:PutObject"}, 248 Resource: resourceArn, 249 }) 250 ``` 251 </textarea> 252 </section> 253 254 <section data-markdown> 255 <textarea data-template> 256 #### Lambda & Permissions 257 ```golang 258 var lambdaFunctions []*sparta.LambdaAWSInfo 259 lambdaFn := sparta.HandleAWSLambda( 260 sparta.LambdaName(transformImage), 261 transformImage, 262 iamRole) 263 // The default timeout is 3 seconds - increase that to 30 264 // so the transform function doesn't timeout 265 lambdaFn.Options = &sparta.LambdaFunctionOptions{ 266 Description: "Stamp assets in S3", 267 MemorySize: 128, 268 Timeout: 30, 269 } 270 // S3 notification configuration 271 lambdaFn.Permissions = append(lambdaFn.Permissions, 272 sparta.S3Permission{ 273 BasePermission: sparta.BasePermission{ 274 SourceArn: s3EventBroadcasterBucket, 275 }, 276 Events: []string{"s3:ObjectCreated:*", 277 "s3:ObjectRemoved:*"}, 278 }) 279 lambdaFunctions = append(lambdaFunctions, lambdaFn) 280 ``` 281 </textarea> 282 </section> 283 284 <section data-markdown data-separator-notes="^Note:"> 285 <textarea data-template> 286 ## Walkthrough 287 288 <div style="font-size: 96px">位</div> 289 Note: 290 * Provision the stack: `go run main.go provision --s3Bucket ${S3_BUCKET}` 291 * Use `aws s3 cp` to upload the ben file to the bucket: `aws s3 cp ./site/ben.jpg s3://$(SPARTA_S3_TEST_BUCKET)/ben.jpg` 292 * Fetch the item JSON info with the /info route: `/info?bucketName=$(SPARTA_S3_TEST_BUCKET)&keyName=xformed_ben.jpg" | jq .` 293 * Download the stamped item using the pre-signed URL 294 Reference: https://github.com/mweagle/SpartaImager/blob/master/README.md 295 </textarea> 296 </section> 297 298 <!-- API Gateway --> 299 <section data-markdown data-separator-notes="^Note:"> 300 <textarea data-template> 301 ### API Gateway 302 303 <img src="./overview/APIGateway.png" alt="Drawing" style="width: 75%"/> 304 305 <small> 306 Source: <a target="_blank" href="https://www.slideshare.net/devopsdaysaustin/2016-serverless-microservices-on-aws-with-api-gateway-and-lambda">Serverless Microservices on AWS</a> 307 </small> 308 309 Note: 310 API Gateway is resource->method->integration->lambda and then back out 311 312 </textarea> 313 </section> 314 315 <!-- Sparta Imager --> 316 <section data-markdown> 317 <textarea data-template> 318 ```bash 319 $ go get -u -v github.com/mweagle/SpartaImager 320 $ cd $GOPATH/src/github.com/mweagle/SpartaImager 321 $ go run main.go provision --level info --s3Bucket $S3_BUCKET 322 ``` 323 </textarea> 324 </section> 325 326 <!-- Code --> 327 <section data-markdown> 328 <textarea data-template> 329 #### API Gateway 330 ```golang 331 apiStage := sparta.NewStage("v1") 332 apiGateway := sparta.NewAPIGateway("SpartaImagerAPI", 333 apiStage) 334 apiGateway.CORSEnabled = true 335 ``` 336 </textarea> 337 </section> 338 339 <section data-markdown> 340 <textarea data-template> 341 #### HTTPS Route 342 ```golang 343 s3ItemInfoLambdaFn := sparta.HandleAWSLambda( 344 sparta.LambdaName(s3ItemInfo), 345 s3ItemInfo, 346 iamDynamicRole) 347 348 // Register the function with the API Gateway 349 apiGatewayResource, _ := api.NewResource("/info", 350 s3ItemInfoLambdaFn) 351 method, err := apiGatewayResource.NewMethod("GET", 352 http.StatusOK) 353 if err != nil { 354 return nil, err 355 } 356 // Whitelist query string params 357 method.Parameters["method.request.querystring.keyName"] = true 358 method.Parameters["method.request.querystring.bucketName"] = true 359 lambdaFunctions = append(lambdaFunctions, s3ItemInfoLambdaFn) 360 ``` 361 </textarea> 362 </section> 363 364 <section data-markdown data-separator-notes="^Note:"> 365 <textarea data-template> 366 ## Walkthrough 367 <div style="font-size: 96px">位</div> 368 369 Note: 370 Follow the Repo steps [README](https://github.com/mweagle/SpartaImager/blob/master/README.md) has an overview of what's going on. 371 - Build the stack 372 - list the bucket: `s3ls $SPARTA_S3_TEST_BUCKET` 373 - Upload the file with aws cp: `aws s3 cp ./site/ben.jpg s3://$SPARTA_S3_TEST_BUCKET/ben.jpg` 374 - `s3ls $SPARTA_S3_TEST_BUCKET` 375 - Hit the API Gateway endpoint to get the presigned URL: `/info?bucketName=$SPARTA_S3_TEST_BUCKET&keyName=xformed_ben.jpg" | jq .` 376 - Download the item with curl 377 - Open it up 378 </textarea> 379 </section> 380 381 <!-- Static Site with CORS --> 382 <!-- API Gateway --> 383 <section data-markdown> 384 <textarea data-template> 385 ### Websites 386 387 * Provision a complete SPA from a repo 388 * Connect API-Gateway with Lambda 389 * Lightweight discovery 390 * [Ben Kehoe on Serverless Service Discovery](https://read.acloud.guru/service-discovery-as-a-service-the-missing-serverless-lynchpin-541d001466f4) 391 </textarea> 392 </section> 393 394 <section data-markdown> 395 <textarea data-template> 396 ```bash 397 $ go get -u -v github.com/mweagle/SpartaHTML 398 $ cd $GOPATH/src/github.com/mweagle/SpartaHTML 399 $ go get -u -v ./... 400 $ go run main.go provision --level info --s3Bucket $S3_BUCKET 401 ``` 402 </textarea> 403 </section> 404 405 <section data-markdown> 406 <textarea data-template> 407 #### Site + API-G 408 ```golang 409 // Register the function with the API Gateway 410 apiStage := sparta.NewStage("v1") 411 apiGateway := sparta.NewAPIGateway("SpartaHTML", apiStage) 412 // Enable CORS s.t. the S3 site can access the resources 413 apiGateway.CORSEnabled = true 414 415 // Provision a new S3 bucket with the resources in the supplied subdirectory 416 s3Site, _ := sparta.NewS3Site("./resources") 417 418 // Deploy it 419 sparta.Main("SpartaHTML", 420 fmt.Sprintf("Sparta app that provisions a CORS-enabled API Gateway together with an S3 site"), 421 spartaLambdaFunctions(apiGateway), 422 apiGateway, 423 s3Site) 424 ``` 425 </textarea> 426 </section> 427 428 <!-- "Discovery" file --> 429 <section data-markdown> 430 <textarea data-template> 431 ### Making Connections 432 433 ```json 434 { 435 "APIGatewayURL": { 436 "Description": "API Gateway URL", 437 "Value": "https://p0ns30kpnd.execute-api.us-west-2.amazonaws.com/v1" 438 } 439 } 440 ``` 441 442 [Manifest.json](https://github.com/mweagle/cloudformationresources/blob/a2efb8eb75fa54dca84d6e63915d4c2cb0210618/zipToS3BucketResource.go#L102) 443 </textarea> 444 </section> 445 446 <!-- Performance --> 447 <section data-markdown> 448 <textarea data-template> 449 ### Performance 450 451 * Developer experience 452 </textarea> 453 </section> 454 455 <section data-markdown> 456 <textarea data-template> 457 ### Developer Experience 458 459 * **standard**: Full CloudFormation `UPDATE_STACK` cycle. 460 * **--inplace**: When safe, concurrent code updates directly via <a href="http://docs.aws.amazon.com/sdk-for-go/api/service/lambda/#Lambda.UpdateFunctionCode">Lambda API</a> 461 </textarea> 462 </section> 463 464 <section data-markdown data-separator-notes="^Note:"> 465 <textarea data-template> 466 ## Walkthrough 467 <div style="font-size: 96px">位</div> 468 469 Note: 470 Use the SpartaHelloWorld app with and without the --inplace argument. Add a new function, try to do an inplace update. 471 </textarea> 472 </section> 473 474 <section data-markdown data-separator-notes="^Note:"> 475 <textarea data-template> 476 ### Testing 477 478 * Use [go test](https://golang.org/pkg/testing/) to test your lambda functions. 479 * See [sparta_test.go](https://github.com/mweagle/Sparta/blob/master/sparta_test.go) 480 * Mock arbitrary event input 481 * Mock API Gateway [requests](https://godoc.org/github.com/mweagle/Sparta/aws/events#NewAPIGatewayMockRequest) 482 483 Note: 484 Sparta shows a sample of this at the root: `go test -v -run TestExplore* .` 485 486 </textarea> 487 </section> 488 489 <section data-markdown> 490 <textarea data-template> 491 ### Observability 492 493 * CloudWatch [Dashboard](https://github.com/mweagle/SpartaXRay), [Metrics](http://gosparta.io/docs/faq/), Logs 494 * [Honeycomb.io](https://github.com/mweagle/SpartaHoneycomb) 495 * [IOPipe](https://www.iopipe.com/) 496 * [AWS Lambda XRay](http://docs.aws.amazon.com/lambda/latest/dg/lambda-x-ray.html#lambda-x-ray-daemon) 497 498 <small><a href="https://speakerdeck.com/smithclay/faas-measurement-fundamentals">Lambda measurement fundamentals</a></small> 499 </textarea> 500 </section> 501 502 <section data-markdown> 503 <textarea data-template> 504 ### CI/CD 505 * Produce a [CodePipeline package](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-action-reference.html) with an optional `--provision` option 506 * Build a CI/CD pipeline with: 507 * [CodePipeline](https://aws.amazon.com/codepipeline/) 508 * [CodeBuild](https://aws.amazon.com/codebuild/) 509 * CloudFormation 510 * See [Serverless, Serverfull, and Weaving Pipelines](https://medium.com/statuscode/serverless-serverfull-and-weaving-pipelines-c9f83eec9227) 511 </textarea> 512 </section> 513 514 <section data-markdown> 515 <textarea data-template> 516 ### Security, Scalability 517 518 * IAMRoles 519 * [SpartaVault](https://github.com/mweagle/SpartaVault) 520 - Use KMS [EnvelopeEncryption](http://docs.aws.amazon.com/kms/latest/developerguide/workflow.html) to manage secrets 521 * [Serverless Artillery](https://github.com/Nordstrom/serverless-artillery) 522 - Lambda max [1000](http://docs.aws.amazon.com/lambda/latest/dg/limits.html) functions 523 - Memory & IO positively correlated 524 - Related: [hey](https://github.com/rakyll/hey), [goad.io](https://goad.io/), [vegeta](https://github.com/tsenart/vegeta) 525 </textarea> 526 </section> 527 528 <section data-markdown> 529 <textarea data-template> 530 ### Alternative Topologies 531 532 <small>Use [TemplateDecorators](https://godoc.org/github.com/mweagle/Sparta#TemplateDecorator) and [WorkflowHooks](https://godoc.org/github.com/mweagle/Sparta#WorkflowHooks) to customize and extend your service</small> 533 534 * [SpartaDocker](https://github.com/mweagle/SpartaDocker) 535 * `Lambda->SQS->ECS Worker Pool` 536 * [Flexibility is a Virtue](https://serverless.zone/flexibility-is-a-virtue-54059d75b1ef) 537 * [SpartaGrafana](https://github.com/mweagle/SpartaGrafana) 538 * `Lambda->Grafana on EC2` 539 * [Serverless Monitoring](https://medium.com/@mweagle/spartagrafana-serverless-monitoring-f86ca6da79ed) 540 541 </textarea> 542 </section> 543 544 <section data-markdown> 545 <textarea data-template> 546 <span style="font-size: 512">馃弳</span> 547 </textarea> 548 </section> 549 550 <!-- CONCLUSION --> 551 <section data-markdown> 552 <textarea data-template> 553 # Pros & Cons 554 </textarea> 555 </section> 556 557 <section data-markdown> 558 <textarea data-template> 559 ## Pros 560 * <span class="fragment fade-in">Serverless can be TTM, financial, and operational win</span> 561 * <span class="fragment fade-in">Go enables developer confidence, team productivity</span> 562 * <span class="fragment fade-in">Sparta supports application evolution: from Serverless, to Container, to VM 563 * Leverage other AWS primitives when needed 564 * Codebase is operationally aware by design 565 </span> 566 </textarea> 567 </section> 568 569 <section data-markdown> 570 <textarea data-template> 571 ## Cons 572 * <span class="fragment fade-in">Go isn't first class</span> <span class="fragment fade-in">- although [binaries are](https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/) 573 * <span class="fragment fade-in">Serverless can be operationally opaque 574 * AWS operational tool usability is uneven 575 * CloudFormation can be verbose</span> 576 * <span class="fragment fade-in">Lacking phased rollout strategies (blue/green, canary)</span> 577 * <span class="fragment fade-in">API-Gateway configuration complexity & performance</span> 578 * <span class="fragment fade-in">Easy to create complex 位 call graphs</span> 579 * <span class="fragment fade-in">Single vendor <a href="https://twitter.com/mweagle/status/913958451474075654">currently</a></span> 580 </textarea> 581 </section> 582 583 <section data-markdown> 584 <textarea data-template> 585 ## Learn More 586 587 * GitHub: <a href="https://github.com/mweagle/Sparta">Sparta</a> 588 * <a href="https://github.com/mweagle/Sparta/blob/master/CHANGES.md">Change Notes</a> 589 * <a href="https://github.com/mweagle?utf8=%E2%9C%93&tab=repositories&q=Sparta&type=&language=">Example Repos<a/> 590 </textarea> 591 </section> 592 593 <!-- THE END --> 594 <section data-markdown> 595 <textarea data-template> 596 ## Sparta 597 598  599 <br /> 600 <smaller>[@mweagle](https://twitter.com/mweagle) on Twitter, [Serverless Slack](https://serverless-forum.slack.com) and [Gophers Slack](https://gophers.slack.com)</smaller> 601 602 <i>Sign Up for Serverless Slack Forum <a href="https://wt-serverless-seattle.run.webtask.io/serverless-forum-signup?webtask_no_cache=1">here</a></i> 603 </textarea> 604 </section> 605 606 <section data-markdown> 607 <textarea data-template> 608 ## PDF 609 Export these slides to [PDF](http://gosparta.io/presentations/training.html?print-pdf) 610 <br /> 611 </textarea> 612 </section> 613 </div> 614 615 </div> 616 617 <script src="/presentations/reveal.js-3.9.2/lib/js/head.min.js"></script> 618 <script src="/presentations/reveal.js-3.9.2/js/reveal.js"></script> 619 620 <script> 621 622 // More info https://github.com/hakimel/reveal.js#configuration 623 Reveal.initialize({ 624 controls: true, 625 progress: true, 626 history: true, 627 center: true, 628 629 transition: 'slide', // none/fade/slide/convex/concave/zoom 630 631 // More info https://github.com/hakimel/reveal.js#dependencies 632 dependencies: [ 633 { src: '/presentations/reveal.js-3.9.2/lib/js/classList.js', condition: function() { return !document.body.classList; } }, 634 { src: '/presentations/reveal.js-3.9.2/plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } }, 635 { src: '/presentations/reveal.js-3.9.2/plugin/markdown/markdown.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } }, 636 { src: '/presentations/reveal.js-3.9.2/plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); } }, 637 { src: '/presentations/reveal.js-3.9.2/plugin/zoom-js/zoom.js', async: true }, 638 { src: '/presentations/reveal.js-3.9.2/plugin/notes/notes.js', async: true } 639 ] 640 }); 641 642 </script> 643 644 </body> 645 </html>