go.chromium.org/luci@v0.0.0-20240309015107-7cdc2e660f33/gae/docs/present/lightning.slide (about)

     1  LUCI/gae: App Engine pseudo-SDK
     2  Fast Tests, Slick API, Happy Hacking
     3  18:00 12 Jul 2016
     4  Tags: appengine, datastore, testing
     5  
     6  Robbie Iannucci
     7  Professional Software Wrestler, Google
     8  iannucci@google.com
     9  https://github.com/luci/gae
    10  
    11  Chrome Infrastructure Team
    12  infra-dev@chromium.org
    13  
    14  * Go on App Engine kicks ass!
    15  
    16  : Hello! Robbie Iannucci, work CIT. Use Go/GAE to build CI microservices.
    17  : QUICK GAE overview (PaaS, Datastore K/V, Memcache, load balancing, scaling, etc.)
    18  : Go/GAE is BEST App Engine
    19  : We made testing Go/GAE super fast+fun
    20  
    21  * Trouble in Paradise?
    22  
    23  *Testing*is*slow*
    24  
    25  - Requires SDK+Python (!!)
    26  - Slow (3-4s minumum overhead per datastore instance)
    27  - Accidental serialization (non-concurrent, race detector ~useless)
    28  
    29  *API*is*less*than*ideal*
    30  
    31  Lacks:
    32  
    33  - automatic object memcaching
    34  - hooks
    35  - embedded IDs
    36  - 'mockable' services
    37  
    38  : Go/GAE 2GTTTGT, but hurdles because Go is recently added
    39  : Uses same Python devserver (impl. speed, maint)
    40  : We miss NDB
    41  : So, as any good engineers would do...
    42  
    43  * Solution
    44  
    45  .background lightning/all_the_things.png
    46  
    47  * Solution
    48  
    49  *High-performance*pure-Go*in-memory*GAE-services*implementation*
    50  
    51  - Concurrent
    52  - Accurate Indexes, Query semantics, Query execution
    53  - Accurate Transaction behavior (including transactional Task Queue)
    54  - Fast tests
    55  
    56  *High-level*service*API*
    57  
    58  - Embedded object identity
    59  - Flexible datatypes/encoding
    60  
    61  : So we built a thing! In-memory pure-Go implementation of the datastore (as well
    62  : other App Engine services). Library lets you access prod services with nice
    63  : API, and get our really fast in-memory fakes in your tests.
    64  
    65  * Code + Numbers
    66  
    67  : So enough run-up, let's take a look at some lightning-style code and numbers.
    68  
    69  * Native way (aetest)
    70  
    71  .code lightning/native_test.go /START OMIT/,/END OMIT/
    72  
    73  : Duplication ("Model")
    74  : Key management
    75  : Takes ~8 seconds == yawn.
    76  
    77  ~7.5 seconds
    78  
    79  * luci/gae way
    80  
    81  .code lightning/gae_test.go /START OMIT/,/END OMIT/
    82  
    83  : Shorter! Model, ID part of model. ID properly typed.
    84  : LUCI/gae can dynamically compute ID too (say if it was a composition of A+B)
    85  : 50 Milliseconds.
    86  
    87  ~0.056 seconds
    88  
    89  * >100x speedup! Less redundancy!
    90  
    91  : Hooray! I can run 100 times as many tests in the same amount of time!
    92  
    93  * But wait, there's more!
    94  
    95  * Features
    96  
    97  *We*actually*use*it*
    98  
    99  700 tests (parallel, clean-slate) run in ~35 seconds.
   100  
   101  *Flexible*Filter*Layer*
   102  
   103  - Transparent caching (based on [[https://github.com/qedus/nds]])
   104  - Recursive buffered transactions
   105  - *<your*logic*here>*
   106  
   107  *Controllable*Eventual*Consistency*
   108  
   109  Allows explicit testing around Eventual Consistency states.
   110  
   111  *Consistent*API*
   112  
   113  Classic GAE, Managed VMs, Cloud Datastore, In-Memory
   114  
   115  * Thanks
   116  
   117  This couldn't have been done without amazing Go packages like:
   118  
   119  - [[https://google.golang.org/appengine]] 😀
   120  - [[https://github.com/steveyen/gkvlite]]
   121  - [[https://github.com/mjibson/goon]]
   122  - [[https://github.com/qedus/nds]]