github.com/pbberlin/tools@v0.0.0-20160910141205-7aa5421c2169/dsu/distributed_unancestored/doc.go (about)

     1  /*  Package distributed_unancestored distributes datastore writes
     2  to several entities; reads are compiled and eventual consistent.
     3  
     4  This is an architecture for high frequency updates.
     5  
     6  Data is saved into various shards.
     7  Contention is reduced.
     8  Those shards can be queried by a conventional index.
     9  On top of that there is a memcache buffer.
    10  
    11  This package is an extension of a demonstration
    12  https://developers.google.com/appengine/articles/sharding_counters
    13  
    14  I added memcache buffering.
    15  I employed the memcache CAS algorithm,
    16  	yet the repetitions upon CAS failure still need to be implemented.
    17  
    18  Another todo is updating the sampling into instance memory.
    19  	We should increase the number of shards based on sampled update frequency.
    20  
    21  Sharded counters use *no* ancestor queries.
    22  Instead, they filter on the properties "Name" and "ShardId"
    23  to find the desired subset.
    24  
    25  Consistence is guaranteed only for changes to one entity group,
    26  or to one entity itself.
    27  => Queries for the total might be slightly stale.
    28  
    29  Race condidtiions are nevertheless prevented by using
    30  datastore transactions.
    31  Brief transaction consideration:
    32  
    33  	Transactions Type A comprise a single entity.
    34  
    35  	Transactions Type B comprise *one* entity group.
    36  
    37  	Transactions Type C comprise up to *five* entity groups
    38  		These are then called cross-group transactions (XGTx)
    39  		XGTx can or cannot employ ancestor queries.
    40  		XGTx use optimistic locking - any other transaction on a particiant
    41  			causes failure
    42  
    43  	Use XGTx and Tx to read CONSISTENT sets of values from >1 entities
    44  
    45  	Any get() or query() always return the state at the BEGIN of tx.
    46  	EVEN if the tx itself has deleted or changed a value,
    47  	downstream gets/queries still return the t_start state.
    48  
    49  	RunInTransaction() retries three times, after conflicts
    50  			=> Make transaction function as idempotent as possible
    51  
    52  Back to this package.
    53  Here we span transactions only over ONE entity
    54  for a read-write operation,
    55  while concurrent reads/writes are prevented from racing.
    56  
    57  Following operations are encapsulated into
    58  a transaction without an entity group
    59  		Increment(): Get/Put       numShards
    60  		Increment(): Get/Put       Shard.I
    61  		AdjustShards:() Get/Put
    62  
    63  
    64  */
    65  package distributed_unancestored