github.com/aarzilli/tools@v0.0.0-20151123112009-0d27094f75e0/dsu/doc.go (about) 1 // Package dsu contains data store utilities; 2 // formeost for the google app engine datastore. 3 // 4 // Package dsu - datastore utiltiies for google's appengine datastore 5 // with memcache buffering. 6 // 7 // Most important feature is the buffering of every get or put 8 // to memcache. 9 // 10 // Another feature is the fairly generic interface and the 11 // generic fields, which can handle images or other binary data. 12 // 13 // github.com/pbberlin/charting.Save/GetImageToDatastore() 14 // show how an expensively rendered image 15 // is persisted to the datastore. 16 // 17 // github.com/pbberlin/big_query.Save/GetChartDataToDatastore() 18 // demonstrate how to save *any* struct type - 19 // by globbing it into a byte vector 20 // and then wrapping it into a dsu.WrapBlob struct. 21 // 22 // ancestored_gb_entries.saveEntry() and ...ListEntries() demonstrate 23 // retrieval by ancestor query 24 // 25 // dsu_persistent_cursor.guestViewCursor demonstrates scanning 26 // a "table" using a serializable cursor. 27 // 28 // dsu_ancestored_urls.save...() and ...list...() 29 // detail how to insert and retrieve consistently or quickly - 30 // using ancestored queries or not 31 // 32 // dsu_distributed_unancestored.Count() and ...Increment() demonstrate 33 // how to distribute ancestored-updates 34 // to several "shards". 35 // 36 // 37 // A daring feature is the memoryInstanceStore. 38 // Data flows now as follows: 39 // 40 // instance[1]Memory < 41 // instance[2]Memory < memCache < dataStore < bigQueryDB 42 // instance[3]Memory < 43 // 44 // The performance characteristics of the layers can be 45 // seen from google's public monitoring service 46 // https://code.google.com/status/appengine: 47 // 48 // instance[x]Memory < memCache < dataStore < bigQueryDB 49 // 0 ms > 2-5 ms < 20-80 ms < ... 50 // 51 // 52 // The instance memory saves 2-5 ms 53 // and it reduces consumption of memcache quota. 54 // i.e. it could be efficient for our charts, 55 // because the charts are *large* 56 // and generated only once a month 57 // and requested lots of times. 58 // It is not limited in size. 59 // 60 // 61 // We want to parametrize the instance memory caching. 62 // We want to parametrize reaching through to memcache. 63 // to avoid stale data at any cost. 64 // 65 // 66 // The invalidation of the instance memory becomes an issue, 67 // when we have multiple module instances. 68 // 69 // Therefore, upon each BufPut() ... 70 // memoryInstanceStore[key_combi] = newCData 71 // we send a http get message to all instances, including ourselves. 72 // The handler functions invalidate the instance cache. 73 // Each receiver checks the senders instance id - 74 // thus the sender avoids invalidation of itself 75 // 76 // 77 // Furthermore we could look into versioning of datastore entries. 78 package dsu