github.com/slspeek/camlistore_namedsearch@v0.0.0-20140519202248-ed6f70f7721a/TODO (about) 1 There are two TODO lists. This file (good for airplanes) and the online bug tracker: 2 3 https://code.google.com/p/camlistore/issues/list 4 5 Offline list: 6 7 -- stop using the "cond" blob router storage type in genconfig, as 8 well as the /bs-and-index/ "replica" storage type, and just let the 9 index register its own AddReceiveHook like the sync handler 10 (pkg/server/sync.go). But whereas the sync handler only synchronously 11 _enqueues_ the blob to replicate, the indexer should synchronously 12 do the ReceiveBlob (ooo-reindex) on it too before returning. 13 But the sync handler, despite technically only synchronously-enqueueing 14 and being therefore async, is still very fast. It's likely the 15 sync handler will therefore send a ReceiveBlob to the indexer 16 at the ~same time the indexer is already indexing it. So the indexer 17 should have some dup/merge suppression, and not do double work. 18 singleflight should work. The loser should still consume the 19 source io.Reader body and reply with the same error value. 20 21 -- ditch the importer.Interrupt type and pass along a context.Context 22 instead, which has its Done channel for cancelation. 23 24 -- S3-only mode doesn't work with a local disk index (kvfile) because 25 there's no directory for us to put the kv in. 26 27 -- fault injection many more places with pkg/fault. maybe even in all 28 handlers automatically somehow? 29 30 -- sync handler's shard validation doesn't retry on error. 31 only reports the errors now. 32 33 -- export blobserver.checkHashReader and document it with 34 the blob.Fetcher docs. 35 36 -- "filestogether" handler, putting related blobs (e.g. files) 37 next to each other in bigger blobs / separate files, and recording 38 offsets of small blobs into bigger ones 39 40 -- diskpacked doesn't seem to sync its index quickly enough. 41 A new blob receieved + process exit + read in a new process 42 doesn't find that blob. kv bug? Seems to need an explicit Close. 43 This feels broken. Add tests & debug. 44 45 -- websocket upload protocol. different write & read on same socket, 46 as opposed to HTTP, to have multiple chunks in flight. 47 48 -- extension to blobserver upload protocol to minimize fsyncs: maybe a 49 client can say "no rush" on a bunch of data blobs first (which 50 still don't get acked back over websocket until they've been 51 fsynced), and then when the client uploads the schema/vivivy blob, 52 that websocket message won't have the "no rush" flag, calling the 53 optional blobserver.Storage method to fsync (in the case of 54 diskpacked/localdisk) and getting all the "uploaded" messages back 55 for the data chunks that were written-but-not-synced. 56 57 -- benchmark uploading a 100MB file to localdisk & diskpacked 58 from camput. 59 60 -- measure FUSE operations, latency, round-trips, performance. 61 see next item: 62 63 -- ... we probaby need a "describe all chunks in file" HTTP handler. 64 then FUSE (when it sees sequential access) can say "what's the 65 list of all chunks in this file?" and then fetch them all at once. 66 see next item: 67 68 -- ... HTTP handler to get multiple blobs at once. multi-download 69 in multipart/mime body. we have this for stat and upload, but 70 not download. 71 72 -- ... if we do blob fetching over websocket too, then we can support 73 cancellation of blob requests. Then we can combine the previous 74 two items: FUSE client can ask the server, over websockets, for a 75 list of all chunks, and to also start streaming them all. assume a 76 high-latency (but acceptable bandwidth) link. the chunks are 77 already in flight, but some might be redundant. once the client figures 78 out some might be redundant, it can issue "stop send" messages over 79 that websocket connection to prevent dups. this should work on 80 both "files" and "bytes" types. 81 82 -- cacher: configurable policy on max cache size. clean oldest 83 things (consider mtime+atime) to get back under max cache size. 84 maybe prefer keeping small things (metadata blobs) too, 85 and only delete large data chunks. 86 87 -- UI: video, at least thumbnailing (use external program, 88 like VLC or whatever nautilus uses?) 89 90 -- rename server.ImageHandler to ThumbnailRequest or something? It's 91 not really a Handler in the normal sense. It's not built once and 92 called repeatedly; it's built for every ServeHTTP request. 93 94 -- unexport more stuff from pkg/server. Cache, etc. 95 96 -- look into garbage from openpgp signing 97 98 -- make leveldb memdb's iterator struct only 8 bytes, pointing to a recycled 99 object, and just nil out that pointer at EOF. 100 101 -- bring in the google glog package to third_party and use it in 102 places that want selective logging (e.g. pkg/index/receive.go) 103 104 -- (Mostly done) verify all ReceiveBlob calls and see which should be 105 blobserver.Receive instead, or ReceiveNoHash. git grep -E 106 "\.ReceiveBlob\(" And maybe ReceiveNoHash should go away and be 107 replaced with a "ReceiveString" method which combines the 108 blobref-from-string and ReceiveNoHash at once. 109 110 -- union storage target. sharder can be thought of a specialization 111 of union. sharder already unions, but has a hard-coded policy 112 of where to put new blobs. union could a library (used by sharder) 113 with a pluggable policy on that. 114 115 -- support for running cammount under camlistored. especially for OS X, 116 where the lifetime of the background daemon will be the same as the 117 user's login session. 118 119 -- website: remove the "Installation" heading for /cmd/*, since 120 they're misleading and people should use "go run make.go" in the 121 general case. 122 123 -- website: add godoc for /server/camlistored (also without a "go get" 124 line) 125 126 -- tests for all cmd/* stuff, perhaps as part of some integration 127 tests. 128 129 -- move most of camput into a library, not a package main. 130 131 -- server cron support: full syncs, camput file backups, integrity 132 checks. 133 134 -- status in top right of UI: sync, crons. (in-progress, un-acked 135 problems) 136 137 -- finish metadata compaction on the encryption blobserver.Storage wrapper. 138 139 -- get security review on encryption wrapper. (agl?) 140 141 -- peer-to-peer server and blobserver target to store encrypted blobs 142 on stranger's hardrives. server will be open source so groups of 143 friends/family can run their own for small circles, or some company 144 could run a huge instance. spray encrypted backup chunks across 145 friends' machines, and have central server(s) present challenges to 146 the replicas to have them verify what they have and how big, and 147 also occasionally say what the SHA-1("challenge" + blob-data) is. 148 149 -- sharing: make camget work with permanode sets too, not just 150 "directory" and "file" things. 151 152 -- sharing: when hitting e.g. http://myserver/share/sha1-xxxxx, if 153 a web browser and not a smart client (Accept header? User-Agent?) 154 then redirect or render a cutesy gallery or file browser instead, 155 still with machine-readable data for slurping. 156 157 -- rethink the directory schema so it can a) represent directories 158 with millions of files (without making a >1MB or >16MB schema blob), 159 probably forming a tree, similar to files. but rather than rolling checksum, 160 just split lexically when nodes get too big. 161 162 -- delete mostly-obsolete camsigd. see big TODO in camsigd.go. 163 164 -- we used to be able live-edit js/css files in server/camlistored/ui when 165 running under the App Engine dev_appserver.py. That's now broken with my 166 latest efforts to revive it. The place to start looking is: 167 server/camlistored/ui/fileembed_appengine.go 168 169 -- should a "share" claim be not a claim but its own permanode, so it 170 can be rescinded? right now you can't really unshare a "haveref" 171 claim. or rather, TODO: verify we support "delete" claims to 172 delete any claim, and verify the share system and indexer all 173 support it. I think the indexer might, but not the share system. 174 Also TODO: "camput delete" or "rescind" subcommand. 175 Also TODO: document share claims in doc/schema/ and on website. 176 177 -- make the -transitive flag for "camput share -transitive" be a tri-state: 178 unset, true, false, and unset should then mean default to true for "file" 179 and "directory" schema blobs, and "false" for other things. 180 181 -- index: static directory recursive sizes: search: ask to see biggest directories? 182 183 -- index: index dates in filenames ("yyyy-mm-dd-Foo-Trip", "yyyy-mm blah", etc). 184 185 -- get webdav server working again, for mounting on Windows. This worked before Go 1 186 but bitrot when we moved pkg/fs to use the rsc/fuse. 187 188 -- work on runsit more, so I can start using this more often. runsit should 189 be able to reload itself, and also watch for binaries changing and restart 190 when binaries change. (or symlinks to binaries) 191 192 -- BUG: osutil paths.go on OS X: should use Library everywhere instead of mix of 193 Library and ~/.camlistore? 194 195 OLD: 196 197 -- add CROS support? Access-Control-Allow-Origin: * + w/ OPTIONS 198 http://hacks.mozilla.org/2009/07/cross-site-xmlhttprequest-with-cors/ 199 200 -- brackup integration, perhaps sans GPG? (requires Perl client?) 201 202 -- blobserver: clean up channel-closing consistency in blobserver interface 203 (most close, one doesn't. all should probably close) 204 205 Android: 206 207 [ ] Fix wake locks in UploadThread. need to hold CPU + WiFi whenever 208 something's enqueued at all and we're running. Move out of the Thread 209 that's uploading itself. 210 [ ] GPG signing of blobs (brad) 211 http://code.google.com/p/android-privacy-guard/ 212 http://www.thialfihar.org/projects/apg/ 213 (supports signing in code, but not an Intent?) 214 http://code.google.com/p/android-privacy-guard/wiki/UsingApgForDevelopment 215 ... mailed the author. 216 217 Client libraries: 218 219 [X] Go 220 [X] JavaScript 221 [/] Python (Brett); but see https://github.com/tsileo/camlipy 222 [ ] Perl 223 [ ] Ruby 224 [ ] PHP