github.com/pbberlin/tools@v0.0.0-20160910141205-7aa5421c2169/os/fsi/dsfs/0_doc.go (about)

     1  // Package dsfs builds a fully distributed
     2  // filesystem layer on top of google datastore.
     3  //
     4  // Architecture
     5  // ==============================
     6  //
     7  // According to http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf
     8  // we must choose the granularity of our entity groups.
     9  //
    10  // We decided on using weakly consistent directory paths.
    11  // Thus, the directory structure can stomach massive updates and inserts.
    12  // But its indexing on property 'dir' may be delayed.
    13  //
    14  // Only each *one* directory is an entity group.
    15  // Applications are forced to partition directories,
    16  // if files *per directory* are changed too frequently.
    17  //
    18  // We cannot allow intermittent "virtual" directories.
    19  // All directories must be explicitly created. Otherwise traversal is impossible.
    20  //
    21  // Direct directory reads are always consistent.
    22  // Only subdirectory queries might be slightly stale.
    23  // Traversals might miss newest directories.
    24  // Traversals might report directories already deleted.
    25  //
    26  //
    27  // In summary: The entire filesystem is extremely parallel,
    28  // and heavily writeable. But it's structural changes
    29  // are not instantly visible to everyone.
    30  //
    31  // Again: Traversal - meaning ReadDir() - is done
    32  // using one global index of the Dir property.
    33  // This index can be queried for equality (direct children),
    34  // or for retrieval of entire subtree.
    35  //
    36  //
    37  // Todo/Consider:
    38  // Add a "block"-layer under file layer,
    39  // so that more than 1MB files can be written?
    40  // At least throw an error before the file is saved?
    41  //
    42  // Mem Caching for files; not just directories - but beware of cost.
    43  //
    44  // Combine with memfs?
    45  // Usage of instance caching with broadcasting instances
    46  // via http request to instances?
    47  //
    48  // Rename is not implemented.
    49  // Rename can be an expensive operation.
    50  //
    51  // RemoveAll and Rename might have to lock
    52  // parts of the filesystem.
    53  // See memfs, for how this could be done.
    54  //
    55  // Nice to have: Links
    56  package dsfs