github.com/qubitproducts/logspray@v0.2.14/indexer/doc.go (about)

     1  // Package indexer processing logs and stores them for
     2  // querying. This is probably confusing several concerns
     3  // and needs work.
     4  //
     5  // On start up, each indexer is assigned a new UUID.
     6  // The indexer creates shards, collections of shard files that span a liited
     7  // time duration (shardDuration). Each shard created by an indexer is assigned
     8  // a UUID.
     9  // Each incoming stream is assigned a UUID. A sharfile is created for each stream.
    10  // Each message in a stream is assigned its ordinal position in the stream
    11  //
    12  // The dataDir should be laid out as follows:
    13  // dataDir/shardID/streamID.pb.log
    14  package indexer