github.com/cellofellow/gopkg@v0.0.0-20140722061823-eec0544a62ad/database/leveldb.chai2010/doc/log_format.txt (about)

     1  The log file contents are a sequence of 32KB blocks.  The only
     2  exception is that the tail of the file may contain a partial block.
     3  
     4  Each block consists of a sequence of records:
     5     block := record* trailer?
     6     record :=
     7  	checksum: uint32	// crc32c of type and data[] ; little-endian
     8  	length: uint16		// little-endian
     9  	type: uint8		// One of FULL, FIRST, MIDDLE, LAST
    10  	data: uint8[length]
    11  
    12  A record never starts within the last six bytes of a block (since it
    13  won't fit).  Any leftover bytes here form the trailer, which must
    14  consist entirely of zero bytes and must be skipped by readers.  
    15  
    16  Aside: if exactly seven bytes are left in the current block, and a new
    17  non-zero length record is added, the writer must emit a FIRST record
    18  (which contains zero bytes of user data) to fill up the trailing seven
    19  bytes of the block and then emit all of the user data in subsequent
    20  blocks.
    21  
    22  More types may be added in the future.  Some Readers may skip record
    23  types they do not understand, others may report that some data was
    24  skipped.
    25  
    26  FULL == 1
    27  FIRST == 2
    28  MIDDLE == 3
    29  LAST == 4
    30  
    31  The FULL record contains the contents of an entire user record.
    32  
    33  FIRST, MIDDLE, LAST are types used for user records that have been
    34  split into multiple fragments (typically because of block boundaries).
    35  FIRST is the type of the first fragment of a user record, LAST is the
    36  type of the last fragment of a user record, and MID is the type of all
    37  interior fragments of a user record.
    38  
    39  Example: consider a sequence of user records:
    40     A: length 1000
    41     B: length 97270
    42     C: length 8000
    43  A will be stored as a FULL record in the first block.
    44  
    45  B will be split into three fragments: first fragment occupies the rest
    46  of the first block, second fragment occupies the entirety of the
    47  second block, and the third fragment occupies a prefix of the third
    48  block.  This will leave six bytes free in the third block, which will
    49  be left empty as the trailer.
    50  
    51  C will be stored as a FULL record in the fourth block.
    52  
    53  ===================
    54  
    55  Some benefits over the recordio format:
    56  
    57  (1) We do not need any heuristics for resyncing - just go to next
    58  block boundary and scan.  If there is a corruption, skip to the next
    59  block.  As a side-benefit, we do not get confused when part of the
    60  contents of one log file are embedded as a record inside another log
    61  file.
    62  
    63  (2) Splitting at approximate boundaries (e.g., for mapreduce) is
    64  simple: find the next block boundary and skip records until we
    65  hit a FULL or FIRST record.
    66  
    67  (3) We do not need extra buffering for large records.
    68  
    69  Some downsides compared to recordio format:
    70  
    71  (1) No packing of tiny records.  This could be fixed by adding a new
    72  record type, so it is a shortcoming of the current implementation,
    73  not necessarily the format.
    74  
    75  (2) No compression.  Again, this could be fixed by adding new record types.