github.com/ledgerwatch/erigon-lib@v1.0.0/kv/Readme.md (about)

     1  #### `Ethdb` package hold's bouquet of objects to access DB
     2  
     3  Words "KV" and "DB" have special meaning here:
     4  
     5  - KV - key-value-style API to access data: let developer manage transactions, stateful cursors.
     6  - DB - object-oriented-style API to access data: Get/Put/Delete/WalkOverTable/MultiPut, managing transactions
     7    internally.
     8  
     9  So, DB abstraction fits 95% times and leads to more maintainable code - because it looks stateless.
    10  
    11  About "key-value-style": Modern key-value databases don't provide Get/Put/Delete methods,
    12  because it's very hard-drive-unfriendly - it pushes developers do random-disk-access which
    13  is [order of magnitude slower than sequential read](https://www.seagate.com/sg/en/tech-insights/lies-damn-lies-and-ssd-benchmark-master-ti/).
    14  To enforce sequential-reads - introduced stateful cursors/iterators - they intentionally look as file-api:
    15  open_cursor/seek/write_data_from_current_position/move_to_end/step_back/step_forward/delete_key_on_current_position/append.
    16  
    17  ## Class diagram:
    18  
    19  ```asciiflow.com
    20  // This is not call graph, just show classes from low-level to high-level. 
    21  // And show which classes satisfy which interfaces.
    22  
    23  +-----------------------------------+   +-----------------------------------+ 
    24  |  github.com/erigonteh/mdbx-go    |   | google.golang.org/grpc.ClientConn |                    
    25  |  (app-agnostic MDBX go bindings)  |   | (app-agnostic RPC and streaming)  |
    26  +-----------------------------------+   +-----------------------------------+
    27                    |                                      |
    28                    |                                      |
    29                    v                                      v
    30  +-----------------------------------+   +-----------------------------------+
    31  |       ethdb/kv_mdbx.go            |   |       ethdb/kv_remote.go          |                
    32  |  (tg-specific MDBX implementaion) |   |   (tg-specific remote DB access)  |              
    33  +-----------------------------------+   +-----------------------------------+
    34                    |                                      |
    35                    |                                      |
    36                    v                                      v    
    37  +----------------------------------------------------------------------------------------------+
    38  |                                       eth/kv_interface.go                                   |  
    39  |         (Common KV interface. DB-friendly, disk-friendly, cpu-cache-friendly.                |
    40  |           Same app code can work with local or remote database.                              |
    41  |           Allows experiment with another database implementations.                           |
    42  |          Supports context.Context for cancelation. Any operation can return error)           |
    43  +----------------------------------------------------------------------------------------------+
    44  
    45  Then:
    46  turbo/snapshotsync/block_reader.go.go
    47  erigon-lib/state/aggregator_v3.go
    48  
    49  Then:
    50  kv_temporal.go
    51  
    52  ```
    53  
    54  ## ethdb.AbstractKV design:
    55  
    56  - InMemory, ReadOnly: `NewMDBX().Flags(mdbx.ReadOnly).InMem().Open()`
    57  - MultipleDatabases, Customization: `NewMDBX().Path(path).WithBucketsConfig(config).Open()`
    58  
    59  
    60  - 1 Transaction object can be used only within 1 goroutine.
    61  - Only 1 write transaction can be active at a time (other will wait).
    62  - Unlimited read transactions can be active concurrently (not blocked by write transaction).
    63  
    64  
    65  - Methods db.Update, db.View - can be used to open and close short transaction.
    66  - Methods Begin/Commit/Rollback - for long transaction.
    67  - it's safe to call .Rollback() after .Commit(), multiple rollbacks are also safe. Common transaction patter:
    68  
    69  ```
    70  tx, err := db.Begin(true, ethdb.RW)
    71  if err != nil {
    72      return err
    73  }
    74  defer tx.Rollback() // important to avoid transactions leak at panic or early return
    75  
    76  // ... code which uses database in transaction
    77   
    78  err := tx.Commit()
    79  if err != nil {
    80      return err
    81  }
    82  ```
    83  
    84  - No internal copies/allocations. It means: 1. app must copy keys/values before put to database. 2. Data after read from
    85    db - valid only during current transaction - copy it if plan use data after transaction Commit/Rollback.
    86  - Methods .Bucket() and .Cursor(), can’t return nil, can't return error.
    87  - Bucket and Cursor - are interfaces - means different classes can satisfy it: for example `MdbxCursor`
    88    and `MdbxDupSortCursor` classes satisfy it.
    89    If your are not familiar with "DupSort" concept, please read [dupsort.md](https://github.com/ledgerwatch/erigon/blob/devel/docs/programmers_guide/dupsort.md)
    90  
    91  
    92  
    93  - If Cursor returns err!=nil then key SHOULD be != nil (can be []byte{} for example).
    94    Then traversal code look as:
    95  
    96  ```go
    97  for k, v, err := c.First(); k != nil; k, v, err = c.Next() {
    98  if err != nil {
    99  return err
   100  }
   101  // logic
   102  }
   103  ``` 
   104  
   105  - Move cursor: `cursor.Seek(key)`
   106  
   107  ## ethdb.Database design:
   108  
   109  - Allows pass multiple implementations
   110  - Allows traversal tables by `db.Walk`
   111  
   112  ## ethdb.TxDb design:
   113  
   114  - holds inside 1 long-running transaction and 1 cursor per table
   115  - method Begin DOESN'T create new TxDb object, it means this object can be passed into other objects by pointer,
   116    and high-level app code can start/commit transactions when it needs without re-creating all objects which holds
   117    TxDb pointer.
   118  - This is reason why txDb.CommitAndBegin() method works: inside it creating new transaction object, pinter to TxDb stays
   119    valid.
   120  
   121  ## How to dump/load table
   122  
   123  Install all database tools: `make db-tools`
   124  
   125  ```
   126  ./build/bin/mdbx_dump -a <datadir>/erigon/chaindata | lz4 > dump.lz4
   127  lz4 -d < dump.lz4 | ./build/bin/mdbx_load -an <datadir>/erigon/chaindata
   128  ```
   129  
   130  ## How to get table checksum
   131  
   132  ```
   133  ./build/bin/mdbx_dump -s table_name <datadir>/erigon/chaindata | tail -n +4 | sha256sum # tail here is for excluding header 
   134  
   135  Header example:
   136  VERSION=3
   137  geometry=l268435456,c268435456,u25769803776,s268435456,g268435456
   138  mapsize=756375552
   139  maxreaders=120
   140  format=bytevalue
   141  database=TBL0001
   142  type=btree
   143  db_pagesize=4096
   144  duplicates=1
   145  dupsort=1
   146  HEADER=END
   147  ```