github.com/cryptohub-digital/blockbook-fork@v0.0.0-20230713133354-673c927af7f1/docs/rocksdb.md (about)

     1  # Data storage in RocksDB
     2  
     3  **Blockbook** stores data the key-value store [RocksDB](https://github.com/facebook/rocksdb/wiki). As there are multiple indexes, Blockbook uses RocksDB **column families** feature to store indexes separately.
     4  
     5  > The database structure is described in golang pseudo types in the form _(name type)_.
     6  >
     7  > Operators used in the description:
     8  >
     9  > - _->_ mapping from key to value.
    10  > - _\+_ concatenation,
    11  > - _[]_ array
    12  >
    13  > Types used in the description:
    14  >
    15  > - _[]byte_ - variable length array of bytes
    16  > - _[32]byte_ - fixed length array of bytes (32 bytes long in this case)
    17  > - _uint32_ - unsigned integer, stored as array of 4 bytes in big endian\*
    18  > - _vint_, _vuint_ - variable length signed/unsigned int
    19  > - _addrDesc_ - address descriptor, abstraction of an address.
    20  >   For Bitcoin type coins it is the transaction output script, stored as variable length array of bytes.
    21  >   For Ethereum type coins it is fixed size array of 20 bytes.
    22  > - _bigInt_ - unsigned big integer, stored as length of the array (1 byte) followed by array of bytes of big int, i.e. _(int_len byte)+(int_value []byte)_. Zero is stored as one byte of value 0.
    23  > - _float32_ - float32 number stored as _uint32_
    24  > - string - string stored as `(len vuint)+(value []byte)`
    25  
    26  **Database structure:**
    27  
    28  The database structure described here is of Blockbook version **0.4.0** (internal data format version 6).
    29  
    30  The database structure for **Bitcoin type** and **Ethereum type** coins is different. Column families used for both types:
    31  
    32  - default, height, addresses, transactions, blockTxs, fiatRates
    33  
    34  Column families used only by **Bitcoin type** coins:
    35  
    36  - addressBalance, txAddresses
    37  
    38  Column families used only by **Ethereum type** coins:
    39  
    40  - addressContracts, internalData, contracts, functionSignatures, blockInternalDataErrors, addressAliases
    41  
    42  **Column families description:**
    43  
    44  - **default**
    45  
    46    Stores internal state in json format, under the key _internalState_.
    47  
    48    Most important internal state values are:
    49  
    50    - coin - which coin is indexed in DB
    51    - data format version - currently 6
    52    - dbState - closed, open, inconsistent
    53  
    54    Blockbook is checking on startup these values and does not allow to run against wrong coin, data format version and in inconsistent state. The database must be recreated if the internal state does not match.
    55  
    56  - **height**
    57  
    58    Maps _block height_ to _block hash_ and additional data about block.
    59  
    60    ```
    61    (height uint32) -> (hash [32]byte)+(time uint32)+(nr_txs vuint)+(size vuint)
    62    ```
    63  
    64  - **addresses**
    65  
    66    Maps _addrDesc+block height_ to _array of transactions with array of input/output indexes_.
    67  
    68    The _block height_ in the key is stored as bitwise complement ^ of the height to sort the keys in the order from newest to oldest.
    69  
    70    As there can be multiple inputs/outputs for the same address in one transaction, each txid is followed by variable length array of input/output indexes.
    71    The index values in the array are multiplied by two, the last element of the array has the lowest bit set to 1.
    72    Input or output is distinguished by the sign of the index, output is positive, input is negative (by operation bitwise complement ^ performed on the number).
    73  
    74    ```
    75    (addrDesc []byte)+(^height uint32) -> []((txid [32]byte)+[](index vint))
    76    ```
    77  
    78  - **addressBalance** (used only by Bitcoin type coins)
    79  
    80    Maps _addrDesc_ to _number of transactions_, _sent amount_, _total balance_ and a list of _unspent transactions outputs (UTXOs)_, ordered from oldest to newest
    81  
    82    ```
    83    (addrDesc []byte) -> (nr_txs vuint)+(sent_amount bigInt)+(balance bigInt)+
    84                         []((txid [32]byte)+(vout vuint)+(block_height vuint)+(amount bigInt))
    85    ```
    86  
    87  - **txAddresses** (used only by Bitcoin type coins)
    88  
    89    Maps _txid_ to _block height_ and array of _input addrDesc_ with _amounts_ and array of _output addrDesc_ with _amounts_, with flag if output is spent. In case of spent output, _addrDesc_len_ is negative (negative sign is achieved by bitwise complement ^).
    90  
    91    ```
    92    (txid []byte) -> (height vuint)+
    93                     (nr_inputs vuint)+[]((addrDesc_len vuint)+(addrDesc []byte)+(amount bigInt))+
    94                     (nr_outputs vuint)+[]((addrDesc_len vint)+(addrDesc []byte)+(amount bigInt))
    95    ```
    96  
    97  - **addressContracts** (used only by Ethereum type coins)
    98  
    99    Maps _addrDesc_ to _total number of transactions_, _number of non contract transactions_, _number of internal transactions_
   100    and array of _contracts_ with _number of transfers_ of given address.
   101  
   102    ```
   103    (addrDesc []byte) -> (total_txs vuint)+(non-contract_txs vuint)+(internal_txs vuint)+
   104                         []((contractAddrDesc []byte)+(type+4*nr_transfers vuint))+
   105                         <(value bigInt) if ERC20> or
   106                           <(nr_values vuint)+[](id bigInt) if ERC721> or
   107                           <(nr_values vuint)+[]((id bigInt)+(value bigInt)) if ERC1155>
   108    ```
   109  
   110  - **internalData** (used only by Ethereum type coins)
   111  
   112    Maps _txid_ to _type (CALL 0 | CREATE 1)_, _addrDesc of created contract for CREATE type_, array of _type (CALL 0 | CREATE 1 | SELFDESTRUCT 2)_, _from addrDesc_, _to addrDesc_, _value bigInt_ and possible _error_.
   113  
   114    ```
   115    (txid []byte) -> (type+2*nr_transfers vuint)+<(addrDesc []byte) if CREATE>+
   116                     []((type byte)+(fromAddrDesc []byte)+(toAddrDesc []byte)+(value bigInt))+
   117                     (error []byte)
   118    ```
   119  
   120  - **blockTxs**
   121  
   122    Maps _block height_ to data necessary for blockchain rollback. Only last 300 (by default) blocks are kept.
   123    The content of value data differs for Bitcoin and Ethereum types.
   124  
   125    - Bitcoin type
   126  
   127    The value is an array of _txids_ and _input points_ in the block.
   128  
   129    ```
   130    (height uint32) -> []((txid [32]byte)+(nr_inputs vuint)+[]((txid [32]byte)+(index vint)))
   131    ```
   132  
   133    - Ethereum type
   134  
   135    The value is an array of transaction data. For each transaction is stored _txid_,
   136    _from_ and _to_ address descriptors and array of contract transfer infos consisting of
   137    _from_, _to_ and _contract_ address descriptors, _type (ERC20 0 | ERC721 1 | ERC1155 2)_ and value (or list of id+value for ERC1155)
   138  
   139    ```
   140    (height uint32) -> [](
   141                          (txid [32]byte)+(from addrDesc)+(to addrDesc)+(nr_contracts vuint)+
   142                          []((from addrDesc)+(to addrDesc)+(contract addrDesc)+(type byte)+
   143                          <(value bigInt) if ERC20 or ERC721> or
   144                            <(nr_values vuint)+[]((id bigInt)+(value bigInt)) if ERC1155>)
   145                         )
   146    ```
   147  
   148  - **transactions**
   149  
   150    Transaction cache, _txdata_ is generated by coin specific parser function PackTx.
   151  
   152    ```
   153    (txid []byte) -> (txdata []byte)
   154    ```
   155  
   156  - **fiatRates**
   157  
   158    Stored daily fiat rates, one day as one entry.
   159  
   160    ```
   161    (timestamp YYYYMMDDhhmmss) -> (nr_currencies vuint)+[]((currency string)+(rate float32))+
   162                                  (nr_tokens vuint)+[]((tokenContract string)+(tokenRate float32))
   163    ```
   164  
   165  - **contracts** (used only by Ethereum type coins)
   166  
   167    Maps contract _addrDesc_ to information about contract - _name_, _symbol_, _type_ (ERC20,ERC721 or ERC1155), _decimals_, _created_ and _destructed_ in block height
   168  
   169    ```
   170    (addrDesc []byte) -> (name string)+(symbol string)+(type string)+(decimals vuint)+
   171                         (createdInBlock vuint)+(destroyedInBlock vuint)
   172    ```
   173  
   174  - **functionSignatures** (used only by Ethereum type coins)
   175  
   176    Database of four byte signatures downloaded from https://www.4byte.directory/.
   177  
   178    ```
   179    (fourBytes uint32)+(id uint32) -> (signatureName string)+[]((parameter string))
   180    ```
   181  
   182  - **blockInternalDataErrors** (used only by Ethereum type coins)
   183  
   184    Errors when fetching internal data from backend. Stored so that the action can be retried.
   185  
   186    ```
   187    (blockHeight uint32) -> (blockHash [32]byte)+(retryCount byte)+(errorMessage []byte)
   188    ```
   189  
   190  - **addressAliases** (used only by Ethereum type coins)
   191  
   192    Maps _address_ to address ENS name.
   193  
   194    ```
   195    (address []byte) -> (ensName []byte)
   196    ```
   197  
   198  **Note:**
   199  The `txid` field as specified in this documentation is a byte array of fixed size with length 32 bytes (_[32]byte_), however some coins may define other fixed size lengths.