github.com/Finschia/finschia-sdk@v0.48.1/store/streaming/file/README.md (about)

     1  # File Streaming Service
     2  This pkg contains an implementation of the [StreamingService](../../../baseapp/streaming.go) that writes
     3  the data stream out to files on the local filesystem. This process is performed synchronously with the message processing
     4  of the state machine.
     5  
     6  ## Configuration
     7  
     8  The `file.StreamingService` is configured from within an App using the `AppOptions` loaded from the app.toml file:
     9  
    10  ```toml
    11  [store]
    12      streamers = [ # if len(streamers) > 0 we are streaming
    13          "file", # name of the streaming service, used by constructor
    14      ]
    15  
    16  [streamers]
    17      [streamers.file]
    18          keys = ["list", "of", "store", "keys", "we", "want", "to", "expose", "for", "this", "streaming", "service"]
    19          write_dir = "path to the write directory"
    20          prefix = "optional prefix to prepend to the generated file names"
    21  ```
    22  
    23  We turn the service on by adding its name, "file", to `store.streamers`- the list of streaming services for this App to employ.
    24  
    25  In `streamers.file` we include three configuration parameters for the file streaming service:
    26  1. `streamers.x.keys` contains the list of `StoreKey` names for the KVStores to expose using this service. 
    27  In order to expose *all* KVStores, we can include `*` in this list. An empty list is equivalent to turning the service off.
    28  2. `streamers.file.write_dir` contains the path to the directory to write the files to.
    29  3. `streamers.file.prefix` contains an optional prefix to prepend to the output files to prevent potential collisions
    30  with other App `StreamingService` output files.
    31  
    32  ##### Encoding
    33  
    34  For each pair of `BeginBlock` requests and responses, a file is created and named `block-{N}-begin`, where N is the block number.
    35  At the head of this file the length-prefixed protobuf encoded `BeginBlock` request is written.
    36  At the tail of this file the length-prefixed protobuf encoded `BeginBlock` response is written.
    37  In between these two encoded messages, the state changes that occurred due to the `BeginBlock` request are written chronologically as
    38  a series of length-prefixed protobuf encoded `StoreKVPair`s representing `Set` and `Delete` operations within the KVStores the service
    39  is configured to listen to.
    40  
    41  For each pair of `DeliverTx` requests and responses, a file is created and named `block-{N}-tx-{M}` where N is the block number and M
    42  is the tx number in the block (i.e. 0, 1, 2...).
    43  At the head of this file the length-prefixed protobuf encoded `DeliverTx` request is written.
    44  At the tail of this file the length-prefixed protobuf encoded `DeliverTx` response is written.
    45  In between these two encoded messages, the state changes that occurred due to the `DeliverTx` request are written chronologically as
    46  a series of length-prefixed protobuf encoded `StoreKVPair`s representing `Set` and `Delete` operations within the KVStores the service
    47  is configured to listen to.
    48  
    49  For each pair of `EndBlock` requests and responses, a file is created and named `block-{N}-end`, where N is the block number.
    50  At the head of this file the length-prefixed protobuf encoded `EndBlock` request is written.
    51  At the tail of this file the length-prefixed protobuf encoded `EndBlock` response is written.
    52  In between these two encoded messages, the state changes that occurred due to the `EndBlock` request are written chronologically as
    53  a series of length-prefixed protobuf encoded `StoreKVPair`s representing `Set` and `Delete` operations within the KVStores the service
    54  is configured to listen to.
    55  
    56  ##### Decoding
    57  
    58  To decode the files written in the above format we read all the bytes from a given file into memory and segment them into proto
    59  messages based on the length-prefixing of each message. Once segmented, it is known that the first message is the ABCI request,
    60  the last message is the ABCI response, and that every message in between is a `StoreKVPair`. This enables us to decode each segment into
    61  the appropriate message type.
    62  
    63  The type of ABCI req/res, the block height, and the transaction index (where relevant) is known
    64  from the file name, and the KVStore each `StoreKVPair` originates from is known since the `StoreKey` is included as a field in the proto message.