github.com/cosmos/cosmos-sdk@v0.50.10/docs/architecture/adr-062-collections-state-layer.md (about)

     1  # ADR 062: Collections, a simplified storage layer for cosmos-sdk modules.
     2  
     3  ## Changelog
     4  
     5  * 30/11/2022: PROPOSED
     6  
     7  ## Status
     8  
     9  PROPOSED - Implemented
    10  
    11  ## Abstract
    12  
    13  We propose a simplified module storage layer which leverages golang generics to allow module developers to handle module
    14  storage in a simple and straightforward manner, whilst offering safety, extensibility and standardisation.
    15  
    16  ## Context
    17  
    18  Module developers are forced into manually implementing storage functionalities in their modules, those functionalities include
    19  but are not limited to:
    20  
    21  - Defining key to bytes formats.
    22  - Defining value to bytes formats.
    23  - Defining secondary indexes.
    24  - Defining query methods to expose outside to deal with storage.
    25  - Defining local methods to deal with storage writing.
    26  - Dealing with genesis imports and exports.
    27  - Writing tests for all the above.
    28  
    29  
    30  This brings in a lot of problems:
    31  - It blocks developers from focusing on the most important part: writing business logic.
    32  - Key to bytes formats are complex and their definition is error-prone, for example:
    33    - how do I format time to bytes in such a way that bytes are sorted?
    34    - how do I ensure when I don't have namespace collisions when dealing with secondary indexes?
    35  - The lack of standardisation makes life hard for clients, and the problem is exacerbated when it comes to providing proofs for objects present in state. Clients are forced to maintain a list of object paths to gather proofs.
    36  
    37  ### Current Solution: ORM
    38  
    39  The current SDK proposed solution to this problem is [ORM](https://github.com/cosmos/cosmos-sdk/blob/main/docs/architecture/adr-055-orm.md).
    40  Whilst ORM offers a lot of good functionality aimed at solving these specific problems, it has some downsides:
    41  - It requires migrations.
    42  - It uses the newest protobuf golang API, whilst the SDK still mainly uses gogoproto. 
    43  - Integrating ORM into a module would require the developer to deal with two different golang frameworks (golang protobuf + gogoproto) representing the same API objects.
    44  - It has a high learning curve, even for simple storage layers as it requires developers to have knowledge around protobuf options, custom cosmos-sdk storage extensions, and tooling download. Then after this they still need to learn the code-generated API.
    45  
    46  ### CosmWasm Solution: cw-storage-plus
    47  
    48  The collections API takes inspiration from [cw-storage-plus](https://docs.cosmwasm.com/docs/1.0/smart-contracts/state/cw-plus/),
    49  which has demonstrated to be a powerful tool for dealing with storage in CosmWasm contracts.
    50  It's simple, does not require extra tooling, it makes it easy to deal with complex storage structures (indexes, snapshot, etc).
    51  The API is straightforward and explicit.
    52  
    53  ## Decision
    54  
    55  We propose to port the `collections` API, whose implementation lives in [NibiruChain/collections](https://github.com/NibiruChain/collections) to cosmos-sdk.
    56  
    57  Collections implements four different storage handlers types:
    58  
    59  - `Map`: which deals with simple `key=>object` mappings.
    60  - `KeySet`: which acts as a `Set` and only retains keys and no object (usecase: allow-lists).
    61  - `Item`: which always contains only one object (usecase: Params)
    62  - `Sequence`: which implements a simple always increasing number (usecase: Nonces)
    63  - `IndexedMap`: builds on top of `Map` and `KeySet` and allows to create relationships with `Objects` and `Objects` secondary keys.
    64  
    65  All the collection APIs build on top of the simple `Map` type.
    66  
    67  Collections is fully generic, meaning that anything can be used as `Key` and `Value`. It can be a protobuf object or not.
    68  
    69  Collections types, in fact, delegate the duty of serialisation of keys and values to a secondary collections API component called `ValueEncoders` and `KeyEncoders`.
    70  
    71  `ValueEncoders` take care of converting a value to bytes (relevant only for `Map`). And offers a plug and play layer which allows us to change how we encode objects, 
    72  which is relevant for swapping serialisation frameworks and enhancing performance.
    73  `Collections` already comes in with default `ValueEncoders`, specifically for: protobuf objects, special SDK types (sdk.Int, sdk.Dec).
    74  
    75  `KeyEncoders` take care of converting keys to bytes, `collections` already comes in with some default `KeyEncoders` for some privimite golang types
    76  (uint64, string, time.Time, ...) and some widely used sdk types (sdk.Acc/Val/ConsAddress, sdk.Int/Dec, ...).
    77  These default implementations also offer safety around proper lexicographic ordering and namespace-collision.
    78  
    79  Examples of the collections API can be found here:
    80  - introduction: https://github.com/NibiruChain/collections/tree/main/examples
    81  - usage in nibiru: [x/oracle](https://github.com/NibiruChain/nibiru/blob/master/x/oracle/keeper/keeper.go#L32), [x/perp](https://github.com/NibiruChain/nibiru/blob/master/x/perp/keeper/keeper.go#L31)
    82  - cosmos-sdk's x/staking migrated: https://github.com/testinginprod/cosmos-sdk/pull/22
    83  
    84  
    85  ## Consequences
    86  
    87  ### Backwards Compatibility
    88  
    89  The design of `ValueEncoders` and `KeyEncoders` allows modules to retain the same `byte(key)=>byte(value)` mappings, making
    90  the upgrade to the new storage layer non-state breaking.
    91  
    92  
    93  ### Positive
    94  
    95  - ADR aimed at removing code from the SDK rather than adding it. Migrating just `x/staking` to collections would yield to a net decrease in LOC (even considering the addition of collections itself).
    96  - Simplifies and standardises storage layers across modules in the SDK.
    97  - Does not require to have to deal with protobuf.
    98  - It's pure golang code.
    99  - Generalisation over `KeyEncoders` and `ValueEncoders` allows us to not tie ourself to the data serialisation framework.
   100  - `KeyEncoders` and `ValueEncoders` can be extended to provide schema reflection.
   101  
   102  ### Negative
   103  
   104  - Golang generics are not as battle-tested as other Golang features, despite being used in production right now.
   105  - Collection types instantiation needs to be improved.
   106  
   107  ### Neutral
   108  
   109  {neutral consequences}
   110  
   111  ## Further Discussions
   112  
   113  - Automatic genesis import/export (not implemented because of API breakage)
   114  - Schema reflection
   115  
   116  
   117  ## References