github.com/voedger/voedger@v0.0.0-20240520144910-273e84102129/design/storage/README.md (about)

     1  ### Story
     2  As a Heeus Architect I want to have a description where and how the data is stored in the Federation
     3  
     4  ### Solution Principles
     5  - current solution is actual for Heeus CE and SE only
     6  - Storage could be:
     7    - `AppPartitionStorage`
     8      - cached
     9      - created\released per each app partition deploy\undeploy
    10      - used by processors, actualizers
    11    - `AppStorages`
    12      - uncached
    13      - created\released per app deploy\undeploy
    14      - used by router to manage BLOBs and certificates
    15  - storage driver name and its params are configured via cmd line
    16  - cache principles
    17    - select by an exact id -> cached always
    18    - `IAppStorage.Get()` and `.GetBatch()` cached, `.Read()` - not
    19  
    20  ### Concepts
    21  ```mermaid
    22  erDiagram
    23  Federation ||..|{ Cluster : has
    24  Cluster ||..|{ Application : has
    25  Cluster ||..|| ClusterStorage : has
    26  Application ||..|{ app_AppPartition : has
    27  app_AppPartition ||..|| PLogPartition : uses
    28  app_AppPartition ||..|{ Workspace : uses
    29  ClusterStorage ||..|{ AppStorage : "has Small Object Storage"
    30  ClusterStorage ||..|| BLOBStorage : has
    31  BLOBStorage ||..|| AmazonS3 : like
    32  BLOBStorage ||..|{ BLOB : has
    33  BLOBStorage ||..|| AppStorage : "can use"
    34  BLOB ||..|| Workspace : "linked to"
    35  AppStorage ||..|| PLog : has
    36  PLog ||..|{ PLogPartition: has
    37  PLogPartition ||..|{ Actualizer : "used e.g. by"
    38  AppStorage ||..|{ Workspace : has
    39  Workspace ||..|| WLog : has
    40  Workspace ||..o{ View : has
    41  Workspace ||..o{ Table : has
    42  WLog ||..|{ QueryProcessor : "used e.g. by"
    43  ```
    44  
    45  ### Components
    46  #### Heeus CE and SE executables
    47  ```mermaid
    48  erDiagram
    49  ce_exe ||..|| ce_appStorageProviderFactory : provides
    50  se_exe ||..|| se_appStorageProviderFactory : provides
    51  ce_exe ||..|| ce_cmdLine : reads
    52  se_exe ||..|| se_cmdLine : reads
    53  ce_appStorageProviderFactory ||..|| ce-se_CLI : "passed to"
    54  se_appStorageProviderFactory ||..|| ce-se_CLI : "passed to"
    55  ce_cmdLine ||..|| ce-se_CLI : "passed to"
    56  se_cmdLine ||..|| ce-se_CLI : "passed to"
    57  
    58  ce-se_CLI ||..|| imetrics_Provide : "calls"
    59  imetrics_Provide ||..|| imetrics_IMetrics : "provides"
    60  
    61  ce-se_CLI ||..|| params : "parses cmd line and provides *struct"
    62  params ||..|| storageCacheSize : has
    63  params ||..|| vvmName : has
    64  params ||..|| driverParams : has
    65  params ||..|| appStorageDriverName : has
    66  os_HostName ||..|| vvmName : "is default for"
    67  
    68  driverParams ||..|| appStorageProviderFactory : "used by"
    69  driverParams ||..|| appStorageProviderFactory : "used by"
    70  appStorageDriverName ||..|| appStorageProviderFactory : "used by"
    71  ce-se_CLI ||..|| appStorageProviderFactory : receives
    72  
    73  appStorageProviderFactory ||..|| ce-se_provideIAppStorageProvider : "passed to and called by"
    74  appStorageDriverName ||..|| ce-se_provideIAppStorageProvider : "passed to"
    75  appStorageDriverName ||..|| appStorageProviderFactory : "used by"
    76  driverParams ||..|| ce-se_provideIAppStorageProvider : "passed to"
    77  ce-se_provideIAppStorageProvider ||..|| istorage_IAppStorageProvider : returns
    78  istorage_IAppStorageProvider ||..|| ce-se_NewAppPartitionStorageFactory : "passed to"
    79  imetrics_IMetrics ||..|| ce-se_NewAppPartitionStorageFactory : "passed to"
    80  storageCacheSize ||..|| ce-se_NewAppPartitionStorageFactory : "passed to"
    81  vvmName ||..|| ce-se_NewAppPartitionStorageFactory : "passed to"
    82  storageCacheSize ||..|| ce-se_NewAppPartitionStorageFactory : "passed to"
    83  istorage_IAppStorageProvider ||..|| ce-se_NewAppPartitionStorageFactory : "passed to"
    84  ce-se_NewAppPartitionStorageFactory ||..|| ce-se_AppPartitionStorages : "returns *struct"
    85  ce-se_AppPartitionStorages ||..|| ce-se_AppPartitionStorageFactory : "has func()"
    86  ce-se_AppPartitionStorages ||..|| ce-se_AppStorages : "has *struct"
    87  ce-se_AppPartitionStorageFactory ||..|| ce-se_AppPartitionStorage : returns
    88  ce-se_AppPartitionStorage ||..|| istorage_IAppStorage : "is cached"
    89  ce-se_AppStorages ||..|| istorage_IAppStorage : "has method to return uncached"
    90  ```
    91  #### ce-se package
    92  ##### AppStorages
    93  ```mermaid
    94  erDiagram
    95  ce-se_NewAppPartitionStorageFactory ||..|| ce-se_AppPartitionStorages : "returns *struct"
    96  ce-se_AppPartitionStorages ||..|| ce-se_AppPartitionStorageFactory : has
    97  ce-se_AppPartitionStorages ||..|| ce-se_AppStorages : has
    98  ce-se_AppStorages ||..|| ce-se_provideBLOBStorage : "used by"
    99  ce-se_AppStorages ||..|| ce-se_provideRouterStorage : "used by"
   100  ce-se_provideBLOBStorage ||..|| iblobstoragestg_IBlobberAppStorage : "creates IAppStorage for sys/blobber app"
   101  iblobstoragestg_IBlobberAppStorage ||..|| istorage_IAppStorage : is
   102  ce-se_provideRouterStorage ||..|| dbcertcache_IRouterAppStorage : "creates IAppStorage for sys/router app"
   103  dbcertcache_IRouterAppStorage ||..|| istorage_IAppStorage : is
   104  iblobstoragestg_IBlobberAppStorage ||..|| iblobstoragestg_Provide : "used by"
   105  iblobstoragestg_Provide ||..|| iblobstorage_IBLOBStorage : returns
   106  
   107  dbcertcache_IRouterAppStorage ||..|| dbcertcache_New : "used by"
   108  dbcertcache_New ||..|| autocert_Cache : "returns"
   109  autocert_Cache ||..|| ihttpimpl_httpProcessor : "used for https by"
   110  
   111  iblobstorage_IBLOBStorage ||..|| ihttpimpl_router : "used by"
   112  ihttpimpl_router ||..|| ihttpimpl_httpProcessor : "contained by"
   113  
   114  iblobstorage_IBLOBStorage{
   115    func WriteBLOB
   116    func ReadBLOB
   117    func QueryBLOBState
   118  }
   119  ```
   120  ##### AppPartitionStorage
   121  ```mermaid
   122  erDiagram
   123  ce-se_NewAppPartitionStorageFactory ||..|| istoragecache_New : "uses"
   124  istoragecache_New ||..|| ce-se_AppPartitionStorageFactory : "used to create"
   125  
   126  ce-se_NewAppPartitionStorageFactory ||..|| ce-se_AppPartitionStorages : "returns *struct"
   127  ce-se_AppPartitionStorages ||..|| ce-se_AppPartitionStorageFactory : has
   128  ce-se_AppPartitionStorages ||..|| ce-se_AppStorages : has
   129  
   130  ce-se_AppPartitionStorageFactory ||..|| ce-se_AppPartitionFactory : "used by"
   131  ce-se_AppPartitionFactory ||..|| app_AppPartition : "to build *struct"
   132  app_AppPartition ||..|| AppPartitionStorage : has
   133  AppPartitionStorage ||..|{ QueryProcessor : "used by"
   134  AppPartitionStorage ||..|| CommandProcessor : "used by"
   135  AppPartitionStorage ||..|{ Actualizer : "used by"
   136  AppPartitionStorage ||..|| istorage_IAppStorage : is
   137  ```