github.com/voedger/voedger@v0.0.0-20240520144910-273e84102129/design/state/ext-engines.md (about)

     1  # Extension Engines
     2  ## Principles
     3  - Extension engine is the component which invokes extensions
     4  - Partition Processor has:
     5    - Extension engine instance(s)
     6    - Extension instance(s)
     7    - Host State (aka State 2.0)
     8  ## Overview
     9  ```mermaid
    10  erDiagram
    11  App ||--|{ AppPartition: has
    12  AppPartition ||--|| CP: has
    13  CP ||--|| HostState: has
    14  CP ||--|{ ExtensionPackage: has
    15  ExtensionPackage ||--|{ ExtensionInstance: has
    16  ExtensionInstance {
    17      string name
    18      string extKind
    19  }
    20  ExtensionInstance }o--|| ExtEngine: invokes
    21  ExtensionPackage ||--|{ ExtEngine: "has (one per engine kind)"
    22  ExtEngine }|--|| HostState: uses
    23  ```
    24  
    25  ## Extension Engine Kinds
    26  Principles:
    27  - Heeus supports different extension engines (Builtin, WASM, LUA)
    28  - Extension engine is constructed with the factory on partition processor initialization stage
    29  
    30  ```mermaid
    31  erDiagram
    32  ExtEngineFactory {
    33      int memLimit
    34  }
    35  ExtEngineFactory ||--|{ ExtEngine: creates
    36  ExtEngine ||--|| WasmEngine: "can be"
    37  ExtEngine ||--|| LuaEngine: "can be"
    38  ExtEngine ||--|| BuiltinEngine: "can be"
    39  WasmEngine ||--|| WasmRuntimeInstance: has
    40  WasmEngine ||--|| WasmState: has
    41  WasmEngine ||--|| ProxyWasm: has
    42  WasmEngine ||--|| ClientMemory: has
    43  WasmEngine ||--|{ UserFunctions: has
    44  ```
    45  
    46  
    47  # WasmEngine
    48  
    49  Principles:
    50  - WasmEngine communicates with User Functions and WasmState through number of functions exported by WASM module (ProxyWasm)
    51  - To communicate with HostState, WasmState uses number of functions implemented by Host and imported by WASM module (ProxyHost)
    52  
    53  ```mermaid
    54  flowchart TD
    55      subgraph Host
    56      ProxyHost
    57      WasmEngine
    58      HostState
    59      ProxyHost-->HostState
    60      WasmEngine-->HostState
    61      end
    62      subgraph WASM
    63      UserFunctions-->WasmState
    64      ProxyWasm-->UserFunctions
    65      WasmState
    66      ProxyWasm
    67      end
    68      WasmEngine-->ProxyWasm
    69      WasmState-->ProxyHost
    70      ProxyWasm-->WasmState
    71  ```
    72  
    73  # WasmEngine: Lifecycle
    74  ```mermaid
    75  sequenceDiagram
    76      participant VVM
    77      participant CP
    78      participant ExtInstance
    79      participant WasmEngine
    80      participant ProxyWasm
    81      VVM->>ExtInstance: Create Instance
    82      VVM->>WasmEngine: Create Instance
    83      WasmEngine->>ProxyWasm: WasmAbiVersion()
    84      WasmEngine->>ProxyWasm: WasmInit()
    85      VVM->>CP: Create Instance (ExtInstances, WasmEngine)
    86      loop commands
    87      CP->>ExtInstance: Invoke(args)
    88      ExtInstance->>WasmEngine: Invoke(name, args)
    89      WasmEngine->>ProxyWasm: WasmRun()
    90      WasmEngine->>ProxyWasm: WasmReset()
    91      end
    92      VVM->>WasmEngine: Finit()
    93      WasmEngine->>ProxyWasm: WasmFinit()
    94  ```
    95  
    96  # ProxyWasm ABI: Detailed
    97  ```mermaid
    98  sequenceDiagram
    99      participant ExtInstance
   100      participant WasmEngine
   101      participant HostState
   102      participant ProxyWasm
   103      participant ProxyHost
   104      participant WasmState
   105      participant WasmExt
   106      ExtInstance->>+WasmEngine: Invoke(name, args)
   107      WasmEngine->>+ProxyWasm: WasmRun(sz_name, kind, arg_ptr)
   108      ProxyWasm-->>WasmState: Reset
   109      ProxyWasm->>WasmExt: UserFunction()
   110      activate WasmExt
   111      WasmExt->>WasmState: KeyBuilder(storage, ...)
   112      WasmState->>WasmExt: WasmKeyBuilder
   113      note right of WasmExt: fill key
   114      WasmExt->>+WasmState: MustExist(key)
   115      WasmState->>+ProxyHost: HostGetElem(key_ptr)
   116      ProxyHost->>HostState: KeyBuilder(storage, ...)
   117      ProxyHost->>HostState: MustExist(key)
   118      HostState->>ProxyHost: IStateElement
   119      ProxyHost->>-WasmState: elem_ptr
   120      WasmState->>-WasmExt: WasmStateElement
   121      WasmExt->>WasmState: ValueBuilder(key)
   122      WasmState->>WasmExt: WasmElementBuilder
   123      note right of WasmExt: fill element
   124      WasmExt->>ProxyWasm: Done Run()
   125      deactivate WasmExt
   126      ProxyWasm->>-WasmEngine: Done WasmRun *intents
   127      WasmEngine->>HostState: NewValue(key1)
   128      WasmEngine->>HostState: NewValue(key2)
   129      WasmEngine->>+ProxyWasm: WasmReset()
   130      ProxyWasm->>+WasmState: Reset()
   131      ProxyWasm->>-WasmEngine: Done WasmReset()
   132      WasmEngine->>-ExtInstance: Done Invoke
   133      ExtInstance->>HostState: Apply
   134  
   135  ```
   136  
   137  # ProxyWasm ABI: Read Detailed
   138  ```mermaid
   139  sequenceDiagram
   140      participant HostState
   141      participant ProxyWasmHost
   142      participant ProxyWasm
   143      participant WasmState
   144      participant WasmExt
   145      WasmExt->>WasmState: KeyBuilder(storage, ...)
   146      WasmState->>WasmExt: WasmKeyBuilder
   147      note right of WasmExt: fill key
   148      WasmExt->>+WasmState: Read(ctx_id, key)
   149      WasmState->>+ProxyWasmHost: HostReadElems(ctx_id, key_ptr)
   150      ProxyWasmHost->>HostState: KeyBuilder(storage, ...)
   151      ProxyWasmHost->>HostState: Read(key, callback)
   152      HostState->>ProxyWasmHost: IStateElement
   153      ProxyWasmHost->>ProxyWasm: WasmOnRead(ctx_id, key_ptr, elem_ptr)
   154      ProxyWasm->>WasmExt: OnStateRead(ctx_id, key, value)
   155      HostState->>ProxyWasmHost: IStateElement
   156      ProxyWasmHost->>-ProxyWasm: WasmOnRead(ctx_id, key_ptr, elem_ptr)
   157      ProxyWasm->>WasmExt: OnStateRead(ctx_id, key, value)
   158      WasmState->>-WasmExt: Done Read
   159  
   160  ```
   161  
   162  
   163  
   164  # Wasm ABI
   165  The list of functions to be exported by WASM:
   166  ```go
   167      WasmAbiVersion_X_Y()
   168  
   169      // Called by host when client is initialized
   170      WasmInit()
   171  
   172      // Called by host to run the extension
   173      // sz_name and arg_ptr must be released by host when ClientRun is finished
   174      // ext_kind: CmdFunction, QueryFunction, Validator, Projector, Etc
   175      WasmRun(sz_name int32, ext_kind int32 arg_ptr int32) (result int32)
   176  
   177      // Called by host to allocate memory on client
   178      WasmMalloc(size int32) (ptr int32)
   179  
   180      // Called by host after Run
   181      WasmReset()
   182  
   183      // Called by host when client is finalized
   184      // Must cleanup the resources
   185      WasmFinit()
   186  
   187      // Called by HostReadElems
   188      WasmOnRead(ctx_id int32, key_ptr int32, element_ptr int32)
   189  ```
   190  
   191  The list of functions to be imported by WASM:
   192  ```go
   193      // Gets element.
   194      // Host calls ClientMalloc to allocate memory in WASM VM
   195      // Host returns elem_ptr which must be released by client
   196      // Returns:
   197      //   0 - ok, element exists
   198      //   1 - ok, element not exists
   199      //   2 - key validation issue
   200      //   3 - i/o error
   201      //   4 - memory allocation issue
   202      HostGetElem(key_ptr int32, elem_ptr* int32) (result int32)
   203  
   204  
   205      // Read elements and calls ClientOnRead for every element read
   206      // Returns:
   207      //   0 - ok
   208      //   2 - key validation issue
   209      //   3 - i/o error
   210      HostReadElems(ctx_id int32, key_ptr int32) (result int32)
   211  ```
   212  
   213  
   214  # References
   215  - [WebAssembly объединит их всех](https://habr.com/ru/post/671048/)