github.com/kaleido-io/firefly@v0.0.0-20210622132723-8b4b6aacb971/docs/architecture/internal_event_sequencing.md (about)

     1  ---
     2  layout: default
     3  title: Internal Event Sequencing
     4  parent: Architecture
     5  nav_order: 6
     6  ---
     7  
     8  # Internal Event Sequencing
     9  {: .no_toc }
    10  
    11  ## Table of contents
    12  {: .no_toc .text-delta }
    13  
    14  1. TOC
    15  {:toc}
    16  
    17  ---
    18  
    19  ## Overview
    20  
    21  ![Internal Event Sequencing](../images/internal_event_sequencing.jpg "Internal Event Sequencing")
    22  
    23  One of the most important roles FireFly has, is to take actions being performed by the local apps, process them, get them confirmed, and then deliver back 
    24  as "stream of consciousness" to the application alongside all the other events that are coming into the application from other FireFly Nodes in the network.
    25  
    26  You might observe the problems solved in this architecture are similar to those in a message queuing system (like Apache Kafka, or a JMS/AMQP provider like ActiveMQ etc.).
    27  
    28  However, we cannot directly replace the internal logic with such a runtime - because FireFly's job is to aggregate data from multiple runtimes that behave similarly to these:
    29  - Private messaging in the Data Exchange
    30  - The blockchain ledger(s) themselves, which are a stream of sequenced events
    31  - The event dispatcher delivering messages to applications that have been sequenced by FireFly
    32  
    33  So FireFly provides the convenient REST based management interface to simplify the world for application developers, by aggregating the data from multiple locations, and delivering it to apps in a deterministic sequence.
    34  
    35  The sequence is made deterministic:
    36  - Globally to all apps within the scope of the ledger, when a Blockchain ledger is used to pin events (see #10)
    37  - Locally for messages delivered through a single FireFly node into the network
    38  - Locally for all messages delivered to applications connected to a FireFly node, across blockchain 
    39  
    40  ## App Instances
    41  
    42  * Broadcast messages to the network
    43  * Ingest ack when message persisted in local messages table
    44  * Consume events via Websocket connection into FireFly
    45  
    46  ## Outbound Sequencers
    47  
    48  * Broadcast or Private through IPFS or Private Data Storage
    49  * Long-running leader-elected jobs listening to the database (via event tables in SQL, etc.)
    50  
    51  ## Inbound Aggregator 
    52  
    53  * Triggered each time an event is detected by the associated plugin.
    54  * It is the responsibility of the plugin to fire events sequentially.  Can be workload managed but **must** be sequential.
    55  
    56  ### Events Table
    57  
    58  * Deliberately lightweight persisted object, that is generated as a byproduct of other persistent actions.
    59  * Records the local sequence of a specific event within the local node.
    60  * The highest level event type is the confirmation of a message, however the table can be extended for more granularity on event types.
    61  
    62  ## Subscription Manager
    63  
    64  * Responsible for filtering and delivering batches of events to the active event dispatchers.
    65  * Records the latest offset confirmed by each dispatcher.
    66  
    67  ## Event Dispatcher
    68  
    69  * Created with leadership election when WebSocket connection is made from an app into FireFly.
    70  * Extensible to other dispatchers (AMQP, etc.).