github.com/kaleido-io/firefly@v0.0.0-20210622132723-8b4b6aacb971/docs/developers/code_hierarchy.md (about)

     1  ---
     2  layout: default
     3  title: FireFly Code Hierarchy
     4  parent: Developers
     5  nav_order: 2
     6  ---
     7  
     8  # Firefly Code Hierarchy
     9  {: .no_toc }
    10  
    11  ## Table of contents
    12  {: .no_toc .text-delta }
    13  
    14  1. TOC
    15  {:toc}
    16  
    17  ---
    18  
    19  Use the following diagram to better understand the hierarchy amongst the core FireFly components, plugins and utility frameworks:
    20  
    21  ```
    22  ┌──────────┐  ┌───────────────┐  
    23  │ cmd      ├──┤ firefly   [Ff]│  - CLI entry point
    24  └──────────┘  │               │  - Creates parent context
    25                │               │  - Signal handling
    26                └─────┬─────────┘
    27                      │
    28  ┌──────────┐  ┌─────┴─────────┐  - HTTP listener (Gorilla mux)
    29  │ internal ├──┤ api       [As]│    * TLS (SSL), CORS configuration etc.
    30  └──────────┘  │ server        │    * WS upgrade on same port
    31                │               │  - REST route definitions
    32                └─────┬─────────┘    * Simple routing logic only, all processing deferred to orchestrator
    33                      │
    34                ┌─────┴─────────┐  - REST route definition framework
    35                │ openapi   [Oa]│    * Standardizes Body, Path, Query, Filter semantics
    36                │ spec          |      - OpenAPI 3.0 (Swagger) generation
    37                └─────┬─────────┘    * Including Swagger. UI
    38                      │
    39                ┌─────┴─────────┐  - WebSocket server
    40                │           [Ws]│    * Developer friendly JSON based protocol business app development
    41                │ websockets    │    * Reliable sequenced delivery
    42                └─────┬─────────┘    * _Event interface [Ei] supports lower level integration with other compute frameworks/transports_
    43                      │
    44                ┌─────┴─────────┐  - Core data types
    45                │ fftypes   [Ft]│    * Used for API and Serialization
    46                │               │    * APIs can mask fields on input via router definition
    47                └─────┬─────────┘
    48                      │
    49                ┌─────┴─────────┐  - Core runtime server. Initializes and owns instances of:
    50                │           [Or]│    * Components: Implement features
    51    ┌───────┬───┤ orchestrator  │    * Plugins:    Pluggable infrastructure services
    52    │       │   │               │  - Exposes actions to router
    53    │       │   └───────────────┘    * Processing starts here for all API calls
    54    │       │
    55    │  Components: Components do the heavy lifting within the engine
    56    │       │
    57    │       │   ┌───────────────┐  - Maintains a view of the entire network
    58    │       ├───┤ network   [Nm]│    * Integrates with network permissioning [NP] plugin
    59    │       │   │ map           │    * Integrates with broadcast plugin
    60    │       │   └───────────────┘    * Handles hierarchy of member identity, node identity and signing identity
    61    │       │
    62    │       │   ┌───────────────┐  - Broadcast of data to all parties in the network
    63    │       ├───┤ broadcast [Bm]│    * Implements dispatcher for batch component
    64    │       │   │ manager       |    * Integrates with public storage interface [Ps] plugin
    65    │       │   └───────────────┘    * Integrates with blockchain interface [Bi] plugin
    66    │       │
    67    │       │   ┌───────────────┐  - Send private data to individual parties in the network
    68    │       ├───┤ private   [Pm]│    * Implements dispatcher for batch component
    69    │       │   │ messaging     |    * Integrates with the data exchange [Dx] plugin
    70    │       │   └──────┬────────┘    * Messages can be pinned and sequenced via the blockchain, or just sent
    71    │       │          │
    72    │       │   ┌──────┴────────┐  - Groups of parties, with isolated data and/or blockchains
    73    │       │   │ group     [Gm]│    * Integrates with data exchange [Dx] plugin
    74    │       │   │ manager       │    * Integrates with blockchain interface [Bi] plugin
    75    │       │   └───────────────┘
    76    │       │
    77    │       │   ┌───────────────┐  - Private data management and validation
    78    │       ├───┤ data      [Dm]│    * Implements dispatcher for batch component
    79    │       │   │ manager       │    * Integrates with data exchange [Dx] plugin
    80    │       │   └──────┬────────┘    * Integrates with blockchain interface [Bi] plugin
    81    │       │          │
    82    │       │   ┌──────┴────────┐  - JSON data shema management and validation (architecture extensible to XML and more)
    83    │       │   │ json      [Jv]│    * JSON Schema validation logic for outbound and inbound messages
    84    │       │   │ validator     │    * Schema propagatation
    85    │       │   └──────┬────────┘    * Integrates with broadcast plugin
    86    │       │          │
    87    │       │   ┌──────┴────────┐  - Binary data addressable via ID or Hash
    88    │       │   │ blobstore [Bs]│    * Integrates with data exchange [Dx] plugin
    89    │       │   │               │    * Hashes data, and maintains mapping to payload references in blob storage
    90    │       │   └───────────────┘    * Integrates with blockchain interface [Bi] plugin
    91    │       │
    92    │       │   ┌───────────────┐  - Private data management and validation
    93    │       ├───┤ event     [Em]│    * Implements dispatcher for batch component
    94    │       │   │ manager       │    * Integrates with data exchange [Dx] plugin
    95    │       │   └──────┬────────┘    * Integrates with blockchain interface [Bi] plugin
    96    │       │          │
    97    │       │   ┌──────┴────────┐  - Handles incoming external data
    98    │       │   │           [Ag]│    * Integrates with data exchange [Dx] plugin
    99    │       │   │ aggregator    │    * Integrates with public storage interface [Ps] plugin
   100    │       │   │               │    * Integrates with blockchain interface [Bi] plugin
   101    │       │   │               │  - Ensures valid events are dispatched only once all data is available
   102    │       │   └──────┬────────┘    * Context aware, to prevent block-the-world scenarios
   103    │       │          │
   104    │       │   ┌──────┴────────┐  - Subscription manager
   105    │       │   │           [Sm]│    * Creation and management of subscriptions
   106    │       │   │ subscription  │    * Creation and management of subscriptions
   107    │       │   │ manager       │    * Message to Event matching logic
   108    │       │   └──────┬────────┘
   109    │       │          │
   110    │       │   ┌──────┴────────┐  - Manages delivery of events to connected applications
   111    │       │   │ event     [Ed]│    * Integrates with data exchange [Dx] plugin
   112    │       │   │ dispatcher    │    * Integrates with blockchain interface [Bi] plugin
   113    │       │   └───────────────┘
   114    │       │
   115    │       │   ┌───────────────┐  - Token operations
   116    │       ├───┤ asset     [Am]│    * NFT coupling with contexts
   117    │       │   │ manager       │    * Transfer coupling with data describing payment reason
   118    │       │   │               │  - ...
   119    │       │   └───────────────┘
   120    │       │
   121    │       │   ┌───────────────┐  - Aggregates messages and data, with rolled up hashes for pinning
   122    │       ├───┤ batch     [Ba]│    * Pluggable dispatchers
   123    │       │   │ manager       │  - Database decoupled from main-line API processing
   124    │       │   │               │    * See architecture diagrams for more info on active/active sequencing
   125    │       │   └──────┬────────┘  - Manages creation of batch processor instances
   126    │       │          │
   127    │       │   ┌──────┴────────┐  - Short lived agent spun up to assemble batches on demand
   128    │       │   │ batch     [Bp]│    * Coupled to an author+type of messages
   129    │       │   │ processor     │  - Builds batches of 100s messages for efficient pinning
   130    │       │   │               │    * Aggregates messages and data, with rolled up hashes for pinning
   131    │       │   └───────────────┘  - Shuts down automatically after a configurable inactivity period
   132    │       ... more TBD
   133    │
   134  Plugins: Each plugin comprises a Go shim, plus a remote agent microservice runtime (if required)
   135    │
   136    │           ┌───────────────┐  - Blockchain Interface
   137    ├───────────┤           [Bi]│    * Transaction submission - including signing key management
   138    │           │ blockchain    │    * Event listening
   139    │           │ interface     │    * Standardized operations, and custom on-chain coupling
   140    │           └─────┬─────────┘
   141    │                 │
   142    │                 ├─────────────────────┬───────────────────┬────────────────────┐
   143    │           ┌─────┴─────────┐   ┌───────┴───────┐   ┌───────┴────────┐   ┌───────┴────────┐
   144    │           │ ethereum      │   │ corda         │   │ fabric         │   │ utdbql [1]     │
   145    │           └───────────────┘   └───────────────┘   └────────────────┘   └────────────────┘
   146    │
   147    │           ┌───────────────┐  - P2P Content Addresssed Filesystem
   148    ├───────────┤ public    [Pi]│    * Payload upload / download
   149    │           │ storage       │    * Payload reference management
   150    │           │ interface     │
   151    │           └─────┬─────────┘
   152    │                 │
   153    │                 ├───────── ... extensible to any shared storage sytem, accessible to all members
   154    │           ┌─────┴─────────┐
   155    │           │ ipfs          │
   156    │           └───────────────┘
   157    │
   158    │           ┌───────────────┐  - Private Data Exchange
   159    ├───────────┤ data      [Dx]│    * Blob storage
   160    │           │ exchange      │    * Private secure messaging
   161    │           └─────┬─────────┘    * Secure file transfer
   162    │                 │
   163    │                 ├─────────────────────┬────────── ... extensible to any private data exchange tech
   164    │           ┌─────┴─────────┐   ┌───────┴───────┐
   165    │           │ httpdirect    │   │ kaleido       │
   166    │           └───────────────┘   └───────────────┘
   167    │
   168    │           ┌───────────────┐  - Identity resolution and mapping
   169    ├───────────┤ identity  [Ii]│    * Resolves opaque identifiers used throughout FireFly
   170    │           │ interface     │    * Maps to and from blockchain signing identities
   171    │           └─────┬─────────┘    * Map API/user identities from authentication, to network/organizational identities
   172    │                 │
   173    │                 ├───────────── ... extensible to DIDs etc.
   174    │           ┌─────┴─────────┐
   175    │           │ onchain       │
   176    │           └───────────────┘
   177    │
   178    │           ┌───────────────┐  - API Authentication and Authorization Interface
   179    ├───────────┤ api auth  [Aa]│    * Authenticates security credentials (OpenID Connect id token JWTs etc.)
   180    │           │               │    * Extracts API/user identity (for identity interface to map)
   181    │           └─────┬─────────┘    * Enforcement point for fine grained API access control
   182    │                 │
   183    │                 ├─────────────────────┬────────── ... extensible other single sign-on technologies
   184    │           ┌─────┴─────────┐   ┌───────┴───────┐
   185    │           │ apikey        │   │ jwt           │
   186    │           └───────────────┘   └───────────────┘
   187    │
   188    │           ┌───────────────┐  - Database Interactions
   189    ├───────────┤ database  [Di]│    * Create, Read, Update, Delete (CRUD) actions
   190    │           │ interace      │    * Filtering and update definition interace
   191    │           └─────┬─────────┘    * Migrations and Indexes
   192    │                 │
   193    │                 ├───────── ... extensible to NoSQL (CouchDB / MongoDB etc.)
   194    │           ┌─────┴─────────┐
   195    │           │ sqlcommon     │
   196    │           └─────┬─────────┘
   197    │                 ├─────────────────────┬───────────────────┐
   198    │           ┌─────┴─────────┐   ┌───────┴───────┐   ┌───────┴────────┐
   199    │           │ postgres      │   │ ql            │   │ sqlite         │
   200    │           └───────────────┘   └───────────────┘   └────────────────┘
   201    │
   202    │           ┌───────────────┐  - Connects the core event engine to external frameworks and applications
   203    ├───────────┤ event     [Ei]│    * Supports long-lived (durable) and ephemeral event subscriptions
   204    │           │ interface     │    * Batching, filtering, all handled in core prior to transport
   205    │           └─────┬─────────┘    * Interface supports connect-in (websocket) and connect-out (broker runtime style) plugins
   206    │                 │
   207    │                 ├───────── ... extensible to integrate off-chain compute framework (Hyperledger Avalon, TEE, ZKP, MPC etc.)
   208    │                 │          ... extensible to additional event delivery brokers/subsystems (Webhooks, Kafka, AMQP etc.)
   209    │           ┌─────┴─────────┐
   210    │           │ websockets    │
   211    │           └───────────────┘
   212    │  ... more TBD
   213  
   214    Additional utility framworks
   215                ┌───────────────┐  - REST API client
   216                │ rest      [Re]│    * Provides convenience and logging
   217                │ client        │    * Standardizes auth, config and retry logic
   218                └───────────────┘    * Built on Resty
   219  
   220                ┌───────────────┐  - WebSocket client
   221                │ wsclient  [Wc]│    * Provides convenience and logging
   222                │               │    * Standardizes auth, config and reconnect logic
   223                └───────────────┘    * Built on Gorilla WebSockets
   224  
   225                ┌───────────────┐  - Translation framework
   226                │ i18n      [In]│    * Every translations must be added to `en_translations.json` - with an `FF10101` key
   227                │               │    * Errors are wrapped, providing extra features from the `errors` package (stack etc.)
   228                └───────────────┘    * Description translations also supported, such as OpenAPI description
   229  
   230                ┌───────────────┐  - Logging framework
   231                │ log       [Lo]│    * Logging framework (logrus) integrated with context based tagging
   232                │               │    * Context is used throughout the code to pass API invocation context, and logging context
   233                └───────────────┘    * Example: Every API call has an ID that can be traced, as well as a timeout
   234  
   235                ┌───────────────┐  - Configuration
   236                │ config    [Co]│    * File and Environment Variable based logging framework (viper)
   237                │               │    * Primary config keys all defined centrally
   238                └───────────────┘    * Plugins integrate by returning their config structure for unmarshaling (JSON tags)
   239  
   240  [1] The "utdbql" blockchain plugin is a simple standalone ordering service, that uses the an in-process database
   241      It does NOT provide a multi-party blockchain, and does NOT provide broadcast to all members in the network
   242  
   243  ```