github.com/kaleido-io/firefly@v0.0.0-20210622132723-8b4b6aacb971/docs/developers/code_hierarchy.md (about) 1 --- 2 layout: default 3 title: FireFly Code Hierarchy 4 parent: Developers 5 nav_order: 2 6 --- 7 8 # Firefly Code Hierarchy 9 {: .no_toc } 10 11 ## Table of contents 12 {: .no_toc .text-delta } 13 14 1. TOC 15 {:toc} 16 17 --- 18 19 Use the following diagram to better understand the hierarchy amongst the core FireFly components, plugins and utility frameworks: 20 21 ``` 22 ┌──────────┐ ┌───────────────┐ 23 │ cmd ├──┤ firefly [Ff]│ - CLI entry point 24 └──────────┘ │ │ - Creates parent context 25 │ │ - Signal handling 26 └─────┬─────────┘ 27 │ 28 ┌──────────┐ ┌─────┴─────────┐ - HTTP listener (Gorilla mux) 29 │ internal ├──┤ api [As]│ * TLS (SSL), CORS configuration etc. 30 └──────────┘ │ server │ * WS upgrade on same port 31 │ │ - REST route definitions 32 └─────┬─────────┘ * Simple routing logic only, all processing deferred to orchestrator 33 │ 34 ┌─────┴─────────┐ - REST route definition framework 35 │ openapi [Oa]│ * Standardizes Body, Path, Query, Filter semantics 36 │ spec | - OpenAPI 3.0 (Swagger) generation 37 └─────┬─────────┘ * Including Swagger. UI 38 │ 39 ┌─────┴─────────┐ - WebSocket server 40 │ [Ws]│ * Developer friendly JSON based protocol business app development 41 │ websockets │ * Reliable sequenced delivery 42 └─────┬─────────┘ * _Event interface [Ei] supports lower level integration with other compute frameworks/transports_ 43 │ 44 ┌─────┴─────────┐ - Core data types 45 │ fftypes [Ft]│ * Used for API and Serialization 46 │ │ * APIs can mask fields on input via router definition 47 └─────┬─────────┘ 48 │ 49 ┌─────┴─────────┐ - Core runtime server. Initializes and owns instances of: 50 │ [Or]│ * Components: Implement features 51 ┌───────┬───┤ orchestrator │ * Plugins: Pluggable infrastructure services 52 │ │ │ │ - Exposes actions to router 53 │ │ └───────────────┘ * Processing starts here for all API calls 54 │ │ 55 │ Components: Components do the heavy lifting within the engine 56 │ │ 57 │ │ ┌───────────────┐ - Maintains a view of the entire network 58 │ ├───┤ network [Nm]│ * Integrates with network permissioning [NP] plugin 59 │ │ │ map │ * Integrates with broadcast plugin 60 │ │ └───────────────┘ * Handles hierarchy of member identity, node identity and signing identity 61 │ │ 62 │ │ ┌───────────────┐ - Broadcast of data to all parties in the network 63 │ ├───┤ broadcast [Bm]│ * Implements dispatcher for batch component 64 │ │ │ manager | * Integrates with public storage interface [Ps] plugin 65 │ │ └───────────────┘ * Integrates with blockchain interface [Bi] plugin 66 │ │ 67 │ │ ┌───────────────┐ - Send private data to individual parties in the network 68 │ ├───┤ private [Pm]│ * Implements dispatcher for batch component 69 │ │ │ messaging | * Integrates with the data exchange [Dx] plugin 70 │ │ └──────┬────────┘ * Messages can be pinned and sequenced via the blockchain, or just sent 71 │ │ │ 72 │ │ ┌──────┴────────┐ - Groups of parties, with isolated data and/or blockchains 73 │ │ │ group [Gm]│ * Integrates with data exchange [Dx] plugin 74 │ │ │ manager │ * Integrates with blockchain interface [Bi] plugin 75 │ │ └───────────────┘ 76 │ │ 77 │ │ ┌───────────────┐ - Private data management and validation 78 │ ├───┤ data [Dm]│ * Implements dispatcher for batch component 79 │ │ │ manager │ * Integrates with data exchange [Dx] plugin 80 │ │ └──────┬────────┘ * Integrates with blockchain interface [Bi] plugin 81 │ │ │ 82 │ │ ┌──────┴────────┐ - JSON data shema management and validation (architecture extensible to XML and more) 83 │ │ │ json [Jv]│ * JSON Schema validation logic for outbound and inbound messages 84 │ │ │ validator │ * Schema propagatation 85 │ │ └──────┬────────┘ * Integrates with broadcast plugin 86 │ │ │ 87 │ │ ┌──────┴────────┐ - Binary data addressable via ID or Hash 88 │ │ │ blobstore [Bs]│ * Integrates with data exchange [Dx] plugin 89 │ │ │ │ * Hashes data, and maintains mapping to payload references in blob storage 90 │ │ └───────────────┘ * Integrates with blockchain interface [Bi] plugin 91 │ │ 92 │ │ ┌───────────────┐ - Private data management and validation 93 │ ├───┤ event [Em]│ * Implements dispatcher for batch component 94 │ │ │ manager │ * Integrates with data exchange [Dx] plugin 95 │ │ └──────┬────────┘ * Integrates with blockchain interface [Bi] plugin 96 │ │ │ 97 │ │ ┌──────┴────────┐ - Handles incoming external data 98 │ │ │ [Ag]│ * Integrates with data exchange [Dx] plugin 99 │ │ │ aggregator │ * Integrates with public storage interface [Ps] plugin 100 │ │ │ │ * Integrates with blockchain interface [Bi] plugin 101 │ │ │ │ - Ensures valid events are dispatched only once all data is available 102 │ │ └──────┬────────┘ * Context aware, to prevent block-the-world scenarios 103 │ │ │ 104 │ │ ┌──────┴────────┐ - Subscription manager 105 │ │ │ [Sm]│ * Creation and management of subscriptions 106 │ │ │ subscription │ * Creation and management of subscriptions 107 │ │ │ manager │ * Message to Event matching logic 108 │ │ └──────┬────────┘ 109 │ │ │ 110 │ │ ┌──────┴────────┐ - Manages delivery of events to connected applications 111 │ │ │ event [Ed]│ * Integrates with data exchange [Dx] plugin 112 │ │ │ dispatcher │ * Integrates with blockchain interface [Bi] plugin 113 │ │ └───────────────┘ 114 │ │ 115 │ │ ┌───────────────┐ - Token operations 116 │ ├───┤ asset [Am]│ * NFT coupling with contexts 117 │ │ │ manager │ * Transfer coupling with data describing payment reason 118 │ │ │ │ - ... 119 │ │ └───────────────┘ 120 │ │ 121 │ │ ┌───────────────┐ - Aggregates messages and data, with rolled up hashes for pinning 122 │ ├───┤ batch [Ba]│ * Pluggable dispatchers 123 │ │ │ manager │ - Database decoupled from main-line API processing 124 │ │ │ │ * See architecture diagrams for more info on active/active sequencing 125 │ │ └──────┬────────┘ - Manages creation of batch processor instances 126 │ │ │ 127 │ │ ┌──────┴────────┐ - Short lived agent spun up to assemble batches on demand 128 │ │ │ batch [Bp]│ * Coupled to an author+type of messages 129 │ │ │ processor │ - Builds batches of 100s messages for efficient pinning 130 │ │ │ │ * Aggregates messages and data, with rolled up hashes for pinning 131 │ │ └───────────────┘ - Shuts down automatically after a configurable inactivity period 132 │ ... more TBD 133 │ 134 Plugins: Each plugin comprises a Go shim, plus a remote agent microservice runtime (if required) 135 │ 136 │ ┌───────────────┐ - Blockchain Interface 137 ├───────────┤ [Bi]│ * Transaction submission - including signing key management 138 │ │ blockchain │ * Event listening 139 │ │ interface │ * Standardized operations, and custom on-chain coupling 140 │ └─────┬─────────┘ 141 │ │ 142 │ ├─────────────────────┬───────────────────┬────────────────────┐ 143 │ ┌─────┴─────────┐ ┌───────┴───────┐ ┌───────┴────────┐ ┌───────┴────────┐ 144 │ │ ethereum │ │ corda │ │ fabric │ │ utdbql [1] │ 145 │ └───────────────┘ └───────────────┘ └────────────────┘ └────────────────┘ 146 │ 147 │ ┌───────────────┐ - P2P Content Addresssed Filesystem 148 ├───────────┤ public [Pi]│ * Payload upload / download 149 │ │ storage │ * Payload reference management 150 │ │ interface │ 151 │ └─────┬─────────┘ 152 │ │ 153 │ ├───────── ... extensible to any shared storage sytem, accessible to all members 154 │ ┌─────┴─────────┐ 155 │ │ ipfs │ 156 │ └───────────────┘ 157 │ 158 │ ┌───────────────┐ - Private Data Exchange 159 ├───────────┤ data [Dx]│ * Blob storage 160 │ │ exchange │ * Private secure messaging 161 │ └─────┬─────────┘ * Secure file transfer 162 │ │ 163 │ ├─────────────────────┬────────── ... extensible to any private data exchange tech 164 │ ┌─────┴─────────┐ ┌───────┴───────┐ 165 │ │ httpdirect │ │ kaleido │ 166 │ └───────────────┘ └───────────────┘ 167 │ 168 │ ┌───────────────┐ - Identity resolution and mapping 169 ├───────────┤ identity [Ii]│ * Resolves opaque identifiers used throughout FireFly 170 │ │ interface │ * Maps to and from blockchain signing identities 171 │ └─────┬─────────┘ * Map API/user identities from authentication, to network/organizational identities 172 │ │ 173 │ ├───────────── ... extensible to DIDs etc. 174 │ ┌─────┴─────────┐ 175 │ │ onchain │ 176 │ └───────────────┘ 177 │ 178 │ ┌───────────────┐ - API Authentication and Authorization Interface 179 ├───────────┤ api auth [Aa]│ * Authenticates security credentials (OpenID Connect id token JWTs etc.) 180 │ │ │ * Extracts API/user identity (for identity interface to map) 181 │ └─────┬─────────┘ * Enforcement point for fine grained API access control 182 │ │ 183 │ ├─────────────────────┬────────── ... extensible other single sign-on technologies 184 │ ┌─────┴─────────┐ ┌───────┴───────┐ 185 │ │ apikey │ │ jwt │ 186 │ └───────────────┘ └───────────────┘ 187 │ 188 │ ┌───────────────┐ - Database Interactions 189 ├───────────┤ database [Di]│ * Create, Read, Update, Delete (CRUD) actions 190 │ │ interace │ * Filtering and update definition interace 191 │ └─────┬─────────┘ * Migrations and Indexes 192 │ │ 193 │ ├───────── ... extensible to NoSQL (CouchDB / MongoDB etc.) 194 │ ┌─────┴─────────┐ 195 │ │ sqlcommon │ 196 │ └─────┬─────────┘ 197 │ ├─────────────────────┬───────────────────┐ 198 │ ┌─────┴─────────┐ ┌───────┴───────┐ ┌───────┴────────┐ 199 │ │ postgres │ │ ql │ │ sqlite │ 200 │ └───────────────┘ └───────────────┘ └────────────────┘ 201 │ 202 │ ┌───────────────┐ - Connects the core event engine to external frameworks and applications 203 ├───────────┤ event [Ei]│ * Supports long-lived (durable) and ephemeral event subscriptions 204 │ │ interface │ * Batching, filtering, all handled in core prior to transport 205 │ └─────┬─────────┘ * Interface supports connect-in (websocket) and connect-out (broker runtime style) plugins 206 │ │ 207 │ ├───────── ... extensible to integrate off-chain compute framework (Hyperledger Avalon, TEE, ZKP, MPC etc.) 208 │ │ ... extensible to additional event delivery brokers/subsystems (Webhooks, Kafka, AMQP etc.) 209 │ ┌─────┴─────────┐ 210 │ │ websockets │ 211 │ └───────────────┘ 212 │ ... more TBD 213 214 Additional utility framworks 215 ┌───────────────┐ - REST API client 216 │ rest [Re]│ * Provides convenience and logging 217 │ client │ * Standardizes auth, config and retry logic 218 └───────────────┘ * Built on Resty 219 220 ┌───────────────┐ - WebSocket client 221 │ wsclient [Wc]│ * Provides convenience and logging 222 │ │ * Standardizes auth, config and reconnect logic 223 └───────────────┘ * Built on Gorilla WebSockets 224 225 ┌───────────────┐ - Translation framework 226 │ i18n [In]│ * Every translations must be added to `en_translations.json` - with an `FF10101` key 227 │ │ * Errors are wrapped, providing extra features from the `errors` package (stack etc.) 228 └───────────────┘ * Description translations also supported, such as OpenAPI description 229 230 ┌───────────────┐ - Logging framework 231 │ log [Lo]│ * Logging framework (logrus) integrated with context based tagging 232 │ │ * Context is used throughout the code to pass API invocation context, and logging context 233 └───────────────┘ * Example: Every API call has an ID that can be traced, as well as a timeout 234 235 ┌───────────────┐ - Configuration 236 │ config [Co]│ * File and Environment Variable based logging framework (viper) 237 │ │ * Primary config keys all defined centrally 238 └───────────────┘ * Plugins integrate by returning their config structure for unmarshaling (JSON tags) 239 240 [1] The "utdbql" blockchain plugin is a simple standalone ordering service, that uses the an in-process database 241 It does NOT provide a multi-party blockchain, and does NOT provide broadcast to all members in the network 242 243 ```