github.com/kaleido-io/firefly@v0.0.0-20210622132723-8b4b6aacb971/README.md (about) 1 # FireFly 2 3 [](https://codecov.io/gh/kaleido-io/firefly) 4 [](https://goreportcard.com/report/github.com/kaleido-io/firefly) 5 6 FireFly is a multiparty system for enterprise data flows, powered by blockchain. It solves all of the layers of complexity that sit between the low level blockchain and high level business processes and user interfaces. FireFly enables developers to build blockchain apps for enterprise up to 100x faster by allowing them to focus on business logic instead of infrastructure. 7 8  9 10 Please see the 11 [Hyperledger FireFly proposal document](https://github.com/kaleido-io/hyperledger-hip/blob/gh-pages/HIPs/firefly.md) 12 for more information about the project goals and architecture. 13 14 ## FireFly repos 15 16 FireFly has a plugin based architecture design, with a microservice runtime footprint. 17 As such there are a number of repos, and the list will grow as the community evolves. 18 19 But not to worry, one of those repos is a CLI designed to get you running with all the components you need in minutes! 20 21 - CLI / Developer experience - https://github.com/hyperledger-labs/firefly-cli 22 - UI Explorer - https://github.com/hyperledger-labs/firefly-ui 23 - Core (this repo) - https://github.com/hyperledger-labs/firefly 24 - HTTP Data Exchange - https://github.com/hyperledger-labs/firefly-dataexchange-https 25 - Ethereum (Hyperledger Besu / Quorum) connector: https://github.com/hyperledger-labs/firefly-ethconnect 26 - Corda connector: https://github.com/hyperledger-labs/firefly-cordaconnect - contributed from Kaleido generation 1 - porting to generation 2 27 - Hyperledger Fabric connector - in design phase, including collaboration with https://github.com/hyperledger-labs/fabric-smart-client 28 29 > Note only the projects that are primarily built to support FireFly are listed here, not all 30 > of the ecosystem of projects that integrate underneath the plugins. See [below](#firefly-code-hierarchy) for 31 > more information on the landscape of plugins and components. 32 33 ## Architecture 34 35 Architecture discussions are currently facilitated by the following Git issues, but as the community evolves (within 36 the Hyperledger foundation) we expect these documents and conversations to move to a better collaboration system. 37 38 - [Node Component Architecture - Periodic Table of Elements](https://github.com/hyperledger-labs/firefly/issues/6) 39 - [Multi-party Event Sequencing for Multi-Party Business Processes](https://github.com/hyperledger-labs/firefly/issues/10) 40 - [Simplest Transaction Flow - Ping Pong](https://github.com/hyperledger-labs/firefly/issues/5) 41 - [Plugin Architecture](https://github.com/hyperledger-labs/firefly/issues/7) 42 - [Internal Event Sequencing Model - The FireFly Stream of Consciousness](https://github.com/hyperledger-labs/firefly/issues/11) 43 44 ## Getting Started 45 46 Use the FireFly CLI for fast bootstrap: https://github.com/hyperledger-labs/firefly-cli 47 48 ## Navigating this repo 49 50 There are **two core codebases** currently active in this repo: 51 52 ### Generation 2: FireFly 53 54 Directories: 55 56 - [internal](./internal): The core Golang implementation code 57 - [pkg](./pkg): Interfaces intended for external project use 58 - [cmd](./cmd): The command line entry point 59 - [solidity_firefly](./solidity_firefly): Ethereum/Solidity smart contract code 60 61 [Full code layout here](#firefly-code-hierarchy) 62 63 This latest generation is re-engineered from the ground up to improve developer experience, runtime performance, and extensibility. 64 65 This means a simplified REST/WebSocket programming model for app development, and a wider range of infrastructure options for deployment. 66 67 It also means a focus on an architecture and code structure for a vibrant open source community. 68 69 A few highlights: 70 71 - Golang codebase 72 - Strong coding standards, including unit test coverage, translation support, logging and more 73 - Fast starting, low memory footprint, multi-threaded runtime 74 - OpenAPI 3.0 API specification (Swagger) 75 - Generated from the API router code, to avoid divergence with the implementation 76 - Active/active HA architecture for the core runtime 77 - Deferring to the core database for state high availability 78 - Exploiting leader election where required 79 - Fully pluggable architecture 80 - Everything from Database through to Blockchain, and Compute 81 - Golang plugin infrastructure to decouple the core code from the implementation 82 - Remote Agent model to decouple code languages, and HA designs 83 - Updated API resource model 84 - `Asset`, `Data`, `Message`, `Event`, `Topic`, `Transaction` 85 - Added flexibility, with simplified the developer experience: 86 - Versioning of data definitions 87 - Introducing a first class `Context` construct link related events into a single sequence 88 - Allow many pieces of data to be attached to a single message, and be automatically re-assembled on arrival 89 - Clearer separation of concerns between the FireFly DB and the Application DB 90 - Better search, filter and query support 91 92 ### Generation 1: Kaleido Asset Trail (KAT) 93 94 Directories: 95 96 - [kat](./kat): The core TypeScript runtime 97 - [solidity_kat](./solidity_kat): Ethereum/Solidity smart contract code 98 - [cordapp_kat](./cordapp_kat): The Corda smart contract (CorDapp) 99 100 This was the original implementation of the multi-party systems API by Kaleido, and is already deployed in a number production projects. 101 102 The codebase distilled years of learning, into a set of patterns for performing blockchain orchestrated data exchange. 103 104 It depends on the following Kaleido services: 105 106 - Blockchain nodes 107 - Ethereum with the Kaleido [Kaleido REST API Gateway](https://docs.kaleido.io/kaleido-services/ethconnect/) 108 - Corda with the Kaleido built-in API for streaming KAT transactions 109 - [Kaleido Event Streams](https://docs.kaleido.io/kaleido-services/event-streams/) 110 - [Kaleido App2App Messaging](https://docs.kaleido.io/kaleido-services/app2app/) 111 - [Kaleido Document Exchange](https://docs.kaleido.io/kaleido-services/document-store/) 112 113 ## FireFly code hierarchy 114 115 ``` 116 ┌──────────┐ ┌───────────────┐ 117 │ cmd ├──┤ firefly [Ff]│ - CLI entry point 118 └──────────┘ │ │ - Creates parent context 119 │ │ - Signal handling 120 └─────┬─────────┘ 121 │ 122 ┌──────────┐ ┌─────┴─────────┐ - HTTP listener (Gorilla mux) 123 │ internal ├──┤ api [As]│ * TLS (SSL), CORS configuration etc. 124 └──────────┘ │ server │ * WS upgrade on same port 125 │ │ - REST route definitions 126 └─────┬─────────┘ * Simple routing logic only, all processing deferred to orchestrator 127 │ 128 ┌─────┴─────────┐ - REST route definition framework 129 │ openapi [Oa]│ * Standardizes Body, Path, Query, Filter semantics 130 │ spec | - OpenAPI 3.0 (Swagger) generation 131 └─────┬─────────┘ * Including Swagger. UI 132 │ 133 ┌─────┴─────────┐ - WebSocket server 134 │ [Ws]│ * Developer friendly JSON based protocol business app development 135 │ websockets │ * Reliable sequenced delivery 136 └─────┬─────────┘ * _Event interface [Ei] supports lower level integration with other compute frameworks/transports_ 137 │ 138 ┌─────┴─────────┐ - Core data types 139 │ fftypes [Ft]│ * Used for API and Serialization 140 │ │ * APIs can mask fields on input via router definition 141 └─────┬─────────┘ 142 │ 143 ┌─────┴─────────┐ - Core runtime server. Initializes and owns instances of: 144 │ [Or]│ * Components: Implement features 145 ┌───────┬───┤ orchestrator │ * Plugins: Pluggable infrastructure services 146 │ │ │ │ - Exposes actions to router 147 │ │ └───────────────┘ * Processing starts here for all API calls 148 │ │ 149 │ Components: Components do the heavy lifting within the engine 150 │ │ 151 │ │ ┌───────────────┐ - Maintains a view of the entire network 152 │ ├───┤ network [Nm]│ * Integrates with network permissioning [NP] plugin 153 │ │ │ map │ * Integrates with broadcast plugin 154 │ │ └───────────────┘ * Handles hierarchy of member identity, node identity and signing identity 155 │ │ 156 │ │ ┌───────────────┐ - Broadcast of data to all parties in the network 157 │ ├───┤ broadcast [Bm]│ * Implements dispatcher for batch component 158 │ │ │ manager | * Integrates with public storage interface [Ps] plugin 159 │ │ └───────────────┘ * Integrates with blockchain interface [Bi] plugin 160 │ │ 161 │ │ ┌───────────────┐ - Send private data to individual parties in the network 162 │ ├───┤ private [Pm]│ * Implements dispatcher for batch component 163 │ │ │ messaging | * Integrates with the data exchange [Dx] plugin 164 │ │ └──────┬────────┘ * Messages can be pinned and sequenced via the blockchain, or just sent 165 │ │ │ 166 │ │ ┌──────┴────────┐ - Groups of parties, with isolated data and/or blockchains 167 │ │ │ group [Gm]│ * Integrates with data exchange [Dx] plugin 168 │ │ │ manager │ * Integrates with blockchain interface [Bi] plugin 169 │ │ └───────────────┘ 170 │ │ 171 │ │ ┌───────────────┐ - Private data management and validation 172 │ ├───┤ data [Dm]│ * Implements dispatcher for batch component 173 │ │ │ manager │ * Integrates with data exchange [Dx] plugin 174 │ │ └──────┬────────┘ * Integrates with blockchain interface [Bi] plugin 175 │ │ │ 176 │ │ ┌──────┴────────┐ - JSON data shema management and validation (architecture extensible to XML and more) 177 │ │ │ json [Jv]│ * JSON Schema validation logic for outbound and inbound messages 178 │ │ │ validator │ * Schema propagatation 179 │ │ └──────┬────────┘ * Integrates with broadcast plugin 180 │ │ │ 181 │ │ ┌──────┴────────┐ - Binary data addressable via ID or Hash 182 │ │ │ blobstore [Bs]│ * Integrates with data exchange [Dx] plugin 183 │ │ │ │ * Hashes data, and maintains mapping to payload references in blob storage 184 │ │ └───────────────┘ * Integrates with blockchain interface [Bi] plugin 185 │ │ 186 │ │ ┌───────────────┐ - Private data management and validation 187 │ ├───┤ event [Em]│ * Implements dispatcher for batch component 188 │ │ │ manager │ * Integrates with data exchange [Dx] plugin 189 │ │ └──────┬────────┘ * Integrates with blockchain interface [Bi] plugin 190 │ │ │ 191 │ │ ┌──────┴────────┐ - Handles incoming external data 192 │ │ │ [Ag]│ * Integrates with data exchange [Dx] plugin 193 │ │ │ aggregator │ * Integrates with public storage interface [Ps] plugin 194 │ │ │ │ * Integrates with blockchain interface [Bi] plugin 195 │ │ │ │ - Ensures valid events are dispatched only once all data is available 196 │ │ └──────┬────────┘ * Context aware, to prevent block-the-world scenarios 197 │ │ │ 198 │ │ ┌──────┴────────┐ - Subscription manager 199 │ │ │ [Sm]│ * Creation and management of subscriptions 200 │ │ │ subscription │ * Creation and management of subscriptions 201 │ │ │ manager │ * Message to Event matching logic 202 │ │ └──────┬────────┘ 203 │ │ │ 204 │ │ ┌──────┴────────┐ - Manages delivery of events to connected applications 205 │ │ │ event [Ed]│ * Integrates with data exchange [Dx] plugin 206 │ │ │ dispatcher │ * Integrates with blockchain interface [Bi] plugin 207 │ │ └───────────────┘ 208 │ │ 209 │ │ ┌───────────────┐ - Token operations 210 │ ├───┤ asset [Am]│ * NFT coupling with contexts 211 │ │ │ manager │ * Transfer coupling with data describing payment reason 212 │ │ │ │ - ... 213 │ │ └───────────────┘ 214 │ │ 215 │ │ ┌───────────────┐ - Aggregates messages and data, with rolled up hashes for pinning 216 │ ├───┤ batch [Ba]│ * Pluggable dispatchers 217 │ │ │ manager │ - Database decoupled from main-line API processing 218 │ │ │ │ * See architecture diagrams for more info on active/active sequencing 219 │ │ └──────┬────────┘ - Manages creation of batch processor instances 220 │ │ │ 221 │ │ ┌──────┴────────┐ - Short lived agent spun up to assemble batches on demand 222 │ │ │ batch [Bp]│ * Coupled to an author+type of messages 223 │ │ │ processor │ - Builds batches of 100s messages for efficient pinning 224 │ │ │ │ * Aggregates messages and data, with rolled up hashes for pinning 225 │ │ └───────────────┘ - Shuts down automatically after a configurable inactivity period 226 │ ... more TBD 227 │ 228 Plugins: Each plugin comprises a Go shim, plus a remote agent microservice runtime (if required) 229 │ 230 │ ┌───────────────┐ - Blockchain Interface 231 ├───────────┤ [Bi]│ * Transaction submission - including signing key management 232 │ │ blockchain │ * Event listening 233 │ │ interface │ * Standardized operations, and custom on-chain coupling 234 │ └─────┬─────────┘ 235 │ │ 236 │ ├─────────────────────┬───────────────────┬────────────────────┐ 237 │ ┌─────┴─────────┐ ┌───────┴───────┐ ┌───────┴────────┐ ┌───────┴────────┐ 238 │ │ ethereum │ │ corda │ │ fabric │ │ utdbql [1] │ 239 │ └───────────────┘ └───────────────┘ └────────────────┘ └────────────────┘ 240 │ 241 │ ┌───────────────┐ - P2P Content Addresssed Filesystem 242 ├───────────┤ public [Pi]│ * Payload upload / download 243 │ │ storage │ * Payload reference management 244 │ │ interface │ 245 │ └─────┬─────────┘ 246 │ │ 247 │ ├───────── ... extensible to any shared storage sytem, accessible to all members 248 │ ┌─────┴─────────┐ 249 │ │ ipfs │ 250 │ └───────────────┘ 251 │ 252 │ ┌───────────────┐ - Private Data Exchange 253 ├───────────┤ data [Dx]│ * Blob storage 254 │ │ exchange │ * Private secure messaging 255 │ └─────┬─────────┘ * Secure file transfer 256 │ │ 257 │ ├─────────────────────┬────────── ... extensible to any private data exchange tech 258 │ ┌─────┴─────────┐ ┌───────┴───────┐ 259 │ │ httpdirect │ │ kaleido │ 260 │ └───────────────┘ └───────────────┘ 261 │ 262 │ ┌───────────────┐ - Identity resolution and mapping 263 ├───────────┤ identity [Ii]│ * Resolves opaque identifiers used throughout FireFly 264 │ │ interface │ * Maps to and from blockchain signing identities 265 │ └─────┬─────────┘ * Map API/user identities from authentication, to network/organizational identities 266 │ │ 267 │ ├───────────── ... extensible to DIDs etc. 268 │ ┌─────┴─────────┐ 269 │ │ onchain │ 270 │ └───────────────┘ 271 │ 272 │ ┌───────────────┐ - API Authentication and Authorization Interface 273 ├───────────┤ api auth [Aa]│ * Authenticates security credentials (OpenID Connect id token JWTs etc.) 274 │ │ │ * Extracts API/user identity (for identity interface to map) 275 │ └─────┬─────────┘ * Enforcement point for fine grained API access control 276 │ │ 277 │ ├─────────────────────┬────────── ... extensible other single sign-on technologies 278 │ ┌─────┴─────────┐ ┌───────┴───────┐ 279 │ │ apikey │ │ jwt │ 280 │ └───────────────┘ └───────────────┘ 281 │ 282 │ ┌───────────────┐ - Database Interactions 283 ├───────────┤ database [Di]│ * Create, Read, Update, Delete (CRUD) actions 284 │ │ interace │ * Filtering and update definition interace 285 │ └─────┬─────────┘ * Migrations and Indexes 286 │ │ 287 │ ├───────── ... extensible to NoSQL (CouchDB / MongoDB etc.) 288 │ ┌─────┴─────────┐ 289 │ │ sqlcommon │ 290 │ └─────┬─────────┘ 291 │ ├─────────────────────┬───────────────────┐ 292 │ ┌─────┴─────────┐ ┌───────┴───────┐ ┌───────┴────────┐ 293 │ │ postgres │ │ ql │ │ sqlite │ 294 │ └───────────────┘ └───────────────┘ └────────────────┘ 295 │ 296 │ ┌───────────────┐ - Connects the core event engine to external frameworks and applications 297 ├───────────┤ event [Ei]│ * Supports long-lived (durable) and ephemeral event subscriptions 298 │ │ interface │ * Batching, filtering, all handled in core prior to transport 299 │ └─────┬─────────┘ * Interface supports connect-in (websocket) and connect-out (broker runtime style) plugins 300 │ │ 301 │ ├───────── ... extensible to integrate off-chain compute framework (Hyperledger Avalon, TEE, ZKP, MPC etc.) 302 │ │ ... extensible to additional event delivery brokers/subsystems (Webhooks, Kafka, AMQP etc.) 303 │ ┌─────┴─────────┐ 304 │ │ websockets │ 305 │ └───────────────┘ 306 │ ... more TBD 307 308 Additional utility framworks 309 ┌───────────────┐ - REST API client 310 │ rest [Re]│ * Provides convenience and logging 311 │ client │ * Standardizes auth, config and retry logic 312 └───────────────┘ * Built on Resty 313 314 ┌───────────────┐ - WebSocket client 315 │ wsclient [Wc]│ * Provides convenience and logging 316 │ │ * Standardizes auth, config and reconnect logic 317 └───────────────┘ * Built on Gorilla WebSockets 318 319 ┌───────────────┐ - Translation framework 320 │ i18n [In]│ * Every translations must be added to `en_translations.json` - with an `FF10101` key 321 │ │ * Errors are wrapped, providing extra features from the `errors` package (stack etc.) 322 └───────────────┘ * Description translations also supported, such as OpenAPI description 323 324 ┌───────────────┐ - Logging framework 325 │ log [Lo]│ * Logging framework (logrus) integrated with context based tagging 326 │ │ * Context is used throughout the code to pass API invocation context, and logging context 327 └───────────────┘ * Example: Every API call has an ID that can be traced, as well as a timeout 328 329 ┌───────────────┐ - Configuration 330 │ config [Co]│ * File and Environment Variable based logging framework (viper) 331 │ │ * Primary config keys all defined centrally 332 └───────────────┘ * Plugins integrate by returning their config structure for unmarshaling (JSON tags) 333 334 [1] The "utdbql" blockchain plugin is a simple standalone ordering service, that uses the an in-process database 335 It does NOT provide a multi-party blockchain, and does NOT provide broadcast to all members in the network 336 337 ``` 338 339 ## API Query Syntax 340 341 REST collections provide filter, `skip`, `limit` and `sort` support. 342 343 - The field in the message is used as the query parameter 344 - When multiple query parameters are supplied these are combined with AND 345 - When the same query parameter is supplied multiple times, these are combined with OR 346 347 ### Example 348 349 `GET` `/api/v1/messages?confirmed=>0&type=broadcast&topic=t1&topic=t2&context=@someprefix&sort=sequence&descending&skip=100&limit=50` 350 351 This states: 352 353 - Filter on `confirmed` greater than 0 354 - Filter on `type` exactly equal to `broadcast` 355 - Filter on `topic` exactly equal to `t1` _or_ `t2` 356 - Filter on `context` containing the case-sensitive string `someprefix` 357 - Sort on `sequence` in `descending` order 358 - Paginate with `limit` of `50` and `skip` of `100` (e.g. get page 3, with 50/page) 359 360 Table of filter operations, which must be the first character of the query string (after the `=` in the above URL path example) 361 362 | Operator | Description | 363 | -------- | --------------------------------- | 364 | (none) | Equal | 365 | `!` | Not equal | 366 | `<` | Less than | 367 | `<=` | Less than or equal | 368 | `>` | Greater than | 369 | `>=` | Greater than or equal | 370 | `@` | Containing - case sensitive | 371 | `!@` | Not containing - case sensitive | 372 | `^` | Containing - case insensitive | 373 | `!^` | Not containing - case insensitive |