go.dedis.ch/onet/v3@v3.2.11-0.20210930124529-e36530bca7ef/README.md (about) 1 [](https://travis-ci.org/dedis/onet) 2 [](https://goreportcard.com/report/github.com/dedis/onet) 3 [](https://coveralls.io/github/dedis/onet) 4 [](https://www.codacy.com/manual/nkcr/onet?utm_source=github.com&utm_medium=referral&utm_content=dedis/onet&utm_campaign=Badge_Grade) 5 6 # The Cothority Overlay Network Library - Onet 7 8 The Overlay-network (Onet) is a library for simulation and deployment of 9 decentralized, distributed protocols. This library offers a framework for 10 research, simulation, and deployment of crypto-related protocols with an emphasis 11 on decentralized, distributed protocols. It offers an abstraction for tree-based 12 communications between thousands of nodes and it is used both in research for 13 testing out new protocols and running simulations, as well as in production to 14 deploy those protocols as a service in a distributed manner. 15 16 **Onet** is developed by [DEDIS/EFPL](http://dedis.epfl.ch) as part of the 17 [Cothority](https://github.com/dedis/cothority) project that aims to deploy a 18 large number of nodes for distributed signing and related projects. In 19 cothority, nodes are commonly named "conodes". A collective authority 20 (cothority) is a set of conodes that work together to handle a distributed, 21 decentralized task. 22 23 Onet allows you to set up the following three elements: 24 25 - _protocols_: a short-lived set of messages being passed back and forth between 26 one or more conodes 27 28 - _services_: define an API usable by client programs and instantiate protocols 29 30 - _apps_: communicate with the service-API of one or more conodes 31 32 We also provide a set of methods to set up and run _simulations_. 33 34 * * * 35 36 <!-- START doctoc.sh generated TOC please keep comment here to allow auto update --> 37 38 <!-- DO NOT EDIT THIS SECTION, INSTEAD RE-RUN doctoc.sh TO UPDATE --> 39 **:book: Table of Contents** 40 41 - [The Cothority Overlay Network Library - Onet](#the-cothority-overlay-network-library---onet) 42 - [General information](#general-information) 43 - [Directories](#directories) 44 - [Version](#version) 45 - [License](#license) 46 - [Contribution](#contribution) 47 - [Contact](#contact) 48 - [Reporting security problems](#reporting-security-problems) 49 - [Components](#components) 50 - [Router](#router) 51 - [Conode](#conode) 52 - [Roster](#roster) 53 - [Protocol](#protocol) 54 - [Service](#service) 55 - [ServiceManager](#servicemanager) 56 - [Tree](#tree) 57 - [Overlay](#overlay) 58 - [TreeNodeInstance](#treenodeinstance) 59 - [App](#app) 60 - [Database Backup and Recovery](#database-backup-and-recovery) 61 - [Backup](#backup) 62 - [Recovery](#recovery) 63 - [Interacting with the database](#interacting-with-the-database) 64 - [Simulation](#simulation) 65 66 <!-- END doctoc.sh generated TOC please keep comment here to allow auto update --> 67 68 # General information 69 70 ## Directories 71 72 - [app](app) - useful libraries if you want to create a CLI app for the 73 cothority 74 75 - [cfgpath](cfgpath) - single package to get the configuration-path 76 77 - [log](log) - everybody needs its own log-library - this one has log-levels, 78 colors, time, ... 79 80 - [network](network) - different type of connections: channels, tcp, tls 81 82 - [simul](simul) - allowing to run your protocols and services on different 83 platforms with up to 50'000 nodes 84 85 ## Version 86 87 The Onet library follows the same development cycle as the one described in 88 the [dedis/cothority](https://github.com/dedis/cothority/tree/master/) project. 89 90 ## License 91 92 This project and all its files are licensed under the 93 GNU Lesser General Public License, version 3 (SPDX short identifier: LGPL-3.0) 94 95 If this license is not suitable for your business or project 96 please contact us for a full commercial license. 97 98 ## Contribution 99 100 If you want to contribute, please have a look at 101 [CONTRIBUTION](https://github.com/dedis/onet/blob/master/CONTRIBUTION) for 102 licensing details and feel free to open a pull request. 103 104 ## Contact 105 106 You can contact us at <https://groups.google.com/forum/#!forum/cothority> or 107 privately at dedis@epfl.ch. 108 109 ## Reporting security problems 110 111 This library is offered as-is without any guarantees. It would need an 112 independent security review before it should be considered ready for use in 113 security-critical applications. If you integrate Onet into your application it 114 is YOUR RESPONSIBILITY to arrange for that audit. 115 116 If you notice a possible security problem, please report it 117 to dedis-security@epfl.ch. 118 119 # Components 120 121 In Onet, you define _Services_ that use _Protocols_ which can send and receive 122 messages. Each _Protocol_ is instantiated when needed as a _ProtocolInstance_. 123 As multiple _Protocols_ can be run at the same time, there can be more than one 124 _ProtocolInstance_ of the same _Protocol_. Onet makes sure all messages get 125 routed to the appropriate _ProtocolInstance_. 126 127 Foreign applications can communicate with Onet over the service-API, which is 128 implemented using protobuf over WebSockets for JavaScript compatibility. 129 130 This chapter provides a high-level description of the cothority framework. Let's 131 start with a picture and then dive into each main components of the 132 library. 133 134  135 136 As you can see there's a bunch of different entities involved. Let's get down 137 the rabbit hole to explain the most important ones! 138 139 ## Router 140 141 The Router handles all incoming and outgoing messages from and to the network. A 142 Router can use different underlying types of connection: 143 144 - _TCP_ which uses regular TCP connections, 145 - _Local_ which uses channels and is mainly for testing purposes, and 146 - _TLS_ which is still in progress. 147 148 ## Conode 149 150 A conode is the main entity of a Cothority server. It holds the Router, the 151 Overlay, and the different Services. Generally, for developing an application 152 using the framework, you would create your Router first, then the Conode, and 153 then finally call `conode.Start()`. 154 155 ## Roster 156 157 A Roster is simply a list of Conodes denoted by their public key and address. A 158 Roster is identified by its ID, which is unique for each list. 159 160 ## Protocol 161 162 A Protocol is an interface where users of the library must implement the logic 163 of the protocol they want to code. It is supposed to be a short term entity that 164 is self-sufficient, i.e. it does not need external access to any other resources 165 of the Cothority framework. A protocol can be launched from SDA itself or by a 166 Service. 167 168 ## Service 169 170 A Service is a long term entity that is created when a Conode is created. It 171 serves different purposes: 172 173 - serving external client requests, 174 - creating and attaching protocols with the Overlay (and launching them), 175 - communicating information to other Services on other Conodes. 176 177 ## ServiceManager 178 179 A ServiceManager is the main interface between the Conode and the Service. It 180 transforms and embed the messages created by the Service to its own format and 181 pass it to the Conode for the sending part; vice versa for the reception part. 182 183 ## Tree 184 185 A Tree is a standard tree data structure where each node - called 186 _TreeNode_ - is denoted by its public key and address. The Tree is constructed 187 out of a Roster. 188 189 ## Overlay 190 191 The Overlay provides an abstraction to communicate over different Trees that the 192 Protocols and Services need. It handles the following tasks: 193 194 - Propagations of the Roster and the Trees between different Conodes 195 - Creation of the Protocol 196 - Dispatching of incoming and outgoing messages to the right Protocol 197 198 ## TreeNodeInstance 199 200 A TreeNodeInstance is created by the Overlay. There is one TreeNodeInstance for 201 each Protocol and it acts as the central point of communication for that 202 Protocol. The TreeNodeInstance offers to its Protocol some common tree methods 203 such as `SendParent`,`SendChild`, `IsRoot` etc. More importantly, it transforms 204 and embeds the message given by the Protocol into its own struct and dispatch it 205 to the Overlay for the sending part; vice versa for the reception part. 206 207 ## App 208 209 An application in the context of Onet is a CLI-program that interacts with one 210 or more conodes through the use of the API defined by one or more services. It 211 is mostly written in go, but in the cothority-repository you can also find 212 libraries to interact in javascript and java. 213 214 # Database Backup and Recovery 215 216 Users of Onet have the option to make use of its built-in database. 217 218 We use [bbolt](https://github.com/etcd-io/bbolt), which supports "fully 219 serializable ACID transactions" to ensure data integrity for Onet users. Users 220 should be able to do the following: 221 222 - Backup data while Onet is running 223 - Recovery from a backup in case of data corruption 224 225 ## Backup 226 227 Users are recommended to perform frequent backups such that data can be 228 recovered if Onet nodes fail. Onet stores all of its data in the context folder, 229 specified by `$CONODE_SERVICE_PATH`. If unset, it defaults to 230 231 - `~/Library/Application Support/conode/data` on macOS, 232 - `$HOME\AppData\Local\Conode` on Windows, or 233 - `~/.local/share/conode` on other Unix/Linux. 234 235 Hence, to backup, it is recommended to use a standard backup tool, such as 236 rsync, and copy the folder to a different physical location periodically. 237 The database keeps a transaction log. 238 239 Performing backups in the middle of a transaction should not be a problem. 240 However, it is still recommended to check the data integrity of the backed-up 241 file using the bbolt CLI, i.e. `bolt check database_name.db`. 242 243 To install the bbolt CLI, see [Bolt Installation](https://github.com/etcd-io/bbolt#installing). 244 245 ## Recovery 246 247 Data corruption is easy to detect as Onet nodes crash when reading from a 248 corrupted database, at startup or during operation. Concretely, the bbolt 249 library would panic 250 ([source](https://github.com/etcd-io/bbolt/blob/386b851495d42c4e02908838373a06d0a533e170/freelist.go#L237)). 251 This behavior is produced by writing a few blocks of random data using `dd` to 252 the database. 253 254 In case of data corruption, the database must be restored from a backup by 255 simply copying the backup copy to the context directory, and then starting the 256 conode again. It is the user's responsibility to make sure that the data is up 257 to date, e.g. by reading the latest data from running Onet nodes. 258 259 ## Interacting with the database 260 261 The primary and recommended methods to interact with the database are 262 [`Load`](https://godoc.org/github.com/dedis/onet#Context.Load) and 263 [`Save`](https://godoc.org/github.com/dedis/onet#Context.Save). If more control 264 on the database is needed, then we can ask the context to return a database 265 handler and bucket name using the function 266 [`GetAdditionalBucket`](https://godoc.org/github.com/dedis/onet#Context.GetAdditionalBucket). 267 268 All the [bbolt functions](https://godoc.org/github.com/etcd-io/bbolt) can be 269 used with the database handler. However, the user should avoid creating new 270 buckets using the bbolt functions and only use `GetAdditionalBucket` to avoid 271 bucket name conflicts. 272 273 # Simulation 274 275 Have a look at the `simul/README.md` for explanations about simulations.