github.com/aychain/blockbook@v0.1.1-0.20181121092459-6d1fc7e07c5b/docs/build.md (about) 1 # Blockbook Build Guide 2 3 ## Setting up your development environment 4 5 Supported environment to develop Blockbook is Linux. Although it is possible build and run Blockbook on macOS 6 or Windows our build process is not prepared for it. But you can still build Blockbook [manually](#manual-build). 7 8 The only dependency required to build Blockbook is Docker. You can see how to install Docker [here](https://docs.docker.com/install/linux/docker-ce/debian/). 9 Manual build require additional dependencies that are described in appropriate section. 10 11 ## Build in Docker environment 12 13 All build operations run in Docker container in order to keep build environment isolated. Makefile in root of repository 14 define few targets used for building, testing and packaging of Blockbook. With Docker image definitions and Debian 15 package templates in *build/docker* and *build/templates* respectively, they are only inputs that make build process. 16 17 Docker build images are created at first execution of Makefile and that information is persisted. (Actually there are 18 created two files in repository – .bin-image and .deb-image – that are used as tags.) Sometimes it is necessary to 19 rebuild Docker images, it is possible by executing `make build-images`. 20 21 ### Building binary 22 23 Just run `make` and that is it. Output binary is stored in *build* directory. Note that although Blockbook is Go application 24 it is dynamically linked with RocksDB dependencies and ZeroMQ. Therefore operating system where Blockbook will be 25 executed still need that dependencies installed. See [Manual build](#manual-build) instructions below or install 26 Blockbook via Debian packages. 27 28 ### Building debug binary 29 30 Standard binary contains no debug symbols. Execute `make build-debug` to get binary for debugging. 31 32 ### Testing 33 34 How to execute tests is described in separate document [here](/docs/testing.md). 35 36 ### Building Debian packages 37 38 Blockbook and particular coin back-end are usually deployed together. They are defined in same place as well. 39 So typical way to build Debian packages is build Blockbook and back-end deb packages by single command. But it is not 40 mandatory, of course. 41 42 > Early releases of Blockbook weren't so friendly for extending. One had to define back-end package, Blockbook package, 43 > back-end configuration and Blockbook configuration as well. There were many options that were duplicated across 44 > configuration files and therefore error prone. 45 > 46 > Actually all configuration options and also build options for both Blockbook and back-end are defined in single JSON 47 > file and all stuff required during build is generated dynamically. 48 49 Makefile targets follow simple pattern, there are few prefixes that define what to build. 50 51 * *deb-blockbook-<coin>* – Build Blockbook package for given coin. 52 53 * *deb-backend-<coin>* – Build back-end package for given coin. 54 55 * *deb-<coin>* – Build both Blockbook and back-end packages for given coin. 56 57 * *all-<coin>* – Similar to deb-<coin> but clean repository and rebuild Docker image before package build. It is useful 58 for production deployment. 59 60 * *all* – Build both Blockbook and back-end packages for all coins. 61 62 Which coins are possible to build is defined in *configs/coins*. Particular coin has to have JSON config file there. 63 64 For example we want to build some packages for Bitcoin and Bitcoin Testnet. 65 66 ```bash 67 # make all-bitcoin deb-backend-bitcoin_testnet 68 ... 69 # ls build/*.deb 70 build/backend-bitcoin_0.16.1-satoshilabs-1_amd64.deb build/backend-bitcoin-testnet_0.16.1-satoshilabs-1_amd64.deb build/blockbook-bitcoin_0.0.6_amd64.deb 71 ``` 72 73 We have built two back-end packages – for Bitcoin and Testnet – and Blockbook package for Bitcoin. Before build have 74 been performed there was cleaned build directory and rebuilt Docker image. 75 76 ### Extra variables 77 78 There are few variables that can be passed to make in order to modify build process. 79 80 In general, build of Blockbook binary require some dependencies. They are downloaded automatically during build process 81 but if you need to build binary repeatedly it consumes a lot of time. Here comes variable *UPDATE_VENDOR* that if is 82 unset says that build process uses *vendor* (i.e. dependencies) from your local repository. For example: 83 `make deb-bitcoin UPDATE_VENDOR=0`. But before the command is executed there must be *vendor* directory populated, 84 you can do it by calling `dep ensure --vendor-only`. See [Manual build](#manual-build) instructions below. 85 86 All build targets allow pass additional parameters to underlying command inside container. It is possible via ARGS 87 variable. For example if you want run only subset of unit-tests, you will perform it by calling: 88 `make test ARGS='-run TestBitcoinRPC' UPDATE_VENDOR=0` 89 90 Common behaviour of Docker image build is that build steps are cached and next time they are executed much faster. 91 Although this is a good idea, when something went wrong you will need to override this behaviour somehow. Execute this 92 command: `make build-images NO_CACHE=true`. 93 94 ### On naming conventions and versioning 95 96 All configuration keys described below are in coin definition file in *configs/coins*. 97 98 **install and data directories** 99 100 Both Blockbook and back-end have separated install and data directories. They use common preffix and are defined in 101 *configs/environ.json* and all templates use them. 102 103 * back-end install directory is */opt/coins/nodes/<coin>*. 104 * back-end data directory is */opt/coins/data/<coin>/backend*. 105 * Blockbook install directory is */opt/coins/blockbook/<coin>*. 106 * Blockbook data directory is */opt/coins/data/<coin>/blockbook*. 107 108 *coin* used above is defined in *coin.alias* in coin definition file. 109 110 **package names** 111 112 Package names are defined in *backend.package_name* and *blockbook.package_name* in coin definition file. We use 113 simple pattern *<prefix>-<coin>* to name packages where *prefix* is either *blockbook* or *backend* and 114 *coin* is made similarly to *coin.alias*. We use convention that coin name uses lowercase characters and dash '-' as 115 a word delimiter. Testnet versions of coins must have *-testnet* suffix. That differs from *coin.alias* because 116 underscore has a special meaning in Debian packaging. For example there are packages *backend-bitcoin* and 117 *blockbook-bitcoin-testnet*. 118 119 **user names** 120 121 User names are defined in *backend.system_user* and *blockbook.system_user* in coin definition file. We follow common 122 Linux conventions, user names use lowercase characters and dash '-' as a word delimiter. 123 124 Back-end user name use coin name only, including testnet services. For example there is *bitcoin* user for both 125 *backend-bitcoin* and *backend-bitcoin-testnet* packages. 126 127 Blockbook user name has *blockbook-* prefix and coin name (made same as back-end version). For example there is 128 *blockbook-bitcoin* user for both *blockbook-bitcoin* and *blockbook-bitcoin-testnet* packages. 129 130 **back-end versioning** 131 132 Since we have to distinguish version of coin distribution and version of our configuration we follow standard Debian 133 package versioning rules (for details see 134 [Debian policy](https://www.debian.org/doc/debian-policy/ch-controlfields.html#version)). There is upstream version 135 and revision both defined in coin definition file in *backend.version* and *backend.package_revision*, respectively. 136 137 **blockbook versioning** 138 139 Blockbook versioning is much simpler. There is only one version defined in *configs/environ.json*. 140 141 ### On back-end building 142 143 Because we don't keep back-end archives inside out repository we download them during build process. Build steps 144 are these: download, verify and extract archive, prepare distribution and make package. 145 146 All configuration keys described below are in coin definition file in *configs/coins*. 147 148 **download archive** 149 150 URL from where is archive downloaded is defined in *backend.binary_url*. 151 152 **verify archive** 153 154 There are three different approaches how is archive verification done. Some projects use PGP sign of archive, some 155 have signed sha256 sums and some don't care about verification at all. So there is option *backend.verification_type* that 156 could be *gpg*, *gpg-sha256* or *sha256* and chooses particular method. 157 158 *gpg* type require file with digital sign and maintainer's public key imported in Docker build image (see below). Sign 159 file is downloaded from URL defined in *backend.verification_source*. Than is passed to gpg in order to verify archvie. 160 161 *gpg-sha256* type require signed checksum file and maintainer's public key imported in Docker build image (see below). 162 Checksum file is downloaded from URL defined in *backend.verification_source*. Then is verified by gpg and passed to 163 sha256sum in order to verify archive. 164 165 *sha256* type is used for coins that don't support verification at all. In *backend.verification_source* is defined 166 hexadecimal string that is compared with output of sha256sum. Although this solution is not secure, it avoid download 167 errors and other surprises at least. 168 169 *gpg* and *gpg-sha256* types require maintainer's public key imported in Docker build image. It is not expected that 170 maintainer's key will change requently while sing or checksum files are changed every release, so it is ideal to 171 store maintainer's key within image definition. Public keys are stored in *build/docker/deb/gpg-keys* directory. Docker 172 image must be rebuilt by calling `make build-images`. 173 174 **extract archive** 175 176 Extraction command is defined in *backend.extract_command*. Content of archive must be extracted to `./backend` directory. 177 See bitcoin.json and vertcoin.json for different approaches. 178 179 **prepare distribution** 180 181 There are two steps in this stage – exclude unnecessary files and generate configuration. 182 183 Some files are not required for server deployment, some binaries have unnecessary dependencies, so it is good idea to 184 extract these files from output package. Files to extract are listed in *backend.exclude_files*. Note that paths are 185 relative to *backend* directory where archive is extracted. 186 187 Configuration is described in [config.md](/docs/config.md). 188 189 ## Manual build 190 191 Instructions below are focused on Debian 9 (Stretch). If you want to use another Linux distribution or operating system 192 like macOS or Windows, please read instructions specific for each project. 193 194 Setup go environment: 195 196 ``` 197 wget https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz && tar xf go1.10.3.linux-amd64.tar.gz 198 sudo mv go /opt/go 199 sudo ln -s /opt/go/bin/go /usr/bin/go 200 # see `go help gopath` for details 201 mkdir $HOME/go 202 export GOPATH=$HOME/go 203 export PATH=$PATH:$GOPATH/bin 204 ``` 205 206 Install RocksDB: https://github.com/facebook/rocksdb/blob/master/INSTALL.md 207 and compile the static_lib and tools 208 209 ``` 210 sudo apt-get update && sudo apt-get install -y \ 211 build-essential git wget pkg-config libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev 212 git clone https://github.com/facebook/rocksdb.git 213 cd rocksdb 214 CFLAGS=-fPIC CXXFLAGS=-fPIC make release 215 ``` 216 217 Setup variables for gorocksdb: https://github.com/tecbot/gorocksdb 218 219 ``` 220 export CGO_CFLAGS="-I/path/to/rocksdb/include" 221 export CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4" 222 ``` 223 224 Install ZeroMQ: https://github.com/zeromq/libzmq 225 226 Install go-dep tool: 227 ``` 228 go get github.com/golang/dep/cmd/dep 229 ``` 230 231 Get blockbook sources, install dependencies, build: 232 233 ``` 234 cd $GOPATH/src 235 git clone https://github.com/trezor/blockbook.git 236 cd blockbook 237 dep ensure -vendor-only 238 go build 239 ``` 240 241 ### Example command 242 243 Blockbook require full node daemon as its back-end. You are responsible for proper installation. Port numbers and 244 daemon configuration are defined in *configs/coins* and *build/templates/backend/config* directories. You should use 245 specific installation process for particular coin you want run (e.g. https://bitcoin.org/en/full-node#other-linux-distributions for Bitcoin). 246 247 When you have running back-end daemon you can start Blockbook. It is highly recommended use ports described in [ports.md](/docs/ports.md) 248 for both Blockbook and back-end daemon. You can use *contrib/scripts/build-blockchaincfg.sh* that will generate 249 Blockbook's blockchain configuration from our coin definition files. 250 251 Example for Bitcoin: 252 ``` 253 contrib/scripts/build-blockchaincfg.sh 254 ./blockbook -sync -blockchaincfg=build/blockchaincfg.json -internal=:9030 -public=:9130 -certfile=server/testcert -logtostderr 255 ``` 256 257 This command starts Blockbook with parallel synchronization and providing HTTP and Socket.IO interface, with database 258 in local directory *data* and established ZeroMQ and RPC connections to back-end daemon specified in configuration 259 file passed to *-blockchaincfg* option. 260 261 Blockbook logs to stderr (option *-logtostderr*) or to directory specified by parameter *-log_dir* . Verbosity of logs can be tuned 262 by command line parameters *-v* and *-vmodule*, for details see https://godoc.org/github.com/golang/glog. 263 264 You can check that Blockbook is running by simple HTTP request: `curl https://localhost:9130`. Returned data is JSON with some 265 run-time information. If port is closed, Blockbook is syncing data.