github.com/yacovm/fabric@v2.0.0-alpha.0.20191128145320-c5d4087dc723+incompatible/docs/source/couchdb_tutorial.rst (about) 1 2 Using CouchDB 3 ============= 4 5 This tutorial will describe the steps required to use the CouchDB as the state 6 database with Hyperledger Fabric. By now, you should be familiar with Fabric 7 concepts and have explored some of the samples and tutorials. 8 9 .. note:: These instructions use the new Fabric chaincode lifecycle introduced 10 in the Fabric v2.0 Alpha release. If you would like to use the previous 11 lifecycle model to use indexes with chaincode, visit the v1.4 12 version of the `Using CouchDB <https://hyperledger-fabric.readthedocs.io/en/release-1.4/couchdb_tutorial.html>`__. 13 14 The tutorial will take you through the following steps: 15 16 #. :ref:`cdb-enable-couch` 17 #. :ref:`cdb-create-index` 18 #. :ref:`cdb-add-index` 19 #. :ref:`cdb-install-instantiate` 20 #. :ref:`cdb-query` 21 #. :ref:`cdb-best` 22 #. :ref:`cdb-pagination` 23 #. :ref:`cdb-update-index` 24 #. :ref:`cdb-delete-index` 25 26 For a deeper dive into CouchDB refer to :doc:`couchdb_as_state_database` 27 and for more information on the Fabric ledger refer to the `Ledger <ledger/ledger.html>`_ 28 topic. Follow the tutorial below for details on how to leverage CouchDB in your 29 blockchain network. 30 31 Throughout this tutorial we will use the `Marbles sample <https://github.com/hyperledger/fabric-samples/blob/master/chaincode/marbles02/go/marbles_chaincode.go>`__ 32 as our use case to demonstrate how to use CouchDB with Fabric and will deploy 33 Marbles to the :doc:`build_network` (BYFN) tutorial network. You should have 34 completed the task :doc:`install`. However, running the BYFN tutorial is not 35 a prerequisite for this tutorial, instead the necessary commands are provided 36 throughout this tutorial to use the network. 37 38 Why CouchDB? 39 ~~~~~~~~~~~~ 40 41 Fabric supports two types of peer databases. LevelDB is the default state 42 database embedded in the peer node and stores chaincode data as simple 43 key-value pairs and supports key, key range, and composite key queries only. 44 CouchDB is an optional alternate state database that supports rich 45 queries when chaincode data values are modeled as JSON. Rich queries are more 46 flexible and efficient against large indexed data stores, when you want to 47 query the actual data value content rather than the keys. CouchDB is a JSON 48 document datastore rather than a pure key-value store therefore enabling 49 indexing of the contents of the documents in the database. 50 51 In order to leverage the benefits of CouchDB, namely content-based JSON 52 queries,your data must be modeled in JSON format. You must decide whether to use 53 LevelDB or CouchDB before setting up your network. Switching a peer from using 54 LevelDB to CouchDB is not supported due to data compatibility issues. All peers 55 on the network must use the same database type. If you have a mix of JSON and 56 binary data values, you can still use CouchDB, however the binary values can 57 only be queried based on key, key range, and composite key queries. 58 59 .. _cdb-enable-couch: 60 61 Enable CouchDB in Hyperledger Fabric 62 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 63 64 CouchDB runs as a separate database process alongside the peer, therefore there 65 are additional considerations in terms of setup, management, and operations. 66 A docker image of `CouchDB <https://hub.docker.com/r/hyperledger/fabric-couchdb/>`__ 67 is available and we recommend that it be run on the same server as the 68 peer. You will need to setup one CouchDB container per peer 69 and update each peer container by changing the configuration found in 70 ``core.yaml`` to point to the CouchDB container. The ``core.yaml`` 71 file must be located in the directory specified by the environment variable 72 FABRIC_CFG_PATH: 73 74 * For docker deployments, ``core.yaml`` is pre-configured and located in the peer 75 container ``FABRIC_CFG_PATH`` folder. However when using docker environments, 76 you typically pass environment variables by editing the 77 ``docker-compose-couch.yaml`` to override the core.yaml 78 79 * For native binary deployments, ``core.yaml`` is included with the release artifact 80 distribution. 81 82 Edit the ``stateDatabase`` section of ``core.yaml``. Specify ``CouchDB`` as the 83 ``stateDatabase`` and fill in the associated ``couchDBConfig`` properties. For 84 more details on configuring CouchDB to work with fabric, refer `here <http://hyperledger-fabric.readthedocs.io/en/master/couchdb_as_state_database.html#couchdb-configuration>`__. 85 To view an example of a core.yaml file configured for CouchDB, examine the 86 BYFN ``docker-compose-couch.yaml`` in the ``HyperLedger/fabric-samples/first-network`` 87 directory. 88 89 .. _cdb-create-index: 90 91 Create an index 92 ~~~~~~~~~~~~~~~ 93 94 Why are indexes important? 95 96 Indexes allow a database to be queried without having to examine every row 97 with every query, making them run faster and more efficiently. Normally, 98 indexes are built for frequently occurring query criteria allowing the data to 99 be queried more efficiently. To leverage the major benefit of CouchDB -- the 100 ability to perform rich queries against JSON data -- indexes are not required, 101 but they are strongly recommended for performance. Also, if sorting is required 102 in a query, CouchDB requires an index of the sorted fields. 103 104 .. note:: 105 106 Rich queries that do not have an index will work but may throw a warning 107 in the CouchDB log that the index was not found. However, if a rich query 108 includes a sort specification, then an index on that field is required; 109 otherwise, the query will fail and an error will be thrown. 110 111 To demonstrate building an index, we will use the data from the `Marbles 112 sample <https://github.com/hyperledger/fabric-samples/blob/master/chaincode/marbles02/go/marbles_chaincode.go>`__. 113 In this example, the Marbles data structure is defined as: 114 115 .. code:: javascript 116 117 type marble struct { 118 ObjectType string `json:"docType"` //docType is used to distinguish the various types of objects in state database 119 Name string `json:"name"` //the field tags are needed to keep case from bouncing around 120 Color string `json:"color"` 121 Size int `json:"size"` 122 Owner string `json:"owner"` 123 } 124 125 126 In this structure, the attributes (``docType``, ``name``, ``color``, ``size``, 127 ``owner``) define the ledger data associated with the asset. The attribute 128 ``docType`` is a pattern used in the chaincode to differentiate different data 129 types that may need to be queried separately. When using CouchDB, it 130 recommended to include this ``docType`` attribute to distinguish each type of 131 document in the chaincode namespace. (Each chaincode is represented as its own 132 CouchDB database, that is, each chaincode has its own namespace for keys.) 133 134 With respect to the Marbles data structure, ``docType`` is used to identify 135 that this document/asset is a marble asset. Potentially there could be other 136 documents/assets in the chaincode database. The documents in the database are 137 searchable against all of these attribute values. 138 139 When defining an index for use in chaincode queries, each one must be defined 140 in its own text file with the extension `*.json` and the index definition must 141 be formatted in the CouchDB index JSON format. 142 143 To define an index, three pieces of information are required: 144 145 * `fields`: these are the frequently queried fields 146 * `name`: name of the index 147 * `type`: always json in this context 148 149 For example, a simple index named ``foo-index`` for a field named ``foo``. 150 151 .. code:: json 152 153 { 154 "index": { 155 "fields": ["foo"] 156 }, 157 "name" : "foo-index", 158 "type" : "json" 159 } 160 161 Optionally the design document attribute ``ddoc`` can be specified on the index 162 definition. A `design document <http://guide.couchdb.org/draft/design.html>`__ is 163 CouchDB construct designed to contain indexes. Indexes can be grouped into 164 design documents for efficiency but CouchDB recommends one index per design 165 document. 166 167 .. tip:: When defining an index it is a good practice to include the ``ddoc`` 168 attribute and value along with the index name. It is important to 169 include this attribute to ensure that you can update the index later 170 if needed. Also it gives you the ability to explicitly specify which 171 index to use on a query. 172 173 174 Here is another example of an index definition from the Marbles sample with 175 the index name ``indexOwner`` using multiple fields ``docType`` and ``owner`` 176 and includes the ``ddoc`` attribute: 177 178 .. _indexExample: 179 180 .. code:: json 181 182 { 183 "index":{ 184 "fields":["docType","owner"] // Names of the fields to be queried 185 }, 186 "ddoc":"indexOwnerDoc", // (optional) Name of the design document in which the index will be created. 187 "name":"indexOwner", 188 "type":"json" 189 } 190 191 In the example above, if the design document ``indexOwnerDoc`` does not already 192 exist, it is automatically created when the index is deployed. An index can be 193 constructed with one or more attributes specified in the list of fields and 194 any combination of attributes can be specified. An attribute can exist in 195 multiple indexes for the same docType. In the following example, ``index1`` 196 only includes the attribute ``owner``, ``index2`` includes the attributes 197 ``owner and color`` and ``index3`` includes the attributes ``owner, color and 198 size``. Also, notice each index definition has its own ``ddoc`` value, following 199 the CouchDB recommended practice. 200 201 .. code:: json 202 203 { 204 "index":{ 205 "fields":["owner"] // Names of the fields to be queried 206 }, 207 "ddoc":"index1Doc", // (optional) Name of the design document in which the index will be created. 208 "name":"index1", 209 "type":"json" 210 } 211 212 { 213 "index":{ 214 "fields":["owner", "color"] // Names of the fields to be queried 215 }, 216 "ddoc":"index2Doc", // (optional) Name of the design document in which the index will be created. 217 "name":"index2", 218 "type":"json" 219 } 220 221 { 222 "index":{ 223 "fields":["owner", "color", "size"] // Names of the fields to be queried 224 }, 225 "ddoc":"index3Doc", // (optional) Name of the design document in which the index will be created. 226 "name":"index3", 227 "type":"json" 228 } 229 230 231 In general, you should model index fields to match the fields that will be used 232 in query filters and sorts. For more details on building an index in JSON 233 format refer to the `CouchDB documentation <http://docs.couchdb.org/en/latest/api/database/find.html#db-index>`__. 234 235 A final word on indexing, Fabric takes care of indexing the documents in the 236 database using a pattern called ``index warming``. CouchDB does not typically 237 index new or updated documents until the next query. Fabric ensures that 238 indexes stay 'warm' by requesting an index update after every block of data is 239 committed. This ensures queries are fast because they do not have to index 240 documents before running the query. This process keeps the index current 241 and refreshed every time new records are added to the state database. 242 243 .. _cdb-add-index: 244 245 246 Add the index to your chaincode folder 247 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 248 249 Once you finalize an index, it is ready to be packaged with your chaincode for 250 deployment by being placed alongside it in the appropriate metadata folder. 251 252 If your chaincode installation and instantiation uses the Hyperledger 253 Fabric Node SDK, the JSON index files can be located in any folder as long 254 as it conforms to this `directory structure <https://fabric-sdk-node.github.io/tutorial-metadata-chaincode.html>`__. 255 During the chaincode installation using the ``client.installChaincode()`` API, 256 include the attribute (``metadataPath``) in the `installation request <https://fabric-sdk-node.github.io/global.html#ChaincodeInstallRequest>`__. 257 The value of the metadataPath is a string representing the absolute path to the 258 directory structure containing the JSON index file(s). 259 260 Alternatively, if you are using the 261 :doc:`commands/peercommand` command to install and instantiate the chaincode, then the JSON 262 index files must be located under the path ``META-INF/statedb/couchdb/indexes`` 263 which is located inside the directory where the chaincode resides. 264 265 The `Marbles sample <https://github.com/hyperledger/fabric-samples/tree/master/chaincode/marbles02/go>`__ below illustrates how the index 266 is packaged with the chaincode which will be installed using the peer commands. 267 268 .. image:: images/couchdb_tutorial_pkg_example.png 269 :scale: 100% 270 :align: center 271 :alt: Marbles Chaincode Index Package 272 273 This sample includes one index named indexOwnerDoc: 274 275 .. code:: json 276 277 {"index":{"fields":["docType","owner"]},"ddoc":"indexOwnerDoc", "name":"indexOwner","type":"json"} 278 279 280 Start the network 281 ----------------- 282 283 :guilabel:`Try it yourself` 284 285 Before installing and instantiating the marbles chaincode, we need to start 286 up the BYFN network. For the sake of this tutorial, we want to operate 287 from a known initial state. The following command will kill any active 288 or stale docker containers and remove previously generated artifacts. 289 Therefore let's run the following command to clean up any 290 previous environments: 291 292 .. code:: bash 293 294 cd fabric-samples/first-network 295 ./byfn.sh down 296 297 298 Now start up the BYFN network with CouchDB by running the following command: 299 300 .. code:: bash 301 302 ./byfn.sh up -c mychannel -s couchdb 303 304 This will create a simple Fabric network consisting of a single channel named 305 `mychannel` with two organizations (each maintaining two peer nodes) and an 306 ordering service while using CouchDB as the state database. 307 308 .. _cdb-install-instantiate: 309 310 Install and define the Chaincode 311 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 312 313 Client applications interact with the blockchain ledger through chaincode. 314 Therefore we need to install a chaincode on every peer that will execute and 315 endorse our transactions. However, before we can interact with our chaincode, 316 the members of the channel need to agree on a chaincode definition that 317 establishes chaincode governance. In the previous section, we demonstrated how 318 to add the index to the chaincode folder so that it is ready for deployment. 319 320 The chaincode needs to be packaged before it can be installed on our peers. 321 We can use the `peer lifecycle chaincode package <http://hyperledger-fabric.readthedocs.io/en/latest/commands/peerlifecycle.html#peer-lifecycle-chaincode-package>`__ command 322 to package the marbles chaincode. 323 324 :guilabel:`Try it yourself` 325 326 Assuming you have started the BYFN network, enter the CLI container: 327 328 .. code:: bash 329 330 docker exec -it cli bash 331 332 Your command prompt will change to something similar to: 333 334 .. code:: bash 335 336 bash-4.4# 337 338 1. Use the following command to package the marbles chaincode from the git 339 repository inside your local container. 340 341 .. code:: bash 342 343 peer lifecycle chaincode package marbles.tar.gz --path github.com/hyperledger/fabric-samples/chaincode/marbles02/go --lang golang --label marbles_1 344 345 This command will create a chaincode package named marbles.tar.gz. 346 347 2. Use the following command to install the chaincode package onto the peer 348 ``peer0.org1.example.com`` in your BYFN network. By default, after starting 349 the BYFN network, the active peer is set to: 350 ``CORE_PEER_ADDRESS=peer0.org1.example.com:7051``: 351 352 .. code:: bash 353 354 peer lifecycle chaincode install marbles.tar.gz 355 356 A successful install command will return the chaincode identifier, similar to 357 the response below: 358 359 .. code:: bash 360 361 2019-04-22 18:47:38.312 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 001 Installed remotely: response:<status:200 payload:"\nJmarbles_1:0907c1f3d3574afca69946e1b6132691d58c2f5c5703df7fc3b692861e92ecd3\022\tmarbles_1" > 362 2019-04-22 18:47:38.312 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 002 Chaincode code package identifier: marbles_1:0907c1f3d3574afca69946e1b6132691d58c2f5c5703df7fc3b692861e92ecd3 363 364 After installing the chaincode on our peer, we need to approve a chaincode 365 definition for our organization, and commit the chaincode definition to the 366 channel. The chaincode definition includes the package identifier that was 367 returned by the install command. We can also use the `peer lifecycle chaincode queryinstalled <http://hyperledger-fabric.readthedocs.io/en/latest/commands/peerlifecycle.html#peer-lifecycle-chaincode-queryinstalled>`__ 368 command to find the package ID of ``marbles.tar.gz``. 369 370 3. Use the following command to query your peer for the package ID of the 371 installed chaincode. 372 373 .. code:: bash 374 375 peer lifecycle chaincode queryinstalled 376 377 The command will return the same package identifier as the install command. 378 You should see output similar to the following: 379 380 .. code:: bash 381 382 Installed chaincodes on peer: 383 Package ID: marbles_1:0907c1f3d3574afca69946e1b6132691d58c2f5c5703df7fc3b692861e92ecd3, Label: marbles_1 384 Package ID: mycc_1:27ef99cb3cbd1b545063f018f3670eddc0d54f40b2660b8f853ad2854c49a0d8, Label: mycc_1 385 386 4. Declare the package ID as an environment variable. Paste the package ID of 387 marbles_1 returned by the ``peer lifecycle chaincode queryinstalled`` command 388 into the command below. The package ID may not be the same for all users, so 389 you need to complete this step using the package ID returned from your console. 390 391 .. code:: bash 392 393 export CC_PACKAGE_ID=marbles_1:0907c1f3d3574afca69946e1b6132691d58c2f5c5703df7fc3b692861e92ecd3 394 395 5. Use the following command to approve a definition of the marbles chaincode 396 for Org1. The approval is distributed within each organization, so the command 397 does not need to target every peer of an organization. 398 399 .. code:: bash 400 401 export ORDERER_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem 402 peer lifecycle chaincode approveformyorg --channelID mychannel --name marbles --version 1.0 --signature-policy "OR('Org1MSP.member','Org2MSP.member')" --init-required --package-id $CC_PACKAGE_ID --sequence 1 --tls true --cafile $ORDERER_CA 403 404 When the command completes successfully you should see something similar to : 405 406 .. code:: bash 407 408 2019-03-18 16:04:09.046 UTC [cli.lifecycle.chaincode] InitCmdFactory -> INFO 001 Retrieved channel (mychannel) orderer endpoint: orderer.example.com:7050 409 2019-03-18 16:04:11.253 UTC [chaincodeCmd] ClientWait -> INFO 002 txid [efba188ca77889cc1c328fc98e0bb12d3ad0abcda3f84da3714471c7c1e6c13c] committed with status (VALID) at 410 411 6. We need a majority of organizations to approve a chaincode definition before 412 it can be committed to the channel. This implies that we need Org2 to approve 413 the chaincode definition as well. Because we do not need Org2 to endorse the 414 chaincode and did not install the package on any Org2 peers, we do not need to 415 provide a packageID as part of the chaincode definition. 416 417 Use the CLI to switch to Org2. Copy and paste the following block of commands 418 as a group into the peer container and run them all at once. 419 420 .. code:: bash 421 422 export CORE_PEER_ADDRESS=peer0.org2.example.com:9051 423 export CORE_PEER_LOCALMSPID=Org2MSP 424 export PEER0_ORG2_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt 425 export CORE_PEER_TLS_ROOTCERT_FILE=$PEER0_ORG2_CA 426 export CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/users/Admin@org2.example.com/msp 427 428 7. You can now approve the chaincode definition for Org2: 429 430 .. code:: bash 431 432 peer lifecycle chaincode approveformyorg --channelID mychannel --name marbles --version 1.0 --signature-policy "OR('Org1MSP.member','Org2MSP.member')" --init-required --sequence 1 --tls true --cafile $ORDERER_CA 433 434 Once a sufficient number of organizations (in this case, a majority) have 435 approved a chaincode definition, one organization can commit the definition to 436 the channel using the `peer lifecycle chaincode commit <http://hyperledger-fabric.readthedocs.io/en/latest/commands/peerlifecycle.html#peer-lifecycle-chaincode-commit>`__ command. 437 438 We are ready to use the chaincode after the chaincode definition has been 439 committed to the channel. Because the marbles chaincode contains an 440 initialization function, we need to use the `peer chaincode invoke <http://hyperledger-fabric.readthedocs.io/en/master/commands/peerchaincode.html?%20chaincode%20instantiate#peer-chaincode-instantiate>`__ command 441 to invoke ``Init()`` before we can use other functions in the chaincode. 442 443 8. Run the following commands to commit the definition of the marbles private 444 data chaincode to the BYFN channel ``mychannel``. This command needs to target 445 peers in Org1 and Org2 to collect endorsements for the commit transaction. 446 447 .. code:: bash 448 449 export ORDERER_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem 450 export ORG1_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt 451 export ORG2_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt 452 peer lifecycle chaincode commit -o orderer.example.com:7050 --channelID mychannel --name marbles --version 1.0 --sequence 1 --signature-policy "OR('Org1MSP.member','Org2MSP.member')" --init-required --tls true --cafile $ORDERER_CA --peerAddresses peer0.org1.example.com:7051 --tlsRootCertFiles $ORG1_CA --peerAddresses peer0.org2.example.com:9051 --tlsRootCertFiles $ORG2_CA 453 454 When the commit transaction completes successfully you should see something 455 similar to: 456 457 .. code:: bash 458 459 2019-04-22 18:57:34.274 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [3da8b0bb8e128b5e1b6e4884359b5583dff823fce2624f975c69df6bce614614] committed with status (VALID) at peer0.org2.example.com:9051 460 2019-04-22 18:57:34.709 UTC [chaincodeCmd] ClientWait -> INFO 002 txid [3da8b0bb8e128b5e1b6e4884359b5583dff823fce2624f975c69df6bce614614] committed with status (VALID) at peer0.org1.example.com:7051 461 462 9. Use the following command to invoke the ``Init`` function to initialize 463 the chaincode. This will start the chaincode on ``peer0.org1.example.com:7051``. 464 465 .. code:: bash 466 467 peer chaincode invoke -o orderer.example.com:7050 --channelID mychannel --name marbles --isInit --tls true --cafile $ORDERER_CA --peerAddresses peer0.org1.example.com:7051 --tlsRootCertFiles $ORG1_CA -c '{"Args":["Init"]}' 468 469 470 Verify index was deployed 471 ------------------------- 472 473 Indexes will be deployed to each peer's CouchDB state database once the 474 chaincode is both installed on the peer and started on the channel. You 475 can verify that the CouchDB index was created successfully by examining the 476 peer log in the Docker container. 477 478 :guilabel:`Try it yourself` 479 480 To view the logs in the peer docker container, open a new Terminal window and 481 run the following command to grep for message confirmation that the index was 482 created. 483 484 :: 485 486 docker logs peer0.org1.example.com 2>&1 | grep "CouchDB index" 487 488 489 You should see a result that looks like the following: 490 491 :: 492 493 [couchdb] CreateIndex -> INFO 0be Created CouchDB index [indexOwner] in state database [mychannel_marbles] using design document [_design/indexOwnerDoc] 494 495 .. note:: If Marbles was not installed on the BYFN peer ``peer0.org1.example.com``, 496 you may need to replace it with the name of a different peer where 497 Marbles was installed. 498 499 .. _cdb-query: 500 501 Query the CouchDB State Database 502 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 503 504 Now that the index has been defined in the JSON file and deployed alongside 505 the chaincode, chaincode functions can execute JSON queries against the CouchDB 506 state database, and thereby peer commands can invoke the chaincode functions. 507 508 Specifying an index name on a query is optional. If not specified, and an index 509 already exists for the fields being queried, the existing index will be 510 automatically used. 511 512 .. tip:: It is a good practice to explicitly include an index name on a 513 query using the ``use_index`` keyword. Without it, CouchDB may pick a 514 less optimal index. Also CouchDB may not use an index at all and you 515 may not realize it, at the low volumes during testing. Only upon 516 higher volumes you may realize slow performance because CouchDB is not 517 using an index and you assumed it was. 518 519 520 Build the query in chaincode 521 ---------------------------- 522 523 You can perform complex rich queries against the chaincode data values using 524 the CouchDB JSON query language within chaincode. As we explored above, the 525 `marbles02 sample chaincode <https://github.com/hyperledger/fabric-samples/blob/master/chaincode/marbles02/go/marbles_chaincode.go>`__ 526 includes an index and rich queries are defined in the functions - ``queryMarbles`` 527 and ``queryMarblesByOwner``: 528 529 * **queryMarbles** -- 530 531 Example of an **ad hoc rich query**. This is a query 532 where a (selector) string can be passed into the function. This query 533 would be useful to client applications that need to dynamically build 534 their own selectors at runtime. For more information on selectors refer 535 to `CouchDB selector syntax <http://docs.couchdb.org/en/latest/api/database/find.html#find-selectors>`__. 536 537 538 * **queryMarblesByOwner** -- 539 540 Example of a parameterized query where the 541 query logic is baked into the chaincode. In this case the function accepts 542 a single argument, the marble owner. It then queries the state database for 543 JSON documents matching the docType of “marble” and the owner id using the 544 JSON query syntax. 545 546 547 Run the query using the peer command 548 ------------------------------------ 549 550 In absence of a client application to test rich queries defined in chaincode, 551 peer commands can be used. Peer commands run from the command line inside the 552 docker container. We will customize the `peer chaincode query <http://hyperledger-fabric.readthedocs.io/en/master/commands/peerchaincode.html?%20chaincode%20query#peer-chaincode-query>`__ 553 command to use the Marbles index ``indexOwner`` and query for all marbles owned 554 by "tom" using the ``queryMarbles`` function. 555 556 :guilabel:`Try it yourself` 557 558 Before querying the database, we should add some data. Run the following 559 command in the peer container as Org1 to create a marble owned by "tom": 560 561 .. code:: bash 562 563 export CORE_PEER_ADDRESS=peer0.org1.example.com:7051 564 export CORE_PEER_LOCALMSPID=Org1MSP 565 export PEER0_ORG1_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt 566 export CORE_PEER_TLS_ROOTCERT_FILE=$PEER0_ORG1_CA 567 export CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp 568 peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C mychannel -n marbles -c '{"Args":["initMarble","marble1","blue","35","tom"]}' 569 570 After an index has been deployed when the chaincode is initialized, it will 571 automatically be utilized by chaincode queries. CouchDB can determine which 572 index to use based on the fields being queried. If an index exists for the 573 query criteria it will be used. However the recommended approach is to 574 specify the ``use_index`` keyword on the query. The peer command below is an 575 example of how to specify the index explicitly in the selector syntax by 576 including the ``use_index`` keyword: 577 578 .. code:: bash 579 580 // Rich Query with index name explicitly specified: 581 peer chaincode query -C mychannel -n marbles -c '{"Args":["queryMarbles", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\"}, \"use_index\":[\"_design/indexOwnerDoc\", \"indexOwner\"]}"]}' 582 583 Delving into the query command above, there are three arguments of interest: 584 585 * ``queryMarbles`` 586 587 Name of the function in the Marbles chaincode. Notice a `shim <https://godoc.org/github.com/hyperledger/fabric-chaincode-go/shim>`__ 588 ``shim.ChaincodeStubInterface`` is used to access and modify the ledger. The 589 ``getQueryResultForQueryString()`` passes the queryString to the shim API ``getQueryResult()``. 590 591 .. code:: bash 592 593 func (t *SimpleChaincode) queryMarbles(stub shim.ChaincodeStubInterface, args []string) pb.Response { 594 595 // 0 596 // "queryString" 597 if len(args) < 1 { 598 return shim.Error("Incorrect number of arguments. Expecting 1") 599 } 600 601 queryString := args[0] 602 603 queryResults, err := getQueryResultForQueryString(stub, queryString) 604 if err != nil { 605 return shim.Error(err.Error()) 606 } 607 return shim.Success(queryResults) 608 } 609 610 * ``{"selector":{"docType":"marble","owner":"tom"}`` 611 612 This is an example of an **ad hoc selector** string which finds all documents 613 of type ``marble`` where the ``owner`` attribute has a value of ``tom``. 614 615 616 * ``"use_index":["_design/indexOwnerDoc", "indexOwner"]`` 617 618 Specifies both the design doc name ``indexOwnerDoc`` and index name 619 ``indexOwner``. In this example the selector query explicitly includes the 620 index name, specified by using the ``use_index`` keyword. Recalling the 621 index definition above :ref:`cdb-create-index`, it contains a design doc, 622 ``"ddoc":"indexOwnerDoc"``. With CouchDB, if you plan to explicitly include 623 the index name on the query, then the index definition must include the 624 ``ddoc`` value, so it can be referenced with the ``use_index`` keyword. 625 626 627 The query runs successfully and the index is leveraged with the following results: 628 629 .. code:: json 630 631 Query Result: [{"Key":"marble1", "Record":{"color":"blue","docType":"marble","name":"marble1","owner":"tom","size":35}}] 632 633 .. _cdb-best: 634 635 Use best practices for queries and indexes 636 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 637 638 Queries that use indexes will complete faster, without having to scan the full 639 database in couchDB. Understanding indexes will allow you to write your queries 640 for better performance and help your application handle larger amounts 641 of data or blocks on your network. 642 643 It is also important to plan the indexes you install with your chaincode. You 644 should install only a few indexes per chaincode that support most of your queries. 645 Adding too many indexes, or using an excessive number of fields in an index, will 646 degrade the performance of your network. This is because the indexes are updated 647 after each block is committed. The more indexes need to be updated through 648 "index warming", the longer it will take for transactions to complete. 649 650 The examples in this section will help demonstrate how queries use indexes and 651 what type of queries will have the best performance. Remember the following 652 when writing your queries: 653 654 * All fields in the index must also be in the selector or sort sections of your query 655 for the index to be used. 656 * More complex queries will have a lower performance and will be less likely to 657 use an index. 658 * You should try to avoid operators that will result in a full table scan or a 659 full index scan such as ``$or``, ``$in`` and ``$regex``. 660 661 In the previous section of this tutorial, you issued the following query against 662 the marbles chaincode: 663 664 .. code:: bash 665 666 // Example one: query fully supported by the index 667 peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarbles", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\"}, \"use_index\":[\"indexOwnerDoc\", \"indexOwner\"]}"]}' 668 669 The marbles chaincode was installed with the ``indexOwnerDoc`` index: 670 671 .. code:: json 672 673 {"index":{"fields":["docType","owner"]},"ddoc":"indexOwnerDoc", "name":"indexOwner","type":"json"} 674 675 Notice that both the fields in the query, ``docType`` and ``owner``, are 676 included in the index, making it a fully supported query. As a result this 677 query will be able to use the data in the index, without having to search the 678 full database. Fully supported queries such as this one will return faster than 679 other queries from your chaincode. 680 681 If you add extra fields to the query above, it will still use the index. 682 However, the query will additionally have to scan the indexed data for the 683 extra fields, resulting in a longer response time. As an example, the query 684 below will still use the index, but will take a longer time to return than the 685 previous example. 686 687 .. code:: bash 688 689 // Example two: query fully supported by the index with additional data 690 peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarbles", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\",\"color\":\"red\"}, \"use_index\":[\"/indexOwnerDoc\", \"indexOwner\"]}"]}' 691 692 A query that does not include all fields in the index will have to scan the full 693 database instead. For example, the query below searches for the owner, without 694 specifying the the type of item owned. Since the ownerIndexDoc contains both 695 the ``owner`` and ``docType`` fields, this query will not be able to use the 696 index. 697 698 .. code:: bash 699 700 // Example three: query not supported by the index 701 peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarbles", "{\"selector\":{\"owner\":\"tom\"}, \"use_index\":[\"indexOwnerDoc\", \"indexOwner\"]}"]}' 702 703 In general, more complex queries will have a longer response time, and have a 704 lower chance of being supported by an index. Operators such as ``$or``, ``$in``, 705 and ``$regex`` will often cause the query to scan the full index or not use the 706 index at all. 707 708 As an example, the query below contains an ``$or`` term that will search for every 709 marble and every item owned by tom. 710 711 .. code:: bash 712 713 // Example four: query with $or supported by the index 714 peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarbles", "{\"selector\":{"\$or\":[{\"docType\:\"marble\"},{\"owner\":\"tom\"}]}, \"use_index\":[\"indexOwnerDoc\", \"indexOwner\"]}"]}' 715 716 This query will still use the index because it searches for fields that are 717 included in ``indexOwnerDoc``. However, the ``$or`` condition in the query 718 requires a scan of all the items in the index, resulting in a longer response 719 time. 720 721 Below is an example of a complex query that is not supported by the index. 722 723 .. code:: bash 724 725 // Example five: Query with $or not supported by the index 726 peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarbles", "{\"selector\":{"\$or\":[{\"docType\":\"marble\",\"owner\":\"tom\"},{"\color\":"\yellow\"}]}, \"use_index\":[\"indexOwnerDoc\", \"indexOwner\"]}"]}' 727 728 The query searches for all marbles owned by tom or any other items that are 729 yellow. This query will not use the index because it will need to search the 730 entire table to meet the ``$or`` condition. Depending the amount of data on your 731 ledger, this query will take a long time to respond or may timeout. 732 733 While it is important to follow best practices with your queries, using indexes 734 is not a solution for collecting large amounts of data. The blockchain data 735 structure is optimized to validate and confirm transactions and is not suited 736 for data analytics or reporting. If you want to build a dashboard as part 737 of your application or analyze the data from your network, the best practice is 738 to query an off chain database that replicates the data from your peers. This 739 will allow you to understand the data on the blockchain without degrading the 740 performance of your network or disrupting transactions. 741 742 You can use block or chaincode events from your application to write transaction 743 data to an off-chain database or analytics engine. For each block received, the block 744 listener application would iterate through the block transactions and build a data 745 store using the key/value writes from each valid transaction's ``rwset``. The 746 :doc:`peer_event_services` provide replayable events to ensure the integrity of 747 downstream data stores. For an example of how you can use an event listener to write 748 data to an external database, visit the `Off chain data sample <https://github.com/hyperledger/fabric-samples/tree/master/off_chain_data>`__ 749 in the Fabric Samples. 750 751 .. _cdb-pagination: 752 753 Query the CouchDB State Database With Pagination 754 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 755 756 When large result sets are returned by CouchDB queries, a set of APIs is 757 available which can be called by chaincode to paginate the list of results. 758 Pagination provides a mechanism to partition the result set by 759 specifying a ``pagesize`` and a start point -- a ``bookmark`` which indicates 760 where to begin the result set. The client application iteratively invokes the 761 chaincode that executes the query until no more results are returned. For more information refer to 762 this `topic on pagination with CouchDB <http://hyperledger-fabric.readthedocs.io/en/master/couchdb_as_state_database.html#couchdb-pagination>`__. 763 764 765 We will use the `Marbles sample <https://github.com/hyperledger/fabric-samples/blob/master/chaincode/marbles02/go/marbles_chaincode.go>`__ 766 function ``queryMarblesWithPagination`` to demonstrate how 767 pagination can be implemented in chaincode and the client application. 768 769 * **queryMarblesWithPagination** -- 770 771 Example of an **ad hoc rich query with pagination**. This is a query 772 where a (selector) string can be passed into the function similar to the 773 above example. In this case, a ``pageSize`` is also included with the query as 774 well as a ``bookmark``. 775 776 In order to demonstrate pagination, more data is required. This example assumes 777 that you have already added marble1 from above. Run the following commands in 778 the peer container to create four more marbles owned by "tom", to create a 779 total of five marbles owned by "tom": 780 781 :guilabel:`Try it yourself` 782 783 .. code:: bash 784 785 peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -c '{"Args":["initMarble","marble2","yellow","35","tom"]}' 786 peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -c '{"Args":["initMarble","marble3","green","20","tom"]}' 787 peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -c '{"Args":["initMarble","marble4","purple","20","tom"]}' 788 peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -c '{"Args":["initMarble","marble5","blue","40","tom"]}' 789 790 In addition to the arguments for the query in the previous example, 791 queryMarblesWithPagination adds ``pagesize`` and ``bookmark``. ``PageSize`` 792 specifies the number of records to return per query. The ``bookmark`` is an 793 "anchor" telling couchDB where to begin the page. (Each page of results returns 794 a unique bookmark.) 795 796 * ``queryMarblesWithPagination`` 797 798 Name of the function in the Marbles chaincode. Notice a `shim <https://godoc.org/github.com/hyperledger/fabric-chaincode-go/shim>`__ 799 ``shim.ChaincodeStubInterface`` is used to access and modify the ledger. The 800 ``getQueryResultForQueryStringWithPagination()`` passes the queryString along 801 with the pagesize and bookmark to the shim API ``GetQueryResultWithPagination()``. 802 803 .. code:: bash 804 805 func (t *SimpleChaincode) queryMarblesWithPagination(stub shim.ChaincodeStubInterface, args []string) pb.Response { 806 807 // 0 808 // "queryString" 809 if len(args) < 3 { 810 return shim.Error("Incorrect number of arguments. Expecting 3") 811 } 812 813 queryString := args[0] 814 //return type of ParseInt is int64 815 pageSize, err := strconv.ParseInt(args[1], 10, 32) 816 if err != nil { 817 return shim.Error(err.Error()) 818 } 819 bookmark := args[2] 820 821 queryResults, err := getQueryResultForQueryStringWithPagination(stub, queryString, int32(pageSize), bookmark) 822 if err != nil { 823 return shim.Error(err.Error()) 824 } 825 return shim.Success(queryResults) 826 } 827 828 829 The following example is a peer command which calls queryMarblesWithPagination 830 with a pageSize of ``3`` and no bookmark specified. 831 832 .. tip:: When no bookmark is specified, the query starts with the "first" 833 page of records. 834 835 :guilabel:`Try it yourself` 836 837 .. code:: bash 838 839 // Rich Query with index name explicitly specified and a page size of 3: 840 peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarblesWithPagination", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\"}, \"use_index\":[\"_design/indexOwnerDoc\", \"indexOwner\"]}","3",""]}' 841 842 The following response is received (carriage returns added for clarity), three 843 of the five marbles are returned because the ``pagsize`` was set to ``3``: 844 845 .. code:: bash 846 847 [{"Key":"marble1", "Record":{"color":"blue","docType":"marble","name":"marble1","owner":"tom","size":35}}, 848 {"Key":"marble2", "Record":{"color":"yellow","docType":"marble","name":"marble2","owner":"tom","size":35}}, 849 {"Key":"marble3", "Record":{"color":"green","docType":"marble","name":"marble3","owner":"tom","size":20}}] 850 [{"ResponseMetadata":{"RecordsCount":"3", 851 "Bookmark":"g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkGoOkOWDSOSANIFk2iCyIyVySn5uVBQAGEhRz"}}] 852 853 .. note:: Bookmarks are uniquely generated by CouchDB for each query and 854 represent a placeholder in the result set. Pass the 855 returned bookmark on the subsequent iteration of the query to 856 retrieve the next set of results. 857 858 The following is a peer command to call queryMarblesWithPagination with a 859 pageSize of ``3``. Notice this time, the query includes the bookmark returned 860 from the previous query. 861 862 :guilabel:`Try it yourself` 863 864 .. code:: bash 865 866 peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarblesWithPagination", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\"}, \"use_index\":[\"_design/indexOwnerDoc\", \"indexOwner\"]}","3","g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkGoOkOWDSOSANIFk2iCyIyVySn5uVBQAGEhRz"]}' 867 868 The following response is received (carriage returns added for clarity). The 869 last two records are retrieved: 870 871 .. code:: bash 872 873 [{"Key":"marble4", "Record":{"color":"purple","docType":"marble","name":"marble4","owner":"tom","size":20}}, 874 {"Key":"marble5", "Record":{"color":"blue","docType":"marble","name":"marble5","owner":"tom","size":40}}] 875 [{"ResponseMetadata":{"RecordsCount":"2", 876 "Bookmark":"g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkmoKkOWDSOSANIFk2iCyIyVySn5uVBQAGYhR1"}}] 877 878 The final command is a peer command to call queryMarblesWithPagination with 879 a pageSize of ``3`` and with the bookmark from the previous query. 880 881 :guilabel:`Try it yourself` 882 883 .. code:: bash 884 885 peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarblesWithPagination", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\"}, \"use_index\":[\"_design/indexOwnerDoc\", \"indexOwner\"]}","3","g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkmoKkOWDSOSANIFk2iCyIyVySn5uVBQAGYhR1"]}' 886 887 The following response is received (carriage returns added for clarity). 888 No records are returned, indicating that all pages have been retrieved: 889 890 .. code:: bash 891 892 [] 893 [{"ResponseMetadata":{"RecordsCount":"0", 894 "Bookmark":"g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkmoKkOWDSOSANIFk2iCyIyVySn5uVBQAGYhR1"}}] 895 896 For an example of how a client application can iterate over 897 the result sets using pagination, search for the ``getQueryResultForQueryStringWithPagination`` 898 function in the `Marbles sample <https://github.com/hyperledger/fabric-samples/blob/master/chaincode/marbles02/go/marbles_chaincode.go>`__. 899 900 .. _cdb-update-index: 901 902 Update an Index 903 ~~~~~~~~~~~~~~~ 904 905 It may be necessary to update an index over time. The same index may exist in 906 subsequent versions of the chaincode that gets installed. In order for an index 907 to be updated, the original index definition must have included the design 908 document ``ddoc`` attribute and an index name. To update an index definition, 909 use the same index name but alter the index definition. Simply edit the index 910 JSON file and add or remove fields from the index. Fabric only supports the 911 index type JSON, changing the index type is not supported. The updated 912 index definition gets redeployed to the peer’s state database when the chaincode 913 is installed and instantiated. Changes to the index name or ``ddoc`` attributes 914 will result in a new index being created and the original index remains 915 unchanged in CouchDB until it is removed. 916 917 .. note:: If the state database has a significant volume of data, it will take 918 some time for the index to be re-built, during which time chaincode 919 invokes that issue queries may fail or timeout. 920 921 Iterating on your index definition 922 ---------------------------------- 923 924 If you have access to your peer's CouchDB state database in a development 925 environment, you can iteratively test various indexes in support of 926 your chaincode queries. Any changes to chaincode though would require 927 redeployment. Use the `CouchDB Fauxton interface <http://docs.couchdb.org/en/latest/fauxton/index.html>`__ or a command 928 line curl utility to create and update indexes. 929 930 .. note:: The Fauxton interface is a web UI for the creation, update, and 931 deployment of indexes to CouchDB. If you want to try out this 932 interface, there is an example of the format of the Fauxton version 933 of the index in Marbles sample. If you have deployed the BYFN network 934 with CouchDB, the Fauxton interface can be loaded by opening a browser 935 and navigating to ``http://localhost:5984/_utils``. 936 937 Alternatively, if you prefer not use the Fauxton UI, the following is an example 938 of a curl command which can be used to create the index on the database 939 ``mychannel_marbles``: 940 941 // Index for docType, owner. 942 // Example curl command line to define index in the CouchDB channel_chaincode database 943 944 .. code:: bash 945 946 curl -i -X POST -H "Content-Type: application/json" -d 947 "{\"index\":{\"fields\":[\"docType\",\"owner\"]}, 948 \"name\":\"indexOwner\", 949 \"ddoc\":\"indexOwnerDoc\", 950 \"type\":\"json\"}" http://hostname:port/mychannel_marbles/_index 951 952 .. note:: If you are using BYFN configured with CouchDB, replace hostname:port 953 with ``localhost:5984``. 954 955 .. _cdb-delete-index: 956 957 Delete an Index 958 ~~~~~~~~~~~~~~~ 959 960 Index deletion is not managed by Fabric tooling. If you need to delete an index, 961 manually issue a curl command against the database or delete it using the 962 Fauxton interface. 963 964 The format of the curl command to delete an index would be: 965 966 .. code:: bash 967 968 curl -X DELETE http://localhost:5984/{database_name}/_index/{design_doc}/json/{index_name} -H "accept: */*" -H "Host: localhost:5984" 969 970 971 To delete the index used in this tutorial, the curl command would be: 972 973 .. code:: bash 974 975 curl -X DELETE http://localhost:5984/mychannel_marbles/_index/indexOwnerDoc/json/indexOwner -H "accept: */*" -H "Host: localhost:5984" 976 977 978 979 .. Licensed under Creative Commons Attribution 4.0 International License 980 https://creativecommons.org/licenses/by/4.0/