github.com/diadata-org/diadata@v1.4.593/documentation/tutorials/blockchainscraper.md (about)

     1  # Write your own blockchain scraper
     2  
     3  ## Add your own blockchain scraper
     4  
     5  In order to add your own scraper for a blockchain supply source, you must adhere to our format. We use Go modules for our scrapers, so that each data provider is living as an independent module.
     6  
     7  ## Directory structure
     8  
     9  The working directory for blockchain scrapers is `diadata/internal/pkg/blockchain-scrapers/blockchains`. Each scrapper should comply with the following folder structure `currencyName/scrappers/currencySymbol/currencySymbol.go` for example for bitcoin will look like `bitcoin/scrapers/btc/btc.go`. You must provide a Dockerfile for your blockchain client and for the Go wrapper that connects to our database. An example dockerfile is provided for the Go bindings with a `bitcoind` client scraping the Bitcoin blockchain. The file should be named `build/Dockerfile-symbol` for this example `build/Dockerfile-btc`
    10  
    11  ```text
    12  FROM golang:1.14 as build
    13  
    14  WORKDIR $GOPATH
    15  
    16  COPY . .
    17  
    18  WORKDIR $GOPATH/src/github.com/diadata-org/diadata/internal/pkg/blockchain-scrapers/blockchains/bitcoin/scrapers/btc
    19  
    20  RUN go install
    21  
    22  FROM gcr.io/distroless/base
    23  
    24  COPY --from=build /go/bin/btc /bin/btc
    25  
    26  CMD ["btc"]
    27  ```
    28  
    29  After cloning a container capable of executing Go, the files from the working directory are copied into the container. Next, the go program located in `diadata/internal/pkg/blockchain-scrapers/blockchains/bitcoin/scrapers/btc` is built and installed and the finished binary is placed into a mininal distroless container. From there it is executed using the statement in the last line.
    30  
    31  ## Blockchain Client
    32  
    33  The bitcoind itself is initialized directly from the file `docker-compose.bitcoin.yml`. Ideally, all blockchain clients are run from there directly
    34  
    35  ```text
    36  version: '3.2'
    37  
    38  services:
    39    bitcoin:
    40      image:
    41        kylemanna/bitcoind
    42      user: "1000"
    43      volumes:
    44        - /home/srv/bitcoin:/bitcoin
    45      command: btc_oneshot -printtoconsole -prune=550 -rpcallowip=::/0 -disablewallet -rpcpassword=mysecretrpcdiapassword -rpcuser=mysecretrpcdiauser
    46      logging:
    47        options:
    48          max-size: "50m"
    49      networks:
    50        - scrapers-network
    51      deploy:
    52        mode: global
    53        restart_policy:
    54          delay: 2s
    55          window: 20s
    56  ```
    57  
    58  Make sure to enable pruning so that only the latest mined blocks are stored in the container. Add persistency in /home/srv for the daemon data, so it can restart without needing to resynch the blockchain.
    59  
    60  ## Go Wrapper
    61  
    62  For your Go code, also add an entry into the docker-compose file so that your Dockerfile is automatically called. An example entry should look something like this
    63  
    64  ```text
    65    btc:
    66      build:
    67        context: ../../../..
    68      build:
    69        context: $GOPATH
    70        dockerfile: $GOPATH/src/github.com/diadata-org/diadata/build/Dockerfile-btc
    71      image: ${DOCKER_HUB_LOGIN}/${STACKNAME}_btc:latest
    72      networks:
    73        - scrapers-network
    74      logging:
    75        options:
    76          max-size: "50m"
    77      secrets:
    78        - api_diadata
    79  ```
    80  
    81  Be sure to add your container to the `scrapers-network` virtual network. Your credentials for the DIA API should be handled by the `secrets` directive in the section of your blockchain client. To initialize the API from Go you can create
    82  
    83  ```go
    84  var api *dia.Client
    85  ```
    86  
    87  and then call
    88  
    89  ```go
    90  var c dia.ConfigApi
    91  configFile := "/run/secrets/api_diadata"
    92  err := gonfig.GetConf(configFile, &c)
    93  if err != nil {
    94    configFile = "../config/secrets/api_diadata.json"
    95    err = gonfig.GetConf(configFile, &c)
    96  }
    97  if err != nil {
    98    log.Println(err)
    99  } else {
   100    log.Println("Loaded secret in", configFile)
   101  }
   102  api = dia.NewClient(&c)
   103  ```
   104  
   105  For sending supply data to DIA, you can use `SendSupply` in the `ApiClient.go` file.
   106  
   107  This is the basic structure of these files. Run your Docker instance by calling
   108  
   109  ```text
   110  docker run <your instance>
   111  ```
   112  
   113  from the main directory. With `docker-compose up` you can test the entire system with your cryptocurrency VM and your Go wrapper in a virtual network.
   114