github.com/TrueBlocks/trueblocks-core/src/apps/chifra@v0.0.0-20241022031540-b362680128f7/internal/scrape/doc.go (about)

     1  // Copyright 2016, 2024 The TrueBlocks Authors. All rights reserved.
     2  // Use of this source code is governed by a license that can
     3  // be found in the LICENSE file.
     4  /*
     5   * Parts of this file were auto generated. Edit only those parts of
     6   * the code inside of 'EXISTING_CODE' tags.
     7   */
     8  
     9  // scrapePkg implements the chifra scrape command.
    10  //
    11  // The chifra scrape application creates TrueBlocks' chunked index of address appearances -- the
    12  // fundamental data structure of the entire system. It also, optionally, pins each chunk of the index
    13  // to IPFS.
    14  //
    15  // chifra scrape is a long running process, therefore we advise you run it as a service or in terminal
    16  // multiplexer such as tmux. You may start and stop chifra scrape as needed, but doing so means the
    17  // scraper will not be keeping up with the front of the blockchain. The next time it starts, it will
    18  // have to catch up to the chain, a process that may take several hours depending on how long ago it
    19  // was last run. See the section below and the "Papers" section of our website for more information
    20  // on how the scraping process works and prerequisites for its proper operation.
    21  //
    22  // You may adjust the speed of the index creation with the --sleep and --block_cnt options. On
    23  // some machines, or when running against some EVM node software, the scraper may overburden the
    24  // hardware. Slowing things down will ensure proper operation. Finally, you may optionally --pin
    25  // each new chunk to IPFS which naturally shards the database among all users. By default, pinning
    26  // is against a locally running IPFS node, but the --remote option allows pinning to an IPFS
    27  // pinning service such as Pinata.
    28  package scrapePkg