github.com/diadata-org/diadata@v1.4.593/documentation/tutorials/write-your-own-rate-scraper.md (about)

     1  # Write your own rate scraper
     2  
     3  These instructions concern writing scrapers for single units characterised by a \(floating point\) number. For scrapers describing the relation between pairs of units, i.e. exchange rates see the instructions in exchangescrapers.md.
     4  
     5  ## Instructions for the addition of a rate scraper
     6  
     7  In order to add your own scraper for a new data source, you must adhere to our format. Create the package file `UpdateMYRATE.go` in the package `/internal/pkg/ratescrapers`. The central method is `UpdateMYRATE()`. This method acts on a RateScraper struct which is defined in RateScraper.go in the ratescrapers package. For instance, for the the Euro Short-Term Rate \(ESTER\) issued by the ECB, `UpdateESTER.go` would look like
     8  
     9  ### Practical advice
    10  
    11  Let's assume you want to scrape a data source that provides floating point information. Create a new file in `exchange-scrapers/` and call it `MySourceScraper.go`. The main difference between scrapers is the `Update()` method, where the actual scraping is done.
    12  
    13  ```go
    14  func (s *RateScraper) UpdateESTER() error {
    15    // scraper code here
    16  }
    17  ```
    18  
    19  Here, the type `RateScraper` from the `ratescrapers` package is something like this:
    20  
    21  ```go
    22  type RateScraper struct {
    23      // signaling channels
    24      shutdown chan nothing
    25      shutdownDone chan nothing
    26      // error handling;
    27      errorLock        sync.RWMutex
    28      error            error
    29      closed           bool
    30      ticker           *time.Ticker
    31      datastore        models.Datastore
    32      chanInterestRate chan *models.InterestRate
    33  }
    34  ```
    35  
    36  The scraped data has to be written into a struct of type `InterestRate` from `pkg/model/types.go`
    37  
    38  ```go
    39  type InterestRate struct {
    40      Symbol             string
    41      Value              float64
    42      PublicationTime    time.Time
    43      EffectiveDate      time.Time  
    44      Source             string
    45  }
    46  ```
    47  
    48  and sent to the channel `chanInterestRate` of `s`. In order to write a new scraper, it is not imperative to understand the architecture of the pathway from top to bottom, but it might be helpful. For a first impression you can have a look at the following [diagram](https://github.com/diadata-org/diadata/tree/master/documentation/tutorials/rate_scraper_diagram_down.pdf).
    49