github.com/abayer/test-infra@v0.0.5/velodrome/transform/README.md (about)

     1  Overview
     2  ========
     3  
     4  Transform is used to create relevant metrics from the data saved in the SQL
     5  database. The SQL database contains the issues and the list of events,
     6  but we may want to calculate additional metrics that reflect the health of
     7  the project.  For example, we may want to understand when certain labels were
     8  applied, and what happened to the pull-request or issues through its lifetime.
     9  
    10  This logic is written as "Plugins". A quick look at the code
    11  ([plugins.go](plugins.go)) explains the interface of a plugin, and the type of
    12  parameters it receives.
    13  
    14  Plugins either wait for:
    15  - Changes to issues (they come by order of modification)
    16  - Events and comments: they come sorted by creation date.
    17  
    18  Note that you will always receives changes to issues before receiving events or
    19  comments.
    20  
    21  The program periodically fetches from the SQL database to find changes and
    22  pushes them to each plugin.
    23  
    24  Walk-through: Creating a new plugin
    25  ===================================
    26  
    27  To create a plugin, you need to implement the interface defined in
    28  [plugins.go](plugins.go), and also register its creation there.
    29  
    30  Let's review an example plugin, [merged.go](merged.go):
    31  
    32  - At registration time, `NewMergedPlugin` tries to get the last measurement from
    33    the InfluxDB, so that it doesn't have to store data it has already written.
    34    Every time a plugin is run, it receives all historical events from the SQL
    35    Database, and it processes all of them, but only stores new ones.
    36  
    37  - We need to implement `ReceiveIssue`, `ReceiveComment`, and `ReceiveIssueEvent`
    38    even if we don't use all of them.
    39  
    40  - When we receive an Event, in this situation, we want to count how many
    41    "merged" events we received, so we just discard all events other than
    42    "merged". Then we discard events that are already in the database because they
    43    we have already processed them. And finally, if the event is of the correct
    44    type and we have not already processed it, we insert the value in the database
    45    through the InfluxDB interface.
    46  
    47  - We make sure the plugin is registered in [plugins.go](plugins.go) or it's
    48    never going to receive any event.
    49  
    50  Testing locally
    51  ===============
    52  
    53  In order to test this program locally, you will need a database set-up in MySQL,
    54  populated with data. Refer to [../fetcher](../fetcher/)
    55  documentation to see how to populate your own database.
    56  
    57  Once it is set-up, you will also need the grafana-stack set-up locally. Refer to
    58  [../grafana-stack](../grafana-stack/) to see how to do that.
    59  
    60  You can then run `transform` to connect to your local instances.