github.com/abayer/test-infra@v0.0.5/metrics/README.md (about)

     1  # Bigquery metrics
     2  
     3  This `metrics-bigquery` job generates metrics that summarize data in our Bigquery
     4  test result database. Each metric is defined with a config file that is consumed
     5  by the `metrics-bigquery` periodic prow job.  Each metric config is a yaml file
     6  like the following:
     7  
     8  ```yaml
     9  # Metric name
    10  metric: failures
    11  # BigQuery query
    12  query: |
    13    #standardSQL
    14    select /* find the most recent time each job passed (may not be this week) */
    15      job,
    16      max(started) latest_pass
    17    from `k8s-gubernator.build.all`
    18    where
    19      result = 'SUCCESS'
    20    group by job
    21  
    22  # JQ filter to make daily results from raw query results
    23  jqfilter: |
    24    [(.[] | select((.latest_pass|length) > 0)
    25    | {(.job): {
    26        latest_pass: (.latest_pass)
    27    }})] | add
    28  
    29  # JQ filter to make influxdb timeseries data points for Velodrome. (Optional)
    30  jqmeasurements: |
    31    [(.[] | select((.latest_pass|length) > 0) | {
    32      measurement: "latest_pass_time",
    33      tags: {
    34        job: (.job)
    35      },
    36      fields: {
    37        job: (.job),
    38        latest_pass: (.latest_pass)
    39    }})]
    40  
    41  ```
    42  
    43  ## Metrics
    44  
    45  * build-stats - number of daily builds and pass rate
    46      - [Config](configs/build-stats.yaml)
    47      - [build-stats-latest.json](http://storage.googleapis.com/k8s-metrics/build-stats-latest.json)
    48  * presubmit-health - presubmit failure rate and timing across PRs
    49      - [Config](configs/presubmit-health.yaml)
    50      - [presubmit-health-latest.json](http://storage.googleapis.com/k8s-metrics/presubmit-health-latest.json)
    51  * failures - find jobs that have been failing the longest
    52      - [Config](configs/failures-config.yaml)
    53      - [failures-latest.json](http://storage.googleapis.com/k8s-metrics/failures-latest.json)
    54  * flakes - find the flakiest jobs this week (and the flakiest tests in each job).
    55      - [Config](configs/flakes-config.yaml)
    56      - [flakes-latest.json](http://storage.googleapis.com/k8s-metrics/flakes-latest.json)
    57  * flakes-daily - find flakes from the previous day. Similar to `flakes`, but creates more granular results for display in Velodrome.
    58      - [Config](configs/flakes-daily-config.yaml)
    59      - [flakes-daily-latest.json](http://storage.googleapis.com/k8s-metrics/flakes-daily-latest.json)
    60  * job-flakes - compute consistency of all jobs
    61      - [Config](configs/job-flakes-config.yaml)
    62      - [job-flakes-latest.json](http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json)
    63  * pr-consistency - calculate PR flakiness for the previous day.
    64      - [Config](configs/pr-consistency-config.yaml)
    65      - [pr-consistency-latest.json](http://storage.googleapis.com/k8s-metrics/pr-consistency-latest.json)
    66  * weekly-consistency - compute overall weekly consistency for PRs
    67      - [Config](configs/weekly-consistency-config.yaml)
    68      - [weekly-consistency-latest.json](http://storage.googleapis.com/k8s-metrics/weekly-consistency-latest.json)
    69  * istio-job-flakes - compute overall weekly consistency for postsubmits
    70      - [Config](configs/istio-flakes.yaml)
    71      - [istio-job-flakes-latest.json](http://storage.googleapis.com/k8s-metrics/istio-job-flakes-latest.json)
    72  
    73  ## Adding a new metric
    74  
    75  To add a new metric, create a PR that adds a new yaml config file
    76  specifying the metric name (`metric`), the bigquery query to execute (`query`), and a
    77  jq filter to filter the data for the daily and latest files (`jqfilter`).
    78  *Optionally*: Include a jqfilter to extract influxdb timeseries measurements
    79  from the raw query results (`jqmeasurements`).
    80  
    81  Run `./bigquery.py --config configs/my-new-config.yaml` and verify that the
    82  output is what you expect.
    83  
    84  Add the new metric to the list above.
    85  
    86  After merging, find the new metric on GCS within 24 hours.
    87  
    88  ## Details
    89  
    90  Each query is run every 24 hours to produce a json
    91  file containing the complete raw query results named with the format
    92  `raw-yyyy-mm-dd.json`. The raw file is then filtered with the associated
    93  jq filter and the results are stored in `daily-yyyy-mm-dd.json`.  These
    94  files are stored in the k8s-metrics GCS bucket in a directory named with
    95  the metric name and persist for a year after their creation. Additionally,
    96  the latest filtered results for a metric are stored in the root of the
    97  k8s-metrics bucket and named with the format `METRICNAME-latest.json`.
    98  
    99  If a config specifies the optional jq filter used to create influxdb timeseries
   100  data points, then the job will use the filter to generate timeseries points from
   101  the raw query results. The points are uploaded to [Velodrome](http://velodrome.k8s.io)'s influxdb instance where they can be used to create graphs and tables.
   102  
   103  ## Consistency
   104  
   105  Consistency means the test, job, pr always produced the same answer. For
   106  example suppose we run a build of a job 5 times at the same commit:
   107  * 5 passing runs, 0 failing runs: consistent
   108  * 0 passing runs, 5 failing runs: consistent
   109  * 1-4 passing runs, 1-4 failing runs: inconsistent aka flaked