github.com/GoogleCloudPlatform/testgrid@v0.0.174/cmd/updater/README.md (about)

     1  
     2  # Testgrid Updater
     3  
     4  This component is responsible for compiling collated GCS results into a [state proto].
     5  
     6  These protos are read by the [Tabulator] component.
     7  
     8  ## Local development
     9  See also [common tips](/cmd/README.md) for running locally.
    10  
    11  ```bash
    12  # --config can take a local file (e.g. `/tmp/testgrid/config`) or GCS file (e.g. `gs://my-testgrid-bucket/config`)
    13  bazelisk run //cmd/updater -- \
    14    --config=gs://my-testgrid-bucket/somewhere/config \
    15    # --test-group=foo,bar \  # If specified, only update these test groups.
    16    # --debug \
    17    # --confirm \
    18  ```
    19  
    20  ### Debugging
    21  
    22  The two most useful flags are `--test-group=foo` and `--confirm=false` (default).
    23  
    24  Setting the `--test-group` flag tells the client to only update a specific group.
    25  This is useful when debugging problems in a specific group.
    26  
    27  The `--confirm` flag controls whether any data is written. Nothing is written by default.
    28  
    29  
    30  ## Update cycles
    31  
    32  Each update cycle the updater:
    33  
    34  * Downloads the specified config proto to get the list of test groups.
    35  * Iterates through each group
    36    - Downloads the existing state proto if present
    37      * Drops the oldest and newest columns
    38      * Old ones are no longer relevant
    39      * New columns may still change
    40    - Grabs the gcs prefix
    41    - Scans GCS under that prefix for results greater than existing ones
    42      * Each job is in a unique GCS\_PREFIX/JOB\_ID folder
    43      * New folders must be monotonically greater than old ones
    44    - Compiles the job result in each folder into a cell mapping
    45    - Converts the cell into the existing state grid proto.
    46      * Appends a new column into the state grid.
    47      * Creates any new rows.
    48      * Appends data to existing rows.
    49  * Determines which (if any) rows have alerts
    50  * Optionally uploads the proto to GCS
    51  
    52  If the `--wait` flag is unset, the job returns at this time.
    53  
    54  Otherwise it repeats after sleeping for that duration.
    55  
    56  [state proto]: /pb/state/state.proto
    57  [Tabulator]: /cmd/tabulator