github.com/yrj2011/jx-test-infra@v0.0.0-20190529031832-7a2065ee98eb/kettle/README.md (about)

     1  # KETTLE -- Kubernetes Extract Tests/Transform/Load Engine
     2  
     3  This collects test results scattered across a variety of GCS buckets,
     4  stores them in a local SQLite database, and outputs newline-delimited
     5  JSON files for import into BigQuery.
     6  
     7  Results are stored in the [k8s-gubernator:build BigQuery dataset](https://bigquery.cloud.google.com/dataset/k8s-gubernator:build),
     8  which is publicly accessible.
     9  
    10  # Running
    11  
    12  Use `pip install -r requirements.txt` to install dependencies.
    13  
    14  # Deploying
    15  
    16  Kettle runs as a pod in the `k8s-gubernator/g8r` cluster
    17  
    18  If you change:
    19  
    20  - `buckets.yaml`: do nothing, it's automatically fetched from GitHub
    21  - `deployment.yaml`: deploy with `make push deploy`
    22  - any code: deploy with `make push update`, revert with `make rollback` if it fails
    23  
    24  # Restarting
    25  
    26  #### Find out when the build started failing
    27  
    28  eg: by looking at the logs
    29  
    30  ```sh
    31  make get-cluster-credentials
    32  kubectl logs -l app=kettle
    33  
    34  # ...
    35  
    36  ==== 2018-07-06 08:19:05 PDT ========================================
    37  PULLED 174
    38  ACK irrelevant 172
    39  EXTEND-ACK  2
    40  gs://kubernetes-jenkins/pr-logs/pull/kubeflow_kubeflow/1136/kubeflow-presubmit/2385 True True 2018-07-06 07:51:49 PDT FAILED
    41  gs://kubernetes-jenkins/logs/ci-cri-containerd-e2e-ubuntu-gce/5742 True True 2018-07-06 07:44:17 PDT FAILURE
    42  ACK "finished.json" 2
    43  Downloading JUnit artifacts.
    44  ```
    45  
    46  Alternatively, navigate to [Gubernator BigQuery page](https://bigquery.cloud.google.com/table/k8s-gubernator:build.all?pli=1&tab=details) (click on “all” on the left and “Details”) and you can see a table showing last date/time the metrics were collected.
    47  
    48  #### Replace pods
    49  
    50  ```sh
    51  kubectl delete pod -l app=kettle
    52  kubectl rollout status deployment/kettle # monitor pod restart status
    53  kubectl get pod -l app=kettle # should show a new pod name
    54  ```
    55  
    56  #### Verify functionality
    57  
    58  You can watch the pod startup and collect data from various GCS buckets by looking at its logs
    59  
    60  ```sh
    61  kubectl logs -f $(kubectl get pod -l app=kettle -oname)
    62  ```
    63  
    64  It might take a couple of hours to be fully functional and start updating BigQuery. You can always go back to the [Gubernator BigQuery page](https://bigquery.cloud.google.com/table/k8s-gubernator:build.all?pli=1&tab=details) and check to see if data collection has resumed.  Backfill should happen automatically.
    65  
    66  # Known Issues
    67  
    68  - Occasionally data from Kettle stops updating, we suspect this is due to a transient hang when contacting GCS ([#8800](https://github.com/kubernetes/test-infra/issues/8800)). If this happens, [restart kettle](#restarting)
    69