github.com/jenkins-x/test-infra@v0.0.7/greenhouse/README.md (about) 1 # Greenhouse 2 3 Greenhouse is our [bazel remote caching](https://docs.bazel.build/versions/master/remote-caching.html) setup. 4 We use this to provide faster build & test presubmits with a Globally shared cache (per repo). 5 6 We have a dashboard with metrics at: [velodrome.k8s.io/dashboard/db/bazel-cache](http://velodrome.k8s.io/dashboard/db/bazel-cache?orgId=1) 7 8 Most Bazel users should probably visit [the official docs](https://docs.bazel.build/versions/master/remote-caching.html) and select one of the options outlined there, with Prow/Kubernetes we are using a custom setup to explore: 9 10 - better support for multiple repos / cache invalidation by changing the cache URL suffix 11 (see also: `images/bootstrap/create_bazel_cache_rcs.sh`) 12 - customized cache eviction / management 13 - integration with our logging and metrics stacks 14 15 16 ## Setup (on a Kubernetes Cluster) 17 We use this with [Prow](./../prow), to set it up we do the following: 18 19 - Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) and [bazel](https://bazel.build/) and Point `KUBECONFIG` at your cluster. 20 - for k8s.io use `make -C prow get-build-cluster-credentials` 21 - Create a dedicated node. We use a GKE node-pool with a single node. Tag this node with label `dedicated=greenhouse` and taint `dedicated=greenhouse:NoSchedule` so your other tasks don't schedule on it. 22 - for k8s.io (running on GKE) this is: 23 ``` 24 gcloud beta container node-pools create greenhouse --cluster=prow --project=k8s-prow-builds --zone=us-central1-f --node-taints=dedicated=greenhouse:NoSchedule --node-labels=dedicated=greenhouse --machine-type=n1-standard-32 --num-nodes=1 25 ``` 26 - if you're not on GKE you'll probably want to pick a node to dedicate and do something like: 27 ``` 28 kubectl label nodes $GREENHOUSE_NODE_NAME dedicated=greenhouse 29 kubectl taint nodes $GREENHOUSE_NODE_NAME dedicated=greenhouse:NoSchedule 30 ``` 31 - Create the Kubernetes service so jobs can talk to it conveniently: `kubectl apply -f greenhouse/service.yaml` 32 - Create a `StorageClass` / `PersistentVolumeClaim` for fast cache storage, we use `kubectl apply -f greenhouse/gce-fast-storage.yaml` for 3TB of pd-ssd storage 33 - Finally build, push, and deploy with `bazel run //greenhouse:production.apply --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64` 34 <!--TODO(bentheelder): make this easier to consume by other users?--> 35 - NOTE: other uses will likely need to tweak this step to their needs, particular the service and storage definitions 36 37 38 ## Optional Setup: 39 - tweak `metrics-service.yaml` and point prometheus at this service to collect metrics