github.com/jenkins-x/test-infra@v0.0.7/greenhouse/README.md (about)

     1  # Greenhouse
     2  
     3  Greenhouse is our [bazel remote caching](https://docs.bazel.build/versions/master/remote-caching.html) setup.
     4  We use this to provide faster build & test presubmits with a Globally shared cache (per repo).
     5  
     6  We have a dashboard with metrics at: [velodrome.k8s.io/dashboard/db/bazel-cache](http://velodrome.k8s.io/dashboard/db/bazel-cache?orgId=1)
     7  
     8  Most Bazel users should probably visit [the official docs](https://docs.bazel.build/versions/master/remote-caching.html) and select one of the options outlined there, with Prow/Kubernetes we are using a custom setup to explore:
     9  
    10  - better support for multiple repos / cache invalidation by changing the cache URL suffix
    11    (see also: `images/bootstrap/create_bazel_cache_rcs.sh`)
    12  - customized cache eviction / management
    13  - integration with our logging and metrics stacks
    14  
    15  
    16  ## Setup (on a Kubernetes Cluster)
    17  We use this with [Prow](./../prow), to set it up we do the following:
    18  
    19   - Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) and [bazel](https://bazel.build/) and Point `KUBECONFIG` at your cluster.
    20     - for k8s.io use `make -C prow get-build-cluster-credentials`
    21   - Create a dedicated node. We use a GKE node-pool with a single node. Tag this node with label `dedicated=greenhouse` and taint `dedicated=greenhouse:NoSchedule` so your other tasks don't schedule on it.
    22     - for k8s.io (running on GKE) this is:
    23     ```
    24     gcloud beta container node-pools create greenhouse --cluster=prow --project=k8s-prow-builds --zone=us-central1-f --node-taints=dedicated=greenhouse:NoSchedule --node-labels=dedicated=greenhouse --machine-type=n1-standard-32 --num-nodes=1
    25     ```
    26     - if you're not on GKE you'll probably want to pick a node to dedicate and do something like:
    27     ```
    28     kubectl label nodes $GREENHOUSE_NODE_NAME dedicated=greenhouse
    29     kubectl taint nodes $GREENHOUSE_NODE_NAME dedicated=greenhouse:NoSchedule
    30     ```
    31   - Create the Kubernetes service so jobs can talk to it conveniently: `kubectl apply -f greenhouse/service.yaml`
    32   - Create a `StorageClass` / `PersistentVolumeClaim` for fast cache storage, we use `kubectl apply -f greenhouse/gce-fast-storage.yaml` for 3TB of pd-ssd storage
    33   - Finally build, push, and deploy with `bazel run //greenhouse:production.apply --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64`
    34     <!--TODO(bentheelder): make this easier to consume by other users?-->
    35     - NOTE: other uses will likely need to tweak this step to their needs, particular the service and storage definitions
    36  
    37  
    38  ## Optional Setup:
    39  - tweak `metrics-service.yaml` and point prometheus at this service to collect metrics