github.com/munnerz/test-infra@v0.0.0-20190108210205-ce3d181dc989/logexporter/README.md (about)

     1  # LogExporter
     2  
     3  LogExporter is a tool that runs post-test on our kubernetes test clusters.
     4  It does the job of computing the set of logfiles to be exported (based on the
     5  node type (master/node), cloud provider, and the node's system services),and
     6  then actually exports them to the GCS path provided to it.
     7  
     8  ## How to run the tool?
     9  
    10  To run the tool as a pod on a single node:
    11  
    12  - Fill in the template variables with the right values in cluster/logexporter-pod.yaml
    13  - `kubectl create -f cluster/logexporter-pod.yaml` (for master, run this as a static pod)
    14  
    15  To run the tool as a daemonset on a k8s cluster:
    16  
    17  - Fill in the template variables with the right values in cluster/logexporter-daemonset.yaml
    18  - `kubectl create -f cluster/logexporter-pod.yaml`
    19  - Delete the daemonset after detecting all work has been done as the pods just sleep after uploading logs
    20  
    21  ## Why not other logging tools?
    22  
    23  Open source logging tools like Elasticsearch, Fluentd and Kibana are mostly centred
    24  around taking in a stream of logs, adding metadata to them, creating individual log
    25  entries that are then indexable/searchable/visualizable. We do not want all this
    26  complicated machinery around the logfiles. Firstly because we don't want to affect
    27  the performance of our tests depending on custom logging tools which can significantly
    28  change from one to another. Secondly, a simple multi-threaded file block upload to GCS
    29  (like this tool does) performs way faster than streaming log entries. Finally, having
    30  logs as files on GCS fits into our current test-infra framework, while using these
    31  tools would make log retrieval an API-oriented process, an unneeded complexity.