github.com/joelanford/operator-sdk@v0.8.2/doc/proposals/metering-operator-metrics.md (about)

     1  ## Auto register operator specific metrics as part of operator-metering
     2  
     3  ### Motivation and goal
     4  
     5  We want to be able to generate the metering reports based on the operator specific Prometheus metrics. In order to be able to do that, operators must be instrumented to expose those metrics, and the operator-sdk should make this as easy as possible. The goal is to have the metering happen based on the usage of each individual operator. Metrics will be based on objects managed by the particular operator.
     6  
     7  ### Overview of the metrics
     8  
     9  To follow both the Prometheus instrumentation [best practices](https://prometheus.io/docs/practices/naming/) as well as the official Kubernetes instrumentation [guide](https://github.com/kubernetes/community/blob/cbe9c8ac5f71a99179d7ffe4a008b9018830af72/contributors/devel/sig-instrumentation/instrumentation.md), the metrics will have the following format:
    10  
    11  ```
    12  crd_kind_info{namespace="namespace",crdkind="instance-name"} 1
    13  ```
    14  
    15  example metric for the memchached-operator would look like this:
    16  
    17  ```
    18  memcached_info{namespace="default",memcached="example-memcached"} 1
    19  ```
    20  
    21  ### kube-state-metrics based solution
    22  
    23  The solution makes use of Kubernetes list/watch to populate a Prometheus metrics registry, kube-state-metrics implements its own registry for performance reasons. [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics#overview) is used because it solves exactly the same problem we are facing but it does it for upstream known resources. The operator-sdk can re-use its functionality to perform the same thing but with custom resources. The kube-state-metrics library can only be used for constant (/static) metrics, metrics that are immutable and thereby entirely regenerated on change. This is perfect for our above mentioned use-case. It is not meant to do e.g. counting in performance critical code paths. Thereby an operator would need kube-state-metrics library for exposing the amount of custom resources that it manages and its details and Prometheus client_golang to expose metrics of its own internals e.g. count of reconciliation loops.
    24  
    25  
    26  ```go
    27  // NewCollectors returns a collection of metrics in the namespaces provided, per the api/kind resource.
    28  // The metrics are registered in the custom generateStore function that needs to be defined.
    29  func NewCollectors(
    30      client *Client,
    31      namespaces []string,
    32      api string,
    33      kind string,
    34      metricsGenerator func(obj interface{}) []*metrics.Metric) (collectors []*kcoll.Collector)
    35  ```
    36  
    37  ```go
    38  // ServeMetrics takes in the collectors that were created and port number on which the metrics will be served.
    39  func ServeMetrics(collectors []*kcoll.Collector, portNumber int) {
    40  
    41  ```
    42  
    43  Note: Due to taking advantage of kube-state-metrics functions and interfaces we cannot use the prometheus/client_golang and we need to register it in the same way as kube-state-metrics does, and expose the `/metrics` and serve it on a port (port `:8389/metrics` for example). For that we will need to also create a [Service](https://kubernetes.io/docs/concepts/services-networking/service/) object or rather update the current Service object.
    44  
    45  ### User facing architecture
    46  
    47  Below is how roughly an example for kube-state-metrics implementation will look like.
    48  
    49  User will have all the below code already generated and included as part of the `main.go` file:
    50  
    51  ```go
    52  	c := metrics.NewCollectors(client, []string{"default"}, resource, kind, MetricsGenerator)
    53  	metrics.ServeMetrics(c)
    54  ```
    55  
    56  with the `MetricsGenerator` function living in the users `pkg/metrics` package:
    57  
    58  ```go
    59  var (
    60  	descMemInfo = ksmetrics.NewMetricFamilyDef(
    61  		"memcached_info",
    62  		"The information of the resource instance.",
    63  		[]string{"namespace", "memcached"},
    64  		nil,
    65  	)
    66  )
    67  
    68   func MetricsGenerator(obj interface{}) []*ksmetrics.Metric {
    69  	ms := []*ksmetrics.Metric{}
    70  	crdp := obj.(*unstructured.Unstructured)
    71   	crd := *crdp
    72  
    73  	lv := []string{crd.GetNamespace(), crd.GetName()}
    74  	m, err := ksmetrics.NewMetric(descMemInfo.Name, descMemInfo.LabelKeys, lv,  float64(1))
    75  	if err != nil {
    76  		fmt.Println(err)
    77  		return ms
    78  	}
    79  	ms = append(ms, m)
    80  	return ms
    81  }
    82  ```
    83  
    84  ### Related work
    85  
    86  In the future if the agreed on kube-state-metrics restructure happens (see https://github.com/kubernetes/kube-state-metrics/issues/579) we can get rid of some of the duplicated functions. But that will probably take a few months and our user facing interface should not change as a result.