github.com/kubewharf/katalyst-core@v0.5.3/docs/proposals/metric/20221202-katalyst-custom-metrics-apiserver.md (about) 1 --- 2 title: Katalyst Custom Metrics APIServer 3 authors: 4 - "waynepeking348" 5 reviewers: 6 - "luomingmeng" 7 - "xuchen-xiaoying" 8 creation-date: 2022-12-02 9 last-updated: 2023-02-23 10 status: implemented 11 --- 12 13 # Katalyst Custom Metrics APIServer 14 15 ## Table of Contents 16 17 <!-- toc --> 18 19 - [Summary](#summary) 20 - [Motivation](#motivation) 21 - [Goals](#goals) 22 - [Non-Goals](#non-goals) 23 - [Proposal](#proposal) 24 - [User Stories](#user-stories) 25 - [Story 1](#story-1) 26 - [Story 2](#story-2) 27 - [Design Overview [Optional]](#design-overview-optional) 28 - [API [Optional]](#api-optional) 29 - [Design Details](#design-details) 30 - [Alternatives](#alternatives) 31 32 <!-- /toc --> 33 34 ## Summary 35 36 Katalyst Custom Metrics APIServer (KCMS for short) is responsible for collecting and storing customized metrics in the production environment, and it implements the standard custom-metrics-apiserver interface for inquiring. It is mainly used by elastic resource recommendation and re-scheduling in the katalyst system. 37 38 ## Motivation 39 40 ### Goals 41 - Provide a customized solution to obtain real-time metrics for systems or modules in katalyst. 42 43 ### Non-Goals 44 - Replace the native custom metrics server. 45 - Works as a general solution for observability. 46 47 ## Proposal 48 In katalyst, many centralized components may depend on customized real-time metrics as critical inputs to generate controlling strategies. 49 50 ### User Stories 51 52 #### Story 1 53 Resource autoscaling (both horizontal and vertical) is a core functionality for cloud-native services. It is usually deployed as a centralized component, and it will dynamically adjust the resource requests for pod or workload level based on real-time running states. To achieve this, the autoscaling module needs to obtain metrics, both for general ones like cpu usage and customized ones like requests qps. Those metrics must be as fresh as possible, and it must be easier enough if we want to add new metrics for more intelligent recommendation strategies. 54 55 #### Story 2 56 Rescheulder or Descheduler is a must-to-have ability in a large cluster, since the running states may change frequently, especially when compared with the time when pods are scheduled. So katalyst will try to rebalance the global scheduling states based on real-time metrics periodically. And the requirements for time-efficiency, stability, reliability, and convenience for new metrics are almost the same as autoscaling. 57 58 ### Design Overview 59 <div align="center"> 60 <picture> 61 <img src="/docs/imgs/custom-metrics-overview.jpg" width=80% title="Katalyst Overview" loading="eager" /> 62 </picture> 63 </div> 64 65 There are four main components in KCMS. 66 - KCMS Agent is embedded as a module in SysAdvisor, and it's responsible for collecting metrics from multiple sources, including raw metrics obtained from malachite, encapsulated metrics generated by SysAdvisor and so on. Those metrics will be exported through standard Prometheus interface and opentelemetry framework. Each metric item will be kept for 5 minutes before GC. 67 - KCMS Collector scrapes metric items from KCMS Agent periodically (default for 5 seconds). It uses label selectors to match KCMS Agent, and always uses Node IP in KCMS Agent Pod as the quering endpoint. After scraping metric, KCMS Collector will send them to KCMS Store. 68 - KCMS Store is an in-memory cache for metrics. It receives metric items from KCMS Collector, and builds indexes in local cache for KCMS Server to query. Each metric item will be kept for 10 minutes before GC. 69 - KCMS Server is actually an extended APIServer, and implements the native custom-metrics-apiserver interface. Native APIServer passes through metric requests to KCMS Server according to the registered APIService, and then KCMS Server proxies the requests to KCMS Store, and returns the responded data list back to Native APIServer. 70 71 ### API 72 KCMS implements the standard custom-metrics-apiserver interface, so users can use metricsClient to refer to those metrics without any extra efforts. For more information about custom-metrics-apiserver (including the basic concepts and common use cases), please refer to [sig-custom-apiserver](https://github.com/kubernetes-sigs/custom-metrics-apiserver). 73 ``` 74 type CustomMetricsProvider interface { 75 // GetMetricByName fetches a particular metric for a particular object. 76 // The namespace will be empty if the metric is root-scoped. 77 GetMetricByName(ctx context.Context, name types.NamespacedName, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValue, error) 78 79 // GetMetricBySelector fetches a particular metric for a set of objects matching 80 // the given label selector. The namespace will be empty if the metric is root-scoped. 81 GetMetricBySelector(ctx context.Context, namespace string, selector labels.Selector, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValueList, error) 82 83 // ListAllMetrics provides a list of all available metrics at 84 // the current time. Note that this is not allowed to return 85 // an error, so it is reccomended that implementors cache and 86 // periodically update this list, instead of querying every time. 87 ListAllMetrics() []CustomMetricInfo 88 } 89 90 // ExternalMetricsProvider is a source of external metrics. 91 // Metric is normally identified by a name and a set of labels/tags. It is up to a specific 92 // implementation how to translate metricSelector to a filter for metric values. 93 // Namespace can be used by the implemetation for metric identification, access control or ignored. 94 type ExternalMetricsProvider interface { 95 GetExternalMetric(ctx context.Context, namespace string, metricSelector labels.Selector, info ExternalMetricInfo) (*external_metrics.ExternalMetricValueList, error) 96 97 ListAllExternalMetrics() []ExternalMetricInfo 98 } 99 100 type MetricsProvider interface { 101 CustomMetricsProvider 102 ExternalMetricsProvider 103 } 104 ``` 105 106 To get information about the detailed custom and external metrics that katalyst supports, please refer to [Katalyst-API](https://github.com/kubewharf/katalyst-api.git), and we plan to provide more metrics along with the implementation of autoscaling and recheduler. 107 108 ### Design Details 109 110 In this section, we will try to introduce the deployment modes that KCMS supports. 111 112 #### StandAlone Mode 113 In StandAlone mode, all centralized components (KCMS Collector, KCMS Store, and KCMS Server) all run in a monolith container. This means that all data dependency is achieved by function calls rather RPC calls. The benefit is performance is strong and deploying is convenient, but reliability is lacking. So it is usually used in testing environments, or in small clusters where it's acceptable to lose some metric. 114 115 <div align="center"> 116 <picture> 117 <img src="/docs/imgs/custom-metrics-monolith-mode.jpg" width=80% title="Katalyst Overview" loading="eager" /> 118 </picture> 119 </div> 120 121 #### Multi Store Mode 122 In Multi Store mode, centralized components run in separated containers, and each component has multiple instances. 123 - For KCMS Collector, only one instance will be the leader while others are cold standbies. Since Collector is stateless, cold standby is enough in case the leader instance crashes and fails to restart. 124 - For KCMS Store, all instances share the same metric contents. KCMS Collector must perform Quorum write and KCMS Server must perform Quorum read to ensure reliability. 125 - For KCMS Serve, all instances are equal to handling querying requests since they are only proxies. 126 Multi Store mode sacrifices some performance to win the most reliability, and is often used in production environments. 127 128 <div align="center"> 129 <picture> 130 <img src="/docs/imgs/custom-metrics-multi-store-mode.jpg" width=80% title="Katalyst Overview" loading="eager" /> 131 </picture> 132 </div> 133 134 #### Future Work: Sharing mode 135 Sharing mode is the future work in case metric items are too large to be stored in one instance. In this mode, both KCMS Collector and KCMS Store will be shared, and each instance will be responsible for an independent piece of items. To ensure reliability, each metric shard will still contain duplicated replicas. 136 137 <div align="center"> 138 <picture> 139 <img src="/docs/imgs/custom-metrics-sharding-mode.jpg" width=80% title="Katalyst Overview" loading="eager" /> 140 </picture> 141 </div> 142 143 ## Alternatives 144 - Expand metrics in the native custom metrics server, but it doesn't support external metrics. Besides, it only contains several metrics in cadvisor, and the cost of adding new metrics is heavy, let along adding labels or something like that. And it doesn't consider reliability, time-efficiency, stability as its main functionalities.