github.com/kubewharf/katalyst-core@v0.5.3/docs/proposals/metric/20221202-katalyst-custom-metrics-apiserver.md (about)

     1  ---
     2  title: Katalyst Custom Metrics APIServer
     3  authors:
     4    - "waynepeking348"
     5  reviewers:
     6    - "luomingmeng"
     7    - "xuchen-xiaoying"
     8  creation-date: 2022-12-02
     9  last-updated: 2023-02-23
    10  status: implemented
    11  ---
    12  
    13  # Katalyst Custom Metrics APIServer
    14  
    15  ## Table of Contents
    16  
    17  <!-- toc -->
    18  
    19  - [Summary](#summary)
    20  - [Motivation](#motivation)
    21      - [Goals](#goals)
    22      - [Non-Goals](#non-goals)
    23  - [Proposal](#proposal)
    24      - [User Stories](#user-stories)
    25          - [Story 1](#story-1)
    26          - [Story 2](#story-2)
    27      - [Design Overview [Optional]](#design-overview-optional)
    28      - [API [Optional]](#api-optional)
    29      - [Design Details](#design-details)
    30  - [Alternatives](#alternatives)
    31  
    32  <!-- /toc -->
    33  
    34  ## Summary
    35  
    36  Katalyst Custom Metrics APIServer (KCMS for short) is responsible for collecting and storing customized metrics in the production environment, and it implements the standard custom-metrics-apiserver interface for inquiring. It is mainly used by elastic resource recommendation and re-scheduling in the katalyst system.
    37  
    38  ## Motivation
    39  
    40  ### Goals
    41  - Provide a customized solution to obtain real-time metrics for systems or modules in katalyst.
    42  
    43  ### Non-Goals
    44  - Replace the native custom metrics server.
    45  - Works as a general solution for observability.
    46  
    47  ## Proposal
    48  In katalyst, many centralized components may depend on customized real-time metrics as critical inputs to generate controlling strategies.
    49  
    50  ### User Stories
    51  
    52  #### Story 1
    53  Resource autoscaling (both horizontal and vertical) is a core functionality for cloud-native services. It is usually deployed as a centralized component, and it will dynamically adjust the resource requests for pod or workload level based on real-time running states. To achieve this, the autoscaling module needs to obtain metrics, both for general ones like cpu usage and customized ones like requests qps. Those metrics must be as fresh as possible, and it must be easier enough if we want to add new metrics for more intelligent recommendation strategies.
    54  
    55  #### Story 2
    56  Rescheulder or Descheduler is a must-to-have ability in a large cluster, since the running states may change frequently, especially when compared with the time when pods are scheduled. So katalyst will try to rebalance the global scheduling states based on real-time metrics periodically. And the requirements for time-efficiency, stability, reliability, and convenience for new metrics are almost the same as autoscaling.
    57  
    58  ### Design Overview
    59  <div align="center">
    60    <picture>
    61      <img src="/docs/imgs/custom-metrics-overview.jpg" width=80% title="Katalyst Overview" loading="eager" />
    62    </picture>
    63  </div>
    64  
    65  There are four main components in KCMS.
    66  - KCMS Agent is embedded as a module in SysAdvisor, and it's responsible for collecting metrics from multiple sources, including raw metrics obtained from malachite, encapsulated metrics generated by SysAdvisor and so on. Those metrics will be exported through standard Prometheus interface and opentelemetry framework. Each metric item will be kept for 5 minutes before GC.
    67  - KCMS Collector scrapes metric items from KCMS Agent periodically (default for 5 seconds). It uses label selectors to match KCMS Agent, and always uses Node IP in KCMS Agent Pod as the quering endpoint. After scraping metric, KCMS Collector will send them to KCMS Store.
    68  - KCMS Store is an in-memory cache for metrics. It receives metric items from KCMS Collector, and builds indexes in local cache for KCMS Server to query. Each metric item will be kept for 10 minutes before GC.
    69  - KCMS Server is actually an extended APIServer, and implements the native custom-metrics-apiserver interface. Native APIServer passes through metric requests to KCMS Server according to the registered APIService, and then KCMS Server proxies the requests to KCMS Store, and returns the responded data list back to Native APIServer.
    70  
    71  ### API
    72  KCMS implements the standard custom-metrics-apiserver interface, so users can use metricsClient to refer to those metrics without any extra efforts. For more information about custom-metrics-apiserver (including the basic concepts and common use cases), please refer to [sig-custom-apiserver](https://github.com/kubernetes-sigs/custom-metrics-apiserver).
    73  ```
    74  type CustomMetricsProvider interface {
    75     // GetMetricByName fetches a particular metric for a particular object.
    76     // The namespace will be empty if the metric is root-scoped.
    77     GetMetricByName(ctx context.Context, name types.NamespacedName, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValue, error)
    78  
    79     // GetMetricBySelector fetches a particular metric for a set of objects matching
    80     // the given label selector. The namespace will be empty if the metric is root-scoped.
    81     GetMetricBySelector(ctx context.Context, namespace string, selector labels.Selector, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValueList, error)
    82  
    83     // ListAllMetrics provides a list of all available metrics at
    84     // the current time. Note that this is not allowed to return
    85     // an error, so it is reccomended that implementors cache and
    86     // periodically update this list, instead of querying every time.
    87     ListAllMetrics() []CustomMetricInfo
    88  }
    89  
    90  // ExternalMetricsProvider is a source of external metrics.
    91  // Metric is normally identified by a name and a set of labels/tags. It is up to a specific
    92  // implementation how to translate metricSelector to a filter for metric values.
    93  // Namespace can be used by the implemetation for metric identification, access control or ignored.
    94  type ExternalMetricsProvider interface {
    95     GetExternalMetric(ctx context.Context, namespace string, metricSelector labels.Selector, info ExternalMetricInfo) (*external_metrics.ExternalMetricValueList, error)
    96  
    97     ListAllExternalMetrics() []ExternalMetricInfo
    98  }
    99  
   100  type MetricsProvider interface {
   101     CustomMetricsProvider
   102     ExternalMetricsProvider
   103  }
   104  ```
   105  
   106  To get information about the detailed custom and external metrics that katalyst supports, please refer to [Katalyst-API](https://github.com/kubewharf/katalyst-api.git), and we plan to provide more metrics along with the implementation of autoscaling and recheduler.
   107  
   108  ### Design Details
   109  
   110  In this section, we will try to introduce the deployment modes that KCMS supports.
   111  
   112  #### StandAlone Mode
   113  In StandAlone mode, all centralized components (KCMS Collector, KCMS Store, and KCMS Server) all run in a monolith container. This means that all data dependency is achieved by function calls rather RPC calls. The benefit is performance is strong and deploying is convenient, but reliability is lacking. So it is usually used in testing environments, or in small clusters where it's acceptable to lose some metric.
   114  
   115  <div align="center">
   116    <picture>
   117      <img src="/docs/imgs/custom-metrics-monolith-mode.jpg" width=80% title="Katalyst Overview" loading="eager" />
   118    </picture>
   119  </div>
   120  
   121  #### Multi Store Mode
   122  In Multi Store mode, centralized components run in separated containers, and each component has multiple instances.
   123  - For KCMS Collector, only one instance will be the leader while others are cold standbies. Since Collector is stateless, cold standby is enough in case the leader instance crashes and fails to restart.
   124  - For KCMS Store, all instances share the same metric contents. KCMS Collector must perform Quorum write and KCMS Server must perform Quorum read to ensure reliability.
   125  - For KCMS Serve, all instances are equal to handling querying requests since they are only proxies.
   126  Multi Store mode sacrifices some performance to win the most reliability, and is often used in production environments.
   127  
   128  <div align="center">
   129    <picture>
   130      <img src="/docs/imgs/custom-metrics-multi-store-mode.jpg" width=80% title="Katalyst Overview" loading="eager" />
   131    </picture>
   132  </div>
   133  
   134  #### Future Work: Sharing mode
   135  Sharing mode is the future work in case metric items are too large to be stored in one instance. In this mode, both KCMS Collector and KCMS Store will be shared, and each instance will be responsible for an independent piece of items. To ensure reliability, each metric shard will still contain duplicated replicas.
   136  
   137  <div align="center">
   138    <picture>
   139      <img src="/docs/imgs/custom-metrics-sharding-mode.jpg" width=80% title="Katalyst Overview" loading="eager" />
   140    </picture>
   141  </div>
   142  
   143  ## Alternatives
   144  - Expand metrics in the native custom metrics server, but it doesn't support external metrics. Besides, it only contains several metrics in cadvisor, and the cost of adding new metrics is heavy, let along adding labels or something like that. And it doesn't consider reliability, time-efficiency, stability as its main functionalities.