github.com/kubewharf/katalyst-core@v0.5.3/docs/proposals/qos-management/reporter-manager/20220515-reporter-manager.md (about) 1 --- 2 title: Katalyst Reporter Manager 3 authors: 4 - "luomingmeng" 5 reviewers: 6 - "waynepeking348" 7 - "csfldf" 8 - "NickrenREN" 9 creation-date: 2022-05-15 10 last-updated: 2023-02-22 11 status: implemented 12 --- 13 14 # Katalyst Reporter Manager 15 16 ## Table of Contents 17 18 <!-- toc --> 19 20 - [Summary](#summary) 21 - [Motivation](#motivation) 22 - [Goals](#goals) 23 - [Non-Goals](#non-goals) 24 - [Proposal](#proposal) 25 - [User Stories](#user-stories) 26 - [Story 1](#story-1) 27 - [Story 2](#story-2) 28 - [Design Overview [Optional]](#design-overview-optional) 29 - [API [Optional]](#api-optional) 30 - [Design Details](#design-details) 31 - [Alternatives](#alternatives) 32 33 <!-- /toc --> 34 35 ## Summary 36 37 Katalyst expands the node resource with a new CRD Custom Node Resource (kcnr for short). This CRD contains both topology aware resources and reclaimed resources, but different fields in CNR may be collected through different sources. For instance, reclaimed resources are collected from SysAdvisor according to container running status, while topology aware resources are collected by each pluggable plugin. If each plugin patches CNR by itself, there may exist many API conflicts, and the QPS for API update requests may rise uncontrollably. 38 39 To solve this, Reporter Manager implements a common framework. Users can implement each resource reporter with a pluggable plugin, and the manager will merge the reported resources into the corresponding fields, and update the CR through one API request. 40 41 ## Motivation 42 43 ### Goals 44 - Implement a common framework to merge different fields for node-level CR, and update through one API request. 45 - Make the resource reporting component pluggable. 46 47 ### Non-Goals 48 - Replace native resource representations in nodes. 49 - Replace device manager or device plugins for scalar resources in nodes. 50 - Implement the general resource, metrics, or device reporting logic. 51 52 ## Proposal 53 54 ### User Stories 55 56 #### Story 1 57 SysAdvisor is the core node-level resource recommendation module in Katalyst, and it will predict the resource requirements for non-reclaimed pods in real-time. If allocatable resources are surplus after matching up with the resource requirements, Katalyst will try to schedule reclaimed pods into this node for better resource utilization. And thus those reclaimed resources should be reported in CNR. 58 59 Since the reporting source is in SysAdvisor, Reporter Manager provides a mechanism for SysAdvisor to register an out-of-tree resource reporting plugin. So the boundary is clear, SysAdvisor is responsible for calculation logics for reclaimed resources, while Reporter Manager is responsible for updating it in CNR. 60 61 #### Story 2 62 In the production environment, we may have a lot of resources or devices that are affiliated with micro topology. To make those micro topology info appreciable in scheduling process, katalyst should report them in CNR. But those resources or devices are usually bounded by specific vendors. If each vendor updates CNR by itself, there may exist many API conflicts, and the QPS for API update requests may rise uncontrollably. 63 64 In this case, Reporter Manager works as a coordinator. Each vendor can implement its own reporter plugin, and register itself into Reporter Manager. The latter will merge all the reporting fields, fix the conflicts, and update CNT through one API request. 65 66 ### Design Overview 67 <div align="center"> 68 <picture> 69 <img src="/docs/imgs/reporter-manager-overview.jpg" width=80% title="Katalyst Overview" loading="eager" /> 70 </picture> 71 </div> 72 73 - Reporter is responsible for updating CR to APIServer. For each CRD, there should exist one corresponding reporter. It receives all reporting fields from the Reporter Plugin Manager, validates the legality, merges them into the CR, and finally updates by clientset. 74 - Reporter Plugin Manager is the core framework. It listens to the registration events from each Reporter Plugin, periodically poll each plugin to obtain reporting info, and dispatch those info to different Reporters. 75 - Reporter Plugin is implemented by each resource or device vendor, and it is registered into Reporter Plugin Manager before it starts. Whenever it is called by Reporter Plugin Manager, it should return the fresh resource or device info for return, along with enough information about which CRD and which field those info are mapped with. 76 77 ### API 78 Reporter Plugin communicates with Reporter Plugin Manager with GPRC, and the protobuf is shown as below. Each Reporter Plugin can report resources for more than one CRD, so it should explicitly nominate the GVR and fields in the protobuf. 79 ``` 80 message Empty { 81 } 82 83 enum FieldType { 84 Spec = 0; 85 Status = 1; 86 Metadata = 2; 87 } 88 89 message ReportContent { 90 k8s.io.apimachinery.pkg.apis.meta.v1.GroupVersionKind groupVersionKind = 1; 91 repeated ReportField field = 2; 92 } 93 94 message ReportField { 95 FieldType fieldType = 1; 96 string fieldName = 2; 97 bytes value = 3; 98 } 99 100 message GetReportContentResponse { 101 repeated ReportContent content = 1; 102 } 103 104 service ReporterPlugin { 105 rpc GetReportContent(Empty) returns (GetReportContentResponse) {} 106 107 rpc ListAndWatchReportContent(Empty) returns (stream GetReportContentResponse) {} 108 } 109 ``` 110 111 Reporter Plugin Manager will implement the interfaces below. 112 ``` 113 type AgentPluginHandler interface { 114 GetHandlerType() string // "ReportePlugin" 115 PluginHandler 116 } 117 118 type PluginHandler interface { 119 // Validate returns an error if the information provided by 120 // the potential plugin is erroneous (unsupported version, ...) 121 ValidatePlugin(pluginName string, endpoint string, versions []string) error 122 // RegisterPlugin is called so that the plugin can be register by any 123 // plugin consumer 124 // Error encountered here can still be Notified to the plugin. 125 RegisterPlugin(pluginName, endpoint string, versions []string) error 126 // DeRegister is called once the pluginwatcher observes that the socket has 127 // been deleted. 128 DeRegisterPlugin(pluginName string) 129 } 130 ``` 131 132 133 Currently, katalyst only supports updating resources for CNR and native Node object. If you want to add a new node-level CRD, and report resources using this mechanism, you should implement a new Reporter in katalyst-core with the following interface. 134 ``` 135 type Reporter interface { 136 // Update receives ReportField list from report manager, the reporter implementation 137 // should be responsible for assembling and updating the specific object 138 Update(ctx context.Context, fields []*v1alpha1.ReportField) error 139 // Run starts the syncing logic of reporter 140 Run(ctx context.Context) 141 } 142 ``` 143 144 ### Design Details 145 To simplify, Reporter Manager only supports FieldType at the top-level of each object, i.e. only Status, Spec, Metadata are supported. If the object has embedded struct in Spec, you should explicitly nominate them in `FieldName` 146 147 For the `FieldName`, we will introduce the details about how to fill up the protobuf for each type. 148 149 #### Map 150 Different Reporter Plugin can report to the same Map field with different keys, and Reporter Manager will merge them together. But if multiple plugins report to the same Map field with the same key, Reporter Manager will always override the former one with the latter one, and the priority is based on the registration orders to make sure the override results are always deterministic. 151 - Plugin A (the former) 152 ``` 153 Field: []*v1alpha1.ReportField{ 154 { 155 FieldType: v1alpha1.FieldType_Status, 156 FieldName: "ReclaimedResourceCapacity", 157 Value: []byte(`&{"cpu": "10"}`), 158 }, 159 } 160 ``` 161 - Plugin B (the latter) 162 ``` 163 Field: []*v1alpha1.ReportField{ 164 { 165 FieldType: v1alpha1.FieldType_Status, 166 FieldName: "ReclaimedResourceCapacity", 167 Value: []byte(`&{"memory": "24Gi"}`), 168 }, 169 } 170 ``` 171 - Result (json) 172 ``` 173 { 174 "reclaimedResourceCapacity": { 175 "cpu": "10", 176 "memory": "24Gi" 177 } 178 } 179 ``` 180 181 #### Slice and Array 182 Different Reporter Plugin can report to the same Slice or Array field, and Reporter Manager will merge them together. 183 - Plugin A 184 ``` 185 Field: []*v1alpha1.ReportField{ 186 { 187 FieldType: v1alpha1.FieldType_Status, 188 FieldName: "ResourceStatus", 189 Value: []byte(`[&{"numa": "numa-1", "available": {"device-1": "12"}}]`), 190 }, 191 } 192 ``` 193 - Plugin B 194 ``` 195 Field: []*v1alpha1.ReportField{ 196 { 197 FieldType: v1alpha1.FieldType_Status, 198 FieldName: "ResourceStatus", 199 Value: []byte(`[&{"numa": "numa-2", "available": {"device-2": "13"}}]`), 200 }, 201 } 202 ``` 203 - Result (json) 204 ``` 205 { 206 "resourceStatus": [ 207 { 208 &{"numa": "numa-1", "available": {"device-1": "12"}}, 209 &{"numa": "numa-2", "available": {"device-2": "13"}} 210 } 211 ] 212 } 213 ``` 214 #### Common Field 215 If different Reporter Plugins try to report to the same common field (int, string, ...), Reporter Manager will always override the former one with the latter one, and the priority is based on the registration orders to make sure the override results are always deterministic. 216 - Plugin A (the former) 217 ``` 218 Field: []*v1alpha1.ReportField{ 219 { 220 FieldType: v1alpha1.FieldType_Status, 221 FieldName: "TopologyStatus", 222 Value: []byte(`&{"sockets": [&{"socketID": 0}]}`), 223 }, 224 } 225 ``` 226 - Plugin B (the latter) 227 ``` 228 Field: []*v1alpha1.ReportField{ 229 { 230 FieldType: v1alpha1.FieldType_Status, 231 FieldName: "TopologyStatus", 232 Value: []byte(`&{"sockets": [&{"socketID": 1}]}`), 233 }, 234 } 235 ``` 236 - Result (json) 237 ``` 238 { 239 "topologyStatus": &{ 240 "sockets": [&{"socketID": 1}] 241 } 242 } 243 ``` 244 ## Alternatives 245 - All CNR resources are collected and updated by a monolith component, but this may cause this monolith component too complicated to maintain. In the meantime, all logics must be implemented in-tree, which will it be inconvenient to extend if new devices are needed.