github.com/oam-dev/kubevela@v1.9.11/design/vela-core/APIServer-Catalog.md (about) 1 # APIServer + Catalog Architecture Design 2 3 ## Summary 4 5 In KubeVela, APIServer provides the RESTful API for external systems (e.g. UI) to manage Vela abstractions like Applications, Definitions; Catalog stores templates to install common-off-the-shell (COTS) capabilities on Kubernetes. 6 7 This doc provides a top-down architecture design for Vela APIServer and Catalog. It clarifies the API interfaces for platform builders to build integration solutions and describes the architecture design in details for the incoming roadmap. Some of the interfaces might have not been implemented yet, but we will follow this design in the future project roadmap. 8 9 ## Motivation 10 11 This design is based on and tries to resolve the following use cases: 12 13 1. UI component wants to discover APIs to integrate with Vela APIServer. 14 1. Users want to manage multiple clusters, catalogues, configuration environments in a single place. 15 1. The management data can be stored in a cloud database like MySQL instead of k8s control plane. 16 1. Because there aren't control logic for those data. This is unlike other Vela resources stored as CR in K8s control plane. 17 1. It is more expensive to host a k8s control plane than MySQL database on cloud. 18 1. Users want to manage third-party capabilities via a well-defined, standard Catalog API interface and Package format. 19 1. Here is the workflow of creating an application: 20 - The user chooses an environment 21 - The user chooses one or more clusters in the environment 22 - The user configures the service deployment configuration such as replica count, instance labels, container image, domain name, port number, etc. 23 24 ## Proposal 25 26 ### 1. Top-down Architecture 27 28 The overall architecture diagram: 29 30  31 32 Here's some explanation of the diagram: 33 34 - UI requests APIServer for data to render the dashboard. 35 - Vela APIServer aggregates data from different kinds of sources. 36 - Vela APIServer can sync with multiple k8s clusters. 37 - Vela APIServer can sync with multiple catalog servers. 38 - For those data not in k8s or catalog servers, Vela APIServer syncs them in MySQL database. 39 40 The above architecture implies that the Vela APIServer could be used to multiple k8s clusters and catalogs. Below is what a deployment of Vela platform would look like: 41 42  43 44 ### 2. API Design 45 46 Below is the overall architecture of API grouping and storage: 47 48  49 50 There are two distinguished layers: 51 - **API layer**: It defines the API discovery and serving endpoints that Vela APIServer implementation must follow. This is the integration point for external system components (e.g. UI) to contact. 52 - **Storage layer**: It describes the storage systems and objects that Vela APIServer syncs data with behind the scene. There are three types of storage: 53 - **K8s cluster**: Vela APIServer manages multiple k8s clusters with regard to the applications and definitions custom resources. 54 - **Catalog server**: Vela APIServer manages multiple catalogs which contain COTS application packages. Currently in our use case the catalogs resides in Git repos. In the future we can extend this to other catalog storage like file server, object storage. 55 - **MySQL database**: Vela APIServer stores global, cross-cluster, cross catalog information in a MySQL database. These data do not exist in k8s or catalog and thus need to be managed by APIServer in a separate database. The database is usually hosted on cloud. 56 57 #### Environment API 58 59 Environment is a collection of configurations of the application and dependent resources. For example, a developer would define `production` and `staging` environments where each have different config of routing, scaling, database connection credentials, etc. 60 61 - `/environments` 62 - Description: List all environments 63 ```json 64 [{"id": "env-1"}, {"id": "env-2"}] 65 ``` 66 - `/environments/<env>` 67 - Description: CRUD operations of an environment. 68 ```json 69 { 70 "id": "env-1", 71 72 // The clusters bound to this environment 73 "clusters": [ 74 { 75 "id": "cluster-1" 76 } 77 ], 78 79 "config": { 80 "foo": "bar" 81 } 82 } 83 ``` 84 85 #### Application API 86 87 Application is a global unit to manage cross-cluster, cross-namespace application deployment. 88 89 - `/applications` 90 - Description: List all applications 91 - `/applications/<app>` 92 - Description: CRUD operations of an application. 93 ```json 94 { 95 "id": "app-1", 96 97 "configuration": { 98 "services": { 99 "testsvc": { 100 "type": "webservice", 101 "image": "crccheck/hello-world" 102 } 103 } 104 }, 105 106 // An application could be deployed to multiple environments 107 "environments": [ 108 { 109 "id": "env-1", 110 // In each env it could deploy to multiple clusters 111 "clusters": [ 112 { 113 "id": "cluster-1", 114 "deployments": [ 115 { 116 "id": "deploy-1", 117 "namespace": "ns-1" 118 } 119 ] 120 } 121 ] 122 } 123 ] 124 } 125 ``` 126 127 #### Cluster API 128 129 - `/clusters` 130 - Description: List all k8s clusters. 131 - `/clusters/<cluster>` 132 - Description: CRUD operations of a k8s cluster 133 ```json 134 { 135 "id": "cluster-1", 136 137 // The definitions indicate the capabilities enabled in this cluster 138 "definitions": [ 139 { 140 "id": "def-1" 141 } 142 ], 143 144 "deployments": [ 145 { 146 "id": "deploy-1" 147 } 148 ] 149 } 150 ``` 151 152 - `/clusters/<cluster-id>/deployments` 153 - Description: List all application deployments on a cluster. 154 - `/clusters/<cluster-id>/deployments/<deploy>` 155 - Description: CRUD operations of an application deployment on a cluster. 156 ```json 157 { 158 "id": "deploy-1", 159 "namespace": "ns-1", 160 "resources": {}, 161 "status": {}, 162 } 163 ``` 164 165 - `/clusters/<cluster-id>/definitions` 166 - Description: List all definitions installed on a cluster. 167 - `/clusters/<cluster-id>/definitions/<def>` 168 - Description: CRUD operations of a definition on a cluster. 169 ```json 170 { 171 "id": "def-1", 172 "namespace": "ns-1", 173 "kind": "TraitDefinition", 174 "spec": { 175 "template": "..." 176 } 177 } 178 ``` 179 180 #### Catalog API 181 182 - `/catalogs` 183 - Description: List all catalogs. 184 - `/catalogs/<catalog>` 185 - Description: CRUD operations of a catalog. 186 ```json 187 { 188 "namespace": "ns-1", 189 "name": "catalog-1", 190 "address": "example.com:8080", 191 "protocols": { 192 "git": { 193 "root_dir": "catalog/" 194 } 195 } 196 } 197 ``` 198 - `/catalogs/<catalog>/sync` 199 - Description: Sync this catalog. 200 - `/catalogs/<catalog>/packages` 201 - Description: List latest-version packages on a catalog. 202 - Param @label: Select packages based on label indexing. 203 Multiple labels could be specified via query parameters. 204 - `/catalogs/<catalog>/packages/<pkg>` 205 - Description: List all versions of a package. 206 - `/catalogs/<catalog>/packages/<pkg>/<version>` 207 - Description: Query the information of a package of specified version. 208 209 ### 3. Catalog Design 210 211 This section will describe the design of the catalog structure, how it interacts with APIServer, and the workflow users install packages. 212 213 #### Catalog structure 214 215 All packages are put under `/catalog/` dir (this is configurable). The directory structure follows a predefined format: 216 217 ```bash 218 /catalog/ # a catalog consists of multiple packages 219 |-- <package> 220 |-- v1.0 # a package consists of multiple versions 221 |-- metadata.yaml 222 |-- definitions/ 223 |-- xxx-workload.yaml 224 |-- xxx-trait.yaml 225 |-- conditions/ 226 |-- check-crd.yaml 227 |-- hooks/ 228 |-- pre-install.yaml 229 |-- modules.yaml 230 |-- v2.0 231 |-- <package> 232 ``` 233 234 The structure of one package version contains: 235 236 - `metadata.yaml`: the metadata of the package. 237 238 ```yaml 239 name: "foo" 240 version: 1.0 241 description: | 242 More details about this package. 243 maintainer: 244 - "@xxx" 245 license: "Apache License Version 2.0" 246 url: "https://..." 247 label: 248 category: foo 249 ``` 250 251 - `definitions`: definition files that describe the capabilities from this package to enable on a cluster. Note that these definitions will compared against a cluster on APIServer side to see if a cluster can install or upgrade this package. 252 253 - `conditions/`: defining conditional checks before deploying this package. For example, check if a CRD with specific version exist, if not then the deployment should fail. 254 255 ```yaml 256 # check-crd.yaml 257 conditions: 258 - target: 259 apiVersion: apiextensions.k8s.io/v1 260 kind: CustomResourceDefinition 261 name: test.example.com 262 fieldPath: spec.versions[0].name 263 op: eq 264 value: "v1" 265 ``` 266 267 - `hooks/`:lifecycle hooks on deploying this package. These are the k8s jobs, and consists of pre-install, post-install, pre-uninstall, post-uninstall. 268 269 ```yaml 270 # pre-install.yaml 271 pre-install: 272 - job: 273 apiVersion: batch/v1 274 kind: Job 275 spec: 276 template: 277 spec: 278 containers: 279 - name: pre-install-job 280 image: "pre-install:v1" 281 command: ["/bin/pre-install"] 282 ``` 283 284 - `modules.yaml`: defining the modules that contain the actual resources, e.g. Helm Charts or Terraform modules. Note that we choose to integrate with existing community solutions instead of inventing our own format. In this way we can adopt the reservoir of community efforts and make the design extensible to more in-house formats as we have observed. 285 286 ```yaml 287 modules: 288 - chart: 289 path: ./charts/ingress-nginx/ # local path in this package 290 remote: 291 repo: https://kubernetes.github.io/ingress-nginx 292 name: ingress-nginx 293 - terraform: 294 path: ./tf_modules/rds/ # local path in this package 295 remote: 296 source: terraform-aws-modules/rds/aws 297 version: "~> 2.0" 298 ``` 299 300 #### Register a catalog in APIServer 301 302 Please refer to `/catalogs/<catalog>` API endpoint above. 303 304 Under the hood, APIServer will scan the catalog repo based on the predefined structure to parse each packages and versions. 305 306 #### Sync a catalog in APIServer 307 308 Please refer to `/catalogs/<catalog>/sync` API endpoint above. 309 310 Under the hood, APIServer will rescan the catalog. 311 312 #### Download a package from a catalog 313 314 Vela APIServer aggregates package information from multiple catalog servers. To download a package, the user first requests the APIServer to find the location of the catalog and the package. Then the user visits the catalog repo directly to download the package data. The workflow is shown as below: 315 316  317 318 In our future roadmap, we will build a catalog controller for each k8s cluster. Then we will add API endpoint to install the package in APIServer which basically creates a CR to trigger the controller to reconcile package installation into the cluster. We choose this instead of APIServer installing the package because in this way we can bypass the APIServer in the package data transfer path and avoid APIServer becoming a single point of failure. 319 320 ## Examples 321 322 ## Considerations 323 324 ### Package parameters 325 326 We can parse the schema of parameters from Helm Chart or Terraform. For example, Helm supports [value schema file](https://www.arthurkoziel.com/validate-helm-chart-values-with-json-schemas/) for input validation and there is an [automation tool](https://github.com/karuppiah7890/helm-schema-gen) to generate the schema. 327 328 ### Package dependency 329 330 Instead of having multiple definitions in one package, we could define that one package correlates to one definition only. But some users will need a bundle of definitions instead of one. For example, a full-observability trait might include a prometheus package, a grafana package, a loki package, and a jaeger package. This is what we call "package bundling". 331 332 To provide a bundle of definitions, we could define package dependency. So a parent package could depend on multiple atomic packages to provide a full-fledged capability. 333 334 Package dependency solution will simplify the structure and provide more atomic packages. But this is not a simple problem and beyond the current scope. We will add this on the future roadmap. 335 336 ### Multi-tenancy 337 338 For initial version we plan to implement APIServer without multi-tenancy. But as an application platform we expect multi-tenancy is a necessary part of Vela. We will keep API compatibility and might add some sort of auth token (e.g. JWT) as a query parameter in the future.