github.com/RedHatInsights/insights-results-aggregator@v1.4.1/docs/architecture.md (about) 1 --- 2 layout: page 3 nav_order: 1 4 --- 5 # Architecture 6 7 Aggregator service consists of three main parts: 8 9 1. Consumer that reads (consumes) Insights OCP messages from specified message 10 broker. Usually Kafka broker is used but it might be possible to develop a 11 interface for different broker. Insights OCP messages are basically encoded 12 in JSON and contain results generated by rule engine. Different consumer can 13 be selected to consume and process DVO Recommendations. 14 2. HTTP or HTTPS server that exposes REST API endpoints that can be used to 15 read list of organizations, list of clusters, read rules results for 16 selected cluster etc. Additionally, basic metrics are exposed as well. Those 17 metrics is configured to be consumed by Prometheus and visualized by 18 Grafana. 19 3. Storage backend which is some instance of SQL database or Redis storage. 20 Currently only PostgreSQL is fully supported, but more SQL databases might 21 be added later. 22 23 ## Whole data flow 24 25 ![data_flow]({{ "assets/customer-facing-services-architecture.png" | relative_url}}) 26 27 1. Event about new data from insights operator is consumed from Kafka. That 28 event contains (among other things) URL to S3 Bucket 29 2. Insights operator data is read from S3 Bucket and Insights OCP rules are 30 applied to that data. Alternatively DVO rules are applied to the same data. 31 3. Results (basically organization ID + cluster name + insights OCP 32 recommendations JSON or DVO recommendations) are stored back into Kafka, but 33 into different topic 34 4. That results are consumed by Insights rules aggregator service that caches 35 them (i.e. stores them into selected database). 36 5. The service provides such data via REST API to other tools, like OpenShift 37 Cluster Manager web UI, OpenShift console, etc. 38 39 Optionally, a so called organization allowlist can be enabled by the configuration variable 40 `enable_org_allowlist`, which enables processing of a .csv file containing organization IDs (path 41 specified by the config variable `org_allowlist`) and allows report processing only for these 42 organizations. This feature is disabled by default, and might be removed altogether in the near 43 future. 44 45 --- 46 **NOTE** 47 48 Detailed information about the exact format of consumed data from Kafka topic is 49 available at 50 51 https://redhatinsights.github.io/insights-data-schemas/ccx_ocp_results_topic.html 52 53 ---