github.com/RedHatInsights/insights-results-aggregator@v1.4.1/docs/architecture.md (about)

     1  ---
     2  layout: page
     3  nav_order: 1
     4  ---
     5  # Architecture
     6  
     7  Aggregator service consists of three main parts:
     8  
     9  1. Consumer that reads (consumes) Insights OCP messages from specified message
    10     broker. Usually Kafka broker is used but it might be possible to develop a
    11     interface for different broker. Insights OCP messages are basically encoded
    12     in JSON and contain results generated by rule engine. Different consumer can
    13     be selected to consume and process DVO Recommendations.
    14  2. HTTP or HTTPS server that exposes REST API endpoints that can be used to
    15     read list of organizations, list of clusters, read rules results for
    16     selected cluster etc. Additionally, basic metrics are exposed as well. Those
    17     metrics is configured to be consumed by Prometheus and visualized by
    18     Grafana.
    19  3. Storage backend which is some instance of SQL database or Redis storage.
    20     Currently only PostgreSQL is fully supported, but more SQL databases might
    21     be added later.
    22  
    23  ## Whole data flow
    24  
    25  ![data_flow]({{ "assets/customer-facing-services-architecture.png" | relative_url}})
    26  
    27  1. Event about new data from insights operator is consumed from Kafka. That
    28     event contains (among other things) URL to S3 Bucket
    29  2. Insights operator data is read from S3 Bucket and Insights OCP rules are
    30     applied to that data. Alternatively DVO rules are applied to the same data.
    31  3. Results (basically organization ID + cluster name + insights OCP
    32     recommendations JSON or DVO recommendations) are stored back into Kafka, but
    33     into different topic
    34  4. That results are consumed by Insights rules aggregator service that caches
    35     them (i.e. stores them into selected database).
    36  5. The service provides such data via REST API to other tools, like OpenShift
    37     Cluster Manager web UI, OpenShift console, etc.
    38  
    39  Optionally, a so called organization allowlist can be enabled by the configuration variable
    40  `enable_org_allowlist`, which enables processing of a .csv file containing organization IDs (path
    41  specified by the config variable `org_allowlist`) and allows report processing only for these
    42  organizations. This feature is disabled by default, and might be removed altogether in the near
    43  future.
    44  
    45  ---
    46  **NOTE**
    47  
    48  Detailed information about the exact format of consumed data from Kafka topic is
    49  available at
    50  
    51  https://redhatinsights.github.io/insights-data-schemas/ccx_ocp_results_topic.html
    52  
    53  ---