github.com/argoproj/argo-cd/v3@v3.2.1/docs/proposals/server-side-pagination.md (about)

     1  ---
     2  title: Server Side Pagination for Applications List and Watch APIs
     3  authors:
     4    - "@alexmt"
     5  sponsors:
     6    - "@jessesuen"
     7  reviewers:
     8    - TBD
     9  approvers:
    10    - TBD
    11  
    12  creation-date: 2024-02-14
    13  last-updated: 2024-02-14
    14  ---
    15  
    16  # Introduce Server Side Pagination for Applications List and Watch APIs
    17  
    18  Improve Argo CD performance by introducing server side pagination for Applications List and Watch APIs.
    19  
    20  ## Open Questions [optional]
    21  
    22  This is where to call out areas of the design that require closure before deciding to implement the
    23  design.
    24  
    25  
    26  ## Summary
    27  
    28  The Argo CD API server currently returns all applications in a single response. This can be a performance
    29  bottleneck when there are a large number of applications. This proposal is to introduce server side pagination
    30  for the Applications List and Watch APIs.
    31  
    32  ## Motivation
    33  
    34  The main motivation for this proposal it to improve the Argo CD UI responsiveness when there are a large number
    35  of applications. The API server memory usage increases with the number of applications however this is not critical
    36  and can be mitigated by increasing memory limits for the API server deployment. The UI however becomes unresponsive
    37  even on a powerful machine when the number of applications increases 2000. The server side pagination will allow
    38  to reduce amount of data returned by the API server and improve the UI responsiveness.
    39  
    40  ### Goals
    41  
    42  * Support server side pagination for Applications List and Watch APIs
    43  
    44  * Leverage pagination in the Argo CD UI to improve responsiveness
    45  
    46  * Leverage pagination in the Argo CD CLI to improve performance and reduce load on the API server
    47  
    48  ### Non-Goals
    49  
    50  * The API Server is known to use a lot of CPU while applying very large RBAC policies to a large number of applications.
    51    Even with pagination API still need to apply RBAC policies to return "last page" response. So the issueis not addressed by this proposal.
    52  
    53  ## Proposal
    54  
    55  **Pagination Cursor**
    56  
    57  It is proposed to add `offset` and `limit` fields for pagination support in Application List API.
    58  The The Watch API is a bit more complex. Both Argo CD user interface and CLI are relying on the Watch API to display real time updates of Argo CD applications.
    59  The Watch API currently supports filtering by a project and an application name. In order to effectively
    60  implement server side pagination for the Watch API we cannot rely on the order of the applications returned by the API server. Instead of
    61  relying on the order it is proposed to rely on the application name and use it as a cursor for pagination. Both the Applications List and Watch
    62  APIs will be extended with the optional `minName` and `maxName` fields. The `minName` field will be used to specify the application name to start from
    63  and the `maxName` field will be used to specify the application name to end at. The fields should be added to the `ApplicationQuery` message
    64  which is used as a request payload for the Applications List and Watch APIs.
    65  
    66  ```proto
    67  message ApplicationQuery { 
    68    // ... existing fields
    69    // New proto fields for server side pagination
    70  	// the application name to start from (app with min name is included in response)
    71  	optional string minName = 9;
    72  	// the application name to end at (app with max name is included in response)
    73  	optional string maxName = 10;
    74    	// offset
    75  	optional int64 offset = 18;
    76  	// limit
    77  	optional int64 limit = 19;
    78  }
    79  ```
    80  
    81  **Server Side Filtering**
    82  
    83  In order to support server side pagination the filtering has to be moved to the server side as well. `ApplicationQuery` message needs to be extended with the following fields:
    84  
    85  ```proto
    86  message ApplicationQuery { 
    87    // ... existing fields
    88    // New proto fields for server side filtering
    89  	// the repos filter
    90  	repeated string repos = 11;
    91  	// the clusters filter
    92  	repeated string clusters = 12;
    93  	// the namespaces filter
    94  	repeated string namespaces = 13;
    95  	// the auth sync filter
    96  	optional bool autoSyncEnabled = 14;
    97  	// the sync status filter
    98  	repeated string syncStatuses = 15;
    99  	// the health status filter
   100  	repeated string healthStatuses = 16;
   101  	// search
   102  	optional string search = 17;
   103  }
   104  ```
   105  
   106  The Argo CD UI should be updated to populate fields in the List and Watch API requests instead of performing filtering on the client side.
   107  
   108  **Applications Stats**
   109  
   110  The Argo CD UI displays the breakdown of the applications by the sync status, health status etc. Stats numbers are calculated on the client side
   111  and rely on the full list of applications returned by the API server. The server side pagination will break the stats calculation. The proposal is to
   112  intoduce a new `stats` field to the Applications List API response. The field will contain the breakdown of the applications by various statuses.
   113  
   114  ```golang
   115  type ApplicationLabelStats struct {
   116  	Key    string   `json:"key" protobuf:"bytes,1,opt,name=key"`
   117  	Values []string `json:"values" protobuf:"bytes,2,opt,name=values"`
   118  }
   119  
   120  // ApplicationListStats holds additional information about the list of applications
   121  type ApplicationListStats struct {
   122  	Total                int64                             `json:"total" protobuf:"bytes,1,opt,name=total"`
   123  	TotalBySyncStatus    map[SyncStatusCode]int64          `json:"totalBySyncStatus,omitempty" protobuf:"bytes,2,opt,name=totalBySyncStatus"`
   124  	TotalByHealthStatus  map[health.HealthStatusCode]int64 `json:"totalByHealthStatus,omitempty" protobuf:"bytes,3,opt,name=totalByHealthStatus"`
   125  	AutoSyncEnabledCount int64                             `json:"autoSyncEnabledCount" protobuf:"bytes,4,opt,name=autoSyncEnabledCount"`
   126  	Destinations         []ApplicationDestination          `json:"destinations" protobuf:"bytes,5,opt,name=destinations"`
   127  	Namespaces           []string                          `json:"namespaces" protobuf:"bytes,6,opt,name=namespaces"`
   128  	Labels               []ApplicationLabelStats           `json:"labels,omitempty" protobuf:"bytes,7,opt,name=labels"`
   129  }
   130  ```
   131  
   132  The `stats` filter should be populated with information about all applications returned by the API server even when single page is loaded.
   133  The Argo CD UI should be updated to use the stats returned by the API server instead of calculating the stats on the client side.
   134  
   135  **Argo CD CLI**
   136  
   137  The Argo CD CLI should be updated to support server side pagination. The `argocd app list` command should be updated to support `--offset` and `--limit` flags.
   138  If the `--offset` and `--limit` flags are not specified the CLI should use pagination to load all applications in batches of 500 applications.
   139  
   140  ### Use cases
   141  
   142  Add a list of detailed use cases this enhancement intends to take care of.
   143  
   144  #### User Server Side Pagination in Argo CD User Interface to improve responsiveness:
   145  As a user, I would like to be able to navigate through the list of applications using the pagination controls.
   146  
   147  #### User Server Side Pagination in Argo CD CLI to reduce load on the API server:
   148  As a user, I would like to use Argo CD CLI to list applications while leveraging the pagination without overloading the API server.
   149  
   150  ### Implementation Details/Notes/Constraints [optional]
   151  
   152  **Application Stats**
   153  
   154  **CLI Backward Compatibility**
   155  
   156  Typically we bump minimal supported API version when we introduce a new feature in CLI. In this case I
   157  suggest to support gracefully missing pagination support in CLI. If the API server returns more applications than
   158  specified in `limit` the CLI should assume pagination is not supported and response has full list of applications.
   159  This way the user can downgrade API server without downgrading CLI.
   160  
   161  ### Security Considerations
   162  
   163  The proposal does not introduce any new security risks.
   164  
   165  ### Risks and Mitigations
   166  
   167  We might need to bump minimal supported API version in CLI to support pagination. The `Implementation Details` section
   168  contains the proposal to avoid doing it.
   169  
   170  ### Upgrade / Downgrade Strategy
   171  
   172  The proposal does not introduce any breaking changes. The API server should gracefully handle requests without pagination fields.
   173  
   174  ## Drawbacks
   175  
   176  None.
   177  
   178  ## Alternatives
   179  
   180  ****