github.com/treeverse/lakefs@v1.24.1-0.20240520134607-95648127bfb0/design/accepted/system_tests.md (about)

     1  # System Tests
     2  
     3  ## Requirements
     4  1. Test the system in the same manner users will be using it.
     5     No ad-hoc initializations or mocking of any kind.
     6  2. Cover the most common use-cases.
     7  3. Portability: The tests don't assume anything on the environment where the lakeFS & DB are running.
     8     lakeFS address is the only **required** argument for running the tests.  
     9  
    10  ## Non-Requirements
    11  1. Test every use-case of the system, including:
    12      a. Every API/gateway endpoint.
    13      b. Every application (e.g. Spark) usage pattern.
    14  2. Avoid repeating unit/integration tests scenarios.
    15  3. Stress testing / benchmarks.
    16  
    17  ## Solution
    18  ### Test
    19  A test consists of one or more calls to the API/gateway and assertions on the actual output.
    20  Test is a standalone unit. It shouldn't rely on any other test to pass to start execution.
    21  
    22  Later on we might consider adding checks on the DB or data lake layer(S3),
    23  by adding information of the underlying system components.
    24  
    25  #### Test-Cases
    26  TBD
    27  
    28  ### Tests binary
    29  Tests will be built into a single binary and run sequentially or in parallel.
    30  
    31  ### When will it run? Where?
    32  As a standalone unit, the tests binary can be executed from anywhere as long as it can reach the lakeFS endpoint.
    33  We encourage developers to run the tests locally as part of the development cycle.
    34  
    35  #### CI
    36  GitHub workflow will be triggered when all other workflows finished successfully and reviewers approved.
    37  The workflow will run the lakeFS app with local Postgres DB on a dedicated server,
    38  execute the tests binary and report the results. Failed run will block the PR merge.
    39  
    40  Underline storage (S3/GCS/Minio/other) will be created once for all test runs.
    41  Tests are orthogonal to the lakeFS setup which could vary in the backing storage layer,
    42  Postgres version and other configurations.
    43  
    44  Objects will be deleted by setting expiry for the entire bucket in the storage layer.
    45  Isolation is kept by setting unique file prefixes which is determined by the lakeFS namespaces.