github.com/replit/upm@v0.0.0-20240423230255-9ce4fc3ea24c/internal/backends/python/regression_tests/README.md (about)

     1  # Regression Tests
     2  
     3  These tests were put in place to make sure we didn't break the guessing
     4  behavior after a big rewrite.
     5  
     6  This test suite tests the new `upm` against the old version of upm: `upm-old`.
     7  
     8  ## Setup
     9  
    10  To set up the test, first you'll need to install the old version of upm by checking out an older version. At the top of the project directory, do:
    11  
    12  ```
    13  git checkout abe5cc5
    14  rm internal/backends/python/pypi_map.gen.go
    15  make upm
    16  go install ./cmd/upm
    17  cp $(which upm) $(dirname $(which upm))/upm-old
    18  ```
    19  
    20  Now comeback to the newer branch:
    21  
    22  ```
    23  git checkout th-python-guess-modules-2
    24  make clean-gen
    25  make upm
    26  make install
    27  ```
    28  
    29  Now you should have access to both `upm` and `upm-old`.
    30  
    31  ## Run a test
    32  
    33  cd into the `regression_tests` directory.
    34  
    35  To test a specific module, pass its name to the test.py script:
    36  
    37  ```
    38  python3 test.py flask
    39  ```
    40  
    41  You should see 4 kinds of outputs:
    42  
    43  * <pkg> ok - means the output of the old matches the new (or we added a manual override in test.py)
    44  * <pkg> failed - means the output of the old does not match the new
    45  * <pkg> added - means the old one did not have an entry for this pkg and the new one does
    46  * <pkg> no-guess - means the old doesn't have an entry and the new couldn't find a guess (based on its heuistic algorithm)
    47  * <pkg> test-errored - means the old does not have an entry for this pkg and the new one error during the package test
    48  * <pkg> missing - means neither the old or the new has an entry for this pkg
    49  
    50  To run the regression tests on the top 10k packages, you can leave out the package name:
    51  
    52  ```
    53  python3 test.py
    54  ```
    55  
    56  This can take up to 30 minutes depending on hardware.
    57  
    58  
    59