github.com/treeverse/lakefs@v1.24.1-0.20240520134607-95648127bfb0/design/open/DOLLAR_PULLS.md (about)

     1  # $PULLS Proposal
     2  
     3  This proposal tries to bridge between two proposals: [pull-request](pull-request.md) and [checks](https://github.com/treeverse/lakeFS/blob/280c5b741df5e1e051958b8adde826364c6df614/design/open/branch_checks.md)
     4  and provide long-running CI/CD processes while laying the foundation for future "pull-request" like feature.
     5  
     6  ## Goals
     7  
     8  * Allow long CI/CD processes
     9  * Enable branch protection feature for merge 
    10  
    11  ## Non-Goals
    12  
    13  * Collaboration
    14    * Reviews / change requests
    15    * Discussions
    16    * Approvals
    17  * Required actions
    18  
    19  ## Suggestion 1: Pull Requests
    20  
    21  ### How it will work
    22  
    23  Introduce a new entity called $PULLS at the repository level. The $PULLS entity is identified by a source branch and destination branch. 
    24  Each source-destination pair can have only a single `OPEN` $PULLS instance.
    25  Define a new hook type - `on-pull` which will run whenever a new $PULLS instance is created.
    26  The actions will run on the source branch, and will allow merging after job is successful.
    27  
    28  #### $PULLS Object
    29  
    30  $PULLS will be saved in the ref-store under the key: 
    31  `$PULLS/<src-branch>/<dest-branch>/<$PULLS-id>`
    32  
    33  The $PULLS object will contain the following data:
    34  
    35  * `Source Branch`
    36  * `Source Ref` - The source branch head at the time of creation
    37  * `Destination Branch`
    38  * `Run ID` - The actions associated run id, may be empty
    39  
    40  #### $PULLS States
    41  
    42  $PULLS will have the following states:
    43  
    44  * `OPEN`
    45  * `CLOSED`
    46  * `MERGED`
    47  
    48  #### Creating a new $PULLS instance
    49  
    50  POST `/api/v1/repositories/{repository}/branches/{sourceBranch}/$PULLS/{destinationBranch}`
    51  
    52  On creation the following occurs:
    53  
    54  1. Check for conflicts between source and destination branch - if conflict is found, do not run actions and display an appropriate message
    55  2. Run actions - and associate RunID with instance
    56  
    57  This will return the $PULLS instance ID, that can be used later for querying and merging etc.
    58  
    59  #### Querying $PULLS
    60  
    61  GET `/api/v1/repositories/{repository}/branches/{sourceBranch}/$PULLS/{destinationBranch}/{id}`
    62  
    63  Will return the status, corresponding actions RunID
    64  
    65  #### Delete $PULLS
    66  
    67  DELETE `/api/v1/repositories/{repository}/branches/{sourceBranch}/$PULLS/{destinationBranch}/{id}`
    68  
    69  #### $PULLS enforcement
    70  
    71  Enforcement will be done in the repository level - using the branch protection rules.
    72  Define a new rule `BranchProtection_RequirePull` which when set, will prohibit merge into branch not via a $PULLS instance.
    73  
    74  #### Merge $PULLS
    75  
    76  Add an optional parameter to the Merge API for the $PULLS id. 
    77  When performing a merge with a $PULLS id, perform the following preliminary checks:
    78  1. Verify source reference id with source branch head
    79  2. Verify actions run successful
    80  3. Attempt to perform merge (Subject to failure on conflict / hooks)
    81  4. Upon successful merge - change $PULLS state to `MERGED`
    82  5. On the event of failure in any of the previous steps - provide a meaningful error
    83  
    84  #### Re-run Actions
    85  
    86  PUT `/api/v1/repositories/{repository}/branches/{sourceBranch}/$PULLS/{destinationBranch}/{id}`
    87  
    88  In the event of failed actions run, it will be possible to re-run the job via the $PULLS context.
    89  This will in fact - create a new trigger for the `on-pull` hook, and update the $PULLS instance with a new run ID.
    90  By "re-running" the actions via the $PULLS, we are able to decouple the $PULLS feature from the hooks re-run requirement,
    91  which can be implemented in the future.
    92  
    93  
    94  ### Required Changes
    95  
    96  #### Branch Protection
    97  
    98  * Add new branch protection rule for $PULLS
    99  * Modify relevant APIs and add check for $PULLS branch protection rule
   100  * UI: On Add, provide options to select rules. Allow editing of rules.
   101  
   102  #### Merge
   103  
   104  * Add checks for $PULLS state
   105  
   106  #### Actions
   107  
   108  * Add a new hook type: `on-pull`
   109  * Implement asynchronous (pollable) hooks
   110  * Implement hooks re-run
   111  
   112  ### UI
   113  
   114  Introduce a new tab called $PULLS, that lists all the repository's $PULLS and their states.
   115  
   116  ![$PULLS view](diagrams/pulls-list.png)
   117  
   118  From this view, it will be possible to create new $PULLS, re-run failed $PULLS and perform merge on successful $PULLS.
   119  
   120  ![$PULLS create](diagrams/pulls-create.png)
   121  
   122  
   123  ## Suggestion 2 (**ACCEPTED**): Keep it minimal - Manual Triggers 
   124  
   125  ### How it will work
   126  
   127  Introduce a new trigger type to lakeFS actions: `manual-trigger`.
   128  Once defined in a lakeFS action, it will be possible to trigger the action 
   129  manually with a mandatory reference (branch/commit/tag).The lakeFS action 
   130  will run on the provided reference and the results will be available in the 
   131  just like any other action run.
   132  
   133  A branch protection rule will be added to the repository level, that will 
   134  enforce actions to be run on the source ref before merge.
   135  
   136  ### Required Changes
   137  
   138  #### Branch Protection
   139  
   140  * Add new branch protection rule for $ACTION_RUN
   141  * Modify relevant APIs and add check for $ACTION_RUN branch protection rule
   142  * UI: On Add, provide options to select rules. Allow editing of rules.
   143  
   144  #### Merge
   145  
   146  * Add checks for Branch Protection rule fulfillment.
   147  
   148  #### Actions
   149  
   150  * Add a new trigger type: `manual-trigger`
   151  * Add an API endpoint for triggering actions:
   152  POST `/repositories/{repository}/refs/{ref}/actions/{action}/trigger`
   153  The request will execute the action on the provided ref if it exists on the 
   154    ref and one of the action triggers is `manual-trigger`. Otherwise, it will 
   155    fail with BadRequest. The request will not wait for the hook execution to 
   156    complete.
   157  
   158  Open questions:
   159  * Should hook re-run be implemented at first?
   160    * We can get away with not implementing it, and implement it in the future.
   161      It means that reruns would occur by pushing a new commit to the source 
   162      branch. Not ideal, but reruns are mostly useful for transient failures.
   163      *Decision*: Start without.
   164  * Should asynchronous (pollable) hooks be implemented at first? 
   165    * Since the HTTP request doesn't wait for the action run to complete, we 
   166      can get away with not implementing it, and implement it in the future.
   167      By decoupling the hook execution from the context of a lakeFS operation 
   168      (pre-merge, pre-commit, etc.), we can now run as long as we want without 
   169      blocking anything. This means that we can run long CI/CD processes, and 
   170      possibly just log that the action is running.
   171      *Decision*: Implement for the first step.