github.com/treeverse/lakefs@v1.24.1-0.20240520134607-95648127bfb0/docs/howto/deploy/upgrade.md (about)

     1  ---
     2  title: Upgrade lakeFS
     3  description: How to upgrade lakeFS to the latest version.
     4  grand_parent: How-To
     5  parent: Install lakeFS
     6  redirect_from: 
     7    - /deploying-aws/upgrade.html
     8    - /reference/upgrade.html
     9    - /howto/upgrade.html
    10  ---
    11  
    12  # Upgrading lakeFS
    13  
    14  Note: For a fully managed lakeFS service with guaranteed SLAs, try [lakeFS Cloud](https://lakefs.cloud)
    15  {: .note }
    16  
    17  Upgrading lakeFS from a previous version usually just requires re-deploying with the latest image (or downloading the latest version if you're using the binary).
    18  If you're upgrading, check whether the [release](https://github.com/treeverse/lakeFS/releases) requires a migration.
    19  
    20  ## When DB migrations are required
    21  
    22  ### lakeFS 0.103.0 or greater
    23  
    24  Version 0.103.0 added support for rolling KV upgrade. This means that users who already migrated to the KV ref-store (versions 0.80.0 and above) no longer have to pass through specific versions for migration.
    25  This includes [ACL migration](https://docs.lakefs.io/reference/access-control-lists.html#migrating-from-the-previous-version-of-acls) which was introduced in lakeFS version 0.98.0.
    26  Running `lakefs migrate up` on the latest lakeFS version will perform all the necessary migrations up to that point.
    27  
    28  ### lakeFS 0.80.0 or greater (KV Migration)
    29  
    30  Starting with version 0.80.2, lakeFS has transitioned from using a PostgreSQL based database implementation to a Key-Value datastore interface supporting
    31  multiple database implementations. More information can be found [here](https://github.com/treeverse/lakeFS/blob/master/design/accepted/metadata_kv/index.md).  
    32  Users upgrading from a previous version of lakeFS must pass through the KV migration version (0.80.2) before upgrading to newer versions of lakeFS.
    33  
    34  > **IMPORTANT: Pre Migrate Requirements**  
    35  > * **Users using OS environment variables for database configuration must define the `connection_string` explicitly or as environment variable before proceeding with the migration.**  
    36  > * **Database storage free capacity of at least twice the amount of the currently used capacity**
    37  > * It is strongly recommended to perform these additional steps:
    38  >   * Commit all uncommitted data on branches
    39  >   * Create a snapshot of your database
    40  > * By default, old database tables are not being deleted by the migration process, and should be removed manually after a successful migration.
    41  > To enable table drop as part of the migration, set the `database.drop_tables` configuration param to `true`
    42  {: .note }
    43  
    44  #### Migration Steps
    45  For each lakeFS instance currently running with the database
    46  1. Modify the `database` section under lakeFS configuration yaml:
    47     1. Add `type` field with `"postgres"` as value
    48     2. Copy the current configuration parameters to a new section called `postgres`
    49  
    50     ```yaml
    51     ---
    52     database:
    53      type: "postgres"
    54      connection_string: "postgres://localhost:5432/postgres?sslmode=disable"
    55      max_open_connections: 20
    56     
    57      postgres:
    58        connection_string: "postgres://localhost:5432/postgres?sslmode=disable"
    59        max_open_connections: 20
    60     ```
    61  
    62  2. Stop all lakeFS instances
    63  3. Using the `lakefs` binary for the new version (0.80.2), run the following:
    64  
    65     ```bash
    66     lakefs migrate up
    67     ```
    68  
    69  4. lakeFS will run the migration process, which in the end should display the following message with no errors:
    70  
    71     ```shell
    72     time="2022-08-10T14:46:25Z" level=info msg="KV Migration took 717.629563ms" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" TempDir=/tmp/kv_migrate_2913402680
    73     ```
    74  
    75  5. It is now possible to remove the old database configuration. The updated configuration should look as such:
    76  
    77     ```yaml
    78     ---
    79     database:
    80      type: "postgres"
    81     
    82      postgres:
    83        connection_string: "postgres://localhost:5432/postgres?sslmode=disable"
    84        max_open_connections: 20
    85     ```
    86   
    87  6. Deploy (or run) the new version of lakeFS.
    88  
    89  ### lakeFS 0.30.0 or greater
    90  
    91  In case migration is required, you first need to stop the running lakeFS service.
    92  Using the `lakefs` binary for the new version, run the following:
    93  
    94  ```bash
    95  lakefs migrate up
    96  ```
    97  
    98  Deploy (or run) the new version of lakeFS.
    99  
   100  Note that an older version of lakeFS cannot run on a migrated database.
   101  
   102  
   103  ### Prior to lakeFS 0.30.0
   104  
   105  **Note:** with lakeFS < 0.30.0, you should first upgrade to 0.30.0 following this guide. Then, proceed to upgrade to the newest version.
   106  {: .note .pb-3 }
   107  
   108  Starting version 0.30.0, lakeFS handles your committed metadata in a [new way](https://docs.google.com/document/d/1jzD7-jun-tdU5BGapmnMBe9ovSzBvTNjXCcVztV07A4/edit?usp=sharing){: target="_blank" }, which is more robust and has better performance.
   109  To move your existing data, you will need to run the following upgrade commands.
   110  
   111  Verify lakeFS version == 0.30.0 (can skip if using Docker)
   112  
   113  ```shell
   114  lakefs --version
   115  ```
   116  
   117  Migrate data from the previous format:
   118  
   119  ```shell
   120  lakefs migrate db
   121  ```
   122  
   123  Or migrate using Docker image:
   124  
   125  ```shell
   126  docker run --rm -it -e LAKEFS_DATABASE_CONNECTION_STRING=<database connection string> treeverse/lakefs:rocks-migrate migrate db
   127  ```
   128  
   129  Once migrated, it is possible to now use more recent lakeFS versions. Please refer to their release notes for more information on ugrading and usage).
   130  
   131  
   132  If you want to start over, discarding your existing data, you need to explicitly state this in your lakeFS configuration file.
   133  To do so, add the following to your configuration (relevant **only** for 0.30.0):
   134  
   135  ```yaml
   136  cataloger:
   137    type: rocks
   138  ```
   139  
   140  ## Data Migration for Version v0.50.0
   141  
   142  If you are using a version before 0.50.0, you must first perform the [previous upgrade to that version](https://docs.lakefs.io/v0.50/reference/upgrade.html#data-migration-for-version-v0500). {: note: .note-warning }