github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/cmd/migrate/README.md (about)

     1  # Loki Migrate Tool
     2  
     3  **WARNING: THIS TOOL IS NOT WELL TESTED, ALWAYS MAKE BACKUPS AND TEST ON LESS IMPORTANT DATA FIRST!**
     4  
     5  This is sort of a bare minimum code hooked directly into the store interfaces within Loki.
     6  
     7  Two stores are created, a source store and destination (abbreviated dest) store.
     8  
     9  Chunks are queried from the source store and written to the dest store, new index entries are created in the dest store as well.
    10  
    11  You should be able to:
    12  
    13  * Migrate between clusters
    14  * Change tenant ID during migration
    15  * Migrate data between schemas
    16  
    17  All data is read and re-written (even when migrating within the same cluster). There are really no optimizations in this code for performance and there are much faster ways to move data depending on what you want to change.
    18  
    19  This is simple and because it uses the storage interfaces, should be complete and should stay working, but it's not optimized to be fast.
    20  
    21  There is however some parallelism built in and there are a few flags to tune this, `migrate -help` for more info
    22  
    23  This does not remove or modify any source data, it only reads existing source data and writes to the destination.
    24  
    25  ## Usage
    26  
    27  Build with
    28  
    29  ```
    30  make migrate
    31  ```
    32  
    33  or
    34  
    35  ```
    36  make migrate-image
    37  ```
    38  
    39  The docker image currently runs and doesn't do anything, it's intended you exec into the container and run commands manually.
    40  
    41  
    42  ### Examples
    43  
    44  Migrate between clusters
    45  
    46  ```
    47  migrate -source.config.file=/etc/loki-us-west1/config/config.yaml -dest.config.file=/etc/loki-us-central1/config/config.yaml -source.tenant=2289 -dest.tenant=2289 -from=2020-06-16T14:00:00-00:00 -to=2020-07-01T00:00:00-00:00
    48  ```
    49  
    50  Migrate tenant ID within a cluster
    51  
    52  ```
    53  migrate -source.config.file=/etc/loki-us-west1/config/config.yaml -dest.config.file=/etc/loki-us-west1/config/config.yaml -source.tenant=fake -dest.tenant=1 -from=2020-06-16T14:00:00-00:00 -to=2020-07-01T00:00:00-00:00
    54  ```
    55  
    56  ### Stopping and restarting
    57  
    58  It's ok to process the same data multiple times, chunks are uniquely addressable, they will just replace each other.
    59  
    60  For boltdb-shipper you will end up with multiple index files which contain duplicate entries,
    61  Loki will handle this without issue and the compactor will reduce the number of files if there are more than 3
    62  (TODO we should make a compactor mode which forces cleanup to a single file)
    63  
    64  You can use the output of the processed sync ranges to help in restarting from a point of already processed data,
    65  however be aware that because of parallel processing, you need to find the last finished time for *ALL* the threads
    66  to determine where processing finished, because of the parallel dispatching of sync ranges the order of messages
    67  will not always be sorted chronologically.
    68  
    69  Also be aware of special considerations for a boltdb-shipper destination outlined below.
    70  
    71  ### batchLen, shardBy, and parallel flags
    72  
    73  The defaults here are probably ok for normal sized computers.
    74  
    75  If sending data from something like a Raspberry Pi, you probably want to run something like `-batchLen=10 -parallel=4` or risk running out of memory.
    76  
    77  The transfer works by breaking up the time range into `shardBy` windows, a window is called a sync range, 
    78  each sync range is then dispatched to one of up to `parallel` worker threads.
    79  
    80  For each sync range, the source index is queried for all the chunks in the sync range, then the list of chunks is processed `batchLen` at a time from the source, 
    81  re-encoded if necessary (such as changing tenant ID), and send to the destination store. You need enough memory to handle having `batchlen` chunks in memory
    82  times the number of `parallel` threads.
    83  
    84  If you have a really huge amount of chunks, many tens or hundreds of thousands per day, you might want to decrease `shardBy` to a smaller window. 
    85  If you don't have many chunks and `shardBy` is too small you will process the same chunks from multiple sync ranges.
    86  
    87  `parallel` can likely be increased up to a point until you saturate your CPU or exhaust memory.
    88  
    89  If you have a lot of really tiny chunks it may make sense to increase `batchLen`, but I'm not sure how much changing this affects the performance. 
    90  
    91  There is not an exact science to tuning these params yet, 
    92  the output window gives information on throughput and you can play around with values to maximize throughput for your data.
    93  
    94  
    95  ### boltdb-shipper
    96  
    97  When the destination index type is boltdb shipper, it's important to make sure index files are uploaded. 
    98  This happens automatically in the background with some timers built into the boltdb-shipper code.
    99  It also happens explicitly when all the sync ranges have been processed and the store is shutdown.
   100  
   101  However, if the process crashes while processing, there may be index files which were not uploaded.
   102  
   103  If restarting after a crash, it's best to overlap the start time with previously processed sync ranges. 
   104  Exactly how much to overlap is hard to say, you could look for the most recently uploaded index file in 
   105  the destination store which is the number of days since the unix epoch, and convert it to seconds to see what day it is.
   106  
   107  e.g. index_18262: 18262 * (60 * 60 * 24) = 1577836800 which is `Wednesday, January 1, 2020 12:00:00 AM`