vitess.io/vitess@v0.16.2/go/vt/vttablet/tabletmanager/vdiff/README.md (about)

     1  
     2  Work In Progress document that describes some key components of vdiff2.
     3  
     4  Most of the vdiff2 code is a reorganization of vdiff, with a few changes which are due to some key functionality differences in vdiff2 from vdiff. See the RFC at https://github.com/vitessio/vitess/issues/10134 for more info.
     5  
     6  ### Key Differences from VDiff2
     7  
     8  * VDiff2 runs on tablets and not on vtctld. There is a single vtctld process that runs synchronously getting data from all target and source shards and compares them. In VDiff2 each shard compares only the data belonging to that target and gets data from related sources.
     9  
    10  * VDiff2 is designed to be resumable. VDiff1 had to be reinvoked in case of any failure. This requires us to switch from using the ResultStreamer to the RowStreamer. The RowStreamer can start from a provided LastPK and also supplies the LastPK in each batch for resumability.
    11  
    12  * VDiff2 runs asynchronously. The vtctld VDiff command will create the vdiff streams, one on each shard. The streams will be run on the current primary and record progress. The vtctld command will also support actions which can manage vdiff's state and report progress.
    13  
    14  ### What stays the same
    15  
    16  The rest of the code mostly a cut/paste with minor refactoring. Instead of the entire code residing in a single vdiff.go file we split it in terms of modules in the tabletmanager/vdiff directory
    17  
    18  * The data comparison algo is identical
    19  
    20  * The mechanism of getting the workflow streams in sync and generating the snapshots is the same, except that only one target is involved per-shard
    21  
    22  
    23  ### Performance Impact
    24  
    25  #### Pros
    26  
    27  * Since it is running on multiple tablets in parallel, VDiff2 should be much faster on a sharded keyspace.
    28  * Being resumable, VDiff2 will be more resilient.
    29    * If VDiff2 is run in resumable mode, then we can take smaller snapshots than the single long-running one that VDiff needs.
    30  * Special provisioning of vtctld servers are not needed.
    31  * Cross-cluster network traffic will be reduced since data being compared by targets will be local
    32  * VDiff2 will eliminate the remaining use case for the `vtctl` (vtctlclient+vtctld) binary -- vdiff's on large tables that could take more than a day to complete.
    33  
    34  
    35  ##### Cons
    36  
    37  * Each source would only have one vstreamer running for each vdiff. Now all targets to which the source contributes will have one vstreamer each. This depends on the sharding configurations of source and target.
    38  * The per-shard comparison happens on the target primary. So if the target is production-serving then there will an extra CPU utilization potentially requiring a larger provisioning on the target.
    39  
    40  ### Setup
    41  
    42  The user invokes `vtctlclient VDiff` on an existing workflow by specifying the target keyspace and workflow and overriding any of the default options that are described in the documentation of the vdiff2 command on the vitessio website.
    43  
    44  ### Database Schema
    45  
    46  A new set of tables have been added to the `_vt` database to support vdiff2.
    47  
    48  #### vdiff
    49  Contains one row per workflow for which vdiff has been requested. Currently only one vdiff can run at a time for a given workflow, so if it has to be restarted its state will be reset. It maintains the state of the vdiff and options specified.
    50  
    51  #### vdiff_table
    52  One row for each table involved in a vdiff. Each table maintains the most recent state: status of each table, last PK, number of rows compared, etc. Any mismatch is flagged immediately and the report of the diff is updated when the diff ends or stops for any reason.
    53  
    54  #### vdiff_log
    55  Free format log of important events in a vdiff such as state changes and errors.
    56  
    57  ### Modules
    58  
    59  This section contains a short note on the key modules in this directory.
    60  
    61  #### Engine (engine.go)
    62  This is a singleton in tabletserver similar to other engines. Each active vdiff has its own controller which is managed by the engine. On a primary, when the engine is opened, vdiff is started/resumed on all active controllers.
    63  
    64  It also has the related table schema.
    65  
    66  #### Controller (controller.go)
    67  
    68  There is one controller for each active vdiff. It maintains the configuration required to run a VDiff including related sources, tablet picker options, an instance of the TabletManagerClient to make rpc calls to other targets, It sets up a WorkflowDiffer which performs the actual diff.
    69  
    70  #### Workflow Differ (workflow_differ.go)
    71  
    72  Sets up all the tables that needed to be diffed using the TableDiffer and serially invokes each diff.
    73  
    74  #### Table Differ (table_differ.go)
    75  
    76  This is the main module that runs a diff on each table, keeps intermediate state and periodically updates this in the `_vt` tables.
    77  
    78  #### Shard Streamer (shard_streamer.go)
    79  
    80  #### Primitive Executor (primitive_executor.go)
    81  
    82  #### Merge Sorter (utils.go)
    83  
    84  ### Miscellaneous Notes
    85  
    86  Since we stream using the standard vstreamer API we automatically subscribe to the lag throttler
    87  mechanism, which will delay the streaming in case the replication lag on the specific shard is above
    88  threshold.
    89  
    90  VDiff2 can recover from the following issues:
    91   * Source or target tablets, which are streaming the data, failing/restarting
    92   * A PRS on the target tablet on which VDiff is running
    93   * User stopping/restarting the operation
    94   * Network failures
    95  
    96