github.com/pingcap/tiflow@v0.0.0-20240520035814-5bf52d54e205/cdc/redo/doc.go (about)

     1  //  Copyright 2021 PingCAP, Inc.
     2  //
     3  //  Licensed under the Apache License, Version 2.0 (the "License");
     4  //  you may not use this file except in compliance with the License.
     5  //  You may obtain a copy of the License at
     6  //
     7  //      http://www.apache.org/licenses/LICENSE-2.0
     8  //
     9  //  Unless required by applicable law or agreed to in writing, software
    10  //  distributed under the License is distributed on an "AS IS" BASIS,
    11  //  See the License for the specific language governing permissions and
    12  //  limitations under the License.
    13  
    14  /*
    15  Package redo provide a redo log for cdc.
    16  
    17  There are three types of log file: meta log file, row log file, ddl log file.
    18  meta file used to store common.LogMeta info (CheckPointTs, ResolvedTs), atomic updated is guaranteed. A rotated file writer is used for other log files.
    19  All files will flush to disk or upload to s3 if enabled every defaultFlushIntervalInMs 1000ms or file size larger than defaultMaxLogSize 64 MB by default.
    20  The log file name is formatted as CaptureID_ChangeFeedID_CreateTime_FileType_MaxCommitTSOfAllEventInTheFile.log if safely wrote or end up with .log.tmp is not.
    21  meta file name is like CaptureID_ChangeFeedID_meta.meta
    22  
    23  Each log file contains batch of model.RowChangedEvent or model.DDLEvent records wrote into different file with defaultMaxLogSize 64 MB.
    24  If larger than 64 MB will auto rotated to a new file.
    25  A record has a length field and a logical Log data. The length field is a 64-bit packed structure holding the length of the remaining logical Log data in its lower
    26  56 bits and its physical padding in the first three bits of the most significant byte. Each record is 8-byte aligned so that the length field is never torn.
    27  
    28  When apply redo log from cli, will select files in the specific dir to open base on the
    29  startTs, endTs send from cli or download logs from s3 first is enabled, then sort the event
    30  records in each file base on commitTs and startTs, after sorted, the new sort file name
    31  should be as CaptureID_ChangeFeedID_CreateTime_FileType_MaxCommitTSOfAllEventInTheFile.log.sort.
    32  */
    33  package redo