github.com/pingcap/ticdc@v0.0.0-20220526033649-485a10ef2652/cdc/processor/doc.go (about)

     1  // Copyright 2021 PingCAP, Inc.
     2  //
     3  // Licensed under the Apache License, Version 2.0 (the "License");
     4  // you may not use this file except in compliance with the License.
     5  // You may obtain a copy of the License at
     6  //
     7  //     http://www.apache.org/licenses/LICENSE-2.0
     8  //
     9  // Unless required by applicable law or agreed to in writing, software
    10  // distributed under the License is distributed on an "AS IS" BASIS,
    11  // See the License for the specific language governing permissions and
    12  // limitations under the License.
    13  
    14  /*
    15  Package processor implements the processor logic based on ETCD worker(pkg/orchestrator).
    16  
    17  There are three main modules: Manager, Processor and TablePipeline(cdc/processor/pipeline).
    18  The Manager's main responsibility is to maintain the Processor's life cycle, like create and destroy the processor instances.
    19  The Processor's main responsibility is to maintain the TablePipeline's life cycle according to the state stored by ETCD,
    20  and calculate the local resolved TS and local checkpoint Ts and put them into ETCD.
    21  The TablePipeline listens to the kv change logs of a specified table(with its mark table if it exists), and sends logs to Sink After sorting and mounting.
    22  
    23  The relationship between the three module is as follows:
    24  
    25  One Capture(with processor role)  -> Processor Manager -> Processor(changefeed1) -> TablePipeline(tableA)
    26                                                        ╲                         ╲
    27                                                         ╲                         -> TablePipeline(tableB)
    28                                                          ╲
    29                                                           ╲
    30                                                            -> Processor(changefeed2) -> TablePipeline(tableC)
    31                                                                                      ╲
    32                                                                                       -> TablePipeline(tableD)
    33  
    34  
    35  */
    36  package processor