github.com/matrixorigin/matrixone@v1.2.0/pkg/queryservice/doc.go (about)

     1  // Copyright 2021 - 2023 Matrix Origin
     2  //
     3  // Licensed under the Apache License, Version 2.0 (the "License");
     4  // you may not use this file except in compliance with the License.
     5  // You may obtain a copy of the License at
     6  //
     7  //      http://www.apache.org/licenses/LICENSE-2.0
     8  //
     9  // Unless required by applicable law or agreed to in writing, software
    10  // distributed under the License is distributed on an "AS IS" BASIS,
    11  // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    12  // See the License for the specific language governing permissions and
    13  // limitations under the License.
    14  
    15  /*
    16  Implement the remote cache of the file service between CN nodes.
    17  When the cache of the file service on this CN node misses, try to
    18  send a cache read request to the CN node which has the key and
    19  return the cached data.
    20  
    21  If the remote node hits the cache, the data will not be read from S3.
    22  Otherwise, continue to read data from S3.
    23  
    24  The broadcast of the cache key is done between CN nodes through
    25  the gossip protocol. In the case that the cluster is running normally,
    26  the CN node sends the added/deleted cache key and broadcasts it
    27  o all other nodes through gossip. When a CN node joins, it will select
    28  one from the original CN node to copy the whole cache key data,
    29  and continue to update the newly added/deleted cache key. When
    30  one CN node goes offline, the data on all nodes will be updated
    31  through the dead node event, and the cache key information on
    32  the corresponding node will be removed.
    33  
    34  The process likes the following:
    35  1. Client sends a read request to a CN instance.
    36  2. CN reads the data from local file service, which will probably hit
    37     the local cache(memory or disk).
    38  3. But, if it does not hit local cache and if remote cache is enabled,
    39     it will gets remote target
    40     from key router, which is implemented by gossip protocol.
    41  4. The local file service sends a remote cache read request to the target
    42     which is got from gossip.
    43  5. The remote file service instance sends back cache data response.
    44  6. If the remote cache hits, the local file service will set the data in
    45     the vector; otherwise, it will read from S3.
    46  
    47       +--------------+
    48       | read request |
    49       +--------------+
    50              |
    51              v
    52     +------------------+           +------------------+          +------------------+
    53     |        CN        |           |        CN        |          |        CN        |
    54     |  +------------+  |           |  +------------+  |          |  +------------+  |
    55     |  |   gossip   |  |           |  |   gossip   |  |          |  |   gossip   |  |
    56     |  |    keys    |  |           |  |    keys    |  |          |  |    keys    |  |
    57     |  +------------+  |           |  +------------|  |          |  +------------|  |
    58     |          ^  |    |           |                  |          |                  |
    59     |          |  |    |           |                  |          |                  |
    60     +----------|--|----+           +------------------+          +------------------+
    61            |   |  |                          |                            |
    62            |  key |                          |                            |
    63            |   |  +--target--+               |                            |
    64            v   +-------+     |               v                            v
    65     +--------------+   |     |        +--------------+             +--------------+
    66     | file-service |---+     |        | file-service |             | file-service |
    67     |              |<--------|        +--------------+             +--------------+
    68     +--------------+                                                     ^  |
    69          ^   |                                                           |  |
    70          |   +-----------------request remote cache target---------------+  |
    71          +------------------------cache data response-----------------------+
    72  
    73  The gossip nodes in cluster works:
    74  
    75                                                                                                   +-------------------+
    76           CN1                               CN2                           CN3                     |       new CN4     |
    77     +-------------+                   +-------------+                +-------------+              |  +-------------+  |
    78     | gossip node |                   | gossip node |                | gossip node |--all items-->|  | gossip node |  |
    79     +-------------+                   +-------------+                +-------------+              |  +-------------+  |
    80         ^      ^ |                       ^   ^   |                       ^     ^                  |                   |
    81         |      | |                       |   |   |                       |     |                  |                   |
    82     set/evict  | +--[item->target(CN1)]--+   |   +--[item->target(CN1)]--+     |                  |                   |
    83       item     |                             |                                 |                  +------------------ +
    84         |      +-------+                     |                                 |                       delete CN 4
    85         |              |                     |                                 |                           |
    86     +--------------+   +---------------------+-------------------------------remove [items->target(CN4)]---+
    87     | file service |
    88     +--------------+
    89  
    90  */
    91  
    92  package queryservice