github.com/weedge/lib@v0.0.0-20230424045628-a36dcc1d90e4/asyncbuffer/readme.md (about)

     1  ### intro
     2  累积数据至buffer中，异步批量处理
     3  
     4  #### 使用场景
     5  
     6  1. Write-behind cache 模式，缓存批量入库，减少网络io, 以及入库磁盘io （补偿机制可以使用WAL(Write-Ahead Logging)的方式顺序写日志,todo）；对于写频率高的场景非常适合，比如投票，股票价格变动，以及课中直播互动场景等
     7  
     8  
     9  #### 配置
    10  ```json
    11      {
    12        "async_buffer": {//用于异步批量处理的buffer配置列表
    13          "nmq_live_user": {//buffer名称
    14              "buffer_win_size": 1,//buffer窗口大小，等于这个值时触发批量操作
    15              "delay_do_ms": 10,//延时10ms做批量处理
    16              "chs": {//多个channel列表
    17                  "nmq": {//channel名称
    18                      "ch_len": 0,//channel的长度
    19                      "sub_worker_num": 1//从channel中取数据的协程数目
    20                  }
    21              }
    22          },
    23          "nmq_live_room": {
    24              "buffer_win_size": 1,
    25              "delay_do_ms": 10,
    26              "chs": {
    27                  "nmq": {
    28                      "ch_len": 0,
    29                      "sub_worker_num": 1
    30                  }
    31              }
    32          },
    33          "nmq_live_org": {
    34              "buffer_win_size": 1,
    35              "delay_do_ms": 10,
    36              "chs": {
    37                  "nmq": {
    38                      "ch_len": 0,
    39                      "sub_worker_num": 1
    40                  }
    41              }
    42          },
    43          "default": {
    44              "buffer_win_size": 1,
    45              "delay_do_ms": 10,
    46              "chs": {
    47                  "default": {
    48                      "ch_len": 0,
    49                      "sub_worker_num": 1
    50                  }
    51              }
    52          }
    53      }
    54    }
    55  
    56  ```
    57  
    58  #### 流程
    59  
    60  ```
    61                                    buffer_win_size
    62                           ---ch1---  [======]  -sub batch worker-
    63  IBuffer -(FormatInput)-> ---ch2---  [======]  -sub batch worker-  -(BatchDo)->(mq,cache,db)
    64                           ---ch3---  [======]  -sub batch worker-
    65                           ....                 .....
    66                       ---flushch---  [======]  -flush worker-
    67  
    68  ```
    69  
    70  #### 使用
    71  ```go
    72  type IBuffer interface {
    73  	BatchDo([][]byte)//批量输出
    74  	FormatInput() (error, []byte)//格式化输入的数据写入buffer
    75  }
    76  ```
    77   1. 自定义根据buffer的实体数据结构, 实现IBuffer中的方法
    78   2. 通过 `SendOneCh(userBufferName, chName, data)` 方法将数据data写入对应的channel buffer中
    79   3. 通过`FlushAll();  FlushOne(bufferName string)` flush全部buffer,flush某个buffer(异步方式)
    80  
    81  #### notice
    82   1. if batchDo panic, bufferData ingore
    83  
    84  #### todo
    85  - [ ] one IBuffer send to multi ch pub and sub batchDo 
    86  - [ ] use multi buffer for multi sub worker to replace mutex lock
    87  
    88  
    89  
    90  #### reference
    91  
    92  1. [什么是 WAL](https://segmentfault.com/a/1190000022512468)
    93  2. [Write-ahead_logging](https://en.wikipedia.org/wiki/Write-ahead_logging)
    94  2. [ARIES:Algorithms_for_Recovery_and_Isolation_Exploiting_Semantics](https://en.wikipedia.org/wiki/Algorithms_for_Recovery_and_Isolation_Exploiting_Semantics)
    95  2. [ARIES Overview, Types of Log Records, ARIES Helper Structures](https://www.youtube.com/watch?v=S9nctHdkggk)
    96  2. [ARIES Database Recovery (CMU Databases Systems / Fall 2019)](https://www.youtube.com/watch?v=4VGkRXVM5fk)
    97  2. [本地事务](http://icyfenix.cn/architect-perspective/general-architecture/transaction/local.html)
    98