github.com/zuoyebang/bitalosdb@v1.1.1-0.20240516111551-79a8c4d8ce20/README.md (about)

     1  ![bitalos](./docs/bitalos.png)
     2  
     3  ### Bitalosdb is a high-performance KV storage engine. [中文版](./README_CN.md)
     4  
     5  ## Introduction
     6  
     7  - Bitalosdb is a high-performance KV storage engine (self-developed), which is based on the creative IO architecture and storage technology and focuses on solving the problem of LSM-Tree read and write amplification. As a substitute for Rocksdb, Bitalosdb can improve read and write performance greatly.
     8  
     9  ## Team
    10  
    11  - Produced: Zuoyebang Company - Platform technical team
    12  
    13  - Author: Xu Ruibo(hustxrb@163.com)
    14  
    15  - Contributors: Xing Fu(wzxingfu@gmail.com), Lu Wenwei(422213023@qq.com), Liu Fang(killcode13@sina.com)
    16  
    17  ## Key Technology
    18  
    19  - Bithash (KV separation technology), significantly reduce write amplification. For bithash, time complexity of retrieval is O(1). GC can be completed independently, and value and index are decoupled.
    20  
    21  - Bitalostree (high-performance compression index technology), basically eliminate read amplification. If B+ Tree has a number of huge Pages, write amplification is a severe problem. With a creative index compression technology, Bitalostree eliminates B+ Tree write amplification, and improves the read performance.
    22  
    23  - Bitalostable (cold and hot data separation technology), stores cold data, which is calculated according to the data scale and access frequency. Storage engine writes cold data to Bitalostable when QPS becomes low. Improve data compression, reduce index memory consumption, and achieve more rational resource utilization. (open source stable edition has basic features, enterprise edition supports more comprehensive hot and cold separation).
    24  
    25  ## Performance
    26  
    27  - This benchmark is based on Bitloasdb version (v5.0) and rocksdb stable version (v7.6.0).
    28  
    29  ### Hardware
    30  
    31  ```
    32  CPU:    Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz
    33  Memory: 384GB
    34  Disk:   2*3.5TB NVMe SSD
    35  ```
    36  
    37  ### Program
    38  
    39  - Benchmark thread number: 8
    40  
    41  - Program cpu cgroup: 8 core
    42  
    43  - Comparison standard: QPS on single-core (multi-core QPS / core number), single-core performance reflects cost advantage better.
    44  
    45  ### Data
    46  
    47  - Key-value spec: key-size=32B、value-size=1KB
    48  
    49  - Comparison dimensions: Total data size(25/50/100GB) * IO ratio(100% random write, 100% random read, 50% random write + 50% random read, 30% random write + 70% random read)
    50  
    51  ### Config
    52  
    53  - rocksdb
    54  
    55  ```
    56  Memtable：256MB
    57  WAL：enable
    58  Cache：8GB
    59  TargetFileSize：256M
    60  L0CompactTrigger：8
    61  L0StopWritesTrigger：24
    62  ```
    63  
    64  - bitalosdb
    65  
    66  ```
    67  Memtable：256MB
    68  WAL：enable
    69  Cache：disable
    70  ```
    71  
    72  ### Result
    73  
    74  - QPS ([Horizontal](./docs/benchmark-qps.png))
    75  
    76  ![benchmark](./docs/benchmark-qps-vertical.png)
    77  
    78  ## Document
    79  
    80  Technical architecture and documentation, refer to the official website: [bitalos.zuoyebang.com](https://bitalos.zuoyebang.com)