github.com/m3db/m3@v1.5.0/src/dbnode/persist/fs/commitlog/README.md (about) 1 # Overview 2 3 The commitlog package contains all the code for reading and writing commitlog files. 4 5 # Implementation 6 7 ## Writer 8 9 The commitlog writer consists of two main components: 10 11 1. The `CommitLog` itself which contains all of the public APIs as well as the logic for managing concurrency, rotating files, and inspecting the status of the commitlog (via the `ActiveLogs` and `QueueLength` APIs). 12 2. The `CommitLogWriter` which contains all of the logic for opening/closing files and writing bytes to disk. 13 14 ### Commitlog 15 16 At a high-level, the `CommitLog` handles writes by inserting them into a queue (buffered channel) in batches and then dequeing the batches in a single-threaded manner and writing all of the individual writes for a batch into the commitlog file one at a time. 17 18 #### Synchronization 19 20 The primary synchronization that happens in the commitlog is via the queue (buffered channel). Many goroutines will insert pending writes into the queue concurrently via the `Write()` and `WriteBatch()` APIs, however, only a single goroutine will pull items off the queue and write them to disk. 21 22 ``` 23 ┌──────────────────────────┐ 24 │Goroutine 1: WriteBatch() ├─────┐ ┌───────────────────────────────────┐ ┌───────────────────────────────────┐ 25 └──────────────────────────┘ │ │ │ │ │ 26 │ │ │ │ │ 27 │ │ │ │ │ 28 ┌──────────────────────────┐ │ ┌─────────────────────────────────────┐ │ Commitlog │ │ Writer │ 29 │Goroutine 2: WriteBatch() │─────┼──────▶│ Commitlog Queue ├──────────▶│ │─────────▶│ │ 30 └──────────────────────────┘ │ └─────────────────────────────────────┘ │ Single-threaded Writer Goroutine │ │ Write to disk │ 31 │ │ │ │ │ 32 │ │ │ │ │ 33 ┌──────────────────────────┐ │ │ │ │ │ 34 │Goroutine 3: WriteBatch() ├─────┘ └───────────────────────────────────┘ └───────────────────────────────────┘ 35 └──────────────────────────┘ 36 ``` 37 *The monopic file that was used to generate the diagram above is located at `docs/code_assets/commitlog/queue.monopic`.* 38 39 Since there is only one goroutine pulling items off of the queue, any state that it alone manages can remain unsynchronized since no other goroutines will interact with it. 40 41 In addition to the queue, the commitlog has two other forms of synchronization: 42 43 1. The `closedState` lock which is an RWLock. An RLock is held for the duration of any operation during which the commitlog must remain open. 44 2. The `flushState` lock. The scope of this lock is very narrow as its only used to protect access to the `lastFlushAt` field. 45 46 #### Rotating Files 47 48 Rotating commitlog files is initiated by the `RotateLogs()` API so that callers can control when this occurs. The `CommitLog` itself will never rotate files on its own without the `RotateLogs()` API being called. 49 50 The commitlog files are not rotated immediately when the `RotateLogs()` method is called because that would require a lot of complicated and expensive synchronization with the `CommitLogWriter` goroutine. Instead, a `rotateLogsEventType` is pushed into the queue and when the single-threaded writer goroutine pulls this event off of the channel it will rotate the commitlog files (since it has exclusive access to them) and then invoke a callback function which notifies the `RotateLogs()` method call (which has been blocked this whole time) to complete and return success to the caller. 51 52 While the `CommitLog` only writes to a single file at once, it maintains two open writers at all times so that they can be "hot-swapped" when the commitlog files need to be rotated. This allows the single-threaded writer goroutine to continue uninterrupted by syscalls and I/O during rotation events which in turn prevents the queue from backing up. Otherwise, rotation events could block the writer for so long (while it waited for a new file to be created) that it caused the queue to back up significantly. 53 54 When a rotation event occurs, instead of waiting for a new file to be opened, the `CommitLog` writer goroutine will swap the primary and secondary `CommitLogWriter` such that the secondary `CommitLogWriter` (which has an empty file) becomes the primary and vice versa. This allows the `CommitLog` writer goroutine to continue writing uninterrupted. 55 56 In the meantime, a goroutine is started in the background that is responsible for resetting the now secondary (formerly primary) `CommitLogWriter` by closing it (which will flush any pending / buffered writes to disk) and re-opening it (which will create a new empty commitlog file in anticipation of the next rotation event). 57 58 Finally, the next time the `CommitLog` attempts to rotate its commitlogs it will need to use the associated `sync.WaitGroup` to ensure that the previously spawned background goroutine has completed resetting the secondary `CommitLogWriter` before it attempts a new hot-swap. 59 60 ### Handling Errors 61 62 The current implementation will panic if any I/O errors are ever encountered while writing bytes to disk or opening/closing files. In the future a "commitlog failure policy" similar to [Cassandra's "stop"](https://github.com/apache/cassandra/blob/6dfc1e7eeba539774784dfd650d3e1de6785c938/conf/cassandra.yaml#L232) may be introduced. 63 64 # Testing 65 66 The commitlog package is tested via: 67 68 1. Standard unit tests 69 2. Property tests 70 3. Concurrency tests 71 72 # File Format 73 74 See `/docs/architecture/m3db/commitlogs.md`.