github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/docs/sources/design-documents/2020-09-Write-Ahead-Log.md (about) 1 --- 2 title: Write-Ahead Logs 3 weight: 30 4 --- 5 ## Write-Ahead Logs 6 7 Author: Owen Diehl - [owen-d](https://github.com/owen-d) ([Grafana Labs](https://grafana.com/)) 8 9 Date: 30/09/2020 10 11 ## Impetus 12 13 Loki already takes numerous steps to ensure the persistence of log data, most notably the use of a configurable replication factor (redundancy) in the ingesters. However, this still leaves much to be desired in persistence guarantees, especially for single binary deployments. This proposal outlines a write ahead log (WAL) in order to complement existing measures by allowing storage/replay of incoming writes via local disk on the ingester components. 14 15 ## Strategy 16 17 We suggest a two pass WAL implementation which includes an initial recording of accepted writes (`segments`) and a subsequent checkpointing (`checkpoints`) which coalesces the first pass into more efficient representations to speed up replaying. 18 19 ### Segments 20 21 Segments are the first pass and most basic WAL. They store individual records of incoming writes that have been accepted and can be used to reconstruct the in memory state of an ingester without any external input. Each segment is some multiple of 32kB and upon filling one segment, a new segment is created. Initially Loki will try 256kb segment sizes, readjusting as necessary. They are sequentially named on disk and are automatically created when a target size is hit as follows: 22 23 ``` 24 data 25 └── wal 26 ├── 000000 27 ├── 000001 28 └── 000002 29 ``` 30 31 ### Truncation 32 33 In order to prevent unbounded growth and remove operations which have been flushed to storage from the WAL, it is regularly truncated and all but the last segment (which is currently active) are deleted at a configurable interval (`ingester.checkpoint-duration`). This is where checkpoints come into the picture. 34 35 ### Checkpoints 36 37 Before truncating the WAL, we advance the WAL segments by one in order to ensure we don't delete the currently writing segment. The directory will look like: 38 39 ``` 40 data 41 └── wal 42 ├── 000000 43 ├── 000001 44 ├── 000002 <- likely not full, no matter 45 └── 000003 <- newly written, empty 46 ``` 47 48 Each in memory stream is iterated across an interval, calculated by `checkpoint_duration / in_memory_streams` and written to the checkpoint. After the checkpoint completes, it is moved from its temp directory to the `ingester.wal-dir`, taking the name of the last segment before it started (`checkpoint.000002`) and then all applicable segments (`00000`, `00001`, `00002`) and any previous checkpoint are deleted. 49 50 Afterwards, it will look like: 51 52 ``` 53 data 54 └── wal 55 ├── checkpoint.000002 <- completed checkpoint 56 └── 000003 <- currently active wal segment 57 ``` 58 59 #### Queueing Checkpoint operations 60 61 It’s possible that one checkpoint operation will start at the same time another is running. In this case, the existing checkpoint operation should disregard its internal ticker and flush its series as fast as possible. Afterwards, the next checkpoint operation can begin. This will likely create a localized spike in IOPS before the amortization of the following checkpoint operation takes over and is another important reason to run the WAL on an isolated disk in order to mitigate noisy neighbor problems. After we've written/moved the current checkpoint, we reap the old one. 62 63 ### WAL Record Types 64 65 #### Streams 66 67 A `Stream` record type is written when an ingester receives a push for a series it doesn't yet have in memory. At a high level, this will contain 68 ```golang 69 type SeriesRecord struct { 70 UserID string 71 Labels labels.Labels 72 Fingerprint uint64 // label fingerprint 73 } 74 ``` 75 76 #### Logs 77 78 A `Logs` record type is written when an ingester receives a push, containing the fingerprint of the series it refers to and a list of `(timestamp, log_line)` tuples, _after_ a `Stream` record type is written, if applicable. 79 ```golang 80 type LogsRecord struct { 81 UserID string 82 Fingperprint uint64 // label fingerprint for the series these logs refer to 83 Entries []logproto.Entry 84 } 85 ``` 86 87 ### Restoration 88 89 Replaying a WAL is done by loading any available checkpoints into memory and then replaying any operations from successively named segments on top (`checkpoint.000003` -> `000004` -> `000005`, etc). It's likely some of these operations will fail because they're already included in the checkpoint (due to delay introduced in our amortizations), but this is ok -- we won't _lose_ any data, only try to write some data twice, which will be ignored. 90 91 ### Deployment 92 93 Introduction of the WAL requires that ingesters have persistent disks which are reconnected across restarts (this is a good fit for StatefulSets in Kubernetes). Additionally, it's recommended that the WAL uses an independent disk such that it's isolated from being affected by or causing noisy neighbor problems, especially during any IOPS spike(s). 94 95 ### Implementation goals 96 97 - Use underlying prometheus wal pkg when possible for consistency & to mitigate undifferentiated heavy lifting. Interfaces handle page alignment & use []byte. 98 - Ensure this package handles arbitrarily long records (log lines in Loki’s case). 99 - Ensure our in memory representations can be efficiently moved to/from `[]byte` in order to generate conversions for fast/efficient loading from checkpoints. 100 - Ensure chunks which have already been flushed to storage are kept around for `ingester.retain-period`, even after a WAL replay. 101 102 ### Alternatives 103 104 #### Use the Cortex WAL 105 106 Since we're not checkpointing from the WAL records but instead doing a memory dump, this isn't bottlenecked by throughput but rather memory size. Therefore we can start with checkpointing by duration rather than accounting for throughput as well. This makes the proposed solution nearly identical to the Cortex WAL approach. The one caveat is that wal segments will accrue between checkpoint operations and may constitute a large amount of data (log throughput varies). We may eventually consider other routes to handle this if duration based checkpointing proves insufficient. 107 108 #### Don't build checkpoints from memory, instead write new WAL elements 109 110 Instead of building checkpoints from memory, this would build the same efficiencies into two distinct WAL Record types: `Blocks` and `FlushedChunks`. The former is a record type which will contain an entire compressed block after it's cut and the latter will contain an entire chunk + the sequence of blocks it holds when it's flushed. This may offer good enough amortization of writes because block cuts are assumed to be evenly distributed & chunk flushes have the same property and use jitter for synchronization. 111 112 This could be used to drop WAL records which have already elapsed the `ingester.retain-period`, allowing for faster WAL replays and more efficient loading. 113 ```golang 114 type FlushRecord struct { 115 Fingerprint uint64 // labels 116 FlushedAt uint64 // timestamp when it was flushed, can be used with `ingester.retain-period` to either keep or discard records on replay 117 LastEntry logproto.Entry // last entry included in the flushed chunk 118 } 119 ``` 120 121 It would also allow building checkpoints without relying on an ingester's internal state, but would likely require multiple WALs, partitioned by record type in order to be able to iterate all `FlushedChunks` -> `Blocks` -> `Series` -> `Samples` such that we could no-op the later (lesser priority) types that are superseded by the former types. The benefits do not seem worth the cost here, especially considering the simpler suggested alternative and the extensibility costs if we need to add new record types if/when the ingester changes internally.