github.com/m3db/m3@v1.5.0/specs/dbnode/flush/FlushVersion.tla (about) 1 ---------------------------- MODULE FlushVersion ---------------------------- 2 EXTENDS Integers, TLC 3 4 \* Constants used to limit the potential search space when 5 \* running the TLC model checker. 6 CONSTANT maxNumWrites 7 CONSTANT maxNumFlushes 8 CONSTANT maxNumTicks 9 10 (*************************************************************************** 11 \* This specification models the version-based flushing/ticking of series 12 \* buffer buckets that is required for the warm/cold write feature. The 13 \* specification models the system by running three different processes: 14 \* 15 \* 1. One process responsible for beginning and ending the flush process. 16 \* This process will begin a flush by looping through all the BucketsInMemory 17 \* and setting their LastSuccessfulFlushVersion to whatever the 18 \* CurrentLastSuccessfulFlushVersion is. It also models both flush 19 \* successes and flush failures. 20 \* Flush Success: Copy all the BucketsInMemory whose 21 |* LastSuccessfulFlushVersion is smaller than 22 \* or equal to the CurrentLastSuccessfulFlushVersion (I.E those 23 \* we could have conceivably known about at flush start time) into 24 \* the set of PersistedBuckets, and then increment the 25 \* CurrentLastSuccessfulFlushVersion. 26 \* 27 \* Flush Failure: Nothing happens and the CurrentLastSuccessfulFlushVersion stays the same. 28 \* 29 \* 30 \* 2. One process responsible for creating new buckets by issuing "writes". It does 31 \* this by storing them in the BucketsInMemory set, as well as in the 32 \* WrittenBuckets set (so we can keep track of them independently of the eviction 33 \* from memory process). 34 \* 35 \* 36 \* 3. One process responsible for the "Ticking" process that will periodically evict 37 \* every bucket from the BucketsInMemory set whose LastSuccessfulFlushVersion is >= 38 \* CurrentLastSuccessfulFlushVersion and also > 0 (because version zero is the bucket 39 \* for unflushed writes). 40 \* 41 \* 42 \* The specification verifies the system by allowing all of these processes to run 43 \* concurrently (as they do in M3DB) and asserting that the invariant that we can 44 \* always read every bucket that has been written from the combination of the 45 \* BucketsInMemory and PersistedBuckets holds true at every step (meaning we never 46 \* lose data due to the Tick evicting data that hasn't been flushed yet.) 47 --algorithm FlushVersion 48 49 variable 50 \* BucketsInMemory, WrittenBuckets, and Persisted buckets are all sets that 51 \* store records in the form: 52 \* 1. ID (int) 53 \* 2. LastSuccessfulFlushVersion (int) 54 \* 55 \* BucketsInMemory stores all the buckets that are currently in the M3DB 56 \* process's memory, WrittenBuckets stores all the buckets that the client 57 \* ever issued (to serve as a source of truth for checking invariants), and 58 \* PersistedBuckets stores all the buckets that have been flushed successfully. 59 BucketsInMemory = {[ID |-> 0, FlushVersion |-> 0]}; 60 WrittenBuckets = {[ID |-> 0, FlushVersion |-> 0]}; 61 PersistedBuckets = {}; 62 \* The last version that was successfully flushed to disk. 63 LastSuccessfulFlushVersion = 0; 64 65 \* Simulate background flushes which either succeed, copy buckets to the 66 \* PersistedBuckets set and then increment the flush version, or fail and 67 \* leave everything as is. 68 process Flush = 0 69 variable 70 CurrentIndex = 0; 71 \* Whether a flush is currently ongoing. 72 FlushInProgress = FALSE; 73 begin 74 flush_loop: while CurrentIndex < maxNumFlushes do 75 either 76 await FlushInProgress = TRUE; 77 78 either 79 \* Flush success 80 \* 81 \* Move all BucketsInMemory that have been flushed (according to 82 \* their flush version) to the PersistedBuckets set but keep them, 83 \* around in memory for now. 84 PersistedBuckets := PersistedBuckets \union { 85 x \in BucketsInMemory: 86 x.FlushVersion <= LastSuccessfulFlushVersion + 1 /\ 87 x.FlushVersion > 0}; 88 LastSuccessfulFlushVersion := LastSuccessfulFlushVersion + 1; 89 or 90 \* Flush failure - No-op 91 LastSuccessfulFlushVersion := LastSuccessfulFlushVersion; 92 end either; 93 94 FlushInProgress := FALSE; 95 or 96 await FlushInProgress = FALSE; 97 98 \* Update the flush version of all BucketsInMemory in memory to the 99 \* current value + 1 since that is what the (possibly partially complete) 100 \* flush could have done to them. 101 BucketsInMemory := { 102 [ID |-> x.ID, FlushVersion |-> LastSuccessfulFlushVersion + 1]: 103 x \in BucketsInMemory}; 104 FlushInProgress := TRUE; 105 end either; 106 107 CurrentIndex := CurrentIndex+1; 108 end while 109 end process 110 111 \* Simulate a process that is writing by putting new buckets into BucketsInMemory. 112 process Write = 1 113 variable 114 CurrentIndex = 0; 115 NextBucketID = 1; 116 begin 117 write_loop: while CurrentIndex < maxNumWrites do 118 \* Write a new bucket. Note that the FlushVersion is always zero for a new bucket. 119 BucketsInMemory := BucketsInMemory \union {[ID |-> NextBucketID, FlushVersion |-> 0]}; 120 WrittenBuckets := WrittenBuckets \union {[ID |-> NextBucketID, FlushVersion |-> 0]}; 121 NextBucketID := NextBucketID + 1; 122 CurrentIndex := CurrentIndex+1; 123 end while 124 end process 125 126 process Tick = 2 127 variable CurrentIndex = 0; 128 129 begin 130 tick_loop: while CurrentIndex < maxNumTicks do 131 \* Evict from memory any block whose flush version is smaller than or equal to the 132 \* current flush version. 133 BucketsInMemory := BucketsInMemory \ { 134 x \in BucketsInMemory: 135 x.FlushVersion <= LastSuccessfulFlushVersion /\ 136 x.FlushVersion > 0}; 137 end while 138 end process 139 140 end algorithm 141 142 143 ***************************************************************************) 144 \* BEGIN TRANSLATION 145 \* Process variable CurrentIndex of process Flush at line 66 col 9 changed to CurrentIndex_ 146 \* Process variable CurrentIndex of process Write at line 114 col 9 changed to CurrentIndex_W 147 VARIABLES BucketsInMemory, WrittenBuckets, PersistedBuckets, 148 LastSuccessfulFlushVersion, pc, CurrentIndex_, FlushInProgress, 149 CurrentIndex_W, NextBucketID, CurrentIndex 150 151 vars == << BucketsInMemory, WrittenBuckets, PersistedBuckets, 152 LastSuccessfulFlushVersion, pc, CurrentIndex_, FlushInProgress, 153 CurrentIndex_W, NextBucketID, CurrentIndex >> 154 155 ProcSet == {0} \cup {1} \cup {2} 156 157 Init == (* Global variables *) 158 /\ BucketsInMemory = {[ID |-> 0, FlushVersion |-> 0]} 159 /\ WrittenBuckets = {[ID |-> 0, FlushVersion |-> 0]} 160 /\ PersistedBuckets = {} 161 /\ LastSuccessfulFlushVersion = 0 162 (* Process Flush *) 163 /\ CurrentIndex_ = 0 164 /\ FlushInProgress = FALSE 165 (* Process Write *) 166 /\ CurrentIndex_W = 0 167 /\ NextBucketID = 1 168 (* Process Tick *) 169 /\ CurrentIndex = 0 170 /\ pc = [self \in ProcSet |-> CASE self = 0 -> "flush_loop" 171 [] self = 1 -> "write_loop" 172 [] self = 2 -> "tick_loop"] 173 174 flush_loop == /\ pc[0] = "flush_loop" 175 /\ IF CurrentIndex_ < maxNumFlushes 176 THEN /\ \/ /\ FlushInProgress = TRUE 177 /\ \/ /\ PersistedBuckets' = ( PersistedBuckets \union { 178 x \in BucketsInMemory: 179 x.FlushVersion <= LastSuccessfulFlushVersion + 1 /\ 180 x.FlushVersion > 0}) 181 /\ LastSuccessfulFlushVersion' = LastSuccessfulFlushVersion + 1 182 \/ /\ LastSuccessfulFlushVersion' = LastSuccessfulFlushVersion 183 /\ UNCHANGED PersistedBuckets 184 /\ FlushInProgress' = FALSE 185 /\ UNCHANGED BucketsInMemory 186 \/ /\ FlushInProgress = FALSE 187 /\ BucketsInMemory' = { 188 [ID |-> x.ID, FlushVersion |-> LastSuccessfulFlushVersion + 1]: 189 x \in BucketsInMemory} 190 /\ FlushInProgress' = TRUE 191 /\ UNCHANGED <<PersistedBuckets, LastSuccessfulFlushVersion>> 192 /\ CurrentIndex_' = CurrentIndex_+1 193 /\ pc' = [pc EXCEPT ![0] = "flush_loop"] 194 ELSE /\ pc' = [pc EXCEPT ![0] = "Done"] 195 /\ UNCHANGED << BucketsInMemory, PersistedBuckets, 196 LastSuccessfulFlushVersion, 197 CurrentIndex_, FlushInProgress >> 198 /\ UNCHANGED << WrittenBuckets, CurrentIndex_W, NextBucketID, 199 CurrentIndex >> 200 201 Flush == flush_loop 202 203 write_loop == /\ pc[1] = "write_loop" 204 /\ IF CurrentIndex_W < maxNumWrites 205 THEN /\ BucketsInMemory' = (BucketsInMemory \union {[ID |-> NextBucketID, FlushVersion |-> 0]}) 206 /\ WrittenBuckets' = (WrittenBuckets \union {[ID |-> NextBucketID, FlushVersion |-> 0]}) 207 /\ NextBucketID' = NextBucketID + 1 208 /\ CurrentIndex_W' = CurrentIndex_W+1 209 /\ pc' = [pc EXCEPT ![1] = "write_loop"] 210 ELSE /\ pc' = [pc EXCEPT ![1] = "Done"] 211 /\ UNCHANGED << BucketsInMemory, WrittenBuckets, 212 CurrentIndex_W, NextBucketID >> 213 /\ UNCHANGED << PersistedBuckets, LastSuccessfulFlushVersion, 214 CurrentIndex_, FlushInProgress, CurrentIndex >> 215 216 Write == write_loop 217 218 tick_loop == /\ pc[2] = "tick_loop" 219 /\ IF CurrentIndex < maxNumTicks 220 THEN /\ BucketsInMemory' = BucketsInMemory \ { 221 x \in BucketsInMemory: 222 x.FlushVersion <= LastSuccessfulFlushVersion /\ 223 x.FlushVersion > 0} 224 /\ pc' = [pc EXCEPT ![2] = "tick_loop"] 225 ELSE /\ pc' = [pc EXCEPT ![2] = "Done"] 226 /\ UNCHANGED BucketsInMemory 227 /\ UNCHANGED << WrittenBuckets, PersistedBuckets, 228 LastSuccessfulFlushVersion, CurrentIndex_, 229 FlushInProgress, CurrentIndex_W, NextBucketID, 230 CurrentIndex >> 231 232 Tick == tick_loop 233 234 Next == Flush \/ Write \/ Tick 235 \/ (* Disjunct to prevent deadlock on termination *) 236 ((\A self \in ProcSet: pc[self] = "Done") /\ UNCHANGED vars) 237 238 Spec == Init /\ [][Next]_vars 239 240 Termination == <>(\A self \in ProcSet: pc[self] = "Done") 241 242 \* END TRANSLATION 243 244 245 \* Invariants, include these when running the model checker. 246 WrittenIDs == {x.ID: x \in WrittenBuckets} 247 ReadableIDs == {x.ID: x \in (BucketsInMemory \union PersistedBuckets)} 248 DoesNotLoseData == WrittenIDs \subseteq ReadableIDs 249 250 ============================================================================= 251 \* Modification History 252 \* Last modified Fri Nov 30 19:08:55 EST 2018 by richardartoul 253 \* Created Fri Nov 30 15:08:04 EST 2018 by richardartoul