github.com/cockroachdb/pebble@v1.1.1-0.20240513155919-3622ade60459/testdata/flushable_ingest (about) 1 # We create multiple SSTs, one of which overlaps with the memtable (scheduling a flush). 2 # Check that the SSTs get ingested to the lowest levels possible. 3 4 batch 5 set a 0 6 ---- 7 8 # The SST below overlaps with memtable and thus should be placed in L0 9 # post flush. 10 build ext1 11 set a 1 12 ---- 13 14 # The SST below overlaps with nothing and thus should be placed in L6 post 15 # flush. 16 build ext2 17 set b 1 18 ---- 19 20 # The SST below doesn't overlap with any SSTs in the LSM and thus can be placed 21 # in L6 post-flush. 22 build ext3 23 set d 1 24 ---- 25 26 # We block the flush, so the SSTs should still be in the flushable queue. 27 blockFlush 28 ---- 29 30 ingest ext1 ext2 ext3 31 ---- 32 33 allowFlush 34 ---- 35 36 lsm 37 ---- 38 39 get 40 a 41 b 42 d 43 ---- 44 a:1 45 b:1 46 d:1 47 48 # We expect 1 WAL for an immutable memtable, 1 file for the ingested ssts, 49 # one for the mutable memtable. We also expect 3 ssts corresponding to the 50 # ingested files. 51 ls 52 ---- 53 000002.log 54 000004.sst 55 000005.sst 56 000006.sst 57 000007.log 58 000008.log 59 CURRENT 60 LOCK 61 MANIFEST-000001 62 OPTIONS-000003 63 ext 64 marker.format-version.000015.016 65 marker.manifest.000001.MANIFEST-000001 66 67 # Test basic WAL replay 68 close 69 ---- 70 71 # In this case only the flushable was holding a reference to the sstables. Even 72 # after the DB is closed, those sstables should still be hanging around. 73 ls 74 ---- 75 000002.log 76 000004.sst 77 000005.sst 78 000006.sst 79 000007.log 80 000008.log 81 CURRENT 82 LOCK 83 MANIFEST-000001 84 OPTIONS-000003 85 ext 86 marker.format-version.000015.016 87 marker.manifest.000001.MANIFEST-000001 88 89 open 90 ---- 91 92 # Make sure that the sstables got flushed in the correct order on a WAL replay. 93 lsm 94 ---- 95 0.1: 96 000004:[a#11,SET-a#11,SET] 97 0.0: 98 000009:[a#10,SET-a#10,SET] 99 000005:[b#12,SET-b#12,SET] 100 000006:[d#13,SET-d#13,SET] 101 102 get 103 a 104 b 105 d 106 ---- 107 a:1 108 b:1 109 d:1 110 111 reset 112 ---- 113 114 # Repeat the steps above without closing Pebble. Note that the final lsm state 115 # will be different because WAL replay just placed the files in L0. 116 batch 117 set a 0 118 ---- 119 120 build ext1 121 set a 1 122 ---- 123 124 build ext2 125 set b 1 126 ---- 127 128 build ext3 129 set d 1 130 ---- 131 132 ingest ext1 ext2 ext3 133 ---- 134 135 lsm 136 ---- 137 0.1: 138 000004:[a#11,SET-a#11,SET] 139 0.0: 140 000009:[a#10,SET-a#10,SET] 141 6: 142 000005:[b#12,SET-b#12,SET] 143 000006:[d#13,SET-d#13,SET] 144 145 reset 146 ---- 147 148 # Test multiple overlapping ingests interleaving batch sets, and then flushing. 149 batch 150 set a 0 151 ---- 152 153 build ext4 154 set a 1 155 ---- 156 157 build ext5 158 set a 2 159 ---- 160 161 blockFlush 162 ---- 163 164 ingest ext4 165 ---- 166 167 allowFlush 168 ---- 169 170 get 171 a 172 ---- 173 a:1 174 175 batch 176 set b 1 177 ---- 178 179 get 180 a 181 b 182 ---- 183 a:1 184 b:1 185 186 # Should get ingested into L0 above the memtable flush. 187 blockFlush 188 ---- 189 190 ingest ext5 191 ---- 192 193 allowFlush 194 ---- 195 196 get 197 a 198 b 199 ---- 200 a:2 201 b:1 202 203 batch 204 set c 1 205 ---- 206 207 flush 208 ---- 209 210 lsm 211 ---- 212 0.2: 213 000007:[a#13,SET-a#13,SET] 214 0.1: 215 000004:[a#11,SET-a#11,SET] 216 0.0: 217 000010:[a#10,SET-a#10,SET] 218 000011:[b#12,SET-b#12,SET] 219 220 # Value of a should be the value of a in the second ingested SST. 221 get 222 a 223 b 224 c 225 ---- 226 a:2 227 b:1 228 c:1 229 230 # Test that non-overlapping ingest still works normally. 231 reset 232 ---- 233 234 batch 235 set a 0 236 ---- 237 238 build ext1 239 set b 1 240 ---- 241 242 build ext2 243 set d 1 244 ---- 245 246 ingest ext1 ext2 247 ---- 248 249 lsm 250 ---- 251 6: 252 000004:[b#11,SET-b#11,SET] 253 000005:[d#12,SET-d#12,SET] 254 255 256 # Verify target level of ingestedFlushable. 257 reset 258 ---- 259 260 batch 261 set a 0 262 ---- 263 264 build ext1 265 set a 1 266 ---- 267 268 build ext2 269 set b 1 270 ---- 271 272 build ext3 273 set d 1 274 ---- 275 276 ingest ext1 ext2 ext3 277 ---- 278 279 lsm 280 ---- 281 0.1: 282 000004:[a#11,SET-a#11,SET] 283 0.0: 284 000009:[a#10,SET-a#10,SET] 285 6: 286 000005:[b#12,SET-b#12,SET] 287 000006:[d#13,SET-d#13,SET] 288 289 290 batch 291 set a 3 292 ---- 293 294 build ext4 295 set a 4 296 ---- 297 298 build ext5 299 set b 5 300 ---- 301 302 ingest ext4 ext5 303 ---- 304 305 # Looking for the sstable with the key a to go into 0.3, and the sstable with 306 # key b to go into 0.0. The sstable doesn't go into L5, because L5 isn't open 307 # yet. 308 lsm 309 ---- 310 0.3: 311 000010:[a#15,SET-a#15,SET] 312 0.2: 313 000014:[a#14,SET-a#14,SET] 314 0.1: 315 000004:[a#11,SET-a#11,SET] 316 0.0: 317 000009:[a#10,SET-a#10,SET] 318 000011:[b#16,SET-b#16,SET] 319 6: 320 000005:[b#12,SET-b#12,SET] 321 000006:[d#13,SET-d#13,SET] 322 323 # Testing whether the new mutable memtable with data is flushed correctly during 324 # WAL replay. 325 reset 326 ---- 327 328 batch 329 set a 0 330 ---- 331 332 # The SST below overlaps with memtable and thus should be placed in L0 333 # post flush. 334 build ext1 335 set a 1 336 ---- 337 338 # The SST below overlaps with nothing and thus should be placed in L6 post 339 # flush. 340 build ext2 341 set b 1 342 ---- 343 344 # The SST below doesn't overlap with any SSTs in the LSM and thus can be placed 345 # in L6 post-flush. 346 build ext3 347 set d 1 348 ---- 349 350 # We block the flush, so the SSTs should still be in the flushable queue. 351 blockFlush 352 ---- 353 354 ingest ext1 ext2 ext3 355 ---- 356 357 # Add another write which should go to the new mutable memtable. 358 batch 359 set f 1 360 ---- 361 362 allowFlush 363 ---- 364 365 lsm 366 ---- 367 368 get 369 a 370 b 371 d 372 f 373 ---- 374 a:1 375 b:1 376 d:1 377 f:1 378 379 # We expect 1 WAL for an immutable memtable, 1 file for the ingested ssts, 380 # one for the mutable memtable. We also expect 3 ssts corresponding to the 381 # ingested files. 382 ls 383 ---- 384 000002.log 385 000004.sst 386 000005.sst 387 000006.sst 388 000007.log 389 000008.log 390 CURRENT 391 LOCK 392 MANIFEST-000001 393 OPTIONS-000003 394 ext 395 marker.format-version.000015.016 396 marker.manifest.000001.MANIFEST-000001 397 398 close 399 ---- 400 401 # In this case only the memtable was holding a reference to the sstables. Even 402 # after the DB is closed, those memtables should still be hanging around. 403 ls 404 ---- 405 000002.log 406 000004.sst 407 000005.sst 408 000006.sst 409 000007.log 410 000008.log 411 CURRENT 412 LOCK 413 MANIFEST-000001 414 OPTIONS-000003 415 ext 416 marker.format-version.000015.016 417 marker.manifest.000001.MANIFEST-000001 418 419 open 420 ---- 421 422 # Make sure that the sstables got flushed in the correct order on a WAL replay. 423 lsm 424 ---- 425 0.1: 426 000004:[a#11,SET-a#11,SET] 427 0.0: 428 000009:[a#10,SET-a#10,SET] 429 000005:[b#12,SET-b#12,SET] 430 000006:[d#13,SET-d#13,SET] 431 000010:[f#14,SET-f#14,SET] 432 433 # Check if the new mutable memtable is using a new log file, and that the 434 # previous log files have been deleted appropriately after the flush. 435 ls 436 ---- 437 000004.sst 438 000005.sst 439 000006.sst 440 000009.sst 441 000010.sst 442 000011.log 443 CURRENT 444 LOCK 445 MANIFEST-000001 446 MANIFEST-000012 447 OPTIONS-000013 448 ext 449 marker.format-version.000015.016 450 marker.manifest.000002.MANIFEST-000012 451 452 # Make sure that the new mutable memtable can accept writes. 453 batch 454 set h 2 455 ---- 456 457 get 458 h 459 ---- 460 h:2 461 462 # Test correct WAL replay with read only mode. We essentially want to make sure 463 # that once a flushable is added to the queue, we create a new mutable memtable 464 # on top of the flushable. Otherwise, we can invert sequence number invariants. 465 reset 466 ---- 467 468 batch 469 set a 0 470 ---- 471 472 # The SST below overlaps with memtable and thus should be placed in L0 473 # post flush. 474 build ext1 475 set a 1 476 ---- 477 478 # The SST below overlaps with nothing and thus should be placed in L6 post 479 # flush. 480 build ext2 481 set b 1 482 ---- 483 484 # The SST below doesn't overlap with any SSTs in the LSM and thus can be placed 485 # in L6 post-flush. 486 build ext3 487 set d 1 488 ---- 489 490 # We block the flush, so the SSTs should still be in the flushable queue. 491 blockFlush 492 ---- 493 494 ingest ext1 ext2 ext3 495 ---- 496 497 # Add another write which should go to the new mutable memtable. 498 batch 499 set a 3 500 ---- 501 502 allowFlush 503 ---- 504 505 lsm 506 ---- 507 508 get 509 a 510 b 511 d 512 ---- 513 a:3 514 b:1 515 d:1 516 517 close 518 ---- 519 520 open readOnly 521 ---- 522 523 get 524 a 525 b 526 d 527 ---- 528 a:3 529 b:1 530 d:1 531 532 # Test with StrictFS 533 reset strictMem 534 ---- 535 536 batch 537 set a 1 538 set b 1 539 ---- 540 541 build ext1 542 set a 2 543 set b 2 544 ---- 545 546 blockFlush 547 ---- 548 549 ingest ext1 550 ---- 551 552 get 553 a 554 b 555 ---- 556 a:2 557 b:2 558 559 ignoreSyncs true 560 ---- 561 562 lsm 563 ---- 564 565 allowFlush 566 ---- 567 568 flush 569 ---- 570 571 # The ingested file is placed above the sstable generated by memtable flush. The 572 # ingested file has a lower file number, but a higher sequence number as 573 # expected. 574 lsm 575 ---- 576 0.1: 577 000004:[a#12,SET-b#12,SET] 578 0.0: 579 000007:[a#10,SET-b#11,SET] 580 581 ls 582 ---- 583 000002.log 584 000004.sst 585 000005.log 586 000006.log 587 000007.sst 588 CURRENT 589 LOCK 590 MANIFEST-000001 591 OPTIONS-000003 592 ext 593 marker.format-version.000015.016 594 marker.manifest.000001.MANIFEST-000001 595 596 close 597 ---- 598 599 # At this point, the changes to the manifest should be lost. Note that 7.sst 600 # is gone because that file was never synced. 601 resetToSynced 602 ---- 603 000002.log 604 000004.sst 605 000005.log 606 000006.log 607 CURRENT 608 LOCK 609 MANIFEST-000001 610 OPTIONS-000003 611 ext 612 ext1 613 marker.format-version.000015.016 614 marker.manifest.000001.MANIFEST-000001 615 616 ignoreSyncs false 617 ---- 618 619 open 620 ---- 621 622 lsm 623 ---- 624 0.1: 625 000004:[a#12,SET-b#12,SET] 626 0.0: 627 000007:[a#10,SET-b#11,SET]