github.com/cockroachdb/pebble@v1.1.2/testdata/iter_histories/iter_optimizations (about) 1 # Test repeated seeks into the same range key, while TrySeekUsingNext=true. 2 # Test for regression fixed in #1849. 3 4 reset 5 ---- 6 7 batch commit 8 range-key-set a c @5 boop 9 range-key-set c e @5 beep 10 ---- 11 committed 2 keys 12 13 combined-iter 14 seek-ge a 15 seek-ge b 16 ---- 17 a: (., [a-c) @5=boop UPDATED) 18 b: (., [a-c) @5=boop) 19 20 # Ensure that no-op optimizations do not reuse range key iterator state across 21 # SetOptions calls. No-op optimizations have the potential to fail to update 22 # RangeKeyChanged(). 23 24 reset 25 ---- 26 27 batch commit 28 range-key-set p s @1 foo 29 ---- 30 committed 1 keys 31 32 combined-iter lower=n@9 upper=x@5 33 seek-lt y@3 34 set-options lower=n@9 upper=x@5 35 seek-lt-limit t o 36 ---- 37 p: (., [p-s) @1=foo UPDATED) 38 . 39 p: valid (., [p-s) @1=foo UPDATED) 40 41 combined-iter lower=n@9 upper=x@5 42 seek-ge o 43 set-options lower=n@9 upper=x@5 44 seek-ge oat 45 ---- 46 p: (., [p-s) @1=foo UPDATED) 47 . 48 p: (., [p-s) @1=foo UPDATED) 49 50 combined-iter lower=n@9 upper=x@5 51 seek-prefix-ge p@5 52 set-options lower=n@9 upper=x@5 53 seek-prefix-ge p 54 ---- 55 p@5: (., [p-"p\x00") @1=foo UPDATED) 56 . 57 p: (., [p-"p\x00") @1=foo UPDATED) 58 59 # Regression test for #1963 / cockroachdb/cockroach#88296. 60 # 61 # The iterators in this test move their bounds monotonically forward 62 # [a,b)→[b,e). This enables the sstable iterator optimization for monotonically 63 # moving bounds (see boundsCmp in sstable/reader.go). With this optimization, 64 # the first seek after the SetBounds may use the fact that the bounds moved 65 # forward monotonically to avoid re-seeking within the index. 66 # 67 # The test cases below exercise a seek to a key, followed by a seek to a smaller 68 # key. The second seek should not make use of the bounds optimization because 69 # doing so may incorrectly skip all keys between the lower bound and the first 70 # seek key. Previously, the code paths that handled block-property filtering on 71 # a two-level iterator could leave the iterator in a state such that the second 72 # seek would improperly also exercise the monotonic bounds optimization. In the 73 # test cases below, this would result in the key 'b' not being found. Each test 74 # case exercises a different combination of seek-ge and seek-prefix-ge. 75 76 reset block-size=1 index-block-size=1 77 ---- 78 79 batch commit 80 set a a 81 set b b 82 set b@4 b@4 83 set z@6 z@6 84 ---- 85 committed 4 keys 86 87 flush 88 ---- 89 90 combined-iter lower=a upper=b point-key-filter=(1,4) 91 seek-ge a 92 set-bounds lower=b upper=e 93 seek-prefix-ge d@5 94 seek-prefix-ge b 95 ---- 96 a: (a, .) 97 . 98 . 99 b: (b, .) 100 101 combined-iter lower=a upper=b point-key-filter=(1,4) 102 seek-ge a 103 set-bounds lower=b upper=e 104 seek-ge d@5 105 seek-prefix-ge b 106 ---- 107 a: (a, .) 108 . 109 . 110 b: (b, .) 111 112 combined-iter lower=a upper=b point-key-filter=(1,4) 113 seek-ge a 114 set-bounds lower=b upper=e 115 seek-ge d@5 116 seek-ge b 117 ---- 118 a: (a, .) 119 . 120 . 121 b: (b, .) 122 123 combined-iter lower=a upper=b point-key-filter=(1,4) 124 seek-ge a 125 set-bounds lower=b upper=e 126 seek-prefix-ge d@5 127 seek-ge b 128 ---- 129 a: (a, .) 130 . 131 . 132 b: (b, .) 133 134 # Test a similar case with range key masking. The previous bug did not apply to 135 # this case, because range-key masking never skips blocks on a seek. 136 137 reset block-size=1 index-block-size=1 138 ---- 139 140 batch commit 141 set a a 142 set b b 143 set b@4 b@4 144 set z@6 z@6 145 range-key-set a z @9 v 146 ---- 147 committed 5 keys 148 149 flush 150 ---- 151 152 combined-iter lower=a upper=b mask-suffix=@10 mask-filter 153 seek-ge a 154 set-bounds lower=b upper=e 155 seek-prefix-ge d@5 156 seek-ge b 157 ---- 158 a: (a, [a-b) @9=v UPDATED) 159 . 160 d@5: (., [d-"d\x00") @9=v UPDATED) 161 b: (b, [b-e) @9=v UPDATED) 162 163 # Test TrySeekUsingNext across no-op SetOptions when reading through an indexed 164 # batch with modifications. The seek-prefix-ges after the first should make use 165 # of the TrySeekUsingNext optimization. 166 # 167 # TODO(jackson): The iterator stats don't signal the use of try-seek-using-next, 168 # so we inspect lastPositioningOp as a proxy since that's the 169 # try-seek-using-next prerequisite that previously regressed. Is there a way to 170 # adapt to this test so that the absence of the try-seek-using-next optimization 171 # is visible in the iterator statistics? 172 # 173 # Regression test for cockroachdb/cockroach#88819. 174 175 reset 176 ---- 177 178 batch commit 179 set b@5 b@5 180 set c@3 c@3 181 set d@9 d@9 182 set e@8 e@8 183 set f@8 f@8 184 ---- 185 committed 5 keys 186 187 flush 188 ---- 189 190 batch name=foo 191 set g@4 g@4 192 ---- 193 wrote 1 keys to batch "foo" 194 195 combined-iter reader=foo name=fooiter 196 inspect lastPositioningOp 197 seek-prefix-ge b@10 198 stats 199 ---- 200 lastPositioningOp="unknown" 201 b@5: (b@5, .) 202 stats: seeked 1 times (1 internal); stepped 0 times (0 internal); blocks: 0B cached, 119B not cached (read time: 0s); points: 1 (3B keys, 3B values) 203 204 mutate batch=foo 205 set h@2 h@2 206 ---- 207 208 iter iter=fooiter 209 set-options 210 inspect lastPositioningOp 211 seek-prefix-ge c@10 212 stats 213 ---- 214 . 215 lastPositioningOp="seekprefixge" 216 c@3: (c@3, .) 217 stats: seeked 2 times (2 internal); stepped 0 times (0 internal); blocks: 0B cached, 119B not cached (read time: 0s); points: 2 (6B keys, 6B values) 218 219 mutate batch=foo 220 set i@1 i@1 221 ---- 222 223 iter iter=fooiter 224 set-options 225 inspect lastPositioningOp 226 seek-prefix-ge d@10 227 stats 228 ---- 229 . 230 lastPositioningOp="seekprefixge" 231 d@9: (d@9, .) 232 stats: seeked 3 times (3 internal); stepped 0 times (0 internal); blocks: 0B cached, 119B not cached (read time: 0s); points: 3 (9B keys, 9B values) 233 234 mutate batch=foo 235 set j@6 j@6 236 ---- 237 238 iter iter=fooiter 239 set-options 240 inspect lastPositioningOp 241 seek-prefix-ge e@10 242 stats 243 ---- 244 . 245 lastPositioningOp="seekprefixge" 246 e@8: (e@8, .) 247 stats: seeked 4 times (4 internal); stepped 0 times (0 internal); blocks: 0B cached, 119B not cached (read time: 0s); points: 4 (12B keys, 12B values) 248 249 # Ensure that a case eligible for TrySeekUsingNext across a SetOptions correctly 250 # sees new batch mutations. The batch iterator should ignore the 251 # TrySeekUsingNext designation. 252 253 reset 254 ---- 255 256 batch commit 257 set b@3 b@3 258 set c@3 c@3 259 ---- 260 committed 2 keys 261 262 batch name=b1 263 ---- 264 wrote 0 keys to batch "b1" 265 266 combined-iter name=i1 reader=b1 267 seek-prefix-ge b@6 268 ---- 269 b@3: (b@3, .) 270 271 mutate batch=b1 272 set b@4 b@4 273 ---- 274 275 iter iter=i1 276 set-options 277 inspect lastPositioningOp 278 seek-prefix-ge b@5 279 ---- 280 . 281 lastPositioningOp="seekprefixge" 282 b@4: (b@4, .) 283 284 # Similar case with SeekGE. 285 286 iter iter=i1 287 seek-ge b@2 288 ---- 289 c@3: (c@3, .) 290 291 mutate batch=b1 292 set c@9 c@9 293 ---- 294 295 iter iter=i1 296 set-options 297 inspect lastPositioningOp 298 seek-ge b@1 299 ---- 300 . 301 lastPositioningOp="seekge" 302 c@9: (c@9, .) 303 304 # Test a case similar to the above, but with an intermediate switch to 305 # range-key-only iteration, so that the batchIter is not re-seeked. 306 307 reset 308 ---- 309 310 batch commit 311 set b@5 b@5 312 set c@3 c@3 313 ---- 314 committed 2 keys 315 316 batch name=b1 317 ---- 318 wrote 0 keys to batch "b1" 319 320 combined-iter name=i1 reader=b1 321 seek-ge b@9 322 ---- 323 b@5: (b@5, .) 324 325 mutate batch=b1 326 set b@6 b@6 327 ---- 328 329 iter iter=i1 330 set-options key-types=range 331 seek-ge b@8 332 set-options key-types=both 333 inspect lastPositioningOp 334 seek-ge b@7 335 ---- 336 . 337 . 338 . 339 lastPositioningOp="invalidate" 340 b@6: (b@6, .) 341 342 reset 343 ---- 344 345 batch commit 346 set b@2 b@2 347 set c@3 c@3 348 ---- 349 committed 2 keys 350 351 batch name=b1 352 ---- 353 wrote 0 keys to batch "b1" 354 355 combined-iter name=i1 reader=b1 356 seek-prefix-ge b@1 357 ---- 358 . 359 360 mutate batch=b1 361 set c@4 c@4 362 ---- 363 364 iter iter=i1 365 set-options 366 inspect lastPositioningOp 367 seek-prefix-ge c@8 368 ---- 369 . 370 lastPositioningOp="seekprefixge" 371 c@4: (c@4, .) 372 373 # Regression test for #2084. 374 # 375 # The optimization added in #2058 began using an enabled TrySeekUsingNext flag 376 # to avoid re-seeking within a level's file metadata. This optimization was 377 # dependent on the invariant that the iterator remained positioned at the 378 # previous seek key, so that a subsequent seek to a larger key does not need to 379 # backtrack. 380 # 381 # This invariant wasn't strictly preserved by the levelIter during SeekPrefixGE 382 # calls. During a SeekPrefixGE, the sstable iterator may return nil despite the 383 # existence of sstable keys greater than the seek key if the sstable's bloom 384 # filter excludes the seek prefix. If the sstable DOES NOT contain any range 385 # tombstones, the levelIter does not advance to the next file if the file's 386 # largest bound has a prefix larger than the seek prefix, returning nil, else it 387 # does advance since the next file could contain the seek prefix. 388 # 389 # However, if the file DOES contain range tombstones, the levelIter returns a 390 # synthetic largest boundary key so that the file remains open until the merging 391 # iterator passes beyond its bounds. This ensures the file's range deletions' 392 # effects on other keys are observed. If another level returned a key greater 393 # than this largest boundary key (eg, because the other level doesn't restrict 394 # results to the seek prefix), the merging iterator could step beyond the 395 # level's synthetic boundary key. This step could advance the levelIter to the 396 # next file, despite its irrelevance to the current prefix. This step would also 397 # break the invariant that the level iterator remained positioned at the seek 398 # key. 399 # 400 # The bug was fixed by comparing the synthetic boundary key to the seek prefix, 401 # avoiding ever Next-ing the level iterator beyond the seek prefix. 402 403 # Set 100 bloom-filter bits per key to ensure the bloom-filter exclusivity 404 # checks successfully exclude prefixes that aren't present. 405 reset bloom-bits-per-key=100 406 ---- 407 408 # [a -d) 409 # b@3 d@1 410 batch commit 411 del-range a d 412 set b@3 b@3 413 set d@1 d@1 414 ---- 415 committed 3 keys 416 417 flush 418 ---- 419 420 # c@0 e@0 421 batch commit 422 del c@0 423 set e@0 e@0 424 ---- 425 committed 2 keys 426 427 flush 428 ---- 429 430 lsm 431 ---- 432 0.1: 433 000007:[c@0#13,DEL-e@0#14,SET] 434 0.0: 435 000005:[a#10,RANGEDEL-d@1#12,SET] 436 437 # The first SeekPrefixGE(b@3) positions each level iterator over their 438 # respective files and correctly finds b@3. 439 # 440 # The second SeekPrefixGE(c@5) seeks in both files. The 0.0 level iterator finds 441 # that its file does not contain the prefix 'c', so it returns nil. Since the file 442 # contains a range deletion, it returns a synthetic boundary key with user key 443 # d@1 to ensure the file stays open until the iterator has moved beyond the 444 # file's bounds. The seek in level 0.1 finds a key with the prefix 'c': a point 445 # tombstone c@0#4,DEL. This gets bubbled up to the Iterator, which skips it 446 # because it's a point tombstone, nexting within 000007 to e@0#5. 447 # 448 # Previously, in the bug highlighted by #2084, the merging iterator would then 449 # see that level 0.0's synthetic boundary key at d@1 was at the top of the heap 450 # and move to the next file in 0.0. The subsequent call to SeekPrefixGE(d@1, 451 # TrySeekUsingNext=true) would incorrectly use the current position within the 452 # 0.0 file metadata (nil), and miss the d@1 key. 453 454 combined-iter 455 seek-prefix-ge b@3 456 seek-prefix-ge c@5 457 seek-prefix-ge d@1 458 ---- 459 b@3: (b@3, .) 460 . 461 d@1: (d@1, .) 462 463 464 # Test an instance where unequal application of TrySeekUsingNext optimizations 465 # among a merging iterator's levels can result in surfacing deleted keys. 466 # Regression test for #2101. 467 468 reset 469 ---- 470 471 batch commit 472 set b b 473 ---- 474 committed 1 keys 475 476 flush 477 ---- 478 479 compact a-h 480 ---- 481 6: 482 000005:[b#10,SET-b#10,SET] 483 484 batch commit 485 set g g 486 ---- 487 committed 1 keys 488 489 flush 490 ---- 491 492 compact a-h 493 ---- 494 6: 495 000005:[b#10,SET-b#10,SET] 496 000007:[g#11,SET-g#11,SET] 497 498 batch commit 499 del-range b d 500 ---- 501 committed 1 keys 502 503 flush 504 ---- 505 506 batch commit 507 set e e 508 ---- 509 committed 1 keys 510 511 flush 512 ---- 513 514 lsm 515 ---- 516 0.0: 517 000009:[b#12,RANGEDEL-d#inf,RANGEDEL] 518 000011:[e#13,SET-e#13,SET] 519 6: 520 000005:[b#10,SET-b#10,SET] 521 000007:[g#11,SET-g#11,SET] 522 523 # The `seek-ge b` could incorrectly return `b` if the level 0.0 levelIter obeys 524 # the TrySeekUsingNext optimization but the level 6 levelIter does not. The 525 # TrySeekUsingNext optimization must be applied equally across all the levels of 526 # a merging iterator. 527 528 combined-iter 529 seek-ge a 530 seek-ge b 531 ---- 532 e: (e, .) 533 e: (e, .) 534 535 # Regression test for #2118, where a MERGE pushes child iterators to the next 536 # key, and possibly past a file that contained a range tombstone that we 537 # should have paused at in a SeekPrefixGE, affecting future TrySeekUsingNexts. 538 # This test constructs this example (suffixes ignored), where square brackets 539 # consist of one SST: 540 # 541 # L0: [(b, MERGE) (c-d, RANGEDEL)] [(m, DEL)] 542 # L6: [(c, SET) (c-e, RANGEKEYSET)] [(j, SET)] 543 # 544 # We create an iterator with L6 filters enabled and create relatively large 545 # bloom filter blocks to reduce the false positive rate. Then we SeekPrefixGE(b) 546 # and end up with the L0 levelIter landing on the (b, MERGE), and the L6 iterator 547 # is exhausted as no SST filter blocks match the prefix. The top-level iterator 548 # then Next()s to find the next internal key at b if there is any, we land 549 # on the pause key at (d, RANGEDELSENTINEL). Crucially since there are no 550 # more items in the mergingIter heap and the merging iter is set to elide 551 # range tombstones, we Next() the level iter again as part of the same top-level 552 # iterator Next(), and land on (m, DEL). The type of the key here doesn't really 553 # matter. 554 # 555 # We then do a SeekPrefixGE(c), and since c > b, in the buggy scenario we 556 # TrySeekUsingNext. The bottom levelIter correctly finds the sstable containing 557 # the set, but the upper levelIter is already past the sstable containing the 558 # rangedel, so it just returns (m, DEL) again, and we surface the (c, SET) that 559 # should have been deleted. 560 561 reset bloom-bits-per-key=100 562 ---- 563 564 batch commit 565 set c@2 foo 566 range-key-set c e @5 bar 567 ---- 568 committed 2 keys 569 570 flush 571 ---- 572 573 compact a-z 574 ---- 575 6: 576 000005:[c#11,RANGEKEYSET-e#inf,RANGEKEYSET] 577 578 batch commit 579 set j k 580 ---- 581 committed 1 keys 582 583 flush 584 ---- 585 586 compact a-z 587 ---- 588 6: 589 000005:[c#11,RANGEKEYSET-e#inf,RANGEKEYSET] 590 000007:[j#12,SET-j#12,SET] 591 592 batch commit 593 del-range c@2 d 594 merge b@2 g 595 ---- 596 committed 2 keys 597 598 flush 599 ---- 600 601 batch commit 602 del m 603 ---- 604 committed 1 keys 605 606 flush 607 ---- 608 609 lsm 610 ---- 611 0.0: 612 000009:[b@2#14,MERGE-d#inf,RANGEDEL] 613 000011:[m#15,DEL-m#15,DEL] 614 6: 615 000005:[c#11,RANGEKEYSET-e#inf,RANGEKEYSET] 616 000007:[j#12,SET-j#12,SET] 617 618 combined-iter upper=z@3 mask-suffix=@3 mask-filter use-l6-filter 619 seek-prefix-ge b@2 620 seek-prefix-ge c@2 621 ---- 622 b@2: (g, .) 623 c@2: (., [c-"c\x00") @5=bar UPDATED) 624 625 # Regression test for Cockroachdb#92205. This test constructs this scenario: 626 # 627 # A DEL in a middle level (L0.0) that we SeekPrefixGE directly for. Note that 628 # this DEL is not deleted by any range deletes; it gets exposed to the 629 # Iterator. There's a key after this DEL in the L0.0 levelIter, and there's a 630 # level above it (L0.1) that has a rangedel deleting that key, but not the DEL 631 # we SeekPrefixGE for. In the lowest level, there's a SET at L6 that is to the 632 # right of the DEL in L0.0, but is also not deleted by the RANGEDEL in L0.1. 633 # Our second SeekPrefixGE will be for this SET. Visualization, where square 634 # brackets are files: 635 # 636 # L0.1 [dd-ee#RANGEDEL] 637 # L0.0 [b#DEL e#SET] 638 # L6 [d#SET] [f#SET g#SET] 639 # 640 # When the Iterator encounters the above DEL internal key in the SeekPrefixGE, it 641 # calls Iterator.nextUserKey in the Iterator.findNextEntry call that was part of the 642 # SeekPrefixGE call. While Iterator.findNextEntry has a conditional to exit 643 # out of the loop if we're in prefix iteration and have gone past the prefix, 644 # this break only happens _after_ nextUserKey() has run. As a result we Next() 645 # the levelIter in L0.0, land on e#SET, and the mergingIter realizes that it 646 # is deleted by the rangedel in a higher level (L0.1). The mergingIter does not 647 # see d#SET because that sstable was excluded by the bloom filter. We then do a relative 648 # seek of all levels below L0.1 to ee (the end key of the rangedel), and in that 649 # process we advance the L6 levelIter to the second file. 650 # 651 # When we do the second SeekPrefixGE for d, the outer Iterator thinks d > b and 652 # so TrySeekUsingNext can work. However, the L6 levelIter has already advanced 653 # past the file containing d#SET, so we don't surface it even though we should 654 # have. 655 656 reset bloom-bits-per-key=100 657 ---- 658 659 batch commit 660 set d@4 foo 661 ---- 662 committed 1 keys 663 664 flush 665 ---- 666 667 compact a-f 668 ---- 669 6: 670 000005:[d@4#10,SET-d@4#10,SET] 671 672 batch commit 673 set f@5 bar 674 set g@5 baz 675 ---- 676 committed 2 keys 677 678 flush 679 ---- 680 681 compact e-k 682 ---- 683 6: 684 000005:[d@4#10,SET-d@4#10,SET] 685 000007:[f@5#11,SET-g@5#12,SET] 686 687 batch commit 688 del b@5 689 set e@4 foobar 690 ---- 691 committed 2 keys 692 693 flush 694 ---- 695 696 batch commit 697 del-range dd ee 698 ---- 699 committed 1 keys 700 701 flush 702 ---- 703 704 lsm 705 ---- 706 0.1: 707 000011:[dd#15,RANGEDEL-ee#inf,RANGEDEL] 708 0.0: 709 000009:[b@5#13,DEL-e@4#14,SET] 710 6: 711 000005:[d@4#10,SET-d@4#10,SET] 712 000007:[f@5#11,SET-g@5#12,SET] 713 714 combined-iter upper=z@3 use-l6-filter 715 seek-prefix-ge b@6 716 seek-prefix-ge d@5 717 ---- 718 . 719 d@4: (foo, .)