From ba157a10fd071e2c1f50772c71fac464c7c40ea8 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Sat, 17 Feb 2024 11:16:15 +0100 Subject: [PATCH 1/4] Add a long test that could find errors in index analysis Add a long test that could find errors in index analysis Update test --- .../queries/0_stateless/02995_index.reference | 125 ++++++++++++++++++ tests/queries/0_stateless/02995_index.sh | 44 ++++++ 2 files changed, 169 insertions(+) create mode 100644 tests/queries/0_stateless/02995_index.reference create mode 100755 tests/queries/0_stateless/02995_index.sh diff --git a/tests/queries/0_stateless/02995_index.reference b/tests/queries/0_stateless/02995_index.reference new file mode 100644 index 00000000000..971ce526db1 --- /dev/null +++ b/tests/queries/0_stateless/02995_index.reference @@ -0,0 +1,125 @@ +12 4 21722 2209341 4 1415 2333 4 61 64 3 +21 1134 11363 58484 1106 1458 1592 136 26 62 32 +22 192 4504 5728 179 291 767 120 47 54 8 +26 196 1327684 5221 195 4140 5661 161 28 49 19 +28 5 2034378 7102 5 325 3255 2 53 60 4 +29 53 45041 45189 45 1580 211 31 55 84 18 +38 424 1600675 4653 424 562 5944 244 60 65 6 +45 17 62743 674873 17 6239 6494 17 65 76 8 +72 1751 1210073 6200 1606 2498 528 859 51 61 11 +79 2 2255228 2255293 2 5495 7057 2 65 65 1 +85 459 1051571 1829638 459 6402 7131 334 32 61 25 +86 10 1748130 1754217 10 4374 7003 10 56 59 4 +91 124 5718 5802 68 282 7113 100 41 63 19 +94 20 1231916 2050003 20 4802 4917 19 53 59 7 +99 2 3665 36667 2 497 697 2 70 71 2 +103 1 2446615 2446615 1 2498 2498 1 58 58 1 +106 33 6149 6699 32 547 826 21 61 61 1 +111 43 2273186 5272 43 492 4923 4 54 72 15 +120 3129 45117 6735 2868 1030 1625 561 59 64 6 +138 2 49243 49374 2 1428 1519 2 47 48 2 +143 100 23321 63639 100 1115 1624 88 51 51 1 +145 1 2447976 2447976 1 6173 6173 1 44 44 1 +153 16 13748 16881 16 1506 1636 16 54 68 9 +159 16457 1525336 7131 12789 1280 6163 2668 24 66 39 +171 5 15042 16698 5 1302 1608 5 65 65 1 +179 6264 1362341 2686 6244 2554 7132 2705 61 67 7 +192 1 1639623 1639623 1 3406 3406 1 32 32 1 +193 1 1429969 1429969 1 7131 7131 1 45 45 1 +207 12 23057 32500 12 1491 1726 12 32 46 7 +221 4060 1366870 6648 3185 4527 5226 687 28 69 36 +228 73 12281 17929 71 1328 2034 63 49 71 18 +229 2 1617478 1723791 2 4590 5578 2 41 42 2 +230 3873 1332729 6949 3649 1330 4703 845 62 65 4 +238 25 2624456 2625673 24 2535 6465 25 58 75 14 +241 154 2554929 2616444 154 2626 7131 148 34 57 17 +248 276 15529 30987 274 1040 1222 136 37 79 27 +254 3018 33966 6635 2837 1057 1622 539 24 60 33 +255 20 1581774 1811334 20 6068 6301 18 33 57 10 +256 5 5145 6841 5 367 376 5 58 58 1 +270 2 2195579 2262119 2 7102 7123 2 33 34 2 +281 32 2379460 616215 32 6042 6086 23 53 64 12 +282 7 1292651 24244 7 1607 2455 6 46 55 5 +286 123 1521935 5269 123 3793 3940 81 40 66 22 +291 21 2419080 3567 21 297 4731 21 54 55 2 +316 4 5221 5616 4 505 558 4 32 35 3 +319 232 56480 63033 230 1599 313 50 33 64 26 +327 15 51647 51894 14 1292 1585 14 47 57 7 +332 24 23484 54948 24 1609 1726 16 32 49 11 +333 1 14189 14189 1 1550 1550 1 63 63 1 +342 49 2579220 2622432 49 4626 6933 48 34 54 14 +346 1979 53016 6735 1818 1334 174 294 26 62 32 +358 45 59058 60844 44 6746 722 40 57 84 15 +363 1198 1260033 2568811 1196 5710 5790 82 55 80 26 +384 150 2361175 476024 150 7008 7123 81 38 64 22 +387 216 5200 6553 199 243 521 120 65 65 1 +392 1877 1607428 2030850 1875 1416 7131 1379 54 66 13 +396 7932 1380803 6186 7769 545 798 1743 24 67 39 +398 3 5183 5213 2 291 352 3 53 59 3 +399 62 51494 59203 61 7073 754 42 55 78 18 +412 2141 1360120 2189792 2136 2491 5658 1371 71 75 5 +413 2 2036037 2064917 2 3963 4666 2 43 45 2 +431 33 2302331 2348449 33 4425 6516 32 69 69 1 +447 59 25125 33094 59 1176 1817 56 53 58 6 +456 1 53157 53157 1 1556 1556 1 26 26 1 +462 5 5456 6280 5 348 4337 5 28 40 5 +472 1 1443716 1443716 1 6122 6122 1 42 42 1 +491 34 1066102 1183673 34 6606 6822 32 46 67 15 +498 896 2230163 3054 895 537 7131 714 24 59 28 +504 108 12281 25180 108 1318 1784 94 55 66 12 +515 22 1588883 2640809 22 6554 6571 15 46 59 12 +518 1 37743 37743 1 1558 1558 1 72 72 1 +530 1 3033 3033 1 561 561 1 59 59 1 +532 21 5721 6355 20 549 665 11 44 50 7 +546 156 2577874 48517 156 1105 324 133 44 51 8 +554 12 1665194 2640066 12 1817 2951 12 57 57 1 +564 3865 2028049 2083433 3722 1115 985 2203 44 84 41 +566 4430 50605 57509 3217 1191 267 459 26 72 39 +567 7 5221 5893 7 333 558 7 32 35 3 +582 1172 1320619 2019743 1172 5819 7131 757 26 63 30 +584 43100 2500 5594 22561 134 4573 1660 48 84 37 +589 18 6046 6068 16 345 564 17 55 62 7 +595 139 1585165 1683606 138 2231 3598 132 54 84 28 +615 3 1056081 1116230 3 5794 5796 2 59 62 3 +619 7 1543114 5241 7 2442 3105 7 41 45 3 +634 2720 1221058 4999 2685 2426 7131 1735 54 60 7 +635 237 2119333 4667 237 561 5999 176 49 60 12 +644 5 1774169 2056171 5 5591 6091 4 33 39 3 +647 8 51632 64403 8 1457 1624 8 26 34 5 +651 1247 1620565 6280 1239 528 792 814 62 63 2 +665 13 4598 4789 13 511 558 11 39 46 7 +679 1560 1613200 25940 1552 1569 3118 781 49 84 35 +704 2 14226 15594 2 1086 1116 2 65 71 2 +715 25 1199352 3490 25 5036 5112 23 34 55 13 +716 1241 61989 6735 1043 1203 1625 395 52 65 14 +730 1819 5560 6170 614 2421 627 293 56 69 14 +736 8 1433153 4941 8 339 4594 8 28 36 5 +749 2 1326176 1339862 2 4339 6213 2 49 50 2 +753 1 53157 53157 1 1556 1556 1 26 26 1 +761 63 1443230 6881 63 3154 3204 26 56 73 14 +762 49 1449596 1968154 49 2437 3753 48 54 62 9 +775 14448 5330 769436 2454 447 6607 655 70 81 12 +789 1 1552458 1552458 1 2441 2441 1 62 62 1 +794 81 5588 6569 80 498 929 41 26 50 15 +839 9 29223 46530 9 1336 1465 9 52 52 1 +844 5 2377545 2377635 5 5129 6321 5 53 69 5 +846 50 2172273 2589295 50 1582 3053 48 64 68 5 +847 2034 56656 63658 1501 1444 838 472 26 63 32 +861 902 5570 6904 643 457 489 37 33 70 34 +873 2360 1519811 50487 2248 1310 1784 316 60 68 9 +879 135 6704 6785 70 279 505 95 35 66 23 +889 5130 2070007 39692 5040 1151 6791 2606 44 66 23 +896 4 511246 859452 4 6554 6561 4 67 71 4 +912 146 1322641 2238040 146 1366 6354 143 59 59 1 +913 46 5495 6867 44 350 565 39 26 43 12 +921 763 1580790 416881 763 6191 7131 509 63 64 2 +925 318 2500952 5025 309 476 6114 182 32 56 21 +931 12 4277 4809 12 238 256 9 63 83 9 +942 954 1331 2228193 952 1121 5047 788 65 70 6 +948 14 1785593 2600431 14 6550 6598 13 34 49 9 +956 5 5755 6023 5 359 411 5 43 48 4 +963 4 3812 3835 4 444 537 4 47 53 4 +978 5 51632 58212 5 1127 1556 5 24 32 5 +980 53 47201 59744 53 1537 1625 36 41 49 9 +987 4593 2020131 763444 3954 256 792 1832 60 64 5 +993 4 1615159 1718339 4 1570 3093 4 62 63 2 diff --git a/tests/queries/0_stateless/02995_index.sh b/tests/queries/0_stateless/02995_index.sh new file mode 100755 index 00000000000..5125d03904e --- /dev/null +++ b/tests/queries/0_stateless/02995_index.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Tags: long, no-debug, no-asan, no-tsan, no-msan, no-ubsan, no-sanitize-coverage + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery " + +DROP TABLE IF EXISTS test; +CREATE TABLE test (a String, b String, c String) ENGINE = MergeTree ORDER BY (a, b, c) SETTINGS index_granularity = 11; + +INSERT INTO test +SELECT round(pow(sipHash64(1, number), 1/4)), round(pow(sipHash64(2, number), 1/6)), round(pow(sipHash64(3, number), 1/10)) +FROM numbers(100000); + +INSERT INTO test +SELECT round(pow(sipHash64(1, number), 1/3)), round(pow(sipHash64(2, number), 1/5)), round(pow(sipHash64(3, number), 1/10)) +FROM numbers(100000); + +INSERT INTO test +SELECT round(pow(sipHash64(1, number), 1/5)), round(pow(sipHash64(2, number), 1/7)), round(pow(sipHash64(3, number), 1/10)) +FROM numbers(100000); + +DETACH TABLE test; +ATTACH TABLE test; +" + +for i in {1..1000} +do + echo " +WITH ${i} AS try +SELECT try, count(), min(a), max(a), uniqExact(a), min(b), max(b), uniqExact(b), min(c), max(c), uniqExact(c) FROM test +WHERE a >= (round(pow(sipHash64(1, try), 1 / (3 + sipHash64(2, try) % 8))) AS a1)::String + AND a <= (a1 + round(pow(sipHash64(3, try), 1 / (3 + sipHash64(4, try) % 8))))::String + AND b >= (round(pow(sipHash64(5, try), 1 / (3 + sipHash64(6, try) % 8))) AS b1)::String + AND b <= (b1 + round(pow(sipHash64(7, try), 1 / (3 + sipHash64(8, try) % 8))))::String + AND c >= (round(pow(sipHash64(9, try), 1 / (3 + sipHash64(10, try) % 8))) AS c1)::String + AND c <= (c1 + round(pow(sipHash64(11, try), 1 / (3 + sipHash64(12, try) % 8))))::String +HAVING count() > 0; +" +done | ${CLICKHOUSE_CLIENT} --multiquery + +${CLICKHOUSE_CLIENT} --multiquery "DROP TABLE test" From 865ae80e7c194bfe257f755334ecc751c7cc1c17 Mon Sep 17 00:00:00 2001 From: Michael Kolupaev Date: Mon, 19 Feb 2024 13:29:54 +0000 Subject: [PATCH 2/4] Counterexample --- .../02996_index_compaction_counterexample.reference | 2 ++ .../0_stateless/02996_index_compaction_counterexample.sql | 6 ++++++ 2 files changed, 8 insertions(+) create mode 100644 tests/queries/0_stateless/02996_index_compaction_counterexample.reference create mode 100644 tests/queries/0_stateless/02996_index_compaction_counterexample.sql diff --git a/tests/queries/0_stateless/02996_index_compaction_counterexample.reference b/tests/queries/0_stateless/02996_index_compaction_counterexample.reference new file mode 100644 index 00000000000..6ed281c757a --- /dev/null +++ b/tests/queries/0_stateless/02996_index_compaction_counterexample.reference @@ -0,0 +1,2 @@ +1 +1 diff --git a/tests/queries/0_stateless/02996_index_compaction_counterexample.sql b/tests/queries/0_stateless/02996_index_compaction_counterexample.sql new file mode 100644 index 00000000000..fd01fbf0dc4 --- /dev/null +++ b/tests/queries/0_stateless/02996_index_compaction_counterexample.sql @@ -0,0 +1,6 @@ +create table b (x Int64, y String) engine MergeTree order by (x, y) settings index_granularity=2; +insert into b values (0, 'a'), (1, 'b'), (1, 'c'); +select count() from b where x = 1 and y = 'b'; +detach table b; +attach table b; +select count() from b where x = 1 and y = 'b'; From 0c113529f965d398808d5e2c756c6e67b6910bee Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Wed, 21 Feb 2024 22:51:26 +0100 Subject: [PATCH 3/4] Update 02996_index_compaction_counterexample.sql --- .../0_stateless/02996_index_compaction_counterexample.sql | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/queries/0_stateless/02996_index_compaction_counterexample.sql b/tests/queries/0_stateless/02996_index_compaction_counterexample.sql index fd01fbf0dc4..1545f83c79f 100644 --- a/tests/queries/0_stateless/02996_index_compaction_counterexample.sql +++ b/tests/queries/0_stateless/02996_index_compaction_counterexample.sql @@ -1,6 +1,8 @@ +DROP TABLE IF EXISTS b; create table b (x Int64, y String) engine MergeTree order by (x, y) settings index_granularity=2; insert into b values (0, 'a'), (1, 'b'), (1, 'c'); select count() from b where x = 1 and y = 'b'; detach table b; attach table b; select count() from b where x = 1 and y = 'b'; +DROP TABLE b; From 72b8880467012842aa20a0a6d35d3e2ed79157d1 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 22 Feb 2024 14:29:24 +0100 Subject: [PATCH 4/4] Make the reference correct --- .../queries/0_stateless/02995_index.reference | 51 ++++++++++--------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/tests/queries/0_stateless/02995_index.reference b/tests/queries/0_stateless/02995_index.reference index 971ce526db1..1e8639caa88 100644 --- a/tests/queries/0_stateless/02995_index.reference +++ b/tests/queries/0_stateless/02995_index.reference @@ -1,36 +1,36 @@ 12 4 21722 2209341 4 1415 2333 4 61 64 3 21 1134 11363 58484 1106 1458 1592 136 26 62 32 -22 192 4504 5728 179 291 767 120 47 54 8 +22 210 4504 5729 196 291 767 124 47 54 8 26 196 1327684 5221 195 4140 5661 161 28 49 19 28 5 2034378 7102 5 325 3255 2 53 60 4 29 53 45041 45189 45 1580 211 31 55 84 18 38 424 1600675 4653 424 562 5944 244 60 65 6 45 17 62743 674873 17 6239 6494 17 65 76 8 -72 1751 1210073 6200 1606 2498 528 859 51 61 11 +72 1862 1210073 6200 1677 2498 528 859 51 61 11 79 2 2255228 2255293 2 5495 7057 2 65 65 1 85 459 1051571 1829638 459 6402 7131 334 32 61 25 86 10 1748130 1754217 10 4374 7003 10 56 59 4 -91 124 5718 5802 68 282 7113 100 41 63 19 +91 165 5718 5802 75 282 7113 112 41 63 22 94 20 1231916 2050003 20 4802 4917 19 53 59 7 99 2 3665 36667 2 497 697 2 70 71 2 103 1 2446615 2446615 1 2498 2498 1 58 58 1 -106 33 6149 6699 32 547 826 21 61 61 1 +106 72 6149 6699 67 527 826 40 61 61 1 111 43 2273186 5272 43 492 4923 4 54 72 15 120 3129 45117 6735 2868 1030 1625 561 59 64 6 138 2 49243 49374 2 1428 1519 2 47 48 2 143 100 23321 63639 100 1115 1624 88 51 51 1 145 1 2447976 2447976 1 6173 6173 1 44 44 1 153 16 13748 16881 16 1506 1636 16 54 68 9 -159 16457 1525336 7131 12789 1280 6163 2668 24 66 39 +159 19952 1525336 7131 12957 1280 6163 2668 24 66 39 171 5 15042 16698 5 1302 1608 5 65 65 1 179 6264 1362341 2686 6244 2554 7132 2705 61 67 7 192 1 1639623 1639623 1 3406 3406 1 32 32 1 193 1 1429969 1429969 1 7131 7131 1 45 45 1 207 12 23057 32500 12 1491 1726 12 32 46 7 -221 4060 1366870 6648 3185 4527 5226 687 28 69 36 +221 5081 1366870 6649 3432 4527 5226 687 24 69 39 228 73 12281 17929 71 1328 2034 63 49 71 18 229 2 1617478 1723791 2 4590 5578 2 41 42 2 -230 3873 1332729 6949 3649 1330 4703 845 62 65 4 +230 3916 1332729 6949 3668 1330 4703 845 62 65 4 238 25 2624456 2625673 24 2535 6465 25 58 75 14 241 154 2554929 2616444 154 2626 7131 148 34 57 17 248 276 15529 30987 274 1040 1222 136 37 79 27 @@ -48,13 +48,14 @@ 332 24 23484 54948 24 1609 1726 16 32 49 11 333 1 14189 14189 1 1550 1550 1 63 63 1 342 49 2579220 2622432 49 4626 6933 48 34 54 14 -346 1979 53016 6735 1818 1334 174 294 26 62 32 +344 1 6486 6486 1 509 509 1 24 24 1 +346 1987 53016 6735 1823 1334 174 294 26 62 32 358 45 59058 60844 44 6746 722 40 57 84 15 363 1198 1260033 2568811 1196 5710 5790 82 55 80 26 384 150 2361175 476024 150 7008 7123 81 38 64 22 -387 216 5200 6553 199 243 521 120 65 65 1 +387 277 5200 6553 252 243 521 130 65 65 1 392 1877 1607428 2030850 1875 1416 7131 1379 54 66 13 -396 7932 1380803 6186 7769 545 798 1743 24 67 39 +396 8181 1380803 6186 7920 545 798 1743 24 67 39 398 3 5183 5213 2 291 352 3 53 59 3 399 62 51494 59203 61 7073 754 42 55 78 18 412 2141 1360120 2189792 2136 2491 5658 1371 71 75 5 @@ -70,48 +71,48 @@ 515 22 1588883 2640809 22 6554 6571 15 46 59 12 518 1 37743 37743 1 1558 1558 1 72 72 1 530 1 3033 3033 1 561 561 1 59 59 1 -532 21 5721 6355 20 549 665 11 44 50 7 +532 26 5721 6355 25 549 665 14 44 50 7 546 156 2577874 48517 156 1105 324 133 44 51 8 554 12 1665194 2640066 12 1817 2951 12 57 57 1 564 3865 2028049 2083433 3722 1115 985 2203 44 84 41 -566 4430 50605 57509 3217 1191 267 459 26 72 39 -567 7 5221 5893 7 333 558 7 32 35 3 +566 4432 50605 57509 3217 1191 267 459 26 72 39 +567 8 5221 5893 7 333 558 8 27 35 4 582 1172 1320619 2019743 1172 5819 7131 757 26 63 30 584 43100 2500 5594 22561 134 4573 1660 48 84 37 -589 18 6046 6068 16 345 564 17 55 62 7 +589 28 6046 6068 19 345 564 27 55 62 8 595 139 1585165 1683606 138 2231 3598 132 54 84 28 615 3 1056081 1116230 3 5794 5796 2 59 62 3 619 7 1543114 5241 7 2442 3105 7 41 45 3 -634 2720 1221058 4999 2685 2426 7131 1735 54 60 7 +634 2722 1221058 4999 2686 2426 7131 1735 54 60 7 635 237 2119333 4667 237 561 5999 176 49 60 12 644 5 1774169 2056171 5 5591 6091 4 33 39 3 647 8 51632 64403 8 1457 1624 8 26 34 5 -651 1247 1620565 6280 1239 528 792 814 62 63 2 +651 1325 1620565 6281 1301 528 792 815 62 63 2 665 13 4598 4789 13 511 558 11 39 46 7 679 1560 1613200 25940 1552 1569 3118 781 49 84 35 704 2 14226 15594 2 1086 1116 2 65 71 2 715 25 1199352 3490 25 5036 5112 23 34 55 13 -716 1241 61989 6735 1043 1203 1625 395 52 65 14 -730 1819 5560 6170 614 2421 627 293 56 69 14 +716 1253 61989 6735 1050 1203 1625 397 52 65 14 +730 2584 5560 6170 634 2421 627 293 56 69 14 736 8 1433153 4941 8 339 4594 8 28 36 5 749 2 1326176 1339862 2 4339 6213 2 49 50 2 753 1 53157 53157 1 1556 1556 1 26 26 1 761 63 1443230 6881 63 3154 3204 26 56 73 14 762 49 1449596 1968154 49 2437 3753 48 54 62 9 -775 14448 5330 769436 2454 447 6607 655 70 81 12 +775 35107 5330 769436 2471 447 6607 656 70 81 12 789 1 1552458 1552458 1 2441 2441 1 62 62 1 -794 81 5588 6569 80 498 929 41 26 50 15 +794 158 5585 6585 155 495 929 67 24 50 20 839 9 29223 46530 9 1336 1465 9 52 52 1 844 5 2377545 2377635 5 5129 6321 5 53 69 5 846 50 2172273 2589295 50 1582 3053 48 64 68 5 -847 2034 56656 63658 1501 1444 838 472 26 63 32 -861 902 5570 6904 643 457 489 37 33 70 34 +847 2577 56656 63658 1582 1444 838 474 26 63 33 +861 1333 5570 6909 839 457 489 37 33 70 34 873 2360 1519811 50487 2248 1310 1784 316 60 68 9 -879 135 6704 6785 70 279 505 95 35 66 23 +879 228 6704 6785 79 279 507 121 35 66 24 889 5130 2070007 39692 5040 1151 6791 2606 44 66 23 896 4 511246 859452 4 6554 6561 4 67 71 4 912 146 1322641 2238040 146 1366 6354 143 59 59 1 -913 46 5495 6867 44 350 565 39 26 43 12 +913 82 5495 6870 78 350 565 67 24 43 15 921 763 1580790 416881 763 6191 7131 509 63 64 2 925 318 2500952 5025 309 476 6114 182 32 56 21 931 12 4277 4809 12 238 256 9 63 83 9 @@ -121,5 +122,5 @@ 963 4 3812 3835 4 444 537 4 47 53 4 978 5 51632 58212 5 1127 1556 5 24 32 5 980 53 47201 59744 53 1537 1625 36 41 49 9 -987 4593 2020131 763444 3954 256 792 1832 60 64 5 +987 6033 2020131 763444 4306 256 792 1832 60 64 5 993 4 1615159 1718339 4 1570 3093 4 62 63 2