Commit Graph

188 Commits

Author SHA1 Message Date
youennL-cs
6526c2a8ab
[RFC] Replacing merge tree new engine (#41005)
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add keyword 'CLEANUP' when OPTIMIZE

* Cleanup uniquely when it's a replacingMergeTree

* Propagate CLEANUP information and change from 'with_cleanup' to 'cleanup'

* Cleanup data flagged as 'is_deleted'

* Fix merge when optimize and add a test

* Fix OPTIMIZE and INSERT + add tests

* New fix for cleanup at the merge

* Cleanup debug logs

* Add the SETTINGS option 'clean_deleted_rows' that can be 'never' or 'always'

* Fix regression bug; Now REplicatedMergeTree can be called as before without 'is_deleted'

* Add Replicated tests

* Disable tag 'long' for our test and cleanup some white spaces

* Update tests

* Fix tests and remove additional useless whitespace

* Fix replica test

* Style clean && add condition check for is_deleted values

* clean_deleted_rows settings is nom an enum

* Add valid default value to the clean_deleted_rows settings

* Update cleanup checkers to use the enum and fix typos in the test

* Fix submodule contrib/AMQP-CPP pointer

* Add missing messages in test reference and remove a print with non derterministic order

* fix replica test reference

* Fix edge case

* Fix a typo for the spell checker

* Fix reference

* Fix a condition to raise an error if is_deleted differ from 0/1 and cleanup

* Change tests file name and update number

* This should fix the ReplacingMergeTree parameter set

* Fix replicated parameters

* Disable allow_deprecated_syntax_for_merge_tree for our new column

* Fix a test

* Remove non deterministic order print in the test

* Test on replicas

* Remove a condition, when checking optional parameters, that should not be sueful since we disabled the deprected_syntaxe

* Revert "Remove a condition, when checking optional parameters, that should not be useful since we disabled the deprected_syntaxe"

This reverts commit b65d64c05e.

* Fix replica management and limit the number of argument to two maximum, due to the possiblity of deprecated table create/attach failing otherwise

* Test a fix for replicated log information error

* Try to add sync to have consistent results

* Change path of replicas that should cause one issue and add few prints in case it's not that

* Get cleanup info on replicas only if information found

* Fix style issues

* Try to avoid replication error 'cannot select parts...' and and replica read/write field order

* Cleanup according to PR reviews
 and add tests on error raised.

* Update src/Storages/MergeTree/registerStorageMergeTree.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* Select ... FINAL don't show rows with is_deleted = true

* Update and fix SELECT ... FINAL merge parameter

* Remove is_deleted rows only on the version inserted when merge

* Fix (master) updates issues

* Revert changes that should not be commited

* Add changes according to review

* Revert changes that should not be commited - part 2

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-02-16 16:03:16 +03:00
Antonio Andelic
8d16fe5793
Merge branch 'master' into add-support-for-settings-alias 2023-02-13 08:46:00 +01:00
Alexey Milovidov
395d6b6bd5
Merge pull request #46282 from CurtizJ/return-back-vertical-merges
Allow vertical merges from compact to wide parts [2]
2023-02-12 05:51:12 +03:00
Anton Popov
1205932a19 add setting to disable vertical merges from compact parts 2023-02-10 16:33:46 +00:00
kssenii
1cbff88a7c Fix integration test 2023-02-04 19:31:19 +01:00
kssenii
a391de18a1 Merge remote-tracking branch 'upstream/master' into allow-single-disk-instead-of-storage-policy 2023-02-02 20:49:41 +01:00
Antonio Andelic
85cfee4bb9 Better alias definition 2023-02-01 13:54:03 +00:00
Antonio Andelic
714fad1529 Add support for settings alias 2023-01-26 14:06:46 +00:00
Robert Schulze
0653f86de9
Various cosmetic cleanups 2023-01-20 10:45:35 +00:00
larryluogit
52ae33dba7
Merge branch 'master' into ftsearch 2023-01-19 11:34:11 -05:00
Han Fei
ed49ebf01a update setting explain 2023-01-13 20:26:08 +01:00
Han Fei
2fb2f503e3 Update src/Storages/MergeTree/MergeTreeSettings.h
Co-authored-by: Sergei Trifonov <svtrifonov@gmail.com>
2023-01-13 20:20:08 +01:00
Han Fei
a258a39eb1 Merge branch 'master' into hanfei/async-cache 2023-01-13 20:17:58 +01:00
Igor Nikonov
1d002e45a9 Fix test & review comments 2023-01-11 12:49:01 +00:00
Igor Nikonov
e75df5e76b Test
+ fix some review comments
2023-01-10 22:14:54 +00:00
Han Fei
bef6168c05 support cache for async inserts block ids 2023-01-10 13:19:12 +01:00
kssenii
2a030c1dc0 Merge remote-tracking branch 'upstream/master' into allow-single-disk-instead-of-storage-policy 2023-01-06 15:58:27 +01:00
HarryLeeIBM
0839ff60ee Initial Checkin of full text search 2022-12-31 18:00:17 -08:00
Dmitry Novik
2ec720c5c4 Add part removal limit for one CleanupThread iteration 2022-12-20 19:15:13 +00:00
kssenii
ea5b06023a Merge remote-tracking branch 'refs/remotes/upstream/master' into allow-single-disk-instead-of-storage-policy 2022-12-14 13:59:51 +01:00
Han Fei
32c7fbec55 use another dedup path for async inserts 2022-11-22 15:12:00 +01:00
kssenii
b5b25c2676 Merge remote-tracking branch 'upstream/master' into allow-single-disk-instead-of-storage-policy 2022-11-07 17:02:41 +01:00
Antonio Andelic
611c2e2bd7 Support for optimizing old parts for entire partition only 2022-10-25 13:40:18 +00:00
Antonio Andelic
05e6b94b54 Polishing 2022-10-19 12:59:28 +00:00
Antonio Andelic
e5408aac99 Simplify logic 2022-10-19 12:34:20 +00:00
Antonio Andelic
e828dce331 Disable auto optimize by default 2022-10-18 12:11:23 +00:00
Antonio Andelic
06b2eeb5a7 Small changes to settings 2022-10-18 09:31:16 +00:00
Antonio Andelic
bd2522fdd0 Merge branch 'master' into optimize-partitions-in-background 2022-10-18 09:13:59 +00:00
Alexey Milovidov
a45f7c3023 Relax "too many parts" thresholds 2022-10-08 00:24:51 +02:00
Alexey Milovidov
169c505603 Relax "too many parts" thresholds 2022-10-08 00:24:51 +02:00
kssenii
01c4299481 Support single disk instead of storage policy 2022-09-27 22:34:31 +02:00
Nikolai Kochetov
3b63719ece
Merge pull request #41619 from den-crane/patch-50
increase default max_suspicious_broken_parts to 100
2022-09-22 13:25:22 +02:00
Denny Crane
75cf53c06e
increase default max_suspicious_broken_parts to 100 2022-09-21 11:10:19 -03:00
Alexey Milovidov
dd412f6145 Fix error 2022-08-28 20:42:11 +02:00
Alexey Milovidov
5ee81598b5 Merge branch 'compress_marks_and_primary_key' of github.com:zhongyuankai/ClickHouse into compress-marks 2022-08-28 04:19:14 +02:00
Antonio Andelic
6f533313e9 Restart everything during attach 2022-08-19 08:17:02 +00:00
Antonio Andelic
b0b9f96837 Use setting for retry period, create independent zookeeper sessions 2022-08-16 13:07:51 +00:00
Alexey Milovidov
0e450787f3 Edit the documentation 2022-08-12 21:24:27 +02:00
Alexey Milovidov
a88e713102 Disable zero-copy replication by default 2022-08-12 21:22:11 +02:00
zhongyuankai
71935b96ef support compress marks and primary key 2022-08-10 18:07:14 +08:00
fastio
e113d697be
Update src/Storages/MergeTree/MergeTreeSettings.h
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2022-08-01 11:04:18 +08:00
fastio
f382a91d54 Auto optimize partition in background if enabled 2022-07-21 10:49:09 +08:00
kssenii
0d2dae0046 Add const, rename settings 2022-06-14 00:47:03 +02:00
kssenii
4859e11c27 Review fixes 2022-06-12 01:15:10 +02:00
kssenii
fd6c84b831 Review fixes 2022-06-10 14:23:13 +02:00
kssenii
7a2676c7ab Clean up broken detached parts with timeout 2022-06-10 12:27:57 +02:00
Azat Khuzhin
dc9ca3d70c
Fix LOGICAL_ERROR in getMaxSourcePartsSizeForMerge during merges (#37413) 2022-05-26 14:14:58 +02:00
Denny Crane
383076ca0e
fix typo 2022-04-10 16:42:43 -03:00
Alexander Tokmakov
287d858fda Merge branch 'master' into mvcc_prototype 2022-03-29 16:24:12 +02:00
taiyang-li
8dbf1c60e7 merge master and fix conflict 2022-03-23 11:36:50 +08:00