Commit Graph

784 Commits

Author SHA1 Message Date
Arthur Passos
ce6877cbb6
Merge branch 'ClickHouse:master' into bring_back_attach_partition_pr 2024-01-26 10:57:11 -03:00
Maksim Kita
2a327107b6 Updated implementation 2024-01-25 14:31:49 +03:00
Maksim Kita
42029f42e7 Table CREATE DROP Poco::Logger memory leak fix 2024-01-25 14:31:18 +03:00
Arthur Passos
dba792dff0
Remove duplicate includ 2024-01-23 15:03:31 -03:00
Arthur Passos
e6b17377d0
Merge branch 'master' into bring_back_attach_partition_pr 2024-01-23 14:52:47 -03:00
Nikolai Kochetov
d64b491b45 Fixing build 2024-01-23 17:00:21 +00:00
Nikolai Kochetov
a26f8f5425
Merge pull request #59076 from ClickHouse/revert-59059-revert-58838-allow-parallel-replicas-for-join-with-analyzer
Revert "Revert "Allow parallel replicas for JOIN with analyzer [part 1].""
2024-01-23 17:01:19 +01:00
Arthur Passos
799a94081b Revert "Revert "Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition""
This reverts commit a1c83e2f51.
2024-01-23 11:40:42 -03:00
Robert Schulze
a1c83e2f51
Revert "Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition" 2024-01-23 15:26:04 +01:00
Nikolai Kochetov
fcbab407b8
Merge branch 'master' into revert-59059-revert-58838-allow-parallel-replicas-for-join-with-analyzer 2024-01-23 12:21:44 +01:00
kssenii
b76630c955 Merge remote-tracking branch 'origin/master' into broken-projections-better-handling 2024-01-23 11:34:58 +01:00
Arthur Passos
24b8bbe9fa
Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition (#39507)
* temp commit

* temp commit

* draft impl for feedback

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* aa

* aa

* Add integ tests and remove partition key restriction

* fix small incosistency in partition id

* style fix

* style fix

* style fix

* use existing DataPartStorageBuilder instead of new one

* Refactor part clone to make it more readable and maintainable

* Add MergeTreeDataPartCloner docs

* define ErrorCodes::BAD_ARGUMENTS

* Rebase

* camel case methods

* address some comments

* yet another rebase?

* Move from integ tests to stateless tests

* address more comments

* add finalize on min_max_idx files

* Add sync option to DistinctPartitionExpCloner

* just a temp test

* revert temp change

* Use echoOn to distinguish test queries

* remove comment

* fix build issue during rebase

* atempt to fix build after rebase

* finally fix build

* clear minmaxidx hyperrectangle before loading it

* Fix error on min_max files deletion where it was being assumed that partition expression contained all columns

* get it to the state it was previously

* add missing include

* getting functional?

* refactoring and renaming

* some more refactoring

* extern bad arguments

* try to fix style

* improvements and docs

* remove duplicate includes

* fix crash

* make tests more stable by ordering

* rebase once again..

* fix

* make ci happy?

* fix rebase issues

* docs

* rebase, but prolly needs to be improved

* refactor out from nasty inheritance to static methods

* fix style

* work around optional

* refactor & integrate some changes

* update column_type

* add tests by dencrane

* set utc

* fix ref file

* fix tests

* use MergeTree instead of SummingMergeTree

* mark MergeTreeDataPart::getBlock as const

* address a few comments

* compute module function name size at compile time

* simplify branching in getPartitionAstFieldsCount

* remove column_indexes argument

* merge getBlock with buildBlock

* add some const specifiers

* small adjustments

* remove no longer needed isNull check

* use std::min and max to update global min max idx

* add some assertions

* forward declare some symbols

* fix grammar

* forward decl

* try to fix build..

* remove IFunction forward decl

* Revert "use std::min and max to update global min max idx"

This reverts commit b2fe79dda7.

* Revert "remove no longer needed isNull check"

This reverts commit 129db2610f.

* Revert "Revert "remove no longer needed isNull check""

This reverts commit 9416087dd8.

* Revert "Revert "use std::min and max to update global min max idx""

This reverts commit 20246d4416.

* remove some comments

* partial use of MonotonicityCheckMatcher

* ranges

* remove KeyDescriptionMonotonicityChecker

* remove duplication of applyfunction

* move functions to anonymous namespace

* move functions to cpp

* Relax partition compatibility requirements by accepting subset, add tests from partitioned to unpartitioned

* updte reference file

* Support for partition by a, b, c to partition by a, b

* refactoring part 1

* refactoring part 2, use hyperrectangle, still not complete

* refactoring part 3, build hyperrectangle with intersection of source & destination min max columns

* Support attaching to table with partition expression of multiple expressions

* add tests

* rename method

* remove some code duplication

* draft impl for replicatedmergetree, need to dive deeper

* ship ref file

* fix impl for replicatedmergetree..

* forbid attach empty partition replicatedmergetree

* Add replicated merge tree integration tests

* add test missing files

* fix black

* do not check for monotonicity of empty partition

* add empty tests & fix replicated

* remove no longer needed buildBlockWithMinMaxINdexes

* remove column logic in buildHyperrectangle

* simplify implementation by using existing methods

* further simplify implementation

* move all MergeTreeDataPartClone private methods to .cpp file

* decrease decomposition

* use different namespaces

* reduce code duplication

* fix style

* address a few comments

* add chassert to assert arguments size on MonotonicityCheckVisitor

* remove deleteMinMaxFiles method

* remove useless checks from sanitycheck

* add tests for attach partition (not id)

* Remove sanityCheckASTPartition and bring back conditional getPartitionIDFromQuery

* remove empty block comment

* small fixes

* fix formatting

* add missing include

* remove duplicate iuncludes

* trigger ci

* reduce some code duplication

* use updated partition id on replicatedmergetree

* fix build

* fix build

* small refactor

* do not use insert increment on fetch part

* remove duplicate includes

* add one more integ test

* black

* black

* rely on partition exp instead of partition id on replicated part fetch to decide if it is a different partition exp

* add one more integ test

* add order by clause

* fix black

---------

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
2024-01-22 21:00:04 +01:00
Nikolai Kochetov
65c5ea31a8
Revert "Revert "Allow parallel replicas for JOIN with analyzer [part 1]."" 2024-01-22 17:09:34 +01:00
Alexander Tokmakov
4315d44517
Revert "Allow parallel replicas for JOIN with analyzer [part 1]." 2024-01-22 13:30:35 +01:00
kssenii
48e489682b Merge remote-tracking branch 'origin/master' into broken-projections-better-handling 2024-01-19 10:49:27 +01:00
MikhailBurdukov
ee5d8c0a27 Review 2024-01-19 09:04:02 +00:00
MikhailBurdukov
f6fb20d6cf Backoff policy for failed mutation. 2024-01-18 15:43:43 +00:00
Nikolai Kochetov
534e086a5d
Update StorageMergeTree.cpp 2024-01-16 13:50:39 +01:00
Nikolai Kochetov
733ceaa775 Allow parallel replicas for JOIN with analyzer. 2024-01-15 18:48:12 +00:00
Kseniia Sumarokova
a6b5b7460b
Merge branch 'master' into broken-projections-better-handling 2024-01-10 16:46:18 +01:00
Anton Popov
8c9cf394b3 Merge remote-tracking branch 'upstream/master' into HEAD 2024-01-06 20:58:17 +00:00
Nikolai Kochetov
7a271f09ed Check if I can remove KeyCondition analysis on AST. 2024-01-03 17:50:46 +00:00
Anton Popov
ce1b978a2d Merge remote-tracking branch 'upstream/master' into HEAD 2024-01-03 16:01:44 +00:00
Alexander Tokmakov
bdada351c8 Revert "Merge pull request #58274 from ClickHouse/revert-58267"
This reverts commit 583b9637c2, reversing
changes made to 224e937620.
2023-12-28 14:07:59 +01:00
Alexey Milovidov
c7efd2afea Revert #58267 2023-12-28 04:09:33 +01:00
Alexander Tokmakov
f5bcfaffa5 disable vertical merges with cleanup 2023-12-27 19:28:50 +01:00
Alexander Tokmakov
01d042c490 Revert "Merge pull request #57932 from ClickHouse/remove-shit-cleanup"
This reverts commit 2d58dc512c, reversing
changes made to 41873dc4a3.
2023-12-27 13:46:06 +01:00
Kseniia Sumarokova
189d0ec147
Merge branch 'master' into broken-projections-better-handling 2023-12-21 17:17:50 +01:00
Alexey Milovidov
7be39a27cc
Revert "Consider lightweight deleted rows when selecting parts to merge" 2023-12-21 01:46:38 +03:00
Nikita Mikhaylov
cb6c9557ff
Merge branch 'master' into remove-old-projections-code 2023-12-20 17:45:38 +01:00
Alexey Milovidov
af32b33e93
Merge pull request #57648 from jewelzqiu/refine-lwd-merge
Consider lightweight deleted rows when selecting parts to merge
2023-12-20 01:10:32 +01:00
Yakov Olkhovskiy
43fcea64b6
Merge pull request #57186 from chenlx0/bytes_uncompressed
Show uncompressed data size in system.tables
2023-12-19 19:04:47 -05:00
Jordi Villar
7c281d9a6f
Allow max_size_to_drop settings in query time (#57452) 2023-12-19 23:16:03 +01:00
Alexey Milovidov
ee502ef6b9 Remove trash (OPTIMIZE CLEANUP) 2023-12-15 22:41:58 +01:00
Nikita Mikhaylov
1780671443
ClickHouse Cloud promotion (#57638)
Co-authored-by: Alexander Sapin <alesapin@gmail.com>
2023-12-15 15:23:23 +01:00
Alexey Milovidov
bf4964b177
Merge pull request #57712 from ClickHouse/clickbench-ci
Add ClickBench to CI
2023-12-13 19:52:44 +01:00
Igor Nikonov
c165be76ab
Parallel replicas: friendly settings (#57542) 2023-12-13 14:42:06 +01:00
Alexey Milovidov
f11b90e7bb Allow buckets without List access 2023-12-13 00:10:58 +01:00
Zhuo Qiu
e34c13b9d9 Consider lightweight deleted rows when selecting parts to merge 2023-12-12 11:40:19 +08:00
kssenii
a57e612cf2 Fxi tests 2023-11-24 16:08:49 +01:00
Chen Lixiang
89fa729235 show uncompressed size in system.tables 2023-11-24 15:54:27 +08:00
Anton Popov
65efb18842 Merge remote-tracking branch 'upstream/master' into HEAD 2023-11-21 17:30:38 +00:00
kssenii
eb7aad0016 Do not consider parts broken if only projections are broken 2023-11-16 12:47:28 +01:00
Alexey Milovidov
c095b1e9ef
Merge pull request #56502 from amosbird/fix-56481
Fix two cases of projection analysis.
2023-11-11 02:29:04 +01:00
Amos Bird
16578d97a3
Fix two cases of projection analysis.
1. Fix projection analysis with partitions.
2. Apply alter conversions during projection analysis.
2023-11-10 10:23:02 +08:00
kssenii
f2c0434c4d Merge remote-tracking branch 'origin/master' into minor-improvements-for-s3-queue 2023-11-06 15:51:11 +01:00
kssenii
da21413354 Better shutdown 2023-11-06 15:47:57 +01:00
vdimir
55a1baf029
Merge pull request #56022 from ClickHouse/vdimir/check_all_tables
Add CHECK query for all the tables
2023-11-03 10:10:42 +01:00
Alexander Sapin
3a1182f6c0 Proper implementation 2023-10-31 18:53:33 +01:00
alesapin
20b8954872 Better process broken parts on table start for replicated tables 2023-10-30 23:16:04 +01:00
vdimir
c1ae369919
wip CHECK query for all tables 2023-10-27 15:22:10 +00:00
vdimir
1d5f75fd71
Merge pull request #53404 from ClickHouse/vdimir/check_table_improvements2
Improve CHECK TABLE system query, support checking part
2023-10-26 10:57:44 +02:00
alesapin
0d68a52c06
Merge pull request #50181 from ClickHouse/repro_49887
Make exception about `ALTER TABLE ... DROP COLUMN|INDEX|PROJECTION` more clear
2023-10-25 15:57:17 +02:00
vdimir
7583394c9c
Support CHECK TABLE ... PART ... query 2023-10-23 12:33:35 +00:00
vdimir
8f0d7954ff
IStorage::checkDataNext returns optional 2023-10-23 10:12:30 +00:00
vdimir
fe95c0d0e4
Improve CHECK TABLE system query
Resubmit PR #52745
2023-10-23 09:35:26 +00:00
alesapin
88f80078e1
Merge branch 'master' into repro_49887 2023-10-20 13:04:00 +02:00
Raúl Marín
f799f5d7a1 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-16 15:07:11 +02:00
Raúl Marín
478f635d60
Merge pull request #53792 from ClickHouse/revert-53782-revert-52762-planner-prepare-filters-for-analysis
Revert "Revert "Planner prepare filters for analysis""
2023-10-16 13:21:54 +02:00
Anton Popov
fb666e137c remove old code of projection analysis 2023-10-12 23:25:20 +00:00
alesapin
e364ed61a6 Proper fix 2023-10-11 17:42:44 +02:00
alesapin
ab09aee754 Initial implementation 2023-10-10 19:59:48 +02:00
Raúl Marín
95d2063e91 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-10 17:29:45 +02:00
Igor Nikonov
0b4d76780f
Merge branch 'master' into revert-53782-revert-52762-planner-prepare-filters-for-analysis 2023-10-09 14:44:00 +02:00
Alexey Milovidov
6e87fa3f67 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-in-memory-data-parts 2023-09-30 05:12:29 +02:00
Alexey Milovidov
6de0250650 Continue removing in-memory data parts 2023-09-30 03:10:59 +02:00
Alexey Milovidov
d8b5713023 Remove the removal of broken detached parts 2023-09-30 01:47:44 +02:00
Nikita Mikhaylov
804ae4c58b Better 2023-09-29 11:16:24 +00:00
Azat Khuzhin
c706101891 Fix throttling of BACKUPs from/to S3 (in case native copy was not used)
In some cases native copy is not possible, and such requests should be
throttled.

v0: copyS3FileNativeWithFallback
v2: revert v0 and pass write_settings
v3: pass read_settings to copyFile()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-09-20 18:28:43 +02:00
alesapin
9dcc41cf83 Fxi 2023-09-19 13:47:32 +02:00
Igor Nikonov
d5ea047ab8 Parallel replicas: cleanup unused params 2023-09-11 21:52:40 +00:00
Igor Nikonov
7a396139df Cleanup: unnecessary SelectQueryInfo usage around distributed 2023-09-08 21:53:38 +00:00
Alexander Sapin
518d4b4357 Merge branch 'master' into alter-table-add-comment 2023-09-06 12:34:04 +02:00
Victor Krasnov
78760639d2 Merge branch 'master' into ADQM-987 2023-09-04 13:59:20 +03:00
Alexey Milovidov
7cb4d52d92 Deprecate metadata cache, part 2 2023-09-03 03:56:11 +02:00
Sergei Trifonov
cf5ea46713
Merge pull request #53338 from azat/throttling-fixes
Fix IO throttling during copying whole directories
2023-09-01 18:29:15 +02:00
Victor Krasnov
52cce27e81 Merge branch 'master' into ADQM-987 2023-09-01 18:00:20 +03:00
Dmitry Novik
5266511281 Merge remote-tracking branch 'origin/master' into revert-53782-revert-52762-planner-prepare-filters-for-analysis 2023-08-31 14:40:53 +00:00
Azat Khuzhin
51e8b16916 Fix IO throttling during copying whole directories
v2: pass local context during ALTER TABLE MOVE PARTITION TO VOLUME/DISK as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-29 23:13:46 +02:00
Nikita Mikhaylov
96cabcae3a Change interface 2023-08-29 16:26:48 +02:00
Raúl Marín
45e8185f9b Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-08-29 11:17:34 +02:00
Victor Krasnov
87c90767d9 Merge branch 'master' into ADQM-987 2023-08-27 09:14:42 +03:00
Nikolai Kochetov
93ab53163a
Revert "Revert "Planner prepare filters for analysis"" 2023-08-24 12:32:56 +02:00
Alexander Tokmakov
011c7f7064
Revert "Planner prepare filters for analysis" 2023-08-24 12:52:03 +03:00
Nikolai Kochetov
3a4a7d70a2
Merge pull request #52762 from kitaisreal/planner-prepare-filters-for-analysis
Planner prepare filters for analysis
2023-08-23 18:59:43 +02:00
Victor Krasnov
8b2342c205 Minor code cleanup: remove some redundant includes of InterpreterAlterQuery.h 2023-08-23 14:16:36 +00:00
Raúl Marín
93dac0c880 Support clang-18 (Wmissing-field-initializers) 2023-08-23 15:53:45 +02:00
Nikolai Kochetov
5ee9b56040 Do not change the snapshot after at reading from MT 2023-08-22 13:50:51 +00:00
Antonio Andelic
ed15babddd
Merge pull request #53198 from ClickHouse/improve-archive-reading
Improve reading from archives
2023-08-21 14:11:25 +02:00
Igor Nikonov
3e9ae1751e Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-17 14:45:37 +00:00
Igor Nikonov
49f7b4acd6 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-16 23:31:49 +00:00
Igor Nikonov
21ef1f1d1c Remove unnecessary Scalars usage for cluster_for_parallel_replicas 2023-08-16 23:24:01 +00:00
Robert Schulze
8a58e0df3e
Merge remote-tracking branch 'rschu1ze/master' into consistency-fixes-for-caches 2023-08-16 21:26:04 +00:00
Alexander Tokmakov
b32bdd1b0a
Merge pull request #53164 from ClickHouse/fixes_for_detached_parts
Fixes for detach/attach partition and broken detached parts cleanup
2023-08-16 00:09:13 +03:00
Igor Nikonov
49134711dc Polishing 2023-08-15 17:37:01 +00:00
Igor Nikonov
9825f8c76e Remove debug logging 2023-08-15 17:03:32 +00:00
Alexander Tokmakov
bf40767f10 fix another race 2023-08-15 14:27:49 +02:00
Igor Nikonov
1678cda8b3 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-15 12:03:35 +00:00
Robert Schulze
f317903073
Merge branch 'ClickHouse:master' into consistency-fixes-for-caches 2023-08-15 10:25:08 +02:00
Igor Nikonov
c94994afcf Propagate setting cluster_for_parallel_replicas to shards 2023-08-14 22:50:46 +00:00