thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-13 18:02:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
79,011 Commits 9,328 Branches 1,783 Tags 2.1 GiB
fc4e947f45
Commit Graph

302 Commits

Author SHA1 Message Date
Amos Bird
e6522e1ebe
JBOD data balancer 2021-03-08 11:10:35 +08:00
Nikolai Kochetov
9a39459888 Refactor ActionsDAG 2021-03-04 20:38:12 +03:00
Amos Bird
93b661ad5a
partition id pruning 2021-03-04 19:43:03 +08:00
Nikolai Kochetov
d328bfa41f Review fixes. Add setting max_optimizations_to_apply. 2021-02-26 19:29:56 +03:00
alesapin
3253638969 Fix backoff for failed background tasks in replicated merge tree 2021-02-11 14:46:18 +03:00
Kruglov Pavel
caef103837
Merge branch 'master' into Add_IStoragePolicy_interface 2021-01-29 14:00:12 +03:00
alesapin
5622e6daa6 Fix rare max_number_of_merges_with_ttl_in_pool limit overrun for non-replicated MergeTree 2021-01-27 14:56:12 +03:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
Alexey Milovidov
8276a1c8d2 Faster parts removal, more safe and efficient interface of IDisk 2021-01-14 19:24:13 +03:00
alexey-milovidov
a08db94343
Revert "Add metrics for part number in MergeTree in ClickHouse" 2021-01-07 16:40:52 +03:00
alexey-milovidov
703e16db5a
Update StorageMergeTree.cpp 2021-01-07 06:38:05 +03:00
weeds085490
5f5b86b485 Merge remote-tracking branch 'origin' into dev/add_metrics_for_parts 2021-01-06 17:32:45 +08:00
alesapin
23156fe6e4 Fix accident stop merges for merge tree 2020-12-22 16:46:09 +03:00
roverxu
b339b9dfd0 fix consistence 2020-12-22 17:38:15 +08:00
Alexey Milovidov
9be5fa9ef2 Merge branch 'master' into Enmk-Optimize_deduplicate 2020-12-20 09:57:10 +03:00
Vasily Nemkov
59fc301344 Fixed test to be less flaky 
Also logging expanded list of columns passed from `DEDUPLICATE BY` to actual deduplication routines.
 2020-12-08 19:44:34 +03:00
Kruglov Pavel
e19eb6f17a
Merge branch 'master' into optimize-data-on-insert 2020-12-08 15:57:46 +03:00
Vasily Nemkov
70ea507dae OPTIMIZE DEDUPLICATE BY columns 
Extended OPTIMIZE ... DEDUPLICATE syntax to allow explicit (or implicit with asterisk/column transformers) list of columns to check for duplicates on.

Following syntax variants are now supported:

OPTIMIZE TABLE table DEDUPLICATE; -- the old one
OPTIMIZE TABLE table DEDUPLICATE BY *;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT (colX, colY);
OPTIMIZE TABLE table DEDUPLICATE BY col1,col2,col3;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex');
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT (colX, colY);

Note that * behaves just like in SELECT: MATERIALIZED, and ALIAS columns are not used for expansion.
Also, it is an error to specify empty list of columns, or write an expression that results in an empty list of columns, or deduplicate by an ALIAS column.
Column transformers other than EXCEPT are not supported.
 2020-12-07 09:44:07 +03:00
Pavel Kruglov
5ae6c6dab9 Fix build error 2020-12-04 20:40:28 +03:00
Pavel Kruglov
905ba78adc Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-12-04 18:56:46 +03:00
Pavel Kruglov
9dbced0474 Pass setting instead of context 2020-12-04 17:01:59 +03:00
Anton Popov
cd1917c7a6
Merge branch 'master' into optimize_final_optimization 2020-12-03 16:52:51 +03:00
alexey-milovidov
f4a61ac3c3
Merge pull request #17527 from ucasFL/spelling 
fix spelling errors
 2020-11-29 13:45:42 +03:00
feng lv
7e3524caa1 fix spelling errors 2020-11-28 08:17:20 +00:00
nikitamikhaylov
72c7cd6693 replace Context& to Settings& 2020-11-25 16:47:32 +03:00
nikitamikhaylov
68bef22fda Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-sequential-consistency 2020-11-23 16:28:35 +03:00
Pavel Kruglov
ca3fe49a2a Make setting global 2020-11-20 17:29:13 +03:00
Nicolae Vartolomei
94293ca3ce Assign UUIDs to parts only when configured to do so 
Avoid breaking backwards compatibility by default for now.
 2020-11-20 13:49:17 +00:00
Pavel Kruglov
4c30857759 Minor change 2020-11-20 01:22:40 +03:00
Nicolae Vartolomei
746f8e45f5 All new parts must have uuids 2020-11-19 13:18:03 +00:00
Pavel Kruglov
547ec19fb3 Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-11-18 12:01:59 +03:00
Pavel Kruglov
8d5e0784d3 Add setting optimize_on_insert 2020-11-12 23:37:23 +03:00
Anton Popov
4cecd89e84 fix dropping of empty parts 2020-11-12 20:37:54 +03:00
Anton Popov
0aa35d53c7 Merge remote-tracking branch 'upstream/master' into HEAD 2020-11-12 14:31:58 +03:00
alesapin
2cac5e5d97
Merge pull request #16875 from CurtizJ/refactor-alter-partition 
Avoid code duplication in alterPartition
 2020-11-12 13:19:59 +03:00
tavplubix
058aa8f85e
Merge pull request #16824 from ClickHouse/replace_stringstreams_with_buffers 
Replace std::*stringstreams with DB::*Buffers
 2020-11-12 01:11:44 +03:00
Anton Popov
2497598a49 Remove empty parts after they were pruned by TTL 2020-11-11 19:19:43 +03:00
Anton Popov
6f0dc08ba7 avoid code duplication in alterPartition 2020-11-11 16:34:07 +03:00
Pavel Kruglov
287d71b363 Fix build 2020-11-11 13:34:32 +03:00
Pavel Kruglov
dd06ff4f94 Minor change 2020-11-11 00:02:11 +03:00
Pavel Kruglov
1c35b47073 Add checking select decision in merge 2020-11-10 23:52:26 +03:00
Pavel Kruglov
c648f62629 Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize_final_optimization 2020-11-10 22:58:21 +03:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
Pavel Kruglov
9120189d8a Add SelectPartsDecision enum class 2020-11-10 17:42:56 +03:00
Nikolai Kochetov
195c941c4e Merge branch 'master' into storage-read-query-plan 2020-11-10 15:02:22 +03:00
Nikolai Kochetov
046830967c Try fix tests. 2020-11-10 13:26:26 +03:00
Alexander Tokmakov
62ff00ee8b use WriteBuffer in formatAST(...) 2020-11-09 19:05:40 +03:00
alesapin
72046313d4 Merge branch 'master' into nvartolomei-drop-part 2020-11-09 10:02:02 +03:00
alexey-milovidov
0e6ae4aff7
Merge pull request #16253 from amosbird/pf 
Prune partition in verbatim way.
 2020-11-08 18:58:02 +03:00
alexey-milovidov
f4ba5f1f9a
Merge pull request #16772 from ClickHouse/fix-stringstream 
Fix "server failed to start" error
 2020-11-08 14:27:08 +03:00
Alexey Milovidov
5314185e25 Merge branch 'master' into azat-optimize_skip_unused_shards-optimization 2020-11-08 00:17:59 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Nikolai Kochetov
c10f733587 Merge branch 'master' into storage-read-query-plan 2020-11-06 15:43:46 +03:00
Amos Bird
2b0085c106
Pruning is different from counting 2020-11-06 19:58:03 +08:00
Amos Bird
30bf5e6d26
Prune partition in verbatim way. 2020-11-06 09:56:13 +08:00
alesapin
da8ac948bf Disallow to drop parts with parallel quorum 2020-11-03 12:24:10 +03:00
alesapin
dac6705995 DROP PART and clear blocks in a single zookeeper op 2020-11-02 20:30:59 +03:00
alesapin
67d22b3e57 Merge branch 'master' into nvartolomei-drop-part 2020-11-02 19:09:09 +03:00
hchen9
feaab260d8 Fix conflict 2020-10-23 12:54:33 -07:00
alesapin
39e47c5338 Less strange lambdas 2020-10-23 11:54:00 +03:00
alesapin
7a63b373f9 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 15:47:11 +03:00
Nikolai Kochetov
7fa045cff8 Merge branch 'master' into storage-read-query-plan 2020-10-22 13:31:10 +03:00
alexey-milovidov
adeba6bdd8
Merge pull request #15074 from amosbird/btc 
Extend trivial count optimization.
 2020-10-22 02:50:57 +03:00
alesapin
c77b154398 Better locks in merge tree 2020-10-21 00:10:55 +03:00
alesapin
ce8ee01d75 Fix background moves start 2020-10-20 14:27:50 +03:00
alesapin
d126aa8a09 Merge branch 'master' into no_background_pool_no_more 2020-10-20 12:54:42 +03:00
alesapin
514a95c292 Fixup 2020-10-19 12:06:51 +03:00
Alexander Tokmakov
026896835c fixes 2020-10-16 15:20:30 +03:00
alesapin
7edbe4d276 Add some comments 2020-10-16 13:12:31 +03:00
alesapin
4c1f5cab13 Sleep less in background task 2020-10-15 19:10:22 +03:00
Pavel Kruglov
25c39689c3 Check expired ttl 2020-10-15 15:15:02 +03:00
alesapin
84b0c22044 fix possible deadlock 2020-10-15 13:54:50 +03:00
alesapin
c5419083e3 Better method name 2020-10-15 10:43:50 +03:00
alesapin
7265792510 Less files 2020-10-14 21:33:41 +03:00
alesapin
a535a089d1 Some style fixes 2020-10-14 19:09:18 +03:00
alesapin
483893cdd4 Slightly refactored code 2020-10-14 17:56:42 +03:00
alesapin
0b14a31ba9 Get rid of background processing pool 2020-10-14 15:44:10 +03:00
alesapin
4014e0f08d Something very similar to working code 2020-10-14 15:32:35 +03:00
alesapin
efd3126b5d Moving pool 2020-10-14 10:22:48 +03:00
Pavel Kruglov
1ccf10505a add tests and comments 2020-10-13 21:26:40 +03:00
Pavel Kruglov
6d9b310f61 Add optimize_skip_merged_partitions 2020-10-13 21:26:35 +03:00
alesapin
fd35368c59 First simple implementation 2020-10-13 17:25:42 +03:00
alesapin
dcf489ac8f Remove unused methods 
(cherry picked from commit 77f5841a46)
 2020-10-13 15:10:59 +03:00
alesapin
ef355e71d6 Buildable code 2020-10-13 15:10:46 +03:00
alesapin
48aaa0ce16 Split merge to separate method 
(cherry picked from commit 14a7a25c4b)
 2020-10-13 15:06:32 +03:00
alesapin
cb7db6d232 Split select and process merges 
(cherry picked from commit 1889eb0ff1)
 2020-10-13 15:04:16 +03:00
Nikolai Kochetov
7e58f99f64 Merge branch 'master' into storage-read-query-plan 2020-10-12 13:12:39 +03:00
Pervakov Grigorii
3a2068f19d Use tmp disk for vertical merge files 2020-10-08 13:49:41 +03:00
Amos Bird
867216103f
Extend trivial count optimization. 2020-10-08 18:08:17 +08:00
hchen9
ed2ce4a6fc Resolve conflict in StorageMemory 2020-10-07 17:19:59 -07:00
Nikolai Kochetov
7e02152b50 Merge branch 'master' into storage-read-query-plan 2020-10-07 14:27:21 +03:00
Nikolai Kochetov
c5cb05f5f3 Try fix tests. 2020-10-07 14:26:29 +03:00
alesapin
8ec58c17f3 More consistent metadata usage 2020-10-05 19:41:46 +03:00
Azat Khuzhin
b838214a35 Pass non-const SelectQueryInfo (and drop mutable qualifiers) 2020-10-02 22:42:35 +03:00
hchen9
b3949db00f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into trivial_count_seq_consistency 2020-10-01 21:15:26 -07:00
Nikolai Kochetov
ec64def384 Use QueryPlan while reading from MergeTree. 2020-10-01 20:34:22 +03:00
hchen9
a5ac39b564 Pass Context parameter for IStorage.totalRows and IStorage.totalBytes 2020-09-30 16:47:42 -07:00
alesapin
f5c38b34ae Fix race condition during write ahead log cleanup and rename 2020-09-25 15:07:38 +03:00
Nicolae Vartolomei
97d0b5ab23 Remove unused query argument 2020-09-08 10:56:10 +01:00
Nicolae Vartolomei
1876374770 ALTER TABLE ... DROP|DETACH PART for MergeTree 2020-09-08 10:56:09 +01:00
alesapin
82c56349a5 Some comments 2020-09-04 13:08:09 +03:00
alesapin
61ecaebcb1 Simplify settings for TTL merges 2020-09-04 09:55:19 +03:00
alesapin
f4c7ff0376 Add fixed size of Merge TTLS 2020-09-03 16:00:13 +03:00
alesapin
232c264d7d Disable force TTL on optimise 2020-08-28 16:45:42 +03:00
alexey-milovidov
0e3b9d0da6
Merge pull request #13747 from ClickHouse/fix-race-condition-in-test 
Fix race condition in test
 2020-08-15 15:55:47 +03:00
Alexey Milovidov
9e6d4802ac Do not drop cache if DROP/RENAME in Atomic database 2020-08-15 09:56:47 +03:00
Alexey Milovidov
ee084f0ec9 Fix race condition in DETACH and background merge 2020-08-15 08:21:02 +03:00
Nikolai Kochetov
09fbce1b1e Merge branch 'master' into refactor-pipes-3 2020-08-04 11:32:34 +03:00
Nikolai Kochetov
2cca4d5fcf Refactor Pipe [part 2]. 2020-08-03 16:54:14 +03:00
alesapin
45894fcf99 Better comments 2020-07-31 15:22:32 +03:00
alexey-milovidov
bbc8473388
Update StorageMergeTree.cpp 2020-07-31 15:00:07 +03:00
alesapin
14928874a1 Fix mutations status check 2020-07-31 14:37:16 +03:00
alexey-milovidov
fc35ce69fa
Merge pull request #12666 from ClickHouse/forward_alter_error_to_client 
Forward exceptions to client if exception happened during ALTER
 2020-07-30 21:36:06 +03:00
alesapin
7a806eb90f
Merge pull request #13017 from ClickHouse/freeze_partition_with_name 
Alter table freeze with verbose output
 2020-07-29 18:42:48 +03:00
alesapin
ba8a29a3a4 Better logging on mutation skip 2020-07-29 16:12:55 +03:00
alesapin
7fc1e45c3b Review fixe 2020-07-29 11:32:52 +03:00
alesapin
c75830e878 Alter table freeze with verbose output 2020-07-28 18:10:36 +03:00
alesapin
355150afce Fix deadlock and better code 2020-07-22 22:29:54 +03:00
alesapin
8c8bdd5070 Fix tests and work with multiple mutations 2020-07-22 18:19:54 +03:00
alesapin
7585fc4b52 Fix style 2020-07-22 16:22:05 +03:00
alesapin
de846e5e3c Forward exception to client during alter 2020-07-22 15:36:19 +03:00
alesapin
4a53264a86 Remove redundant and duplicated code 2020-07-13 19:19:08 +03:00
Anton Popov
416d9648c2 remove questionable functionality 2020-07-07 12:58:14 +03:00
Anton Popov
4422df2e37 Merge remote-tracking branch 'upstream/master' into HEAD 2020-07-02 20:18:21 +03:00
Vladimir Chebotarev
bb8da71eff Moves task shall be started if new storage policy needs them. 2020-06-30 15:36:44 +03:00
Anton Popov
a43cb93be5 remove questionable functionality 2020-06-29 18:46:50 +03:00
Alexey Milovidov
97ad23b905 Allow to ALTER partition key in some cases 2020-06-28 22:39:31 +03:00
alesapin
6f1824f0ea Correct merge with master 2020-06-26 14:30:23 +03:00
alesapin
e9c47dc89c Merge branch 'master' into CurtizJ-polymorphic-parts 2020-06-26 14:27:19 +03:00
alesapin
292016e045 Add missed lock 2020-06-23 11:04:43 +03:00
alesapin
c666763cc0 Remove unused method better 2020-06-18 20:09:06 +03:00
alesapin
e888dafdc2 Remove unused method 2020-06-18 20:03:42 +03:00
alesapin
54e5fe7dbc Less locks 2020-06-18 19:19:40 +03:00
alesapin
d79982f497 Better locks in Storages 2020-06-18 19:10:47 +03:00
alesapin
aab4ce6394 Truncate with metadata 2020-06-18 13:29:13 +03:00
alesapin
ed8f3b2fc4 TTL in storage in memory metadata 2020-06-17 16:39:26 +03:00
alesapin
eca6e9087e Fix race condition 2020-06-17 14:52:19 +03:00
alesapin
ab61abccc1 Partition key in StorageInMemoryMetadata 2020-06-17 13:34:23 +03:00
alesapin
1ddeb3d149 Buildable getSampleBlock in StorageInMemoryMetadata 2020-06-16 18:51:29 +03:00
alesapin
71f99a274d Compileable getSampleBlockWithColumns in StorageInMemoryMetadata 2020-06-16 17:25:08 +03:00
alesapin
0bcd22008a Get column dependencies in StorageInMemoryMetadata 2020-06-16 15:19:21 +03:00
Anton Popov
b19d48a11c Merge remote-tracking branch 'upstream/master' into HEAD 2020-06-16 06:37:55 +03:00
alesapin
36ba0192df Metadata in read and write methods of IStorage 2020-06-15 22:08:58 +03:00
alesapin
33a74a3ea0 TTL methods in StorageInMemoryMetadata 2020-06-15 20:50:53 +03:00
alesapin
ec933d9d03 Better naming 2020-06-10 12:09:51 +03:00
alesapin
d2fcf5aea5 Fixes for gcc 2020-06-09 20:28:29 +03:00
alesapin
f7a397242f Merge branch 'master' into consistent_metadata4 2020-06-09 18:59:59 +03:00
alesapin
223f45685f Review fixes 2020-06-09 05:03:22 +03:00
Anton Popov
66e31d4311 in-memory parts: several fixes 2020-06-05 23:49:24 +03:00
alesapin
abaf47f0cd Make metadata single structure 2020-06-05 20:29:40 +03:00