thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 19:32:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
104,173 Commits 9,319 Branches 1,783 Tags 2.1 GiB
b43ddde50a
Commit Graph

340 Commits

Author SHA1 Message Date
Alexander Tokmakov
76156af5cc cancel merges on drop partition 2021-06-24 17:07:43 +03:00
alesapin
47b29092b4 Fix bug in MergerMutator parts selector 2021-06-22 16:47:42 +03:00
Anton Popov
662d5b8495 fix ALTER MODIFY COLUMN of columns, that participates in TTL expressions 2021-06-21 18:20:18 +03:00
Alexander Tokmakov
9a88b9dacc set maxtid on drop part 2021-06-08 21:17:18 +03:00
Alexander Tokmakov
9915bd0a8b Merge branch 'master' into mvcc_prototype 2021-06-03 00:52:57 +03:00
Alexander Tokmakov
9a9e95172f add test with mv 2021-06-02 23:03:44 +03:00
Alexander Tokmakov
5692db736c Merge branch 'master' into mvcc_prototype 2021-05-31 21:11:22 +03:00
kssenii
a0d8d689d3 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-27 15:44:12 +03:00
alesapin
6e29404a75 Add max_parts_to_merge_at_once setting 2021-05-25 16:21:17 +03:00
kssenii
3dee003f9b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-20 19:20:09 +03:00
Amos Bird
c68ef31065
Skip data finalization when doing projection materialization. 2021-05-20 00:28:50 +08:00
Alexander Tokmakov
529d1aeb19 fix merges of uncommitted parts 2021-05-18 20:07:29 +03:00
Alexander Tokmakov
f2ab5a05c6 Merge branch 'master' into mvcc_prototype 2021-05-18 13:36:38 +03:00
Alexander Tokmakov
1fd6142c1a try enable merges 2021-05-18 13:35:48 +03:00
kssenii
9b8df78fdd Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-17 17:42:05 +03:00
Alexander Tokmakov
970ad94851 Merge branch 'master' into fix_intersecting_parts 2021-05-14 13:17:02 +03:00
Alexander Tokmakov
df2849be90 Merge branch 'master' into mvcc_prototype 2021-05-13 21:48:36 +03:00
Alexander Tokmakov
2da12f0b59 fix intersection of drop range and merge 2021-05-13 17:04:36 +03:00
Alexander Tokmakov
e114c7eb8b fix virtual parts in REPLACE_RANGE 2021-05-13 14:29:59 +03:00
kssenii
0527f0ea33 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-12 16:54:18 +03:00
Nikolai Kochetov
29d7038e30
Add ProjectionDescription::Type enum. Remove some checks. 2021-05-11 18:12:27 +08:00
Amos Bird
264cff6415
Projections 
TODO (suggested by Nikolai)

1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies

3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)

Also need to figure out how prewhere works for projections, and
row_filter_policies.

wip
 2021-05-11 18:12:23 +08:00
Alexander Tokmakov
eef899ae63 better code, add test 2021-05-10 21:03:37 +03:00
kssenii
9ec92ec514 Fix tests, less manual concatination of paths 2021-05-05 18:39:30 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
Alexander Tokmakov
3422bd1742 check parts visibility for select 2021-04-08 20:20:45 +03:00
Anton Popov
173d2ea1f4 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-16 02:50:14 +03:00
alesapin
06eb2d8dfd
Merge branch 'master' into s3_zero_copy_replication 2021-03-13 22:32:54 +03:00
Azat Khuzhin
69b2b2a159 Fix fsync_part_directory for horizontal merge 2021-03-11 21:41:27 +03:00
Anton Popov
df6663dcb6 refactoring of serializations 2021-03-09 20:02:26 +03:00
Anton Popov
bc417cf54a refactoring of serializations 2021-03-09 17:46:52 +03:00
Anton Ivashkin
e69124a0a6 Merge master 2021-03-04 13:26:40 +03:00
alesapin
9ebf1b4fad Get rid of separate minmax index fields 2021-03-02 13:33:54 +03:00
Anton Ivashkin
c891cf4557 Fixes by review response 2021-02-26 12:48:57 +03:00
Anton Ivashkin
e64c63c611 Merge master 2021-02-05 20:10:06 +03:00
Pavel Kovalenko
b7151aa754 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskDecorator.h
#	src/Disks/IDisk.h
#	src/Disks/S3/DiskS3.cpp
 2021-02-03 14:22:18 +03:00
Kruglov Pavel
caef103837
Merge branch 'master' into Add_IStoragePolicy_interface 2021-01-29 14:00:12 +03:00
Anton Popov
c7070da85a better abstractions in disk interface 2021-01-26 17:49:35 +03:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
Anton Ivashkin
357d98eb36 Merge master 2021-01-20 12:23:03 +03:00
Anton Ivashkin
eba98b04b0 Zero copy replication over S3: Hybrid storage support 2021-01-18 19:16:45 +03:00
Pavel Kovalenko
1e3a059f64 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskCacheWrapper.cpp
#	src/Disks/S3/DiskS3.cpp
 2021-01-18 13:39:49 +03:00
Alexey Milovidov
24c8e53440 Merge branch 'master' into multiple-nested 2021-01-16 16:28:40 +03:00
Alexey Milovidov
8276a1c8d2 Faster parts removal, more safe and efficient interface of IDisk 2021-01-14 19:24:13 +03:00
Pavel Kovalenko
b09862b7b9 Ability to backup-restore metadata files for DiskS3 (fixes and tests) 2021-01-12 20:18:40 +03:00
Anton Popov
36ae0e4d35 Merge remote-tracking branch 'upstream/master' into HEAD 2021-01-11 13:51:12 +03:00
Alexey Milovidov
6eb5a5f4d9 Remove useless code 2021-01-10 03:28:59 +03:00
Azat Khuzhin
b1f08f5c27 Rename FileSyncGuard to DirectorySyncGuard 2021-01-07 20:26:18 +03:00
Anton Popov
11283e3d81 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-25 21:25:59 +03:00
alexey-milovidov
491f481713
Merge pull request #18481 from ClickHouse/disable_write_with_aio 
Disable write with AIO even for big merges
 2020-12-25 04:24:43 +03:00
alesapin
5d94b7eec0 Remove non working code 2020-12-24 19:30:53 +03:00
Anton Popov
518cac6150 fix for review 2020-12-23 13:11:24 +03:00
Anton Popov
78c4b9f2bc do not allow merge from wide to compact parts 2020-12-23 03:16:15 +03:00
Anton Popov
b6ff6300b2 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-22 18:06:21 +03:00
Alexey Milovidov
9be5fa9ef2 Merge branch 'master' into Enmk-Optimize_deduplicate 2020-12-20 09:57:10 +03:00
Anton Popov
a42b00c9aa Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-17 20:43:23 +03:00
alesapin
849db47f8a Better exception messages 2020-12-16 17:31:17 +03:00
Vasily Nemkov
59fc301344 Fixed test to be less flaky 
Also logging expanded list of columns passed from `DEDUPLICATE BY` to actual deduplication routines.
 2020-12-08 19:44:34 +03:00
Anton Popov
d7aad3bf79 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-07 19:13:01 +03:00
Vasily Nemkov
70ea507dae OPTIMIZE DEDUPLICATE BY columns 
Extended OPTIMIZE ... DEDUPLICATE syntax to allow explicit (or implicit with asterisk/column transformers) list of columns to check for duplicates on.

Following syntax variants are now supported:

OPTIMIZE TABLE table DEDUPLICATE; -- the old one
OPTIMIZE TABLE table DEDUPLICATE BY *;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT (colX, colY);
OPTIMIZE TABLE table DEDUPLICATE BY col1,col2,col3;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex');
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT (colX, colY);

Note that * behaves just like in SELECT: MATERIALIZED, and ALIAS columns are not used for expansion.
Also, it is an error to specify empty list of columns, or write an expression that results in an empty list of columns, or deduplicate by an ALIAS column.
Column transformers other than EXCEPT are not supported.
 2020-12-07 09:44:07 +03:00
Pavel Kruglov
9dbced0474 Pass setting instead of context 2020-12-04 17:01:59 +03:00
Anton Popov
cd1917c7a6
Merge branch 'master' into optimize_final_optimization 2020-12-03 16:52:51 +03:00
Anton Popov
b384beb564 Merge remote-tracking branch 'upstream/master' into HEAD 2020-11-23 17:46:51 +03:00
Pavel Kruglov
ca3fe49a2a Make setting global 2020-11-20 17:29:13 +03:00
Nicolae Vartolomei
040aba9f85 Add uuid.txt to checksums for parts stored on disk 
We are breaking backwards compatibility anyway (but agted by a setting)
 2020-11-20 13:49:17 +00:00
Nicolae Vartolomei
94293ca3ce Assign UUIDs to parts only when configured to do so 
Avoid breaking backwards compatibility by default for now.
 2020-11-20 13:49:17 +00:00
Pavel Kruglov
4c30857759 Minor change 2020-11-20 01:22:40 +03:00
Nicolae Vartolomei
746f8e45f5 All new parts must have uuids 2020-11-19 13:18:03 +00:00
Nicolae Vartolomei
425dc4b11b Add unique identifiers IMergeTreeDataPart structure 
For now uuids are not generated at all, they are present only if the
part is updated manually (as you can see in the integration test).

The only place where they can be seen today by an end user is in
`system.parts` table. I was looking for hiding this column behind an
option but couldn't find an easy way to do that.

Likely this is also required for WAL, but need to think how not to break
compatibility.

Relates to #13574, https://github.com/ClickHouse/ClickHouse/issues/13574

Next 1: In the upcoming PR the plan is to integrate de-duplication based on
these fingerprints in the query pipeline.

Next 2: We'll enable automatic generation of uuids and come up with a
way for conditionally sending uuids when processing distributed queries
only when part movement is in progress.
 2020-11-19 13:14:25 +00:00
Pavel Kruglov
c648f62629 Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize_final_optimization 2020-11-10 22:58:21 +03:00
Pavel Kruglov
9120189d8a Add SelectPartsDecision enum class 2020-11-10 17:42:56 +03:00
Anton Popov
245c395a68 Merge remote-tracking branch 'upstream/master' into HEAD 2020-11-06 22:00:32 +03:00
alesapin
40fc512e79 Merge branch 'master' into no_background_pool_no_more 2020-10-29 12:53:34 +03:00
alesapin
8e8bdeb5d7
Merge pull request #16434 from ClickHouse/fix_fake_race_on_merges_list 
Fix fake race condition on system.merges merge_algorithm
 2020-10-28 15:51:53 +03:00
alesapin
60f2d822d7 Fix fake race condition on system.merges merge_algorithm 2020-10-27 18:27:12 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell 2020-10-27 12:04:03 +01:00
Anton Popov
a249f0c95e Merge remote-tracking branch 'upstream/master' into HEAD 2020-10-23 22:05:00 +03:00
Anton Popov
06dc0155e5 allow to read subhcolumns from other storages 2020-10-23 17:44:23 +03:00
alesapin
7a63b373f9 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 15:47:11 +03:00
alesapin
4b65d5469f Fix some unrelated performance issues in select parts for merge 2020-10-22 15:41:01 +03:00
Vladimir Chebotarev
aa5f207fd4
Added disable_merges option for volumes in multi-disk configuration (#13956) 
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-10-20 18:10:24 +03:00
Pavel Kruglov
d6fc6d35fa Fix style 2020-10-15 19:08:24 +03:00
Pavel Kruglov
25c39689c3 Check expired ttl 2020-10-15 15:15:02 +03:00
alesapin
dccf51b316 Remove redundant logging 2020-10-15 14:08:39 +03:00
alesapin
4014e0f08d Something very similar to working code 2020-10-14 15:32:35 +03:00
alesapin
efd3126b5d Moving pool 2020-10-14 10:22:48 +03:00
Pavel Kruglov
51807599e7 fix style 2020-10-13 23:10:37 +03:00
Pavel Kruglov
1ccf10505a add tests and comments 2020-10-13 21:26:40 +03:00
Pavel Kruglov
6d9b310f61 Add optimize_skip_merged_partitions 2020-10-13 21:26:35 +03:00
alesapin
fd35368c59 First simple implementation 2020-10-13 17:25:42 +03:00
Pervakov Grigorii
3a2068f19d Use tmp disk for vertical merge files 2020-10-08 13:49:41 +03:00
alesapin
387b97aba5 Merge branch 'master' into codecs_for_nested_types 2020-09-21 11:06:28 +03:00
alexey-milovidov
4e77c2e48e
Update MergeTreeDataMergerMutator.cpp 2020-09-19 01:29:31 +03:00
Amos Bird
d842cb704f
Allow mutations to work with key constraints. 2020-09-19 02:40:02 +08:00
alesapin
e96a3ac5f3 DataType in enumerate streams 2020-09-18 14:37:58 +03:00
alesapin
6f8da7dfdf Merge branch 'master' into fix_compression_codec_perf 2020-09-15 09:59:45 +03:00
alesapin
e96256d367 Trying to improve performance 2020-09-14 22:15:25 +03:00
Nikita Mikhaylov
ddea62211d
Merge pull request #11948 from CurtizJ/fsync-parts 
Experiment. Allow to turn on fsync on inserts, merges and fetches.
 2020-09-14 22:37:14 +04:00
Amos Bird
c2d79bc5cc
Add merge_algorithm to system.merges 2020-09-13 10:00:03 +08:00
Anton Popov
af4089c606 Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-11 00:39:21 +03:00
alesapin
f274ffc9d1 Add comments 2020-09-07 10:59:14 +03:00
Anton Popov
3b9be70c8c Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-05 18:53:55 +03:00
alesapin
79ef72178e Less garbage 2020-09-04 17:18:49 +03:00
alesapin
f48d654d35 Less garbage 2020-09-04 17:15:34 +03:00
alesapin
fecb2f1311 Less copypaste 2020-09-04 17:08:43 +03:00
alesapin
f8aa6cfe8a
Merge pull request #14438 from ClickHouse/apply_ttl_if_not_calculated 
Apply TTL if it's not calculated for part
 2020-09-04 15:32:50 +03:00
alesapin
4079636f81 Merge 2020-09-04 13:52:51 +03:00
alesapin
61ecaebcb1 Simplify settings for TTL merges 2020-09-04 09:55:19 +03:00
alesapin
15a02321bf Merge branch 'fix_select_parts_for_merge' into control_ttl_merges_in_pool 2020-09-03 17:16:29 +03:00
alesapin
e48e4309b2 Better names 2020-09-03 16:29:18 +03:00
alesapin
565555e55c Fix bug in parts selection 2020-09-03 16:04:28 +03:00
alesapin
13248a744b Fix bug in parts selection 2020-09-03 16:02:24 +03:00
alesapin
f4c7ff0376 Add fixed size of Merge TTLS 2020-09-03 16:00:13 +03:00
alesapin
aa47d0aabc Merge branch 'apply_ttl_if_not_calculated' into recompression_in_background 2020-09-03 12:07:03 +03:00
alesapin
acc0ee0657 Apply TTL if it's not calculated for part 2020-09-03 11:59:41 +03:00
alesapin
dcbddbb8d0 Add recompression ttls tests and fix bugs 2020-09-02 19:15:41 +03:00
alesapin
128cb7ce22 Don't select already selected parts 2020-09-02 15:16:12 +03:00
alesapin
aac466ab1c Merge branch 'better_ttl_merges_selection' into recompression_in_background 2020-09-02 13:30:04 +03:00
alesapin
fbb37c37df Simplier interface 2020-09-02 11:28:46 +03:00
alesapin
b08056fa8c Better selection of Merges with TTL 2020-09-02 11:18:50 +03:00
Anton Popov
4fe57fc169 Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-02 01:29:44 +03:00
alesapin
c37a19f7b0 Better 2020-09-01 14:23:38 +03:00
alesapin
f0dc5a3085 First working test 2020-09-01 13:49:53 +03:00
alesapin
05b10048a6
Merge pull request #14116 from ClickHouse/parts_default_compression 
Save parts default compression
 2020-09-01 09:43:42 +03:00
alesapin
46f833b7df Some changes 2020-08-31 22:50:42 +03:00
alesapin
034f1a895d Merge branch 'master' into recompression_in_background 2020-08-31 11:10:14 +03:00
alesapin
232c264d7d Disable force TTL on optimise 2020-08-28 16:45:42 +03:00
alesapin
2fc80189af Add default compression codec to merge tree data part 2020-08-26 18:29:46 +03:00
Anton Popov
4df5c72878 Merge remote-tracking branch 'upstream/master' into HEAD 2020-08-24 12:57:27 +03:00
alexey-milovidov
7b53a0ef33 Revert "Added allow_merges option for volumes in multi-disk configuration (#13402)" 
This reverts commit 1e2616542a.
 2020-08-21 18:44:29 +03:00
Vladimir Chebotarev
1e2616542a
Added allow_merges option for volumes in multi-disk configuration (#13402) 2020-08-21 12:04:13 +03:00
Nikolai Kochetov
9b67cd9faf Merge branch 'master' into refactor-pipes-3 2020-08-10 10:50:17 +03:00
Alexey Milovidov
edd89a8610 Fix half of typos 2020-08-08 03:47:03 +03:00
Nikolai Kochetov
d83c7cc0e6 Merge branch 'master' into refactor-pipes-3 2020-08-06 17:32:50 +03:00
Nikolai Kochetov
20e63d2271 Refactor Pipe [part 6] 2020-08-06 15:24:05 +03:00
Alexey Milovidov
d3c0635fc0 Merge branch 'master' into stack-frams-size-limit 2020-08-02 04:35:09 +03:00
Nikolai Kochetov
39530f837e Remove TreeExecutorBlockInputStream. 2020-07-31 16:23:19 +03:00
Alexey Milovidov
c491b2c153 Lower stack frame size 2020-07-30 23:58:08 +03:00
alexey-milovidov
4c3241808d
Merge pull request #12728 from ClickHouse/prevent-inconsistent-merge-tree-settings 
Safe logic for incorrect MergeTreeSettings
 2020-07-30 17:55:39 +03:00
Alexander Kazakov
2bde393499
Merging #12548 - Correction to merge_with_ttl_timeout logic by @excitoon (#12982) 
* Fixed `merge_with_ttl_timeout` logic.
* Separate TTL-merge timers for each partition

Co-authored-by: Vladimir Chebotarev <vladimir.chebotarev@gmail.com>
 2020-07-28 17:38:34 +03:00
Alexey Milovidov
798ef1b9f9 Adjustments; added test result 2020-07-27 23:18:23 +03:00
alesapin
1707f84a44 Less strict check and rare rename bug 2020-07-27 12:42:37 +03:00
alesapin
e0bf5913e7 Fix sticking mutations 2020-07-26 17:21:57 +03:00
Alexey Milovidov
6b4be2b9fe Safe logic for incorrect MergeTreeSettings #10897 2020-07-23 15:31:09 +03:00
Artem Zuikov
2afd123eda
Refactoring: extract TreeOptimizer from SyntaxAnalyzer (#12645) 2020-07-22 20:13:05 +03:00
Nikolai Kochetov
cab9146ff3 Remove SortDescription from IBlockInputStream. 2020-07-21 11:05:52 +03:00
alesapin
84f8bf1e14
Merge pull request #10697 from CurtizJ/polymorphic-parts 
Polymorphic parts (in-memory format)
 2020-07-08 13:25:45 +03:00
Nikita Mikhaylov
d31ed58f01 done 2020-07-06 17:33:31 +03:00
Anton Popov
ca346ea13c rename fsync-related settings 2020-07-03 02:41:37 +03:00
Anton Popov
b2aa565a37 allow to turn on fsync on inserts, merges and fetches 2020-06-27 17:52:16 +03:00