thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 03:42:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
54,670 Commits 9,315 Branches 1,783 Tags 2.1 GiB
13e4579052
Commit Graph

2592 Commits

Author SHA1 Message Date
alesapin
dca35b1913 Add awesome test for fetch 2020-10-27 15:24:10 +03:00
kssenii
a48bc83763 Fix style 2020-10-27 12:04:07 +00:00
Pavel Kruglov
331545fe0d Fix typo 2020-10-27 14:29:36 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell 2020-10-27 12:04:03 +01:00
kssenii
db03cd7dd7 Much more optimal queues setup 2020-10-27 11:00:01 +00:00
kssenii
7beddceadd Fix bug that caused awful CPU usage 2020-10-27 10:02:27 +00:00
kssenii
cd0dbcb9d8 Minimize event loop lifetime 2020-10-27 10:01:39 +00:00
alesapin
d6130f13ef Add table to system tables 2020-10-27 10:21:49 +03:00
alesapin
880f4bbd05 System fetches 2020-10-26 19:38:35 +03:00
Alexander Kuzmenkov
5a70e26c70 Merge remote-tracking branch 'origin/master' into HEAD 2020-10-26 19:21:13 +03:00
alesapin
9ed4668dbb Refactor common part of background list 2020-10-26 15:40:55 +03:00
alesapin
47ebc9f74e Add separate pool for fetches 2020-10-26 14:02:47 +03:00
alesapin
d5d0819698 Merge branch 'master' into no_background_pool_no_more 2020-10-26 12:58:51 +03:00
alexey-milovidov
3aa49204d6
Update MergeTreeSettings.cpp 2020-10-26 07:39:20 +03:00
hchen9
d9bc486c0e Fix StorageJoin and StorageSet 2020-10-25 20:57:19 -07:00
hchen9
c25a3e4e7b Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into trivial_count_seq 2020-10-25 20:51:09 -07:00
Alexey Milovidov
d5e277f080 Better exceptions rethrow 2020-10-26 06:35:57 +03:00
hchen9
824c04f873 Fix conflict code 2020-10-25 18:12:17 -07:00
Alexey Milovidov
22a89fca01 Fix processing very large entries in queue 2020-10-25 05:57:03 +03:00
Alexey Milovidov
a2d888f19f Fix processing very large entries in queue 2020-10-25 05:57:03 +03:00
Ivan
1d170f5745
ASTTableIdentifier Part #1: improve internal representation of ASTIdentifier name (#16149) 
* Use only |name_parts| as primary name source

* Restore legacy logic for table restoration

* Fix build

* Fix tests

* Add pytest server config

* Fix tests

* Fixes due to review
 2020-10-24 21:46:10 +03:00
alexey-milovidov
c579611420
Merge pull request #16306 from azat/Set-Join-total-rows-bytes 
Add total_rows/total_bytes support for Set/Join
 2020-10-24 07:26:32 +03:00
alexey-milovidov
6037982343
Merge pull request #16284 from ClickHouse/primary_key_without_order_by 
Add ability to specify PRIMARY KEY without ORDER BY
 2020-10-24 06:05:04 +03:00
Alexey Milovidov
13b8dce792 Update contributors 2020-10-24 00:26:22 +03:00
hchen9
feaab260d8 Fix conflict 2020-10-23 12:54:33 -07:00
Azat Khuzhin
4d1385ef19 Add total_rows/total_bytes support for Set/Join 2020-10-23 21:11:55 +03:00
alesapin
bc331a8029 Better set 2020-10-23 19:34:41 +03:00
alesapin
b28894c43b More compatible way 2020-10-23 19:31:51 +03:00
alesapin
59ca0d071c Fix comment 2020-10-23 14:17:37 +03:00
alesapin
69f2f9838e Add ability to specify PRIMARY KEY without ORDER BY 2020-10-23 14:16:27 +03:00
alesapin
39e47c5338 Less strange lambdas 2020-10-23 11:54:00 +03:00
Alexander Kuzmenkov
5cbf645de4 Merge remote-tracking branch 'origin/master' into tmp 2020-10-22 19:48:02 +03:00
alesapin
3c559c699f Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 17:16:46 +03:00
alesapin
1d07ece5d1 Fix clang tidy warning 2020-10-22 17:15:56 +03:00
alesapin
7a63b373f9 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 15:47:11 +03:00
alesapin
4b65d5469f Fix some unrelated performance issues in select parts for merge 2020-10-22 15:41:01 +03:00
Nikolai Kochetov
a42fd18390 Try fix tests. 2020-10-22 15:01:22 +03:00
Nikolai Kochetov
7fa045cff8 Merge branch 'master' into storage-read-query-plan 2020-10-22 13:31:10 +03:00
alesapin
945dd3aa59 Merge branch 'master' into blind_fix_of_read_in_order 2020-10-22 10:37:37 +03:00
alexey-milovidov
adeba6bdd8
Merge pull request #15074 from amosbird/btc 
Extend trivial count optimization.
 2020-10-22 02:50:57 +03:00
Alexander Tokmakov
ac42199ae5 fix race on StorageReplicatedMergeTree::data_parts_exchange_endpoint 2020-10-21 22:25:22 +03:00
Kruglov Pavel
0b75885c8a
Merge branch 'master' into select_final 2020-10-21 20:42:39 +03:00
Pavel Kruglov
89fdeb4e15 Fix style, move setting and add checking level>0 2020-10-21 20:35:31 +03:00
Nikolai Kochetov
a0da588f06 Merge branch 'master' into new-block-for-functions-4 2020-10-21 16:16:58 +03:00
alexey-milovidov
0b7430dda1
Merge pull request #16206 from ClickHouse/fix-memory-tracking 
Fix multiple issues with memory tracking
 2020-10-21 14:34:22 +03:00
alesapin
c53f59dece
Merge pull request #15984 from ClickHouse/fix_insert_mv_dangling_ref 
Fix possibly dangling reference to target table of MV
 2020-10-21 12:13:51 +03:00
alesapin
11ea6d7911 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-21 11:39:48 +03:00
alesapin
b45d42bb9c Maybe devirtualization can help 2020-10-21 11:35:36 +03:00
alesapin
9f47c82bbd Sleep a little when pool is full 2020-10-21 11:31:57 +03:00
alesapin
b1a13d4884 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-21 10:30:09 +03:00
Nikolai Kochetov
68c9a57788 Merge branch 'master' into new-block-for-functions-4 2020-10-21 09:17:25 +03:00
Alexey Milovidov
2619efadc8 Fix multiple issues with memory tracking 2020-10-21 03:31:34 +03:00
alexey-milovidov
8084ce75cb
Merge pull request #16105 from azat/allow_nondeterministic_optimize_skip_unused_shards 
Add allow_nondeterministic_optimize_skip_unused_shards
 2020-10-21 00:16:26 +03:00
alesapin
c77b154398 Better locks in merge tree 2020-10-21 00:10:55 +03:00
alesapin
8097c696de Trying another fix 2020-10-20 23:05:20 +03:00
alexey-milovidov
75a02112bc
Merge pull request #15592 from sundy-li/readonly-metrics 
sub ReadonlyReplica metric when detach readonly tables
 2020-10-20 19:17:52 +03:00
alesapin
439bbd9861 Blind performance fix 2020-10-20 18:30:46 +03:00
Vladimir Chebotarev
aa5f207fd4
Added disable_merges option for volumes in multi-disk configuration (#13956) 
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-10-20 18:10:24 +03:00
Nikolai Kochetov
a1d5adf8d5 Merge branch 'master' into new-block-for-functions-4 2020-10-20 16:16:08 +03:00
alesapin
30ca17b09b Comment to settings 2020-10-20 16:12:25 +03:00
alesapin
7256d01579 Normal backoff value instead of 10 seconds 2020-10-20 16:11:01 +03:00
Alexander Kuzmenkov
84908df6d8 fix setting parent from HTTP traceparent 2020-10-20 14:35:13 +03:00
alesapin
ce8ee01d75 Fix background moves start 2020-10-20 14:27:50 +03:00
alesapin
d126aa8a09 Merge branch 'master' into no_background_pool_no_more 2020-10-20 12:54:42 +03:00
alexey-milovidov
5e46e7c670
Merge pull request #15723 from Jokser/idempotent-part-move 
Possibility to move part to another disk/volume if first attempt was failed
 2020-10-20 04:26:50 +03:00
alexey-milovidov
124379cccc
Update IMergeTreeDataPart.h 2020-10-20 04:24:30 +03:00
alexey-milovidov
26517ff08d
Update IMergeTreeDataPart.h 2020-10-20 04:23:23 +03:00
Nikolai Kochetov
bc58637ec2 Fixing build. 2020-10-19 21:37:44 +03:00
Alexander Kuzmenkov
6dcfd26b29 Merge remote-tracking branch 'origin/master' into tmp 2020-10-19 21:36:30 +03:00
Alexander Kuzmenkov
defcd29b31
Merge pull request #16036 from ClickHouse/stress_test_improvemets 
Minor improvements of stress test
 2020-10-19 16:12:30 +03:00
Nicolae Vartolomei
af76ff1dbf RFC: Add metadata section to MergeTreeWriteAheadLog 
Current WAL format doesn't seem to be extendable without breaking
reverse compatibility.

This PR proposes to introduces a metadata section
for each action in the WAL to which optional fields can be added.

Adding new fields is trivial and doesn't have to break backwards
compatibility if these are not critical for correctness.

Metadata contains the minimum version of the reader allowed
to interpret it.

Removing fields is not supported.
 2020-10-19 12:16:31 +01:00
alesapin
514a95c292 Fixup 2020-10-19 12:06:51 +03:00
alesapin
cb139abe2a Better schedule after job already running 2020-10-19 11:47:15 +03:00
alesapin
e09713ece1 Merge branch 'master' into no_background_pool_no_more 2020-10-19 10:58:31 +03:00
alesapin
963bc57835
Merge pull request #16011 from ClickHouse/fix_alter_add_and_modify_order_by 
Fix alter sorting key for ReplicatedVersionedCollapsingMergeTree
 2020-10-19 10:31:13 +03:00
Azat Khuzhin
9b8abd44ab Add allow_nondeterministic_optimize_skip_unused_shards 2020-10-17 01:07:02 +03:00
Alexander Tokmakov
c19d2efd88 add more holders 2020-10-16 20:16:58 +03:00
alesapin
518ba2ceb2 Safe random generator 2020-10-16 16:48:12 +03:00
Alexander Tokmakov
026896835c fixes 2020-10-16 15:20:30 +03:00
alesapin
c22c760ac1 Fix unit test build 2020-10-16 13:18:56 +03:00
alesapin
7edbe4d276 Add some comments 2020-10-16 13:12:31 +03:00
Pavel Kovalenko
a474bdcabf Merge remote-tracking branch 'origin/master' into idempotent-part-move 2020-10-16 13:02:40 +03:00
alesapin
0fb1f74b2e Trying to schedule better 2020-10-16 11:00:15 +03:00
alesapin
280d0c55c2 Merge branch 'master' into no_background_pool_no_more 2020-10-16 10:25:02 +03:00
Alexey Milovidov
6c3aad34be Avoid stack overflow in materialized views, part 2: sanity check 2020-10-16 03:19:45 +03:00
Alexey Milovidov
ffd3d2519e Avoid stack overflow in materialized views, part 1 2020-10-16 03:13:17 +03:00
Kruglov Pavel
5aba639430 Update test 2020-10-15 22:01:18 +03:00
sundy-li
153be93544 sub ReadonlyReplica when detach readonly tables 2020-10-15 22:01:18 +03:00
Pavel Kruglov
6c1e0e3df8 Change default to false 2020-10-15 19:40:20 +03:00
alesapin
4c1f5cab13 Sleep less in background task 2020-10-15 19:10:22 +03:00
Pavel Kruglov
d6fc6d35fa Fix style 2020-10-15 19:08:24 +03:00
Pavel Kruglov
ee8a9d2cf1 Don't touch MySQL database if it's unnecessary 2020-10-15 18:57:17 +03:00
Alexander Kuzmenkov
bc7b5f6bb4 Merge remote-tracking branch 'origin/master' into tmp 2020-10-15 17:15:05 +03:00
Vitaly Baranov
5f261cec3c
Merge pull request #15199 from filimonov/non_delimited_protobuf 
ProtobufSingle format
 2020-10-15 16:58:39 +03:00
alesapin
629506445d Better backoff policy 2020-10-15 16:57:50 +03:00
Pavel Kovalenko
ed61c5681b Use 'moving' directory instead of 'detached' when move part to another disk/volume. 2020-10-15 16:55:13 +03:00
alesapin
60d14352f8 Merge branch 'master' into no_background_pool_no_more 2020-10-15 16:34:14 +03:00
Alexander Kuzmenkov
f366b366ae
Merge pull request #15987 from filimonov/background-pool-count_no_work_done-bug 
Fix the bug when NOTHING_TO_DO events wrongly increment count_no_work_done
 2020-10-15 16:11:57 +03:00
alesapin
2002289003 Fix bug in alter primary key for replicated versioned collapsing merge tree 2020-10-15 16:02:39 +03:00
Alexander Kuzmenkov
b4378b004f Merge remote-tracking branch 'origin/master' into tmp 2020-10-15 16:01:41 +03:00
Pavel Kruglov
f5fac575f4 don't postprocess single parts 2020-10-15 15:22:41 +03:00
Pavel Kruglov
25c39689c3 Check expired ttl 2020-10-15 15:15:02 +03:00
tavplubix
1003de75ed
Update StorageMaterializedView.cpp 2020-10-15 15:06:44 +03:00
alesapin
dccf51b316 Remove redundant logging 2020-10-15 14:08:39 +03:00
tavplubix
6dc2f00347
Update StorageMaterializedView.cpp 2020-10-15 13:57:47 +03:00
alesapin
84b0c22044 fix possible deadlock 2020-10-15 13:54:50 +03:00
alesapin
374949051c Add thread safety 2020-10-15 13:41:36 +03:00
alesapin
e84eda176a Add test and slightly refactored 2020-10-15 13:22:02 +03:00
alesapin
583d7042e9 Get rid of some deadlocks 2020-10-15 11:29:18 +03:00
Nikolai Kochetov
e8038ae84f
Merge pull request #15857 from amosbird/e2 
Refactor processors.
 2020-10-15 10:45:00 +03:00
alesapin
c5419083e3 Better method name 2020-10-15 10:43:50 +03:00
alesapin
3394c22c03 Fix race and clang-tidy 2020-10-15 10:39:27 +03:00
Mikhail Filimonov
ace2e99cb3
Fix the bug when NOTHING_TO_DO events wrongly increment count_no_work_done 2020-10-14 23:08:47 +02:00
alesapin
3e3505258f No sleep when we have job 2020-10-14 22:33:36 +03:00
alesapin
ae8ed99805 Missed change 2020-10-14 21:35:03 +03:00
alesapin
7265792510 Less files 2020-10-14 21:33:41 +03:00
alesapin
a535a089d1 Some style fixes 2020-10-14 19:09:18 +03:00
alesapin
fc25b6cd72 Remove background processing pool 2020-10-14 18:26:00 +03:00
alesapin
6a4911dc57 Remove some garbage 2020-10-14 18:05:14 +03:00
alesapin
483893cdd4 Slightly refactored code 2020-10-14 17:56:42 +03:00
Vitaly Baranov
997570780d
Merge branch 'master' into non_delimited_protobuf 2020-10-14 16:26:57 +03:00
alesapin
0b14a31ba9 Get rid of background processing pool 2020-10-14 15:44:10 +03:00
alesapin
4014e0f08d Something very similar to working code 2020-10-14 15:32:35 +03:00
Alexander Tokmakov
72b1339656 Revert "Revert "Write structure of table functions to metadata"" 
This reverts commit c65d1e5c70.
 2020-10-14 15:19:29 +03:00
tavplubix
c65d1e5c70
Revert "Write structure of table functions to metadata" 2020-10-14 13:59:29 +03:00
alesapin
efd3126b5d Moving pool 2020-10-14 10:22:48 +03:00
alexey-milovidov
f60ccb4edf
Merge pull request #14295 from ClickHouse/write_structure_of_table_functions 
Write structure of table functions to metadata
 2020-10-13 23:56:09 +03:00
Pavel Kruglov
51807599e7 fix style 2020-10-13 23:10:37 +03:00
Pavel Kruglov
44c2b138f3 Fix style 2020-10-13 22:53:36 +03:00
Pavel Kruglov
be0cb31d21 Add tests and comments 2020-10-13 21:55:03 +03:00
Pavel Kruglov
1ccf10505a add tests and comments 2020-10-13 21:26:40 +03:00
Pavel Kruglov
6d9b310f61 Add optimize_skip_merged_partitions 2020-10-13 21:26:35 +03:00
Pavel Kruglov
8200bab859 Add setting do_not_merge_across_partitions 2020-10-13 17:54:52 +03:00
alesapin
fd35368c59 First simple implementation 2020-10-13 17:25:42 +03:00
alesapin
dcf489ac8f Remove unused methods 
(cherry picked from commit 77f5841a46)
 2020-10-13 15:10:59 +03:00
alesapin
ef355e71d6 Buildable code 2020-10-13 15:10:46 +03:00
alesapin
48aaa0ce16 Split merge to separate method 
(cherry picked from commit 14a7a25c4b)
 2020-10-13 15:06:32 +03:00
alesapin
cb7db6d232 Split select and process merges 
(cherry picked from commit 1889eb0ff1)
 2020-10-13 15:04:16 +03:00
Nikita Mikhaylov
835c480056
Merge pull request #15858 from ClickHouse/fix_alter_default_validation 
Fix alter default validation
 2020-10-13 12:47:18 +03:00
tavplubix
6c1d59cd49
Merge pull request #15743 from azat/Atomic-DROP-TABLE-hang-fix 
Fix DROP TABLE hangs with Atomic engine
 2020-10-13 12:40:36 +03:00
alesapin
321a7ae6bf
Merge pull request #15852 from ClickHouse/fix_reverse_processor 
Fix block structure mismatch after ALTER
 2020-10-13 09:49:26 +03:00
Amos Bird
ca7a5ccd0d
More resize 2020-10-13 09:56:35 +08:00
Azat Khuzhin
bba44d0f0e Add IF EXISTS for removing inner table of the materialized view 
Instead of playing with has_inner_table flag, which looks a bit icky.
 2020-10-12 21:46:07 +03:00
Azat Khuzhin
d16adddb41 Fix drop of materialized view with inner table in Atomic database 
Materialized view drop the inner table (if any) drop the drop() method,
and this will lead to essure recursive drop() from the drop worker
thread, which will hang:

    3  std::__1::condition_variable::wait<> ()
    4  DB::DatabaseCatalog::waitTableFinallyDropped (this=0x7ffff7831500, uuid=...) at ../src/Interpreters/DatabaseCatalog.cpp:828
    5  DB::InterpreterDropQuery::executeToTable (this=this@entry=0x7fff16ff88a0, table_id_=..., query=...) at ../src/Interpreters/InterpreterDropQuery.cpp:135
    6  DB::InterpreterDropQuery::execute (this=this@entry=0x7fff16ff88a0) at ../contrib/libcxx/include/string:1474
    7  DB::executeDropQuery (kind=kind@entry=DB::ASTDropQuery::Drop, global_context=..., target_table_id=...) at ../src/Storages/StorageMaterializedView.cpp:156
    8  DB::StorageMaterializedView::drop (this=0x7ffefc348f40) at ../src/Storages/StorageMaterializedView.cpp:169
    9  DB::DatabaseCatalog::dropTableFinally (this=this@entry=0x7ffff7831500, table=...) at ../src/Interpreters/DatabaseCatalog.cpp:775
    10 DB::DatabaseCatalog::dropTableDataTask (this=0x7ffff7831500) at ../src/Interpreters/DatabaseCatalog.cpp:745

Fix this by dropping the inner table just before scheduling job the the
drop worker thread.
 2020-10-12 21:46:07 +03:00
Alexander Kuzmenkov
6200433e12 Merge origin/master into tmp (using imerge) 2020-10-12 21:15:31 +03:00
Mikhail Filimonov
87a8ba63f7
Few more CR fixes 2020-10-12 15:05:40 +02:00
tavplubix
748b75ab1b
Merge pull request #15767 from ClickHouse/fix_materialize_mysql_count_v2 
Try fix select count() for MaterializeMySQL (another way)
 2020-10-12 14:41:38 +03:00
Nikolai Kochetov
2a62a91af4
Merge pull request #15662 from ClickHouse/fix-15529 
Fix read from Buffer table with empty nested table with different types.
 2020-10-12 14:12:11 +03:00
alesapin
6732e5c217 Fix alter default validation 2020-10-12 14:02:35 +03:00
Nikolai Kochetov
7e58f99f64 Merge branch 'master' into storage-read-query-plan 2020-10-12 13:12:39 +03:00
Amos Bird
47fcd8bffb
Fix some problems introduced by refactoring. 2020-10-12 17:58:09 +08:00
Amos Bird
d2dcfc3f0d
Refactor processors. 2020-10-12 17:30:05 +08:00
alesapin
5b5dd4fb28 Fix order by desc after alter 2020-10-12 11:28:47 +03:00
Mikhail Filimonov
f6b00f2cb6
After CR fixes 2020-10-12 07:52:09 +02:00
Mikhail Filimonov
6df42c580b
ProtobufSingle output format 2020-10-12 07:51:45 +02:00
feng lv
8d1780c725 Merge branch 'master' of github.com:ClickHouse/ClickHouse into add-mutation-for-storagememory 2020-10-11 20:54:15 +08:00
alexey-milovidov
10e9d14466
Merge pull request #15818 from ClickHouse/style-pragma-once 
Check for #pragma once in headers
 2020-10-11 13:14:09 +03:00
alexey-milovidov
1187903b44
Merge pull request #15813 from ClickHouse/remove-useless-metrics 
Remove useless metrics to avoid confusion
 2020-10-11 01:11:17 +03:00
Alexey Milovidov
269b6383f5 Check for #pragma once in headers 2020-10-10 21:37:02 +03:00
Alexey Milovidov
5b482f4191 Cleanups 2020-10-10 19:31:10 +03:00
Alexey Milovidov
06fcf437fe Addition to prev. revision 2020-10-10 19:02:13 +03:00
Alexey Milovidov
9f15daf45c Remove useless metrics to avoid confusion 2020-10-10 18:52:44 +03:00
robot-clickhouse
554fa482e6 Auto version update to [20.11.1.1] [54442] 2020-10-10 18:39:58 +03:00
Nikolai Kochetov
76a04fb4b4
Merge pull request #15762 from ClickHouse/new-block-for-functions 
Use `ColumnsWithTypeAndName` instead of `Block` for function calls
 2020-10-10 08:50:38 +03:00
Nikita Mikhaylov
6ac5431804
Merge pull request #15783 from nikitamikhaylov/copier-backquotes 
Merging  #14939
 2020-10-10 02:32:27 +03:00
alexey-milovidov
0fd007ad9e
Merge pull request #15601 from alexelex/alexelex-master 
Improvement of Quorum Inserts in ClickHouse
 2020-10-09 22:43:53 +03:00
alesapin
35165c8ea4
Merge pull request #15756 from CurtizJ/fix-prewhere-with-virtuals 
Fix prewhere with virtual columns
 2020-10-09 18:07:41 +03:00
nikitamikhaylov
88a5302759 Merge branch 'master' of github.com:ClickHouse/ClickHouse into copier-backquotes 2020-10-09 16:26:31 +03:00
Alexandra Latysheva
f6f33168aa fix compilation error 2020-10-09 12:11:21 +00:00
Alexandra Latysheva
0594a77b57 fix thread restart for parallel quorum inserts 2020-10-09 11:20:20 +00:00
Nikolai Kochetov
a7fb2e38a5 Use ColumnWithTypeAndName as function argument instead of Block. 2020-10-09 10:41:28 +03:00
Alexander Tokmakov
a6178cc876 try fix select count() for MaterializeMySQL 2020-10-08 23:39:24 +03:00
tavplubix
7b5f2b2bb0
Merge pull request #15701 from ClickHouse/better_clone_replica 
Improvements in lost replica recovery
 2020-10-08 23:19:23 +03:00
Alexandra Latysheva
6877f4aa9c for parallel and non-parallel quorum insertion at the same time 2020-10-08 15:35:41 +00:00
tavplubix
6065a5b65c
Update StorageReplicatedMergeTree.cpp 2020-10-08 16:37:32 +03:00
feng lv
677787ff0e fix 
fix
 2020-10-08 21:34:09 +08:00
Anton Popov
1a6d2bc538 fix prewhere with virtuals columns 2020-10-08 15:59:47 +03:00
feng lv
ae544940ce format 2020-10-08 19:16:53 +08:00
feng lv
ff1a314609 fix 2020-10-08 19:15:11 +08:00
Pervakov Grigorii
3a2068f19d Use tmp disk for vertical merge files 2020-10-08 13:49:41 +03:00
alexey-milovidov
c0cb18b445
Merge pull request #15740 from azat/kafka-memory-accounting 
Take memory allocations from librdkafka threads into account
 2020-10-08 13:12:54 +03:00
Amos Bird
867216103f
Extend trivial count optimization. 2020-10-08 18:08:17 +08:00
feng lv
3bcb9b8db5 fix 2020-10-08 17:18:56 +08:00
feng lv
3690294ea4 fix 2020-10-08 15:19:37 +08:00
hchen9
ed2ce4a6fc Resolve conflict in StorageMemory 2020-10-07 17:19:59 -07:00
Azat Khuzhin
1f51de362f Take memory allocations from librdkafka threads into account 2020-10-07 23:18:20 +03:00
alexey-milovidov
e465ce3d49
Merge pull request #15642 from azat/force_data_skipping_indices 
Implement force_data_skipping_indices setting
 2020-10-07 23:14:51 +03:00
Alexander Tokmakov
ffb19fa4bc fix 2020-10-07 22:47:31 +03:00
Azat Khuzhin
3cd71f3357 Cleanup interceptors for librdkafka in StorageKafka 
Wrap them into StorageKafkaInterceptors to allow access to private
fields and add logging inside interceptors if something fails.

This is also preparation for ThreadStatus interceptor.
 2020-10-07 21:42:13 +03:00
Nikolai Kochetov
54e77e906e
Merge pull request #15717 from ClickHouse/fix-array-join-optimisation-for-mv 
Fix array join optimisation for MV
 2020-10-07 20:19:35 +03:00
Latysheva Alexandra
f43ad338ed
Merge pull request #12 from alexelex/quorum_inserts_2 
style check
 2020-10-07 18:56:14 +06:00
Alexandra Latysheva
8f8872dc42 style check 2020-10-07 12:24:46 +00:00
Latysheva Alexandra
6c26f77380
Merge pull request #11 from alexelex/quorum_inserts_2 
changing the condition for updateQuorum, specialized it and reduced t…
 2020-10-07 17:49:20 +06:00
Pavel Kovalenko
69c126f1f1 Possibility to move part to another disk/volume if first attempt was failed. 2020-10-07 14:35:28 +03:00
Alexandra Latysheva
a43cac2c1a changing the condition for updateQuorum, specialized it and reduced the number of calls to the zk 2020-10-07 11:28:48 +00:00
Nikolai Kochetov
7e02152b50 Merge branch 'master' into storage-read-query-plan 2020-10-07 14:27:21 +03:00
Nikolai Kochetov
c5cb05f5f3 Try fix tests. 2020-10-07 14:26:29 +03:00
tavplubix
041015545c
Merge pull request #15537 from ClickHouse/mutation_hangs_after_replace_partition 
Fix mutation may hang after REPLACE/DROP PARTITION
 2020-10-07 14:25:29 +03:00
Nikolai Kochetov
df02573c92 Fix ARRAY JOIN optimisation when reading from MV. 2020-10-07 13:11:58 +03:00
Nikolai Kochetov
5f7aeddfe5
Merge pull request #15682 from ClickHouse/akz/storage_memory_less_locks 
Less locking in StorageMemory: totalRows + totalBytes
 2020-10-07 12:59:20 +03:00
alexey-milovidov
e00961f882
Merge pull request #15657 from Jokser/disk-s3-write-error-handling 
Proper error handling during insert into MergeTree with S3
 2020-10-07 11:17:32 +03:00
Alexander Tokmakov
47e9fe9ff1 fix 2020-10-07 03:05:48 +03:00
Alexander Tokmakov
ea16370024 improvements in lost replica recovery 2020-10-07 02:29:33 +03:00
Azat Khuzhin
75e612fc16 Use full featured parser for force_data_skipping_indices 2020-10-07 01:44:14 +03:00
Azat Khuzhin
ef6d12967f Implement force_data_skipping_indices setting 2020-10-07 01:42:31 +03:00
Latysheva Alexandra
991e51a80e
Merge pull request #10 from alexelex/quorum_inserts_2 
style check fix
 2020-10-07 04:38:30 +06:00
Alexandra Latysheva
9077c92862 style check fix 2020-10-06 22:36:42 +00:00
Latysheva Alexandra
85bcc07c0b
Merge pull request #9 from alexelex/quorum_inserts_2 
Quorum inserts [version 3]
 2020-10-07 04:04:41 +06:00
Alexandra Latysheva
644091459f style fixes and remove small changes 2020-10-06 22:01:30 +00:00
Alexandra Latysheva
f549ecf9d1 insert_quorum_parallel: status nodes in /quorum/parallel and allow merges 2020-10-06 21:49:48 +00:00
Alexander Kazakov
cabab90ab2 Get back to moving args 2020-10-06 17:54:22 +03:00
Alexander Kazakov
ae2c106f94 Minor fixes 2020-10-06 17:04:08 +03:00
Alexander Kazakov
e54ff6e60a Lock-less totalRows/totalBytes + more clear postponed init 2020-10-06 16:45:17 +03:00
Pavel Kovalenko
69b4bc6f33 Proper error handling during insert into MergeTree with S3. 2020-10-06 12:38:00 +03:00
Nikolai Kochetov
7caf6da363 Fix build. 2020-10-06 11:56:21 +03:00
Nikolai Kochetov
8320efc189 Fix read from Buffer table with empty nested table with different types. 2020-10-06 11:24:12 +03:00
Nikolai Kochetov
d968f276b5 Fix buffer table. 2020-10-06 11:21:05 +03:00
Nikolai Kochetov
f9bf1e3406 Merge branch 'master' into storage-read-query-plan 2020-10-06 09:50:10 +03:00
Latysheva Alexandra
d495a2fb2d
Merge pull request #7 from alexelex/quorum_inserts_2 
build checker fix (style) and change comments
 2020-10-06 02:33:29 +06:00
Alexandra Latysheva
41e0148d72 build checker fix (style) and change comments 2020-10-05 20:30:50 +00:00
Alexander Tokmakov
418766e92b fix another bug in mutations 2020-10-05 22:16:28 +03:00
Latysheva Alexandra
3b952aff7d
Merge pull request #6 from alexelex/quorum_inserts_2 
fix test_00814
 2020-10-05 23:01:18 +06:00
Alexandra Latysheva
2be62bbd1b fix test_00814 2020-10-05 16:58:26 +00:00
alesapin
8ec58c17f3 More consistent metadata usage 2020-10-05 19:41:46 +03:00
Alexander Tokmakov
30a18f7567 make it better 2020-10-05 16:52:03 +03:00
feng lv
55247bf3a9 Merge branch 'master' of github.com:ClickHouse/ClickHouse into add-mutation-for-storagememory 2020-10-05 21:48:34 +08:00
Latysheva Alexandra
18007622a1
Merge pull request #5 from alexelex/quorum_inserts_2 
style check fix
 2020-10-05 19:00:39 +06:00
Alexandra Latysheva
aa391a47eb style check fix 2020-10-05 12:59:07 +00:00
Latysheva Alexandra
7e7cedd73f
Merge pull request #4 from alexelex/quorum_inserts_2 
style check fix
 2020-10-05 16:43:14 +06:00
Alexandra Latysheva
0662432c5c style check 2020-10-05 10:41:49 +00:00
Latysheva Alexandra
a63084adaf
Merge pull request #3 from alexelex/quorum_inserts_2 
Quorum inserts 2
 2020-10-05 03:28:58 +06:00
Alexandra Latysheva
d54b6c9542 add question in comments 2020-10-04 20:32:08 +00:00
Alexandra Latysheva
ff8cbe6745 remove debug things 2020-10-04 20:19:05 +00:00
Alexandra Latysheva
8263e62298 working copy (with some debug info) 2020-10-04 19:55:39 +00:00
feng lv
6289f44fd4 multiversion storage for StorageMemory 2020-10-05 00:28:36 +08:00
hchen9
cbf3af401d Fix comments 2020-10-04 02:42:03 -07:00
alexey-milovidov
53ce5e3892
Merge pull request #15551 from azat/for-user-limits-over-distributed 
Pass through *_for_user settings via Distributed with cluster-secure
 2020-10-04 04:18:20 +03:00
alexey-milovidov
9551562889
Merge pull request #15442 from ClickHouse/add_alter_version 
Add ability to alter version column in VersionedCollapsingMergeTree.
 2020-10-03 05:40:15 +03:00
Azat Khuzhin
f25c1742b8 Pass through *_for_user settings via Distributed with cluster-secure 
In cluster-secure case the user on shards (remote) is equal to the user
on the initiator, so those settings can be safely applied.
 2020-10-03 02:04:47 +03:00
Azat Khuzhin
b838214a35 Pass non-const SelectQueryInfo (and drop mutable qualifiers) 2020-10-02 22:42:35 +03:00
Azat Khuzhin
587cde853e Avoid skipping unused shards twice (for query processing stage and read itself) 2020-10-02 22:42:09 +03:00
alesapin
77c9f7af79
Merge pull request #15532 from ClickHouse/substitute_defaults_recursively 
Fix defaults substitution for absent columns which depend on other defaults
 2020-10-02 21:42:01 +03:00
Alexander Tokmakov
6b2da861dc Merge branch 'master' into write_structure_of_table_functions 2020-10-02 20:46:58 +03:00
Alexander Tokmakov
5f6bbe8967 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-10-02 20:14:08 +03:00
Alexander Tokmakov
32776da69a Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-10-02 20:09:37 +03:00
Azat Khuzhin
9cb3c743bd
Convert to python3 (#15007) 2020-10-02 19:54:07 +03:00
hchen9
1f85114325 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into trivial_count_seq_consistency 2020-10-02 09:10:44 -07:00
Azat Khuzhin
21deb6812c
Drop unused code for numeric_limits<int128> in MergeTreeDataSelectExecutor (#15519) 2020-10-02 16:46:20 +03:00
alesapin
17b86dec96 Recursive defaults 2020-10-02 15:38:50 +03:00
Alexander Kuzmenkov
a9a7b065a0 Merge remote-tracking branch 'origin/master' into HEAD 2020-10-02 14:38:47 +03:00
Nikolai Kochetov
5ea4dc0850 Try fix tests. 2020-10-02 14:25:16 +03:00
hchen9
b3949db00f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into trivial_count_seq_consistency 2020-10-01 21:15:26 -07:00
Nikolai Kochetov
ea131989be Try fix test. 2020-10-01 21:47:20 +03:00
Nikolai Kochetov
ec64def384 Use QueryPlan while reading from MergeTree. 2020-10-01 20:34:22 +03:00
Nicolae Vartolomei
eb1d253355 Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it 2020-10-01 17:59:50 +01:00
Alexander Kuzmenkov
244687bfc1 Merge remote-tracking branch 'origin/master' into HEAD 2020-10-01 18:41:48 +03:00
Nicolae Vartolomei
7197a0c803 Possibly fix the issue with quorum 2020-10-01 11:38:50 +01:00
alesapin
8bc9ede711 Fix clang tidy 2020-10-01 09:53:47 +03:00
alexey-milovidov
9808d0be81
Merge pull request #15376 from excitoon-favorites/fixs3storage 
Fixed compression in S3 storage
 2020-10-01 04:30:32 +03:00
hchen9
b4494578d7 Support select_sequential_consistency in StorageReplicatedMergeTree.totalRows and totalBytes 2020-09-30 16:50:58 -07:00
hchen9
a5ac39b564 Pass Context parameter for IStorage.totalRows and IStorage.totalBytes 2020-09-30 16:47:42 -07:00
Alexandra Latysheva
e89a56969f first part 2020-09-30 23:16:27 +00:00
Artem Zuikov
0ac18a382f
Fix JOIN with GROUP BY over StorageMerge (#15242) 2020-09-30 23:11:27 +03:00
alesapin
97c2579fb9 Use separate function 2020-09-30 22:50:23 +03:00
alesapin
ac9f0ff4da More strict check for version column 2020-09-30 22:44:35 +03:00
Nikolai Kochetov
2efde179e9 Merge branch 'master' into storage-read-query-plan 2020-09-30 15:23:46 +03:00
Nikolai Kochetov
5ac6bc071d QueryPlan for StorageBuffer and StorageMaterializedView read. 2020-09-30 15:22:57 +03:00
Nikolai Kochetov
576ffadb17 Fix explain for ISourceStep. 2020-09-30 15:22:30 +03:00
Vladimir Chebotarev
a38c7ff2a8 Somehow uncompressed PUT works and gzipped PUT doesn't, in S3 storage. 2020-09-30 15:04:42 +03:00
Vladimir Chebotarev
698768add0 Fixed format argument in S3 storage when compression method is selected. 2020-09-30 13:55:44 +03:00
alexey-milovidov
0c07c2d3a3
Merge pull request #15089 from ClickHouse/codecs_for_nested_types 
Codecs for nested types
 2020-09-30 07:18:19 +03:00
alexey-milovidov
412a54e356
Merge pull request #14693 from hustnn/zk_exist_fix2 
Fix potential memory leak caused by zookeeper exist calling
 2020-09-30 04:48:11 +03:00
alexey-milovidov
b6bccfc4f4
Merge pull request #14776 from Vxider/disable_set_and_join_persistency 
Support memory only mode for StorageJoin and StorageSet.
 2020-09-30 04:35:43 +03:00
alexey-milovidov
b658f348a7
Update MergeTreeDataPartWriterWide.cpp 2020-09-30 01:23:42 +03:00
alesapin
65a05a970d Better message on error 2020-09-29 15:07:01 +03:00
tavplubix
4578d89819
Merge pull request #15348 from ClickHouse/replicate_default_args 
Continuation of #14791
 2020-09-28 18:53:02 +03:00
Alexander Kuzmenkov
218d86de7e Merge remote-tracking branch 'origin/master' into tmp 2020-09-28 16:29:51 +03:00
Nikita Mikhaylov
a4832ed5ce
Merge branch 'master' into HyaZz-master 2020-09-28 13:55:45 +03:00
Nikita Mikhaylov
73573472e9
Merge pull request #15299 from CurtizJ/fix-mysql-hung 
Fix hang of queries with a lot of subqueries to same mysql table
 2020-09-28 12:51:35 +03:00
alesapin
bc662ac4e0
Merge pull request #15293 from amosbird/bc3 
Replace useless multiset with unordered_set
 2020-09-28 10:30:04 +03:00
hustnn
a7024134dd Merge remote-tracking branch 'upstream/master' into zk_exist_fix2 2020-09-28 14:44:55 +08:00
Alexander Tokmakov
b0d99217fb Merge branch 'master' into write_structure_of_table_functions 2020-09-27 14:26:47 +03:00
Alexander Tokmakov
aa5e5013f2 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-09-27 14:11:53 +03:00
Alexander Tokmakov
34addcf61f add review suggestions 2020-09-26 22:18:28 +03:00
alesapin
59f2bbe2a3
Merge pull request #15304 from ClickHouse/fix_race_condition_on_rename 
Fix race condition between write ahead log cleanup and rename
 2020-09-26 10:43:26 +03:00
feng lv
504d577639 Merge branch 'master' of github.com:ClickHouse/ClickHouse into add-mutation-for-storagememory 2020-09-26 15:19:33 +08:00
alexey-milovidov
d1fce06292
Merge pull request #15297 from hexiaoting/build_bugfix 
fix build bug for merge_selector2.cpp
 2020-09-26 07:32:33 +03:00
Nikolai Kochetov
dea90009e3 Fix build 2020-09-25 16:03:12 +03:00
Alexander Tokmakov
68669b8a92 Merge branch 'master' into replicate_default_args 2020-09-25 15:52:17 +03:00
alesapin
f5c38b34ae Fix race condition during write ahead log cleanup and rename 2020-09-25 15:07:38 +03:00
Anton Popov
085f63a9bf fix hang of queries with a lot of subqueries to same mysql table 2020-09-25 14:12:48 +03:00
tavplubix
b96157e32b
Merge branch 'master' into enable_atomic_database_by_default 2020-09-25 13:42:39 +03:00
hexiaoting
4666ae3211 fix build bug for merge_selector2.cpp 2020-09-25 17:59:19 +08:00
Amos Bird
81d08b59e5
Replace useless multiset with unordered_set 2020-09-25 16:38:09 +08:00
alexey-milovidov
47d150b116
Revert "Revert "Avoid deadlocks in Log/TinyLog"" 2020-09-25 02:29:16 +03:00
alexey-milovidov
13529265c4
Revert "Avoid deadlocks in Log/TinyLog" 2020-09-25 02:28:57 +03:00
Alexander Tokmakov
fb826557b7 Merge branch 'master' into enable_atomic_database_by_default 2020-09-25 02:04:01 +03:00
Alexander Kuzmenkov
813a876e93
Merge pull request #14962 from ClickHouse/log-avoid-deadlock 
Avoid deadlocks in Log/TinyLog
 2020-09-24 13:54:11 +03:00
alesapin
96c419974c Merge branch 'master' into codecs_for_nested_types 2020-09-24 10:28:20 +03:00
Alexander Tokmakov
f0f91f84af Merge branch 'master' into enable_atomic_database_by_default 2020-09-23 21:50:15 +03:00
alesapin
87b64baea5 Fix again 2020-09-23 16:21:06 +03:00
alesapin
f404925397 More optimal 2020-09-23 15:19:45 +03:00
alesapin
1eb8f0a495 Fix bug in storage buffer during alter 2020-09-23 15:06:54 +03:00
alesapin
74671c909c Merge branch 'master' into codecs_for_nested_types 2020-09-23 13:12:23 +03:00
alesapin
bb4b739f43 Fix PVS-Studio warning 2020-09-23 13:11:48 +03:00
Alexander Kuzmenkov
f6316e2612 Merge remote-tracking branch 'origin/master' into tmp 2020-09-23 11:53:13 +03:00
filimonov
cc24ef9f83
Better debug message from MergeTreeDataSelectExecutor 
See #15168
 2020-09-22 21:35:29 +02:00
Alexander Tokmakov
7fcf20e48c enable more tests 2020-09-22 17:29:09 +03:00
alesapin
7015f272e7 Better comment 2020-09-22 16:48:38 +03:00
Alexander Kuzmenkov
679c0988a5 Merge remote-tracking branch 'origin/master' into HEAD 2020-09-22 16:25:52 +03:00
Nikolai Kochetov
62ef40394a
Merge pull request #14856 from ClickHouse/query-plan-for-creating-sets 
Query plan for creating sets
 2020-09-22 16:19:20 +03:00
alesapin
fd394f699c Better comments 2020-09-22 15:49:55 +03:00
alesapin
96d06c6ae6 Less changes 2020-09-22 15:16:15 +03:00
feng lv
c0b5f847d9 add test 
fix

fix
 2020-09-22 20:08:56 +08:00
Alexander Kuzmenkov
dde4cf70e1 Merge remote-tracking branch 'origin/master' into HEAD 2020-09-22 14:03:59 +03:00
Alexander Kuzmenkov
7b64ca33b1 Merge remote-tracking branch 'origin/master' into tmp 2020-09-22 14:02:20 +03:00
Alexander Kuzmenkov
bfc3be9e43 Merge remote-tracking branch 'origin/aku/tcp-versio' into tmp 2020-09-22 14:02:15 +03:00
feng lv
5d7a77c207 add mutation support for StorageMemory 2020-09-22 17:23:46 +08:00
hustnn
35f1db0043 Fix potential memory leak caused by zookeeper exist calling 2020-09-22 09:28:12 +08:00
Alexander Tokmakov
91cbd49e85 fix tests 2020-09-22 00:09:50 +03:00
Alexey Milovidov
1e65105b06 Merge branch 'master' into log-avoid-deadlock 2020-09-21 21:09:06 +03:00
alesapin
e87b961095 Trying to add same logic for compact parts 2020-09-21 20:35:09 +03:00
alesapin
fbd6f10ff7 Use default codec instead of throwing exception 2020-09-21 19:07:14 +03:00
alesapin
8517dda807 First working test 2020-09-21 17:47:10 +03:00
alesapin
4debccb7fe Almost working version 2020-09-21 17:22:13 +03:00
alesapin
a3e999784b Buildable code 2020-09-21 14:24:10 +03:00
Alexander Tokmakov
13e41cdd20 try enable atomic database by default 2020-09-21 13:55:34 +03:00
alesapin
387b97aba5 Merge branch 'master' into codecs_for_nested_types 2020-09-21 11:06:28 +03:00
alesapin
7fc5a2dae1 Some intermediate stage 2 2020-09-21 10:18:23 +03:00
alesapin
09f6a454ae Some intermediate stage 2020-09-21 10:17:58 +03:00
alexey-milovidov
3e0509af69
Merge pull request #13573 from amosbird/rms 
specific ReplicatedMergeTree settings
 2020-09-21 01:36:33 +03:00
alexey-milovidov
85483f8532
Merge pull request #14853 from ClickHouse/akz/optimized_index_binary_search 
Optimized marks selection algorithm for continuous marks ranges
 2020-09-20 19:48:45 +03:00
alexey-milovidov
f7cae4a57d
Merge pull request #14975 from ClickHouse/fix_merge_tree_settings_check 
Fix check for merge tree settings sanity on server startup
 2020-09-20 19:43:40 +03:00
alesapin
5ffee8808a
Alter remove column properties and TTLs (#14742) 2020-09-20 16:27:33 +03:00
philip.han
3122455b4d BloomFilter is used just in specific cases of indexof() 2020-09-20 19:04:33 +09:00
alexey-milovidov
1474b80ecc
Merge pull request #15008 from ClickHouse/clang-tidy-11-without-enabling 
Fix terribly wrong code
 2020-09-20 02:00:54 +03:00
alexey-milovidov
cd7354117c
Update StorageJoin.cpp 2020-09-19 19:38:36 +03:00
Alexander Tokmakov
de540c6955 minor fixes 2020-09-19 15:59:37 +03:00
Alexey Milovidov
76147cc0c1 Generate ya.make 2020-09-19 15:52:17 +03:00
Alexey Milovidov
3003ebabc2 Merge branch 'master' into Vxider-disable_set_and_join_persistency 2020-09-19 15:51:40 +03:00
alexey-milovidov
eb9ee723c3
Merge pull request #11913 from ClickHouse/compact-parts-by-default 
Enable compact parts by default for small parts
 2020-09-19 15:31:15 +03:00
Alexey Milovidov
4d06bc62b1 Allow concurrent reads while writing to TinyLog 2020-09-19 14:02:11 +03:00
Alexey Milovidov
111acdc63b Continuation 2020-09-19 14:02:11 +03:00
Alexey Milovidov
bc8fc3e280 Simplification 2020-09-19 14:02:11 +03:00
Alexey Milovidov
66ea3c0c43 Remove useless method 2020-09-19 14:02:11 +03:00
Alexey Milovidov
8193da847b Simplification 2020-09-19 14:02:11 +03:00
Alexey Milovidov
3bb0088c86 Whitespace 2020-09-19 14:02:11 +03:00
Alexey Milovidov
dc7e3b75d3 Whitespace 2020-09-19 14:02:11 +03:00
Alexey Milovidov
8a8284ce31 Avoid deadlocks in Log/TinyLog 2020-09-19 14:02:11 +03:00
alexey-milovidov
1fcebce926
Merge pull request #15000 from Jokser/disable-ttl-move-on-insert 
Option to disable TTL move on data part insert
 2020-09-19 13:45:02 +03:00
alexey-milovidov
3463d97f8c
Merge pull request #14973 from amosbird/bm2 
Ignore key constraints when doing mutations.
 2020-09-19 13:44:30 +03:00
Alexey Milovidov
ef5363b87f Fix terribly wrong code 2020-09-19 02:05:13 +03:00
alexey-milovidov
4e77c2e48e
Update MergeTreeDataMergerMutator.cpp 2020-09-19 01:29:31 +03:00
Alexey Milovidov
24b334258b Resolve review comment 2020-09-18 22:25:56 +03:00
Alexey Milovidov
e1ffa07a39 Resolve review comments 2020-09-18 22:08:53 +03:00
Alexey Milovidov
21ca9fce63 Merge branch 'master' into storages-small-improvements 2020-09-18 22:03:19 +03:00
alexey-milovidov
322e63971d
Update MergeTreeIndexConditionBloomFilter.cpp 2020-09-18 21:47:32 +03:00
Amos Bird
d842cb704f
Allow mutations to work with key constraints. 2020-09-19 02:40:02 +08:00
Alexander Tokmakov
1ca9a92b21 Merge branch 'master' into write_structure_of_table_functions 2020-09-18 21:09:23 +03:00
Pavel Kovalenko
da04a130ed Add option to disable ttl move on data part insert - minor fixes. 2020-09-18 20:45:30 +03:00
Pavel Kovalenko
6b4b5898a7 Renaming. 2020-09-18 18:41:14 +03:00
Pavel Kovalenko
0da19ab46d Add option to disable ttl move on data part insert 2020-09-18 18:30:00 +03:00
Nikolai Kochetov
b26f11c00c Support StorageDistributed::read for QueryPlan. 2020-09-18 17:16:53 +03:00
alesapin
e38b537017
Merge pull request #11684 from ClickHouse/manual-write-duplicate-parts-to-replicas 
Don't ignore duplicate parts written to replicas
 2020-09-18 16:50:00 +03:00
Alexander Kazakov
1ee2e3d2b3 Review fix 2020-09-18 16:03:48 +03:00
Vxider
c9249f77c7 rename persistency to persistent 2020-09-18 20:58:27 +08:00
Vxider
0c535062fc rename disable_persistency to persistency 2020-09-18 20:48:55 +08:00
Nikolai Kochetov
50674a320f Refactor IStorage::read with query plan. 2020-09-18 14:39:07 +03:00
alesapin
e96a3ac5f3 DataType in enumerate streams 2020-09-18 14:37:58 +03:00
Vitaly Baranov
f43714cd76
Merge pull request #14880 from vitlibar/fix-removing-live-view-after-watching 
Fix removing a live view after watching and after timeout has passed.
 2020-09-18 14:17:44 +03:00
Nikolai Kochetov
2491f0e1c5 Merge branch 'master' into storage-read-query-plan 2020-09-18 14:07:03 +03:00
Nikolai Kochetov
c7aff19937
Merge pull request #14221 from hagen1778/settings-leaf-limits 
[settings]: introduce new query complexity settings for leaf-nodes
 2020-09-18 14:05:10 +03:00
philip.han
cbedd44a60 Fix indexOf() to use BloomFilter 2020-09-18 17:43:07 +09:00
alesapin
96a693baec Better message 2020-09-18 10:46:23 +03:00
alesapin
e6914acb3b Fix check for merge tree settings on server startup 2020-09-18 10:44:17 +03:00
alesapin
3536ac1b17
Merge pull request #14926 from ClickHouse/better-drop-replicated-table 
Better drop for Replicated tables
 2020-09-18 09:51:39 +03:00
roman
ddca262fe6 fix review comments 2020-09-17 20:54:21 +01:00
Alexey Milovidov
7fb4dfea2c Small improvements for IStorage::rename 2020-09-17 22:50:43 +03:00
alesapin
447205c525 More comments 2020-09-17 22:30:45 +03:00
alesapin
dc677b93fb Comments and fix test 2020-09-17 22:30:17 +03:00
alexey-milovidov
7152fec241
Update JoinSettings.cpp 2020-09-17 21:24:14 +03:00
Alexander Tokmakov
41e99cf261 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-09-17 20:28:50 +03:00
alesapin
b07c00ef7c Ugly, but probably working implementation 2020-09-17 18:33:50 +03:00
alexey-milovidov
84eece69ba
Merge pull request #14876 from amosbird/ns 
Get rid of query settings after initialization.
 2020-09-17 17:49:25 +03:00
Alexey Milovidov
c1e9ce2cf2 Remove questionable code 2020-09-17 17:38:06 +03:00
Alexander Kuzmenkov
fb64cf210a straighten the protocol version 2020-09-17 17:37:29 +03:00
Vitaly Baranov
3356d75b23
Merge pull request #13156 from azat/cluster-secure 
Secure inter-cluster query execution (with initial_user as current query user) [v3]
 2020-09-17 17:11:00 +03:00
alesapin
d09fd8ca8b Add comment 2020-09-17 17:01:17 +03:00
Nikolai Kochetov
0bf4e8e6e9 Support reading with QueryPlan for StorageView. 2020-09-17 16:22:24 +03:00
Alexander Kuzmenkov
a374541214 straighten the protocol version 2020-09-17 15:15:05 +03:00
alesapin
6601f4f786 Better errors, fix merge 2020-09-17 15:10:06 +03:00
alesapin
6c369c8dc6 Fix merge error 2020-09-17 15:01:03 +03:00
Alexey Milovidov
a5560cd62d Better drop replicated tables 2020-09-17 14:27:17 +03:00
alesapin
40b2f203b6 Merge branch 'master' into manual-write-duplicate-parts-to-replicas 2020-09-17 13:21:00 +03:00
roman
b41421cb1c [settings]: introduce new query complexity settings for leaf-nodes 
The new setting should allow to control query complexity on leaf nodes
excluding the final merging stage on the root-node. For example, distributed
query that reads 1k rows from 5 shards will breach the `max_rows_to_read=5000`,
while effectively every shard reads only 1k rows. With setting `max_rows_to_read_leaf=1500`
this limit won't be reached and query will succeed since every shard reads
not more that ~1k rows.
 2020-09-17 10:37:05 +01:00
Alexander Kuzmenkov
6217dd77b5 Merge remote-tracking branch 'origin/master' into tmp 2020-09-17 12:35:10 +03:00
alesapin
f104c382f8
Merge pull request #14887 from azat/StorageFile-write-to-fd 
Fix SIGSEGV for an attempt to INSERT into StorageFile(fd)
 2020-09-17 10:25:02 +03:00
alesapin
5f9e7ed169
Merge pull request #14834 from ClickHouse/fix_compression_codec_perf 
Fix performance degradation during parts selection for merge
 2020-09-17 10:09:41 +03:00
alesapin
73544a3781
Merge pull request #14845 from ClickHouse/fix_alias_array 
Fix recursive column defaults
 2020-09-17 10:02:39 +03:00
Azat Khuzhin
138e953429 Fix SIGSEGV for an attempt to INSERT into StorageFile(fd) 2020-09-17 01:26:34 +03:00
Anton Popov
8e9828d2ff Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-16 21:07:35 +03:00
Alexey Milovidov
26eb49b48d Fix typo 2020-09-16 20:02:55 +03:00
Amos Bird
96a202c0fb
Get rid of query settings after initialization. 2020-09-16 22:35:39 +08:00
alesapin
cbe85327e2 Check stack size 2020-09-16 16:24:07 +03:00
Vitaly Baranov
d99a011d07 Fix removing a live view after watching and after timeout has passed. 2020-09-16 15:53:58 +03:00
alesapin
97baa67f2b
Merge pull request #14839 from Jokser/distributed-storage-multiple-volumes 
Don't throw exception if Distributed storage has multi-volume storage policy configuration
 2020-09-16 10:13:14 +03:00
Nikolai Kochetov
13edfeea1f Fix build. 2020-09-15 20:23:49 +03:00
Nikolai Kochetov
e29f27e333
Merge pull request #14815 from ClickHouse/storage-read-query-plan 
Return query plan from storage.
 2020-09-15 18:34:11 +03:00
Vitaly Baranov
74d5b43fe6
Merge pull request #14828 from vitlibar/change-columns-of-system-user-directories 
Add 'params' column to system.user_directories table.
 2020-09-15 18:25:50 +03:00
Alexander Kazakov
7465e00163 Optimized marks selection algorithm for continuous marks ranges 2020-09-15 17:22:32 +03:00
Alexander Tokmakov
0ecf8e595f cleanup code, add test 2020-09-15 16:41:53 +03:00
alesapin
e20c54998c Better exception on non physical columns 2020-09-15 16:29:11 +03:00
Anton Popov
3e855daefd Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-15 15:31:29 +03:00
alesapin
8828a78174 Fix alias on default columns 2020-09-15 14:17:58 +03:00
Vxider
e136eba3f8 fix style check 2020-09-15 18:58:16 +08:00
Nikolai Kochetov
6eac433c55 Review fixes. 2020-09-15 13:40:39 +03:00
Artem Zuikov
51ba12c2c3
Try speedup build (#14809) 2020-09-15 12:55:57 +03:00
Nikolai Kochetov
c1f61984e5
Merge pull request #14513 from amosbird/mf1 
Binary operator monotonicity
 2020-09-15 12:45:23 +03:00
Alexander Tokmakov
3671c91c37 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-09-15 12:43:01 +03:00
Pavel Kovalenko
01ab28a182 Don't throw exception if Distributed storage has multi-volume storage policy configuration. 2020-09-15 12:26:56 +03:00
Nikolai Kochetov
8c2aa7ce8c Added comment. 2020-09-15 12:22:45 +03:00
Vxider
1c82d2eb23 use table-level setting 2020-09-15 17:16:10 +08:00
alesapin
6f8da7dfdf Merge branch 'master' into fix_compression_codec_perf 2020-09-15 09:59:45 +03:00
Vitaly Baranov
0f53b449fd Add 'params' column to system.user_directories table. 2020-09-15 01:51:53 +03:00
Azat Khuzhin
0159c74f21 Secure inter-cluster query execution (with initial_user as current query user) [v3] 
Add inter-server cluster secret, it is used for Distributed queries
inside cluster, you can configure in the configuration file:

  <remote_servers>
      <logs>
          <shard>
              <secret>foobar</secret> <!-- empty -- works as before -->
              ...
          </shard>
      </logs>
  </remote_servers>

And this will allow clickhouse to make sure that the query was not
faked, and was issued from the node that knows the secret. And since
trust appeared it can use initial_user for query execution, this will
apply correct *_for_user (since with inter-server secret enabled, the
query will be executed from the same user on the shards as on initator,
unlike "default" user w/o it).

v2: Change user to the initial_user for Distributed queries if secret match
v3: Add Protocol::Cluster package
v4: Drop Protocol::Cluster and use plain Protocol::Hello + user marker
v5: Do not use user from Hello for cluster-secure (superfluous)
 2020-09-15 01:36:28 +03:00
Alexander Tokmakov
da6beb1a91 Merge branch 'master' into enable_some_tests_with_database_atomic 2020-09-14 23:00:14 +03:00
Alexander Tokmakov
a812f7eb75 allow using Atomic database inside MaterializeMySQL 2020-09-14 22:25:02 +03:00
alesapin
e96256d367 Trying to improve performance 2020-09-14 22:15:25 +03:00
Nikita Mikhaylov
ddea62211d
Merge pull request #11948 from CurtizJ/fsync-parts 
Experiment. Allow to turn on fsync on inserts, merges and fetches.
 2020-09-14 22:37:14 +04:00
Nikolai Kochetov
bcfc83022a Return query plan from storage. 2020-09-14 17:13:58 +03:00
Vxider
407bccdefa Support default arguments for Replicated engine 2020-09-14 15:46:27 +08:00
alesapin
06575970aa
Merge pull request #14705 from amosbird/misc1 
Add merge_algorithm to system.merges
 2020-09-14 09:58:30 +03:00
Vxider
b5498e6ed2 update code style 2020-09-14 09:39:32 +08:00
Vxider
cbf8282701 add disable_set_and_join_persistency 2020-09-13 23:11:18 +08:00
Amos Bird
5cc8fd395c
Fix empty key segfault 2020-09-13 21:55:16 +08:00
Amos Bird
34b9547ce1
Binary operator monotonicity 2020-09-13 21:55:12 +08:00
Amos Bird
c2d79bc5cc
Add merge_algorithm to system.merges 2020-09-13 10:00:03 +08:00
Vitaly Baranov
70b0fe88b1
Merge pull request #14604 from vitlibar/no-detach-threads-2 
Use join() instead of detach() for threads #2
 2020-09-12 02:07:46 +03:00
alesapin
2c4047b280 Revert accident changes 2020-09-11 20:07:00 +03:00
alesapin
6dd75182f0 Better 2020-09-11 20:04:55 +03:00
alesapin
a4c43e51b9 Add a test 2020-09-11 20:04:55 +03:00
alesapin
a5f8899874 First implementation 2020-09-11 20:04:55 +03:00
Anton Popov
a64331d79f fix syncing of WAL 2020-09-11 16:09:26 +03:00
tavplubix
1f47b1ff6b
Merge pull request #14704 from ClickHouse/fix_misleading_message_uuid_macro 
Fix misleading exception message about uuid macro
 2020-09-11 12:45:32 +03:00
Anton Popov
f2a5216e97 add waiting for fsync in WAL 2020-09-11 02:28:50 +03:00
Anton Popov
af4089c606 Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-11 00:39:21 +03:00
tavplubix
4581e3a18a
Merge pull request #14563 from bharatnc/ncb/zk-conn-fail-do-cleanup 
ReplicatedMergeTree - cleanup data directory after Zookeeper exceptions
 2020-09-10 19:19:57 +03:00
bharatnc
3b3b7bd5e7 minor changes to comment 2020-09-10 09:09:48 -07:00
tavplubix
a6c9dd0465
Merge pull request #14664 from ClickHouse/fix_missed_database_when_altering_mv 
Fix missed database name when altering MaterializedView
 2020-09-10 17:55:31 +03:00
Alexander Tokmakov
f19f02028b fix misleading exception message about uuid macro 2020-09-10 17:50:10 +03:00
Alexander Tokmakov
20b3823770 do not remove directory on attach 2020-09-10 13:21:13 +03:00
Anton Popov
b9bf67b6ac allow to randomize part type in MergeTree 2020-09-10 04:27:36 +03:00
Alexander Tokmakov
7080d56470 fix missed database name when altering mv 2020-09-09 22:59:34 +03:00
Vitaly Baranov
2a9ab48279 Use join() instead of detach() for the no_users_thread in StorageLiveView. 2020-09-09 22:39:27 +03:00
Alexander Tokmakov
dee1fefeb4 add more tests 2020-09-09 20:39:49 +03:00
bharatnc
fe6d6f16ea Merge remote-tracking branch 'upstream/master' into ncb/zk-conn-fail-do-cleanup 2020-09-09 09:25:34 -07:00
alesapin
bd14c79b0b
Merge pull request #14494 from ClickHouse/recompression_in_background 
Recompression TTL
 2020-09-09 17:46:01 +03:00
Alexander Kuzmenkov
e729dc5552 Merge remote-tracking branch 'origin/master' into tmp 2020-09-09 17:33:11 +03:00
Alexander Tokmakov
d278d849e4 Merge branch 'master' into write_structure_of_table_functions 2020-09-09 17:12:35 +03:00
Alexander Tokmakov
b68782d285 enable more tests with Atomic database 2020-09-09 16:32:50 +03:00
Winter Zhang
3d1d64ec60
Fix currentDatabase function cannot be used in ON CLUSTER ddl query. (#14211) 2020-09-09 12:58:59 +03:00
Anton Popov
32135d96f9
Merge pull request #12183 from CurtizJ/polymorphic-parts-2 
Support codecs in compact parts
 2020-09-09 12:34:13 +03:00
alesapin
485b104898 Use max ttl and add introspection to system parts about recompression 2020-09-09 12:15:42 +03:00
alesapin
a696cf12f6 Merge branch 'master' into recompression_in_background 2020-09-09 10:12:03 +03:00
bharatnc
f37814b367 Merge remote-tracking branch 'upstream/master' into ncb/zk-conn-fail-do-cleanup 2020-09-08 23:45:20 -07:00
alexey-milovidov
f2e9a09b11
Merge pull request #14585 from ClickHouse/join-processor 
Remove Join from ExpressionActions
 2020-09-08 23:56:09 +03:00
Anton Popov
a419267dc6 minor fixes 2020-09-08 19:28:49 +03:00
robot-clickhouse
8def718661 Auto version update to [20.10.1.1] [54440] 2020-09-08 17:04:41 +03:00
Alexander Kuzmenkov
c62bf555d2 Merge origin/master into tmp (using imerge) 2020-09-08 16:26:38 +03:00
alesapin
f0f7f2b5d2 Merge branch 'master' into recompression_in_background 2020-09-08 15:43:37 +03:00
alesapin
6ad91d1018
Merge pull request #14490 from ClickHouse/control_ttl_merges_in_pool 
Control ttl merges in pool
 2020-09-08 15:42:20 +03:00
Nikolai Kochetov
39aad9979a Remove Join from ExpressionActions. 2020-09-08 13:40:53 +03:00
Nicolae Vartolomei
a62e96c7aa nit: fix argument names for clang-10 2020-09-08 10:56:10 +01:00
Nicolae Vartolomei
97d0b5ab23 Remove unused query argument 2020-09-08 10:56:10 +01:00
Nicolae Vartolomei
1876374770 ALTER TABLE ... DROP|DETACH PART for MergeTree 2020-09-08 10:56:09 +01:00
Ilya Yatsishin
11a247d2f4
Merge pull request #14556 from ClickHouse/fix-arcadia-2 
Lower binary size in "Arcadia" build
 2020-09-08 12:08:35 +03:00
Nikolai Kochetov
c3eeb8f71e
Merge pull request #14412 from ClickHouse/pipes-and-creating-sets 
Pipes and creating sets
 2020-09-08 10:47:56 +03:00
alesapin
4364bff3bc
Merge pull request #12761 from kssenii/rabbitmq-improvements 
RabbitMQ improvements
 2020-09-08 09:38:25 +03:00
alesapin
8ff0aedebe Merge branch 'control_ttl_merges_in_pool' into recompression_in_background 2020-09-08 09:34:03 +03:00
alesapin
405a204514 Lower timeout for TTL merges 2020-09-08 09:33:00 +03:00
alesapin
f4846e2ef1
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Anton Popov <pad11rus@gmail.com>
 2020-09-08 09:30:54 +03:00
bharatnc
0c00b992d5 StorageReplicatedMergeTree - cleanup data dir after Zk exceptions 
It's possible for `getZooKeeper()` to timeout if  zookeeper host(s) can't
be reached. In such cases `Poco::Exception` is thrown after a connection
timeout - refer to `src/Common/ZooKeeper/ZooKeeperImpl.cpp:866` for more info.

Side effect of this is that the CreateQuery gets interrupted and it exits.
But the data Directories for the tables being created aren't cleaned up.
This unclean state will hinder table creation on any retries and will
complain that the Directory for table already exists.

To achieve a clean state on failed table creations, catch this error if
the exception is of type Poco::Exception and call `dropIfEmpty()` method,
then proceed throwing the exception. Without this, the Directory for the
tables need to be manually deleted before retrying the CreateQuery.
 2020-09-07 18:09:09 -07:00
Alexey Milovidov
e3924b8057 Fix "Arcadia" 2020-09-08 01:14:13 +03:00
alexey-milovidov
e1104135c8
Merge pull request #13939 from fastio/master 
Multithreading implementation of Kafka engine
 2020-09-07 20:12:06 +03:00
Nicolae Vartolomei
6060a1ab57 ALTER TABLE ... DROP|DETACH PART for ReplicatedMergeTree 2020-09-07 17:01:45 +01:00
kssenii
40c8290dde Tiny fixes, better tests 2020-09-07 14:48:50 +00:00
alesapin
17650e803f Tiny improvements 2020-09-07 14:08:53 +03:00
alesapin
f6237dc334 Better name 2020-09-07 12:32:45 +03:00
alesapin
4ce975c512 Merge branch 'master' into kssenii-rabbitmq-improvements 2020-09-07 11:36:08 +03:00
alesapin
f274ffc9d1 Add comments 2020-09-07 10:59:14 +03:00
alesapin
ba70de63f8 Fix diff 2020-09-07 10:28:04 +03:00
alesapin
7f301eab63 Merge branch 'master' into recompression_in_background 2020-09-07 09:31:16 +03:00
Dmitry
9639846ca9 Merge branch 'master' into system_events_show_zero_values 2020-09-06 14:55:14 +03:00
Peng Jian
fd82272cfe Fix code style, and update docs for Kafka engine 2020-09-06 11:17:19 +08:00
Peng Jian
e6bfd9d586 1. Add new setting for Kafka engine, named kafka_thread_per_consumer which default value is false. 2. Create separate thread pool for Kafka engine. 2020-09-06 11:17:19 +08:00
Peng Jian
de0a40aedc fix code style 2020-09-06 11:17:19 +08:00
Peng Jian
61ac52cf29 Multithreading implementation of Kafka engine 2020-09-06 11:17:18 +08:00
Anton Popov
3b9be70c8c Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-05 18:53:55 +03:00
Nikolai Kochetov
499405504f Merge branch 'master' into pipes-and-creating-sets 2020-09-04 23:41:20 +03:00
Anton Popov
c73bb98002 fix clang-tidy 2020-09-04 19:50:40 +03:00
Anton Popov
f1cf03f1bb Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-04 17:40:22 +03:00
alesapin
37a2bd0bfd less debug 2020-09-04 17:23:32 +03:00
alesapin
de911aaabe Merge branch 'master' into recompression_in_background 2020-09-04 17:21:27 +03:00
alesapin
2d1042614e Less comments 2020-09-04 17:19:41 +03:00
alesapin
79ef72178e Less garbage 2020-09-04 17:18:49 +03:00
alesapin
f48d654d35 Less garbage 2020-09-04 17:15:34 +03:00
alesapin
fecb2f1311 Less copypaste 2020-09-04 17:08:43 +03:00
alesapin
ea7168580b Fixes 2020-09-04 16:55:07 +03:00
Alexander Tokmakov
ed5f827708 Merge branch 'master' into write_structure_of_table_functions 2020-09-04 16:38:57 +03:00
Anton Popov
68913eab62 better reading from compact parts with differents codecs 2020-09-04 15:48:55 +03:00
alesapin
f8aa6cfe8a
Merge pull request #14438 from ClickHouse/apply_ttl_if_not_calculated 
Apply TTL if it's not calculated for part
 2020-09-04 15:32:50 +03:00
alesapin
f9dd4cc98d Merge branch 'control_ttl_merges_in_pool' into recompression_in_background 2020-09-04 14:32:25 +03:00
alesapin
e42d0f60da Fix several bugs 2020-09-04 14:27:27 +03:00
alesapin
4079636f81 Merge 2020-09-04 13:52:51 +03:00
alesapin
6f5ba4d8e5 Fix ya.make 2020-09-04 13:31:29 +03:00
alesapin
69b31ab90d More comments 2020-09-04 13:29:55 +03:00
alesapin
82c56349a5 Some comments 2020-09-04 13:08:09 +03:00
Alexander Tokmakov
b840d741d0 Merge branch 'master' into write_structure_of_table_functions 2020-09-04 13:00:07 +03:00
Nikolai Kochetov
c761970bc5 Merge branch 'master' into pipes-and-creating-sets 2020-09-04 11:38:35 +03:00
Nikolai Kochetov
b7e53208d1 Fix tests. 2020-09-04 11:36:47 +03:00
alesapin
c96a2b4f22
Merge pull request #14444 from ClickHouse/fix_select_parts_for_merge 
Fix select parts for merge
 2020-09-04 09:55:48 +03:00
alesapin
61ecaebcb1 Simplify settings for TTL merges 2020-09-04 09:55:19 +03:00
Anton Popov
6883ee7eea create less compressed streams while writing compact parts 2020-09-04 01:38:17 +03:00
Anton Popov
d5da58918e create less compressed streams while writing compact parts 2020-09-04 01:04:46 +03:00
Anton Popov
98011f6fc3 Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-03 17:53:05 +03:00
alesapin
0dab4d9485 Merge branch 'fix_select_parts_for_merge' into control_ttl_merges_in_pool 2020-09-03 17:41:23 +03:00
alesapin
bd24c3a059 Fix build useless binary 2020-09-03 17:38:56 +03:00
Kseniia Sumarokova
4b7c303eaf
Merge branch 'master' into rabbitmq-improvements 2020-09-03 17:26:04 +03:00
alesapin
15a02321bf Merge branch 'fix_select_parts_for_merge' into control_ttl_merges_in_pool 2020-09-03 17:16:29 +03:00
alesapin
55adb6f9f2
Update src/Storages/MergeTree/MergeSelector.h 
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2020-09-03 16:55:34 +03:00
alesapin
e48e4309b2 Better names 2020-09-03 16:29:18 +03:00
alesapin
565555e55c Fix bug in parts selection 2020-09-03 16:04:28 +03:00
alesapin
13248a744b Fix bug in parts selection 2020-09-03 16:02:24 +03:00
alesapin
f4c7ff0376 Add fixed size of Merge TTLS 2020-09-03 16:00:13 +03:00
alesapin
aa47d0aabc Merge branch 'apply_ttl_if_not_calculated' into recompression_in_background 2020-09-03 12:07:03 +03:00
alesapin
acc0ee0657 Apply TTL if it's not calculated for part 2020-09-03 11:59:41 +03:00
kssenii
e1ef558a2d Fixes 2020-09-03 06:50:20 +00:00
alexey-milovidov
edea940e17
Update StorageDistributed.cpp 2020-09-03 04:39:36 +03:00
Azat Khuzhin
fffeeeba06 Force WithMergeableStateAfterAggregation via distributed_group_by_no_merge (convert to UInt64) 
Possible values:
- 1 - Do not merge aggregation states from different servers for distributed query processing - in case it is for certain that there are different keys on different shards.
- 2 - same as 1 but also apply ORDER BY and LIMIT stages
 2020-09-03 00:52:51 +03:00
Azat Khuzhin
10b4f3b41f Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key 
Previous set of QueryProcessingStage does not allow to do this.
But after WithMergeableStateAfterAggregation had been introduced the
following queries can be optimized too under
optimize_distributed_group_by_sharding_key:
- GROUP BY sharding_key LIMIT
- GROUP BY sharding_key LIMIT BY
- GROUP BY sharding_key ORDER BY

And right now it is still not supports:
- WITH TOTALS (looks like it can be supported)
- WITH ROLLUP (looks like it can be supported)
- WITH CUBE
- SETTINGS extremes=1 (looks like it can be supported)
But will be implemented separatelly.

vX: fixes
v2: fix WITH *
v3: fix extremes
v4: fix LIMIT OFFSET (and make a little bit cleaner)
v5: fix HAVING
v6: fix ORDER BY
v7: rebase against 20.7
v8: move out WithMergeableStateAfterAggregation
v9: add optimize_distributed_group_by_sharding_key into test names
 2020-09-03 00:52:51 +03:00
Azat Khuzhin
4043be3121 Add new query processing stage WithMergeableStateAfterAggregation 
Process query until the stage where the aggregate functions were
calculated and finalized.

It will be used for optimize_distributed_group_by_sharding_key.

v2: fix aliases
v3: Fix protocol ABI breakage due to WithMergeableStateAfterAggregation
    Conditions >= for QueryProcessingStage::Enum has been verified, and they
    are ok (in InterpreterSelectQuery).
 2020-09-03 00:51:08 +03:00
Alexander Tokmakov
b1d6ac0d14 Merge branch 'master' into write_structure_of_table_functions 2020-09-02 20:04:50 +03:00
alesapin
dcbddbb8d0 Add recompression ttls tests and fix bugs 2020-09-02 19:15:41 +03:00
Alexander Tokmakov
4bf3f9e4a3 use StorageProxy for StorageMaterializeMySQL 2020-09-02 18:56:21 +03:00
Alexander Tokmakov
ee7b8a797a better code, fix Distributed format of StorageFile 2020-09-02 18:07:53 +03:00
alexey-milovidov
8e2fba5be1
Merge pull request #12567 from amosbird/fmi 
table function view
 2020-09-02 17:07:57 +03:00
Nikolai Kochetov
92c937db8b Remove CreatingSetsBlockInputStream 2020-09-02 16:13:13 +03:00
alexey-milovidov
c009ace7a2
Update StorageSystemEvents.cpp 2020-09-02 15:28:29 +03:00
alesapin
128cb7ce22 Don't select already selected parts 2020-09-02 15:16:12 +03:00
alesapin
aac466ab1c Merge branch 'better_ttl_merges_selection' into recompression_in_background 2020-09-02 13:30:04 +03:00
Dmitry
6f18cd772c fixes 2020-09-02 13:07:23 +03:00
Dmitry
2180cdc400 system_events_show_zero_values setting added 2020-09-02 13:00:49 +03:00
alesapin
fbb37c37df Simplier interface 2020-09-02 11:28:46 +03:00
alesapin
b08056fa8c Better selection of Merges with TTL 2020-09-02 11:18:50 +03:00
Amos Bird
6bd753d85d
TableFunction view. 2020-09-02 16:02:10 +08:00
Anton Popov
4fe57fc169 Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-02 01:29:44 +03:00
Anton Popov
26d75f7602 do fsync for WAL 2020-09-02 01:25:10 +03:00
kssenii
6682c62a90 Fixes 2020-09-01 20:07:34 +00:00
Alexander Tokmakov
41dd393824 fixes 2020-09-01 20:55:35 +03:00
Alexander Tokmakov
56695727b2 Merge branch 'master' into write_structure_of_table_functions 2020-09-01 20:15:13 +03:00
bharatnc
f0d1c6c794 Merge remote-tracking branch 'upstream/master' into ncb/fix-alter-live-lock 2020-09-01 09:06:10 -07:00
Anton Popov
3cadc9033a fsyncs for metadata files of part 2020-09-01 18:53:22 +03:00
Alexander Tokmakov
c7f355a8f3 add test 2020-09-01 18:24:21 +03:00
Alexander Tokmakov
3a9779adb4 slightly better code 2020-09-01 17:41:49 +03:00
alesapin
c37a19f7b0 Better 2020-09-01 14:23:38 +03:00
kssenii
15f735bb24 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into rabbitmq-improvements 2020-09-01 10:51:34 +00:00
alesapin
f0dc5a3085 First working test 2020-09-01 13:49:53 +03:00
Alexander Tokmakov
b6093d9a86 fix query header 2020-09-01 13:37:42 +03:00
kssenii
c2fb72abf8 Better mv, more comments 2020-09-01 07:58:41 +00:00
alesapin
b751319f9e Merge branch 'master' into recompression_in_background 2020-09-01 10:31:10 +03:00
alesapin
05b10048a6
Merge pull request #14116 from ClickHouse/parts_default_compression 
Save parts default compression
 2020-09-01 09:43:42 +03:00
alesapin
e28b477f79
Merge pull request #14263 from bharatnc/ncb/disallow-codec-for-alias 
Disallow codec for ALIAS columns types
 2020-09-01 09:42:53 +03:00
bharatnc
c3dd968931 fix ALTER LIVE VIEW lock issue 
This PR fixes a lock issue that happens while executing
`ALTER LIVE VIEW` query with the `REFRESH` command that
results in a exception. The problem is that lock is currently
being acquired in `InterpreterALterQuery.cpp`
in the `InterpreterAlterQuery::execute()` method and lock
is again being reacquired in `StorageLiveView.cpp` in the
` StorageLiveView::refresh` method. This removes that extra
lock.

Before fix:

```sql
--create table

CREATE TABLE test0 (
    c0 UInt64
) ENGINE = MergeTree() PARTITION BY c0 ORDER BY c0;

-- enable experimental_live_view

:) SET allow_experimental_live_view=1

-- create live view;

:) CREATE LIVE VIEW live1 AS SELECT * FROM table0;

-- alter live view results in exception

:) ALTER LIVE VIEW live1 REFRESH;

...
...

Received exception from server (version 20.8.1):
Code: 49. DB::Exception: Received from localhost:9000. DB::Exception: RWLockImpl::getLock(): RWLock is already locked in exclusive mode.

```

After fix:

```sql
:)  ALTER LIVE VIEW live1 REFRESH;

ALTER LIVE VIEW live1
    REFRESH

Ok.

0 rows in set. Elapsed: 0.016 sec.
```
 2020-08-31 22:17:58 -07:00
Anton Popov
25140b9bd5 fsync MergeTree format file 2020-09-01 04:39:36 +03:00
robot-clickhouse
e2fa0eae2f Auto version update to [20.9.1.1] [54439] 2020-08-31 23:07:41 +03:00
alesapin
46f833b7df Some changes 2020-08-31 22:50:42 +03:00
Alexander Tokmakov
f0a5f19dae try use another wrapper for table functions 2020-08-31 22:03:26 +03:00
bharatnc
c377c228ef fix tests 2020-08-31 10:22:53 -07:00
bharatnc
c4e235b000 Merge remote-tracking branch 'upstream/master' into ncb/disallow-codec-for-alias 2020-08-31 09:01:07 -07:00
bharatnc
caed8cd474 change error code to BAD_ARGUMENTS (36) 2020-08-31 09:00:32 -07:00
Alexander Kuzmenkov
bc8765d5ad
Merge pull request #14095 from azat/DistributedFilesToInsert-fix 
Fix DistributedFilesToInsert metric (zeroed when it should not)
 2020-08-31 18:58:30 +03:00
alexey-milovidov
f5a38fa78d
Merge pull request #14203 from donge/master 
fix issue #14202
 2020-08-31 17:48:39 +03:00
alesapin
b20a0bc254 Add recompression flag in ReplicatedEntry 2020-08-31 16:42:42 +03:00
alesapin
067eb4599d Fix empty columns case 2020-08-31 16:39:27 +03:00
alesapin
adc18f4d3f Write with recompression TTL 2020-08-31 16:29:31 +03:00
alesapin
42c210fcba Recompress TTLs in memory metadata 2020-08-31 15:12:51 +03:00
alesapin
4834bed35b Add recompression TTL parser 2020-08-31 14:35:53 +03:00
kssenii
e57d1c827f Better shutdown 2020-08-31 10:00:28 +00:00
kssenii
647cf5718e Better settings 2020-08-31 09:27:42 +00:00
alesapin
034f1a895d Merge branch 'master' into recompression_in_background 2020-08-31 11:10:14 +03:00
alesapin
5b0822902b
Merge pull request #14155 from amosbird/fzk 
Allow FETCH PARTITION from other zookeepers
 2020-08-31 09:54:51 +03:00
bharatnc
5c83b09a2f disallow CODEC setting for column type ALIAS 
This commit adds checks in place during table creation
and updates to ensure that we don't allow `CODEC` for
ALIAS (`default_type` column) like:

```sql
CREATE TABLE compression_codec_on_alias
(
    `c0` ALIAS c1 CODEC(ZSTD),
    `c1` UInt64
)
ENGINE = MergeTree()
PARTITION BY c0
ORDER BY c1;
```

After these safeguards in place, when trying to create/update column
codec, we will get excaptions like this:

```sql

-- create

CREATE TABLE compression_codec_on_alias
(
    `c0` ALIAS c1 CODEC(ZSTD),
    `c1` UInt64
)
ENGINE = MergeTree()
PARTITION BY c0
ORDER BY c1

Received exception from server (version 20.8.1):
Code: 377. DB::Exception: Received from localhost:9000. DB::Exception: Cannot specify codec for column type ALIAS.

0 rows in set. Elapsed: 0.006 sec.

-- modify

ALTER TABLE compression_codec_on_alias
    ADD COLUMN `c3` ALIAS c2 CODEC(ZSTD) AFTER c2

Received exception from server (version 20.8.1):
Code: 377. DB::Exception: Received from localhost:9000. DB::Exception: Cannot specify codec for column type ALIAS.

0 rows in set. Elapsed: 0.005 sec.

```
 2020-08-30 23:45:53 -07:00
alexey-milovidov
b1efc5df01
Merge pull request #14225 from amosbird/bf1 
Fix bug in mark inclusion search.
 2020-08-30 02:58:22 +03:00
alesapin
2d33a4029b
Merge pull request #14220 from ClickHouse/remove_redundant_flag 
Disable force TTL on optimize final
 2020-08-29 10:40:36 +03:00
Amos Bird
591a4d60d4
Fix bug in mark inclusion search. 2020-08-29 09:46:46 +08:00
Alexander Kuzmenkov
79e0b184ac A test + reinterpretAsUUID 2020-08-28 23:40:23 +03:00
alesapin
ebc163c924 Merge branch 'master' into parts_default_compression 2020-08-28 23:11:42 +03:00
alesapin
10c7a6c45e
Add ability to specify Default codec for columns (#14049) 
* Add ability to specify DefaultCompression codec which correspond to settings specified in config.xml

* Fix style

* Rename DefaultCompression to simple Default

* Fix compression codec

* Better codec description representation

* Less strange code and one method

* Fix delta
 2020-08-28 20:40:45 +03:00
alesapin
65c33f0802
Merge pull request #13280 from amosbird/as 
ALTER MODIFY SAMPLE BY
 2020-08-28 19:32:09 +03:00
alesapin
232c264d7d Disable force TTL on optimise 2020-08-28 16:45:42 +03:00
alesapin
1ea00d5e7a Fix test and better messages 2020-08-28 14:12:51 +03:00
alesapin
b599d0a1ff Simplify function 2020-08-28 13:47:29 +03:00
alesapin
0896b49533 Small fixes 2020-08-28 13:18:56 +03:00
kssenii
4e0c619721 Global refactoring 2020-08-28 10:03:54 +00:00
alesapin
77faf9587f Better interface 2020-08-28 12:07:20 +03:00
Dongdong Yang
7e48edf42c fix issue #14202 2020-08-28 13:29:15 +08:00
Alexander Kuzmenkov
4612794678 opentelemetry context propagation 2020-08-28 04:21:08 +03:00
Alexey Milovidov
5763737d97 Fixups 2020-08-28 03:53:22 +03:00
Alexey Milovidov
ed1d120de0 Added another test; fixup 2020-08-28 03:28:37 +03:00
Alexey Milovidov
c4e8aaac16 Fixups 2020-08-28 03:07:51 +03:00
Alexey Milovidov
6082697c4d Support for quorum 2020-08-28 02:39:12 +03:00
Alexey Milovidov
cdba5e727c Fix mistake 2020-08-28 02:30:07 +03:00
Alexey Milovidov
88db4938f5 Fix error; refinements 2020-08-28 02:22:00 +03:00
Alexey Milovidov
c04dc4e722 Merge branch 'master' into manual-write-duplicate-parts-to-replicas 2020-08-28 01:56:11 +03:00
Alexander Tokmakov
b521ca9b18 save structure for all table functions 2020-08-28 00:05:52 +03:00
Alexander Kuzmenkov
2bef406200 remove accidental changes 2020-08-27 21:49:06 +03:00
Amos Bird
1476a9e236
Add replicated_merge_tree_settings table 2020-08-27 22:41:59 +08:00
Amos Bird
05a5a13e08
specific ReplicatedMergeTree settings 2020-08-27 22:37:41 +08:00
Amos Bird
078b14610d
ALTER MODIFY SAMPLE BY 2020-08-27 22:31:30 +08:00
Amos Bird
0c1cf22c00
Allow FETCH PARTITION from other zookeepers 2020-08-27 22:19:18 +08:00
alesapin
32db38b4d2 Fix compression codec in WAL 2020-08-27 17:06:14 +03:00
alesapin
d426751e26 Default codec if part is not stored on disk 2020-08-27 16:32:23 +03:00
alesapin
0ff5e5412e Merge branch 'master' into parts_default_compression 2020-08-27 16:01:37 +03:00
alexey-milovidov
ea0122de7e
Merge pull request #14120 from ClickHouse/parallel-insert-more-storages 
Support parallel INSERT for more table engines
 2020-08-27 15:08:56 +03:00
alexey-milovidov
9baa0fbf81
Merge pull request #14129 from CurtizJ/fix-performance 
Slightly optimize very short queries with LowCardinality
 2020-08-27 15:07:19 +03:00
alesapin
1225c09a02 Fix default codec for in-memory parts 2020-08-27 11:35:55 +03:00
alesapin
886a7b485e Fixed codec in clone part 2020-08-27 09:49:41 +03:00
bharatnc
ffb73adb11 fix style check 2020-08-26 15:31:48 -07:00
bharatnc
593f054b10 minor fix exception message 2020-08-26 15:31:48 -07:00
bharatnc
3a35aedb56 add comments explaining index_granularity_bytes safe guard 2020-08-26 15:31:48 -07:00
bharatnc
4ac9b59cec format file 2020-08-26 15:31:48 -07:00
bharatnc
37416045ba setting min_index_granularity_bytes 
This PR adds a new setting called `min_index_granularity_bytes`.

If the `index_granularity_bytes` is > 0 and is lesser than the
`min_index_granularity_bytes`, throw an exception inside the
`registerStorageMergeTree.cpp` file.
 2020-08-26 15:31:48 -07:00
Azat Khuzhin
a588947fe2 Fix DistributedFilesToInsert metric (zeroed when it should not) 
CurrentMetrics::Increment add amount for specified metric only for the
lifetime of the object, but this is not the intention, since
DistributedFilesToInsert is a gauge and after #10263 it can exit from
the callback (and enter again later, for example after SYSTEM STOP
DISTRIBUTED SEND it will always exit from it, until SYSTEM START
DISTRIBUTED SEND).

So make Increment member of a class (this will also fix possible issues
with substructing value on DROP TABLE).
 2020-08-27 00:43:00 +03:00
Alexander Tokmakov
969940b4c9 write table tructure for table function remote(...) 2020-08-26 23:55:40 +03:00
alesapin
44eb702fbe Remove redundant code 2020-08-26 22:41:57 +03:00
alesapin
3a680dc8fb Fix removal bug 2020-08-26 22:40:04 +03:00
kssenii
4fecfdbe2f Better & cleaner 2020-08-26 18:52:45 +00:00
Anton Popov
eeb78bf291 slightly optimize very short queries with LowCardinality 2020-08-26 21:46:18 +03:00
Alexey Milovidov
2a09aa53cc Support parallel INSERT for more table engines 2020-08-26 19:41:30 +03:00
alesapin
1fb93a8bf2 Remove read helpers 2020-08-26 18:57:06 +03:00
alesapin
18eb0dbcaa Better remove 2020-08-26 18:39:11 +03:00
alesapin
2fc80189af Add default compression codec to merge tree data part 2020-08-26 18:29:46 +03:00
alexey-milovidov
c157b7a685
Merge pull request #14056 from ClickHouse/alternative-14043 
Alternative implementation of #14043
 2020-08-26 13:15:30 +03:00
alesapin
21d78f8513 Better codec description representation 2020-08-26 11:45:13 +03:00
alesapin
7428c46737 Merge branch 'master' into recompressing_in_background 2020-08-26 10:28:03 +03:00
alexey-milovidov
d3a9389465
Merge branch 'master' into bharatnc-ncb/timezones-table 2020-08-26 04:26:37 +03:00
Alexey Milovidov
0e021c6151 Merge branch 'master' into manual-write-duplicate-parts-to-replicas 2020-08-26 03:24:35 +03:00
alexey-milovidov
c1c5474730
Merge pull request #14051 from javisantana/fix/info_in_read_only_table 
adds some info to the exception about zookeeper path
 2020-08-26 02:27:01 +03:00
Alexey Milovidov
0101fa0af9 Fix error 2020-08-26 02:12:16 +03:00
Alexey Milovidov
63feb75306 Some tweaks 2020-08-25 23:49:22 +03:00