thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-19 12:52:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
110,391 Commits 9,333 Branches 1,784 Tags 2.2 GiB
b541ec485d
Commit Graph

892 Commits

Author SHA1 Message Date
kssenii
9461e225bb Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-27 22:47:47 +03:00
kssenii
2647b88a66 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-27 19:38:19 +00:00
alesapin
490ca93162
Merge pull request #30372 from ClickHouse/update_keeper_config 
Updatable keeper configuration.
 2021-10-25 10:37:03 +03:00
kssenii
7de5fca075 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-24 17:04:24 +00:00
kssenii
39b9e9c258 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-22 15:40:41 +03:00
alesapin
d849f56cd4 Fix typo and update NuRaft 2021-10-22 11:24:18 +03:00
mergify[bot]
b043a04767
Merge branch 'master' into update_keeper_config 2021-10-22 07:12:42 +00:00
Kseniia Sumarokova
da0021c82f
Merge pull request #30412 from ClickHouse/kssenii-fix-test 
Follow-up for #30282
 2021-10-21 12:48:10 +03:00
mergify[bot]
aee446352a
Merge branch 'master' into update_keeper_config 2021-10-21 07:48:23 +00:00
kssenii
220791b21c Fix 2021-10-20 17:15:07 +00:00
Nikolai Kochetov
8986de7e50
Merge pull request #30384 from amosbird/projection-fix21 
Fix crash when minmax_count projection is used with primary key in partition expr
 2021-10-20 16:17:13 +03:00
mergify[bot]
35b5cf23cb
Merge branch 'master' into update_keeper_config 2021-10-20 09:57:09 +00:00
Kseniia Sumarokova
5324cc8359
Merge pull request #30282 from kssenii/fix-local-less-threads 
Less threads in clickhouse-local, fix Ok. printing
 2021-10-19 22:54:28 +03:00
Amos Bird
aa8bc93be8
Fix build 2021-10-19 23:35:59 +08:00
kssenii
e53335bc6f Better way 2021-10-19 08:21:38 +00:00
alesapin
f2d266acce Merge branch 'master' into update_keeper_config 2021-10-19 10:16:57 +03:00
alesapin
bfe2a937eb At least able to start 2021-10-18 18:27:51 +03:00
alesapin
6d24ca4c3a Review fixes 2021-10-18 12:13:24 +03:00
kssenii
710ee96878 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-18 08:31:25 +00:00
kssenii
1cc511789d Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into fix-local-less-threads 2021-10-18 08:30:59 +00:00
alesapin
f1fe96e194 Merge branch 'master' into debug_keeper 2021-10-18 10:16:07 +03:00
alexey-milovidov
7beffb04ad
Merge pull request #29495 from ClickHouse/improvement_for_strange_mutations 
Minor improvement for strange mutations
 2021-10-17 20:57:06 +03:00
kssenii
4a0bdd296d Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-16 18:35:48 +03:00
kssenii
9525437499 Less threads in local, fix Ok. printing 2021-10-16 10:25:39 +00:00
Nikolai Kochetov
c668696047
Merge pull request #30171 from ClickHouse/remove-stream-interfaces 
Remove stream interfaces
 2021-10-16 09:34:01 +03:00
kssenii
ccb02cac56 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-15 12:36:30 +00:00
kssenii
881ae8617e Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-15 15:09:56 +03:00
Maksim Kita
7beaeb8b13
Merge branch 'master' into indexcache 2021-10-15 12:59:09 +03:00
alesapin
90ff7f05fd Start keeper asynchronously if has connection to other nodes 2021-10-14 13:21:41 +03:00
Nikolai Kochetov
3d3e143c29
Merge pull request #30001 from ClickHouse/remove-streams-from-formats 
Remove streams from formats.
 2021-10-14 12:39:07 +03:00
Nikolai Kochetov
2957971ee3 Remove some last streams. 2021-10-13 21:22:02 +03:00
tavplubix
a69b0d9818
Follow-up to #29901 (#30003) 
* more informative logs

* Update ZooKeeper.cpp

* fix
 2021-10-13 13:21:07 +03:00
kssenii
1c8b1b1133 Better 2021-10-12 09:33:07 +00:00
Nikolai Kochetov
ec18340351 Remove streams from formats. 2021-10-11 19:11:50 +03:00
kssenii
fde6f0507d Merge remote-tracking branch 'origin/retriable-http' into disk-async-read 2021-10-10 20:23:44 +03:00
kssenii
f3f820c6e0 HTTP read buffer retries 2021-10-08 15:41:55 +03:00
kssenii
d6e1d0503d Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-07 16:21:00 +03:00
Alexander Tokmakov
5365cc686f Merge branch 'master' into improvement_for_strange_mutations 2021-10-07 13:26:58 +03:00
Maksim Kita
395119339f Merge branch 'master' into rename-common 2021-10-02 17:15:10 +03:00
Maksim Kita
c966806205
Merge pull request #29529 from kitaisreal/external-xml-loaders-reload-with-server-configuration-support 
ExternalXMLLoaders reload with server configuration support
 2021-10-02 16:26:28 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Nikita Mikhaylov
9756b8dc33
Added an ability to execute more merges and mutations than threads, added new settings (#29140) 2021-10-01 00:26:24 +03:00
Maksim Kita
7edf63162f ExternalXMLLoaders reload with server configuration support 2021-09-29 15:52:58 +03:00
Alexander Tokmakov
d120fdf595 check if query context exist before using cache 2021-09-28 21:42:41 +03:00
Alexander Tokmakov
72b1b2e360 Merge branch 'master' into mvcc_prototype 2021-09-23 22:53:27 +03:00
kssenii
1b9565dfa8 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-09-22 22:08:44 +03:00
Kseniia Sumarokova
17b1dd4bc6
Merge branch 'master' into better-retries-2 2021-09-21 13:55:53 +03:00
kssenii
975033e6e3 Fix 2021-09-21 09:44:54 +00:00
Anton Popov
83e45adad6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-20 20:04:50 +03:00
kssenii
f312760dca Better 2021-09-20 19:57:15 +03:00
tavplubix
66bc619c3e
Merge pull request #28983 from ClickHouse/func_zookeeper_session_uptime 
Add function zookeeperSessionUptime(), fix some flaky tests
 2021-09-20 14:47:49 +03:00
Maksim Kita
ca395e9678 Fixed tests 2021-09-17 18:43:00 +03:00
Maksim Kita
de20e04dfd Added executable user defined functions 2021-09-17 18:42:59 +03:00
Anton Popov
99175f7acc minor enhancements in async inserts 2021-09-16 20:55:34 +03:00
Alexander Tokmakov
5c605189ac Merge branch 'master' into func_zookeeper_session_uptime 2021-09-15 21:10:06 +03:00
Anton Popov
fc17936c12 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-14 23:02:30 +03:00
Alexander Tokmakov
ee8cb93004 add function zookeeperSessionUptime() 2021-09-13 15:19:37 +03:00
alexey-milovidov
ea13a8b562
Merge pull request #28659 from myrrc/improvement/tostring_to_magic_enum 
Improving CH type system with concepts
 2021-09-12 15:26:29 +03:00
Mike Kot
c4385a2c40 Simplifying exception messages 2021-09-11 00:40:18 +02:00
Anton Popov
8e5b3b2f6c Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-09 15:12:34 +03:00
Anton Popov
1f41f4541f change timeouts in async insert to milliseconds 2021-09-08 18:30:04 +03:00
mergify[bot]
64fb384656
Merge branch 'master' into global-merge-executor 2021-09-08 13:15:38 +00:00
Nikita Mikhaylov
3b7fa06540 Review fixes 2021-09-08 12:42:03 +00:00
Nikita Mikhaylov
6062dd0021 Better 2021-09-08 00:21:21 +00:00
Vitaly Baranov
70c6623036
Merge branch 'master' into governance/session_log 2021-09-07 10:12:54 +03:00
Vitaly Baranov
bcc31f1f3e Remove unnecessary changes. 2021-09-07 01:37:28 +03:00
Nikita Mikhaylov
ea0fbf81af Renaming 2021-09-06 12:01:16 +00:00
Anton Popov
c9a0cc5ce9 fix race on table schema in insert-select 2021-09-03 17:23:21 +03:00
Nikita Mikhaylov
a6fe91ca47 get rid of half of allocations 2021-09-02 17:40:29 +00:00
Nikita Mikhaylov
7f21cd7f3d save 2021-09-02 10:39:27 +00:00
Anton Popov
5e421ab272 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-02 02:42:50 +03:00
Anton Popov
7c42ce8370 improvements of async inserts 2021-09-02 02:18:09 +03:00
Nikita Mikhaylov
ceab6feb2a Fix test 2021-09-01 12:35:37 +00:00
Nikita Mikhaylov
f8d4f04294 Merge upstream/master into global-merge-executor (using imerge) 2021-08-31 11:52:11 +00:00
alesapin
525999145e
Merge pull request #26791 from ClickHouse/async-reads 
Experiment with asynchronous readers
 2021-08-31 13:17:45 +03:00
Maksim Kita
4ebd0ae941
Merge pull request #28102 from ClickHouse/executable-table-function 
Merging #23192
 2021-08-31 12:00:50 +03:00
Anton Popov
36ac20681e refactor AsynchronousInsertQueue 2021-08-31 05:16:02 +03:00
Nikita Mikhaylov
c4416906c8 done 2021-08-30 19:37:03 +00:00
Vasily Nemkov
c902afddde Added system.session_log table 
Which logs all the info about LogIn, LogOut and LogIn Failure events.
Additional info that is logged:
- User name
- event type (LogIn, LogOut, LoginFailure)
- Event date\time\time with microseconds
- authentication type (same as for IDENTIFIED BY of CREATE USER statement)
- array of active settings profiles upon login
- array of active roles upon login
- array of changed settings with corresponding values
- client address and port
- interface (TCP\HTTP\MySQL\PostgreSQL, etc.)
- client info (name, version info)
- optional LoginFailure reason text message.

Added some tests to verify that events are properly saved with all necessary info via following interfaces:
- TCP
- HTTP
- MySQL

Known limitations
- Not tested against named HTTP sessions, PostgreSQL and gRPC, hence those are not guaranteed to work 100% properly.
 2021-08-30 18:28:28 +03:00
Anton Popov
7aec53bb8f handle all exceptions during async inserts 2021-08-28 01:07:28 +03:00
Anton Popov
9e67943878 add waiting for async inserts 2021-08-28 00:29:10 +03:00
Alexey Milovidov
7df2532d60 Fix style 2021-08-27 04:29:37 +03:00
Alexey Milovidov
864d01e564 Add pread_fake_async method 2021-08-27 03:35:04 +03:00
Alexey Milovidov
46732e781f Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-27 01:57:48 +03:00
Anton Popov
6d3274c22c Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-26 03:31:46 +03:00
Alexey Milovidov
d8250ab3d6 Preparation 2021-08-26 01:14:42 +03:00
Maksim Kita
6ccdde4a88 Updated TableFunctionExecutable 2021-08-25 22:30:22 +03:00
Alexey Milovidov
8f57216180 Progress on development 2021-08-25 00:45:58 +03:00
alesapin
95cf0634dc Clickhouse-keeper: renames and comments 2021-08-24 15:30:31 +03:00
Vitaly Baranov
63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00
Amos Bird
a2256b1307
Dedicated Mark/Uncompressed cache for skip indices 2021-08-23 15:27:31 +08:00
alexey-milovidov
e620ea15bd
Merge pull request #27900 from amosbird/newf 
getServerPort function
 2021-08-21 22:40:25 +03:00
Amos Bird
997acdc39e
getPort function 2021-08-21 20:33:36 +08:00
alexey-milovidov
5ac6a99542
Merge pull request #27927 from ClickHouse/fix_27832 
Сheck cluster name before creating Distributed
 2021-08-21 10:40:23 +03:00
Alexander Tokmakov
8c6dd18917 check cluster name before creating Distributed 2021-08-20 14:55:04 +03:00
Kruglov Pavel
5a282e8947
Merge pull request #27875 from vitlibar/fix-shutdown-named-session-storage 
Fix shutdown of NamedSessionStorage.
 2021-08-20 11:50:43 +03:00
Vitaly Baranov
9fecda940d Fix shutdown of NamedSessionStorage. 2021-08-19 17:27:03 +03:00
Vitaly Baranov
3b09640d94 Use sessions more. 2021-08-19 02:22:09 +03:00
Vitaly Baranov
65ee9a1272
Merge pull request #26864 from vitlibar/refactor-sessions 
Introduce sessions
 2021-08-19 01:38:51 +03:00
Maksim Kita
75cb85846f
Merge pull request #27804 from amosbird/fixshutdown 
Proper shutdown global context
 2021-08-18 22:11:38 +03:00
Vitaly Baranov
fabd7193bd Code cleanups and improvements. 2021-08-18 14:24:52 +03:00
Vasily Nemkov
51ffc33457 Introduce sessions. 
This is required to add system.session_log table.
 2021-08-18 14:24:52 +03:00
Amos Bird
523de98e2d
Proper shutdown global context 2021-08-18 15:23:32 +08:00
Raúl Marín
d1d47658f5 Merge remote-tracking branch 'blessed/master' into materialization_log 2021-08-10 10:49:05 +02:00
benbiti
039d4673ed fix metric BackgroundMessageBrokerSchedulePoolTask 2021-08-09 15:56:59 +08:00
Anton Popov
3a0d4807a5 Merge branch 'async-insert' of git://github.com/abyss7/ClickHouse into merging-20557 2021-08-06 16:20:02 +03:00
mergify[bot]
dc57254982
Merge branch 'master' into improve_create_or_replace 2021-08-03 11:39:07 +00:00
Raúl Marín
9f32ecca89 Merge remote-tracking branch 'blessed/master' into materialization_log 2021-08-02 16:21:10 +02:00
Nikolai Kochetov
6951e8147d
Merge pull request #27020 from amosbird/shardlevelconstness 
Shard-level const column
 2021-08-02 13:45:20 +03:00
alesapin
181f93c60a
Merge pull request #24997 from evillique/nlp 
NLP functions
 2021-08-02 10:50:40 +03:00
Amos Bird
479d4fa991
Add shardNum() and shardCount() functions 2021-07-31 15:45:39 +08:00
Vitaly Baranov
99929981ab
Merge pull request #25687 from MaxWk/profile-default-database 
add setting profile_default_database
 2021-07-31 08:37:02 +03:00
alesapin
4746002776 Add an ability to build ClickHouse without NLP functions 2021-07-30 16:30:30 +03:00
alesapin
916594fe23 Merge branch 'master' into evillique-nlp 2021-07-30 13:03:10 +03:00
Alexander Tokmakov
47995a053e fix system.zookeeper_log initialization 2021-07-29 17:48:49 +03:00
Alexander Tokmakov
c74bfbf991 Merge branch 'master' into mvcc_prototype 2021-07-28 22:21:48 +03:00
Raúl Marín
4a09c54a04 Merge remote-tracking branch 'blessed/master' into materialization_log 2021-07-28 10:38:47 +02:00
万康
0c203dd43d add missing changes 2021-07-28 14:05:22 +08:00
Raúl Marín
f0125262b7 Add views and it's tables to system.query_log 2021-07-27 16:29:33 +02:00
Raúl Marín
ea5c02a605 WIP 2021-07-27 16:26:27 +02:00
Raúl Marín
f34cb886cc Add materialization log accessor to Context 2021-07-27 16:23:46 +02:00
kevin wan
a56a3c793c
Merge branch 'master' into profile-default-database 2021-07-27 21:08:08 +08:00
mergify[bot]
2929cf3ba8
Merge branch 'master' into system_zookeeper_log 2021-07-26 08:40:55 +00:00
mergify[bot]
044be267d6
Merge branch 'master' into improve_create_or_replace 2021-07-26 08:38:48 +00:00
Vitaly Baranov
db97921b5b Changing default roles affects new sessions only. 2021-07-23 23:23:14 +03:00
Vitaly Baranov
19d5a6ab2f
Merge pull request #26714 from vitlibar/new-function-current-profiles 
New functions currentProfiles(), enabledProfiles(), defaultProfiles().
 2021-07-23 09:10:29 +03:00
Vitaly Baranov
7afcc65060 Add new functions currentProfiles(), enabledProfiles(), defaultProfiles(). 2021-07-22 22:20:53 +03:00
Vitaly Baranov
4f1926550b
Merge pull request #26429 from vitlibar/remove-mysql-wire-context 
Remove MySQLWireContext
 2021-07-19 12:21:24 +03:00
万康
d826115f2a fix code style 2021-07-18 14:56:26 +08:00
Vitaly Baranov
0f8b196682 Remove MySQLWireContext. 2021-07-16 22:21:20 +03:00
Vitaly Baranov
6c74b44729 Fix possible crash when login as dropped user. 2021-07-15 19:01:34 +03:00
Alexander Tokmakov
f0ed6be269 Merge branch 'master' into system_zookeeper_log 2021-07-13 20:49:37 +03:00
Ivan Lezhankin
69e7f082fe Merge remote-tracking branch 'upstream/master' into async-insert 2021-07-13 17:15:32 +03:00
Nikita Mikhaylov
c255f152aa
Merge pull request #26051 from ClickHouse/fix_21184 
Fix sequence_id in MySQL protocol
 2021-07-13 16:03:25 +03:00
Ivan Lezhankin
ab26aed6f9 Merge remote-tracking branch 'upstream/master' into async-insert 2021-07-13 13:54:09 +03:00
Alexey Milovidov
29e8b783ba Comments 2021-07-13 03:59:52 +03:00
万康
60f556be2d add default_database to user 2021-07-12 22:35:07 +08:00
mergify[bot]
b7ae4e4ced
Merge branch 'master' into async-insert 2021-07-12 09:21:37 +00:00
Artur
8113580237 Add warning in debug mode 2021-07-09 14:40:32 +00:00
Alexander Tokmakov
4165ba2a01 add system.zookeeper_log 2021-07-09 17:05:35 +03:00
Artur
146a45a8b3 Adding system.warnings 2021-07-09 09:27:51 +00:00
tavplubix
8333a09258
Update Context.cpp 2021-07-08 13:18:54 +03:00
Alexander Tokmakov
1a470fb777 fix sequence_id in MySQL protocol 2021-07-07 20:03:28 +03:00
alesapin
f0fc12d0dd Merge branch 'master' into evillique-nlp 2021-07-07 12:14:49 +03:00
Alexander Tokmakov
5e44fa32b4 Merge branch 'master' into improve_create_or_replace 2021-07-06 11:50:43 +03:00
Alexander Tokmakov
aa3f0b2032 fix 2021-07-02 19:39:55 +03:00
Maksim Kita
b3e3a3cde0 Fixed tests 2021-07-02 16:36:02 +03:00
Maksim Kita
da8c957167 Aggregator added CompiledExpressionCache 2021-07-01 22:56:36 +03:00
Maksim Kita
5b3fc9b49e Added comment 2021-06-30 01:46:57 +03:00
Maksim Kita
b2177ab723 ExpressionCache destruction fix 2021-06-29 23:40:15 +03:00
Ivan Lezhankin
d7126682b0 Fix shared build 2021-06-29 15:05:46 +03:00
Ivan Lezhankin
1bb41593d1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-28 22:55:56 +03:00
Amos Bird
4da1b8154a
Fix data race in getClusters() 2021-06-26 22:15:57 +08:00
Nikolay Degterinsky
491ddc859d Fixed spelling and code style, added generated ya.make files 2021-06-19 18:52:09 +00:00
Nikolay Degterinsky
014dbcbf49 Moved Lemmatizers & SynonymExtensions to Interpreters 2021-06-19 17:57:46 +00:00
alesapin
9e77ca7182
Merge branch 'master' into nlp 2021-06-17 21:47:41 +03:00
Ivan Lezhankin
37365589ed Merge branch 'master' into async-insert 2021-06-17 16:57:45 +03:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Alexey Milovidov
447d7bb8cd Minor changes 2021-06-14 07:13:35 +03:00
Alexander Tokmakov
6f5cf773ef Merge branch 'master' into mvcc_prototype 2021-06-09 12:50:54 +03:00
Ivan Lezhankin
973cea73b7 Merge branch 'master' into async-insert 2021-06-08 15:10:30 +03:00
Alexander Tokmakov
ceef0665e2 fix 2021-06-08 13:01:49 +03:00
Nikolay Degterinsky
fb217ca916 added LemmaGen lemmatizer 2021-06-05 00:52:35 +00:00
mergify[bot]
d68d5e0ef1
Merge branch 'master' into rename-const-context-ptr 2021-06-04 14:41:41 +00:00
Nikolay Degterinsky
b22d5327da minor ContextPtr fix 2021-06-04 00:33:41 +00:00
Nikolay Degterinsky
480da5838e merged and resolved the conflict in Context.cpp 2021-06-03 20:23:14 +00:00
Nikolay Degterinsky
ed12fb5604 added WordNet synonyms extensions 2021-06-03 19:28:12 +00:00
Nikolay Degterinsky
d12deb8128 implemented function synonyms() for plain extension type 2021-06-03 02:20:42 +00:00
Alexander Tokmakov
9915bd0a8b Merge branch 'master' into mvcc_prototype 2021-06-03 00:52:57 +03:00
alesapin
4a1bf51ad1
Merge branch 'master' into limited_fetches 2021-06-02 23:01:47 +03:00
Ivan Lezhankin
791cbcf0b1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-02 18:50:37 +03:00
Ivan Lezhankin
84c23dc060 Merge branch 'master' into async-insert 2021-06-02 18:06:21 +03:00
Nikolai Kochetov
dbaa6ffc62 Rename ContextConstPtr to ContextPtr. 2021-06-01 15:20:52 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs 
Poco::File to std::filesystem
 2021-05-31 23:17:02 +03:00
Alexander Tokmakov
5692db736c Merge branch 'master' into mvcc_prototype 2021-05-31 21:11:22 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
alesapin
2985def91d Add tests 2021-05-27 15:54:47 +03:00
alesapin
42b2383e01 Basic throttler implementation 2021-05-26 23:37:44 +03:00
kssenii
73f16ee9ee Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-26 23:08:08 +03:00
Maksim Kita
0eabdc7241 CompileExpression cached functions with context fix 2021-05-25 21:46:40 +03:00
kssenii
3dee003f9b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-20 19:20:09 +03:00
Alexander Tokmakov
19337dc227 add config 2021-05-18 22:40:34 +03:00
alesapin
31181095e7 Review fixes 2021-05-18 17:08:56 +03:00
Alexander Tokmakov
f2ab5a05c6 Merge branch 'master' into mvcc_prototype 2021-05-18 13:36:38 +03:00
kssenii
9b8df78fdd Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-17 17:42:05 +03:00
Azat Khuzhin
aeb2c5d88a Preserve dictionaries until storage shutdown 
v2: Context::loadEmbeddedDictionaries()
v3: Context::loadDictionaries()
 2021-05-13 22:03:00 +03:00
Alexander Tokmakov
df2849be90 Merge branch 'master' into mvcc_prototype 2021-05-13 21:48:36 +03:00
kssenii
0527f0ea33 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-12 16:54:18 +03:00
kssenii
393b026f2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-11 16:49:23 +03:00
Amos Bird
264cff6415
Projections 
TODO (suggested by Nikolai)

1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies

3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)

Also need to figure out how prewhere works for projections, and
row_filter_policies.

wip
 2021-05-11 18:12:23 +08:00
Alexey Milovidov
b2ca5cd98b Merge branch 'master' into normalize-bigint 2021-05-11 02:05:40 +03:00
Maksim Kita
c79d7eae21 Refactored interfaces 2021-05-08 17:39:35 +03:00
Alexey Milovidov
54fb40c041 Fix OpenTelemetry 2021-05-05 01:42:14 +03:00
kssenii
3386dc32d3 Change in Functions/* 2021-04-29 00:38:58 +03:00
Ivan Lezhankin
484528ba2c Fix problem with use-after-free inside shared_ptr 2021-04-21 16:19:28 +03:00
Ivan Lezhankin
ee194928d2 CC 2021-04-21 16:19:28 +03:00
Ivan Lezhankin
a36596e303 Add settings 2021-04-19 22:16:34 +03:00
Ivan Lezhankin
567fb44b07 Works with trivial inserts 2021-04-19 20:49:02 +03:00
Ivan Lezhankin
6ed9e34750 [WIP] 2021-04-19 17:51:26 +03:00
Ivan Lezhankin
9e5d274467 Merge remote-tracking branch 'upstream/master' into async-insert 2021-04-14 20:16:05 +03:00
alexey-milovidov
f8ce5547aa
Merge pull request #21318 from azat/servers-pool-wait 
Fix SIGSEGV by waiting servers thread pool
 2021-04-14 14:15:07 +03:00
Nikita Mikhaylov
c333c3dedb review fixes 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
36a8419f60 style 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
4843f86329 use only one connection between initiator and worker 2021-04-13 22:39:42 +03:00
Azat Khuzhin
d59bdfd45d Fix one more lock-order-inversion 
TSan report [1]:

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=11314)
  Cycle in lock order graph: M183938897938677368 (0x000000000000) => M2505 (0x7b9000002008) => M183938897938677368

  Mutex M2505 acquired here while holding mutex M183938897938677368 in thread T6:
    0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a327b6)
    1 std::__1::__libcpp_recursive_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:385:10 (clickhouse-tsan+0x17cdb689)
    2 std::__1::recursive_mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:71:14 (clickhouse-tsan+0x17cdb689)
    3 std::__1::unique_lock<std::__1::recursive_mutex>::unique_lock(std::__1::recursive_mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:119:61 (clickhouse-tsan+0x11e3506f)
    4 DB::Context::getLock() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:517:12 (clickhouse-tsan+0x11e3506f)
    5 DB::Context::getSchedulePool() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1517:17 (clickhouse-tsan+0x11e3506f)
    6 DB::IBackgroundJobExecutor::start() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:158:42 (clickhouse-tsan+0x12be1cda)
    7 DB::StorageMergeTree::startup() obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:112:29 (clickhouse-tsan+0x129ed46e)
    8 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_2::operator()(std::__1::shared_ptr<DB::IStorage> const&) const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:230:16 (clickhouse-tsan+0x11d71fba)
    9 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3::operator()() const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:238:56 (clickhouse-tsan+0x11d71fba)
    10 decltype(std::__1::forward<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(fp)()) std::__1::__invoke<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x11d71fba)
    11 void std::__1::__invoke_void_return_wrapper<void>::__call<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x11d71fba)
    12 std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x11d71fba)
    13 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x11d71fba)
    14 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b3b8e0)
    15 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b3b8e0)
    16 ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b3b8e0)
    17 void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3e600)
    18 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(fp0)...)) std::__1::__invoke_constexpr<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3682:1 (clickhouse-tsan+0x8b3e600)
    19 decltype(auto) std::__1::__apply_tuple_impl<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::__tuple_indices<std::__1::tuple<>&...>) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1415:1 (clickhouse-tsan+0x8b3e600)
    20 decltype(auto) std::__1::apply<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1424:1 (clickhouse-tsan+0x8b3e600)
    21 ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()::operator()() obj-x86_64-linux-gnu/../src/Common/ThreadPool.h:178:13 (clickhouse-tsan+0x8b3e600)
    22 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3e561)
    23 void std::__1::__invoke_void_return_wrapper<void>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x8b3e561)
    24 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x8b3e561)
    25 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x8b3e561)
    26 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b39205)
    27 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b39205)
    28 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b39205)
    29 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3cea8)
    30 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3cea8)
    31 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:280:5 (clickhouse-tsan+0x8b3cea8)
    32 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:291:5 (clickhouse-tsan+0x8b3cea8)

  Mutex M183938897938677368 previously acquired by the same thread here:
    0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a327b6)
    1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse-tsan+0x17cdb4f9)
    2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse-tsan+0x17cdb4f9)
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse-tsan+0x12be1ca9)
    4 DB::IBackgroundJobExecutor::start() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:155:21 (clickhouse-tsan+0x12be1ca9)
    5 DB::StorageMergeTree::startup() obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:112:29 (clickhouse-tsan+0x129ed46e)
    6 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_2::operator()(std::__1::shared_ptr<DB::IStorage> const&) const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:230:16 (clickhouse-tsan+0x11d71fba)
    7 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3::operator()() const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:238:56 (clickhouse-tsan+0x11d71fba)
    8 decltype(std::__1::forward<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(fp)()) std::__1::__invoke<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x11d71fba)
    9 void std::__1::__invoke_void_return_wrapper<void>::__call<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x11d71fba)
    10 std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x11d71fba)
    11 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x11d71fba)
    12 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b3b8e0)
    13 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b3b8e0)
    14 ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b3b8e0)
    15 void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3e600)
    16 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(fp0)...)) std::__1::__invoke_constexpr<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3682:1 (clickhouse-tsan+0x8b3e600)
    17 decltype(auto) std::__1::__apply_tuple_impl<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::__tuple_indices<std::__1::tuple<>&...>) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1415:1 (clickhouse-tsan+0x8b3e600)
    18 decltype(auto) std::__1::apply<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1424:1 (clickhouse-tsan+0x8b3e600)
    19 ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()::operator()() obj-x86_64-linux-gnu/../src/Common/ThreadPool.h:178:13 (clickhouse-tsan+0x8b3e600)
    20 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3e561)
    21 void std::__1::__invoke_void_return_wrapper<void>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x8b3e561)
    22 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x8b3e561)
    23 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x8b3e561)
    24 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b39205)
    25 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b39205)
    26 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b39205)
    27 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3cea8)
    28 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3cea8)
    29 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:280:5 (clickhouse-tsan+0x8b3cea8)
    30 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:291:5 (clickhouse-tsan+0x8b3cea8)

  Mutex M183938897938677368 acquired here while holding mutex M2505 in main thread:
    0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a327b6)
    1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse-tsan+0x17cdb4f9)
    2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse-tsan+0x17cdb4f9)
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse-tsan+0x12be261b)
    4 DB::IBackgroundJobExecutor::finish() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:167:21 (clickhouse-tsan+0x12be261b)
    5 DB::IBackgroundJobExecutor::~IBackgroundJobExecutor() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:185:5 (clickhouse-tsan+0x12be261b)
    6 DB::StorageMergeTree::~StorageMergeTree() obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:174:1 (clickhouse-tsan+0x129ed768)
    7 std::__1::default_delete<DB::StorageMergeTree>::operator()(DB::StorageMergeTree*) const obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:1397:5 (clickhouse-tsan+0x12e48b0b)
    8 std::__1::__shared_ptr_pointer<DB::StorageMergeTree*, std::__1::shared_ptr<DB::StorageMergeTree>::__shared_ptr_default_delete<DB::StorageMergeTree, DB::StorageMergeTree>, std::__1::allocator<DB::StorageMergeTree> >::__on_zero_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2565:5 (clickhouse-tsan+0x12e48b0b)
    9 std::__1::__shared_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2475:9 (clickhouse-tsan+0x125b53ea)
    10 std::__1::__shared_weak_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2517:27 (clickhouse-tsan+0x125b53ea)
    11 std::__1::shared_ptr<DB::IStorage>::~shared_ptr() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3212:19 (clickhouse-tsan+0x125b53ea)
    12 DB::SystemLog<DB::AsynchronousMetricLogElement>::~SystemLog() obj-x86_64-linux-gnu/../src/Interpreters/SystemLog.h:118:7 (clickhouse-tsan+0x125b53ea)
    13 std::__1::allocator<DB::AsynchronousMetricLog>::destroy(DB::AsynchronousMetricLog*) obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:891:15 (clickhouse-tsan+0x125b1dd8)
    14 void std::__1::allocator_traits<std::__1::allocator<DB::AsynchronousMetricLog> >::__destroy<DB::AsynchronousMetricLog>(std::__1::integral_constant<bool, true>, std::__1::allocator<DB::AsynchronousMetricLog>&, DB::AsynchronousMetricLog*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:539:21 (clickhouse-tsan+0x125b1dd8)
    15 void std::__1::allocator_traits<std::__1::allocator<DB::AsynchronousMetricLog> >::destroy<DB::AsynchronousMetricLog>(std::__1::allocator<DB::AsynchronousMetricLog>&, DB::AsynchronousMetricLog*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:487:14 (clickhouse-tsan+0x125b1dd8)
    16 std::__1::__shared_ptr_emplace<DB::AsynchronousMetricLog, std::__1::allocator<DB::AsynchronousMetricLog> >::__on_zero_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2611:9 (clickhouse-tsan+0x125b1dd8)
    17 std::__1::__shared_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2475:9 (clickhouse-tsan+0x125904ff)
    18 std::__1::__shared_weak_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2517:27 (clickhouse-tsan+0x125904ff)
    19 std::__1::shared_ptr<DB::AsynchronousMetricLog>::~shared_ptr() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3212:19 (clickhouse-tsan+0x125904ff)
    20 DB::SystemLogs::~SystemLogs() obj-x86_64-linux-gnu/../src/Interpreters/SystemLog.cpp:155:1 (clickhouse-tsan+0x125904ff)
    21 std::__1::__optional_destruct_base<DB::SystemLogs, false>::reset() obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:245:21 (clickhouse-tsan+0x11e43655)
    22 DB::ContextShared::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:443:21 (clickhouse-tsan+0x11e43655)
    23 DB::Context::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2251:13 (clickhouse-tsan+0x11e3be37)
    24 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5::operator()() const obj-x86_64-linux-gnu/../programs/server/Server.cpp:892:5 (clickhouse-tsan+0x8abacc2)
    25 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::invoke() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:97:9 (clickhouse-tsan+0x8abacc2)
    26 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::~basic_scope_guard() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:47:28 (clickhouse-tsan+0x8abacc2)
    27 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:1395:1 (clickhouse-tsan+0x8ab5cba)
    28 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b4ac7b)
    29 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa97be)
    30 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b67093)
    31 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8aa838e)
    32 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8aa68f9)

  Mutex M2505 previously acquired by the same thread here:
    0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a327b6)
    1 std::__1::__libcpp_recursive_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:385:10 (clickhouse-tsan+0x17cdb689)
    2 std::__1::recursive_mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:71:14 (clickhouse-tsan+0x17cdb689)
    3 std::__1::lock_guard<std::__1::recursive_mutex>::lock_guard(std::__1::recursive_mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse-tsan+0x11e4363f)
    4 DB::ContextShared::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:438:21 (clickhouse-tsan+0x11e4363f)
    5 DB::Context::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2251:13 (clickhouse-tsan+0x11e3be37)
    6 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5::operator()() const obj-x86_64-linux-gnu/../programs/server/Server.cpp:892:5 (clickhouse-tsan+0x8abacc2)
    7 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::invoke() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:97:9 (clickhouse-tsan+0x8abacc2)
    8 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::~basic_scope_guard() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:47:28 (clickhouse-tsan+0x8abacc2)
    9 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:1395:1 (clickhouse-tsan+0x8ab5cba)
    10 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b4ac7b)
    11 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa97be)
    12 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b67093)
    13 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8aa838e)
    14 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8aa68f9)

  [1]: https://clickhouse-test-reports.s3.yandex.net/21318/f3b1ad0f5d1024275674e1beac24251ae97c8453/functional_stateful_tests_(thread).html#fail1

v2: Convert ContextSharedPart::system_logs to std::unique_ptr (to avoid copy ctor)
v3: Fix readability-identifier-naming,-warnings-as-errors for system_logs_
v4: fix conflicts
 2021-04-13 21:37:38 +03:00
Azat Khuzhin
3056d2c5d1 Fix data-race during server shutdown the context 
Found with 01737_clickhouse_server_wait_server_pool:

==================
WARNING: ThreadSanitizer: data race (pid=13248)
  Write of size 1 at 0x7b9000003a38 by main thread:
    0 std::__1::__optional_destruct_base<DB::SystemLogs, false>::reset() obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:246:24 (clickhouse-tsan+0x11e3043e)
    1 DB::ContextShared::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:441:21 (clickhouse-tsan+0x11e3043e)
    2 DB::Context::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2249:13 (clickhouse-tsan+0x11e28c17)
    3 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5::operator()() const obj-x86_64-linux-gnu/../programs/server/Server.cpp:892:5 (clickhouse-tsan+0x8ab1a32)
    4 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::invoke() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:97:9 (clickhouse-tsan+0x8ab1a32)
    5 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::~basic_scope_guard() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:47:28 (clickhouse-tsan+0x8ab1a32)
    6 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:1395:1 (clickhouse-tsan+0x8aacb39)
    7 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b3446b)
    8 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa04be)
    9 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b50883)
    10 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8a9f08e)
    11 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8a9d5f9)

  Previous read of size 1 at 0x7b9000003a38 by thread T2 (mutexes: write M2504):
    0 std::__1::__optional_storage_base<DB::SystemLogs, false>::has_value() const obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:295:22 (clickhouse-tsan+0x11e25348)
    1 std::__1::optional<DB::SystemLogs>::operator bool() const obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:938:64 (clickhouse-tsan+0x11e25348)
    2 DB::Context::getQueryLog() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1875:10 (clickhouse-tsan+0x11e25348)
    3 DB::executeQueryImpl(char const*, char const*, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:657:50 (clickhouse-tsan+0x12653751)
    4 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:904:30 (clickhouse-tsan+0x12651308)
    5 DB::TCPHandler::runImpl() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:289:24 (clickhouse-tsan+0x12f04b45)
    6 DB::TCPHandler::run() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1500:9 (clickhouse-tsan+0x12f13907)
    7 Poco::Net::TCPServerConnection::start() obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse-tsan+0x15b1f722)
    8 Poco::Net::TCPServerDispatcher::run() obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:113:19 (clickhouse-tsan+0x15b1fe4e)
    9 Poco::PooledThread::run() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse-tsan+0x15c86fe1)
    10 Poco::(anonymous namespace)::RunnableHolder::run() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse-tsan+0x15c8557f)
    11 Poco::ThreadImpl::runnableEntry(void*) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse-tsan+0x15c83d87)

  Location is heap block of size 7296 at 0x7b9000002000 allocated by main thread:
    0 operator new(unsigned long) <null> (clickhouse-tsan+0x8a9aae7)
    1 std::__1::__unique_if<DB::ContextShared>::__unique_single std::__1::make_unique<DB::ContextShared>() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2068:28 (clickhouse-tsan+0x11e15c2c)
    2 DB::Context::createShared() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:503:32 (clickhouse-tsan+0x11e15c2c)
    3 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:426:27 (clickhouse-tsan+0x8aa19ee)
    4 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b3446b)
    5 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa04be)
    6 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b50883)
    7 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8a9f08e)
    8 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8a9d5f9)

  Mutex M2504 (0x7b9000002008) created at:
    0 pthread_mutex_init <null> (clickhouse-tsan+0x8a0d37d)
    1 std::__1::__libcpp_recursive_mutex_init(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:370:10 (clickhouse-tsan+0x17cc4d93)
    2 std::__1::recursive_mutex::recursive_mutex() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:56:14 (clickhouse-tsan+0x17cc4d93)
    3 DB::ContextShared::ContextShared() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:394:5 (clickhouse-tsan+0x11e40bc3)
    4 std::__1::__unique_if<DB::ContextShared>::__unique_single std::__1::make_unique<DB::ContextShared>() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2068:32 (clickhouse-tsan+0x11e15c37)
    5 DB::Context::createShared() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:503:32 (clickhouse-tsan+0x11e15c37)
    6 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:426:27 (clickhouse-tsan+0x8aa19ee)
    7 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b3446b)
    8 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa04be)
    9 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b50883)
    10 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8a9f08e)
    11 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8a9d5f9)

  Thread T2 'TCPHandler' (tid=13643, running) created by main thread at:
    0 pthread_create <null> (clickhouse-tsan+0x8a0bf0b)
    1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable> >) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:202:6 (clickhouse-tsan+0x15c83827)
    2 Poco::Thread::start(Poco::Runnable&) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:128:2 (clickhouse-tsan+0x15c84f6c)
    3 Poco::PooledThread::start() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:85:10 (clickhouse-tsan+0x15c873e2)
    4 Poco::ThreadPool::ThreadPool(int, int, int, int) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:252:12 (clickhouse-tsan+0x15c873e2)
    5 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:843:22 (clickhouse-tsan+0x8aa8e5f)
    6 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b3446b)
    7 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa04be)
    8 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b50883)
    9 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8a9f08e)
    10 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8a9d5f9)

SUMMARY: ThreadSanitizer: data race obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:246:24 in std::__1::__optional_destruct_base<DB::SystemLogs, false>::reset()

v2: fix deadlock by calling SystemLogs::shutdown w/o Context lock
 2021-04-13 21:37:38 +03:00
Pavel Kruglov
e4f3678ae9 Fix build 2021-04-12 12:28:46 +03:00
alexey-milovidov
d21ffd83a7
Merge branch 'master' into catboost-reload 2021-04-11 13:29:36 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
alexey-milovidov
5469cc15c1
Trigger CI 2021-04-10 23:43:56 +03:00
Alexander Tokmakov
f270fa1843 rollback on exception 2021-04-09 15:53:51 +03:00
alesapin
2987bbc948 Small improvements 2021-04-07 16:52:11 +03:00
alesapin
1c8e33414f Merge branch 'master' into johnskopis/dynamic-interserver-creds-v20 2021-04-06 16:28:46 +03:00
kssenii
9dba529494 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-04-05 13:14:51 +00:00
Alexander Tokmakov
f596906f47 add transaction counters 2021-04-05 14:41:34 +03:00
Pavel Kruglov
dff7834af0 Fix config comparison 2021-04-02 13:17:28 +03:00
Pavel Kruglov
cc65a1db97 Update paths to the model configs in config reloading 2021-04-01 13:30:05 +03:00
alesapin
5e3ebb3e0e
Merge pull request #22274 from ClickHouse/rename_keeper 
Rename NuKeeper and TestKeeper to Keeper in all places
 2021-03-30 10:39:22 +03:00
tavplubix
3c0f5a57ec
Merge pull request #21535 from ClickHouse/distributed_ddl_improvements 
Distributed DDL improvements
 2021-03-29 22:40:11 +03:00
Ivan Lezhankin
de05803177 Merge branch 'master' into async-insert 2021-03-29 20:59:02 +03:00
alesapin
be132a32a2 More renames 2021-03-29 11:24:56 +03:00
alexey-milovidov
f90a328568
Merge pull request #22109 from azat/nested-epoll-wait 
Disable async_socket_for_remote/use_hedged_requests for buggy linux kernels
 2021-03-29 05:29:10 +03:00
alexey-milovidov
3bc170e5ec
Merge pull request #22122 from fastio/fix_thread_pool_name_exception 
The thread pool name should be not longer than 15 chars
 2021-03-29 05:24:55 +03:00
Alexey Milovidov
2a8ac01cdb Rename as suggested by Kita 2021-03-28 22:24:28 +03:00
Alexey Milovidov
50f712e198 Integrate mmap cache to the infrastructure 2021-03-28 04:10:30 +03:00
Azat Khuzhin
8e445c5e55 Disable async_socket_for_remote/use_hedged_requests for buggy linux kernels 
async_socket_for_remote/use_hedged_requests uses nested epolls, which
may not be reliable in 5.5+ [1], but it has been fixed in shortly in
5.7+ [2] or
5.6.13+.

  [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=339ddb53d373
  [2]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c54a6a44bf3
 2021-03-25 23:39:30 +03:00
Peng Jian
2ede490152 The thread pool name should not longer than 15 chars 2021-03-25 21:57:47 +08:00
tavplubix
5dfd0045a6
Merge branch 'master' into distributed_ddl_improvements 2021-03-22 18:13:19 +03:00
kssenii
8a650c4408 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-03-22 13:38:25 +00:00
Ivan Lezhankin
4f1fad97ab Merge remote-tracking branch 'upstream/master' into async-insert 2021-03-19 23:41:01 +03:00
Nikolai Kochetov
c3c393a7aa Merge branch 'master' into refactor-actions-dag 2021-03-18 14:33:07 +03:00
Ivan Lezhankin
346818266a [WIP] 2021-03-17 17:11:47 +03:00
kssenii
34dea63248 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-03-17 13:32:43 +00:00
kssenii
9662a492c9 Proper bridge start 2021-03-17 08:21:04 +00:00
Anton Popov
840417c957
Merge pull request #21685 from Avogar/reload-cluster-config 
Update clusters only if their configs were updated
 2021-03-15 20:09:58 +03:00
tavplubix
6a265f468e
Merge branch 'master' into distributed_ddl_improvements 2021-03-15 11:24:14 +03:00
Vitaly Baranov
d6e0342c30 Improvements in implementations of the classes AccessRights and GrantedRoles. 2021-03-14 19:12:35 +03:00
Pavel Kruglov
ee483d89e5 Fix style 2021-03-13 23:18:49 +03:00
Pavel Kruglov
58e415c7d4 Update clusters only if their configs were updated 2021-03-13 00:17:19 +03:00
Denis Glazachev
290a6d273e
Add Kerberos support for authenticating existing users when accessing over HTTP 2021-03-11 23:41:10 +03:00
Alexander Tokmakov
83b3e4e0f5 allow to use Cluster from Replicated db 2021-03-09 20:05:24 +03:00
Nikolai Kochetov
15921fbfcb Remove index by name from ActionsDAG 2021-03-03 23:01:07 +03:00
alesapin
a48bb7e3a0 Fix redundant zookeeper reconnect and possibility of multiple zookeeper session for a single ClickHouse server 2021-02-26 22:05:25 +03:00
Alexander Tokmakov
2a36d6cb55 review suggestions 2021-02-20 02:41:58 +03:00
Alexander Tokmakov
7b54b892b5 fix 2021-02-16 17:05:58 +03:00
Alexander Tokmakov
bf6f64a3fb Merge branch 'master' into database_replicated 2021-02-16 01:28:19 +03:00
alesapin
f80137626a
Merge pull request #19580 from ClickHouse/in_memory_raft 
In memory coordination inside ClickHouse
 2021-02-13 10:19:23 +03:00
alesapin
5f32f65211 Merge branch 'master' into in_memory_raft 2021-02-11 23:09:29 +03:00
Azat Khuzhin
935870b2c2 Add separate config directive for Buffer profile 
If you push data via Buffer engine then all your queries will be done
from one user, however this is not always desired behavior, since this
will not allow to limit queries with max_concurrent_queries_for_user and
similar.
 2021-02-10 21:40:26 +03:00
alesapin
9667bdcbd2 Merge branch 'master' into in_memory_raft 2021-02-09 16:06:31 +03:00
alesapin
15ce3dc112 Fix access control manager destruction order 2021-02-09 11:24:39 +03:00
Alexander Tokmakov
5e0cbfe972 Merge branch 'master' into database_replicated 2021-02-08 23:01:12 +03:00
Alexander Tokmakov
78c1d69b8c better code 2021-02-08 22:36:17 +03:00
tavplubix
ac477d9850
Merge pull request #19771 from ClickHouse/thread_state_improvements 
Minor code improvements around ThreadStatus
 2021-02-08 22:34:55 +03:00
alesapin
d5ee7d33e9 Merge branch 'master' into in_memory_raft 2021-02-08 10:30:31 +03:00
Alexander Tokmakov
87502d0220 Merge branch 'thread_state_improvements' into database_replicated 2021-02-03 20:19:35 +03:00
Alexander Tokmakov
d010f97db0 Merge branch 'master' into database_replicated 2021-02-03 20:13:25 +03:00
Aleksei Semiglazov
921518db0a CLICKHOUSE-606: query deduplication based on parts' UUID 
* add the query data deduplication excluding duplicated parts in MergeTree family engines.

query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1

allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.

data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.

NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.

* add _part_uuid virtual column, allowing to use UUIDs in predicates.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>

address comments
 2021-02-02 16:53:39 +00:00
alesapin
d265e3b419 Less timeouts 2021-02-02 11:02:25 +03:00
alesapin
365bf65f5a Fix install script 2021-02-01 17:14:59 +03:00
alesapin
57c9b6c864 Fix build without nuraft 2021-02-01 16:18:17 +03:00
alesapin
0aca40d8cb Merge branch 'master' into in_memory_raft 2021-02-01 14:30:11 +03:00
alexey-milovidov
d93dad0ec3
Merge pull request #19722 from azat/background_message_broker_schedule_pool_size 
Add separate pool for message brokers (RabbitMQ and Kafka)
 2021-01-31 06:07:45 +03:00
alesapin
8116d039f9 Merge branch 'master' into in_memory_raft 2021-01-29 18:55:46 +03:00
Alexander Tokmakov
004517009e fix 2021-01-29 18:13:27 +03:00
Alexey Milovidov
5a92e633a1 Remove useless headers 2021-01-29 10:37:46 +03:00
Azat Khuzhin
acde56ff8a Add separate pool for message brokers (RabbitMQ and Kafka) 
background_message_broker_schedule_pool_size was introduced in #13939,
but never used actually, use it for RabbitMQ and Kafka.
 2021-01-27 21:08:41 +03:00
alesapin
a33963e211 Better raft server startup 2021-01-27 20:54:25 +03:00
Alexander Tokmakov
f20d5e3b41 fix 2021-01-26 20:51:25 +03:00
alesapin
3935d51b14 Fix segfault 2021-01-26 17:23:10 +03:00
alesapin
817eb100a1 Better shutdown 2021-01-26 17:08:31 +03:00
alesapin
71dca6dc00 Tidy fixes 2021-01-26 11:17:19 +03:00
alesapin
dea4b5009b Some server initialization 2021-01-25 15:29:12 +03:00
Alexander Tokmakov
3bd4d97353 Merge branch 'master' into database_replicated 2021-01-25 14:19:04 +03:00
alesapin
173b6fefdf Merge branch 'master' into in_memory_raft 2021-01-25 12:03:28 +03:00
alexey-milovidov
ba3e064a63
Merge pull request #19371 from kssenii/test-coverage-with-factories 
test coverage with factories
 2021-01-25 07:11:09 +03:00
kssenii
0d940c6137 Add AggregateFunctionCombinatorFactory, better names 2021-01-23 12:07:37 +00:00
alesapin
4a06bf1d79 Merge branch 'fix_one_more_tiny_race' into in_memory_raft 2021-01-22 15:56:43 +03:00
kssenii
daab2c91bb Better 2021-01-21 21:15:11 +00:00
Nikita Mikhaylov
dca0cbf4eb
Merge pull request #19313 from nikitamikhaylov/another-race-cache-dictionary 
Fixed race between copy-constructor and addQueryAccessInfo
 2021-01-21 21:02:36 +03:00
kssenii
c1702f34ee Add factories info into system.query_log 2021-01-21 15:46:37 +00:00
alesapin
4aa11b3494 Remove zkutil namespace from TestKeeperStorage 2021-01-21 18:09:48 +03:00
alesapin
a241793a14 Merge branch 'decompose_test_keeper_storage' into in_memory_raft 2021-01-20 15:51:54 +03:00
Nikita Mikhailov
cf88874088 done 2021-01-20 15:30:41 +03:00
alesapin
6c6bf60937 Rename function 2021-01-19 18:23:32 +03:00
alesapin
ac64a13392 Split storage and requests processing 2021-01-19 17:22:28 +03:00
alesapin
6896558424 Some code movements 2021-01-18 19:03:26 +03:00
Alexander Tokmakov
7f97a11c84 Merge branch 'master' into database_replicated 2021-01-18 17:09:39 +03:00
Alexander Tokmakov
5f0f8ae3cd fix duplicate UUIDs of LiveView on server startup 2021-01-13 21:13:46 +03:00
John Skopis
9bcc5a6175 Support interserver credential rotation 
Restarting a server instance to change the interserver password results
in many replicas being out of sync until all clusters are using the new
credential.

This commit adds dynamic credential loading for both the client
(Replicated* tables) and server (InterserverIOHTTPHandler).

This commit also adds the ability to rotate credentials, i.e. accept more
than one credential during a credential change.

state0 (no auth):

    <interserver_http_credentials />

state1 (auth+allow_empty migration):

    <interserver_http_credentials>
        <user>admin</user>
        <password>222</password>
        <allow_empty>true</allow_empty>
    </interserver_http_credentials>

state2 (auth+new admin password migration):

    <interserver_http_credentials>
        <user>admin</user>
        <password>333</password>
        <users>
            <admin>222</admin>
        </users>
    </interserver_http_credentials>
 2021-01-04 09:59:09 +00:00
Nikita Mikhaylov
4165a58ce4
Merge pull request #11617 from nikitamikhaylov/parallel-parsing-input-format 
Parallel formatting
 2020-12-31 21:22:16 +04:00
Nikita Mikhailov
c5f92e5096 better formatfactory 2020-12-30 06:07:30 +03:00
Alexander Tokmakov
21c3fc0e16 check settings constraints in setProfile(...) 2020-12-24 20:48:54 +03:00
Nikita Mikhailov
dcfbe782c6 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-23 05:20:22 +03:00
Amos Bird
2a28c127eb
Better code for real 2020-12-18 14:13:59 +08:00
Amos Bird
2f92d57af9
Better code. 2020-12-18 14:13:58 +08:00
Amos Bird
81d7f8ef2d
Need proper locking 2020-12-18 14:13:58 +08:00
Amos Bird
272311f6bd
Extensive information in system.query_log 2020-12-18 14:13:58 +08:00
Nikita Mikhailov
6a55e018e4 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-18 02:25:32 +03:00
alesapin
b347fa322c Trying to fix races one more time 2020-12-16 15:58:00 +03:00
nikitamikhaylov
12e624fd9a fix tests 2020-12-15 00:56:48 +03:00
Nikita Mikhaylov
e0addac6fc save changes 2020-12-15 00:56:46 +03:00
alesapin
278bbf6d51 Merge branch 'master' into complete_zk_api 2020-12-04 16:48:19 +03:00
Alexey Milovidov
a2eb7c96df Fix ugly bug 2020-12-03 19:51:27 +03:00
Alexander Tokmakov
1a4bd67736 fixes 2020-12-01 20:20:42 +03:00
Alexander Tokmakov
9e3fd3c170 Merge branch 'master' into database_replicated 2020-11-27 17:08:34 +03:00
Alexander Tokmakov
f1a52a609b separate DatabaseReplicatedDDLWorker 2020-11-27 17:04:03 +03:00
alesapin
d199a2bf76 Merge branch 'master' into complete_zk_api 2020-11-27 14:55:55 +03:00
alexey-milovidov
dfae1efbbd
Merge pull request #17070 from fastio/master 
Support multiple ZooKeeper clusters
 2020-11-27 10:38:01 +03:00
alesapin
310307fa72 Replace eventfd with pipes 2020-11-25 11:18:15 +03:00
Alexander Kuzmenkov
1d7dcef4a0 Merge remote-tracking branch 'origin/master' into HEAD 2020-11-24 21:45:10 +03:00
Alexander Tokmakov
5ccf975a4a Merge branch 'master' into database_replicated 2020-11-24 14:38:02 +03:00
alesapin
b8fb44a7b9 Merge with master 2020-11-23 11:05:50 +03:00
Alexander Tokmakov
7ab4445e99 try another approach 2020-11-20 19:06:27 +03:00
Amos Bird
1d9d586e20
Make global_context consistent. 2020-11-20 18:23:14 +08:00
alesapin
1aba61be7f Merge branch 'master' into complete_zk_api 2020-11-19 13:32:15 +03:00
Peng Jian
a0683ce460 Support mulitple ZooKeeper clusters 2020-11-19 15:44:47 +08:00
Alexander Kuzmenkov
1570320e20 fixes for context hierarchy 2020-11-18 20:43:18 +03:00
alesapin
2cd66f23a3 Merge branch 'master' into complete_zk_api 2020-11-18 17:50:50 +03:00
nikitamikhaylov
6f4e5bcaac style 2020-11-17 17:39:34 +03:00
nikitamikhaylov
ee8b3339a1 done 2020-11-17 17:39:34 +03:00
Alexander Tokmakov
0731bfb7be Merge branch 'master' into database_replicated 2020-11-12 21:04:18 +03:00
alexey-milovidov
c9c8f50683
Merge pull request #16903 from azat/no-trace-collector-without-trace_log 
Disable TraceCollector if trace_log is disabled
 2020-11-12 19:43:27 +03:00
alesapin
4aad6a597c Merge branch 'correctly_send_close_request' into complete_zk_api 2020-11-12 17:46:02 +03:00
tavplubix
058aa8f85e
Merge pull request #16824 from ClickHouse/replace_stringstreams_with_buffers 
Replace std::*stringstreams with DB::*Buffers
 2020-11-12 01:11:44 +03:00
Azat Khuzhin
719230ecd9 Disable TraceCollector if trace_log is disabled 2020-11-11 21:29:05 +03:00
alesapin
6398271432 Merge branch 'master' into complete_zk_api 2020-11-11 16:57:24 +03:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
alesapin
932ea0ef8b Merge branch 'master' into complete_zk_api 2020-11-10 13:56:11 +03:00
Amos Bird
aa8e6db786
reload auxiliary zookeepers configuration 2020-11-10 15:56:37 +08:00
Alexey Milovidov
dba57529f8 Merge branch 'master' of github.com:yandex/ClickHouse into fix-stringstream 2020-11-07 22:03:38 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Alexey Milovidov
28c6dea1a2 LOGICAL_ERROR was mistakenly used for some ErrorCodes #12735 2020-11-06 21:02:16 +03:00
alexey-milovidov
ee3e289ec2
Merge pull request #15983 from ClickHouse/no_background_pool_no_more 
Replace BackgroundProcessingPool with SchedulePool task and ThreadPool.
 2020-11-05 21:38:22 +03:00
alexey-milovidov
654d127ce6
Merge pull request #16630 from sundy-li/patch-6 
better change the config
 2020-11-04 11:36:07 +03:00
alexey-milovidov
74ebe952d1
Update Context.cpp 2020-11-04 11:32:20 +03:00
alesapin
a02b4322b6 Merge branch 'master' into complete_zk_api 2020-11-03 13:13:07 +03:00
sundy-li
af5ecbef27 set zookeeper config 2020-11-03 17:57:24 +08:00
alesapin
101aa9e931 Merge branch 'master' into no_background_pool_no_more 2020-11-02 10:49:19 +03:00
alesapin
c2525ef211 Server and client pinging each other 2020-10-30 17:16:47 +03:00
alesapin
40fc512e79 Merge branch 'master' into no_background_pool_no_more 2020-10-29 12:53:34 +03:00
alesapin
1c9e24b5e2 Merge branch 'master' into system_fetches_table 2020-10-28 10:20:04 +03:00
alesapin
3824fcb076 Rename file and some tweaks 2020-10-27 15:47:42 +03:00
alesapin
880f4bbd05 System fetches 2020-10-26 19:38:35 +03:00
Alexander Kuzmenkov
5a70e26c70 Merge remote-tracking branch 'origin/master' into HEAD 2020-10-26 19:21:13 +03:00
Alexander Kuzmenkov
6552613118
Merge pull request #16280 from filimonov/clickhouse-local-tmp-folder 
clickhouse-local can work without tmp directory
 2020-10-26 19:10:15 +03:00
Alexander Kuzmenkov
79a6be08d1
Update src/Interpreters/Context.cpp 2020-10-26 19:09:53 +03:00
alesapin
9ed4668dbb Refactor common part of background list 2020-10-26 15:40:55 +03:00
Alexander Tokmakov
636da1318b Merge branch 'master' into database_replicated 2020-10-26 15:00:43 +03:00
alesapin
d5d0819698 Merge branch 'master' into no_background_pool_no_more 2020-10-26 12:58:51 +03:00
alesapin
f97e6beb70 Remove message broker pool from context 2020-10-25 15:34:18 +03:00
Mikhail Filimonov
f3a18864d5
After CR fixes - better & simpler. Issue in Context::shutdown fixed 2020-10-23 16:57:06 +02:00
Denny Crane
b310d0abb4
Fix for incorrect error message 
max_[table/partition]_size_to_drop can be changed without restart since https://github.com/ClickHouse/ClickHouse/pull/7779
 2020-10-22 15:22:17 -03:00
Alexander Kuzmenkov
5cbf645de4 Merge remote-tracking branch 'origin/master' into tmp 2020-10-22 19:48:02 +03:00
Alexander Kuzmenkov
145e2b012f cleanup 2020-10-22 19:47:20 +03:00
alesapin
7a63b373f9 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 15:47:11 +03:00
Alexander Tokmakov
72dbb43235 Merge branch 'master' into fork-valbaturin-replication-db-engine 2020-10-20 19:14:54 +03:00
Vladimir Chebotarev
aa5f207fd4
Added disable_merges option for volumes in multi-disk configuration (#13956) 
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-10-20 18:10:24 +03:00
alesapin
7edbe4d276 Add some comments 2020-10-16 13:12:31 +03:00
alesapin
0fb1f74b2e Trying to schedule better 2020-10-16 11:00:15 +03:00
alesapin
4c1f5cab13 Sleep less in background task 2020-10-15 19:10:22 +03:00
alesapin
60d14352f8 Merge branch 'master' into no_background_pool_no_more 2020-10-15 16:34:14 +03:00
Alexander Kuzmenkov
b4378b004f Merge remote-tracking branch 'origin/master' into tmp 2020-10-15 16:01:41 +03:00
alesapin
3394c22c03 Fix race and clang-tidy 2020-10-15 10:39:27 +03:00
alesapin
a535a089d1 Some style fixes 2020-10-14 19:09:18 +03:00
alesapin
0b14a31ba9 Get rid of background processing pool 2020-10-14 15:44:10 +03:00
Alexander Tokmakov
72b1339656 Revert "Revert "Write structure of table functions to metadata"" 
This reverts commit c65d1e5c70.
 2020-10-14 15:19:29 +03:00
tavplubix
c65d1e5c70
Revert "Write structure of table functions to metadata" 2020-10-14 13:59:29 +03:00
alexey-milovidov
f60ccb4edf
Merge pull request #14295 from ClickHouse/write_structure_of_table_functions 
Write structure of table functions to metadata
 2020-10-13 23:56:09 +03:00
Alexander Kuzmenkov
6200433e12 Merge origin/master into tmp (using imerge) 2020-10-12 21:15:31 +03:00
alexey-milovidov
a8708338f2
Merge pull request #15809 from amosbird/c1 
add print_query_id support for native client
 2020-10-11 13:29:35 +03:00
Amos Bird
ff665c362f
Add print query id support to client 2020-10-11 10:09:08 +08:00
Alexey Milovidov
9f15daf45c Remove useless metrics to avoid confusion 2020-10-10 18:52:44 +03:00
Alexander Tokmakov
6b2da861dc Merge branch 'master' into write_structure_of_table_functions 2020-10-02 20:46:58 +03:00
Alexander Kuzmenkov
218d86de7e Merge remote-tracking branch 'origin/master' into tmp 2020-09-28 16:29:51 +03:00
tavplubix
ffc0499952
Merge pull request #15243 from ClickHouse/tavplubix-patch-1 
Try fix race on Context::settings
 2020-09-28 11:44:27 +03:00
Alexander Tokmakov
b0d99217fb Merge branch 'master' into write_structure_of_table_functions 2020-09-27 14:26:47 +03:00
alesapin
97671a1771 Ugly fix for default database race 2020-09-25 14:27:00 +03:00
tavplubix
5ff5ca1fa6
Update Context.cpp 2020-09-24 20:08:15 +03:00
Alexander Kuzmenkov
7b64ca33b1 Merge remote-tracking branch 'origin/master' into tmp 2020-09-22 14:02:20 +03:00
Alexander Kuzmenkov
bfc3be9e43 Merge remote-tracking branch 'origin/aku/tcp-versio' into tmp 2020-09-22 14:02:15 +03:00
alexey-milovidov
3e0509af69
Merge pull request #13573 from amosbird/rms 
specific ReplicatedMergeTree settings
 2020-09-21 01:36:33 +03:00
Vitaly Baranov
46ab2f76cd
Merge pull request #14919 from vitlibar/add-login-function-to-access-storage 
Add login() function to IAccessStorage
 2020-09-20 16:58:22 +03:00
Vitaly Baranov
dbc837c148 IAccessStorage now can override the login() function. 2020-09-19 01:04:20 +03:00
Alexander Tokmakov
1ca9a92b21 Merge branch 'master' into write_structure_of_table_functions 2020-09-18 21:09:23 +03:00
Vitaly Baranov
3356d75b23
Merge pull request #13156 from azat/cluster-secure 
Secure inter-cluster query execution (with initial_user as current query user) [v3]
 2020-09-17 17:11:00 +03:00
Alexander Kuzmenkov
6217dd77b5 Merge remote-tracking branch 'origin/master' into tmp 2020-09-17 12:35:10 +03:00
sundy-li
544b2cb20d add configChanged method for zookeeper 
fix logic error && skip reload testkeeper
 2020-09-17 13:33:45 +08:00
sundyli
9c2a8ea6d0 dynamic zookeeper config 2020-09-17 13:26:36 +08:00
Azat Khuzhin
a35c1f8ed0 Do not check client's host when user comes from interserver secret 2020-09-16 00:35:49 +03:00
Azat Khuzhin
0159c74f21 Secure inter-cluster query execution (with initial_user as current query user) [v3] 
Add inter-server cluster secret, it is used for Distributed queries
inside cluster, you can configure in the configuration file:

  <remote_servers>
      <logs>
          <shard>
              <secret>foobar</secret> <!-- empty -- works as before -->
              ...
          </shard>
      </logs>
  </remote_servers>

And this will allow clickhouse to make sure that the query was not
faked, and was issued from the node that knows the secret. And since
trust appeared it can use initial_user for query execution, this will
apply correct *_for_user (since with inter-server secret enabled, the
query will be executed from the same user on the shards as on initator,
unlike "default" user w/o it).

v2: Change user to the initial_user for Distributed queries if secret match
v3: Add Protocol::Cluster package
v4: Drop Protocol::Cluster and use plain Protocol::Hello + user marker
v5: Do not use user from Hello for cluster-secure (superfluous)
 2020-09-15 01:36:28 +03:00
Vitaly Baranov
70b0fe88b1
Merge pull request #14604 from vitlibar/no-detach-threads-2 
Use join() instead of detach() for threads #2
 2020-09-12 02:07:46 +03:00
Vitaly Baranov
2a9ab48279 Use join() instead of detach() for the no_users_thread in StorageLiveView. 2020-09-09 22:39:27 +03:00
Alexander Tokmakov
d278d849e4 Merge branch 'master' into write_structure_of_table_functions 2020-09-09 17:12:35 +03:00
Alexander Kuzmenkov
c62bf555d2 Merge origin/master into tmp (using imerge) 2020-09-08 16:26:38 +03:00
Alexander Kuzmenkov
d0a9926e7d fixes and some docs 2020-09-08 16:19:27 +03:00
alesapin
bea15175d6
Merge pull request #14496 from Jokser/disk-s3-shutdown 
Disable S3 requests processing during context shutdown to speed up termination process
 2020-09-08 15:21:54 +03:00
Peng Jian
fd82272cfe Fix code style, and update docs for Kafka engine 2020-09-06 11:17:19 +08:00
Peng Jian
e6bfd9d586 1. Add new setting for Kafka engine, named kafka_thread_per_consumer which default value is false. 2. Create separate thread pool for Kafka engine. 2020-09-06 11:17:19 +08:00
Pavel Kovalenko
9d11d4dd44 Variable inline. 2020-09-04 21:45:22 +03:00
Pavel Kovalenko
c933f72adb Disable S3 requests processing during context shutdown to speed up termination process. 2020-09-04 17:17:27 +03:00
Alexander Tokmakov
56695727b2 Merge branch 'master' into write_structure_of_table_functions 2020-09-01 20:15:13 +03:00
Alexander Tokmakov
3a9779adb4 slightly better code 2020-09-01 17:41:49 +03:00
Alexander Kuzmenkov
c02f9e1dd1 remove some debug output to pass the tests 2020-08-29 01:48:48 +03:00
Alexander Kuzmenkov
063cc386a3 cosmetic fixes 2020-08-28 22:02:50 +03:00
alesapin
1ea00d5e7a Fix test and better messages 2020-08-28 14:12:51 +03:00
Alexander Kuzmenkov
4612794678 opentelemetry context propagation 2020-08-28 04:21:08 +03:00
Alexander Kuzmenkov
54b3047d19 fixup 2020-08-27 21:44:20 +03:00
Amos Bird
05a5a13e08
specific ReplicatedMergeTree settings 2020-08-27 22:37:41 +08:00
Amos Bird
0c1cf22c00
Allow FETCH PARTITION from other zookeepers 2020-08-27 22:19:18 +08:00
alexey-milovidov
7b53a0ef33 Revert "Added allow_merges option for volumes in multi-disk configuration (#13402)" 
This reverts commit 1e2616542a.
 2020-08-21 18:44:29 +03:00
Vladimir Chebotarev
1e2616542a
Added allow_merges option for volumes in multi-disk configuration (#13402) 2020-08-21 12:04:13 +03:00
Alexander Kuzmenkov
05ad9b9fff opentelemetry wip 2020-08-20 23:59:58 +03:00
tavplubix
4aff48dc3f
Merge pull request #10851 from zhang2014/feature/materialize_mysql_database 
ISSUES-4006 support MaterializeMySQL database engine
 2020-08-12 15:59:28 +03:00
Alexey Milovidov
12f66fa82c Fix 99% of typos 2020-08-08 04:01:47 +03:00
zhang2014
a3bc18bed2 ISSUES-4006 try resolve review comment 2020-08-06 12:03:27 +08:00
zhang2014
035760df9d ISSUES-4006 refactor for external ddl 2020-08-04 10:31:26 +08:00
Alexey Milovidov
1c039d2cf3 Attempt to fix "Arcadia" build 2020-08-03 04:44:58 +03:00
Alexey Milovidov
6e63a62a65 Attempt to fix "Arcadia" build 2020-08-03 03:14:59 +03:00
Vitaly Baranov
56665a15f7 Rework and rename the template class SettingsCollection => BaseSettings. 2020-07-31 20:54:18 +03:00
Vitaly Baranov
e40a8ac176 Use names of the settings instead of their indices in SettingsConstraints and settings profiles. 2020-07-31 19:11:27 +03:00
Gleb Novikov
aac97957ce Fixed integration test_tmp_policy 2020-07-26 21:27:36 +03:00
Gleb Novikov
7f5b6fba78 Generic volume is coming... 
1. SingleDiskVolume for temporary volumes
2. Generic VolumePtr in StoragePolicies
3. Removed max_data_part_size in system.storage_policies, added volume_type
 2020-07-26 21:27:36 +03:00
Alexander Kuzmenkov
4398c2daf8 boop the CI 2020-07-23 23:44:30 +03:00
Denis Glazachev
9effacfbc1 Merge branch 'master' into ldap-per-user-authentication 
* master: (1102 commits)
  Update README.md
  Update README.md
  Update README.md
  Update index.md
  [docs] add intrdocution for statements page (#12189)
  Revert "Run perf tests with memory sampling (for allocations >1M)"
  Sanitize LINK_LIBRARIES property for the directories (#12160)
  [docs] refactor Domains overview (#12186)
  DOCS-647: toStartOfSecond (#12190)
  [docs] add intrdocution for commercial page (#12187)
  DOCSUP-1348 Russian translation for new functions (#133) (#12194)
  changelog fixes
  Update index.md (#12191)
  Update zh kafka.md title (#12192)
  Added test for #3767
  style fix for #12152
  Tests for fixed issues #10846 and #7347
  changelog fixes
  [docs] introduction for special table engines (#12170)
  [docs] introduction for third-party interfaces (#12175)
  ...

# Conflicts:
#	src/Access/ya.make
#	src/Common/ErrorCodes.cpp
 2020-07-08 00:42:09 +04:00
Alexey Milovidov
1462a66d1e Fix typos 2020-06-27 22:05:00 +03:00
Val
cbcd1bea0e provide better comments and information 2020-06-27 17:00:44 +03:00
Val
5e076b464e add replicated db snapshot, integration test, repl alter queries, etc 
add an option to create replicated tables within replicated db without specifying zk path and replica id
add replicated sch pool
disable replication of alter queries for replicated tables in replicated dbs
snapshot prototype. amend of replicated db workflow
add prototype of integration tests for replicated db
 2020-06-27 16:58:15 +03:00
Vitaly Baranov
29a9d4187f Fix using current database while checking access rights. 2020-06-24 17:31:05 +03:00
Denis Glazachev
5db60202b6 Merge branch 'master' into ldap-per-user-authentication 2020-06-19 00:11:08 +04:00
Alexey Milovidov
1ab599b0a0 Remove "current_password" but keep it for Arcadians 2020-06-15 05:17:08 +03:00
Alexey Milovidov
bbe5f4c909 Revert "Remove "current_password" because it is harmful" 
This reverts commit 66ccb2f6b1.
 2020-06-15 05:13:41 +03:00
Alexey Milovidov
66ccb2f6b1 Remove "current_password" because it is harmful 2020-06-15 04:12:01 +03:00
Alexey Milovidov
6c278fee61 Better exception message 2020-06-14 04:07:47 +03:00
Alexander Kuzmenkov
e354bbc78c
Merge pull request #11588 from ClickHouse/aku/async-log 
Add system.asynchronous_metric_log
 2020-06-11 16:10:35 +03:00
Denis Glazachev
9e3a28a6b8 Merge branch 'master' into ldap-per-user-authentication 
* master: (414 commits)
  Update file.md
  Update merge.md
  Update dictionary.md
  Update external-data.md
  Update distributed.md
  Update null.md
  Update set.md
  Update join.md
  Update url.md
  Update view.md
  Update materializedview.md
  Update memory.md
  Update buffer.md
  Update generate.md
  removed a sentence about global lock during rename (#11577)
  greatCircleAngle en translation (#11584)
  Update configuration-files.md
  try fix flacky test
  Update why.html
  Update rich.html
  ...

# Conflicts:
#	src/Common/ErrorCodes.cpp
#	utils/ci/jobs/quick-build/run.sh
 2020-06-11 03:06:17 +04:00
Denis Glazachev
848330b37a Expect <ldap_servers> in main config.xml 2020-06-11 02:48:15 +04:00
Alexander Kuzmenkov
e4be52f35f Add system.asynchronous_metric_log 2020-06-10 22:17:54 +03:00
Alexander Tokmakov
08073903ed fix query_id of http queries 2020-06-10 17:24:42 +03:00
Alexander Kuzmenkov
fafa953ee4 fixup 2020-06-06 04:23:21 +03:00
Alexander Kuzmenkov
eaaf005a03
Merge pull request #11410 from ClickHouse/aku/logical-error-synonyms 
Remove assorted synonyms of LOGICAL_ERROR
 2020-06-05 13:10:40 +03:00
Alexander Kuzmenkov
07e4bb7050 Remove assorted synonyms of LOGICAL_ERROR. 
We don't need any special handling for them on the client, and, on the
contrary, have to handle them as logical errors in tests.
 2020-06-03 16:18:42 +03:00
Denis Glazachev
c427524bc8 Simplefy ExternalAuthenticators exposure to isCorrectPassword() 2020-06-03 01:02:31 +04:00
Pervakov Grigorii
bc9381406f Add default credentials and custom headers for s3 table functions. 2020-06-02 20:48:49 +03:00
Denis Glazachev
c61cbe8e98 Merge branch 'master' into ldap-per-user-authentication 
* master: (114 commits)
  Update PushingToViewsBlockOutputStream.cpp
  Update PushingToViewsBlockOutputStream.cpp
  make clang-10 happy
  Fix sync_async test (remove timeout)
  CLICKHOUSEDOCS-631: temporary_files_codec, join_on_disk_max_files_to_merge settings. (#11242)
  Suppress output of cancelled queries in clickhouse-client #9473
  Better log messages in ConfigReloader
  fix select from StorageJoin
  Fix unit tests under MSan
  Added test.
  Fix build.
  Fix arguments for AggregateFunctionQuantile/
  Update style.md
  Add a guide on error messages.
  Report dictionary name on dictionary load errors.
  more types in ASOF JOIN (#11301)
  Fix part_log test
  Update test.
  Add perftest.
  Parallel processing for PushingToViewsBlockOutputStream::writeSuffix
  ...
 2020-06-02 12:42:29 +04:00
Denis Glazachev
e8144976fe Merge branch 'master' into ldap-per-user-authentication 2020-05-31 15:09:16 +04:00
Alexey Milovidov
25f941020b Remove namespace pollution 2020-05-31 00:57:37 +03:00
Alexander Tokmakov
c071e0a473 add context to DatabaseCatalog methods 2020-05-29 02:01:18 +03:00
Denis Glazachev
2863de750e Merge branch 'master' into ldap-per-user-authentication 2020-05-28 01:30:52 +04:00
Denis Glazachev
d74f1357d4 Add LDAP authentication support 2020-05-28 01:06:33 +04:00