Nikita Mikhaylov
3954eff276
do not use cluster from config on remote replicas
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
b05718feac
style
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
c9ae4cb467
multiplem pipes on remote
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
024374a2ec
review fixes
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
f36a715c32
delete unused code
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
a15757a9c9
better renaming
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
09a62e713a
rename to s3Cluster
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
a743442a17
build fixes
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
507cb8514a
review fixes
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
75230e3701
better
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
7a68820342
style
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
7b95ff579d
fix S3 test
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
4465a0627f
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
704fb04941
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
c333c3dedb
review fixes
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
efef179b89
remove prints
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
36a8419f60
style
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
8a4b5a586e
fixed uuid
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
7276b40556
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
2555ae5d3f
better processing stage
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
587fbdd10d
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
fea3aaafe6
simplify storages3dist
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
4843f86329
use only one connection between initiator and worker
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
ebf1e2fa9c
fix build
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
3ab17233cf
cleanup
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
b3094412b1
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
c1d1313dd8
remote prints
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
851644d8bf
fix test
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
e601a432e5
better[2]
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
cef9e19eb2
better
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
a5d55781b1
remote debug prints
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
9b7a6e66d0
add ifded
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
4e4b383214
added hash of itiator address
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
2549468c14
better
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
0be3fa178b
save
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
64b4cd0e63
save
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
31b4f9b17f
save
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
44ca65a9a4
save
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
081ea84a41
save
2021-04-13 22:39:41 +03:00
Alexey Milovidov
df44476307
Fix error
2021-04-13 22:32:02 +03:00
Alexey Milovidov
f7da292d2a
Merge branch 'master' into array-has-any-all-generic
2021-04-13 22:16:33 +03:00
Alexander Tokmakov
05e4c8d3ef
fix attach mv in atomic db
2021-04-13 22:13:26 +03:00
Alexey Milovidov
97611faad0
Whitespace
2021-04-13 22:06:24 +03:00
Alexey Milovidov
03662165f3
Comment
2021-04-13 22:03:52 +03:00
Alexey Milovidov
8bd5578c92
Less amount of template instantiations
2021-04-13 22:02:07 +03:00
Alexey Milovidov
42412f9a08
extern template
2021-04-13 21:59:55 +03:00
Alexey Milovidov
fb98915435
More safe CPU dispatching
2021-04-13 21:54:46 +03:00
Alexey Milovidov
b00c66cb36
More safe CPU dispatching
2021-04-13 21:53:55 +03:00
Azat Khuzhin
d59bdfd45d
Fix one more lock-order-inversion
...
TSan report [1]:
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=11314)
Cycle in lock order graph: M183938897938677368 (0x000000000000) => M2505 (0x7b9000002008) => M183938897938677368
Mutex M2505 acquired here while holding mutex M183938897938677368 in thread T6:
0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a327b6)
1 std::__1::__libcpp_recursive_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:385:10 (clickhouse-tsan+0x17cdb689)
2 std::__1::recursive_mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:71:14 (clickhouse-tsan+0x17cdb689)
3 std::__1::unique_lock<std::__1::recursive_mutex>::unique_lock(std::__1::recursive_mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:119:61 (clickhouse-tsan+0x11e3506f)
4 DB::Context::getLock() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:517:12 (clickhouse-tsan+0x11e3506f)
5 DB::Context::getSchedulePool() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1517:17 (clickhouse-tsan+0x11e3506f)
6 DB::IBackgroundJobExecutor::start() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:158:42 (clickhouse-tsan+0x12be1cda)
7 DB::StorageMergeTree::startup() obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:112:29 (clickhouse-tsan+0x129ed46e)
8 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_2::operator()(std::__1::shared_ptr<DB::IStorage> const&) const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:230:16 (clickhouse-tsan+0x11d71fba)
9 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3::operator()() const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:238:56 (clickhouse-tsan+0x11d71fba)
10 decltype(std::__1::forward<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(fp)()) std::__1::__invoke<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x11d71fba)
11 void std::__1::__invoke_void_return_wrapper<void>::__call<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x11d71fba)
12 std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x11d71fba)
13 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x11d71fba)
14 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b3b8e0)
15 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b3b8e0)
16 ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b3b8e0)
17 void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3e600)
18 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(fp0)...)) std::__1::__invoke_constexpr<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3682:1 (clickhouse-tsan+0x8b3e600)
19 decltype(auto) std::__1::__apply_tuple_impl<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::__tuple_indices<std::__1::tuple<>&...>) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1415:1 (clickhouse-tsan+0x8b3e600)
20 decltype(auto) std::__1::apply<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1424:1 (clickhouse-tsan+0x8b3e600)
21 ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()::operator()() obj-x86_64-linux-gnu/../src/Common/ThreadPool.h:178:13 (clickhouse-tsan+0x8b3e600)
22 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3e561)
23 void std::__1::__invoke_void_return_wrapper<void>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x8b3e561)
24 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x8b3e561)
25 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x8b3e561)
26 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b39205)
27 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b39205)
28 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b39205)
29 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3cea8)
30 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3cea8)
31 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:280:5 (clickhouse-tsan+0x8b3cea8)
32 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:291:5 (clickhouse-tsan+0x8b3cea8)
Mutex M183938897938677368 previously acquired by the same thread here:
0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a327b6)
1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse-tsan+0x17cdb4f9)
2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse-tsan+0x17cdb4f9)
3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse-tsan+0x12be1ca9)
4 DB::IBackgroundJobExecutor::start() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:155:21 (clickhouse-tsan+0x12be1ca9)
5 DB::StorageMergeTree::startup() obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:112:29 (clickhouse-tsan+0x129ed46e)
6 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_2::operator()(std::__1::shared_ptr<DB::IStorage> const&) const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:230:16 (clickhouse-tsan+0x11d71fba)
7 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3::operator()() const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:238:56 (clickhouse-tsan+0x11d71fba)
8 decltype(std::__1::forward<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(fp)()) std::__1::__invoke<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x11d71fba)
9 void std::__1::__invoke_void_return_wrapper<void>::__call<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x11d71fba)
10 std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x11d71fba)
11 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x11d71fba)
12 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b3b8e0)
13 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b3b8e0)
14 ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b3b8e0)
15 void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3e600)
16 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(fp0)...)) std::__1::__invoke_constexpr<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3682:1 (clickhouse-tsan+0x8b3e600)
17 decltype(auto) std::__1::__apply_tuple_impl<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::__tuple_indices<std::__1::tuple<>&...>) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1415:1 (clickhouse-tsan+0x8b3e600)
18 decltype(auto) std::__1::apply<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1424:1 (clickhouse-tsan+0x8b3e600)
19 ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()::operator()() obj-x86_64-linux-gnu/../src/Common/ThreadPool.h:178:13 (clickhouse-tsan+0x8b3e600)
20 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3e561)
21 void std::__1::__invoke_void_return_wrapper<void>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x8b3e561)
22 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x8b3e561)
23 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x8b3e561)
24 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b39205)
25 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b39205)
26 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b39205)
27 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3cea8)
28 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3cea8)
29 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:280:5 (clickhouse-tsan+0x8b3cea8)
30 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:291:5 (clickhouse-tsan+0x8b3cea8)
Mutex M183938897938677368 acquired here while holding mutex M2505 in main thread:
0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a327b6)
1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse-tsan+0x17cdb4f9)
2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse-tsan+0x17cdb4f9)
3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse-tsan+0x12be261b)
4 DB::IBackgroundJobExecutor::finish() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:167:21 (clickhouse-tsan+0x12be261b)
5 DB::IBackgroundJobExecutor::~IBackgroundJobExecutor() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:185:5 (clickhouse-tsan+0x12be261b)
6 DB::StorageMergeTree::~StorageMergeTree() obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:174:1 (clickhouse-tsan+0x129ed768)
7 std::__1::default_delete<DB::StorageMergeTree>::operator()(DB::StorageMergeTree*) const obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:1397:5 (clickhouse-tsan+0x12e48b0b)
8 std::__1::__shared_ptr_pointer<DB::StorageMergeTree*, std::__1::shared_ptr<DB::StorageMergeTree>::__shared_ptr_default_delete<DB::StorageMergeTree, DB::StorageMergeTree>, std::__1::allocator<DB::StorageMergeTree> >::__on_zero_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2565:5 (clickhouse-tsan+0x12e48b0b)
9 std::__1::__shared_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2475:9 (clickhouse-tsan+0x125b53ea)
10 std::__1::__shared_weak_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2517:27 (clickhouse-tsan+0x125b53ea)
11 std::__1::shared_ptr<DB::IStorage>::~shared_ptr() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3212:19 (clickhouse-tsan+0x125b53ea)
12 DB::SystemLog<DB::AsynchronousMetricLogElement>::~SystemLog() obj-x86_64-linux-gnu/../src/Interpreters/SystemLog.h:118:7 (clickhouse-tsan+0x125b53ea)
13 std::__1::allocator<DB::AsynchronousMetricLog>::destroy(DB::AsynchronousMetricLog*) obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:891:15 (clickhouse-tsan+0x125b1dd8)
14 void std::__1::allocator_traits<std::__1::allocator<DB::AsynchronousMetricLog> >::__destroy<DB::AsynchronousMetricLog>(std::__1::integral_constant<bool, true>, std::__1::allocator<DB::AsynchronousMetricLog>&, DB::AsynchronousMetricLog*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:539:21 (clickhouse-tsan+0x125b1dd8)
15 void std::__1::allocator_traits<std::__1::allocator<DB::AsynchronousMetricLog> >::destroy<DB::AsynchronousMetricLog>(std::__1::allocator<DB::AsynchronousMetricLog>&, DB::AsynchronousMetricLog*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:487:14 (clickhouse-tsan+0x125b1dd8)
16 std::__1::__shared_ptr_emplace<DB::AsynchronousMetricLog, std::__1::allocator<DB::AsynchronousMetricLog> >::__on_zero_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2611:9 (clickhouse-tsan+0x125b1dd8)
17 std::__1::__shared_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2475:9 (clickhouse-tsan+0x125904ff)
18 std::__1::__shared_weak_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2517:27 (clickhouse-tsan+0x125904ff)
19 std::__1::shared_ptr<DB::AsynchronousMetricLog>::~shared_ptr() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3212:19 (clickhouse-tsan+0x125904ff)
20 DB::SystemLogs::~SystemLogs() obj-x86_64-linux-gnu/../src/Interpreters/SystemLog.cpp:155:1 (clickhouse-tsan+0x125904ff)
21 std::__1::__optional_destruct_base<DB::SystemLogs, false>::reset() obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:245:21 (clickhouse-tsan+0x11e43655)
22 DB::ContextShared::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:443:21 (clickhouse-tsan+0x11e43655)
23 DB::Context::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2251:13 (clickhouse-tsan+0x11e3be37)
24 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5::operator()() const obj-x86_64-linux-gnu/../programs/server/Server.cpp:892:5 (clickhouse-tsan+0x8abacc2)
25 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::invoke() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:97:9 (clickhouse-tsan+0x8abacc2)
26 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::~basic_scope_guard() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:47:28 (clickhouse-tsan+0x8abacc2)
27 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:1395:1 (clickhouse-tsan+0x8ab5cba)
28 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b4ac7b)
29 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa97be)
30 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b67093)
31 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8aa838e)
32 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8aa68f9)
Mutex M2505 previously acquired by the same thread here:
0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a327b6)
1 std::__1::__libcpp_recursive_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:385:10 (clickhouse-tsan+0x17cdb689)
2 std::__1::recursive_mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:71:14 (clickhouse-tsan+0x17cdb689)
3 std::__1::lock_guard<std::__1::recursive_mutex>::lock_guard(std::__1::recursive_mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse-tsan+0x11e4363f)
4 DB::ContextShared::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:438:21 (clickhouse-tsan+0x11e4363f)
5 DB::Context::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2251:13 (clickhouse-tsan+0x11e3be37)
6 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5::operator()() const obj-x86_64-linux-gnu/../programs/server/Server.cpp:892:5 (clickhouse-tsan+0x8abacc2)
7 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::invoke() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:97:9 (clickhouse-tsan+0x8abacc2)
8 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::~basic_scope_guard() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:47:28 (clickhouse-tsan+0x8abacc2)
9 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:1395:1 (clickhouse-tsan+0x8ab5cba)
10 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b4ac7b)
11 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa97be)
12 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b67093)
13 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8aa838e)
14 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8aa68f9)
[1]: https://clickhouse-test-reports.s3.yandex.net/21318/f3b1ad0f5d1024275674e1beac24251ae97c8453/functional_stateful_tests_(thread).html#fail1
v2: Convert ContextSharedPart::system_logs to std::unique_ptr (to avoid copy ctor)
v3: Fix readability-identifier-naming,-warnings-as-errors for system_logs_
v4: fix conflicts
2021-04-13 21:37:38 +03:00
Azat Khuzhin
a61ae26729
Fix lock-order-inversion during system.*_log shutting down
...
As TSan reports [1]:
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=24429)
Cycle in lock order graph: M152695175523663992 (0x000000000000) => M2505 (0x7b9000002008) => M152695175523663992
Mutex M2505 acquired here while holding mutex M152695175523663992 in thread T7:
0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a301b6)
1 std::__1::__libcpp_recursive_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:385:10 (clickhouse-tsan+0x17cd6e89)
2 std::__1::recursive_mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:71:14 (clickhouse-tsan+0x17cd6e89)
3 std::__1::unique_lock<std::__1::recursive_mutex>::unique_lock(std::__1::recursive_mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:119:61 (clickhouse-tsan+0x11e32a9f)
4 DB::Context::getLock() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:517:12 (clickhouse-tsan+0x11e32a9f)
5 DB::Context::getSchedulePool() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1517:17 (clickhouse-tsan+0x11e32a9f)
6 DB::IBackgroundJobExecutor::start() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:158:42 (clickhouse-tsan+0x12bde50a)
7 DB::StorageMergeTree::startup() obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:112:29 (clickhouse-tsan+0x129e9e1e)
8 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_2::operator()(std::__1::shared_ptr<DB::IStorage> const&) const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:230:16 (clickhouse-tsan+0x11d6fa2a)
9 DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3::operator()() const obj-x86_64-linux-gnu/../src/Databases/DatabaseOrdinary.cpp:238:56 (clickhouse-tsan+0x11d6fa2a)
10 decltype(std::__1::forward<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(fp)()) std::__1::__invoke<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x11d6fa2a)
11 void std::__1::__invoke_void_return_wrapper<void>::__call<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&>(DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x11d6fa2a)
12 std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x11d6fa2a)
13 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::DatabaseOrdinary::startupTables(ThreadPoolImpl<ThreadFromGlobalPool>&)::$_3, void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x11d6fa2a)
14 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b39350)
15 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b39350)
16 ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b39350)
17 void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3c070)
18 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(fp0)...)) std::__1::__invoke_constexpr<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3682:1 (clickhouse-tsan+0x8b3c070)
19 decltype(auto) std::__1::__apply_tuple_impl<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::__tuple_indices<std::__1::tuple<>&...>) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1415:1 (clickhouse-tsan+0x8b3c070)
20 decltype(auto) std::__1::apply<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&, std::__1::tuple<>&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&) obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1424:1 (clickhouse-tsan+0x8b3c070)
21 ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()::operator()() obj-x86_64-linux-gnu/../src/Common/ThreadPool.h:178:13 (clickhouse-tsan+0x8b3c070)
22 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3bfd1)
23 void std::__1::__invoke_void_return_wrapper<void>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()&>(void&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse-tsan+0x8b3bfd1)
24 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()>::operator()() obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse-tsan+0x8b3bfd1)
25 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse-tsan+0x8b3bfd1)
26 std::__1::__function::__policy_func<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse-tsan+0x8b36c75)
27 std::__1::function<void ()>::operator()() const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse-tsan+0x8b36c75)
28 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:247:17 (clickhouse-tsan+0x8b36c75)
29 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:73 (clickhouse-tsan+0x8b3a918)
30 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse-tsan+0x8b3a918)
31 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:280:5 (clickhouse-tsan+0x8b3a918)
32 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:291:5 (clickhouse-tsan+0x8b3a918)
Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
Mutex M152695175523663992 acquired here while holding mutex M2505 in main thread:
0 pthread_mutex_lock <null> (clickhouse-tsan+0x8a301b6)
1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse-tsan+0x17cd6cf9)
2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse-tsan+0x17cd6cf9)
3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse-tsan+0x12bdee4b)
4 DB::IBackgroundJobExecutor::finish() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:167:21 (clickhouse-tsan+0x12bdee4b)
5 DB::IBackgroundJobExecutor::~IBackgroundJobExecutor() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:185:5 (clickhouse-tsan+0x12bdee4b)
6 DB::StorageMergeTree::~StorageMergeTree() obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:174:1 (clickhouse-tsan+0x129ea118)
7 std::__1::default_delete<DB::StorageMergeTree>::operator()(DB::StorageMergeTree*) const obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:1397:5 (clickhouse-tsan+0x12e4433b)
8 std::__1::__shared_ptr_pointer<DB::StorageMergeTree*, std::__1::shared_ptr<DB::StorageMergeTree>::__shared_ptr_default_delete<DB::StorageMergeTree, DB::StorageMergeTree>, std::__1::allocator<DB::StorageMergeTree> >::__on_zero_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2565:5 (clickhouse-tsan+0x12e4433b)
9 std::__1::__shared_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2475:9 (clickhouse-tsan+0x125b355a)
10 std::__1::__shared_weak_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2517:27 (clickhouse-tsan+0x125b355a)
11 std::__1::shared_ptr<DB::IStorage>::~shared_ptr() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3212:19 (clickhouse-tsan+0x125b355a)
12 DB::SystemLog<DB::AsynchronousMetricLogElement>::~SystemLog() obj-x86_64-linux-gnu/../src/Interpreters/SystemLog.h:118:7 (clickhouse-tsan+0x125b355a)
13 std::__1::allocator<DB::AsynchronousMetricLog>::destroy(DB::AsynchronousMetricLog*) obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:891:15 (clickhouse-tsan+0x125aff68)
14 void std::__1::allocator_traits<std::__1::allocator<DB::AsynchronousMetricLog> >::__destroy<DB::AsynchronousMetricLog>(std::__1::integral_constant<bool, true>, std::__1::allocator<DB::AsynchronousMetricLog>&, DB::AsynchronousMetricLog*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:539:21 (clickhouse-tsan+0x125aff68)
15 void std::__1::allocator_traits<std::__1::allocator<DB::AsynchronousMetricLog> >::destroy<DB::AsynchronousMetricLog>(std::__1::allocator<DB::AsynchronousMetricLog>&, DB::AsynchronousMetricLog*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:487:14 (clickhouse-tsan+0x125aff68)
16 std::__1::__shared_ptr_emplace<DB::AsynchronousMetricLog, std::__1::allocator<DB::AsynchronousMetricLog> >::__on_zero_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2611:9 (clickhouse-tsan+0x125aff68)
17 std::__1::__shared_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2475:9 (clickhouse-tsan+0x1258e74f)
18 std::__1::__shared_weak_count::__release_shared() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2517:27 (clickhouse-tsan+0x1258e74f)
19 std::__1::shared_ptr<DB::AsynchronousMetricLog>::~shared_ptr() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3212:19 (clickhouse-tsan+0x1258e74f)
20 DB::SystemLogs::~SystemLogs() obj-x86_64-linux-gnu/../src/Interpreters/SystemLog.cpp:155:1 (clickhouse-tsan+0x1258e74f)
21 std::__1::__optional_destruct_base<DB::SystemLogs, false>::reset() obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:245:21 (clickhouse-tsan+0x11e41085)
22 DB::ContextShared::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:443:21 (clickhouse-tsan+0x11e41085)
23 DB::Context::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2251:13 (clickhouse-tsan+0x11e39867)
24 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5::operator()() const obj-x86_64-linux-gnu/../programs/server/Server.cpp:892:5 (clickhouse-tsan+0x8ab8732)
25 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::invoke() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:97:9 (clickhouse-tsan+0x8ab8732)
26 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::~basic_scope_guard() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:47:28 (clickhouse-tsan+0x8ab8732)
27 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:1395:1 (clickhouse-tsan+0x8ab3839)
28 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b464ab)
29 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa71be)
30 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b628c3)
31 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8aa5d8e)
32 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8aa42f9)
[1]: https://clickhouse-test-reports.s3.yandex.net/21318/38be9ff43ac4c46ce6e803fc125d910bde1d4c71/functional_stateful_tests_(thread).html#fail1
2021-04-13 21:37:38 +03:00
Azat Khuzhin
3056d2c5d1
Fix data-race during server shutdown the context
...
Found with 01737_clickhouse_server_wait_server_pool:
==================
WARNING: ThreadSanitizer: data race (pid=13248)
Write of size 1 at 0x7b9000003a38 by main thread:
0 std::__1::__optional_destruct_base<DB::SystemLogs, false>::reset() obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:246:24 (clickhouse-tsan+0x11e3043e)
1 DB::ContextShared::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:441:21 (clickhouse-tsan+0x11e3043e)
2 DB::Context::shutdown() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2249:13 (clickhouse-tsan+0x11e28c17)
3 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5::operator()() const obj-x86_64-linux-gnu/../programs/server/Server.cpp:892:5 (clickhouse-tsan+0x8ab1a32)
4 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::invoke() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:97:9 (clickhouse-tsan+0x8ab1a32)
5 ext::basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_5>::~basic_scope_guard() obj-x86_64-linux-gnu/../base/common/../ext/scope_guard.h:47:28 (clickhouse-tsan+0x8ab1a32)
6 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:1395:1 (clickhouse-tsan+0x8aacb39)
7 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b3446b)
8 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa04be)
9 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b50883)
10 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8a9f08e)
11 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8a9d5f9)
Previous read of size 1 at 0x7b9000003a38 by thread T2 (mutexes: write M2504):
0 std::__1::__optional_storage_base<DB::SystemLogs, false>::has_value() const obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:295:22 (clickhouse-tsan+0x11e25348)
1 std::__1::optional<DB::SystemLogs>::operator bool() const obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:938:64 (clickhouse-tsan+0x11e25348)
2 DB::Context::getQueryLog() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1875:10 (clickhouse-tsan+0x11e25348)
3 DB::executeQueryImpl(char const*, char const*, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:657:50 (clickhouse-tsan+0x12653751)
4 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:904:30 (clickhouse-tsan+0x12651308)
5 DB::TCPHandler::runImpl() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:289:24 (clickhouse-tsan+0x12f04b45)
6 DB::TCPHandler::run() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1500:9 (clickhouse-tsan+0x12f13907)
7 Poco::Net::TCPServerConnection::start() obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse-tsan+0x15b1f722)
8 Poco::Net::TCPServerDispatcher::run() obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:113:19 (clickhouse-tsan+0x15b1fe4e)
9 Poco::PooledThread::run() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse-tsan+0x15c86fe1)
10 Poco::(anonymous namespace)::RunnableHolder::run() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse-tsan+0x15c8557f)
11 Poco::ThreadImpl::runnableEntry(void*) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse-tsan+0x15c83d87)
Location is heap block of size 7296 at 0x7b9000002000 allocated by main thread:
0 operator new(unsigned long) <null> (clickhouse-tsan+0x8a9aae7)
1 std::__1::__unique_if<DB::ContextShared>::__unique_single std::__1::make_unique<DB::ContextShared>() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2068:28 (clickhouse-tsan+0x11e15c2c)
2 DB::Context::createShared() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:503:32 (clickhouse-tsan+0x11e15c2c)
3 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:426:27 (clickhouse-tsan+0x8aa19ee)
4 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b3446b)
5 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa04be)
6 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b50883)
7 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8a9f08e)
8 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8a9d5f9)
Mutex M2504 (0x7b9000002008) created at:
0 pthread_mutex_init <null> (clickhouse-tsan+0x8a0d37d)
1 std::__1::__libcpp_recursive_mutex_init(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:370:10 (clickhouse-tsan+0x17cc4d93)
2 std::__1::recursive_mutex::recursive_mutex() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:56:14 (clickhouse-tsan+0x17cc4d93)
3 DB::ContextShared::ContextShared() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:394:5 (clickhouse-tsan+0x11e40bc3)
4 std::__1::__unique_if<DB::ContextShared>::__unique_single std::__1::make_unique<DB::ContextShared>() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2068:32 (clickhouse-tsan+0x11e15c37)
5 DB::Context::createShared() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:503:32 (clickhouse-tsan+0x11e15c37)
6 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:426:27 (clickhouse-tsan+0x8aa19ee)
7 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b3446b)
8 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa04be)
9 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b50883)
10 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8a9f08e)
11 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8a9d5f9)
Thread T2 'TCPHandler' (tid=13643, running) created by main thread at:
0 pthread_create <null> (clickhouse-tsan+0x8a0bf0b)
1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable> >) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:202:6 (clickhouse-tsan+0x15c83827)
2 Poco::Thread::start(Poco::Runnable&) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:128:2 (clickhouse-tsan+0x15c84f6c)
3 Poco::PooledThread::start() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:85:10 (clickhouse-tsan+0x15c873e2)
4 Poco::ThreadPool::ThreadPool(int, int, int, int) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:252:12 (clickhouse-tsan+0x15c873e2)
5 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:843:22 (clickhouse-tsan+0x8aa8e5f)
6 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse-tsan+0x15b3446b)
7 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:342:25 (clickhouse-tsan+0x8aa04be)
8 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse-tsan+0x15b50883)
9 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:134:20 (clickhouse-tsan+0x8a9f08e)
10 main obj-x86_64-linux-gnu/../programs/main.cpp:368:12 (clickhouse-tsan+0x8a9d5f9)
SUMMARY: ThreadSanitizer: data race obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:246:24 in std::__1::__optional_destruct_base<DB::SystemLogs, false>::reset()
v2: fix deadlock by calling SystemLogs::shutdown w/o Context lock
2021-04-13 21:37:38 +03:00
Alexey Milovidov
cdbc9d9fa0
Merge branch 'master' into speedup-intdiv
2021-04-13 20:43:54 +03:00
Alexey Milovidov
c01756014e
More generic implementation of has
2021-04-13 20:08:49 +03:00
Kruglov Pavel
6350f734dc
Merge branch 'master' into datasketches-uniq
2021-04-13 19:34:15 +03:00
alesapin
6d8efe5d5b
Merge pull request #22992 from ClickHouse/ssl_keeper
...
Add SSL to keeper (both client and internal)
2021-04-13 19:15:07 +03:00
tavplubix
1525e38a3c
Merge pull request #22990 from ClickHouse/tavplubix-patch-1
...
Fix excessive warning in StorageDistributed with cross-replication
2021-04-13 18:58:12 +03:00
tavplubix
da681d9907
Merge pull request #22998 from ClickHouse/fix_22989
...
Fix assertion when filtering tables in StorageMerge
2021-04-13 18:57:24 +03:00
alexey-milovidov
46e1da03fb
Update KeyCondition.cpp
2021-04-13 18:47:11 +03:00
pingyu
8244e9fc94
address comments
2021-04-13 23:09:05 +08:00
Alexey Milovidov
83a78a5aa7
Fix unfinished code in GatherUtils #20272
2021-04-13 16:56:39 +03:00
Alexander Tokmakov
fdf3cf378f
fix testkeeper multi response
2021-04-13 15:49:40 +03:00
alesapin
60792843ea
Less garbage
2021-04-13 15:04:18 +03:00
alesapin
b115b8af3d
Something working
2021-04-13 14:55:08 +03:00
songenjie
7c32cc1e18
fix case style for local variable
2021-04-13 17:34:04 +08:00
hexiaoting
4f36ac7f64
Fix error
2021-04-13 17:31:49 +08:00
alexey-milovidov
1945d01719
Merge pull request #21972 from kssenii/nanodbc
...
Switch to nanodbc
2021-04-13 12:23:57 +03:00
Maksim Kita
305d3d6b16
Merge pull request #22978 from ClickHouse/filter-by-virtual-columns-fix
...
Fix trivial mistake in filtering by virtual columns
2021-04-13 11:42:16 +03:00
Maksim Kita
e687fc320f
Merge pull request #22997 from ClickHouse/increase-max-uri-size
...
Increase `max_uri_size` to 1 MiB by default
2021-04-13 11:26:23 +03:00
alesapin
cbd8a57fda
Merge branch 'master' into ssl_keeper
2021-04-13 10:34:23 +03:00
Nikolai Kochetov
e731dfe650
Merge pull request #22991 from ClickHouse/untuple-and-subquery
...
Fix subquery with untuple.
2021-04-13 09:53:13 +03:00
songenjie
564136ec46
[clickhouse][server][dll][alter]support fetch part
2021-04-13 12:40:33 +08:00
hexiaoting
15f3a441e3
Fix error
2021-04-13 11:41:43 +08:00
hexiaoting
77c460e8d1
Merge remote-tracking branch 'origin/master' into dev-sumcount
2021-04-13 11:14:14 +08:00
hexiaoting
52bdec396b
Merge remote-tracking branch 'origin/master' into dev_nested_upadte
2021-04-13 09:56:47 +08:00
alexey-milovidov
ee9aaab4bc
Merge pull request #22937 from Enmk/formatDateTime_toDateTime64_fixes
...
Fixes: formatDateTime and toDateTime64
2021-04-13 01:40:25 +03:00
alexey-milovidov
9e88f5fbd3
Update ReadHelpers.h
2021-04-13 01:39:15 +03:00
alexey-milovidov
a780e8e7e1
Update ReadHelpers.h
2021-04-13 01:38:59 +03:00
alexey-milovidov
e2edecc8e2
Merge pull request #22434 from Avogar/catboost-reload
...
Update paths to the catboost model configs in config reloading
2021-04-13 01:21:04 +03:00
alexey-milovidov
d239c66fd3
Merge pull request #21511 from azat/optimize_skip_unused_shards-improvement
...
Rewrite IN in query for remote shards to exclude values that do not belong to the shard
2021-04-13 01:18:11 +03:00
Alexey Milovidov
1b8a054880
Fix error
2021-04-13 00:11:30 +03:00
Alexey Milovidov
d5580a8e71
Dynamic dispatch for intDiv
2021-04-12 23:02:42 +03:00
Alexey Milovidov
ff0d3860d4
Fix typos + minor improvement #22314
2021-04-12 23:02:16 +03:00
alesapin
56f0e290de
Merge pull request #22945 from ClickHouse/increase-fetches-pool-size
...
More conservative value of `background_fetches_pool_size` setting
2021-04-12 22:41:04 +03:00
Azat Khuzhin
a497d4d462
Do not perform optimize_skip_unused_shards for cluster with one node
2021-04-12 22:18:31 +03:00
Alexander Tokmakov
ad414bab05
fix assertion when filtering tables in StorageMerge
2021-04-12 22:04:26 +03:00
Alexey Milovidov
e260b2a60a
Increase max_uri_size to 1 MiB by default
2021-04-12 21:06:47 +03:00
Alexey Milovidov
fafe8c00fe
Fix warning #22987
2021-04-12 20:57:03 +03:00
Denny Crane
eaa092cae2
Update MergeTreeData.cpp
...
Better error message.
2021-04-12 10:58:06 -03:00
alesapin
bc8ebb1028
Simplify build
2021-04-12 16:11:43 +03:00
alesapin
338ff1615c
Simplier config
2021-04-12 15:40:01 +03:00
alesapin
ef34c95c7e
Add SSL to keeper (both client and internal)
2021-04-12 15:25:52 +03:00
Nikolai Kochetov
f6d86d6032
Merge branch 'master' into add-read-from-mt-step
2021-04-12 15:23:32 +03:00
Nikolai Kochetov
0448e1415f
Fix subquery with untuple.
2021-04-12 15:15:55 +03:00
tavplubix
a995962e6a
Update StorageDistributed.cpp
2021-04-12 14:58:24 +03:00
Vladimir
4f34bc4e73
Merge pull request #22973 from kreuzerkrieg/Fix_Poco_HTTP_Client
...
Fix an error handling in Poco HTTP Client for AWS.
2021-04-12 13:40:44 +03:00
Dmitry Krylov
df60724847
Fix constness
2021-04-12 20:29:58 +10:00
Dmitry Krylov
a3d57bd5af
Check number of arguments
2021-04-12 20:29:35 +10:00
Nikolai Kochetov
7019a9a659
Merge pull request #22673 from nvartolomei/nv/exp-arrow-stream
...
Experiment with ArrowStream streaming
2021-04-12 13:29:25 +03:00
Pavel Kruglov
baccd09efe
Fix synchronization
2021-04-12 13:05:46 +03:00
Pavel Kruglov
e4f3678ae9
Fix build
2021-04-12 12:28:46 +03:00
Nikolai Kochetov
28d6e3f56c
Merge pull request #22977 from azat/combinator-fix
...
Fix combinators with common prefix name (State and SimpleState) with libstdc++
2021-04-12 12:27:01 +03:00
Maksim Kita
b9fe85d537
Merge pull request #22964 from ClickHouse/uniq-up-to-ubsan
...
Check out of range values in FieldVisitorConverToNumber
2021-04-12 11:50:19 +03:00
alesapin
fea0ec5046
Some code
2021-04-12 11:10:23 +03:00
Maksim Kita
3078fd7d79
Merge pull request #22983 from borodark/master
...
Error message reads better
2021-04-12 10:56:10 +03:00
Azat Khuzhin
260a77118f
Apply idle_connnection_timeout/poll_interval after each query
...
Since right now it is impossible to change those settings, only via
user profile.
2021-04-12 10:39:28 +03:00
Azat Khuzhin
79bd8d4d3f
Respect optimize_skip_unused_shards_rewrite_in with optimize_skip_unused_shards_limit
2021-04-12 10:37:28 +03:00
Azat Khuzhin
e439914d38
Fix optimized cluster logic for optimize_skip_unused_shards
2021-04-12 10:37:28 +03:00
Azat Khuzhin
fbb386dca5
Rewrite IN in query for remote shards to exclude values that does not belongs to shard
...
v2: fix optimize_skip_unused_shards_rewrite_in for sharding_key wrapped into function
v3: fix column name for optimize_skip_unused_shards_rewrite_in
v4: fix optimize_skip_unused_shards_rewrite_in with Null
v5:
- squash with Remove query argument for IStreamFactory::createForShard()
- use proper column after function execution (using sharding_key_column_name)
- update the test reference since (X) now is tuple(X)
2021-04-12 10:37:28 +03:00
Azat Khuzhin
773212529b
Make Cluster::SlotToShards public
2021-04-12 10:32:04 +03:00
Dmitry Krylov
68bd27f3c5
Fix code style
2021-04-12 12:12:40 +10:00
Dmitry Krylov
35472bcc2f
Update to new IFunction::create()
2021-04-12 11:37:44 +10:00
Egor O'Sten
c8b572ed59
Readability of error message for ORDER BY
2021-04-11 20:42:05 -04:00
Egor O'Sten
c77dd36ca9
Readability of error message for ORDER BY
2021-04-11 20:40:00 -04:00
Dmitry Krylov
582f6d3a72
Remove unneeded branches of code
2021-04-12 10:23:31 +10:00
Dmitry Krylov
c2f326e2ae
Remove unneeded branches of code
2021-04-12 10:23:31 +10:00
Dmitry Krylov
ce1b9b20d5
Remove unneeded branches of code
2021-04-12 10:23:31 +10:00
Dmitry Krylov
15ea9b3f22
Move arrayFold functionality to arrayFold.cpp
2021-04-12 10:23:31 +10:00
Dmitry Krylov
517ef8ebf4
Bug with string functions in arrayFold fixed
2021-04-12 10:23:31 +10:00
Dmitry Krylov
e3502dabc7
Remove debug messages
2021-04-12 10:23:31 +10:00
Dmitry Krylov
1861817bfc
Remove debug messages
2021-04-12 10:23:31 +10:00
Dmitry Krylov
9446df026b
Cleaning code
2021-04-12 10:23:31 +10:00
Dmitry Krylov
b19a06ba48
Fix to work with empty arrays
2021-04-12 10:23:31 +10:00
Dmitry Krylov
fbd43aeea8
Draft: taking into accout input arrays variable length
2021-04-12 10:23:31 +10:00
Dmitry Krylov
2e8a296cc9
Draft: very simple variant
2021-04-12 10:23:31 +10:00
Dmitry Krylov
7c47832405
Experiment: attempt to shrink arrays
2021-04-12 10:23:31 +10:00
kssenii
df84e4e93a
Fix
2021-04-11 23:39:18 +00:00
kssenii
7a287e6fe9
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into nanodbc
2021-04-11 21:36:08 +00:00
Kseniia Sumarokova
71484fcfbd
Merge branch 'master' into postgresql-protocol-with-row-policy
2021-04-12 00:12:49 +03:00
kssenii
792d57d630
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into postgresql-protocol-with-row-policy
2021-04-11 21:10:33 +00:00
Alexey Milovidov
22720dd7a4
Fix trivial mistake in filtering by virtual columns
2021-04-11 22:39:22 +03:00
Azat Khuzhin
0bf6b61b59
Fix combinators with common prefix name (State and SimpleState) with libstdc++
...
Previously sort order of the std::unordered_map in libstdc++ was
different and any *SimpleState() reports an error that function does not
exists.
Fix this by using proper order in container, and use std::vector over
std::unordered_map, since there linear traversing anyway in the single
method -- tryFindSuffix()
Note that test is not required, since it either fail with unknown
function or not.
2021-04-11 22:10:02 +03:00
Maksim Kita
b571656fd7
Merge pull request #22931 from Enmk/dateDiff_DateTime64_fix
...
Fixed dateDiff for DateTime64
2021-04-11 21:36:24 +03:00
Maksim Kita
aa6183af86
Merge pull request #22971 from azat/max_part_removal_threads
...
Respect max_part_removal_threads for ReplicatedMergeTree
2021-04-11 21:14:47 +03:00
Vasily Nemkov
ffd808f877
Removed redundant access specifier
2021-04-11 17:38:11 +03:00
Vasily Nemkov
0f7c83b5ac
Fixed parsing DateTime64 as unix timestamp with subsecond precision scaled to integer.
...
e.g. toDateTime64('1111111111222', 3)
2021-04-11 17:28:58 +03:00
kreuzerkrieg
228a0a0647
Fix an error handling in Poco HTTP Client for AWS.
2021-04-11 17:22:19 +03:00
Azat Khuzhin
b24d1251de
Introduce helper for removing one part from fs for ReplicatedMergeTree
2021-04-11 14:38:46 +03:00
alexey-milovidov
0f6fc5bd72
Merge pull request #22962 from kitaisreal/storage-external-distributed-arcadia-fix
...
StorageExternalDistributed arcadia fix
2021-04-11 13:34:03 +03:00
alexey-milovidov
d21ffd83a7
Merge branch 'master' into catboost-reload
2021-04-11 13:29:36 +03:00
Maksim Kita
20d5d550c4
Merge pull request #22960 from kitaisreal/dictionary-added-decimal256-attribute-type-support
...
Dictionary added Decimal256 attribute type support
2021-04-11 13:07:00 +03:00
Azat Khuzhin
940ff6cb76
Do not use c-style cast for std::min(max_part_removal_threads)
2021-04-11 12:22:39 +03:00
Maksim Kita
8dbf721e52
Merge pull request #22961 from kitaisreal/dictionaries-standardize-exceptions
...
Dictionaries standardize exceptions
2021-04-11 12:05:59 +03:00
Azat Khuzhin
6e71757814
Fix thread accounting for max_part_removal_threads
2021-04-11 11:43:30 +03:00
Azat Khuzhin
dfe3699e77
Respect max_part_removal_threads for ReplicatedMergeTree
2021-04-11 11:33:45 +03:00
pingyu
251804e9aa
fix style check error
2021-04-11 14:23:39 +08:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Alexey Milovidov
d369da6643
Update tests
2021-04-11 00:52:42 +03:00
alexey-milovidov
5469cc15c1
Trigger CI
2021-04-10 23:43:56 +03:00
Alexey Milovidov
1f2226b9db
Check out of range values in FieldVisitorConverToNumber
2021-04-10 23:38:14 +03:00
alexey-milovidov
e9b39c40f8
Merge pull request #22947 from azat/inactive_parts_to_throw_insert-fix
...
Fix inactive_parts_to_throw_insert=0 with inactive_parts_to_delay_insert>0
2021-04-10 23:21:16 +03:00
Denis Glazachev
5e8fe847a2
Remove unneeded changes
...
Fix GCC build
2021-04-10 23:23:18 +04:00
Maksim Kita
685da4e234
StorageExternalDistributed arcadia fix
2021-04-10 21:52:17 +03:00
Maksim Kita
b43a709fec
Dictionaries standardize exceptions
2021-04-10 21:48:36 +03:00
Maksim Kita
7df43891c1
Dictionary added Decimal256 attribute type support
2021-04-10 19:53:21 +03:00
pingyu
df78a1b3fd
address review comments
2021-04-10 23:00:34 +08:00
changvvb
03a2ba315b
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into support-apple-m1
2021-04-10 21:31:38 +08:00
alexey-milovidov
2d4a8c4787
Merge pull request #22885 from traceon/gcc-macos-compilation-fixes
...
Fix vanilla GCC compilation in macOS
2021-04-10 14:17:11 +03:00
alexey-milovidov
969d9f495b
Merge pull request #22939 from CurtizJ/fix-const-map-conversion
...
Fix conversion from const map with nulls
2021-04-10 14:13:58 +03:00
Maksim Kita
8b34a54c2d
Merge pull request #22938 from kitaisreal/aggregate-function-allocates-memory-in-arena-default-impl-fix
...
IAggreagteFunction allocatesMemoryInArena removed default implementation
2021-04-10 14:08:59 +03:00
Azat Khuzhin
a6703c5ac2
Fix inactive_parts_to_throw_insert=0 with inactive_parts_to_delay_insert>0
2021-04-10 09:57:08 +03:00
Alexey Milovidov
173c0e5c43
More conservative value of background_fetches_pool_size setting
2021-04-09 23:45:14 +03:00
Alexander Kuzmenkov
e44b3822e3
Merge pull request #21850 from fastio/handle_errors_for_kafka_engine
...
Handle errors for Kafka engine
2021-04-09 22:59:40 +03:00
Anton Popov
329c9033a8
fix conversion from const map with nulls
2021-04-09 20:50:10 +03:00
Maksim Kita
d9e9224188
IAggreagteFunction allocatesMemoryInArena removed default implementation
2021-04-09 20:12:06 +03:00
Vasily Nemkov
c882d290bd
Fixed parsing string to DateTime64 for large values.
...
Before fix:
SELECT toDateTime64('2201-01-12 12:12:12', 3, 'UTC')
1970-03-26 09:10:48.237
After fix:
SELECT toDateTime64('2201-01-12 12:12:12', 3, 'UTC')
2201-01-12 12:12:12.000
2021-04-09 19:39:54 +03:00
Vasily Nemkov
4dcc1b2d2e
Fixed formatDateTime for DateTime64 arguments
2021-04-09 19:39:54 +03:00
changvvb
facbb0368b
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into support-apple-m1
2021-04-09 23:19:19 +08:00
Alexander Kuzmenkov
0264124146
Merge pull request #21942 from ucasFL/distributed_depth
...
Add settings max_distributed_depth
2021-04-09 15:52:58 +03:00
Alexander Kuzmenkov
5db8e4c111
Update SystemLog.h
2021-04-09 15:48:09 +03:00
Vasily Nemkov
4a784bd2e3
Fixed dateDiff for DateTime64
...
Added tests that verify fix
- against DateTime64 vals of equal scale
- DateTime64 vs DateTime
- DateTime vs DateTime64
- DateTime64 vs Date
- Date vs DateTime64
2021-04-09 15:29:16 +03:00
kssenii
72d89bc933
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into nanodbc
2021-04-09 11:13:01 +00:00
Nikolai Kochetov
d0b5615641
Add comments.
2021-04-09 12:35:21 +03:00
Maksim Kita
0cacca5b6e
Merge pull request #22833 from ClickHouse/trivial-fix
...
Fix what looks like a trivial mistake
2021-04-09 12:02:16 +03:00
Nikolai Kochetov
28ca191102
Merge pull request #22087 from ClickHouse/better-filter-push-down
...
Better filter push down
2021-04-09 10:22:17 +03:00
alesapin
4b0f973f93
Merge pull request #18215 from filimonov/quorum-dedup-issue
...
Fix issue with quorum retries behaviour
2021-04-09 10:19:50 +03:00
alexey-milovidov
42b865c9c1
Merge pull request #22829 from azat/dist-send-log
...
Slightly improve logging messages for Distributed async sends
2021-04-09 09:51:53 +03:00
alexey-milovidov
ee9b53a342
Merge pull request #22863 from kitaisreal/data-type-low-cardinality-format-tsv-parsing-issue
...
DataTypeLowCardinality format tsv parsing issue
2021-04-09 09:50:50 +03:00
alexey-milovidov
f7edcdf7c8
Merge pull request #22867 from azat/sparse_hashed-bytes_allocated-fix
...
Fix bytes_allocated for sparse_hashed dictionaries
2021-04-09 09:50:04 +03:00
alexey-milovidov
bcb8da031d
Merge pull request #22860 from ClickHouse/fix_coordination_build_macos
...
Fix Coordination MacOS build
2021-04-09 09:48:44 +03:00
alexey-milovidov
b02a82250b
Merge pull request #22858 from vitlibar/fix-hanging-in-temporary-live-view-cleaner
...
Fix hanging in TemporaryLiveViewCleaner
2021-04-09 09:25:38 +03:00
alexey-milovidov
8af2352e27
Merge pull request #22876 from nikitamikhaylov/mann-whitney-over-window
...
`mannWhitneyUTest` and `rankCorr` with window functions
2021-04-09 09:17:27 +03:00
Denis Glazachev
e8464d61f1
Fix vanilla GCC compilation in macOS
2021-04-09 04:28:24 +04:00
Nikita Mikhaylov
86933749bd
done
2021-04-09 00:22:18 +03:00
Anton Popov
1c28878f5d
Merge pull request #22726 from azat/mt-reverse-total-rows-approx-fix
...
Fix approx total rows accounting for reverse reading from MergeTree
2021-04-09 00:16:49 +03:00
Azat Khuzhin
93c9a02480
Fix bytes_allocated for sparse_hashed dictionaries
2021-04-08 23:29:29 +03:00
Nikolai Kochetov
3426bc3906
Merge pull request #22763 from CurtizJ/fix-having-push-down
...
Fix pushdown of having
2021-04-08 21:53:50 +03:00
Maksim Kita
942399983c
DataTypeLowCardinality format tsv parsing issue
2021-04-08 20:12:41 +03:00
Nikolai Kochetov
3ea186358e
Fix fuzz crash.
2021-04-08 18:00:32 +03:00
kssenii
80fc675258
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into postgresql-protocol-with-row-policy
2021-04-08 14:46:56 +00:00
alesapin
07442b1170
Fix stupid sed
2021-04-08 17:24:05 +03:00
alesapin
c28a3b860c
Fix Coordination darwin build
2021-04-08 17:17:57 +03:00
Nikolai Kochetov
794d7c89b6
Remove ReverseRowsStep
2021-04-08 16:18:17 +03:00
Vitaly Baranov
5bcb7a0fed
Fix hanging in TemporaryLiveViewCleaner
2021-04-08 16:14:49 +03:00
Nikolai Kochetov
febb187da0
Added test,
2021-04-08 15:54:27 +03:00
Alexander Kuzmenkov
4a66548c5d
write distributed depth as VarUInt to save space, because it's small
2021-04-08 15:17:47 +03:00
Nikolai Kochetov
7ffbeac9df
Add info about indexes to ReadFromMergeTree step.
2021-04-08 14:48:54 +03:00
Alexander Kuzmenkov
dc7a2b7f88
Merge pull request #22761 from ClickHouse/aku/flush-log
...
add more messages when flushing the logs
2021-04-08 13:50:17 +03:00
alesapin
e762d02e37
More const
2021-04-08 13:38:40 +03:00
alesapin
e17444757b
Fix rare case when quorum insert is not really quorum because of deduplication
2021-04-08 13:35:38 +03:00
changvvb
b070090ca7
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into support-apple-m1
2021-04-08 18:11:16 +08:00
Nikolai Kochetov
118090396f
Filter column after filter push down now always the first one..
2021-04-08 12:54:39 +03:00
Alexey Milovidov
8fb6e6fbf2
Fix mistake
2021-04-08 11:51:22 +03:00
Nikolai Kochetov
7c5a9133df
Add index info to ReadFromStorageStep.
2021-04-08 11:19:04 +03:00
alesapin
1533f9b9aa
Merge pull request #14113 from johnskopis/dynamic-interserver-creds-v20
...
Support interserver credential rotation
2021-04-08 11:01:40 +03:00
alexey-milovidov
b8e239c211
Merge pull request #22821 from kitaisreal/functions-external-dictionaries-standardize-exception-throw
...
Functions ExternalDictionaries standardize exception throw
2021-04-08 10:41:41 +03:00
alexey-milovidov
ed367855d5
Merge pull request #22822 from kitaisreal/file-dictionary-source-fix-absolute-path
...
FileDictionarySource fix absolute file path
2021-04-08 10:34:07 +03:00
Azat Khuzhin
c27b931f6a
Slightly improve logging messages for Distributed async sends
...
- add took time (in ms)
- add rows/bytes
2021-04-08 08:10:39 +03:00
Nicolae Vartolomei
7259834fad
More Arrow like code style
2021-04-07 23:33:25 +03:00
Nicolae Vartolomei
4ea363006b
Experiment with ArrowStream streaming
2021-04-07 23:33:25 +03:00
Maksim Kita
ef7571c226
Merge pull request #22521 from kitaisreal/dictionaries-update-field-fix
...
FlatDictionary, HashedDictionary update_field optimizations
2021-04-07 23:29:53 +03:00
alesapin
c85a60d795
Merge pull request #22743 from ClickHouse/fix_potential_segfault_on_startup
...
Fix potential segfault on Keeper startup
2021-04-07 23:10:29 +03:00
Maksim Kita
ff04292ac8
Update FileDictionarySource.cpp
2021-04-07 21:42:09 +03:00
Maksim Kita
f7cc15fdcd
FileDictionarySource fix absolute file path
2021-04-07 21:38:24 +03:00
Maksim Kita
caff65f1c2
Functions ExternalDictionaries standardize exception throw
2021-04-07 20:52:12 +03:00
alesapin
b2a370ad49
Fix PVS
2021-04-07 19:06:20 +03:00
alesapin
98871f8c5a
More clear messages
2021-04-07 17:05:40 +03:00
Anton Popov
fa9a19cbdd
fix pushdown of having
2021-04-07 17:02:59 +03:00
alesapin
2a6560b1c6
Better interface
2021-04-07 16:59:18 +03:00
alesapin
2987bbc948
Small improvements
2021-04-07 16:52:11 +03:00
Ivan
aa47eb7d5e
Fix checkpoint position after putting it on buffer end ( #22518 )
2021-04-07 16:22:38 +03:00
Alexander Kuzmenkov
fecd5f3435
more debug messages
2021-04-07 16:16:19 +03:00
Nikita Mikhaylov
48af7a898c
Merge pull request #22299 from nikitamikhaylov/keen-wolf-storagefile-column-oriented
...
Merging #21302
2021-04-07 16:01:12 +03:00
kssenii
15ae912b56
Fix postgresql protocol with row policy
2021-04-07 12:56:09 +00:00
Alexander Kuzmenkov
06bb58cb69
Merge pull request #22704 from ClickHouse/aku/window-multiple
...
fix window functions with multiple input streams and no sorting
2021-04-07 15:15:47 +03:00
Vladimir
9fe20c1628
Revert "Move conditions from JOIN ON to WHERE"
2021-04-07 14:57:20 +03:00
Kruglov Pavel
5b52657428
Merge pull request #22725 from azat/buffer-lock-metrics
...
Add metric to track how much time is spend during waiting for Buffer layer lock
2021-04-07 14:35:57 +03:00
alesapin
feff1175f4
Merge branch 'master' into johnskopis/dynamic-interserver-creds-v20
2021-04-07 13:55:38 +03:00
alesapin
a1164a7e4c
More consistent
2021-04-07 13:21:53 +03:00
alesapin
36c0e601a9
Better non-dirty fix
2021-04-07 13:18:07 +03:00
Ilya Yatsishin
ed2d82603e
Merge pull request #22588 from foolchi/fix_dist_map_select
...
fix dist map type select
2021-04-07 11:53:23 +03:00
Vladimir
864ab4a7ab
Merge pull request #21370 from vdimir/fix-merge-engine-join-group-by
2021-04-07 11:52:27 +03:00
Vladimir
db0550013c
Merge pull request #22325 from azat/dist-async-insert-header-check
...
Check only column name and type for async distributed blocks
2021-04-07 11:51:12 +03:00
alesapin
05eeec16c1
Fix potential segfault on Keeper startup
2021-04-07 11:49:10 +03:00
Maksim Kita
7baafcb567
Merge pull request #22630 from kitaisreal/direct-dictionary-dict-get-multiple-columns-optimization
...
DirectDictionary dictGet multiple columns optimization
2021-04-07 10:29:42 +03:00
alesapin
9fd251eaaf
Merge pull request #22514 from ClickHouse/merge_tree_deduplication
...
Non replicated merge tree deduplication
2021-04-07 10:19:43 +03:00
alexey-milovidov
80cb871e20
Merge pull request #22684 from ClickHouse/exception-safety-in-zk-client-2
...
Always call event.set for zk client callbacks.
2021-04-07 07:20:41 +03:00
alexey-milovidov
a7e216969a
Merge pull request #22592 from azat/uncaught-exceptions
...
Fix some uncaught exceptions (in SCOPE_EXIT) under memory pressure
2021-04-07 07:17:26 +03:00
hexiaoting
ee24d9a5cc
Merge remote-tracking branch 'origin/master' into dev_nested_upadte
2021-04-07 10:42:51 +08:00
hexiaoting
8c60f95551
Fix error
2021-04-07 10:39:01 +08:00
feng lv
78afb5b23c
fix
2021-04-07 01:58:57 +00:00
alesapin
c77a006325
Merge pull request #22707 from ClickHouse/don_t_fsync_on_log_rotate
...
Remove redundant fsync on coordination logs rotation
2021-04-06 23:51:28 +03:00
kssenii
0b39480d51
Better
2021-04-06 20:01:23 +00:00
alexey-milovidov
5d672d4529
Update FormatFactory.cpp
2021-04-06 22:23:16 +03:00
Nikita Mikhailov
71a2c85d9a
better
2021-04-06 22:23:16 +03:00
Nikita Mikhailov
af85ce2fd1
better
2021-04-06 22:23:16 +03:00
Nikita Mikhailov
37f48d13b4
add test
2021-04-06 22:23:16 +03:00
Nikita Mikhailov
282775b10c
save
2021-04-06 22:23:16 +03:00
keenwolf
1a4961499b
the Native format is not supported after test
2021-04-06 22:23:16 +03:00
keenwolf
94c8e7af1a
remove const for value-copy-return
2021-04-06 22:23:16 +03:00
keenwolf
3d3923b518
update comments
2021-04-06 22:23:16 +03:00
keenwolf
b95f54edce
Small fix
2021-04-06 22:23:16 +03:00
keenwolf
139bb678a9
Branch with Format isColumnOriented() or not
2021-04-06 22:23:15 +03:00
keenwolf
4eaf718fd2
fix the getColumsForNames() to bring the whole column info from metadata
2021-04-06 22:23:15 +03:00
keenwolf
9d6e2b4bad
update comments
2021-04-06 22:23:15 +03:00
keenwolf
de365a5852
fix
2021-04-06 22:23:15 +03:00
keenwolf
6f9bad682f
Only read needed columns for formats as parquet etc
2021-04-06 22:23:15 +03:00
Maksim Kita
e38206ee02
Merge pull request #22721 from kitaisreal/memory-storage-sync-comments-and-code
...
MemoryStorage sync comments and code
2021-04-06 21:56:41 +03:00
Azat Khuzhin
60b000ed51
Fix approx total rows accounting for reverse reading from MergeTree
2021-04-06 21:13:51 +03:00
Azat Khuzhin
c4a7e81287
Add metric to track how much time is spend during waiting for Buffer layer lock
...
It uses very fast CLOCK_MONOTONIC_COARSE, so this should not be a
problem.
Also note that there is no sense in using microseconds/nanoseconds since
accuracy of CLOCK_MONOTONIC_COARSE usually milliseconds.
2021-04-06 21:13:24 +03:00
Azat Khuzhin
f157278b72
Safer SCOPE_EXIT
...
It executes the code in the dtor, that should never throw.
2021-04-06 21:10:37 +03:00
Azat Khuzhin
b25a4e066c
Lock MEMORY_LIMIT_EXCEEDED in ThreadStatus::detachQuery()
...
Found with fuzzer [1]:
<Fatal> BaseDaemon: (version 21.5.1.6440, build id: 3B097C902DDAA35688D90750552ED499DC5D10A0) (from thread 8012) Terminate called for uncaught exception:
Code: 241, e.displayText() = DB::Exception: Memory limit (for user) exceeded: would use 153.51 MiB (attempt to allocate chunk of 4194368 bytes), maximum: 150.00 MiB, Stack trace (when copying this message, always include the lines below):
0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x26fdcdd9 in /usr/bin/clickhouse
1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:57: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xad0df02 in /usr/bin/clickhouse
2. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/string:1444: DB::Exception::Exception<char const*, char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, long&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*&&, char const*&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, long&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0xacc7bef in /usr/bin/clickhouse
3. ./obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:219: MemoryTracker::alloc(long) @ 0xacc65eb in /usr/bin/clickhouse
4. ./obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:0: MemoryTracker::alloc(long) @ 0xacc5dad in /usr/bin/clickhouse
5. ./obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:0: MemoryTracker::alloc(long) @ 0xacc5dad in /usr/bin/clickhouse
6. ./obj-x86_64-linux-gnu/../src/Common/AllocatorWithMemoryTracking.h:35: AllocatorWithMemoryTracking<DB::Field>::allocate(unsigned long) @ 0xad0a2fe in /usr/bin/clickhouse
7. void std::__1::vector<DB::Field, AllocatorWithMemoryTracking<DB::Field> >::__push_back_slow_path<DB::Field>(DB::Field&&) @ 0x11712a51 in /usr/bin/clickhouse
8. ./obj-x86_64-linux-gnu/../src/Interpreters/ThreadStatusExt.cpp:356: DB::ThreadStatus::detachQuery(bool, bool) @ 0x1f5d5237 in /usr/bin/clickhouse
9. ./obj-x86_64-linux-gnu/../src/Processors/Executors/PipelineExecutor.cpp:0: void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) @ 0x20c488e6 in /usr/bin/clickhouse
10. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:0: ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xad9f6cc in /usr/bin/clickhouse
11. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:1655: void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) @ 0xada8264 in /usr/bin/clickhouse
12. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
13. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
(version 21.5.1.6440)
[1]: https://clickhouse-test-reports.s3.yandex.net/22583/69296876005c0fa171c755f8b224e4d58192c402/stress_test_(address).html#fail1
2021-04-06 21:10:36 +03:00
alesapin
95881bf2dd
Fix duplicated headers
2021-04-06 19:31:24 +03:00
Alexander Kuzmenkov
d9ad06bd6d
fix ubsan
2021-04-06 19:24:56 +03:00
alesapin
b4fc13de8e
fix style
2021-04-06 17:02:56 +03:00
alesapin
4633afe962
Some fixes
2021-04-06 16:56:14 +03:00
alesapin
d913f327ed
Buildable code
2021-04-06 16:42:38 +03:00
alesapin
1c8e33414f
Merge branch 'master' into johnskopis/dynamic-interserver-creds-v20
2021-04-06 16:28:46 +03:00
changvvb
55a2ea4a79
GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(DateLUTWithTimeZoneAndTimeRange);
2021-04-06 21:26:51 +08:00
Amos Bird
a89e3df40a
Disable table function view in expression ( #21465 )
2021-04-06 16:08:29 +03:00
Nikolai Kochetov
9f39f5d52d
Add more counters to MergeTreeDataSelectExecutor
2021-04-06 15:39:55 +03:00
alesapin
a40209e84d
Remove strange fsync on coordination logs rotation
2021-04-06 15:25:15 +03:00
Alexander Kuzmenkov
585da2c8d0
Merge remote-tracking branch 'origin/master' into HEAD
2021-04-06 14:58:21 +03:00
Alexey Milovidov
f70c9e8752
Merge branch 'master' into protocol-compression-auto
2021-04-06 14:53:31 +03:00
alexey-milovidov
3708821689
Merge pull request #22655 from azat/system.errors-invalid-read
...
Fix impossible invalid-read for system.errors accounting
2021-04-06 14:48:33 +03:00
Alexander Kuzmenkov
06e7a9f8f2
fix window functions with multiple input streams and no sorting
2021-04-06 14:43:27 +03:00
Maksim Kita
af19f89273
MemoryStorage sync comments and code
2021-04-06 13:41:48 +03:00
Nikolai Kochetov
6102652c99
Merge branch 'master' into better-filter-push-down
2021-04-06 13:38:03 +03:00
Kseniia Sumarokova
ad85467128
Merge pull request #22668 from kssenii/postgres-add-bool
...
Add bool type in postgres engine
2021-04-06 13:29:33 +03:00
alesapin
53a7836a9b
Fix typos
2021-04-06 13:16:34 +03:00
Alexander Kuzmenkov
4402ec3101
Merge pull request #22324 from ClickHouse/aku/perf-stable
...
make some perf test queries more stable
2021-04-06 13:15:38 +03:00
alesapin
6456a1507c
Support alter setting
2021-04-06 13:14:44 +03:00
vdimir
735154c81a
Handle not plain where tree in StorageMerge modifySelect
2021-04-06 12:29:29 +03:00
alesapin
86a843bb51
Merge pull request #21509 from kssenii/library-bridge
...
clickhouse-library-bridge for library dictionary source
2021-04-06 12:26:08 +03:00
Nikolai Kochetov
44700d97e9
Add setting to disable query plan optimization. Fix perftest.
2021-04-06 11:25:54 +03:00
tavplubix
8b1d320633
Merge pull request #22217 from kssenii/replicas-shards-for-mysql-and-postgres
...
Replicas and shards for mysql and postgres storages
2021-04-06 11:05:24 +03:00
alesapin
3cd571f92b
Merge pull request #22669 from ClickHouse/fix_two_bugs_with_mutation_wait
...
Fix two stupid bugs in mutation wait
2021-04-06 11:04:16 +03:00
changvvb
1f8bd034a3
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into support-apple-m1
2021-04-06 15:55:03 +08:00
changvvb
3a9d063f12
Fix unit test
2021-04-06 15:54:35 +08:00
Azat Khuzhin
27d4fbd13b
Compare Block itself for distributed async INSERT batches
...
INSERT into Distributed with insert_distributed_sync=1 stores the
distributed batches on the disk for sending in background.
But types may be a little bit different for the Distributed and it's
underlying table, so the initiator need to know whether conversion is
required or not.
Before this patch those on disk distributed batches contains header,
which includes dumpStructure() for the block in that batch, however it
checks not only names and types and plus dumpStructure() is a debug
method.
So instead of storing string representation for the block header we
should store empty block in the file header (note, that we cannot store
the empty block not in header, since this will require reading all
blocks from file, due to some trickery of the readers interface).
Note, that this patch also contains tiny refactoring:
- s/header/distributed_header/
v1: dumpNamesAndTypes()
v2: dump empty block into the batch itself
v3: move empty block into the header
2021-04-06 10:05:21 +03:00
Amos Bird
46c9affa83
Fix CTE in view.
2021-04-06 11:32:16 +08:00
Anton Popov
c5354885a6
Merge pull request #22654 from azat/log-nested-fix
...
[RFC] Fix LOGICAL_ERROR for Log with nested types w/o columns in the SELECT clause
2021-04-06 00:34:58 +03:00
kssenii
68154fe10a
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into postgres-add-bool
2021-04-05 20:27:04 +00:00
Maksim Kita
de71553c9d
Merge pull request #22593 from zlobober/introduce_distributed_write
...
Introduce IStorage::distributedWrite method for distributed INSERT SELECTS
2021-04-05 22:59:57 +03:00
kssenii
dc42d5189d
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres
2021-04-05 15:36:35 +00:00
alesapin
c830397317
Fix two stupid bugs in mutation wait
2021-04-05 18:08:43 +03:00
kssenii
cc2fdc6c62
Add bool type
2021-04-05 14:49:52 +00:00
kssenii
89a2e94364
Fixes
2021-04-05 14:08:49 +00:00
feng lv
f6e4e13f89
fix conflict
2021-04-05 14:08:30 +00:00
feng lv
56073db22d
max distributed depth
...
Add settings max_distributed_depth
fix style
fix
fix
fix
fix test
fix
fix
2021-04-05 14:00:54 +00:00
kssenii
9dba529494
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge
2021-04-05 13:14:51 +00:00
