thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-24 16:42:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
130,634 Commits 9,168 Branches 1,781 Tags 2.1 GiB
57d7a58dbe
Commit Graph

130634 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexey Milovidov
355fa8dbab
Merge pull request #51866 from ClickHouse/rschu1ze-azure-darwin2 
Merging #51326.
 2023-12-10 17:49:16 +01:00
Nikolay Degterinsky
8b5c694684 Fix 2023-12-10 16:45:17 +00:00
Azat Khuzhin
2707e38c69 Fix dead-lock in DatabaseMaterializedPostgreSQL after interface changes 
Deadlock for ATTACH:

    /src/ch/clickhouse/src/Common/ThreadFuzzer.cpp:0::pthread_mutex_lock
    /src/ch/clickhouse/contrib/llvm-project/libcxx/src/mutex.cpp:39::std::__1::mutex::lock()
    /src/ch/clickhouse/src/Databases/DatabaseOrdinary.cpp:240::DB::DatabaseOrdinary::waitTableStarted(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const
    /src/ch/clickhouse/src/Databases/DatabasesCommon.cpp:203::DB::DatabaseWithOwnTablesBase::tryGetTable(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>) const
    /src/ch/clickhouse/src/Databases/PostgreSQL/DatabaseMaterializedPostgreSQL.cpp:335::DB::DatabaseMaterializedPostgreSQL::attachTableUnlocked(std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::IStorage> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
    /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__mutex_base💯:DB::IDatabase::attachTable(std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::IStorage> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
    /src/ch/clickhouse/contrib/llvm-project/libcxx/include/string:1499::DB::DatabaseMaterializedPostgreSQL::createTable(std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::IStorage> const&, std::__1::shared_ptr<DB::IAST> const&)
    /src/ch/clickhouse/contrib/llvm-project/libcxx/include/string:1499::DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /src/ch/clickhouse/src/Interpreters/InterpreterCreateQuery.cpp:0::DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /src/ch/clickhouse/src/Interpreters/InterpreterCreateQuery.cpp:1751::DB::InterpreterCreateQuery::execute()
    /src/ch/clickhouse/src/Interpreters/executeQuery.cpp:0::DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*)
    /src/ch/clickhouse/src/Interpreters/executeQuery.cpp:1287::DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum)
    /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:612::DB::TCPHandler::runImpl()
    /src/ch/clickhouse/src/Server/TCPHandler.cpp:2294::DB::TCPHandler::run()
    /src/ch/clickhouse/base/poco/Net/src/TCPServerConnection.cpp:57::Poco::Net::TCPServerConnection::start()
    /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48::Poco::Net::TCPServerDispatcher::run()
    /src/ch/clickhouse/base/poco/Foundation/src/ThreadPool.cpp:202::Poco::PooledThread::run()
    /src/ch/clickhouse/base/poco/Foundation/include/Poco/SharedPtr.h:139::Poco::ThreadImpl::runnableEntry(void*)

CI: https://s3.amazonaws.com/clickhouse-test-reports/55271/ece8c86ebc02908e36786ca32d68f9ce0ddcb35d/integration_tests__tsan__[1_6].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 16:06:18 +01:00
Azat Khuzhin
a6cf2297ef Fix Context use-after-free in ServerAsynchronousMetrics 
TSan found [1]:

    WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=598)
      Write of size 8 at 0x7ffcf9ad9cb0 by main thread:
        0 DB::AsynchronousMetrics::~AsynchronousMetrics() build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 (clickhouse+0xf38ed6e) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        1 DB::ServerAsynchronousMetrics::~ServerAsynchronousMetrics() build_docker/./src/Interpreters/ServerAsynchronousMetrics.h:10:7 (clickhouse+0xf2b2220) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        2 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./programs/server/Server.cpp:1995:1 (clickhouse+0xf2b2220)
        3 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d6c535e) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        4 DB::Server::run() build_docker/./programs/server/Server.cpp:397:25 (clickhouse+0xf29d341) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        5 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d6e50b4) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        6 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:203:20 (clickhouse+0xf29a4a3) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        7 main build_docker/./programs/main.cpp:505:12 (clickhouse+0x72d72a0) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)

      Previous read of size 8 at 0x7ffcf9ad9cb0 by thread T727:
        0 DB::AsynchronousMetrics::update(std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Common/AsynchronousMetrics.cpp:1559:5 (clickhouse+0xf38dec5) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        1 DB::AsynchronousMetrics::run() build_docker/./src/Common/AsynchronousMetrics.cpp:354:13 (clickhouse+0xf38f996) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        2 DB::AsynchronousMetrics::start()::$_0::operator()() const build_docker/./src/Common/AsynchronousMetrics.cpp:273:62 (clickhouse+0xf3921ca) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        3 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&>(DB::AsynchronousMetrics::start()::$_0&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3921ca)
        4 decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1 (clickhouse+0xf3921ca)
        5 decltype(auto) std::__1::apply[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1 (clickhouse+0xf3921ca)
        6 ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:253:13 (clickhouse+0xf3921ca)
        7 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(DB::AsynchronousMetrics::start()::$_0&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3921ca)
        8 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf3921ca)
        9 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf3921ca)
        10 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf3921ca)
        11 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xf305b2e) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        12 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0xf305b2e)
        13 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:421:13 (clickhouse+0xf305b2e)
        14 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:183:73 (clickhouse+0xf30c6d1) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        15 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf30c6d1)
        16 void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0xf30c6d1)
        17 void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0xf30c6d1)

    SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 in DB::AsynchronousMetrics::~AsynchronousMetrics()

Or MSan [2]:

    ==573==WARNING: MemorySanitizer: use-of-uninitialized-value
        0 0x55b57079112d in std::__1::weak_ptr<DB::Context const>::lock() const build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1645:20
        1 0x55b57079112d in DB::WithContextImpl<std::__1::shared_ptr<DB::Context const>>::getContext() const build_docker/./src/Interpreters/Context_fwd.h:41:28
        2 0x55b57079112d in DB::ServerAsynchronousMetrics::updateImpl(std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, DB::AsynchronousMetricValue, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, DB::AsynchronousMetricValue>>>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Interpreters/ServerAsynchronousMetrics.cpp:260:63
        3 0x55b559540309 in DB::AsynchronousMetrics::update(std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Common/AsynchronousMetrics.cpp:1559:5
        4 0x55b55954258c in DB::AsynchronousMetrics::run() build_docker/./src/Common/AsynchronousMetrics.cpp:354:13
        5 0x55b559549111 in DB::AsynchronousMetrics::start()::$_0::operator()() const build_docker/./src/Common/AsynchronousMetrics.cpp:273:62
        6 0x55b559549111 in decltype(std::declval<DB::AsynchronousMetrics::start()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&>(DB::AsynchronousMetrics::start()::$_0&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
        7 0x55b559549111 in decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1
        8 0x55b559549111 in decltype(auto) std::__1::apply[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1
        9 0x55b559549111 in ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:253:13
        10 0x55b559549111 in decltype(std::declval<DB::AsynchronousMetrics::start()::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(DB::AsynchronousMetrics::start()::$_0&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
        11 0x55b559549111 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9
        12 0x55b559549111 in std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12
        13 0x55b559549111 in void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16
        14 0x55b5593eb38a in std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16
        15 0x55b5593eb38a in std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12
        16 0x55b5593eb38a in ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:421:13
        17 0x55b5593f9a0a in void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:183:73
        18 0x55b5593f9a0a in decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
        19 0x55b5593f9a0a in void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5
        20 0x55b5593f9a0a in void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5
        21 0x7f7ff3899ac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2) (BuildId: a43bfc8428df6623cd498c9c0caeb91aec9be4f9)
        22 0x7f7ff392ba3f  (/lib/x86_64-linux-gnu/libc.so.6+0x126a3f) (BuildId: a43bfc8428df6623cd498c9c0caeb91aec9be4f9)

      Member fields were destroyed
        0 0x55b541a72c9d in __sanitizer_dtor_callback_fields (/usr/bin/clickhouse+0x7c6dc9d) (BuildId: 57941f7730deefd0a2028f1d9f3e173472a4aa76)
        1 0x55b559304b83 in std::__1::weak_ptr<DB::Context const>::~weak_ptr() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1397:26
        2 0x55b559304b83 in std::__1::weak_ptr<DB::Context const>::~weak_ptr() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1553:1
        3 0x55b559304b83 in DB::WithContextImpl<std::__1::shared_ptr<DB::Context const>>::~WithContextImpl() build_docker/./src/Interpreters/Context_fwd.h:30:8
        4 0x55b559304b83 in DB::ServerAsynchronousMetrics::~ServerAsynchronousMetrics() build_docker/./src/Interpreters/ServerAsynchronousMetrics.h:10:7
        5 0x55b559304b83 in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./programs/server/Server.cpp:1995:1
        6 0x55b5789f579e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8
        7 0x55b5592d762a in DB::Server::run() build_docker/./programs/server/Server.cpp:397:25
        8 0x55b578a3efdf in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9
        9 0x55b5592d0489 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:203:20
        10 0x55b541acd4ab in main build_docker/./programs/main.cpp:505:12
        11 0x7f7ff382ed8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: a43bfc8428df6623cd498c9c0caeb91aec9be4f9)

SUMMARY: MemorySanitizer: use-of-uninitialized-value build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1645:20 in std::__1::weak_ptr<DB::Context const>::lock() const

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/52717/fcdead023c4350233ef1e0f7f82a71653ed62229/stress_test__tsan_.html
  [2]: https://s3.amazonaws.com/clickhouse-test-reports/52717/fcdead023c4350233ef1e0f7f82a71653ed62229/stress_test__msan_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 16:03:52 +01:00
Azat Khuzhin
8f5a60eb5a More descriptive comment for destroying context prio global thread pool 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 15:39:43 +01:00
Vitaly Baranov
a058a26b7e
Merge pull request #57454 from vitlibar/fix-rwlock 
Fix RWLock inconsistency after write lock timeout
 2023-12-10 14:09:29 +01:00
Azat Khuzhin
ce9fd2c57a Fix test_dictionaries_update_and_reload/test.py::test_reload_while_loading flakiness 
On CI this test fails due to start_time == prev_start_time:

            start_time, duration = get_loading_start_time("slow"), get_loading_duration("slow")
    >       assert start_time > prev_start_time
    E       assert time.struct_time(tm_year=2023, tm_mon=12, tm_mday=9, tm_hour=23, tm_min=42, tm_sec=2, tm_wday=5, tm_yday=343, tm_isdst=-1) > time.struct_time(tm_year=2023, tm_mon=12, tm_mday=9, tm_hour=23, tm_min=42, tm_sec=2, tm_wday=5, tm_yday=343, tm_isdst=-1)

The reason I guess is that there is sleep(0.5), while the
loading_start_time is DateTime not DateTime64, so you cannot distinguish
values if the difference is less then one second

CI: https://s3.amazonaws.com/clickhouse-test-reports/57710/7af1c0885daaf1e41470c5fdd92abfc7b6b2befc/integration_tests__asan__[3_4].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 08:36:10 +01:00
Azat Khuzhin
ece8c86ebc Fix attaching table in Atomic engine after attachTableUnlocked() interface 2023-12-10 08:28:10 +01:00
Azat Khuzhin
fcdead023c Proper server termination in case of intermediate failures during initialization 
For instance on CI it fails because KeeperDispatch::initialize() failed,
but there is already proper KeeperDispatch::shutdown() in dtor, the
problem is that you first need to shutdown global context, that will
terminate all thread pools, and only after you can join global thread
pool.

CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/6ccbc2ea75c39234dff748cfb147bd44d65d3e75/integration_tests__asan__[3_4].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 08:01:40 +01:00
Bharat Nallan Chakravarthy
bfe2ffbce9 use the EVP interface to satisfy s390x builds 2023-12-09 21:31:21 -08:00
Vitaly Baranov
ab2f60ccef sleep() function now can be cancelled with KILL QUERY. 2023-12-10 03:19:25 +01:00
Alexey Milovidov
3d03a44f15 Merge branch 'master' into rschu1ze-azure-darwin2 2023-12-10 01:43:35 +01:00
Alexey Milovidov
b2572fceda Add a test 2023-12-10 01:42:43 +01:00
Alexey Milovidov
100b36e8a9 Do not check in replication 2023-12-10 01:32:42 +01:00
Alexey Milovidov
33b0ec7d55 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-10 01:21:43 +01:00
Nikolay Degterinsky
fc8656ee9e Fix unary operators parsing 2023-12-09 22:44:06 +00:00
Alexey Milovidov
ecbae2d8fd
Merge pull request #57711 from ClickHouse/musl-8 
Switch Fast Test to Musl
 2023-12-09 23:23:44 +01:00
Azat Khuzhin
4f9d90761b Attach tables lazily only for clickhouse-local 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
c14721bb03 Move lazy table initialization interface into IDatabase 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
f75484a0b1 Make IDatabase::attachTableUnlocked() virtual and use it for overrides (mostly) 2023-12-09 23:08:29 +01:00
Azat Khuzhin
cc9afe6f6e Replace --no-system-tables with loading virtual tables of system database lazily 
First of all the problem is that that simple 'SELECT 1' cannot be runned
without system.one, which makes --no-system-tables almost useless:

    $ ./clickhouse-debug local --no-system-tables -q "select 1"
    Code: 81. DB::Exception: Database system does not exist. (UNKNOWN_DATABASE)

Secondly there are just too much flags, and this one
(--no-system-tables) is too damn specific.

This patch should improve startup time of the clickhouse-local almost
3x in debug builds.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
6ccbc2ea75 Move io_uring reader into the Context from static to make it's thread joinable 
v2: fix for standalone keeper build
CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/72b1052f7c2d453308262924e767ab8dc2206933/stateless_tests__debug__[4_5].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 22:50:48 +01:00
Azat Khuzhin
7af1c0885d Do not call add*() from addBatchSparseSinglePlace() if there are no rows 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 22:47:25 +01:00
Azat Khuzhin
472a31e581 Fix SIGSEGV for aggregation of sparse columns with any() RESPECT NULL 
Add proper check into any() RESPECT NULL

Fixes: https://github.com/ClickHouse/ClickHouse/issues/57709
Fixes: https://github.com/ClickHouse/ClickHouse/pull/57189 (cc @Algunenano)
CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/7ec9c6a7f7490b5075407410dc193a2534216e15/fuzzer_astfuzzerdebug/report.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 22:47:15 +01:00
Igor Nikonov
87e47bdbee
Merge pull request #57467 from ClickHouse/rm-select 
Remove select() usage
 2023-12-09 22:27:28 +01:00
Alexey Milovidov
f3def573c6 Switch Fast Test to Musl 2023-12-09 21:54:09 +01:00
Alexey Milovidov
2eeb383579
Merge pull request #57180 from ClickHouse/musl-7 
Check what will happen if we build ClickHouse with Musl
 2023-12-09 21:47:28 +01:00
Vitaly Baranov
179a0a2e8f Fixed condition and add more chasserts for consistency checks. 2023-12-09 20:22:48 +01:00
Alexey Milovidov
f0df110b15 Enable some libraries 2023-12-09 19:30:53 +01:00
Alexey Milovidov
9c7bb7d914
Merge pull request #57693 from den-crane/test/http_content-type_test 
test for #33308
 2023-12-09 19:16:07 +01:00
Alexey Milovidov
b6e3e5691c Fix Musl build 2023-12-09 18:58:35 +01:00
Yakov Olkhovskiy
63619b5935 allow IPv6 to UInt128 conversion and binary arithmetic 2023-12-09 17:57:32 +00:00
Azat Khuzhin
b7edde3621 Shutdown embedded keeper even if it was not started successfully 
Since even when it does not, there are still few threads left that
should be joined (request, response, snapshot, snapshot s3):

There are some problems with keeper and invalid config, that got stuck in:

    libc.so.6`___lldb_unnamed_symbol3457 + 231
    libc.so.6`pthread_cond_timedwait + 571
    clickhouse`std::__1::__libcpp_condvar_timedwait[abi:v15000](__cv=0x00007f475c41a708, __m=0x00007f475c41a6b0, __ts=0x00007f45a99f1440) at __threading_support:341:10
    clickhouse`std::__1::condition_variable::__do_timed_wait(this=0x00007f475c41a708, lk=0x00007f45a99f1738, tp=time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1L, 1000000000L> > > @ 0x00007f45a99f1488) at condition_variable.cpp:76:14
    clickhouse`std::__1::cv_status std::__1::condition_variable::wait_for<long long, std::__1::ratio<1l, 1000000000l> >(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __d=0x00007f45a99f1590) at __mutex_base:457:9
    clickhouse`void std::__1::condition_variable::__do_timed_wait<std::__1::chrono::steady_clock>(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __tp=time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1L, 1000000000L> > > @ 0x00007f45a99f15a8) at __mutex_base:512:5
    clickhouse`std::__1::cv_status std::__1::condition_variable::wait_until<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __t=0x00007f45a99f1698) at __mutex_base:415:5
    clickhouse`bool std::__1::condition_variable::wait_until<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> >, bool ConcurrentBoundedQueue<DB::KeeperStorage::RequestForSession>::popImpl<true>(DB::KeeperStorage::RequestForSession&, std::__1::optional<unsigned long>)::'lambda'()>(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __t=0x00007f45a99f1698, __pred=(unnamed class) @ 0x00007f45a99f1650)::'lambda'()) at __mutex_base:427:13
    clickhouse`bool std::__1::condition_variable::wait_for[abi:v15000]<long long, std::__1::ratio<1l, 1000l>, bool ConcurrentBoundedQueue<DB::KeeperStorage::RequestForSession>::popImpl<true>(DB::KeeperStorage::RequestForSession&, std::__1::optional<unsigned long>)::'lambda'()>(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __d=0x00007f45a99f1720, __pred=(unnamed class) @ 0x00007f45a99f16b8)::'lambda'()) at __mutex_base:471:12
    clickhouse`bool ConcurrentBoundedQueue<DB::KeeperStorage::RequestForSession>::popImpl<true>(this=0x00007f475c41a680, x=0x00007f45a99f1ba0, timeout_milliseconds= Has Value=true ) at ConcurrentBoundedQueue.h:75:50
    clickhouse`ConcurrentBoundedQueue<DB::KeeperStorage::RequestForSession>::tryPop(this=0x00007f475c41a680, x=0x00007f45a99f1ba0, milliseconds=5000) at ConcurrentBoundedQueue.h:161:16
    clickhouse`DB::KeeperDispatcher::requestThread(this=0x00007f475c8fac18) at KeeperDispatcher.cpp:88:33
    clickhouse`DB::KeeperDispatcher::initialize(this=0x00007f45a99f2020)::$_0::operator()() const at KeeperDispatcher.cpp:349:52
    clickhouse`decltype(__f=0x00007f45a99f2020)::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&) at invoke.h:394:23
    clickhouse`decltype(__f=0x00007f45a99f2020, __t=size=0, (null)=__tuple_indices<> @ 0x00007f45a99f1d28) std::__1::__apply_tuple_impl[abi:v15000]<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&, std::__1::tuple<>&>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) at tuple:1789:1
    clickhouse`decltype(__f=0x00007f45a99f2020, __t=size=0) std::__1::apply[abi:v15000]<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&, std::__1::tuple<>&>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&, std::__1::tuple<>&) at tuple:1798:1
    clickhouse`ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(this=0x00007f475c9703c0)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'()::operator()() at ThreadPool.h:245:13
    clickhouse`decltype(__f=0x00007f475c9703c0)::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'()&>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&) at invoke.h:394:23
    clickhouse`void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(__args=0x00007f475c9703c0)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'()&) at invoke.h:479:9
    clickhouse`std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'(), void ()>::operator(this=0x00007f475c9703c0)[abi:v15000]() at function.h:235:12
    clickhouse`void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(__buf=0x00007f45a99f22d8)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) at function.h:716:16
    clickhouse`std::__1::__function::__policy_func<void ()>::operator(this=0x00007f45a99f22d8)[abi:v15000]() const at function.h:848:16
    clickhouse`std::__1::function<void ()>::operator(this=0x00007f45a99f22d8)() const at function.h:1187:12
    clickhouse`ThreadPoolImpl<std::__1::thread>::worker(this=0x00007f475c8229c0, thread_it=std::__1::list<std::__1::thread, std::__1::allocator<std::__1::thread> >::iterator @ 0x00007f45a99f23a8) at ThreadPool.cpp:427:13
    clickhouse`void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(this=0x00007f45cea0c8a8)>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const at ThreadPool.cpp:180:73
    clickhouse`decltype(__f=0x00007f45cea0c8a8)()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) at invoke.h:394:23
    clickhouse`void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(__t=size=2, (null)=__tuple_indices<> @ 0x00007f45a99f2408)>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) at thread:284:5
    clickhouse`void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()> >(__vp=0x00007f45cea0c8a0) at thread:295:5
    libc.so.6`___lldb_unnamed_symbol3481 + 755
    libc.so.6`___lldb_unnamed_symbol3865 + 11

This should fix the problem with
test_keeper_incorrect_config/test.py::test_invalid_configs test.

CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/72b1052f7c2d453308262924e767ab8dc2206933/integration_tests__asan__[1_6].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 17:54:16 +01:00
Azat Khuzhin
55d5a3affa Correctly wait background threads 
There are some places that make it possible to trigger use-after-free
from threads because some global variable had been destroyed before, for
example some logger.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 17:54:16 +01:00
Alexey Milovidov
730463aab8 Merge branch 'master' into musl-7 2023-12-09 17:47:41 +01:00
Alexey Milovidov
7e61bce8e5
Merge pull request #57706 from ClickHouse/deprecate-live-view-2 
Deprecate LIVE VIEW
 2023-12-09 17:43:00 +01:00
Alexey Milovidov
c77183a597
Merge pull request #57480 from azat/dist/async-INSERT-fixes 
Fix possible distributed sends stuck due to "No such file or directory" (during recovering batch from disk)
 2023-12-09 17:11:35 +01:00
Alexey Milovidov
56cd9196b0
Merge pull request #57641 from azat/system.stack_trace-fix 
Avoid hangs of system.stack_trace by correctly prohibit parallel read from it
 2023-12-09 17:07:48 +01:00
Alexey Milovidov
0d1bdff347 Deprecate LIVE VIEW 2023-12-09 16:57:42 +01:00
Alexey Milovidov
0157066fc5 Merge branch 'master' into musl-7 2023-12-09 15:19:38 +01:00
Alexey Milovidov
41f126eba0
Merge pull request #57694 from ClickHouse/revert-57229-revert-57222-update-sentry 
Revert "Revert "Update Sentry""
 2023-12-09 15:17:31 +01:00
Kseniia Sumarokova
69adfffa02
Merge pull request #57588 from ClickHouse/materialized-postgres-escape-publication 
Add a test for #55251
 2023-12-09 12:14:28 +01:00
Bharat Nallan Chakravarthy
64ae040954 fix test 2023-12-08 20:47:31 -08:00
Nikolai Kochetov
5e1dccf1f4 Fixing tests 2023-12-08 22:36:01 +00:00
Alexey Milovidov
89cce44814 Cut useless code 2023-12-08 23:20:20 +01:00
Nikolay Degterinsky
c5e52d260f
Merge pull request #57553 from evillique/fix-local-crash 
Fix crash in clickhouse-local
 2023-12-08 22:57:52 +01:00
Alexey Milovidov
f5451d181c
Merge pull request #57698 from ClickHouse/alexey-milovidov-patch-8 
Update build-cross-osx.md
 2023-12-08 22:35:28 +01:00
Alexey Milovidov
57b740f84b Disable iconv in libxml2 2023-12-08 22:14:24 +01:00
Nikolai Kochetov
a76d9c259f Fixing style 2023-12-08 21:06:18 +00:00
Alexey Milovidov
00aca9a3d4
Merge pull request #57634 from ClickHouse/fix-logs-and-exceptions-messages-formatting-4 
Fix "logs and exception messages formatting", part 4
 2023-12-08 22:02:02 +01:00