ClickHouse

mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-09-30 21:50:49 +00:00

ClickHouse® is a real-time analytics DBMS

ai analytics big-data clickhouse cpp dbms distributed-database hacktoberfest mpp olap rust sql

Go to file

Azat Khuzhin 30f96dc525 Fix possible UAF from jobs in the ThreadPool on shutdown ThreadPoolImpl<>::worker signaling that job is done while still storing std::function<> object, and this can lead to problems on shutdown, since in in this cast std::function<> can refers to some global/static object that had been already destroyed (typical example is Logger). I believe that this is exactly what TSAN reports about (decoded manually, since llvm-symbolizer does not work in the test env): <details> ``` - 2020-09-20 17:44:43 Write of size 8 at 0x7b1000008f78 by main thread (mutexes: write M1432): operator delete(void, unsigned long) ??:0:0 Poco::Logger::~Logger() /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:39:1 non-virtual thunk to Poco::Logger::~Logger() /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:0:0 Poco::RefCountedObject::release() const /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/RefCountedObject.h:82:24 Poco::Logger::shutdown() /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:346:16 Poco::AutoLoggerShutdown::~AutoLoggerShutdown() /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:459:4 cxa_at_exit_wrapper(void) ??:0:0 ``` ``` - 2020-09-20 17:44:43 Previous atomic read of size 4 at 0x7b1000008f78 by thread T116: __tsan_atomic32_load ??:0:0 int std::__1::__cxx_atomic_load<int>(std::__1::__cxx_atomic_base_impl<int> const, std::__1::memory_order) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:970:12 std::__1::__atomic_base<int, false>::load(std::__1::memory_order) const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:1487:17 std::__1::__atomic_base<int, false>::operator int() const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:1491:53 Poco::Logger::is(int) const /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/Logger.h:1264:9 MemoryTracker::logPeakMemoryUsage() const /build/obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:59:5 MemoryTracker::~MemoryTracker() /build/obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:46:13 DB::ThreadGroupStatus::~ThreadGroupStatus() /build/obj-x86_64-linux-gnu/../src/Common/ThreadStatus.h:51:7 std::__1::__shared_ptr_emplace<DB::ThreadGroupStatus, std::__1::allocator<DB::ThreadGroupStatus> >::__on_zero_shared() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3602:23 std::__1::__shared_count::__release_shared() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3440:9 std::__1::__shared_weak_count::__release_shared() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3482:27 std::__1::shared_ptr<DB::ThreadGroupStatus>::~shared_ptr() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:4207:19 DB::PipelineExecutor::executeImpl(unsigned long)::$_4::~$_4() /build/obj-x86_64-linux-gnu/../src/Processors/Executors/PipelineExecutor.cpp:720:34 ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'()::~() /build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.h:161:54 std::__1::__compressed_pair_elem<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), 0, false>::~__compressed_pair_elem() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2188:8 std::__1::__function::__alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), std::__1::allocator<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::Pip> /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1559:38 std::__1::__function::__func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), std::__1::allocator<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineE> /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1706:10 std::__1::__function::__value_func<void ()>::~__value_func() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1828:19 std::__1::function<void ()>::~function() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2460:43 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void>) /build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:268:5 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const /build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:116:73 decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleI> /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3519:1 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'(> /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:273:5 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned lon> /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:284:5 __tsan_thread_start_func ??:0:0 ``` ``` - 2020-09-20 17:44:43 Mutex M1432 (0x0000181213a8) created at: pthread_mutex_init ??:0:0 Poco::MutexImpl::MutexImpl() /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Mutex_POSIX.cpp:64:6 Poco::Mutex::Mutex() /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Mutex.cpp:34:8 __cxx_global_var_init /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:28:15 _GLOBAL__sub_I_Logger.cpp /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:0:0 __libc_csu_init ??:0:0 ``` </details>		2020-09-20 23:46:28 +03:00
.github	Update PULL_REQUEST_TEMPLATE.md	2020-08-25 19:58:02 +03:00
base	Fix terribly wrong code	2020-09-19 02:05:13 +03:00
benchmark	Fix typos	2020-08-16 15:11:02 +03:00
cmake	Remove OpenCL	2020-09-19 17:20:00 +03:00
contrib	Update jemalloc to include patch for percpu arena	2020-09-17 21:31:39 +03:00
debian	Drop python from build dependencies	2020-09-19 00:20:54 +03:00
docker	Remove obsolete code from performance test	2020-09-19 19:27:37 +03:00
docs	Merge pull request #14838 from BayoNet/docs/CLICKHOUSEDOCS-758-fixing-links	2020-09-20 23:08:55 +03:00
programs	Merge pull request #15008 from ClickHouse/clang-tidy-11-without-enabling	2020-09-20 02:00:54 +03:00
src	Fix possible UAF from jobs in the ThreadPool on shutdown	2020-09-20 23:46:28 +03:00
tests	Merge pull request #15033 from amosbird/fixstringkey	2020-09-20 19:51:24 +03:00
utils	Fix terribly wrong code	2020-09-19 02:05:13 +03:00
website	Fix bug on the website	2020-09-04 00:05:55 +03:00
.arcignore	Added .arcignore	2020-05-21 09:17:03 +03:00
.clang-format	Fixed wrong code around Memory Profiler	2020-03-03 03:24:44 +03:00
.clang-tidy	Enable clang-tidy for programs and utils	2020-05-18 04:19:50 +03:00
.editorconfig	Changed tabs to spaces in editor configs and in style guide [#CLICKHOUSE-3].	2017-04-01 11:35:09 +03:00
.gitattributes	Decreased warning threshold [#MOBMET-3248]	2016-06-07 11:23:15 +03:00
.gitignore	Updated gitignore-files.	2020-08-06 18:05:32 -07:00
.gitmodules	Update jemalloc to include patch for percpu arena	2020-09-17 21:31:39 +03:00
.potato.yml	Update comments in .potato.yml	2019-10-20 12:50:33 +08:00
.vimrc	Changed tabs to spaces in editor configs and in style guide [#CLICKHOUSE-3].	2017-04-01 11:35:09 +03:00
AUTHORS	Update AUTHORS	2020-01-23 17:36:05 +03:00
CHANGELOG.md	Update CHANGELOG.md	2020-09-14 21:57:50 +03:00
CMakeLists.txt	Remove OpenCL	2020-09-19 17:20:00 +03:00
CODE_OF_CONDUCT.md	Add minimal code of conduct #9676	2020-03-16 12:44:28 +03:00
CONTRIBUTING.md	Update CONTRIBUTING.md	2020-01-27 21:03:30 +03:00
docker-compose.yml	Updated docker-compose.yml #1025	2017-07-26 20:05:32 +03:00
format_sources	allow several <graphite> targets (#603 )	2017-03-21 23:08:09 +04:00
LICENSE	Update LICENSE	2020-01-23 17:38:30 +03:00
PreLoad.cmake	Disable ninja for CLion	2019-09-11 10:23:20 +03:00
README.md	Update event links	2020-09-14 18:27:15 +03:00
release	Added auto-generated machine-readable file with list of stable versions	2020-05-02 21:53:55 +03:00
SECURITY.md	[security docs] actually, only admins can create advisories	2020-07-10 16:51:37 +03:00
uncrustify.cfg	Better .clang-format and uncrustify.cfg	2018-11-29 15:45:34 +03:00
ya.make	Changes required for auto-sync with Arcadia	2020-04-16 15:31:57 +03:00

README.md

ClickHouse is an open-source column-oriented database management system that allows generating analytical data reports in real time.

Useful Links

Official website has quick high-level overview of ClickHouse on main page.
Tutorial shows how to set up and query small ClickHouse cluster.
Documentation provides more in-depth information.
YouTube channel has a lot of content about ClickHouse in video format.
Slack and Telegram allow to chat with ClickHouse users in real-time.
Blog contains various ClickHouse-related articles, as well as announcements and reports about events.
Code Browser with syntax highlight and navigation.
Yandex.Messenger channel shares announcements and useful links in Russian.
Contacts can help to get your questions answered if there are any.
You can also fill this form to meet Yandex ClickHouse team in person.

Upcoming Events

eBay migrating from Druid on September 23, 2020.
ClickHouse for Edge Analytics on September 29, 2020.