thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
107,141 Commits 9,326 Branches 1,783 Tags 2.1 GiB
d66beb9ac4
Commit Graph

56 Commits

Author SHA1 Message Date
Anton Popov
839cd614fb fix memory leak in azure sdk 2023-01-31 01:01:10 +00:00
Zhiguo Zhou
823cb8a8f0 Call std::condition_variable::notify_one w/o lock in ThreadPool 
Calling notify_one function with lock held would let the notified
thread immediately block again, and increase the lock contention.
This commit moves the calls of notify_one function out of critical
sections of ThreadPool, whose lock (ThreadPoolImpl::mutex) is most
extensively contended in some workloads.
 2022-12-21 13:50:03 +08:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Frank Chen
d9a516288c Resolve comments 2022-09-01 11:56:10 +08:00
Frank Chen
83cbdef3c6 Fix context re-initialization for ThreadPool 2022-08-30 12:26:23 +08:00
Frank Chen
92f7ca3616 Move TracingContextOnThread::current() out of class for simplicity 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:23:56 +08:00
Frank Chen
bb00dcc19b Remove using namespace from header 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:20:13 +08:00
Frank Chen
03007606ba Fix format 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-24 17:01:31 +08:00
Frank Chen
cd19366b44 Move classes into DB::OpenTelemetry namespace 2022-08-24 16:41:40 +08:00
Frank Chen
efc6a60a60 Clean code 2022-08-24 15:59:44 +08:00
Frank Chen
035896fa16 Fix build check 2022-08-02 20:18:42 +08:00
Frank Chen
da57a993e4 Fix CI 2022-07-09 13:43:10 +08:00
Frank Chen
93dc109e36 Fix code 2022-07-07 17:44:19 +08:00
Frank Chen
32931857f0 Propagate tracing context across threads 2022-07-07 17:41:43 +08:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to 
Replace where possible by std::lock_guard which is more light-weight.
 2022-06-28 19:19:06 +00:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings 
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
 2022-04-20 10:29:05 +02:00
Alexey Milovidov
ab80b94f45
Update ThreadPool.cpp 2022-04-13 15:18:49 +03:00
Alexey Milovidov
f0fad98a73 Reset thread name in thread pool 2022-04-11 01:03:24 +02:00
alexey-milovidov
a089f9918c
Update ThreadPool.cpp 2022-02-04 03:54:33 +03:00
Alexey Milovidov
78eebd5c7c Fix parallel loading of data parts 2022-02-04 02:29:46 +03:00
Tomáš Hromada
445b0ba7cc
Expose all GlobalThreadPool config params (#31285) 2021-11-12 16:24:47 +03:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask (#25165) 
Introduced MergeTask and MutateTask
 2021-09-17 00:19:58 +03:00
Nikita Mikhaylov
8a0765c60d Review fixes (partially) 2021-09-07 12:45:39 +00:00
Nikita Mikhaylov
74c0b61e53 Better 2021-09-06 18:33:31 +00:00
Nikita Mikhaylov
0249015515 added priority queue 2021-09-06 11:37:51 +00:00
Nikita Mikhaylov
25a879044b Allow allocations to allocate a thread 2021-09-04 17:23:05 +00:00
Nikita Mikhaylov
cc7c221fad Own PriorityQueue + prettifying the code 2021-09-02 21:31:32 +00:00
Alexey Milovidov
ba45bd834f Close #27816 2021-08-19 11:22:57 +03:00
alexey-milovidov
a27808be98
Revert "Do not miss exceptions from the ThreadPool" 2021-08-19 11:17:47 +03:00
Azat Khuzhin
83d871b701 Do not miss exceptions from the ThreadPool 2021-08-08 19:25:43 +03:00
Azat Khuzhin
e3cbd4a5df Improve "Cannot schedule a task" error message 2021-05-31 10:24:05 +03:00
Nikita Mikhaylov
da496cba01
Update ThreadPool.cpp 2021-05-25 15:58:04 +03:00
Nikita Mikhaylov
ba6595cd17 add more comments 2021-05-25 14:59:29 +03:00
Nikita Mikhaylov
343d30002d done 2021-05-25 14:59:29 +03:00
Azat Khuzhin
590a6b9855 Add ThreadPool::finished() 2021-04-28 22:33:29 +03:00
Alexander Tokmakov
73e96250a8 add comments 2021-01-13 15:19:41 +03:00
Alexander Tokmakov
ff6c3c75c2 add protection from unsafe allocations 2021-01-12 18:41:24 +03:00
Anton Popov
57a1642035 fix race in aggregation with combinator distinct 2020-12-22 14:30:29 +03:00
tavplubix
044f143463
Update ThreadPool.cpp 2020-11-06 18:53:33 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell 2020-10-27 12:04:03 +01:00
Azat Khuzhin
789434994a Reset job on failure too in ThreadPool::worker() 
job should be reseted before decrementing scheduled_jobs to
ensure that the Job destroyed before wait() returns.
 2020-09-29 10:56:36 +03:00
Alexander Kuzmenkov
478c7309d4
Merge pull request #14843 from ClickHouse/aku/global-pool 
Exception on double init of global thread pool
 2020-09-22 13:44:39 +03:00
alexey-milovidov
10d3a1d730
Update ThreadPool.cpp 2020-09-21 20:49:15 +03:00
Azat Khuzhin
30f96dc525 Fix possible UAF from jobs in the ThreadPool on shutdown 
ThreadPoolImpl<>::worker signaling that job is done while still storing
std::function<> object, and this can lead to problems on shutdown, since
in in this cast std::function<> can refers to some global/static object
that had been already destroyed (typical example is Logger).

I believe that this is exactly what TSAN reports about (decoded
manually, since llvm-symbolizer does not work in the test env):

<details>

```

- 2020-09-20 17:44:43   Write of size 8 at 0x7b1000008f78 by main thread (mutexes: write M1432):
    operator delete(void*, unsigned long)
    ??:0:0

    Poco::Logger::~Logger()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:39:1
    non-virtual thunk to Poco::Logger::~Logger()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:0:0

    Poco::RefCountedObject::release() const
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/RefCountedObject.h:82:24
    Poco::Logger::shutdown()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:346:16

    Poco::AutoLoggerShutdown::~AutoLoggerShutdown()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:459:4

    cxa_at_exit_wrapper(void*)
    ??:0:0

```

```
- 2020-09-20 17:44:43   Previous atomic read of size 4 at 0x7b1000008f78 by thread T116:

    __tsan_atomic32_load
    ??:0:0

    int std::__1::__cxx_atomic_load<int>(std::__1::__cxx_atomic_base_impl<int> const*, std::__1::memory_order)
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:970:12
    std::__1::__atomic_base<int, false>::load(std::__1::memory_order) const
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:1487:17
    std::__1::__atomic_base<int, false>::operator int() const
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:1491:53
    Poco::Logger::is(int) const
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/Logger.h:1264:9
    MemoryTracker::logPeakMemoryUsage() const
    /build/obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:59:5

    MemoryTracker::~MemoryTracker()
    /build/obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:46:13

    DB::ThreadGroupStatus::~ThreadGroupStatus()
    /build/obj-x86_64-linux-gnu/../src/Common/ThreadStatus.h:51:7
    std::__1::__shared_ptr_emplace<DB::ThreadGroupStatus, std::__1::allocator<DB::ThreadGroupStatus> >::__on_zero_shared()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3602:23

    std::__1::__shared_count::__release_shared()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3440:9
    std::__1::__shared_weak_count::__release_shared()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3482:27
    std::__1::shared_ptr<DB::ThreadGroupStatus>::~shared_ptr()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:4207:19
    DB::PipelineExecutor::executeImpl(unsigned long)::$_4::~$_4()
    /build/obj-x86_64-linux-gnu/../src/Processors/Executors/PipelineExecutor.cpp:720:34
    ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'()::~()
    /build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.h:161:54
    std::__1::__compressed_pair_elem<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), 0, false>::~__compressed_pair_elem()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2188:8
    std::__1::__function::__alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), std::__1::allocator<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::Pip>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1559:38
    std::__1::__function::__func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), std::__1::allocator<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineE>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1706:10

    std::__1::__function::__value_func<void ()>::~__value_func()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1828:19
    std::__1::function<void ()>::~function()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2460:43
    ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>)
    /build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:268:5

    void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const
    /build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:116:73
    decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleI>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3519:1
    void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'(>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:273:5
    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned lon>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:284:5

    __tsan_thread_start_func
    ??:0:0
```

```
- 2020-09-20 17:44:43   Mutex M1432 (0x0000181213a8) created at:
    pthread_mutex_init
    ??:0:0

    Poco::MutexImpl::MutexImpl()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Mutex_POSIX.cpp:64:6

    Poco::Mutex::Mutex()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Mutex.cpp:34:8

    __cxx_global_var_init
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:28:15
    _GLOBAL__sub_I_Logger.cpp
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:0:0

    __libc_csu_init
    ??:0:0
```

</details>
 2020-09-20 23:46:28 +03:00
Alexander Kuzmenkov
bcea99f2e5
Update ThreadPool.cpp 2020-09-16 11:59:58 +03:00
Alexander Kuzmenkov
5afb19faf1
Update ThreadPool.cpp 2020-09-15 19:58:09 +03:00
Alexander Kuzmenkov
a792850ecd
Update ThreadPool.cpp 2020-09-15 18:05:42 +03:00
Alexander Kuzmenkov
d943bac1a4 Exception on double init of global thread pool 2020-09-15 13:30:16 +03:00
Vitaly Baranov
668653600c Use SettingMaxThreads only in Settings, call getNumberOfPhysicalCPUCores() instead of SettingMaxThreads::getAuto(). 2020-07-31 19:11:27 +03:00
bharatnc
9be7ab0341 fix style check - removed unused LOGICAL_ERROR 2020-06-23 13:06:22 -07:00