Commit Graph

761 Commits

Author SHA1 Message Date
bharatnc
24ea6b2380 TraceCollector - time s&us using same timespec 2020-09-25 10:39:37 -07:00
bharatnc
b0d4fe5114 TraceLog - add field event_time_microseconds 2020-09-25 10:39:37 -07:00
tavplubix
b96157e32b
Merge branch 'master' into enable_atomic_database_by_default 2020-09-25 13:42:39 +03:00
alexey-milovidov
13529265c4
Revert "Avoid deadlocks in Log/TinyLog" 2020-09-25 02:28:57 +03:00
Alexander Tokmakov
fb826557b7 Merge branch 'master' into enable_atomic_database_by_default 2020-09-25 02:04:01 +03:00
Alexey Milovidov
dce6a436f3 Fix strange code 2020-09-24 19:35:17 +03:00
Vitaly Baranov
0605808ed1
Merge pull request #14492 from vitlibar/use-previous-users-xml-if-failed-to-reload
Use previous users.xml if failed to reload
2020-09-24 16:25:19 +03:00
Alexander Kuzmenkov
813a876e93
Merge pull request #14962 from ClickHouse/log-avoid-deadlock
Avoid deadlocks in Log/TinyLog
2020-09-24 13:54:11 +03:00
Alexander Tokmakov
f0f91f84af Merge branch 'master' into enable_atomic_database_by_default 2020-09-23 21:50:15 +03:00
Alexander Tokmakov
a8266a3d3c fix DDDL with cross-replication and Atomic 2020-09-23 21:28:59 +03:00
Alexander Kuzmenkov
679c0988a5 Merge remote-tracking branch 'origin/master' into HEAD 2020-09-22 16:25:52 +03:00
Alexander Kuzmenkov
dde4cf70e1 Merge remote-tracking branch 'origin/master' into HEAD 2020-09-22 14:03:59 +03:00
Alexander Kuzmenkov
478c7309d4
Merge pull request #14843 from ClickHouse/aku/global-pool
Exception on double init of global thread pool
2020-09-22 13:44:39 +03:00
Alexander Kuzmenkov
c0241b617a Merge remote-tracking branch 'origin/master' into HEAD 2020-09-22 13:14:31 +03:00
Alexey Milovidov
1e65105b06 Merge branch 'master' into log-avoid-deadlock 2020-09-21 21:09:06 +03:00
alexey-milovidov
e29c4c3cc4
Merge pull request #15024 from ClickHouse/fix_renameat2_einval_on_macos
Fix EINVAL in renameat2 on MacOS
2020-09-21 21:08:17 +03:00
alexey-milovidov
10d3a1d730
Update ThreadPool.cpp 2020-09-21 20:49:15 +03:00
tavplubix
fd2cc343e6
Update renameat2.cpp 2020-09-21 15:37:34 +03:00
tavplubix
450c1a5709
Update renameat2.cpp 2020-09-21 13:19:02 +03:00
Alexander Tokmakov
ef85ce90c3 fix build 2020-09-21 13:17:33 +03:00
Azat Khuzhin
30f96dc525 Fix possible UAF from jobs in the ThreadPool on shutdown
ThreadPoolImpl<>::worker signaling that job is done while still storing
std::function<> object, and this can lead to problems on shutdown, since
in in this cast std::function<> can refers to some global/static object
that had been already destroyed (typical example is Logger).

I believe that this is exactly what TSAN reports about (decoded
manually, since llvm-symbolizer does not work in the test env):

<details>

```

- 2020-09-20 17:44:43   Write of size 8 at 0x7b1000008f78 by main thread (mutexes: write M1432):
    operator delete(void*, unsigned long)
    ??:0:0

    Poco::Logger::~Logger()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:39:1
    non-virtual thunk to Poco::Logger::~Logger()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:0:0

    Poco::RefCountedObject::release() const
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/RefCountedObject.h:82:24
    Poco::Logger::shutdown()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:346:16

    Poco::AutoLoggerShutdown::~AutoLoggerShutdown()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:459:4

    cxa_at_exit_wrapper(void*)
    ??:0:0

```

```
- 2020-09-20 17:44:43   Previous atomic read of size 4 at 0x7b1000008f78 by thread T116:

    __tsan_atomic32_load
    ??:0:0

    int std::__1::__cxx_atomic_load<int>(std::__1::__cxx_atomic_base_impl<int> const*, std::__1::memory_order)
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:970:12
    std::__1::__atomic_base<int, false>::load(std::__1::memory_order) const
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:1487:17
    std::__1::__atomic_base<int, false>::operator int() const
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:1491:53
    Poco::Logger::is(int) const
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/Logger.h:1264:9
    MemoryTracker::logPeakMemoryUsage() const
    /build/obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:59:5

    MemoryTracker::~MemoryTracker()
    /build/obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:46:13

    DB::ThreadGroupStatus::~ThreadGroupStatus()
    /build/obj-x86_64-linux-gnu/../src/Common/ThreadStatus.h:51:7
    std::__1::__shared_ptr_emplace<DB::ThreadGroupStatus, std::__1::allocator<DB::ThreadGroupStatus> >::__on_zero_shared()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3602:23

    std::__1::__shared_count::__release_shared()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3440:9
    std::__1::__shared_weak_count::__release_shared()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3482:27
    std::__1::shared_ptr<DB::ThreadGroupStatus>::~shared_ptr()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:4207:19
    DB::PipelineExecutor::executeImpl(unsigned long)::$_4::~$_4()
    /build/obj-x86_64-linux-gnu/../src/Processors/Executors/PipelineExecutor.cpp:720:34
    ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'()::~()
    /build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.h:161:54
    std::__1::__compressed_pair_elem<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), 0, false>::~__compressed_pair_elem()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2188:8
    std::__1::__function::__alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), std::__1::allocator<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::Pip>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1559:38
    std::__1::__function::__func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_4>(DB::PipelineExecutor::executeImpl(unsigned long)::$_4&&)::'lambda'(), std::__1::allocator<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineE>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1706:10

    std::__1::__function::__value_func<void ()>::~__value_func()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1828:19
    std::__1::function<void ()>::~function()
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2460:43
    ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>)
    /build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:268:5

    void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const
    /build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:116:73
    decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(fp0)...)) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleI>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3519:1
    void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'(>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:273:5
    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned lon>
    /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:284:5

    __tsan_thread_start_func
    ??:0:0
```

```
- 2020-09-20 17:44:43   Mutex M1432 (0x0000181213a8) created at:
    pthread_mutex_init
    ??:0:0

    Poco::MutexImpl::MutexImpl()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Mutex_POSIX.cpp:64:6

    Poco::Mutex::Mutex()
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Mutex.cpp:34:8

    __cxx_global_var_init
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:28:15
    _GLOBAL__sub_I_Logger.cpp
    /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Logger.cpp:0:0

    __libc_csu_init
    ??:0:0
```

</details>
2020-09-20 23:46:28 +03:00
alexey-milovidov
7b35a6a59a
Merge pull request #15028 from ClickHouse/fix-bar
Fix buffer overflow in "bar" function
2020-09-20 19:04:15 +03:00
alexey-milovidov
1474b80ecc
Merge pull request #15008 from ClickHouse/clang-tidy-11-without-enabling
Fix terribly wrong code
2020-09-20 02:00:54 +03:00
Alexey Milovidov
dae3b40a55 Fix buffer overflow in "bar" function 2020-09-19 22:15:16 +03:00
Alexey Milovidov
059646743b Remove OpenCL 2020-09-19 17:20:00 +03:00
Alexander Tokmakov
46aa307d0a fix EINVAL in renameat2 on MacOS 2020-09-19 17:19:06 +03:00
Vitaly Baranov
33efb36714 Use previous users.xml if failed to reload. 2020-09-19 16:21:12 +03:00
Alexey Milovidov
4d06bc62b1 Allow concurrent reads while writing to TinyLog 2020-09-19 14:02:11 +03:00
alexey-milovidov
c461d782e4
Merge pull request #14929 from ClickHouse/mlock-code-segment
Only mlock code segment
2020-09-19 13:46:49 +03:00
Alexey Milovidov
ef5363b87f Fix terribly wrong code 2020-09-19 02:05:13 +03:00
Alexey Milovidov
7d0f3db639 Fix annoying "Arcadia" 2020-09-18 23:19:41 +03:00
Alexey Milovidov
8a8e4f8a41 Attempt to speed-up clang build 2020-09-18 21:58:57 +03:00
Alexey Milovidov
6e6909f47d Fix "Arcadia" 2020-09-18 21:54:54 +03:00
Alexey Milovidov
72c2201287 Fix "Arcadia" 2020-09-18 17:39:33 +03:00
Alexander Kuzmenkov
a9b2bff8ba Merge remote-tracking branch 'origin/master' into HEAD 2020-09-18 13:43:29 +03:00
Alexey Milovidov
1752bbc1ed Fix "Arcadia" build 2020-09-18 01:29:15 +03:00
Alexey Milovidov
a783fe110f Fix build 2020-09-17 21:15:19 +03:00
Alexey Milovidov
eb523f1a7c Merge branch 'master' into mlock-code-segment 2020-09-17 21:15:09 +03:00
Alexey Milovidov
20d07ed405 Update comment 2020-09-17 21:04:11 +03:00
Alexander Kuzmenkov
fb64cf210a straighten the protocol version 2020-09-17 17:37:29 +03:00
Vitaly Baranov
3356d75b23
Merge pull request #13156 from azat/cluster-secure
Secure inter-cluster query execution (with initial_user as current query user) [v3]
2020-09-17 17:11:00 +03:00
alexey-milovidov
496df5b3e9
Merge pull request #14678 from sundy-li/patch-2
dynamic zookeeper config when session expired
2020-09-17 17:05:22 +03:00
Alexey Milovidov
068e8576b7 Corrections 2020-09-17 15:53:52 +03:00
Alexey Milovidov
8bce20076c Only mlock code segment 2020-09-17 15:39:37 +03:00
Nikolai Kochetov
959d755a1b
Update Visitor.h 2020-09-17 15:08:55 +03:00
Nikolai Kochetov
225b85624e Merge branch 'master' into inline-gather-utils 2020-09-17 11:17:32 +03:00
sundy-li
544b2cb20d add configChanged method for zookeeper
fix logic error && skip reload testkeeper
2020-09-17 13:33:45 +08:00
Alexander Kuzmenkov
bcea99f2e5
Update ThreadPool.cpp 2020-09-16 11:59:58 +03:00
Alexander Kuzmenkov
5afb19faf1
Update ThreadPool.cpp 2020-09-15 19:58:09 +03:00
Alexander Kuzmenkov
a792850ecd
Update ThreadPool.cpp 2020-09-15 18:05:42 +03:00