Commit Graph

4083 Commits

Author SHA1 Message Date
Nikita Taranov
453ee52f86
Merge pull request #67235 from ClickHouse/fix_oss_fuzz_build
Fix oss-fuzz build
2024-08-01 10:50:12 +00:00
Robert Schulze
4b40e948c7
Merge pull request #66475 from rschu1ze/bump-rocksdb
Bump RocksDB from v6.29.5 to v7.10.2
2024-07-31 09:14:42 +00:00
Robert Schulze
72d6467fd2
Bump Azure to 1.13 2024-07-30 10:43:21 +00:00
Antonio Andelic
2ea4bfe04d
Merge pull request #67098 from ClickHouse/add-numactl
Try calculating available memory if ClickHouse is bound to subset of NUMA nodes
2024-07-30 08:30:58 +00:00
Robert Schulze
07507e0cec
Merge remote-tracking branch 'ClickHouse/master' into bump-rocksdb 2024-07-29 13:41:55 +00:00
Robert Schulze
294fa1dea6
Merge remote-tracking branch 'ClickHouse/master' into bump-icu 2024-07-29 10:32:37 +00:00
Nikita Taranov
109bd52296 impl 2024-07-26 23:05:56 +01:00
Antonio Andelic
2519f9ed42 Only support archs 2024-07-26 12:08:16 +02:00
Antonio Andelic
287cce7d21 Fixes 2024-07-26 09:20:15 +02:00
Antonio Andelic
6557387148 Try calcualting memory with numactl if membind used 2024-07-25 11:49:02 +02:00
Michael Kolupaev
1b2fd51e09
Merge pull request #66977 from ClickHouse/uwo
Apply libunwind fix
2024-07-25 03:40:19 +00:00
Michael Kolupaev
36e59a1b70 Final commit hash 2024-07-25 01:58:27 +00:00
Alexey Milovidov
1914b37058
Merge pull request #65521 from JackyWoo/add_statistics_cmsketch
Add count-min sketches as statistics type
2024-07-24 00:46:23 +00:00
Michael Kolupaev
1cbbbd1077 Apply libunwind fix 2024-07-23 17:35:12 +00:00
Antonio Andelic
1be54641c1 Revert libunwind patch 2024-07-22 11:03:19 +02:00
Robert Schulze
2a68867a20
Merge remote-tracking branch 'ClickHouse/master' into bump-icu 2024-07-22 08:19:07 +00:00
Robert Schulze
25a22d6201
Merge remote-tracking branch 'ClickHouse/master' into bump-icu 2024-07-19 12:43:11 +00:00
Yong Wang
3ef2cbdec5 openssl-dynamic-build-ppc64le : fix openssl dynamic build for ppc64le 2024-07-18 11:21:52 -07:00
Alexey Milovidov
ec87bfc956 Merge branch 'update-tidy-for-clang-19' into clickhouse-fix-msan-report-in-epoll 2024-07-17 05:49:49 +02:00
Alexey Milovidov
4d88abfcfa Merge branch 'master' into uwu 2024-07-16 06:04:58 +02:00
JackyWoo
245359e536 Merge branch 'master' into add_statistics_cmsketch 2024-07-16 09:45:31 +08:00
Alexey Milovidov
6d617e4c4d Merge branch 'master' of github.com:ClickHouse/ClickHouse into clickhouse-fix-msan-report-in-epoll 2024-07-15 18:30:30 +02:00
Robert Schulze
2083c65005
Merge remote-tracking branch 'ClickHouse/master' into bump-icu 2024-07-15 15:31:07 +00:00
Robert Schulze
e7becedb03
Merge remote-tracking branch 'ClickHouse/master' into bump-rocksdb 2024-07-15 06:54:16 +00:00
Robert Schulze
2132ce52e0
Bump ICU to 75 2024-07-15 06:49:02 +00:00
Antonio Andelic
7d62a19f4c
Merge pull request #66439 from azat/jemalloc-clock-assertion
[RFC] Fix jemalloc assertion due to non-monotonic CLOCK_MONOTONIC_COARSE
2024-07-15 06:45:29 +00:00
Alexey Milovidov
41327260ba Fix MSan report in GRPC 2024-07-15 03:47:35 +02:00
Alexey Milovidov
cae3f63545 Merge branch 'master' into uwu 2024-07-15 01:22:38 +02:00
Robert Schulze
5b6956ea23
Bump rocksdb to v7.10.2 2024-07-14 12:38:28 +00:00
Robert Schulze
b5cb264b01
Bump ICU to 71 2024-07-14 12:17:25 +00:00
Robert Schulze
ad86e3060a
OpenSSL: Follow-up to #66064
More commits were added to the upstream fix after #66064 has been merged
in ClickHouse. This PR adds these commits to keep things in-sync.
2024-07-14 11:48:13 +00:00
Azat Khuzhin
669ce7cd40 Enable jemalloc heap profiler by default for debug builds as well
Since the issue with jemalloc assertion is clear, let's revert that
workaround

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-14 08:21:13 +02:00
Azat Khuzhin
22611781b0 Fix jemalloc assertion due to non-monotonic CLOCK_MONOTONIC_COARSE
Recently one tricky assertion of jemalloc had been discovered [1]:

    Failed assertion: "nstime_compare(&decay->epoch, new_time) <= 0"

  [1]: https://github.com/ClickHouse/ClickHouse/issues/66193

And as it turns out it is really possible for CLOCK_MONOTONIC_COARSE to
go backwards, in a nutshell it can be done with ADJ_FREQUENCY, you can
find example here [2]. And I can't trigger this issue for non-coarse
clocks.

  [2]: https://gist.github.com/azat/7ea7f50ed75591b1af2d675a240ea94c?permalink_comment_id=5119222#gistcomment-5119222

But, jemalloc do not call clock_gettime() that frequently (I've verified
it), so it can use non-coarse version - CLOCK_MONOTONIC

I've also measured the latency of CLOCK_MONOTONIC and
CLOCK_MONOTONIC_COARSE, and it is 20ns vs 4ns per call [3], so make this
change affect performance you need really frequently calls of
clock_gettime.

  [3]: https://gist.github.com/azat/622fa1f9a5d8e7d546ee9d294501961d?permalink_comment_id=5119245#gistcomment-5119245

Interesting, that this bug started to appears only after jemalloc heap
profiler had been enabled by default [4], no clue why (I would believe
more in a more frequent calls to clock_adjtime(ADJ_FREQUENCY), but I
can't verify this)

  [4]: https://github.com/ClickHouse/ClickHouse/pull/65702

To be continued...

Fixes: https://github.com/ClickHouse/ClickHouse/issues/66193
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-14 08:21:13 +02:00
Robert Schulze
264d7d760f
Bump rocksdb to 7.0.4 2024-07-13 16:43:39 +00:00
Nikita Mikhaylov
8182ec8762
Merge pull request #66292 from ClickHouse/bump-azure
Bump `azure-sdk-for-cpp`
2024-07-11 12:58:50 +00:00
Nikita Mikhaylov
9c10d4f402
Merge pull request #66291 from ClickHouse/bump-pocketfft
Bump `pocketfft`
2024-07-11 11:50:21 +00:00
Robert Schulze
1ebe1da7b9
Merge pull request #66216 from rschu1ze/bump-rocksdb2
Bump RocksDB
2024-07-11 09:52:59 +00:00
Azat Khuzhin
e6e8b065bc Fix possible deadlock for jemalloc with enabled profiler
ClickHouse uses JEMALLOC_PROF_LIBGCC - _Unwind_Backtrace() for jemalloc,
which calls _Unwind_Backtrace() during bootstrap of jemalloc, and if,
dlsym() has allocations it will lead to deadlock:

    0  __lll_lock_wait (futex=futex@entry=0x16516410 <init_lock+64>, private=0) at lowlevellock.c:52
    1  0x00007ffff7f9d0a3 in __GI___pthread_mutex_lock (mutex=0x16516410 <init_lock+64>) at ../nptl/pthread_mutex_lock.c:80
    2  0x000000000ba6ec23 in pthread_mutex_lock (arg=arg@entry=0x16516410 <init_lock+64>) at ./build/RelWithDebInfo/./src/Common/ThreadFuzzer.cpp:445
    3  0x0000000014215fd4 in malloc_mutex_lock_final (mutex=0x165163d0 <init_lock>) at ./contrib/jemalloc/include/jemalloc/internal/mutex.h:151
    4  malloc_mutex_lock_slow (mutex=0x165163d0 <init_lock>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/mutex.c:90
    5  0x00000000141defc6 in malloc_mutex_lock (tsdn=0x0, mutex=<optimized out>) at ./contrib/jemalloc/include/jemalloc/internal/mutex.h:217
    6  malloc_init_hard () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2118
    7  0x00000000141d05db in malloc_init () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:298
    8  imalloc_init_check (sopts=<optimized out>, dopts=<optimized out>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2658
    9  imalloc (sopts=<optimized out>, dopts=<optimized out>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2689
    10 calloc (num=num@entry=1, size=size@entry=32) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2852
    11 0x00007ffff7c42c05 in _dlerror_run (operate=operate@entry=0x7ffff7c42490 <dlsym_doit>, args=args@entry=0x7fffffffda40) at dlerror.c:148
    12 0x00007ffff7c42525 in __dlsym (handle=<optimized out>, name=0xe7f568 "dl_iterate_phdr") at dlsym.c:70
    13 0x000000001408e1a0 in (anonymous namespace)::getOriginalDLIteratePHDR () at ./build/RelWithDebInfo/./base/base/phdr_cache.cpp:44
    14 dl_iterate_phdr (callback=0x16287a60 <libunwind::findUnwindSectionsByPhdr(dl_phdr_info*, unsigned long, void*)>, data=0x7fffffffdb58) at ./build/RelWithDebInfo/./base/base/phdr_cache.cpp:64
    15 0x0000000016288468 in libunwind::LocalAddressSpace::findUnwindSections (this=<optimized out>, targetAddr=371765377, info=...) at ./contrib/libunwind/src/AddressSpace.hpp:605
    16 libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister (this=this@entry=0x7fffffffde70, isReturnAddress=false) at ./contrib/libunwind/src/UnwindCursor.hpp:2553
    17 0x0000000016287617 in __unw_init_local (cursor=cursor@entry=0x7fffffffde70, context=context@entry=0x7fffffffddc8) at ./build/RelWithDebInfo/./contrib/libunwind/src/libunwind.cpp:91
    18 0x000000001628b094 in _Unwind_Backtrace (callback=0x14226f40 <prof_unwind_init_callback>, ref=0x0) at ./build/RelWithDebInfo/./contrib/libunwind/src/UnwindLevel1-gcc-ext.c:106
    19 0x000000001421a7e0 in prof_boot2 (tsd=tsd@entry=0x7ffff7c357c0, base=<optimized out>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/prof.c:693
    20 0x00000000141df3e7 in malloc_init_hard () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2151
    21 0x00000000141d96f1 in malloc_init () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:298
    22 nallocx (size=32, flags=0) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:4011
    23 0x000000000b9fdd9b in _ZN6Memory23getActualAllocationSizeITpTkNSt3__17same_asISt11align_val_tEEJEQsr2DBE16OptionalArgumentIDpT_EEEmmS5_ (size=32) at ./src/Common/memory.h:197
    24 _ZN6Memory11trackMemoryITpTkNSt3__17same_asISt11align_val_tEEJEQsr2DBE16OptionalArgumentIDpT_EEEmmR15AllocationTraceS5_ (size=32, trace=...) at ./src/Common/memory.h:208
    25 operator new (size=size@entry=32) at ./build/RelWithDebInfo/./src/Common/new_delete.cpp:53
    26 0x000000001540e0a6 in google::protobuf::internal::ShutdownData::get () at ./build/RelWithDebInfo/./contrib/google-protobuf/src/google/protobuf/message_lite.cc:679
    27 google::protobuf::internal::OnShutdownRun (f=0x15407420 <google::protobuf::internal::DestroyString(void const*)>, arg=0x167a6368 <google::protobuf::internal::fixed_address_empty_string>) at ./build/RelWithDebInfo/./contrib/google-protobuf/src/google/protobuf/message_lite.cc:697
    28 0x0000000015407560 in global constructors keyed to 000101 () at ./contrib/google-protobuf/src/google/protobuf/generated_message_util.h:180
    29 0x000000001629101d in __libc_csu_init ()
    30 0x00007ffff7dba010 in __libc_start_main (main=0x66e3b00 <main(int, char**)>, argc=4, argv=0x7fffffffe258, init=0x16290fd0 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe248) at ../csu/libc-start.c:264
    31 0x00000000066e302e in _start ()

And this is indeed what happens for glibc prio 2.34, since only in [2]
the allocation had been removed

  [2]: https://sourceware.org/git/?p=glibc.git;a=commit;h=fada9018199c21c469ff0e731ef75c6020074ac9

But, jemalloc provides also JEMALLOC_PROF_LIBUNWIND, yes it is not llvm
libunwind, but gnu one, but ClickHouse version of llvm libunwind
provides unw_backtrace(), so we can use it, and it will not be called
during jemalloc bootstrap, so deadlock should not be possible.

Funny thing that for this deadlock the workaround is to enable
background_thread, because a thread created for it will initialize TLS for
__dlerror().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-10 16:05:36 +02:00
Nikita Mikhaylov
2b091983e8 Bump Azure to ea3e19a7be 2024-07-09 19:59:49 +00:00
Nikita Mikhaylov
0d54151cb8 Make the pocketfft to point to the upstream/master branch 2024-07-09 19:55:37 +00:00
Robert Schulze
55468caeae
Fix ARM build 2024-07-09 16:38:22 +00:00
Vitaly Baranov
32b9f7940f
Merge pull request #66191 from vitlibar/grpc-avoid-using-harmful-function-rand
Avoid using harmful function rand() in grpc
2024-07-09 13:42:20 +00:00
Robert Schulze
3f1c0b9966
Merge remote-tracking branch 'ClickHouse/master' into add_statistics_cmsketch 2024-07-09 10:48:19 +00:00
Kruglov Pavel
52b3d1a244
Merge pull request #66130 from slvrtrn/update-avro-submodule-with-array-block-size-fix
Update AVRO submodule with the array block size fix
2024-07-09 07:58:04 +00:00
Nikita Taranov
a9546fe551
Merge pull request #66147 from ClickHouse/fix_mac_build
Playing minesweeper with build system
2024-07-08 22:05:17 +00:00
Robert Schulze
bd42a096b7
Bump rocksdb to v6.29.5 2024-07-08 16:52:36 +00:00
Robert Schulze
1bfafa42be
Bump rocksdb to v6.28.2 2024-07-08 14:47:35 +00:00
Robert Schulze
9509802866
Bump rocksdb to v6.27.3 2024-07-08 13:57:03 +00:00
Robert Schulze
31f9bed442
Bump rocksdb to 6.26.1 2024-07-08 13:49:05 +00:00
Robert Schulze
adcee80b2d
Bump rocksdb to v6.25.3 2024-07-08 13:41:34 +00:00