Commit Graph

4041 Commits

Author SHA1 Message Date
Antonio Andelic
7d62a19f4c
Merge pull request #66439 from azat/jemalloc-clock-assertion
[RFC] Fix jemalloc assertion due to non-monotonic CLOCK_MONOTONIC_COARSE
2024-07-15 06:45:29 +00:00
Robert Schulze
ad86e3060a
OpenSSL: Follow-up to #66064
More commits were added to the upstream fix after #66064 has been merged
in ClickHouse. This PR adds these commits to keep things in-sync.
2024-07-14 11:48:13 +00:00
Azat Khuzhin
669ce7cd40 Enable jemalloc heap profiler by default for debug builds as well
Since the issue with jemalloc assertion is clear, let's revert that
workaround

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-14 08:21:13 +02:00
Azat Khuzhin
22611781b0 Fix jemalloc assertion due to non-monotonic CLOCK_MONOTONIC_COARSE
Recently one tricky assertion of jemalloc had been discovered [1]:

    Failed assertion: "nstime_compare(&decay->epoch, new_time) <= 0"

  [1]: https://github.com/ClickHouse/ClickHouse/issues/66193

And as it turns out it is really possible for CLOCK_MONOTONIC_COARSE to
go backwards, in a nutshell it can be done with ADJ_FREQUENCY, you can
find example here [2]. And I can't trigger this issue for non-coarse
clocks.

  [2]: https://gist.github.com/azat/7ea7f50ed75591b1af2d675a240ea94c?permalink_comment_id=5119222#gistcomment-5119222

But, jemalloc do not call clock_gettime() that frequently (I've verified
it), so it can use non-coarse version - CLOCK_MONOTONIC

I've also measured the latency of CLOCK_MONOTONIC and
CLOCK_MONOTONIC_COARSE, and it is 20ns vs 4ns per call [3], so make this
change affect performance you need really frequently calls of
clock_gettime.

  [3]: https://gist.github.com/azat/622fa1f9a5d8e7d546ee9d294501961d?permalink_comment_id=5119245#gistcomment-5119245

Interesting, that this bug started to appears only after jemalloc heap
profiler had been enabled by default [4], no clue why (I would believe
more in a more frequent calls to clock_adjtime(ADJ_FREQUENCY), but I
can't verify this)

  [4]: https://github.com/ClickHouse/ClickHouse/pull/65702

To be continued...

Fixes: https://github.com/ClickHouse/ClickHouse/issues/66193
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-14 08:21:13 +02:00
Nikita Mikhaylov
8182ec8762
Merge pull request #66292 from ClickHouse/bump-azure
Bump `azure-sdk-for-cpp`
2024-07-11 12:58:50 +00:00
Nikita Mikhaylov
9c10d4f402
Merge pull request #66291 from ClickHouse/bump-pocketfft
Bump `pocketfft`
2024-07-11 11:50:21 +00:00
Robert Schulze
1ebe1da7b9
Merge pull request #66216 from rschu1ze/bump-rocksdb2
Bump RocksDB
2024-07-11 09:52:59 +00:00
Azat Khuzhin
e6e8b065bc Fix possible deadlock for jemalloc with enabled profiler
ClickHouse uses JEMALLOC_PROF_LIBGCC - _Unwind_Backtrace() for jemalloc,
which calls _Unwind_Backtrace() during bootstrap of jemalloc, and if,
dlsym() has allocations it will lead to deadlock:

    0  __lll_lock_wait (futex=futex@entry=0x16516410 <init_lock+64>, private=0) at lowlevellock.c:52
    1  0x00007ffff7f9d0a3 in __GI___pthread_mutex_lock (mutex=0x16516410 <init_lock+64>) at ../nptl/pthread_mutex_lock.c:80
    2  0x000000000ba6ec23 in pthread_mutex_lock (arg=arg@entry=0x16516410 <init_lock+64>) at ./build/RelWithDebInfo/./src/Common/ThreadFuzzer.cpp:445
    3  0x0000000014215fd4 in malloc_mutex_lock_final (mutex=0x165163d0 <init_lock>) at ./contrib/jemalloc/include/jemalloc/internal/mutex.h:151
    4  malloc_mutex_lock_slow (mutex=0x165163d0 <init_lock>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/mutex.c:90
    5  0x00000000141defc6 in malloc_mutex_lock (tsdn=0x0, mutex=<optimized out>) at ./contrib/jemalloc/include/jemalloc/internal/mutex.h:217
    6  malloc_init_hard () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2118
    7  0x00000000141d05db in malloc_init () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:298
    8  imalloc_init_check (sopts=<optimized out>, dopts=<optimized out>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2658
    9  imalloc (sopts=<optimized out>, dopts=<optimized out>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2689
    10 calloc (num=num@entry=1, size=size@entry=32) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2852
    11 0x00007ffff7c42c05 in _dlerror_run (operate=operate@entry=0x7ffff7c42490 <dlsym_doit>, args=args@entry=0x7fffffffda40) at dlerror.c:148
    12 0x00007ffff7c42525 in __dlsym (handle=<optimized out>, name=0xe7f568 "dl_iterate_phdr") at dlsym.c:70
    13 0x000000001408e1a0 in (anonymous namespace)::getOriginalDLIteratePHDR () at ./build/RelWithDebInfo/./base/base/phdr_cache.cpp:44
    14 dl_iterate_phdr (callback=0x16287a60 <libunwind::findUnwindSectionsByPhdr(dl_phdr_info*, unsigned long, void*)>, data=0x7fffffffdb58) at ./build/RelWithDebInfo/./base/base/phdr_cache.cpp:64
    15 0x0000000016288468 in libunwind::LocalAddressSpace::findUnwindSections (this=<optimized out>, targetAddr=371765377, info=...) at ./contrib/libunwind/src/AddressSpace.hpp:605
    16 libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister (this=this@entry=0x7fffffffde70, isReturnAddress=false) at ./contrib/libunwind/src/UnwindCursor.hpp:2553
    17 0x0000000016287617 in __unw_init_local (cursor=cursor@entry=0x7fffffffde70, context=context@entry=0x7fffffffddc8) at ./build/RelWithDebInfo/./contrib/libunwind/src/libunwind.cpp:91
    18 0x000000001628b094 in _Unwind_Backtrace (callback=0x14226f40 <prof_unwind_init_callback>, ref=0x0) at ./build/RelWithDebInfo/./contrib/libunwind/src/UnwindLevel1-gcc-ext.c:106
    19 0x000000001421a7e0 in prof_boot2 (tsd=tsd@entry=0x7ffff7c357c0, base=<optimized out>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/prof.c:693
    20 0x00000000141df3e7 in malloc_init_hard () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2151
    21 0x00000000141d96f1 in malloc_init () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:298
    22 nallocx (size=32, flags=0) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:4011
    23 0x000000000b9fdd9b in _ZN6Memory23getActualAllocationSizeITpTkNSt3__17same_asISt11align_val_tEEJEQsr2DBE16OptionalArgumentIDpT_EEEmmS5_ (size=32) at ./src/Common/memory.h:197
    24 _ZN6Memory11trackMemoryITpTkNSt3__17same_asISt11align_val_tEEJEQsr2DBE16OptionalArgumentIDpT_EEEmmR15AllocationTraceS5_ (size=32, trace=...) at ./src/Common/memory.h:208
    25 operator new (size=size@entry=32) at ./build/RelWithDebInfo/./src/Common/new_delete.cpp:53
    26 0x000000001540e0a6 in google::protobuf::internal::ShutdownData::get () at ./build/RelWithDebInfo/./contrib/google-protobuf/src/google/protobuf/message_lite.cc:679
    27 google::protobuf::internal::OnShutdownRun (f=0x15407420 <google::protobuf::internal::DestroyString(void const*)>, arg=0x167a6368 <google::protobuf::internal::fixed_address_empty_string>) at ./build/RelWithDebInfo/./contrib/google-protobuf/src/google/protobuf/message_lite.cc:697
    28 0x0000000015407560 in global constructors keyed to 000101 () at ./contrib/google-protobuf/src/google/protobuf/generated_message_util.h:180
    29 0x000000001629101d in __libc_csu_init ()
    30 0x00007ffff7dba010 in __libc_start_main (main=0x66e3b00 <main(int, char**)>, argc=4, argv=0x7fffffffe258, init=0x16290fd0 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe248) at ../csu/libc-start.c:264
    31 0x00000000066e302e in _start ()

And this is indeed what happens for glibc prio 2.34, since only in [2]
the allocation had been removed

  [2]: https://sourceware.org/git/?p=glibc.git;a=commit;h=fada9018199c21c469ff0e731ef75c6020074ac9

But, jemalloc provides also JEMALLOC_PROF_LIBUNWIND, yes it is not llvm
libunwind, but gnu one, but ClickHouse version of llvm libunwind
provides unw_backtrace(), so we can use it, and it will not be called
during jemalloc bootstrap, so deadlock should not be possible.

Funny thing that for this deadlock the workaround is to enable
background_thread, because a thread created for it will initialize TLS for
__dlerror().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-10 16:05:36 +02:00
Nikita Mikhaylov
2b091983e8 Bump Azure to ea3e19a7be 2024-07-09 19:59:49 +00:00
Nikita Mikhaylov
0d54151cb8 Make the pocketfft to point to the upstream/master branch 2024-07-09 19:55:37 +00:00
Robert Schulze
55468caeae
Fix ARM build 2024-07-09 16:38:22 +00:00
Vitaly Baranov
32b9f7940f
Merge pull request #66191 from vitlibar/grpc-avoid-using-harmful-function-rand
Avoid using harmful function rand() in grpc
2024-07-09 13:42:20 +00:00
Kruglov Pavel
52b3d1a244
Merge pull request #66130 from slvrtrn/update-avro-submodule-with-array-block-size-fix
Update AVRO submodule with the array block size fix
2024-07-09 07:58:04 +00:00
Nikita Taranov
a9546fe551
Merge pull request #66147 from ClickHouse/fix_mac_build
Playing minesweeper with build system
2024-07-08 22:05:17 +00:00
Robert Schulze
bd42a096b7
Bump rocksdb to v6.29.5 2024-07-08 16:52:36 +00:00
Robert Schulze
1bfafa42be
Bump rocksdb to v6.28.2 2024-07-08 14:47:35 +00:00
Robert Schulze
9509802866
Bump rocksdb to v6.27.3 2024-07-08 13:57:03 +00:00
Robert Schulze
31f9bed442
Bump rocksdb to 6.26.1 2024-07-08 13:49:05 +00:00
Robert Schulze
adcee80b2d
Bump rocksdb to v6.25.3 2024-07-08 13:41:34 +00:00
Robert Schulze
312dd82425
Bump rocksdb to v6.24.2 2024-07-08 12:58:59 +00:00
Robert Schulze
068da21837
Merge pull request #66068 from rschu1ze/bump-rocksdb
Bump rocksdb to v6.23.3
2024-07-08 12:17:40 +00:00
Robert Schulze
5943d60f26
Merge pull request #66097 from azat/build/src-dir-fix
Avoid using source directory for generated files
2024-07-08 09:20:39 +00:00
Vitaly Baranov
7b19076ebe Avoid using harmful function rand() in grpc. 2024-07-07 22:31:45 +02:00
Vitaly Baranov
42945a6d4a
Merge pull request #66137 from vitlibar/switch-contrib-orc-to-better-commit
Switch submodule contrib/orc to a proper commit in the main branch.
2024-07-07 17:38:08 +00:00
Nikita Taranov
0c13289ea3
Merge branch 'master' into fix_mac_build 2024-07-07 16:31:49 +02:00
Nikita Taranov
6e8edf008f try fix 2024-07-05 18:36:50 +02:00
Robert Schulze
72b3ebd180
Merge pull request #66136 from rschu1ze/bump-s2
Bump s2geometry again
2024-07-05 14:05:31 +00:00
Vitaly Baranov
40fd150279 Switch submodule contrib/orc to proper commit in the main branch. 2024-07-05 15:33:50 +02:00
Robert Schulze
d3f23c2753
Bump s2geometry again 2024-07-05 13:29:34 +00:00
slvrtrn
3ce470c57f Update AVRO submodule with the array block size fix 2024-07-05 14:42:43 +02:00
Vitaly Baranov
f315b36323
Merge pull request #66069 from vitlibar/add-prometheus-protocols-protobufs
Add protobufs for Prometheus remote-write/remote-read protocols
2024-07-05 10:41:04 +00:00
Robert Schulze
3621b35591
Merge pull request #66094 from rschu1ze/bump-s2geometry
Bump s2geometry to latest master
2024-07-05 10:30:20 +00:00
Robert Schulze
4dca031e9d
Merge pull request #66100 from rschu1ze/bump-azure
Bump Azure to 1.12
2024-07-05 09:11:00 +00:00
Vitaly Baranov
1ef5bca591 Fix cmake function PROTOBUF_GENERATE_CPP(): now it returns correct paths in SRCS and HDRS
even if input ".proto" files are located in sibling directories.
2024-07-04 23:07:19 +02:00
Vitaly Baranov
e428542b2e Add prometheus protobufs. 2024-07-04 23:06:14 +02:00
Antonio Andelic
ec6739120c
Merge pull request #66041 from ClickHouse/try-disabling-jemalloc-background-threads
Try disabling jemalloc background threads
2024-07-04 20:30:39 +00:00
Robert Schulze
ffe1f8fea0
Bump Azure to 1.12 2024-07-04 19:24:30 +00:00
Robert Schulze
1d8389ddee
Bump rocksdb to v6.23.3 2024-07-04 19:19:34 +00:00
Robert Schulze
597810b69d
Fix s390x build 2024-07-04 18:46:09 +00:00
Robert Schulze
632f292fd1
Merge pull request #66082 from rschu1ze/bmp-vectorscan11
Bump vectorscan to 5.4.11
2024-07-04 18:11:10 +00:00
Robert Schulze
1bd515cc1f
Merge remote-tracking branch 'ClickHouse/master' into bump-s2geometry 2024-07-04 15:43:47 +00:00
Azat Khuzhin
c49d26bc23 Avoid using source directory for generated files
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-04 17:20:16 +02:00
Robert Schulze
11a30d6d60
Bump s2geometry to latest master 2024-07-04 14:20:06 +00:00
Antonio Andelic
613ed1ebbf Merge branch 'master' into try-disabling-jemalloc-background-threads 2024-07-04 13:58:26 +02:00
Antonio Andelic
c93d8cbb66 Fixes 2024-07-04 13:57:47 +02:00
Nikita Mikhaylov
e2a494fa60
Merge pull request #66064 from rschu1ze/mysql-openssl-tsan
Maybe fix tsan assert in `test_mysql_killed_while_insert_8_0`
2024-07-04 11:17:42 +00:00
Robert Schulze
33b7afc1b4
Bump vectorscan to 5.4.11 2024-07-04 10:30:24 +00:00
Robert Schulze
dbac2212a0
Merge pull request #66072 from rschu1ze/llvm-15-burning-some-trash
Clean-up custom LLVM 15 patches
2024-07-04 09:06:38 +00:00
Robert Schulze
43b088dbd0
Merge pull request #66056 from rschu1ze/bmp-vectorscan
Bump vectorscan to 5.4.10.1
2024-07-04 08:59:50 +00:00
Robert Schulze
eb7ab5128d
Clean-up custom LLVM 15 patches 2024-07-03 21:07:59 +00:00