Commit Graph

4221 Commits

Author SHA1 Message Date
Nikita Mikhaylov
ff8ce505d7
Revert "Bump rocksdb from v8.10 to v9.4 + enable jemalloc and liburing" 2024-08-08 00:20:16 +02:00
Yakov Olkhovskiy
b246f7c17c merge master 2024-08-07 16:25:31 +00:00
Robert Schulze
681014d99d
Merge pull request #67904 from rschu1ze/bump-rocksdb3
Bump rocksdb from v8.10 to v9.4 + enable jemalloc and liburing
2024-08-07 14:34:48 +00:00
Robert Schulze
e07f8b4807
Merge pull request #67933 from mzhukova/dev/mzhukova/upd-qpl-1.6.0
Upgrade QPL to v1.6.0
2024-08-07 09:40:49 +00:00
Robert Schulze
7af334bd46
Merge pull request #67942 from rschu1ze/fix-memoverflow-in-usearch
Fix memory corruption in usearch
2024-08-07 08:38:14 +00:00
Robert Schulze
26b622540f
Merge pull request #64489 from ClickHouse/zup
Update zlib-ng from 2.0.2 to 2.1.7
2024-08-06 19:23:05 +00:00
Zhukova, Maria
36b6adbb30 qpl-cmake: Better wording on ISA-L copy and removed non-required linking 2024-08-06 12:15:23 -07:00
Robert Schulze
9594a9baff
Fix memory corruption in usearch 2024-08-06 18:43:44 +00:00
Robert Schulze
7a5b30d955
Re-add forked usearch repo 2024-08-06 18:39:18 +00:00
Robert Schulze
302bd5fdc6
Remove usearch submodule 2024-08-06 18:37:26 +00:00
Robert Schulze
df0dac2f5b
Enable jemalloc and liburing in rocksdb 2024-08-06 11:17:41 +00:00
Robert Schulze
2a2ba4e685
Bump rocksdb to 9.4.0 2024-08-06 10:49:22 +00:00
Robert Schulze
e7110be48c
Bump rocksdb to v9.3.1 2024-08-06 10:43:21 +00:00
Robert Schulze
b3f5ddcd65
Bump rocksdb to 9.2.1 2024-08-06 10:40:54 +00:00
Robert Schulze
f0a6ae2205
Bump rocksdb to v9.1.1 2024-08-06 10:35:33 +00:00
Robert Schulze
7e87650a00
Bump rocksdb to v9.0.1 2024-08-06 10:32:12 +00:00
Robert Schulze
4a8be15ca8
Bump rocksdb to v8.11.4 2024-08-06 10:26:37 +00:00
Michael Kolupaev
8703941a0f Apply https://github.com/ClickHouse/zlib-ng/pull/18 2024-08-05 22:36:30 +00:00
Robert Schulze
d0bc728d52
Bump rocksdb to v8.9.1 2024-08-05 08:11:40 +00:00
Robert Schulze
cebdc5ecf6
Bump rocksdb to v8.0.0 2024-08-05 08:11:23 +00:00
Alexey Milovidov
ce58549592
Merge pull request #67557 from rschu1ze/icu-s390x
Re-enable ICU on s390/x
2024-08-04 17:51:41 +00:00
Robert Schulze
ee8079b039
Merge remote-tracking branch 'ClickHouse/master' into icu-s390x 2024-08-04 09:38:36 +00:00
Azat Khuzhin
9d0e066cda Bump NuRaft (to properly catch thread exceptions)
Refs: https://github.com/ClickHouse/NuRaft/pull/75
Refs: https://github.com/eBay/NuRaft/pull/525
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-08-03 23:41:50 +02:00
Nikita Taranov
453ee52f86
Merge pull request #67235 from ClickHouse/fix_oss_fuzz_build
Fix oss-fuzz build
2024-08-01 10:50:12 +00:00
Robert Schulze
dde274f6fa
Re-enable ICU on s390/x 2024-07-31 18:08:45 +00:00
Robert Schulze
4b40e948c7
Merge pull request #66475 from rschu1ze/bump-rocksdb
Bump RocksDB from v6.29.5 to v7.10.2
2024-07-31 09:14:42 +00:00
Robert Schulze
72d6467fd2
Bump Azure to 1.13 2024-07-30 10:43:21 +00:00
Antonio Andelic
2ea4bfe04d
Merge pull request #67098 from ClickHouse/add-numactl
Try calculating available memory if ClickHouse is bound to subset of NUMA nodes
2024-07-30 08:30:58 +00:00
Robert Schulze
07507e0cec
Merge remote-tracking branch 'ClickHouse/master' into bump-rocksdb 2024-07-29 13:41:55 +00:00
Robert Schulze
294fa1dea6
Merge remote-tracking branch 'ClickHouse/master' into bump-icu 2024-07-29 10:32:37 +00:00
Nikita Taranov
109bd52296 impl 2024-07-26 23:05:56 +01:00
Antonio Andelic
2519f9ed42 Only support archs 2024-07-26 12:08:16 +02:00
Antonio Andelic
287cce7d21 Fixes 2024-07-26 09:20:15 +02:00
Antonio Andelic
6557387148 Try calcualting memory with numactl if membind used 2024-07-25 11:49:02 +02:00
Michael Kolupaev
1b2fd51e09
Merge pull request #66977 from ClickHouse/uwo
Apply libunwind fix
2024-07-25 03:40:19 +00:00
Michael Kolupaev
36e59a1b70 Final commit hash 2024-07-25 01:58:27 +00:00
Alexey Milovidov
1914b37058
Merge pull request #65521 from JackyWoo/add_statistics_cmsketch
Add count-min sketches as statistics type
2024-07-24 00:46:23 +00:00
Michael Kolupaev
1cbbbd1077 Apply libunwind fix 2024-07-23 17:35:12 +00:00
Michael Kolupaev
51ebbff79e Merge remote-tracking branch 'origin/master' into zup 2024-07-22 20:52:54 +00:00
Antonio Andelic
1be54641c1 Revert libunwind patch 2024-07-22 11:03:19 +02:00
Robert Schulze
2a68867a20
Merge remote-tracking branch 'ClickHouse/master' into bump-icu 2024-07-22 08:19:07 +00:00
Zhukova, Maria
1347bc3218 update qpl-cmake to reflect changes in QPL 1.5.0-1.6.0 2024-07-19 14:20:19 -07:00
Zhukova, Maria
277dbfa057 update QPL to 1.6.0 + missing header fix 2024-07-19 14:20:18 -07:00
Robert Schulze
25a22d6201
Merge remote-tracking branch 'ClickHouse/master' into bump-icu 2024-07-19 12:43:11 +00:00
Yong Wang
3ef2cbdec5 openssl-dynamic-build-ppc64le : fix openssl dynamic build for ppc64le 2024-07-18 11:21:52 -07:00
Alexey Milovidov
ec87bfc956 Merge branch 'update-tidy-for-clang-19' into clickhouse-fix-msan-report-in-epoll 2024-07-17 05:49:49 +02:00
Yakov Olkhovskiy
f07a395bf1 Merge branch 'master' into ci-fuzzer-enable 2024-07-17 03:43:57 +00:00
Alexey Milovidov
4d88abfcfa Merge branch 'master' into uwu 2024-07-16 06:04:58 +02:00
JackyWoo
245359e536 Merge branch 'master' into add_statistics_cmsketch 2024-07-16 09:45:31 +08:00
Yakov Olkhovskiy
7de367d692 Merge branch 'master' into ci-fuzzer-enable 2024-07-15 20:58:42 +00:00
Alexey Milovidov
6d617e4c4d Merge branch 'master' of github.com:ClickHouse/ClickHouse into clickhouse-fix-msan-report-in-epoll 2024-07-15 18:30:30 +02:00
Robert Schulze
2083c65005
Merge remote-tracking branch 'ClickHouse/master' into bump-icu 2024-07-15 15:31:07 +00:00
Robert Schulze
e7becedb03
Merge remote-tracking branch 'ClickHouse/master' into bump-rocksdb 2024-07-15 06:54:16 +00:00
Robert Schulze
2132ce52e0
Bump ICU to 75 2024-07-15 06:49:02 +00:00
Antonio Andelic
7d62a19f4c
Merge pull request #66439 from azat/jemalloc-clock-assertion
[RFC] Fix jemalloc assertion due to non-monotonic CLOCK_MONOTONIC_COARSE
2024-07-15 06:45:29 +00:00
Alexey Milovidov
41327260ba Fix MSan report in GRPC 2024-07-15 03:47:35 +02:00
Alexey Milovidov
cae3f63545 Merge branch 'master' into uwu 2024-07-15 01:22:38 +02:00
Robert Schulze
5b6956ea23
Bump rocksdb to v7.10.2 2024-07-14 12:38:28 +00:00
Robert Schulze
b5cb264b01
Bump ICU to 71 2024-07-14 12:17:25 +00:00
Robert Schulze
ad86e3060a
OpenSSL: Follow-up to #66064
More commits were added to the upstream fix after #66064 has been merged
in ClickHouse. This PR adds these commits to keep things in-sync.
2024-07-14 11:48:13 +00:00
Azat Khuzhin
669ce7cd40 Enable jemalloc heap profiler by default for debug builds as well
Since the issue with jemalloc assertion is clear, let's revert that
workaround

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-14 08:21:13 +02:00
Azat Khuzhin
22611781b0 Fix jemalloc assertion due to non-monotonic CLOCK_MONOTONIC_COARSE
Recently one tricky assertion of jemalloc had been discovered [1]:

    Failed assertion: "nstime_compare(&decay->epoch, new_time) <= 0"

  [1]: https://github.com/ClickHouse/ClickHouse/issues/66193

And as it turns out it is really possible for CLOCK_MONOTONIC_COARSE to
go backwards, in a nutshell it can be done with ADJ_FREQUENCY, you can
find example here [2]. And I can't trigger this issue for non-coarse
clocks.

  [2]: https://gist.github.com/azat/7ea7f50ed75591b1af2d675a240ea94c?permalink_comment_id=5119222#gistcomment-5119222

But, jemalloc do not call clock_gettime() that frequently (I've verified
it), so it can use non-coarse version - CLOCK_MONOTONIC

I've also measured the latency of CLOCK_MONOTONIC and
CLOCK_MONOTONIC_COARSE, and it is 20ns vs 4ns per call [3], so make this
change affect performance you need really frequently calls of
clock_gettime.

  [3]: https://gist.github.com/azat/622fa1f9a5d8e7d546ee9d294501961d?permalink_comment_id=5119245#gistcomment-5119245

Interesting, that this bug started to appears only after jemalloc heap
profiler had been enabled by default [4], no clue why (I would believe
more in a more frequent calls to clock_adjtime(ADJ_FREQUENCY), but I
can't verify this)

  [4]: https://github.com/ClickHouse/ClickHouse/pull/65702

To be continued...

Fixes: https://github.com/ClickHouse/ClickHouse/issues/66193
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-14 08:21:13 +02:00
Yakov Olkhovskiy
698b5ce7fa Merge branch 'master' into ci-fuzzer-enable 2024-07-14 05:31:56 +00:00
Robert Schulze
264d7d760f
Bump rocksdb to 7.0.4 2024-07-13 16:43:39 +00:00
Nikita Mikhaylov
8182ec8762
Merge pull request #66292 from ClickHouse/bump-azure
Bump `azure-sdk-for-cpp`
2024-07-11 12:58:50 +00:00
Nikita Mikhaylov
9c10d4f402
Merge pull request #66291 from ClickHouse/bump-pocketfft
Bump `pocketfft`
2024-07-11 11:50:21 +00:00
Robert Schulze
1ebe1da7b9
Merge pull request #66216 from rschu1ze/bump-rocksdb2
Bump RocksDB
2024-07-11 09:52:59 +00:00
Azat Khuzhin
e6e8b065bc Fix possible deadlock for jemalloc with enabled profiler
ClickHouse uses JEMALLOC_PROF_LIBGCC - _Unwind_Backtrace() for jemalloc,
which calls _Unwind_Backtrace() during bootstrap of jemalloc, and if,
dlsym() has allocations it will lead to deadlock:

    0  __lll_lock_wait (futex=futex@entry=0x16516410 <init_lock+64>, private=0) at lowlevellock.c:52
    1  0x00007ffff7f9d0a3 in __GI___pthread_mutex_lock (mutex=0x16516410 <init_lock+64>) at ../nptl/pthread_mutex_lock.c:80
    2  0x000000000ba6ec23 in pthread_mutex_lock (arg=arg@entry=0x16516410 <init_lock+64>) at ./build/RelWithDebInfo/./src/Common/ThreadFuzzer.cpp:445
    3  0x0000000014215fd4 in malloc_mutex_lock_final (mutex=0x165163d0 <init_lock>) at ./contrib/jemalloc/include/jemalloc/internal/mutex.h:151
    4  malloc_mutex_lock_slow (mutex=0x165163d0 <init_lock>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/mutex.c:90
    5  0x00000000141defc6 in malloc_mutex_lock (tsdn=0x0, mutex=<optimized out>) at ./contrib/jemalloc/include/jemalloc/internal/mutex.h:217
    6  malloc_init_hard () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2118
    7  0x00000000141d05db in malloc_init () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:298
    8  imalloc_init_check (sopts=<optimized out>, dopts=<optimized out>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2658
    9  imalloc (sopts=<optimized out>, dopts=<optimized out>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2689
    10 calloc (num=num@entry=1, size=size@entry=32) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2852
    11 0x00007ffff7c42c05 in _dlerror_run (operate=operate@entry=0x7ffff7c42490 <dlsym_doit>, args=args@entry=0x7fffffffda40) at dlerror.c:148
    12 0x00007ffff7c42525 in __dlsym (handle=<optimized out>, name=0xe7f568 "dl_iterate_phdr") at dlsym.c:70
    13 0x000000001408e1a0 in (anonymous namespace)::getOriginalDLIteratePHDR () at ./build/RelWithDebInfo/./base/base/phdr_cache.cpp:44
    14 dl_iterate_phdr (callback=0x16287a60 <libunwind::findUnwindSectionsByPhdr(dl_phdr_info*, unsigned long, void*)>, data=0x7fffffffdb58) at ./build/RelWithDebInfo/./base/base/phdr_cache.cpp:64
    15 0x0000000016288468 in libunwind::LocalAddressSpace::findUnwindSections (this=<optimized out>, targetAddr=371765377, info=...) at ./contrib/libunwind/src/AddressSpace.hpp:605
    16 libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister (this=this@entry=0x7fffffffde70, isReturnAddress=false) at ./contrib/libunwind/src/UnwindCursor.hpp:2553
    17 0x0000000016287617 in __unw_init_local (cursor=cursor@entry=0x7fffffffde70, context=context@entry=0x7fffffffddc8) at ./build/RelWithDebInfo/./contrib/libunwind/src/libunwind.cpp:91
    18 0x000000001628b094 in _Unwind_Backtrace (callback=0x14226f40 <prof_unwind_init_callback>, ref=0x0) at ./build/RelWithDebInfo/./contrib/libunwind/src/UnwindLevel1-gcc-ext.c:106
    19 0x000000001421a7e0 in prof_boot2 (tsd=tsd@entry=0x7ffff7c357c0, base=<optimized out>) at ./build/RelWithDebInfo/./contrib/jemalloc/src/prof.c:693
    20 0x00000000141df3e7 in malloc_init_hard () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:2151
    21 0x00000000141d96f1 in malloc_init () at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:298
    22 nallocx (size=32, flags=0) at ./build/RelWithDebInfo/./contrib/jemalloc/src/jemalloc.c:4011
    23 0x000000000b9fdd9b in _ZN6Memory23getActualAllocationSizeITpTkNSt3__17same_asISt11align_val_tEEJEQsr2DBE16OptionalArgumentIDpT_EEEmmS5_ (size=32) at ./src/Common/memory.h:197
    24 _ZN6Memory11trackMemoryITpTkNSt3__17same_asISt11align_val_tEEJEQsr2DBE16OptionalArgumentIDpT_EEEmmR15AllocationTraceS5_ (size=32, trace=...) at ./src/Common/memory.h:208
    25 operator new (size=size@entry=32) at ./build/RelWithDebInfo/./src/Common/new_delete.cpp:53
    26 0x000000001540e0a6 in google::protobuf::internal::ShutdownData::get () at ./build/RelWithDebInfo/./contrib/google-protobuf/src/google/protobuf/message_lite.cc:679
    27 google::protobuf::internal::OnShutdownRun (f=0x15407420 <google::protobuf::internal::DestroyString(void const*)>, arg=0x167a6368 <google::protobuf::internal::fixed_address_empty_string>) at ./build/RelWithDebInfo/./contrib/google-protobuf/src/google/protobuf/message_lite.cc:697
    28 0x0000000015407560 in global constructors keyed to 000101 () at ./contrib/google-protobuf/src/google/protobuf/generated_message_util.h:180
    29 0x000000001629101d in __libc_csu_init ()
    30 0x00007ffff7dba010 in __libc_start_main (main=0x66e3b00 <main(int, char**)>, argc=4, argv=0x7fffffffe258, init=0x16290fd0 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe248) at ../csu/libc-start.c:264
    31 0x00000000066e302e in _start ()

And this is indeed what happens for glibc prio 2.34, since only in [2]
the allocation had been removed

  [2]: https://sourceware.org/git/?p=glibc.git;a=commit;h=fada9018199c21c469ff0e731ef75c6020074ac9

But, jemalloc provides also JEMALLOC_PROF_LIBUNWIND, yes it is not llvm
libunwind, but gnu one, but ClickHouse version of llvm libunwind
provides unw_backtrace(), so we can use it, and it will not be called
during jemalloc bootstrap, so deadlock should not be possible.

Funny thing that for this deadlock the workaround is to enable
background_thread, because a thread created for it will initialize TLS for
__dlerror().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-10 16:05:36 +02:00
Nikita Mikhaylov
2b091983e8 Bump Azure to ea3e19a7be 2024-07-09 19:59:49 +00:00
Nikita Mikhaylov
0d54151cb8 Make the pocketfft to point to the upstream/master branch 2024-07-09 19:55:37 +00:00
Robert Schulze
55468caeae
Fix ARM build 2024-07-09 16:38:22 +00:00
Vitaly Baranov
32b9f7940f
Merge pull request #66191 from vitlibar/grpc-avoid-using-harmful-function-rand
Avoid using harmful function rand() in grpc
2024-07-09 13:42:20 +00:00
Robert Schulze
3f1c0b9966
Merge remote-tracking branch 'ClickHouse/master' into add_statistics_cmsketch 2024-07-09 10:48:19 +00:00
Kruglov Pavel
52b3d1a244
Merge pull request #66130 from slvrtrn/update-avro-submodule-with-array-block-size-fix
Update AVRO submodule with the array block size fix
2024-07-09 07:58:04 +00:00
Nikita Taranov
a9546fe551
Merge pull request #66147 from ClickHouse/fix_mac_build
Playing minesweeper with build system
2024-07-08 22:05:17 +00:00
Robert Schulze
bd42a096b7
Bump rocksdb to v6.29.5 2024-07-08 16:52:36 +00:00
Robert Schulze
1bfafa42be
Bump rocksdb to v6.28.2 2024-07-08 14:47:35 +00:00
Robert Schulze
9509802866
Bump rocksdb to v6.27.3 2024-07-08 13:57:03 +00:00
Robert Schulze
31f9bed442
Bump rocksdb to 6.26.1 2024-07-08 13:49:05 +00:00
Robert Schulze
adcee80b2d
Bump rocksdb to v6.25.3 2024-07-08 13:41:34 +00:00
Robert Schulze
312dd82425
Bump rocksdb to v6.24.2 2024-07-08 12:58:59 +00:00
Robert Schulze
068da21837
Merge pull request #66068 from rschu1ze/bump-rocksdb
Bump rocksdb to v6.23.3
2024-07-08 12:17:40 +00:00
Robert Schulze
5943d60f26
Merge pull request #66097 from azat/build/src-dir-fix
Avoid using source directory for generated files
2024-07-08 09:20:39 +00:00
Vitaly Baranov
7b19076ebe Avoid using harmful function rand() in grpc. 2024-07-07 22:31:45 +02:00
Vitaly Baranov
42945a6d4a
Merge pull request #66137 from vitlibar/switch-contrib-orc-to-better-commit
Switch submodule contrib/orc to a proper commit in the main branch.
2024-07-07 17:38:08 +00:00
Nikita Taranov
0c13289ea3
Merge branch 'master' into fix_mac_build 2024-07-07 16:31:49 +02:00
Yakov Olkhovskiy
6e5e680797 bump libprotobuf-mutator, fix build 2024-07-06 05:42:21 +00:00
Nikita Taranov
6e8edf008f try fix 2024-07-05 18:36:50 +02:00
Robert Schulze
72b3ebd180
Merge pull request #66136 from rschu1ze/bump-s2
Bump s2geometry again
2024-07-05 14:05:31 +00:00
Vitaly Baranov
40fd150279 Switch submodule contrib/orc to proper commit in the main branch. 2024-07-05 15:33:50 +02:00
Robert Schulze
d3f23c2753
Bump s2geometry again 2024-07-05 13:29:34 +00:00
slvrtrn
3ce470c57f Update AVRO submodule with the array block size fix 2024-07-05 14:42:43 +02:00
Vitaly Baranov
f315b36323
Merge pull request #66069 from vitlibar/add-prometheus-protocols-protobufs
Add protobufs for Prometheus remote-write/remote-read protocols
2024-07-05 10:41:04 +00:00
Robert Schulze
3621b35591
Merge pull request #66094 from rschu1ze/bump-s2geometry
Bump s2geometry to latest master
2024-07-05 10:30:20 +00:00
Robert Schulze
4dca031e9d
Merge pull request #66100 from rschu1ze/bump-azure
Bump Azure to 1.12
2024-07-05 09:11:00 +00:00
Vitaly Baranov
1ef5bca591 Fix cmake function PROTOBUF_GENERATE_CPP(): now it returns correct paths in SRCS and HDRS
even if input ".proto" files are located in sibling directories.
2024-07-04 23:07:19 +02:00
Vitaly Baranov
e428542b2e Add prometheus protobufs. 2024-07-04 23:06:14 +02:00
Antonio Andelic
ec6739120c
Merge pull request #66041 from ClickHouse/try-disabling-jemalloc-background-threads
Try disabling jemalloc background threads
2024-07-04 20:30:39 +00:00
Michael Kolupaev
1eb7ca7479 Merge remote-tracking branch 'origin/master' into zup 2024-07-04 19:25:23 +00:00
Robert Schulze
ffe1f8fea0
Bump Azure to 1.12 2024-07-04 19:24:30 +00:00
Robert Schulze
1d8389ddee
Bump rocksdb to v6.23.3 2024-07-04 19:19:34 +00:00
Michael Kolupaev
b04e6c806b Merge remote-tracking branch 'origin/master' into uwu 2024-07-04 19:02:47 +00:00
Robert Schulze
597810b69d
Fix s390x build 2024-07-04 18:46:09 +00:00
Robert Schulze
632f292fd1
Merge pull request #66082 from rschu1ze/bmp-vectorscan11
Bump vectorscan to 5.4.11
2024-07-04 18:11:10 +00:00
Robert Schulze
1bd515cc1f
Merge remote-tracking branch 'ClickHouse/master' into bump-s2geometry 2024-07-04 15:43:47 +00:00
Azat Khuzhin
c49d26bc23 Avoid using source directory for generated files
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-04 17:20:16 +02:00
Robert Schulze
11a30d6d60
Bump s2geometry to latest master 2024-07-04 14:20:06 +00:00
Antonio Andelic
613ed1ebbf Merge branch 'master' into try-disabling-jemalloc-background-threads 2024-07-04 13:58:26 +02:00
Antonio Andelic
c93d8cbb66 Fixes 2024-07-04 13:57:47 +02:00
Nikita Mikhaylov
e2a494fa60
Merge pull request #66064 from rschu1ze/mysql-openssl-tsan
Maybe fix tsan assert in `test_mysql_killed_while_insert_8_0`
2024-07-04 11:17:42 +00:00
Robert Schulze
33b7afc1b4
Bump vectorscan to 5.4.11 2024-07-04 10:30:24 +00:00
Robert Schulze
dbac2212a0
Merge pull request #66072 from rschu1ze/llvm-15-burning-some-trash
Clean-up custom LLVM 15 patches
2024-07-04 09:06:38 +00:00
Robert Schulze
43b088dbd0
Merge pull request #66056 from rschu1ze/bmp-vectorscan
Bump vectorscan to 5.4.10.1
2024-07-04 08:59:50 +00:00
JackyWoo
0c5821e5b8 Merge branch 'master' into add_statistics_cmsketch
# Conflicts:
#	docs/en/engines/table-engines/mergetree-family/mergetree.md
#	src/Storages/Statistics/Statistics.cpp
#	src/Storages/Statistics/Statistics.h
#	src/Storages/Statistics/StatisticsTDigest.h
#	src/Storages/Statistics/StatisticsUniq.h
#	src/Storages/Statistics/TDigestStatistics.cpp
#	tests/queries/0_stateless/02864_statistics_uniq.sql
2024-07-04 10:25:53 +08:00
Robert Schulze
eb7ab5128d
Clean-up custom LLVM 15 patches 2024-07-03 21:07:59 +00:00
Robert Schulze
9737c5bab4
Probably fix tsan assert in test_mysql_killed_while_insert_8_0 2024-07-03 17:38:53 +00:00
Robert Schulze
8b14754005
Fix ARM build (upgrade sysroot) 2024-07-03 16:29:47 +00:00
Robert Schulze
39a371b27d
Bump vectorscan 2024-07-03 15:07:31 +00:00
Antonio Andelic
5fd36059e4 Try disabling background threads 2024-07-03 15:28:01 +02:00
Michael Kolupaev
ab4d445656 Merge remote-tracking branch 'origin/master' into uwu 2024-07-02 04:37:46 +00:00
Azat Khuzhin
9c7db6711b Fix possible issues with MySQL client protocol TLS connections
Occasionally, 02479_mysql_connect_to_self fails on CI [1].

  [1]: https://github.com/ClickHouse/ClickHouse/issues/50911

The problem was indeed query profiler and EINTR, but not in a way you
may think.

For such failures you may see the following trace in trace_log:

    contrib/openssl/crypto/bio/bss_sock.c:127::sock_read
    contrib/openssl/crypto/bio/bio_meth.c:121::bread_conv
    contrib/openssl/crypto/bio/bio_lib.c:285::bio_read_intern
    contrib/openssl/crypto/bio/bio_lib.c:311::BIO_read
    contrib/openssl/ssl/record/methods/tls_common.c:398::tls_default_read_n
    contrib/openssl/ssl/record/methods/tls_common.c:575::tls_get_more_records
    contrib/openssl/ssl/record/methods/tls_common.c:1122::tls_read_record
    contrib/openssl/ssl/record/rec_layer_s3.c:645::ssl3_read_bytes
    contrib/openssl/ssl/s3_lib.c:4527::ssl3_read_internal
    contrib/openssl/ssl/s3_lib.c:4551::ssl3_read
    contrib/openssl/ssl/ssl_lib.c:2343::ssl_read_internal
    contrib/openssl/ssl/ssl_lib.c:2357::SSL_read
    contrib/mariadb-connector-c/libmariadb/secure/openssl.c:729::ma_tls_read
    contrib/mariadb-connector-c/libmariadb/ma_tls.c:90::ma_pvio_tls_read
    contrib/mariadb-connector-c/libmariadb/ma_pvio.c:250::ma_pvio_read
    contrib/mariadb-connector-c/libmariadb/ma_pvio.c:297::ma_pvio_cache_read
    contrib/mariadb-connector-c/libmariadb/ma_net.c:373::ma_real_read
    contrib/mariadb-connector-c/libmariadb/ma_net.c:427::ma_net_read
    contrib/mariadb-connector-c/libmariadb/mariadb_lib.c:192::ma_net_safe_read
    contrib/mariadb-connector-c/libmariadb/mariadb_lib.c:2138::mthd_my_read_query_result
    contrib/mariadb-connector-c/libmariadb/mariadb_lib.c:2212::mysql_real_query
    src/Common/mysqlxx/Query.cpp:56::mysqlxx::Query::executeImpl()
    src/Common/mysqlxx/Query.cpp:73::mysqlxx::Query::use()
    src/Processors/Sources/MySQLSource.cpp:50::DB::MySQLSource::Connection::Connection()

After which the connection will fail with:

    Code: 1000. DB::Exception: Received from localhost:9000. DB::Exception: mysqlxx::ConnectionLost: Lost connection to MySQL server during query (127.0.0.1:9004). (POCO_EXCEPTION)

But, if you will take a look at ma_tls_read() you will see that it has
proper retries for SSL_ERROR_WANT_READ (and EINTR is just a special case
of it), but still, for some reason it fails.

And the reason is the units of the read/write timeout, ma_tls_read()
calls poll(read_timeout) in case of SSL_ERROR_WANT_READ, but, it
incorrectly assume that the timeout is in milliseconds, but that timeout
was in seconds, this bug had been fixed in [2], and now it works like a
charm!

  [2]: https://github.com/ClickHouse/mariadb-connector-c/pull/17

I've verified it with patching openssl library:

    diff --git a/crypto/bio/bss_sock.c b/crypto/bio/bss_sock.c
    index 82f7be85ae..3d2f3926a0 100644
    --- a/crypto/bio/bss_sock.c
    +++ b/crypto/bio/bss_sock.c
    @@ -124,7 +124,24 @@ static int sock_read(BIO *b, char *out, int outl)
                 ret = ktls_read_record(b->num, out, outl);
             else
     # endif
    -            ret = readsocket(b->num, out, outl);
    +        {
    +            static int i = 0;
    +            static int j = 0;
    +            if (!(++j % 5))
    +            {
    +                fprintf(stderr, "sock_read: inject EAGAIN with ret=0\n");
    +                ret = 0;
    +                errno = EAGAIN;
    +            }
    +            else if (!(++i % 3))
    +            {
    +                fprintf(stderr, "sock_read: inject EAGAIN with ret=-1\n");
    +                ret = -1;
    +                errno = EAGAIN;
    +            }
    +            else
    +                ret = readsocket(b->num, out, outl);
    +        }
             BIO_clear_retry_flags(b);
             if (ret <= 0) {
                 if (BIO_sock_should_retry(ret))

And before this patch (well, not the patch itself, but the referenced
patch in mariadb-connector-c) if you will pass read_write_timeout=1 it
will fail:

    SELECT * FROM mysql('127.0.0.1:9004', system, one, 'default', '', SETTINGS connect_timeout = 100, connection_wait_timeout = 100, read_write_timeout=1)
    Code: 1000. DB::Exception: Received from localhost:9000. DB::Exception: mysqlxx::ConnectionLost: Lost connection to MySQL server during query (127.0.0.1:9004). (POCO_EXCEPTION)

But after, it always works:

    $ ch benchmark -c10 -q "SELECT * FROM mysql('127.0.0.1:9004', system, one, 'default', '', SETTINGS connection_pool_size=1, connect_timeout = 100, connection_wait_timeout = 100, read_write_timeout=1)"
    ^CStopping launch of queries. SIGINT received.

    Queries executed: 478.

    localhost:9000, queries: 478, QPS: 120.171, RPS: 120.171, MiB/s: 0.001, result RPS: 120.171, result MiB/s: 0.001.

    0.000%          0.014 sec.
    10.000%         0.058 sec.
    20.000%         0.065 sec.
    30.000%         0.073 sec.
    40.000%         0.079 sec.
    50.000%         0.087 sec.
    60.000%         0.089 sec.
    70.000%         0.091 sec.
    80.000%         0.095 sec.
    90.000%         0.100 sec.
    95.000%         0.102 sec.
    99.000%         0.105 sec.
    99.900%         0.140 sec.
    99.990%         0.140 sec.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-01 18:39:43 +02:00
Antonio Andelic
8f9beffc65 No jemalloc profiler for non-Linux 2024-06-28 16:04:45 +02:00
Michael Kolupaev
840edd99fa Apply https://github.com/ClickHouse/libunwind/pull/27 too 2024-06-27 21:53:23 +00:00
Michael Kolupaev
36f9b41e7d Merge remote-tracking branch 'origin/master' into uwu 2024-06-27 21:48:37 +00:00
Antonio Andelic
c9fa974472 Use background thread 2024-06-27 08:46:10 +02:00
Antonio Andelic
35820eaa7b Build jemalloc with profiler 2024-06-26 11:13:18 +02:00
JackyWoo
1601420dfd Merge branch 'master' into add_statistics_cmsketch 2024-06-25 10:05:15 +08:00
Alexey Milovidov
49634db3ba Update submodule 2024-06-25 02:51:57 +02:00
Alexey Milovidov
3b377fabe7 Update submodule 2024-06-25 02:49:40 +02:00
JackyWoo
15e20f56fa Add statistics cmsketch 2024-06-24 16:38:56 +08:00
Robert Schulze
605d51700a
Merge pull request #65529 from rschu1ze/openssl-temp-official-fix
OpenSSL: Replace temporary fix for unsynchronized access by official fix
2024-06-23 15:24:28 +00:00
Michael Kolupaev
ad246312d7 Switch to final commit hash 2024-06-21 23:05:41 +00:00
Michael Kolupaev
92ff09f303 There's no Unwind_AppleExtras.cpp anymore 2024-06-21 22:53:38 +00:00
Robert Schulze
0a60dc1672
OpenSSL: Replace temporary fix for unsynchronized access by official fix 2024-06-21 12:57:44 +00:00
Robert Schulze
df857ff17a
Bump re2 to latest HEAD 2024-06-21 10:34:30 +00:00
Michael Kolupaev
06781efcb7 Switch to a cleaner update, hopefully fix builds 2024-06-20 22:59:36 +00:00
Michael Kolupaev
8115926eb6 Fix ARM some more 2024-06-20 22:18:14 +00:00
Michael Kolupaev
62d6e3d339 Fix ARM 2024-06-20 22:18:14 +00:00
Michael Kolupaev
3ab8ba0d4a Update zlib-ng from 2.0.2 to 2.1.6 2024-06-20 22:18:14 +00:00
Michael Kolupaev
424224b202 Set _LIBUNWIND_REMEMBER_STACK_ALLOC explicitly just in case 2024-06-20 22:05:16 +00:00
Michael Kolupaev
8bb60101c4 Update libunwind to 18.1.7 2024-06-20 22:01:16 +00:00
Robert Schulze
fd52e19b3e
Suppress leaksan false positive in OpenSSL 2024-06-19 08:30:47 +00:00
Robert Schulze
abe424b24b
Merge pull request #65111 from rschu1ze/reenable-session-caching
Re-enable OpenSSL session caching
2024-06-18 16:39:38 +00:00
Alexander Tokmakov
b150b0f5fa
Revert "Fix AWS ECS" 2024-06-17 23:11:59 +02:00
Robert Schulze
cdcfcb99fc
Merge remote-tracking branch 'rschu1ze/master' into reenable-session-caching 2024-06-17 20:51:11 +00:00
Robert Schulze
186bd0cc3d
Temporary fix for tsan issue openssl#24629 2024-06-17 12:20:54 +00:00
Alexey Milovidov
2c1918908a Revert "Debug AWS"
This reverts commit 20ca8f8714.
2024-06-16 17:12:55 +02:00
Alexey Milovidov
b41bb96605 Revert "Debug AWS"
This reverts commit 642dc35a76.
2024-06-16 17:12:19 +02:00
Alexey Milovidov
37eb815b17 Fix error 2024-06-16 17:12:05 +02:00
Alexey Milovidov
642dc35a76 Debug AWS 2024-06-16 15:21:17 +02:00