Commit Graph

4133 Commits

Author SHA1 Message Date
Michael Kolupaev
f6f79e188d Apply libunwind changes needed for musl 2024-08-14 00:23:18 +00:00
Robert Schulze
2ffcc97af2
Merge pull request #63675 from rschu1ze/vector-search
Initial implementation of vector similarity index
2024-08-13 15:06:20 +00:00
Robert Schulze
218421c255
Remove Annoy indexes
Annoy indexes fell out of favor in the community, at least when it comes
to vector databases. Such indexes work okay-ish low dimensions but they
suffers badly from a curse of dimensionality which makes them inapt for
a high number of dimensions.

Now that Annoy is gone, issue (*) also disappears and we can drop
'no-ubsan', 'no-cpu-aarch64', and 'no-asan' from tests.

(*) spotify/annoy#456
2024-08-12 15:24:49 +00:00
Robert Schulze
d7211f9d12
Fix CMake integration of usearch and annoy
Registers usearch and annoy properly via configure_config.cmake and
config.h.in like all other 3rd party libs, instead of (mis)using
target_compile_definitions.
2024-08-12 15:24:18 +00:00
Robert Schulze
a39b9cf643
Un-screw usearch's build description
No directory 'SimSIMD-map' exists, the build only worked because SimSIMD
support in usearch was (accidentally?) disabled. This commit corrects
the build description. SimSIMD support in usearch will be enabled by a
later commit.
2024-08-12 15:24:14 +00:00
Robert Schulze
037a1006fd
Merge remote-tracking branch 'ClickHouse/master' into ci-fuzzer-enable 2024-08-12 12:28:32 +00:00
Robert Schulze
4502862033
Fix no-SSE3 build 2024-08-11 08:35:47 +00:00
Robert Schulze
eec5fe087c
Fix CMake for QATlib 2024-08-09 08:50:52 +00:00
Robert Schulze
a497a23914
Fix CMake for QPL 2024-08-09 08:44:22 +00:00
Robert Schulze
47f429a524
Proper CMake for libfiu 2024-08-09 08:04:24 +00:00
Alexey Milovidov
2e6ecf6913
Merge pull request #66311 from azat/rdkafka-fix-data-race
Fix possible data-race StorageKafka with statistics_interval_ms>0
2024-08-08 18:56:22 +02:00
Robert Schulze
5a17d93bf3
Fix freebsd build 2024-08-08 07:29:20 +00:00
Robert Schulze
5c97205742
Reapply "Bump rocksdb from v8.10 to v9.4 + enable jemalloc and liburing"
This reverts commit ff8ce505d7.
2024-08-08 07:23:10 +00:00
Azat Khuzhin
301ac5dab7 Fix possible data-race StorageKafka with statistics_interval_ms>0
The problem here is that ignorelist did not work by some reason, if I
will look at the ignored functions it should not contain any TSan
interseption code, while it does:

    $ lldb-13 clickhouse
    (lldb) target create "clickhouse"
    disas -n rd_avg_rollover
    Current executable set to '/home/azat/ch/tmp/tsan-test/clickhouse' (x86_64).
    (lldb) disas -n rd_avg_rollover
    clickhouse`rd_kafka_stats_emit_avg:
    clickhouse[0x1cbf84a7] <+39>:  leaq   0x30(%r15), %r12
    clickhouse[0x1cbf84ab] <+43>:  movq   %r12, %rdi
    clickhouse[0x1cbf84ae] <+46>:  callq  0x1ccdad40                ; rdk_thread_mutex_lock at tinycthread.c:111
    clickhouse[0x1cbf84b3] <+51>:  leaq   0x58(%r15), %rdi
    clickhouse[0x1cbf84b7] <+55>:  callq  0x71b5390                 ; __tsan_read4
    clickhouse[0x1cbf84bc] <+60>:  cmpl   $0x0, 0x58(%r15)
    clickhouse[0x1cbf84c1] <+65>:  je     0x1cbf8595                ; <+277> [inlined] rd_avg_rollover + 238 at rdavg.h
    clickhouse[0x1cbf84c7] <+71>:  leaq   -0xc8(%rbp), %rdi
    clickhouse[0x1cbf84ce] <+78>:  xorl   %esi, %esi
    clickhouse[0x1cbf84d0] <+80>:  callq  0x1ccdac80                ; rdk_thread_mutex_init at tinycthread.c:62
    clickhouse[0x1cbf84d5] <+85>:  leaq   0x5c(%r15), %rdi
    clickhouse[0x1cbf84d9] <+89>:  callq  0x71b5390                 ; __tsan_read4

    (lldb) disas -n rd_avg_calc
    clickhouse`rd_kafka_broker_ops_io_serve:
    clickhouse[0x1cbdf086] <+1990>: leaq   0x5a4(%rbx), %rdi
    clickhouse[0x1cbdf08d] <+1997>: callq  0x71b5390                 ; __tsan_read4
    clickhouse[0x1cbdf092] <+2002>: cmpl   $0x0, 0x5a4(%rbx)
    clickhouse[0x1cbdf099] <+2009>: je     0x1cbdf12b                ; <+2155> [inlined] rd_kafka_broker_timeout_scan + 719 at rdkafka_broker.c

I guess the reason is that they had been inlined

So now rd_avg_calc() guarded with a mutex.

Refs: https://github.com/ClickHouse/librdkafka/pull/11
Fixes: https://github.com/ClickHouse/ClickHouse/issues/60939
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-08-08 07:43:29 +02:00
Nikita Mikhaylov
ff8ce505d7
Revert "Bump rocksdb from v8.10 to v9.4 + enable jemalloc and liburing" 2024-08-08 00:20:16 +02:00
Yakov Olkhovskiy
b246f7c17c merge master 2024-08-07 16:25:31 +00:00
Robert Schulze
681014d99d
Merge pull request #67904 from rschu1ze/bump-rocksdb3
Bump rocksdb from v8.10 to v9.4 + enable jemalloc and liburing
2024-08-07 14:34:48 +00:00
Robert Schulze
e07f8b4807
Merge pull request #67933 from mzhukova/dev/mzhukova/upd-qpl-1.6.0
Upgrade QPL to v1.6.0
2024-08-07 09:40:49 +00:00
Robert Schulze
7af334bd46
Merge pull request #67942 from rschu1ze/fix-memoverflow-in-usearch
Fix memory corruption in usearch
2024-08-07 08:38:14 +00:00
Robert Schulze
26b622540f
Merge pull request #64489 from ClickHouse/zup
Update zlib-ng from 2.0.2 to 2.1.7
2024-08-06 19:23:05 +00:00
Zhukova, Maria
36b6adbb30 qpl-cmake: Better wording on ISA-L copy and removed non-required linking 2024-08-06 12:15:23 -07:00
Robert Schulze
9594a9baff
Fix memory corruption in usearch 2024-08-06 18:43:44 +00:00
Robert Schulze
7a5b30d955
Re-add forked usearch repo 2024-08-06 18:39:18 +00:00
Robert Schulze
302bd5fdc6
Remove usearch submodule 2024-08-06 18:37:26 +00:00
Robert Schulze
df0dac2f5b
Enable jemalloc and liburing in rocksdb 2024-08-06 11:17:41 +00:00
Robert Schulze
2a2ba4e685
Bump rocksdb to 9.4.0 2024-08-06 10:49:22 +00:00
Robert Schulze
e7110be48c
Bump rocksdb to v9.3.1 2024-08-06 10:43:21 +00:00
Robert Schulze
b3f5ddcd65
Bump rocksdb to 9.2.1 2024-08-06 10:40:54 +00:00
Robert Schulze
f0a6ae2205
Bump rocksdb to v9.1.1 2024-08-06 10:35:33 +00:00
Robert Schulze
7e87650a00
Bump rocksdb to v9.0.1 2024-08-06 10:32:12 +00:00
Robert Schulze
4a8be15ca8
Bump rocksdb to v8.11.4 2024-08-06 10:26:37 +00:00
Michael Kolupaev
8703941a0f Apply https://github.com/ClickHouse/zlib-ng/pull/18 2024-08-05 22:36:30 +00:00
Robert Schulze
d0bc728d52
Bump rocksdb to v8.9.1 2024-08-05 08:11:40 +00:00
Robert Schulze
cebdc5ecf6
Bump rocksdb to v8.0.0 2024-08-05 08:11:23 +00:00
Alexey Milovidov
ce58549592
Merge pull request #67557 from rschu1ze/icu-s390x
Re-enable ICU on s390/x
2024-08-04 17:51:41 +00:00
Robert Schulze
ee8079b039
Merge remote-tracking branch 'ClickHouse/master' into icu-s390x 2024-08-04 09:38:36 +00:00
Azat Khuzhin
9d0e066cda Bump NuRaft (to properly catch thread exceptions)
Refs: https://github.com/ClickHouse/NuRaft/pull/75
Refs: https://github.com/eBay/NuRaft/pull/525
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-08-03 23:41:50 +02:00
Nikita Taranov
453ee52f86
Merge pull request #67235 from ClickHouse/fix_oss_fuzz_build
Fix oss-fuzz build
2024-08-01 10:50:12 +00:00
Robert Schulze
dde274f6fa
Re-enable ICU on s390/x 2024-07-31 18:08:45 +00:00
Robert Schulze
4b40e948c7
Merge pull request #66475 from rschu1ze/bump-rocksdb
Bump RocksDB from v6.29.5 to v7.10.2
2024-07-31 09:14:42 +00:00
Robert Schulze
72d6467fd2
Bump Azure to 1.13 2024-07-30 10:43:21 +00:00
Antonio Andelic
2ea4bfe04d
Merge pull request #67098 from ClickHouse/add-numactl
Try calculating available memory if ClickHouse is bound to subset of NUMA nodes
2024-07-30 08:30:58 +00:00
Robert Schulze
07507e0cec
Merge remote-tracking branch 'ClickHouse/master' into bump-rocksdb 2024-07-29 13:41:55 +00:00
Robert Schulze
294fa1dea6
Merge remote-tracking branch 'ClickHouse/master' into bump-icu 2024-07-29 10:32:37 +00:00
Nikita Taranov
109bd52296 impl 2024-07-26 23:05:56 +01:00
Antonio Andelic
2519f9ed42 Only support archs 2024-07-26 12:08:16 +02:00
Antonio Andelic
287cce7d21 Fixes 2024-07-26 09:20:15 +02:00
Antonio Andelic
6557387148 Try calcualting memory with numactl if membind used 2024-07-25 11:49:02 +02:00
Michael Kolupaev
1b2fd51e09
Merge pull request #66977 from ClickHouse/uwo
Apply libunwind fix
2024-07-25 03:40:19 +00:00
Michael Kolupaev
36e59a1b70 Final commit hash 2024-07-25 01:58:27 +00:00