Commit Graph

2666 Commits

Author SHA1 Message Date
Robert Schulze
0d80874d40
Merge pull request #38068 from ClickHouse/clang-tsa
Support for Clang Thread Safety Analysis (TSA)
2022-06-21 20:19:33 +02:00
Suzy Wang
1a23dfa827 Update thrift to 0.13.0 2022-06-21 06:54:54 -07:00
Saulius Valatka
a7152fe5ec bump liburing 2022-06-21 13:20:46 +03:00
Saulius Valatka
7b5e13a9eb implement pending request queue to prevent CQ overflows, more error checking, reduce size of io_uring 2022-06-21 13:17:56 +03:00
Saulius Valatka
6bbabf59d9 simplify build, add read method randomized tests, fix typos 2022-06-21 09:59:46 +03:00
Saulius Valatka
3f755a2505 add liburing to build 2022-06-21 09:59:46 +03:00
zvonand
22af00b757 rename variable + fix handling of ENABLE_LIBRARIES 2022-06-20 23:53:47 +05:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA)
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
2022-06-20 16:13:25 +02:00
Rafael Acevedo
51035ee300
Update librdkafka
Update to get fixes from https://github.com/ClickHouse/librdkafka/pull/5

Refs https://github.com/ClickHouse/ClickHouse/issues/38151
2022-06-20 07:19:58 -03:00
zvonand
832fd6e0a9 Added tests + minor updates 2022-06-19 23:10:28 +05:00
Alexey Milovidov
0cf88e0950
Revert "ClickHouse's boringssl module updated to the official version of the FIPS compliant." 2022-06-18 23:16:18 +03:00
alesapin
0ae0cc54aa
Merge pull request #38106 from ClickHouse/tsan_less_flaky_tests
Some fixes for tests with tsan
2022-06-18 14:22:43 +02:00
Alexey Milovidov
d43492a512
Merge pull request #37795 from DevTeamBK/curl_upgrade
Upgrade curl to 7.83.1
2022-06-18 04:14:09 +03:00
alesapin
ca33ff93cb
Merge pull request #37555 from ClickHouse/revert-37534-revert-37036-keeper-preprocess-operations
Add support for preprocessing ZooKeeper operations and real-time digest in `clickhouse-keeper`
2022-06-16 17:29:00 +02:00
Suzy Wang
b80b9ec8b9 Merge branch 'master' into curl_upgrade 2022-06-16 08:09:15 -07:00
Daniel Kutenin
a769dea8ef
Merge branch 'ClickHouse:master' into master 2022-06-15 19:12:10 +01:00
Alexander Tokmakov
9cac78b498 make tests with tsan less flaky 2022-06-15 19:54:46 +02:00
Maksim Kita
5729210f22
Merge pull request #37797 from kitaisreal/libunwind-update-version
libunwind update version
2022-06-15 19:32:42 +02:00
Danila Kutenin
08e3f77a9c Optimize most important parts with NEON SIMD
First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.

That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
2022-06-15 13:19:29 +00:00
mergify[bot]
68d043869c
Merge branch 'master' into libunwind-update-version 2022-06-15 11:43:01 +00:00
Maksim Kita
1c841fadcd Updated libunwind 2022-06-15 10:59:23 +02:00
zvonand
c149c916ec initial setup 2022-06-15 11:49:55 +05:00
mergify[bot]
17cc36e0e2
Merge branch 'master' into grpc_update_to_PR9 2022-06-15 02:10:38 +00:00
Antonio Andelic
b7bd5a8eb1 Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-14 12:51:35 +00:00
Maksim Kita
4a37e36d78 Updated libunwind 2022-06-14 12:52:52 +02:00
Antonio Andelic
e5504f1b33 Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-13 15:43:10 +00:00
Robert Schulze
027fe1db9d
Merge pull request #37970 from ClickHouse/color_diagnostics
Always disable --color-diagnostics for LLVM
2022-06-13 13:32:19 +02:00
Maksim Kita
fb54a1cb6a Updated libunwind 2022-06-13 11:44:45 +02:00
Maksim Kita
8907c4b5b0 Updated libunwind 2022-06-13 11:44:45 +02:00
Maksim Kita
25a886a78c libunwind update version fix sanitizers 2022-06-13 11:44:45 +02:00
Maksim Kita
d8c4af725a libunwind update version 2022-06-13 11:44:45 +02:00
Robert Schulze
0696ee3758
Always disable --color-diagnostics for LLVM
The flag is not only not recognized on MacOS, it is not recognized by
the gold linker in general.
2022-06-10 10:39:38 +02:00
Antonio Andelic
2c37fe3d7b Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-06 12:39:55 +00:00
Peng Liu
6de7590af4 compile the new libgsasl with supportting three modes(privacy/integrity/authentication) 2022-06-04 23:30:38 +08:00
Bharat Nallan Chakravarthy
5fb9336159 add ssl version to build options 2022-06-04 15:31:08 +05:30
Alexey Milovidov
4387379090
Merge pull request #37745 from DevTeamBK/libxml2_upgrade
Upgrade libxml2 to 2.9.14
2022-06-02 18:39:09 +03:00
Antonio Andelic
7ee47fc5bb Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-02 13:38:52 +00:00
Alexey Milovidov
eef6a5ec96
Revert "Remove resursive submodules" 2022-06-02 09:41:12 +03:00
Suzy Wang
17fb5d8068 upgrade curl to 7.83.1 2022-06-01 21:30:02 -07:00
Robert Schulze
7813cf5a8b
Merge pull request #37719 from ClickHouse/lib-dependency-mgmt
Fix build with -DENABLE_LIBRARIES=0
2022-06-01 16:53:49 +02:00
Suzy Wang
9bef38effb Upgrade libxml2 to 2.9.14 2022-06-01 06:10:05 -07:00
mergify[bot]
4aefcf29e3
Merge branch 'revert-37534-revert-37036-keeper-preprocess-operations' into keeper-real-time-digest 2022-06-01 10:17:49 +00:00
Robert Schulze
393b97763a
Make SSL a mandatory dependency for now
- SSL is a dependency of too many libs + unit tests (via poco crypto
  which requires SSL)

- optional SSL is desirable but right now, turning off SSL (via
  -DENABLE_LIBRARIES=0 or =DENABLE_SSL=0) breaks the build

- therefore make SSL mandatory for now + add a TODO comment
2022-06-01 11:26:24 +02:00
Robert Schulze
933f98a900
Removed warning flags in contribs
warnings are disabled for all contribs in contrib/CMakeLists.txt already
2022-06-01 11:02:39 +02:00
Robert Schulze
a4e037c728
Remove unused M_LIBRARY link 2022-06-01 11:02:39 +02:00
Robert Schulze
12871a43e1
Cosmetics 2022-06-01 11:02:39 +02:00
Robert Schulze
249fe561f4
Fix build with -DENABLE_LIBRARIES=0 / -DENABLE_REPLXX=0
Replxx: When disabled via -DENABLE_LIBRARIES=0 or -DENABLE_REPLXX (the
latter was undocumented) the build broke because replxx symbols were
used since [0] in header LineReader.h. This header should in theory
stay clean of replxx but doesn't for efficiency reasons.

This change makes compilation of replxx mandatory. As replxx is quite
small, I guess this is okay. (The alternative is to litter the code
with ifdefs for non-replxx and a replxx paths.)

[0] https://github.com/ClickHouse/ClickHouse/pull/33201
2022-06-01 10:02:28 +02:00
HarryLeeIBM
93be56a151 Update grpc submodule to PR 9 2022-05-31 14:12:44 -07:00
alesapin
65057bf8c4
Merge pull request #37616 from ClickHouse/remove-resursive-submodules
Remove resursive submodules
2022-05-31 11:58:04 +02:00
Robert Schulze
557bb2d235
Disable amqp-cpp and cassandra build if libuv is disabled
On MacOS/GCC, the libuv build is disabled due to a compiler bug. This
is now propagated to dependent libraries amqp-cpp and cassandra.
Oddly enough, the Mac/GCC build was broken since at least Jan 2022
without someone noticing.
2022-05-31 10:34:03 +02:00
Alexey Milovidov
7281d51e45 Remove recursive submodules 2022-05-28 04:01:03 +02:00
tchepavel
42cd854688
Merge branch 'master' into nats-integration 2022-05-27 16:05:08 +03:00
Robert Schulze
7543841438
Merge pull request #37518 from ClickHouse/bump-cctz-to-2022-05-15
Bump cctz to 2022-05-15
2022-05-25 22:14:41 +02:00
Robert Schulze
90deef1c3c
Bump cctz to 2022-05-15 2022-05-25 12:21:05 +02:00
Kseniia Sumarokova
02d162c25d
Merge branch 'master' into nats-integration 2022-05-25 10:04:39 +02:00
Antonio Andelic
fe72cd7478 Merge branch 'master' into keeper-real-time-digest 2022-05-25 07:41:31 +00:00
tchepavel
43226a57fc
Merge branch 'master' into nats-integration 2022-05-24 18:15:11 +03:00
Alexander Gololobov
7d0ed7e51a Remove eigen library 2022-05-24 10:24:50 +02:00
Alexey Milovidov
dd2b2380fa Add a comment #37078 2022-05-24 04:22:51 +02:00
tchepavel
d9436ec7dd
Merge branch 'master' into nats-integration 2022-05-23 14:30:09 +03:00
Antonio Andelic
8cb6101a3c Merge branch 'master' into keeper-real-time-digest 2022-05-23 10:34:20 +00:00
mergify[bot]
cc1362511a
Merge branch 'master' into patch-2 2022-05-23 00:12:44 +00:00
Yakov Olkhovskiy
d878f193d8
Merge pull request #37013 from mnutt/hashid
Add hashid support
2022-05-21 17:14:54 -04:00
Michael Nutt
23dbf1b257 Merge branch 'master' into hashid 2022-05-20 08:42:01 -04:00
Antonio Andelic
e5f7f5f689 Update NuRaft commit 2022-05-20 09:59:13 +00:00
Alexander Gololobov
4350935377 Re-enable SSE2 for PowerPC 2022-05-20 08:51:48 +02:00
Alexander Gololobov
4885940378 Removed ENABLE_MKL option, because we cannot depend on non-contrib libs 2022-05-20 08:51:48 +02:00
Michael Nutt
e453132db8 remove hashid define guard 2022-05-18 15:26:54 -04:00
Antonio Andelic
4a91ab6560 Use NuRaft with PreAppendLogs 2022-05-16 13:38:49 +00:00
tchepavel
7317c76d3c
Merge branch 'master' into nats-integration 2022-05-15 14:41:22 +03:00
Robert Schulze
eb9f11da3d
Replace variable ARCH_ARM by ARCH_AARCH64
Both are equivalent and removing the duplication was a leftover of PR #36614
2022-05-15 09:48:02 +02:00
Michael Nutt
8bff9b8ce9
Merge branch 'master' into hashid 2022-05-14 09:52:05 +09:00
Yakov Olkhovskiy
f5562f354a update poco 2022-05-13 17:34:29 -04:00
Alexander Gololobov
fe2aa1861f
Merge pull request #37033 from qieqieplus/array-distance-functions
Distance/ Norm functions for array
2022-05-13 17:28:19 +02:00
Nikolai Kochetov
b2c363a38a
Merge pull request #37157 from ClickHouse/try-enable-cache-for-https-session
Enable DNS cache for HTTPSClientSession.
2022-05-13 15:42:30 +02:00
tchepavel
cc94378698 Add nats-io cmake; finally fix publishing 2022-05-12 19:11:32 +03:00
Nikolai Kochetov
77fb1b84ac Enable DNS cache for HTTPSClientSession. 2022-05-12 15:21:29 +00:00
tchepavel
d9a19910a8 Add nats-io cmake; move subscribe 2022-05-12 15:32:51 +03:00
tchepavel
f53ec6a3a5 Preparing for first draft 2022-05-12 14:08:45 +03:00
tchepavel
5f8fcb748f Select, MV, Insert working 2022-05-12 13:58:50 +03:00
tchepavel
30ff0de33e Change submodule to patched 2022-05-12 13:58:50 +03:00
tchepavel
944a729fc5 Copy Rabbit to become NATS 2022-05-12 13:58:49 +03:00
Michael Nutt
2ff13c4e5d
Merge branch 'master' into hashid 2022-05-12 03:12:10 +09:00
Alexander Gololobov
b34a55c9e9
Merge branch 'master' into array-distance-functions 2022-05-11 16:55:02 +02:00
Yakov Olkhovskiy
6d3a54a044
Merge pull request #36467 from olevino/wyhash
Wyhash
2022-05-11 09:57:09 -04:00
Alexander Gololobov
118dd64c93 Only enable MPL2-licensed code in eigen library 2022-05-11 14:45:10 +02:00
qieqieplus
5f9eee976f fix & format 2022-05-11 16:14:43 +08:00
Yuriy Chernyshov
6d28b22687
Fix jemalloc compatibility with LLVM libunwind
jemalloc provides support for two different libunwind flavors: the original HP libunwind and the one coming with gcc / g++ / libstdc++.

The latter is identified by `JEMALLOC_PROF_LIBGCC` and provides `_Unwind_Backtrace` method instead of `unw_backtrace`.
At the time ClickHouse uses LLVM libunwind which follows libgcc's way of backtracing.

ClickHouse has to provide `unw_backtrace` method by the means of [commit 8e2b31e](8e2b31e766).

While this PR does not allow complete remove of the patch (as ClickHouse itself uses unw_backtrace directly), it definitely sorts the things out.
2022-05-10 16:14:59 +03:00
Yakov Olkhovskiy
c8c6deb2ff
Update CMakeLists.txt 2022-05-09 08:48:51 -04:00
Sean Lafferty
fe5afa6b37
Support ZSTD compression for Arrow (#35486) 2022-05-09 13:41:08 +02:00
qieqieplus
b00a17ca38 Merge branch 'master' into array-distance-functions 2022-05-09 15:15:07 +08:00
Yakov Olkhovskiy
09765028ae
Update CMakeLists.txt 2022-05-09 01:15:18 -04:00
Michael Nutt
c16ce7657e add hashid support 2022-05-08 06:42:51 +09:00
mergify[bot]
1b7b1e852b
Merge branch 'master' into bump-minizip 2022-05-02 12:14:04 +00:00
Alexey Milovidov
a31e425872
Merge pull request #36798 from ClickHouse/poco-update-58
pull poco #58
2022-05-01 13:48:29 +03:00
Alexey Milovidov
6c75b63953
Merge pull request #35914 from DevTeamBK/FIPS_compliance
ClickHouse's boringssl module updated to the official version of the FIPS compliant.
2022-04-29 21:08:51 +03:00
Yakov Olkhovskiy
970191234f pull poco #58 2022-04-29 09:48:31 -04:00
Nikita Mikhaylov
0c001df252
Better error message from NuRaft in case of CRC mismatch (#36746) 2022-04-28 17:19:28 +02:00
Meena Renganathan
ccd6f5a572 Merge branch 'master' of https://github.com/DevTeamBK/ClickHouse into FIPS_compliance 2022-04-28 07:33:12 -07:00