Commit Graph

1273 Commits

Author SHA1 Message Date
Robert Schulze
fa6deaa0f7
Merge pull request #38294 from ClickHouse/fix_system_dot_licences_on_mac
Fix system.licenses on mac
2022-06-23 09:33:07 +02:00
Robert Schulze
a26aba792e
Assume GNU tools are installed when crosscompiling 2022-06-22 20:07:26 +02:00
Robert Schulze
1030dd305a
Correct brew install command 2022-06-22 16:59:03 +02:00
Robert Schulze
287a7f6e98
Let CMake check the presence of GNU utils
In list-licenses.sh it's too late ... the build simply continues if
list-licenses.sh returns with a non-zero exit code.
2022-06-21 23:49:29 +02:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA)
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
2022-06-20 16:13:25 +02:00
Robert Schulze
948b5b7804
Merge pull request #38170 from ClickHouse/llvm-bump-tools
Bump minimum / maximum LLVM to 12 / 14
2022-06-17 13:28:54 +02:00
Robert Schulze
26b122d9a9
Bump minimum / maximum LLVM to 12 / 14
- the CI builds were recently upgraded to Clang 14

- this commit bumps versions of other LLVM tools needed for the build

- this is important for people who have multiple LLVM versions installed
  via their package manager
2022-06-17 09:49:08 +02:00
Mikhail f. Shiryaev
06dd85f921
Update version to 22.7.1.1 2022-06-16 17:15:22 +02:00
Alexey Milovidov
c26abbfdc5
Merge pull request #38093 from danlark1/master
Optimize most important parts with NEON SIMD
2022-06-16 08:26:56 +03:00
Danila Kutenin
08e3f77a9c Optimize most important parts with NEON SIMD
First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.

That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
2022-06-15 13:19:29 +00:00
mergify[bot]
4bd61950db
Merge branch 'master' into stripping 2022-06-14 20:48:52 +00:00
Robert Schulze
c39f4a862c
Retain .clickhouse.hash when stripping 2022-06-14 18:14:35 +00:00
Robert Schulze
35c3944d1a
Warn when gold is used instead of lld 2022-06-13 11:31:08 +00:00
Robert Schulze
bc6f30fd40
Move binary hash to ELF section ".ClickHouse.hash" 2022-06-13 08:46:23 +00:00
Robert Schulze
b27e6cfed2
Strip less aggressively to make the embedded hash survive
- It was noticed that in (*), the crashstack says "There is no
  information about the reference checksum."

- The binaries are pulled via docker hub and upon inspection they indeed
  lack the hash embedded as ELF section ".note.ClickHouse.hash" in the
  clickhouse binary. This is weird because docker hub images are
  "official" builds which should trigger the hash embedding.

- Turns out that the docker hub binaries are also stripped which was too
  aggressive. We now no longer remove sections ".comment" and ".note"
  which are anyways only 140 bytes in size, i.e. binary size still goes
  down (on my stystem) from 2.1 GB to 0.47 + 0.40 GB binary + dbg info.

(*) https://playground.lodthe.me/ba75d494-95d1-4ff6-a0ad-60c138636c9b
2022-06-10 18:33:55 +02:00
Alexey Milovidov
74f5301225
Merge pull request #37379 from azat/fix-debug-symbols
Fix debug symbols in packages
2022-05-22 00:19:34 +03:00
Alexander Gololobov
692c19b80d
Merge pull request #37369 from ClickHouse/fix_eigen_build
Fixes for eigen library build
2022-05-20 23:39:31 +02:00
Azat Khuzhin
6b7dd76fac Fix debug symbols in packages
- before: usr/lib/debug/usr/bin/clickhouse.debug/clickhouse.debug
- after : usr/lib/debug/usr/bin/clickhouse.debug

Note, clickhouse_make_empty_debug_info_for_nfpm() is fine.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-05-20 17:39:27 +03:00
Robert Schulze
b475fbc9a7
Merge pull request #37300 from ClickHouse/cmake-cleanup-pt3
Various cmake cleanups
2022-05-20 10:02:36 +02:00
Robert Schulze
9b0acaa0ca
Update cmake/ccache.cmake
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
2022-05-20 09:57:25 +02:00
Robert Schulze
7c0e467b05
Small update of comment for SOURCE_DATE_EPOCH workaround 2022-05-20 09:46:19 +02:00
Alexander Gololobov
4350935377 Re-enable SSE2 for PowerPC 2022-05-20 08:51:48 +02:00
Alexey Milovidov
7c9df33bf8
Update warnings.cmake 2022-05-20 03:12:44 +03:00
Mikhail f. Shiryaev
d0fe794fe5
Update version to 22.6.1.1 2022-05-19 10:06:09 +02:00
Robert Schulze
b48cd13c07
Remove unused file 2022-05-18 22:25:34 +02:00
Robert Schulze
45463c27cc
Update cmake/ccache.cmake
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
2022-05-18 12:43:43 +02:00
Robert Schulze
6ce8947417
GLOB with CONFIGURE_DEPENDS
Globbing generally misses to pick up files which were added/deleted
after CMake's configure. This is a nuissance but can be alleviated using
CONFIGURE_DEPENDS (available since CMake 3.12) which adds a check for
new/deleted files before each compile and - if necessary - restarts the
configuration. On my system, the check takes < 0.1 sec.

(Side note: CONFIGURE_DEPENDS is not guaranteed to work accross all
generators, but at least it works for Ninja which everyone @CH seems to
use.)
2022-05-18 10:34:10 +02:00
Robert Schulze
30a01506ed
Remove weird check for not-Clang
The check activated ccache unconditionally for all non-Clang compilers
(= GCC) while allowing ancient ccache versions for these. Perhaps there
was a reason for that in the past but it's simpler to only require a
minimum ccache version.

To simplify further, also require at least ccache 3.3 (released in 2016)
instead of 3.2.1 (released in 2014).
2022-05-17 20:38:54 +02:00
Robert Schulze
4aff310a1c
Made the ccache integration more straightforward
The new logic is simpler and should do the same as before.
2022-05-17 20:38:54 +02:00
Robert Schulze
92259335b0
Use existing variable COMPILER_CLANG to identify Clang 2022-05-17 19:35:22 +02:00
Robert Schulze
6db32a3ed1
Fix externally set compiler launchers
The compiler launcher (ccache, distcc) can be set externally via
-DCMAKE_CXX_COMPILER_LAUNCHER=<tool>. We previously silently ignored
this setting and continued without any launcher (e.g. ccache). Changed
this to now respect the externally specified launcher.
2022-05-17 19:28:05 +02:00
Robert Schulze
c0a4af295c
Cosmetics 2022-05-17 19:06:13 +02:00
Robert Schulze
34af1cb116
Throw option WEVERYTHING out
WEVERYTHING enables on Clang literally every warning. People on the
internet are divided if this is a good thing or not but ClickHouse
compiles with -Weverything + some exceptions for noisy warnings since at
least a year.

I tried to build with WEVERYTHING = OFF and the build was badly broken.
It seems nobody actually turns WEVERYTHING off. Actually, why would one
if the CI builds (configured with WEVERYTHING = ON) potentially generate
errors not generated in local development.

To simplify the build scripts and to remove the need to maintain two
sets of compiler warnings, I made WEVERYTHING the default and threw
WEVERYTHING = OFF out.
2022-05-17 15:52:53 +02:00
Robert Schulze
95d28973aa
Cosmetics 2022-05-15 16:07:10 +02:00
Robert Schulze
28ae7ad9c6
Print used linker always and not only on Linux 2022-05-15 16:07:10 +02:00
Robert Schulze
7000bdb3e8
Remove condition which is always true 2022-05-15 16:07:10 +02:00
Robert Schulze
7c11f41038
Removed funny objcopy/strip finding logic on Apple
On Darwin, the build script tries to
1. use llvm-objcopy/llvm-strip from $PATH,
2. if not found by 1., use standard objcopy/strip from $PATH

The brew install instructions recommends to set $PATH to brew's binary
dir, so 2. will find something (assuming binutils is installed from
brew). If $PATH additionally points to brew's LLVM binary dir (which is
different from brew's binary dir), 1. will find the llvm versions of the
tools.

This commit removes additional logic which repeats above steps in a more
implicit way by calling brew internally and figuring out the paths once
more if 1. and 2. cannot find them in the $PATH. This removes
duplication and simplifies the script. Maybe it even helps with
reproducibility.
2022-05-15 16:06:54 +02:00
Robert Schulze
9add41efce
Log locations of ar, ranlib and install-name-tool
Makes the output consistent with other build tools
2022-05-15 16:04:32 +02:00
Robert Schulze
a982e95caf
Harmonize and clarify error messasges 2022-05-15 16:04:32 +02:00
Robert Schulze
d505bcd510
Bump AppleClang to 12.0
- previous XCode 10.2 / Clang 7.0 was horribly outdated

- XCode 12.0 corresponds to the minimally required vanilla Clang version 12.0

- remove passing of "-fchar8_t" flag (with Clang >= 9, it is part of -std=c++20)
2022-05-15 16:04:32 +02:00
Robert Schulze
ef6138a2ee
Extract minimum compiler versions 2022-05-15 16:04:32 +02:00
Robert Schulze
ce50a1531f
Abort if unsupported compiler 2022-05-15 16:04:32 +02:00
Robert Schulze
3a5c16ebbd
Fail in systems with unknown CMAKE_SYSTEM_NAME 2022-05-15 09:48:54 +02:00
Robert Schulze
e06db56c67
Move ppc cpu features out of the x86 block 2022-05-15 09:48:48 +02:00
Robert Schulze
3eb964a9f5
Simplify: Move warnings stuff into cmake/warnings.cmake 2022-05-15 09:48:03 +02:00
Robert Schulze
48e8c6185c
Remove unused check for -Wshadow
Looks like an unused leftover
2022-05-15 09:48:03 +02:00
Robert Schulze
0795b8cdb1
Fix check if ccache is already setup 2022-05-15 09:48:02 +02:00
Robert Schulze
eb9f11da3d
Replace variable ARCH_ARM by ARCH_AARCH64
Both are equivalent and removing the duplication was a leftover of PR #36614
2022-05-15 09:48:02 +02:00
Alexander Gololobov
1dc3d04837 Removed __SSE2__ flag for PPC64 2022-05-12 16:20:19 +02:00