libunwind is reentrant and signal safe, and works faster then then
gcc_eh (plus it has some custom patches for problems that have been
found during it's usage in ClickHouse).
gcc_eh may be missing in the system (if gcc was not installed), and
even if it exists clickhouse uses -nodefaultlibs, so some care should be
made to make it work.
Also this library is tiny and there shouln't be any problem to require
it always (there is already tendency to require some contrib libraries,
i.e. poco).
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Cross-compiling on Linux for Mac failed with CMake parameters
-DCMAKE_BUILD_TYPE=None -DENABLE_CLICKHOUSE_SELF_EXTRACTING=1
-DENABLE_TESTS=0 (see below). This happened e.g. in #51243.
The problem was that ThinLTO enabled/disabled depends on ENABLE_TESTS
(see the top-level CMakeLists.txt). If ENABLE_TESTS=0 then ThinLTO is
activated. On Linux, building/linking works with or without ThinLTO but
on Mac building/linking the self-extracting compressor binary doesn’t
work if ThinLTO is on. This is quite weird, as a workaround restrict
ThinLTO to Linux.
-------
[185/187] Linking CXX static library base/glibc-compatibility/libglibc-compatibility.a
[186/187] Linking CXX static library contrib/zstd-cmake/lib_zstd.a
[187/187] Linking CXX executable utils/self-extracting-executable/pre_compressor
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/repo/ch4/build
[0/2] Re-checking globbed directories...
[108/108] Linking CXX executable utils/self-extracting-executable/pre_compressor
FAILED: utils/self-extracting-executable/pre_compressor
: && /usr/bin/clang++-16 --target=x86_64-apple-darwin -std=c++20 -fdiagnostics-color=always -Xclang -fuse-ctor-homing -Wno-enum-constexpr-conversion -fsized-deallocation -gdwarf-aranges -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt -fasynchronous-unwind-tables -ffile-prefix-map=/home/ubuntu/repo/ch4=. -falign-functions=32 -mbranches-within-32B-boundaries -stdlib=libc++ -fdiagnostics-absolute-paths -fstrict
-vtable-pointers -Wall -Wextra -Wframe-larger-than=65536 -Weverything -Wpedantic -Wno-zero-length-array -Wno-c++98-compat-pedantic -Wno-c++98-compat -Wno-c++20-compat -Wno-sign-conversion -Wno-implicit-int-conversion -Wno-implicit-int-float-conversion -Wno-ctad-maybe-unsupported -Wno-disabled-macro-expansion -Wno-documentation-unknown-command -Wno-double-promotion -Wno-exit-time-destructors -Wno-float-equal -Wn
o-global-constructors -Wno-missing-prototypes -Wno-missing-variable-declarations -Wno-padded -Wno-switch-enum -Wno-undefined-func-template -Wno-unused-template -Wno-vla -Wno-weak-template-vtables -Wno-weak-vtables -Wno-thread-safety-negative -Wno-enum-constexpr-conversion -Wno-unsafe-buffer-usage -O2 -g -DNDEBUG -O3 -g -gdwarf-4 -flto=thin -fwhole-program-vtables -isysroot /home/ubuntu/repo/ch4/cmake/darwin/..
/toolchain/darwin-x86_64 -mmacosx-version-min=10.15 -Wl,-headerpad_max_install_names --ld-path=/home/ubuntu/cctools/bin/x86_64-apple-darwin-ld -rdynamic -Wl,-U,_inside_main -flto=thin -fwhole-program-vtables utils/self-extracting-executable/CMakeFiles/pre_compressor.dir/compressor.cpp.o -o utils/self-extracting-executable/pre_compressor contrib/zstd-cmake/lib_zstd.a contrib/libcxx-cmake/libcxx.a contrib/lib
cxxabi-cmake/libcxxabi.a -nodefaultlibs -lc -lm -lpthread -ldl && :
clang: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
ld: warning: ignoring file utils/self-extracting-executable/CMakeFiles/pre_compressor.dir/compressor.cpp.o, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0xDE 0xC0 0x17 0x0B 0x00 0x00 0x00 0x00 0x14 0x00 0x00 0x00 0x88 0x3E 0x03 0x00 )
ld: warning: ignoring file contrib/zstd-cmake/lib_zstd.a, building for macOS-x86_64 but attempting to link with file built for macOS-x86_64
ld: warning: ignoring file contrib/libcxxabi-cmake/libcxxabi.a, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x21 0x3C 0x61 0x72 0x63 0x68 0x3E 0x0A 0x23 0x31 0x2F 0x31 0x32 0x20 0x20 0x20 )
ld: warning: ignoring file contrib/libcxx-cmake/libcxx.a, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x21 0x3C 0x61 0x72 0x63 0x68 0x3E 0x0A 0x23 0x31 0x2F 0x31 0x32 0x20 0x20 0x20 )
Undefined symbols for architecture x86_64:
"_main", referenced from:
implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
woboq codebrowser uses clang tooling, which adds clang system includes
(in Linux::AddClangSystemIncludeArgs()), because none of (-nostdinc,
-nobuiltininc) is set.
And later it will complain with -Wpoison-system-directories for added by
itself includes in InitHeaderSearch::AddUnmappedPath(), because they are
starts from one of the following:
- /usr/include
- /usr/local/include
The interesting thing here is that it got broken only after upgrading to
llvm 16 (in #49678), and the reason for this is that clang 15 build has
system includes that does not trigger the warning -
"/usr/lib/clang/15.0.7/include", while clang 16 has
"/usr/include/clang/16.0.4/include"
So let's simply disable this warning, but only for woboq.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
"The -fexperimental-new-pass-manager and -fno-legacy-pass-manager flags
have been removed. These flags have been silently ignored since Clang
15."
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
LTO in Rust produces multiple definition of `rust_eh_personality' (and
few others), and to overcome this --allow-multiple-definition has been
added.
Query for benchmark:
SELECT ignore(BLAKE3(materialize('Lorem ipsum dolor sit amet, consectetur adipiscing elit'))) FROM numbers(1000000000) FORMAT `Null`
upstream : Elapsed: 2.494 sec. Processed 31.13 million rows, 249.08 MB (12.48 million rows/s., 99.86 MB/s.)
upstream + rust lto: Elapsed: 13.56 sec. Processed 191.9 million rows, 1.5400 GB (14.15 million rows/s., 113.22 MB/s.)
llvm BLAKE3 : Elapsed: 3.053 sec. Processed 43.24 million rows, 345.88 MB (14.16 million rows/s., 113.28 MB/s.)
Note, I thought about simply replacing it with BLAKE3 from LLVM, but:
- this will not solve LTO issues for Rust (and in future more libraries
could be added)
- it makes integrating_rust_libraries.md useless (and there is even blog
post)
So instead I've decided to add this quirk (--allow-multiple-definition)
to fix builds.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
This reverts PRs #42470, #47673 and #47744. The problem was that after
the switch to ld64.lld, server binaries build in Debug mode no longer
came up, see the discussion after (*). My attempt to fix it with
`-no_compact_unwind` didn't help.
I also tried
- `-keep_dwarf_unwind`,
- `-unwindlib` (to use the OS unwinder),
- the unwinder in contrib (CMake variable EXCEPTION_HANDLING_LIBRARY)
but w/o success. Just tons of wasted time.
Rolling back to lld as linker on Mac. This will bring back many linker
warnings (#42282) but I rather accept that (temporarily, maybe someone
can figure out how to fix them) than have a broken Debug binary.
(*) https://github.com/ClickHouse/ClickHouse/pull/42470#issuecomment-1312344068
Right now it works for host platforms because of gcc package, that
includes gcc-cross sysroot.
Use bundled sysroot from contrib instead.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Right now it works for host platforms because of gcc package, that
includes gcc-cross sysroot.
Use bundled sysroot from contrib instead.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Public flags, especially -Wxx (i.e. -Wno-XX) can hide some warnings,
that had been added in the main cmake rules of ClickHouse.
This patch had been tested manually with the following patch:
```patch
diff --git a/contrib/jemalloc-cmake/CMakeLists.txt b/contrib/jemalloc-cmake/CMakeLists.txt
index d5ea69d4926..7e79fba0c16 100644
--- a/contrib/jemalloc-cmake/CMakeLists.txt
+++ b/contrib/jemalloc-cmake/CMakeLists.txt
@@ -158,6 +158,7 @@ target_include_directories(_jemalloc SYSTEM PRIVATE
"${CMAKE_CURRENT_BINARY_DIR}/${JEMALLOC_INCLUDE_PREFIX}/jemalloc/internal")
target_compile_definitions(_jemalloc PRIVATE -DJEMALLOC_NO_PRIVATE_NAMESPACE)
+target_compile_options(_jemalloc INTERFACE -Wno-error) # also PUBLIC had been checked
if (CMAKE_BUILD_TYPE_UC STREQUAL "DEBUG")
target_compile_definitions(_jemalloc PRIVATE
```
And cmake gave:
CMake Error at cmake/sanitize_targets.cmake:69 (message):
_jemalloc set INTERFACE_COMPILE_OPTIONS to -Wno-error. This is forbidden.
Call Stack (most recent call first):
cmake/sanitize_targets.cmake:79 (sanitize_interface_flags)
CMakeLists.txt:595 (include)
Fixes: #12447
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
We currently only log a compiler-generated "build id" at startup which
is different for each build. That makes it useless to determine the
exact source code state in tests (e.g. BC test) and from user log files
(e.g. if someone compiled an intermediate version of ClickHouse).
Current log message:
Starting ClickHouse 22.10.1.1 with revision 54467, build id: 6F35820328F89C9F36E91C447FF9E61CAF0EF019, PID 42633
New log message:
Starting ClickHouse 22.10.1.1 (revision 54467, git hash: b6b1f7f763f94ffa12133679a6f80342dd1c3afe, build id: 47B12BE61151926FBBD230DE42F3B7A6652AC482), PID 981813
Right now LINKER_NAME contains version so it does not work:
Aug 27 13:02:10 -- Using linker: /usr/bin/ld.lld-14
Without this patch:
$ time gdb ~/ch/clickhouse/.bld-release/programs/clickhouse -batch
real 1m20.564s
user 1m18.622s
sys 0m1.940s
$ time lldb-13 clickhouse -batch
real 0m9.353s
user 1m31.035s
sys 0m8.346s
With:
$ time gdb clickhouse -batch
real 0m4.769s
user 0m3.969s
sys 0m0.800s
$ time lldb clickhouse -batch
real 0m6.703s
user 0m45.383s
sys 0m4.934s
So by just adding 4.15MiB to the binary size, gdb start up time improved
17 times.
$ objdump -j .gdb_index -h .bld-release/programs/clickhouse
.bld-release/programs/clickhouse: file format elf64-x86-64
Sections:
Idx Name Size VMA LMA File off Algn
40 .gdb_index 00426d74 0000000000000000 0000000000000000 7fb43443 2**0
CONTENTS, READONLY, DEBUGGING
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Without ADDITIONAL_CLEAN_FILES it reports an error:
Cleaning... ninja: error: remove(contrib/llvm/llvm/NATIVE): Directory not empty
ninja: error: remove(/bld/contrib/llvm/llvm/NATIVE): Directory not empty
0 files.
Note, that ADDITIONAL_CLEAN_FILES had been added since cmake 3.15.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Right now cmake add the following options only if USE_STATIC_LIBRARIES
is OFF:
- SPLIT_SHARED_LIBRARIES
- CLICKHOUSE_SPLIT_BINARY
And this breaks the following usage:
$ cmake ..
$ cat > debug-build-cache.cmake
set(USE_STATIC_LIBRARIES OFF CACHE BOOL "")
set(SPLIT_SHARED_LIBRARIES ON CACHE BOOL "")
set(CLICKHOUSE_SPLIT_BINARY ON CACHE BOOL "")
^D
$ cmake -C debug-build-cache.cmake ..
CMake Error at CMakeLists.txt:83 (message):
Defining SPLIT_SHARED_LIBRARIES=1 without USE_STATIC_LIBRARIES=0 has no
effect.
Since with this initial cache we have the following:
- USE_STATIC_LIBRARIES=OFF (because it was already set)
- SPLIT_SHARED_LIBRARIES=ON (was not set before, so new value)
- CLICKHOUSE_SPLIT_BINARY (was not set before, also new value)
Yes this is not the common usage, but it seems that it is pretty easy to
avoid.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
