Commit Graph

965 Commits

Author SHA1 Message Date
Azat Khuzhin
6e9985a94d Remove removed in clang 16 -fexperimental-new-pass-manager
"The -fexperimental-new-pass-manager and -fno-legacy-pass-manager flags
have been removed. These flags have been silently ignored since Clang
15."

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-09 12:22:14 +02:00
Han Fei
64b67b9a37 Merge branch 'master' into hanfei/fiu 2023-05-09 01:14:30 +02:00
robot-clickhouse-ci-2
40922d197f
Merge pull request #49600 from azat/build/rust-lto
RFC: Improve performance of BLAKE3 by 11% by enabling LTO for Rust
2023-05-07 06:14:57 +02:00
Alexey Milovidov
1c007d8bd6 Merge branch 'master' into no-export-dynamic 2023-05-07 04:22:03 +02:00
Alexey Milovidov
d3c3d8b8e4 Remove export of dynamic symbols 2023-05-06 23:52:16 +02:00
Azat Khuzhin
1107988a82 Improve performance of BLAKE3 by 11% by enabling LTO for Rust
LTO in Rust produces multiple definition of `rust_eh_personality' (and
few others), and to overcome this --allow-multiple-definition has been
added.

Query for benchmark:

    SELECT ignore(BLAKE3(materialize('Lorem ipsum dolor sit amet, consectetur adipiscing elit'))) FROM numbers(1000000000) FORMAT `Null`

  upstream           : Elapsed: 2.494 sec. Processed 31.13 million rows, 249.08 MB (12.48 million rows/s., 99.86 MB/s.)
  upstream + rust lto: Elapsed: 13.56 sec. Processed 191.9 million rows, 1.5400 GB (14.15 million rows/s., 113.22 MB/s.)
  llvm BLAKE3        : Elapsed: 3.053 sec. Processed 43.24 million rows, 345.88 MB (14.16 million rows/s., 113.28 MB/s.)

Note, I thought about simply replacing it with BLAKE3 from LLVM, but:
- this will not solve LTO issues for Rust (and in future more libraries
  could be added)
- it makes integrating_rust_libraries.md useless (and there is even blog
  post)

So instead I've decided to add this quirk (--allow-multiple-definition)
to fix builds.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-06 22:28:56 +02:00
Alexey Milovidov
355d5af786 Remove some code 2023-05-06 18:33:45 +02:00
Han Fei
d608ba41e7 try to fix 2023-05-02 00:05:36 +02:00
Robert Schulze
550e430c28
A better way of excluding ISA-L on non-x86
Follow-up to #49256. More 'modern', i.e. uses a CMake TARGET exists
check instead of an internal variable.
2023-05-01 20:42:57 +00:00
Jordi
be702ee584 Disable ISA-L on aarch64 architectures 2023-04-27 12:15:37 +02:00
alesapin
0fcf19efd3 Missed brackets 2023-04-25 15:47:47 +02:00
alesapin
bb21461145 Try to fix with pie 2023-04-25 15:45:23 +02:00
Han Fei
fc101a33cd
Update CMakeLists.txt
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
2023-04-16 16:01:41 +02:00
Han Fei
862c7b43af forbid gwpsan in debug mode to rescue stress tests 2023-04-15 12:54:54 +02:00
Robert Schulze
609b892879
Fold several Clang-related configurations under one if 2023-04-11 16:27:32 +00:00
Robert Schulze
1477887259
Initial removal of logic under COMPILER_GCC 2023-04-11 16:27:32 +00:00
Mikhail f. Shiryaev
7ff582b963
Deprecate ENABLE_CCACHE and replace it by COMPILER_CACHE 2023-03-29 23:33:31 +02:00
Robert Schulze
1845bfc644
Revert change to ld64.lld linker on Mac
This reverts PRs #42470, #47673 and #47744. The problem was that after
the switch to ld64.lld, server binaries build in Debug mode no longer
came up, see the discussion after (*). My attempt to fix it with
`-no_compact_unwind` didn't help.

I also tried
- `-keep_dwarf_unwind`,
- `-unwindlib` (to use the OS unwinder),
- the unwinder in contrib (CMake variable EXCEPTION_HANDLING_LIBRARY)
but w/o success. Just tons of wasted time.

Rolling back to lld as linker on Mac. This will bring back many linker
warnings (#42282) but I rather accept that (temporarily, maybe someone
can figure out how to fix them) than have a broken Debug binary.

(*) https://github.com/ClickHouse/ClickHouse/pull/42470#issuecomment-1312344068
2023-03-27 15:26:54 +02:00
Robert Schulze
5d5308e060
CMake cleanup: Remove configuration of CMAKE_SHARED_LINKER_FLAGS
Follow-up to #44828
2023-03-26 17:59:39 +00:00
Robert Schulze
9b3e57070f
Fix broken cross-compiled macos builds 2023-03-20 08:33:33 +00:00
Robert Schulze
1499b77592
Restore native macos build 2023-03-19 19:59:22 +01:00
Han Fei
722e6bb2e8
Merge pull request #47680 from ClickHouse/rs/uncruftify-build
CMake: Remove further cruft from build
2023-03-17 22:58:46 +01:00
Robert Schulze
23281c7348
Merge pull request #47669 from ClickHouse/rs/more-gold-removal
Follow-up to #47660: Further removal of gold linker support
2023-03-17 14:58:23 +01:00
Robert Schulze
f72a337074
Remove cruft from build
No need to check compiler flags, clang >= 15 supports all of them.
2023-03-17 13:44:04 +00:00
Robert Schulze
f8980c582e
CMake: More removal of gold linker (follow-up to #47660)
+ fix a linker warning
2023-03-17 11:01:46 +00:00
Robert Schulze
22d496e8e6
CMake: Build ClickHouse w/o GNU extensions 2023-03-16 15:02:52 +00:00
Robert Schulze
348a40aa5d
Compile with C++23 2023-03-10 19:11:58 +00:00
Amos Bird
7b4f489837
Update flags 2023-02-28 22:45:33 +08:00
bkuschel
55e9322612 Fixes for OpenSSL and s390x 2023-02-17 16:20:21 -05:00
Suzy Wang
20f8cf233b remove comma 2023-02-09 19:53:00 +00:00
Suzy Wang
64313d7020
Merge branch 'master' into z-build-0120 2023-02-09 14:49:22 -05:00
Suzy Wang
716d2c4ffb Update as suggested 2023-02-02 20:41:06 +00:00
Suzy Wang
19d26828a0 s390x build support 2023-01-20 21:16:55 +00:00
Han Fei
29b20775f7 not compile gwp in non-linux environment 2023-01-17 23:06:44 +01:00
Robert Schulze
27f5aad49e
What happens if I remove 156 lines of code? 2023-01-03 18:51:16 +00:00
Robert Schulze
cfb6feffde
What happens if I remove these 139 lines of code? 2023-01-03 18:35:31 +00:00
Alexey Milovidov
4651a538a8 Fix extremely slow stack traces in debug build 2022-12-24 20:31:04 +01:00
Azat Khuzhin
28737a503c Configure rustc compiler properly
Right now it works for host platforms because of gcc package, that
includes gcc-cross sysroot.

Use bundled sysroot from contrib instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-14 20:57:29 +01:00
Azat Khuzhin
67fa185611
Revert "Builtin skim" 2022-12-14 17:17:19 +03:00
Alexey Milovidov
df17d1b71b
Merge branch 'master' into builtin-skim 2022-12-13 03:33:42 +03:00
Azat Khuzhin
3dfc86d3f8 Enable -Werror unconditionally (will enable it for FreeBSD)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-12 08:05:57 +01:00
Azat Khuzhin
8570f15688 Configure rustc compiler properly
Right now it works for host platforms because of gcc package, that
includes gcc-cross sysroot.

Use bundled sysroot from contrib instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-11 15:52:00 +01:00
Boris Kuschel
abcc114768
Scarier OpenSSL options 2022-12-08 04:59:49 -08:00
Boris Kuschel
cc3314acbe
Merge branch 'master' into openssl_support 2022-12-08 07:40:43 -05:00
Nikita Taranov
b81ad6aaf7
Add google benchmark to contrib (#43779)
* add google benchmark to contrib

* rework integer_hash_tables_and_hashes

* update readme

* keep benchmarks near the benchmarked code

* fix fasttests build

* rm old target

* fix
2022-12-08 13:38:08 +01:00
Boris Kuschel
853da17f76 Fix merge, default value 2022-12-07 13:22:46 -05:00
Boris Kuschel
623f23f55d OpenSSL cmake changes 2022-12-07 13:22:46 -05:00
Alexey Milovidov
75dbd6828a
Merge pull request #43082 from Algunenano/werror_base
Add global flags to base/ libraries
2022-11-22 19:18:44 +01:00
Azat Khuzhin
c3f3078884 Sanitize thirdparty libraries for public flags
Public flags, especially -Wxx (i.e. -Wno-XX) can hide some warnings,
that had been added in the main cmake rules of ClickHouse.

This patch had been tested manually with the following patch:

    ```patch
    diff --git a/contrib/jemalloc-cmake/CMakeLists.txt b/contrib/jemalloc-cmake/CMakeLists.txt
    index d5ea69d4926..7e79fba0c16 100644
    --- a/contrib/jemalloc-cmake/CMakeLists.txt
    +++ b/contrib/jemalloc-cmake/CMakeLists.txt
    @@ -158,6 +158,7 @@ target_include_directories(_jemalloc SYSTEM PRIVATE
         "${CMAKE_CURRENT_BINARY_DIR}/${JEMALLOC_INCLUDE_PREFIX}/jemalloc/internal")

     target_compile_definitions(_jemalloc PRIVATE -DJEMALLOC_NO_PRIVATE_NAMESPACE)
    +target_compile_options(_jemalloc INTERFACE -Wno-error) # also PUBLIC had been checked

     if (CMAKE_BUILD_TYPE_UC STREQUAL "DEBUG")
         target_compile_definitions(_jemalloc PRIVATE
    ```

And cmake gave:

    CMake Error at cmake/sanitize_targets.cmake:69 (message):
      _jemalloc set INTERFACE_COMPILE_OPTIONS to -Wno-error.  This is forbidden.
    Call Stack (most recent call first):
      cmake/sanitize_targets.cmake:79 (sanitize_interface_flags)
      CMakeLists.txt:595 (include)

Fixes: #12447
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-16 07:20:24 +01:00
Raúl Marín
5caba7000a Try a different approach 2022-11-15 17:53:24 +01:00
Raúl Marín
764020423b Include global flags in default libs 2022-11-10 11:46:26 +01:00
Lloyd-Pottiger
7419a3bd58 fix
Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>
2022-10-29 18:58:11 +08:00
Robert Schulze
da5a2e2db0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup
Physical merge conflicts:
- src/Common/ZooKeeper/ZooKeeperImpl.cpp
- src/Core/config_core.h.in
- src/Functions/FunctionsAES.h
- src/Functions/config_functions.h.in
- src/configure_config.cmake

Logical merge conflicts:
- Functions/tryDecrypt.cpp
2022-10-06 08:43:25 +00:00
Alexey Milovidov
25d13b89e9
Merge pull request #41142 from MeenaRenganathan22/OpenSSL_InHouse
Changes to support the OpenSSL In-house build.
2022-10-05 23:02:04 +03:00
Alexey Milovidov
a46206571e
Update CMakeLists.txt 2022-10-05 23:00:49 +03:00
Alexey Milovidov
89b1a5188f
Update CMakeLists.txt 2022-10-05 22:59:44 +03:00
BoloniniD
3a3ca51980 Fixes after review 2022-10-02 20:52:05 +03:00
root
7e4edfeb5a Merge branch 'master' of https://github.com/MeenaRenganathan22/ClickHouse into OpenSSL_InHouse 2022-09-30 15:50:46 -07:00
root
bb1771e159 Update the macro ENABLE_CH_BUNDLE_BORINGSSLL to ENABLE_EXTERNAL_OPENSSL 2022-09-30 10:35:49 -07:00
BoloniniD
55c79230b3 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-09-29 23:53:25 +03:00
Robert Schulze
06507c40de
${ConfigIncludePath} --> ${CONFIG_INCLUDE_PATH} 2022-09-28 08:28:47 +00:00
Robert Schulze
823d8fb6cd
Move git calls back into git.cmake
+ renamed the file from originally "git_status.cmake" to "git.cmake"
  (because we not longer run only "git status")
2022-09-27 11:45:58 +00:00
Robert Schulze
ec35ff9cd6
Log git hash during startup
We currently only log a compiler-generated "build id" at startup which
is different for each build. That makes it useless to determine the
exact source code state in tests (e.g. BC test) and from user log files
(e.g. if someone compiled an intermediate version of ClickHouse).

Current log message:
  Starting ClickHouse 22.10.1.1 with revision 54467, build id: 6F35820328F89C9F36E91C447FF9E61CAF0EF019, PID 42633

New log message:
  Starting ClickHouse 22.10.1.1 (revision 54467, git hash: b6b1f7f763f94ffa12133679a6f80342dd1c3afe, build id: 47B12BE61151926FBBD230DE42F3B7A6652AC482), PID 981813
2022-09-26 14:12:36 +00:00
BoloniniD
0df426d329 Corrosion fixes + review fixes 2022-09-16 00:05:21 +03:00
BoloniniD
147dfac11e Try using Corrosion 2022-09-12 23:05:41 +03:00
Meena Renganathan
51d6611b96 Committing the ClickHouse core changes and other libraries to support OpenSSL. BoringSSL is still set as default 2022-09-12 09:05:38 -07:00
BoloniniD
e8bcbcd016
Merge branch 'master' into BLAKE3 2022-09-09 11:48:31 +03:00
BoloniniD
d635c3f9be Disable BLAKE3 for aarch64-darwin 2022-09-06 21:22:40 +03:00
Azat Khuzhin
2d3fb36a62 Fix debug symbols (Add a quirk to force clang emit .debug_aranges with ThinLTO)
Wrap a linker into a script that will add some settings (`-mllvm
-generate-arange-section`) in case of ThinLTO to emit `.debug_aranges`
symbols.

Dicussion in the LLVM can be found here [1].

  [1]: https://discourse.llvm.org/t/clang-does-not-produce-full-debug-aranges-section-with-thinlto/64898

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-02 23:22:25 +02:00
Alexander Tokmakov
69387acffa
Revert "Support for DWARF-5 in in house DWARF parser" 2022-08-29 14:25:53 +03:00
Alexey Milovidov
8448242785
Merge pull request #40716 from azat/fix-gdb-index
Fix gdb index (makes gdb 17x faster)
2022-08-29 00:13:58 +03:00
Alexey Milovidov
ae97e880d7
Merge pull request #40710 from azat/DWARF-5
Support for DWARF-5 in in house DWARF parser
2022-08-29 00:13:33 +03:00
Alexey Milovidov
ddff5e9145
Merge pull request #40712 from ClickHouse/ci-cmake-self-extractable
cmake: Disable default ENABLE_CLICKHOUSE_SELF_EXTRACTING
2022-08-28 17:12:19 +03:00
Azat Khuzhin
19178a8f18 Fix gdb index (makes gdb 17x faster)
Right now LINKER_NAME contains version so it does not work:

    Aug 27 13:02:10 -- Using linker: /usr/bin/ld.lld-14

Without this patch:

    $ time gdb ~/ch/clickhouse/.bld-release/programs/clickhouse -batch

    real    1m20.564s
    user    1m18.622s
    sys     0m1.940s

    $ time lldb-13 clickhouse -batch
    real    0m9.353s
    user    1m31.035s
    sys     0m8.346s

With:

    $ time gdb clickhouse -batch
    real    0m4.769s
    user    0m3.969s
    sys     0m0.800s

    $ time lldb clickhouse -batch
    real    0m6.703s
    user    0m45.383s
    sys     0m4.934s

So by just adding 4.15MiB to the binary size, gdb start up time improved
17 times.

    $ objdump -j .gdb_index -h .bld-release/programs/clickhouse

    .bld-release/programs/clickhouse:     file format elf64-x86-64

    Sections:
    Idx Name          Size      VMA               LMA               File off  Algn
     40 .gdb_index    00426d74  0000000000000000  0000000000000000  7fb43443  2**0
                      CONTENTS, READONLY, DEBUGGING

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-28 15:01:34 +02:00
Azat Khuzhin
7f65075dcb Fix usage of SHA1 for BuildID
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-28 09:02:10 +02:00
Yakov Olkhovskiy
b5bb1be15a pass ENABLE_CLICKHOUSE_SELF_EXTRACTING to native build 2022-08-27 23:29:32 +00:00
Azat Khuzhin
444acb96c8 Do not pin DWARF-4 during build
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-27 22:28:15 +02:00
Alexey Milovidov
91ed3ebe9a Remove the remaining parts of "unbundled" build 2022-08-27 02:18:47 +02:00
Robert Schulze
cadfdc40ef
fix: CMake option ENABLE_CCACHE is not propagated to native build
- fixes #40439
2022-08-23 18:59:51 +00:00
Alexey Milovidov
a716d78264 Support build with clang-16 2022-08-13 06:06:01 +02:00
BoloniniD
29084d92d8 Fix CMake for ENABLE_RUST 2022-08-02 20:44:12 +03:00
BoloniniD
b161773f71 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-08-02 20:25:25 +03:00
Robert Schulze
d834d9e137
Minor: Better variable name output 2022-07-31 07:44:25 +00:00
Alexey Milovidov
36e6500e54
Merge branch 'master' into BLAKE3 2022-07-30 23:14:05 +03:00
Robert Schulze
199e254777
Merge remote-tracking branch 'origin/master' into no-split-binary 2022-07-28 15:54:22 +00:00
Yakov Olkhovskiy
da42ec7d28 some refactoring 2022-07-24 21:08:43 -04:00
Alexey Milovidov
071374b152 Remove SPLIT_BINARY 2022-07-24 01:15:54 +02:00
Yakov Olkhovskiy
e7438cc433
Merge branch 'master' into cmake-self-extracting-executable 2022-07-21 00:25:08 -04:00
Yakov Olkhovskiy
b766552481 add native build for cross-compilation 2022-07-20 23:09:05 -04:00
Azat Khuzhin
0e95dba3ef Fix make clean (due to crosscompile of llvm)
Without ADDITIONAL_CLEAN_FILES it reports an error:

    Cleaning... ninja: error: remove(contrib/llvm/llvm/NATIVE): Directory not empty
    ninja: error: remove(/bld/contrib/llvm/llvm/NATIVE): Directory not empty
    0 files.

Note, that ADDITIONAL_CLEAN_FILES had been added since cmake 3.15.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-20 17:34:10 +03:00
Azat Khuzhin
21fa5e6ece Add check for CLICKHOUSE_SPLIT_BINARY=1 with USE_STATIC_LIBRARIES=1
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-18 11:42:49 +03:00
Azat Khuzhin
30018ce5d3 Fix running cmake with predefined cache (for development only)
Right now cmake add the following options only if USE_STATIC_LIBRARIES
is OFF:
- SPLIT_SHARED_LIBRARIES
- CLICKHOUSE_SPLIT_BINARY

And this breaks the following usage:

    $ cmake ..
    $ cat > debug-build-cache.cmake
    set(USE_STATIC_LIBRARIES              OFF CACHE BOOL "")
    set(SPLIT_SHARED_LIBRARIES            ON  CACHE BOOL "")
    set(CLICKHOUSE_SPLIT_BINARY           ON  CACHE BOOL "")
    ^D
    $ cmake -C debug-build-cache.cmake ..
    CMake Error at CMakeLists.txt:83 (message):
      Defining SPLIT_SHARED_LIBRARIES=1 without USE_STATIC_LIBRARIES=0 has no
      effect.

Since with this initial cache we have the following:

- USE_STATIC_LIBRARIES=OFF (because it was already set)
- SPLIT_SHARED_LIBRARIES=ON (was not set before, so new value)
- CLICKHOUSE_SPLIT_BINARY (was not set before, also new value)

Yes this is not the common usage, but it seems that it is pretty easy to
avoid.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-17 12:18:24 +03:00
Alexey Milovidov
d8b1b44a29
Merge branch 'master' into align-branch-within-32b 2022-07-13 16:53:34 +03:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_"
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.

As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.

In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).

(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
2022-07-11 19:36:18 +02:00
Azat Khuzhin
e8f5cd3c68 Add separate option to omit symbols from heavy contrib
Sometimes it is useful to build contrib with debug symbols for further
debugging.

With everything turned ON (i.e. debug build) I got 3.3GB vs 3.0GB w/o
this patch, 9% bloat, thoughts about this is this OK or not for you, if
not STRIP_DEBUG_SYMBOLS_HEAVY_CONTRIB can be OFF by default (regardless
of build type).

P.S. aws debug symbols adds just 1.7%.
v2: rename STRIP_HEAVY_DEBUG_SYMBOLS
v3: OMIT_HEAVY_DEBUG_SYMBOLS
v4: documentation had been removed
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-02 06:32:03 +03:00
Robert Schulze
bb358617e1
Better naming for stuff related to splitted debug symbols
The previous name was slightly misleading, e.g. it is not about
"intalling stripped binaries" but about splitting debug symbols from the
binary.
2022-06-30 23:41:27 +02:00
BoloniniD
6ddcec0906
Merge branch 'master' into BLAKE3 2022-06-28 16:53:06 +03:00
Robert Schulze
bc46cef63c
Minor follow-up
- change ELF section name to ".clickhouse.hash" (lowercase seems
  standard)

- more expressive/concise integrity check messages at startup
2022-06-14 08:52:13 +00:00
Robert Schulze
e25fbb9292
Log official/non-official build and integrated/separate debug symbols
The goal is to find out why some of the binaries with official name
(BuilderDebRelease, BuilderBinRelease) produced by CI still contain no
hash section.

(also, strip and objcopy are mandatory tools and we don't need to check
for their existence at this point)
2022-06-13 12:34:09 +00:00
BoloniniD
b05ee41d25 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-06-06 16:03:10 +03:00