First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.
That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
- It was noticed that in (*), the crashstack says "There is no
information about the reference checksum."
- The binaries are pulled via docker hub and upon inspection they indeed
lack the hash embedded as ELF section ".note.ClickHouse.hash" in the
clickhouse binary. This is weird because docker hub images are
"official" builds which should trigger the hash embedding.
- Turns out that the docker hub binaries are also stripped which was too
aggressive. We now no longer remove sections ".comment" and ".note"
which are anyways only 140 bytes in size, i.e. binary size still goes
down (on my stystem) from 2.1 GB to 0.47 + 0.40 GB binary + dbg info.
(*) https://playground.lodthe.me/ba75d494-95d1-4ff6-a0ad-60c138636c9b
- before: usr/lib/debug/usr/bin/clickhouse.debug/clickhouse.debug
- after : usr/lib/debug/usr/bin/clickhouse.debug
Note, clickhouse_make_empty_debug_info_for_nfpm() is fine.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Globbing generally misses to pick up files which were added/deleted
after CMake's configure. This is a nuissance but can be alleviated using
CONFIGURE_DEPENDS (available since CMake 3.12) which adds a check for
new/deleted files before each compile and - if necessary - restarts the
configuration. On my system, the check takes < 0.1 sec.
(Side note: CONFIGURE_DEPENDS is not guaranteed to work accross all
generators, but at least it works for Ninja which everyone @CH seems to
use.)
The check activated ccache unconditionally for all non-Clang compilers
(= GCC) while allowing ancient ccache versions for these. Perhaps there
was a reason for that in the past but it's simpler to only require a
minimum ccache version.
To simplify further, also require at least ccache 3.3 (released in 2016)
instead of 3.2.1 (released in 2014).
The compiler launcher (ccache, distcc) can be set externally via
-DCMAKE_CXX_COMPILER_LAUNCHER=<tool>. We previously silently ignored
this setting and continued without any launcher (e.g. ccache). Changed
this to now respect the externally specified launcher.
WEVERYTHING enables on Clang literally every warning. People on the
internet are divided if this is a good thing or not but ClickHouse
compiles with -Weverything + some exceptions for noisy warnings since at
least a year.
I tried to build with WEVERYTHING = OFF and the build was badly broken.
It seems nobody actually turns WEVERYTHING off. Actually, why would one
if the CI builds (configured with WEVERYTHING = ON) potentially generate
errors not generated in local development.
To simplify the build scripts and to remove the need to maintain two
sets of compiler warnings, I made WEVERYTHING the default and threw
WEVERYTHING = OFF out.
On Darwin, the build script tries to
1. use llvm-objcopy/llvm-strip from $PATH,
2. if not found by 1., use standard objcopy/strip from $PATH
The brew install instructions recommends to set $PATH to brew's binary
dir, so 2. will find something (assuming binutils is installed from
brew). If $PATH additionally points to brew's LLVM binary dir (which is
different from brew's binary dir), 1. will find the llvm versions of the
tools.
This commit removes additional logic which repeats above steps in a more
implicit way by calling brew internally and figuring out the paths once
more if 1. and 2. cannot find them in the $PATH. This removes
duplication and simplifies the script. Maybe it even helps with
reproducibility.
- previous XCode 10.2 / Clang 7.0 was horribly outdated
- XCode 12.0 corresponds to the minimally required vanilla Clang version 12.0
- remove passing of "-fchar8_t" flag (with Clang >= 9, it is part of -std=c++20)
- Properly handle the case that we are on an unsupported but unlisted
arch, e.g. mips. Before, we would simply continue
configuration/compilation with no architecture set.
- CMake variable "ARCH_ARM" could in theory be replaced by
"ARCH_AARCH64". This would need refactoring in dependent CMakeLists,
therefore not doing it now.
Really just for convenience so the developer can choose appropriate
values for CMake options PARALLEL_COMPILE_JOBS and PARALLEL_LINK_JOBS.
Might also help to find out why builds are slow.
