* Replicate poco into base/poco/
* De-register poco submodule
* Build poco from ClickHouse
* Exclude poco from stylecheck
* Exclude poco from whitespace check
* Exclude poco from typo check
* Remove x bit from sources/headers (the style check complained)
* Exclude poco from duplicate include check
* Fix fasttest
* Remove contrib/poco-cmake/*
* Simplify poco build descriptions
* Remove poco stuff not used by ClickHouse
* Glob poco sources
* Exclude poco from clang-tidy
I decided to bind fuzzy to a regular C-R, so that everyone will notice
it, and keep an old behaviour for expect tests, or if someone do not
need fuzzy search (or want blasignly fast search, since skim is slightly
slower).
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Note, that it can the fail the client if the skim itself will fail,
however I haven't seen it panicd, so let's try.
P.S. about adding USE_SKIM into configure header instead of just compile
option for target, it is better, because it allows not to recompile lots
of C++ headers, since we have to add skim library as PUBLIC. But anyway
this will be resolved in a different way, but separatelly.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Note, that it can the fail the client if the skim itself will fail,
however I haven't seen it panicd, so let's try.
P.S. about adding USE_SKIM into configure header instead of just compile
option for target, it is better, because it allows not to recompile lots
of C++ headers, since we have to add skim library as PUBLIC.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
I found few difference in sk/fzf and after this patch I think that it
should behave better.
So the following commands will be used by default:
- fzf --read0 --tac --tiebreak=index --height=30%
- sk --read0 --tac --tiebreak=-score --height=30%
Those two more or less allows true fuzzy reserve search.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Because safeExit() does not includes header with defines, it does not
know about THREAD_SANITIZER.
And it also fixes Azure blob storage, actually everything is fine with
the sdk itself, the problem is only in TSan that intercepts _exit() and
report leak, even thoug that tread will be joined later.
Refs: #23056 (#23616)
Fixes: #38474Closes: #42640Fixes: #42638Fixes: #34988
Cc: @alexey-milovidov, @tavplubix
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
This is like Alt-# in readline, it is useful when you need to look
something else, and need to save current query/command somewhere, and
commented lin the history is a good option.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Remove strange code
* Even more code removal
* Fix style
* Remove even more code
* Simplify code by making it slower
* Attempt to do something
* Attempt to do something
* Well do something with this horrible trash
* Add a test
clang-tidy now also checks code in header files. Because the analyzer
finds tons of issues, activate the check only for directory "base/" (see
file ".clang-tidy"). All other directories, in particular "src/" are
left to future work.
While many findings were fixed, some were not (and suppressed instead).
Reasons for this include: a) the file is 1:1 copypaste of a 3rd-party
lib (e.g. pcg_extras.h) and fixing stuff would make upgrades/fixes more
difficult b) a fix would have broken lots of using code
getauxval() from glibc-compatibility did not work always correctly:
- It does not work after setenv(), and this breaks vsyscalls,
like sched_getcpu() [1] (and BaseDaemon.cpp always set TZ if timezone
is defined, which is true for CI [2]).
Also note, that fixing setenv() will not fix LSan,
since the culprit is getauxval()
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1163404
[2]: ClickHouse#32928 (comment)
- Another think that is definitely broken is LSan (Leak Sanitizer), it
relies on worked getauxval() but it does not work if __environ is not
initialized yet (there is even a commit about this).
And because of, at least, one leak had been introduced [3]:
[3]: ClickHouse#33840
Fix this by using /proc/self/auxv with fallback to environ solution to
make it compatible with environment that does not allow reading from
auxv (or no procfs).
v2: add fallback to environ solution
v3: fix return value for __auxv_init_procfs()
(cherry picked from commit f187c3499a)
v4: more verbose message on errors, CI founds [1]:
AUXV already has value (529267711)
[1]: https://s3.amazonaws.com/clickhouse-test-reports/39103/2325f7e8442d1672ce5fb43b11039b6a8937e298/stress_test__memory__actions_.html
v5: break at AT_NULL
v6: ignore AT_IGNORE
v7: suppress TSan and remove superior check to avoid abort() in case of race
v8: proper suppressions (not inner function but itself)
Refs: #33957
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
getauxval() from glibc-compatibility did not work always correctly:
- It does not work after setenv(), and this breaks vsyscalls,
like sched_getcpu() [1] (and BaseDaemon.cpp always set TZ if timezone
is defined, which is true for CI [2]).
Also note, that fixing setenv() will not fix LSan,
since the culprit is getauxval()
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1163404
[2]: ClickHouse#32928 (comment)
- Another think that is definitely broken is LSan (Leak Sanitizer), it
relies on worked getauxval() but it does not work if __environ is not
initialized yet (there is even a commit about this).
And because of, at least, one leak had been introduced [3]:
[3]: ClickHouse#33840
Fix this by using /proc/self/auxv with fallback to environ solution to
make it compatible with environment that does not allow reading from
auxv (or no procfs).
v2: add fallback to environ solution
v3: fix return value for __auxv_init_procfs()
Refs: #33957
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.
As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.
In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).
(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
- TSA is a static analyzer build by Google which finds race conditions
and deadlocks at compile time.
- It works by associating a shared member variable with a
synchronization primitive that protects it. The compiler can then
check at each access if proper locking happened before. A good
introduction are [0] and [1].
- TSA requires some help by the programmer via annotations. Luckily,
LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
std::shared_mutex and std::scoped_lock. This commit enables them
(--> contrib/libcxx-cmake/CMakeLists.txt).
- Further, this commit adds convenience macros for the low-level
annotations for use in ClickHouse (--> base/defines.h). For
demonstration, they are leveraged in a few places.
- As we compile with "-Wall -Wextra -Weverything", the required compiler
flag "-Wthread-safety-analysis" was already enabled. Negative checks
are an experimental feature of TSA and disabled
(--> cmake/warnings.cmake). Compile times did not increase noticeably.
- TSA is used in a few places with simple locking. I tried TSA also
where locking is more complex. The problem was usually that it is
unclear which data is protected by which lock :-(. But there was
definitely some weird code where locking looked broken. So there is
some potential to find bugs.
*** Limitations of TSA besides the ones listed in [1]:
- The programmer needs to know which lock protects which piece of shared
data. This is not always easy for large classes.
- Two synchronization primitives used in ClickHouse are not annotated in
libcxx:
(1) std::unique_lock: A releaseable lock handle often together with
std::condition_variable, e.g. in solve producer-consumer problems.
(2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
considered a design flaw + typically it is slower than a standard
mutex. In this commit, one std::recursive_mutex was converted to
std::mutex and annotated with TSA.
- For free-standing functions (e.g. helper functions) which are passed
shared data members, it can be tricky to specify the associated lock.
This is because the annotations use the normal C++ rules for symbol
resolution.
[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.
__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
Replxx: When disabled via -DENABLE_LIBRARIES=0 or -DENABLE_REPLXX (the
latter was undocumented) the build broke because replxx symbols were
used since [0] in header LineReader.h. This header should in theory
stay clean of replxx but doesn't for efficiency reasons.
This change makes compilation of replxx mandatory. As replxx is quite
small, I guess this is okay. (The alternative is to litter the code
with ifdefs for non-replxx and a replxx paths.)
[0] https://github.com/ClickHouse/ClickHouse/pull/33201
