Commit Graph

132 Commits

Author SHA1 Message Date
Alexander Tokmakov
3f6594f4c6 forbid old ctor of Exception 2023-01-23 22:18:05 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449)
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
2023-01-24 00:13:58 +03:00
Antonio Andelic
fbda86ddc8 PR fixes 2022-11-30 13:24:08 +01:00
Antonio Andelic
a6f38cb1cc
Merge branch 'master' into keeper-prometheus 2022-11-24 12:12:08 +01:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus 2022-11-21 09:10:10 +01:00
Azat Khuzhin
eb3d21831d Provide full stacktrace in case of uncaught exception during server start
Without it, it is hard to understand where the problem is.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-18 13:24:57 +01:00
Azat Khuzhin
953457de28 Remove POCO_CLICKHOUSE_PATCH
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-10 22:41:26 +01:00
Antonio Andelic
8aeb406d82 Fix standalone build 2022-11-10 10:49:00 +01:00
Antonio Andelic
36d763011b Compile standalone keeper 2022-11-10 09:23:52 +01:00
Antonio Andelic
a0c12c2428 Define LOGICAL_ERROR 2022-11-09 16:06:25 +01:00
Antonio Andelic
1ed3930809 Add more metrics for Keeper 2022-11-09 15:51:41 +01:00
Antonio Andelic
c92ae5a385 Add suport for AsyncMetrics in standalone Keeper 2022-11-09 13:46:03 +01:00
Antonio Andelic
e82a5d43b5 Extract HTTPContext for HTTPServer 2022-11-09 09:02:04 +01:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Antonio Andelic
7c7395575f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-17 11:02:36 +00:00
Robert Schulze
fd86829824
Consolidate config_core.h into config.h
Less duplication, less confusion ...
2022-09-28 13:31:57 +00:00
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH}
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Robert Schulze
588a5e5a42
Simplify a bit 2022-09-27 07:47:18 +00:00
Robert Schulze
99725e68d1
Fix standalone keeper build 2022-09-26 19:28:27 +00:00
Robert Schulze
9711950c35
Fix build 2022-09-26 15:05:36 +00:00
Antonio Andelic
eefbbf53e8 Extract S3 logic 2022-09-21 11:53:54 +00:00
Dmitry Novik
29b5d023ee Replace WITH_TEXT_LOG by WITHOUT_TEXT_LOG 2022-09-08 19:16:37 +02:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_"
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.

As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.

In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).

(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
2022-07-11 19:36:18 +02:00
Antonio Andelic
37f799550b
Merge pull request #38072 from lingpeng0314/master
Add Keeper related monitoring data
2022-07-10 09:39:57 +02:00
lingpeng0314
cca94f01ae add function tryGetKeeperDispatcher for context
- to make sure other case will get error if dispatcher not initialized
2022-07-04 13:35:08 +08:00
lingpeng0314
03a5ce542d Remove Exception in getKeeperDispatcher
- to handle the nil case by caller
2022-07-01 11:27:40 +08:00
Robert Schulze
bb358617e1
Better naming for stuff related to splitted debug symbols
The previous name was slightly misleading, e.g. it is not about
"intalling stripped binaries" but about splitting debug symbols from the
binary.
2022-06-30 23:41:27 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA)
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
2022-06-20 16:13:25 +02:00
lingpeng0314
1d463718ca Change getKeeperDispatcher error type 2022-06-20 17:11:15 +08:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
Antonio Andelic
cf022542bd Merge branch 'master' into keeper-recovery-mode 2022-04-26 06:50:02 +00:00
Antonio Andelic
4e389d8df1 Rename to hostname, add tests 2022-04-22 08:16:14 +00:00
Antonio Andelic
6ac3784af7 Add config example 2022-04-21 13:49:36 +00:00
Antonio Andelic
70f3d3a863 Merge branch 'master' into keeper-recovery-mode 2022-04-19 12:32:48 +00:00
Antonio Andelic
3e77340a81
Allow clickhouse-keeper to serve 4-letter commands before quorum (#35992)
* Use async start for clichkouse-keeper

* Return non active server for 4lw commands if quorum not achieved
2022-04-19 14:03:00 +02:00
Antonio Andelic
cd548aeb30 Refactor test for multiple cluster size 2022-04-19 08:08:36 +00:00
Antonio Andelic
6c3bf0a5d3 Format files 2022-04-19 08:08:36 +00:00
Antonio Andelic
0e1ba927bd Add argument for force recovery 2022-04-19 08:08:35 +00:00
alesapin
c95a9971bc Merge branch 'master' into clickhouse-keeper 2022-03-25 11:22:57 +01:00
alesapin
e790a73081 Simplify strip for new packages 2022-03-23 15:14:30 +01:00
alesapin
a4ab73619f
Fix UBSan build 2022-03-22 15:05:31 +01:00
alesapin
96c0e9fddf Better cmake 2022-03-11 15:47:07 +01:00
alesapin
c5a456802f Fix build 2022-03-04 19:39:11 +01:00
alesapin
0eb7d28192 Building small keeper binary 2022-03-03 21:27:46 +01:00
Azat Khuzhin
798da0c314 Introduce safeExit() helper (_exit() compatible with TSan)
v2: add missing defines.h header
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-02 22:17:17 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-01 14:30:03 +03:00
alesapin
b8c6481d4c Fix clickhouse server start when replicated access storage depend on keeper 2022-01-25 16:40:12 +03:00
JackyWoo
569ce62e8d use session_timeout as session timeout uper bound 2021-12-30 17:18:51 +08:00
JackyWoo
19b5394be8 fix tests 2021-12-30 15:49:48 +08:00
JackyWoo
d35e5f8319 add lower bound session timeout to keeper 2021-12-29 20:59:01 +08:00
Kevin Michel
ffc1fca296
Start/stop servers when listen_host/*_port changes
This allows starting and stopping separately each protocol server
without restarting ClickHouse.

This also allows adding or removing `listen_host` entries, which
start and stops servers for all enabled ports.

When stopping a server, the listening socket is immediately closed
(and available for another server).

Protocols with persistent connections try to wait for any currently
running query to finish before closing the connection, but idle
connection are closed quickly (depending on how often the protocol
is polled).

An extra ProfileEvent is added, `MainConfigLoads`, it is
incremented every time the configuration is reloaded. This helps
when trying to assess whether the new configuration was applied.
2021-12-24 08:26:02 +01:00
Azat Khuzhin
57c027be0e Remove arcadia build support 2021-12-11 21:25:23 +03:00
alesapin
dc6afadea3 Fix default keeper port 2021-11-25 14:01:30 +03:00
alesapin
5f3eca4cd1 Merge branch 'master' into JackyWoo_add_4_letter_words_commands 2021-11-17 12:59:20 +03:00
Tomáš Hromada
445b0ba7cc
Expose all GlobalThreadPool config params (#31285) 2021-11-12 16:24:47 +03:00
JackyWoo
a60663e33d add 4lw commands to keeper 2021-10-27 20:26:42 +08:00
alesapin
18cceedc00 Fix build and update for clickhouse-keeper 2021-10-19 17:29:49 +03:00
alesapin
6d24ca4c3a Review fixes 2021-10-18 12:13:24 +03:00
alesapin
90ff7f05fd Start keeper asynchronously if has connection to other nodes 2021-10-14 13:21:41 +03:00
Azat Khuzhin
a5a2c5ef8a Move KEEPER_DEFAULT_PATH into separate header 2021-10-03 14:34:03 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Alexey Milovidov
e513a5db32 Change <yandex> to <clickhouse> in configs 2021-09-20 01:38:53 +03:00
alesapin
47b995467e
Merge pull request #28080 from ClickHouse/some_renames_in_keeper
Clickhouse-keeper: renames and comments
2021-08-26 10:16:21 +03:00
alesapin
ea5fbbedec Fix clickhouse keeper jepsen tests 2021-08-25 19:15:56 +03:00
alesapin
95cf0634dc Clickhouse-keeper: renames and comments 2021-08-24 15:30:31 +03:00
Alexander Tokmakov
31d75c9c38 fix split build 2021-08-18 15:15:31 +03:00
Alexander Tokmakov
4d71f65082 fix build 2021-08-17 16:24:14 +03:00
Alexander Tokmakov
9e9fa043ca minor improvements, add test 2021-08-16 21:30:53 +03:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Benjamin Naecker
c3506bf16d Adds a better way to include binary resources
- Uses a small assembly file to include binary resources, rather than
  objcopy
- Updates `base/common/getResource.cpp` for this new method of inclusion
- Removes linux-only guards in CMake files, as this solution is
  cross-platform.

The resulting binary resources are available in the ClickHouse server
binary on Linux, macOS, and illumos platforms. FreeBSD has not been
tested, but will likely work as well.
2021-06-09 14:03:30 -07:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
Azat Khuzhin
c343f85b64 Do not built clickhouse-keeper w/o NuRaft
Otherwise it will fail because of unused argument and unreachable code.
2021-05-23 08:37:56 +03:00
alesapin
31181095e7 Review fixes 2021-05-18 17:08:56 +03:00
alesapin
c9bef9d332 Better comments 2021-05-17 12:38:44 +03:00
alesapin
76e94a0583 Better 2021-05-17 12:00:31 +03:00
alesapin
e5cc36f031 Add comment 2021-05-17 11:58:42 +03:00
alesapin
ee46850112 Better cmake for keeper 2021-05-13 15:26:10 +03:00
alesapin
df22534dbc Fix cmake 2021-05-13 11:42:10 +03:00
alesapin
4e3f103748 Missed config file 2021-05-12 17:09:05 +03:00
alesapin
43ee9f0a3a Check for directory owner 2021-05-12 17:05:44 +03:00
alesapin
41eadf9127 Something working 2021-05-12 16:04:34 +03:00
alesapin
ffb4e1af8f Some copy-paste 2021-05-12 13:39:07 +03:00