thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-20 13:15:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
161,287 Commits 9,342 Branches 1,785 Tags 2.2 GiB
8a04621f43
Commit Graph

1225 Commits

Author SHA1 Message Date
Antonio Andelic
2988e13050 Free bitmask 2024-07-25 13:55:01 +02:00
Antonio Andelic
6557387148 Try calcualting memory with numactl if membind used 2024-07-25 11:49:02 +02:00
Antonio Andelic
d78cfd030f Use cgroups as source 2024-07-23 10:33:12 +02:00
Antonio Andelic
1c3f7d0fd0 Small fix 2024-07-22 15:18:13 +02:00
Antonio Andelic
1d1dc9c7ad
Merge branch 'master' into read-cgroup-memory-usage-async-metrics 2024-07-21 10:37:01 +01:00
Alex Katsman
eb51dc8980 Don't start GWP allocations until init is finished 2024-07-19 13:34:50 +00:00
Antonio Andelic
9a43183eb3 Finish background memory thread 2024-07-18 10:40:47 +02:00
Antonio Andelic
21009577d8 Dedicated memory background thread 2024-07-18 09:01:20 +02:00
Alexey Milovidov
4217d029c0 Fix something 2024-07-18 03:33:36 +02:00
Antonio Andelic
97f4ec2adb Read cgroup memory usage in async metrics thread 2024-07-17 16:59:35 +02:00
Alexey Milovidov
f5db89faa2 Merge branch 'update-tidy-for-clang-19' into format-settings-parsing 2024-07-17 05:53:36 +02:00
Alexander Tokmakov
09dc79ad3a Merge branch 'master' into reject_poco 2024-07-15 20:17:28 +02:00
Alexey Milovidov
9745a5dd74 Merge branch 'master' into format-settings-parsing 2024-07-12 23:19:54 +02:00
serxa
bf125d5da6 Merge branch 'master' into fix-use-concurrency-control 2024-07-12 08:44:12 +00:00
Alexander Tokmakov
243edcc8aa add logs and metrics about rejected connections in Poco 2024-07-12 01:11:18 +02:00
Raúl Marín
a339dd020f Reduce dependencies on MergeTreeSettings.h 2024-07-09 19:39:41 +02:00
Alexey Milovidov
278f2bb14f Merge branch 'master' of github.com:ClickHouse/ClickHouse into format-settings-parsing 2024-07-05 22:57:11 +02:00
pufit
4195ce2183
Merge branch 'master' into pufit/startup-scripts 2024-07-05 13:52:58 -04:00
Anton Popov
907a365926
Merge pull request #66098 from CurtizJ/better-sanitizer-message 
More precise warning message about sanitizers
 2024-07-05 14:15:28 +00:00
Alexey Milovidov
dbcc2c6979 Merge branch 'master' into format-settings-parsing 2024-07-05 15:53:23 +02:00
Anton Popov
93afc8e613 more precise warning message about sanitizers 2024-07-04 15:11:29 +00:00
Antonio Andelic
613ed1ebbf Merge branch 'master' into try-disabling-jemalloc-background-threads 2024-07-04 13:58:26 +02:00
Antonio Andelic
c93d8cbb66 Fixes 2024-07-04 13:57:47 +02:00
Antonio Andelic
c86cdbb243 Remove scary jemalloc log 2024-07-03 11:05:34 +02:00
Alexey Milovidov
0a493b88f2 Merge branch 'master' into format-settings-parsing 2024-06-29 04:25:34 +02:00
Alexander Tokmakov
4748e2929f
Merge pull request #65570 from ClickHouse/keeper_az 
Improve ZooKeeper load balancing
 2024-06-28 22:57:20 +00:00
vdimir
77bdd5da8d
Merge branch 'master' into mongodb_refactoring 2024-06-27 12:24:04 +02:00
vdimir
27a72fd5c4
Merge branch 'master' into pufit/startup-scripts 2024-06-27 12:14:33 +02:00
Alexander Tokmakov
43cb2f6af7 remove trash 2024-06-26 02:03:22 +02:00
alesapin
12a129d838
Merge pull request #65545 from ClickHouse/add_customizeable_config_reloader_interval 
Add ability to change config reload interval
 2024-06-25 14:18:54 +00:00
Alexey Milovidov
3b149d1e0b Merge branch 'master' into format-settings-parsing 2024-06-25 03:18:37 +02:00
Sema Checherinda
fb2f0e78f7
Merge pull request #63985 from ianton-ru/protocols-tls-extend-params 
More parameters for tls layer in protocols
 2024-06-24 12:41:56 +00:00
alesapin
fc10bf544f Merge remote-tracking branch 'origin/master' into add_customizeable_config_reloader_interval 2024-06-24 11:38:02 +02:00
Kirill Nikiforov
edb3d0ab42
fix ssl, fix after master merge 2024-06-24 00:45:15 +03:00
Robert Schulze
4e1a68f33d
Show compiled expressions cache settings size in system.server_settings 2024-06-23 10:55:52 +00:00
alesapin
104dc8fa3e Fix server settings 2024-06-21 19:16:27 +02:00
alesapin
0dc01f4a94 Add ability to change config reload interval 2024-06-21 18:24:20 +02:00
Alexey Milovidov
9dcaade3bb Merge branch 'master' into format-settings-parsing 2024-06-21 16:02:53 +02:00
pufit
3786ca72d0 docs, review fixes 2024-06-21 02:02:30 -04:00
pufit
e1248d8d4d Merge branch 'master' into pufit/startup-scripts 
# Conflicts:
#	src/Core/ServerSettings.h
 2024-06-20 21:32:11 -04:00
Sema Checherinda
e36a641a80 Merge branch 'master' into protocols-tls-extend-params 2024-06-20 19:02:12 +02:00
Kseniia Sumarokova
784f66cf2f
Merge pull request #65239 from ClickHouse/tracing-try-2 
Initialize global profiler for Poco::ThreadPool (needed for keeper, etc)
 2024-06-19 09:00:46 +00:00
Alexey Milovidov
11d54f4809 Pass-through settings from the client 2024-06-18 15:18:53 +02:00
Sergei Trifonov
a17d9d92d7
Merge branch 'master' into workload-for-merges 2024-06-14 18:35:43 +02:00
Alexander Tokmakov
bc8c8ffd55
Merge pull request #64986 from ClickHouse/fix_init_order 
Fix initialization order (ServerUUID/ZooKeeper)
 2024-06-14 12:32:05 +00:00
kssenii
fb32a99578 Initialize global trace collector for Poco::ThreadPool 2024-06-13 19:20:58 +02:00
serxa
08a293a91f Merge branch 'master' into workload-for-merges 2024-06-11 21:33:25 +00:00
Antonio Andelic
f51ed35200 Merge branch 'master' into fix-gwp-asan 2024-06-11 16:57:50 +02:00
kssenii
571ddb66df Merge remote-tracking branch 'origin/master' into named-collecitons-in-keeper 2024-06-10 13:17:32 +02:00
Antonio Andelic
c2c5992434 Merge branch 'master' into fix-gwp-asan 2024-06-10 12:50:08 +02:00
Alexander Tokmakov
01ece14035 fix initialization order (ServerUUID/ZooKeeper) 2024-06-07 16:23:40 +02:00
serxa
7594e06ea2 Merge branch 'master' into workload-for-merges 2024-06-07 13:58:41 +00:00
Antonio Andelic
bfb1c4c793 better 2024-06-06 11:29:42 +02:00
Antonio Andelic
1d3cf17053 Fix global trace collector 2024-06-06 11:18:31 +02:00
pufit
86bbcb0037 Startup scripts 2024-06-06 01:42:07 -04:00
Antonio Andelic
5df91804f6 Add probability setting for GWPAsan 2024-06-03 11:54:37 +02:00
kssenii
0571485ef6 Merge remote-tracking branch 'origin/master' into named-collecitons-in-keeper 2024-05-30 19:45:01 +02:00
kssenii
83c0c9bcde Move everything to NamedCollecitonFactory 2024-05-30 19:43:05 +02:00
Robert Schulze
7f4c68c0b8
Port internal changes back to public repo 2024-05-29 08:45:28 +00:00
serxa
c04d7bc2b7 Merge branch 'master' into workload-for-merges 2024-05-28 10:44:59 +00:00
Alexander Tokmakov
3ee2307024
Revert "Refactoring of Server.h: Isolate server management from other logic" 2024-05-27 10:04:19 +02:00
TTPO100AJIEX
251010f109 Move protocol-server and inter-server management into separate classes 
Co-authored-by: Alex Koledaev <ax3l3rator@gmail.com>
 2024-05-24 02:14:26 +03:00
Anton Ivashkin
983fa64e4f Fix CertificateReloader usage for custom protocols 2024-05-23 17:53:23 +02:00
Sergei Trifonov
dd22140b57
Merge branch 'master' into fix-use-concurrency-control 2024-05-22 11:36:20 +02:00
Francisco Javier Jurado Moreno
e1caea6ab5 Split attached table count into attached tables, views and dictionaries 2024-05-21 11:29:07 +02:00
serxa
96e19ac548 add merge_workload and mutation_workload settings for server and merge tree 2024-05-17 17:10:40 +00:00
Han Fei
116055b532 refine load part logic 2024-05-08 15:04:16 +02:00
Alexey Milovidov
ed1bbd0bd0 Merge branch 'master' into local-less-directories 2024-05-03 02:35:44 +02:00
alesapin
eacbf5fe10 Fix premature server listen for connections 2024-04-30 14:32:03 +02:00
Alexey Milovidov
39118ef461 Hopefully better 2024-04-27 18:47:03 +02:00
Anton Popov
b50117780c Merge remote-tracking branch 'upstream/master' into HEAD 2024-04-17 13:05:46 +00:00
Robert Schulze
7dc05d092b
Merge remote-tracking branch 'rschu1ze/master' into be-less-boring-32 2024-04-15 16:35:10 +00:00
Anton Popov
9e54a833ed fix build 2024-04-12 18:41:40 +00:00
Anton Popov
47c653f87c better retries in azure sdk 2024-04-12 15:39:14 +00:00
Alexander Tokmakov
d8e97b51bf Merge branch 'master' into cannot_allocate_thread 2024-04-10 21:21:42 +02:00
Nikita Taranov
a2147b8ded
Merge branch 'master' into keep_alive_max_reqs 2024-04-09 21:32:36 +02:00
Robert Schulze
dca6e0abbd
Fix the build (after boringssl submodule removal) 2024-04-08 11:02:41 +00:00
Alexander Tokmakov
c53b20a770 fix 2024-04-03 20:57:12 +02:00
Alexander Tokmakov
98ac8031e0 add fault injection 2024-04-03 19:49:23 +02:00
alesapin
f5c5146153 [RFC] Try to add global Real and CPU trace collector 2024-04-02 18:04:53 +02:00
Sergei Trifonov
b252062767
Merge branch 'master' into fix-use-concurrency-control 2024-04-01 18:32:40 +02:00
Nikita Taranov
d7b34a80bb stash 2024-03-22 16:09:14 +00:00
serxa
c31b958f88 Merge branch 'master' into async-loader-waiters-limit 2024-03-21 10:58:22 +00:00
Raúl Marín
de855ca917 Reduce header dependencies 2024-03-19 17:04:29 +01:00
Han Fei
8a54c85f3c
Merge pull request #61049 from hanfei1991/hanfei/check-limit-periodically 
Check cgroups memory limit update periodically
 2024-03-18 15:15:46 +01:00
Robert Schulze
fe86741059
Some fixups 2024-03-18 10:44:56 +00:00
serxa
dd6c1ac19a add diagnostics for concurrency control 2024-03-15 18:40:50 +00:00
Robert Schulze
4e12b5dc91
Some fixups 2024-03-13 16:54:20 +00:00
Robert Schulze
ce665eae30
Doc fixups 2024-03-13 15:13:17 +00:00
alesapin
9b1d8a2f52 fix style 2024-03-13 14:15:34 +01:00
alesapin
6203d45b96 working version with comments 2024-03-13 13:12:48 +01:00
Han Fei
c7c64636d0 add a test 2024-03-12 23:31:33 +01:00
serxa
5b796e1f42 Merge branch 'master' into async-loader-waiters-limit 2024-03-10 15:20:27 +00:00
Sema Checherinda
5c41727725 http connections pools 2024-03-10 10:00:18 +01:00
Han Fei
c01a6775d7 fix style 2024-03-09 02:28:39 +01:00
Han Fei
4997f95426 fix build 2024-03-08 17:30:58 +01:00
Han Fei
c92fb8700f Merge branch 'master' into hanfei/check-limit-periodically 2024-03-08 03:59:41 +01:00
serxa
0f0ea422f2 separate limits on number of waiting and executing queries 2024-03-07 17:05:54 +00:00
Han Fei
7fd13df8a5 check memory limit periodically 2024-03-07 17:21:03 +01:00
Robert Schulze
b5a7ef5fd3
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-hdr 2024-03-05 11:24:28 +00:00
Alexey Milovidov
a7430004b3 Fix real time query profiler on AArch64 2024-03-04 18:13:49 +01:00
Alexey Milovidov
de6c5d3a40
Merge pull request #60552 from ClickHouse/pc2 
Userspace page cache again
 2024-03-04 17:47:35 +03:00
Robert Schulze
af6755f241
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-hdr 2024-03-01 11:45:21 +00:00
Michael Kolupaev
42437a2ae1 Userspace page cache (#53770) 
* Userspace page cache

* Maybe it'll build this time, who knows.

* 'auto' went out of fashion, I guess

* Documentation, tsan workaround, metric 'UnreclaimableRss', disable page cache in the test that uses DatabaseOrdinary

* Moved CachedInMemoryReadBufferFromFile to object store level, changed settings, addressed other comments.

* Fix

* Another fix

* Fix restricted seek, fix ppc64le build

* Don't allow page cache with file cache

* Adjust tests a little

* Fix clang-tidy

* Conflicts

* Comments

* Maybe unbroke AsynchronousBoundedReadBuffer

* SettingsChangesHistory.h

* Fix warning in test
 2024-02-29 12:28:32 +00:00
Alexander Tokmakov
15e1191f74
Revert "Userspace page cache (#53770)" 
This reverts commit 7b55c61551.
 2024-02-29 13:20:17 +01:00
Michael Kolupaev
7b55c61551
Userspace page cache (#53770) 
* Userspace page cache

* Maybe it'll build this time, who knows.

* 'auto' went out of fashion, I guess

* Documentation, tsan workaround, metric 'UnreclaimableRss', disable page cache in the test that uses DatabaseOrdinary

* Moved CachedInMemoryReadBufferFromFile to object store level, changed settings, addressed other comments.

* Fix

* Another fix

* Fix restricted seek, fix ppc64le build

* Don't allow page cache with file cache

* Adjust tests a little

* Fix clang-tidy

* Conflicts

* Comments

* Maybe unbroke AsynchronousBoundedReadBuffer

* SettingsChangesHistory.h

* Fix warning in test
 2024-02-29 11:27:32 +00:00
Kirill Nikiforov
2744fc0c2d rename dns_cache_max_size to dns_cache_max_entries 2024-02-29 02:32:13 +03:00
Robert Schulze
3b18eb4f17
Fix clang-tidy in some headers 2024-02-28 22:47:34 +00:00
Kirill Nikiforov
8b314a3ac8
Implement system.dns_cache table and dns_cache_max_size server setting (#60257) 
* system.dns_cache table WIP

* system.dns_cache table

* rollback unexpected contrib changes

* rollback unexpected changes

* add docs

* code style fixes

* optimize dns cache exporting

* stateful test for system.dns_cache table

* stateful test for system.dns_cache table

* cr fixes

* stateful test for system.dns_cache table

* stateful test for system.dns_cache table

* implement system.dns_cache table

* system.dns_cache test

* system.dns_cache fix docs

* system.dns_cache prettify

* dns_cache_max_size settings

* fix system.dns_cache test

* Update dns_cache.md

* fix doc

Co-authored-by: pufit <pufit@clickhouse.com>

---------

Co-authored-by: pufit <pufit@clickhouse.com>
 2024-02-26 15:48:02 -05:00
Robert Schulze
048f7ec3e9
Merge pull request #59516 from rschu1ze/cgroup-observing-memory-tracker 
Update memory tracker periodically with cgroup memory usage
 2024-02-26 15:27:19 +01:00
János Benjamin Antal
7cad0050c0
Disambiguate alter commands list (#59532) 
* Add optional parentheses to TTL element and alter command

* Add some basic tests

* Extend tests with more test cases

* Add one more test case

* Add server setting to control new formatting behavior

* Automatic style fix

* Fix segfault in `clickhouse-format` related to new server setting

* Remove format_ttl_expressions_with_parentheses

* Fix unit test compilation failures

* Introduce global flag to control the new formatting behavior

* Revert "Fix segfault in `clickhouse-format` related to new server setting"

This reverts commit d7131a3145.

* Revert accidental changes

* Revert accidental include reorder

* Revert accidental changes

* Enable new format in client and local server

---------

Co-authored-by: robot-clickhouse <robot-clickhouse@users.noreply.github.com>
 2024-02-22 19:17:15 +01:00
Alexander Tokmakov
64a80f1011
Fix default path when path is not specified in config (#59654) 
* Update Server.cpp

* Update SentryWriter.cpp

* Update Keeper.cpp

* Update SentryWriter.cpp
 2024-02-21 20:33:39 +01:00
Sergei Trifonov
6e3fddf4a3
Merge pull request #59313 from ClickHouse/abstract-concurrency-control 
Split `ISlotControl` from `ConcurrencyControl`
 2024-02-21 15:04:03 +01:00
Robert Schulze
ba9661a5d4
Merge remote-tracking branch 'rschu1ze/master' into cgroup-observing-memory-tracker 2024-02-19 09:54:08 +00:00
Alexey Milovidov
509d8ee99c Apply review comments 2024-02-17 01:54:54 +01:00
Alexey Milovidov
39f363ba6e Print CPU flags at startup 2024-02-16 13:09:07 +01:00
Alexey Milovidov
5b83e771ed Print CPU flags at startup 2024-02-16 13:06:57 +01:00
Konstantin Bogdanov
914fd27d5b
Add PlacementInfo 2024-02-14 10:19:36 +01:00
Robert Schulze
7b7ea6d477
Merge remote-tracking branch 'rschu1ze/master' into cgroup-observing-memory-tracker 2024-02-14 08:56:22 +00:00
Azat Khuzhin
e6d96e4325 Better warning for disabled kernel.task_delayacct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-02-10 14:49:43 +01:00
Robert Schulze
f6f1fd4d26
Initial implementation 2024-02-08 16:06:44 +00:00
Konstantin Bogdanov
5894fa9860
0 means number of cores 2024-02-06 19:10:34 +01:00
Konstantin Bogdanov
5969668503
Concurrent table creation in DatabaseReplicated 2024-02-01 09:51:52 +01:00
serxa
ba85642453 split ISlotControl from ConcurrencyControl 2024-01-28 20:26:55 +00:00
Sergei Trifonov
4fb511bfb3
Merge pull request #59137 from ClickHouse/fix-async-loader-shutdown-deadlock 
Fix not started database shutdown deadlock
 2024-01-26 11:21:13 +01:00
Maksim Kita
2a327107b6 Updated implementation 2024-01-25 14:31:49 +03:00
serxa
8728ab768f Fix not started database shutdown deadlock 2024-01-23 19:01:57 +00:00
Robert Schulze
e23e7e6b73
Merge pull request #53710 from rschu1ze/system-reload-asynchronous-metrics 
Add statement `SYSTEM RELOAD ASYNCHRONOUS METRICS`
 2024-01-23 11:56:25 +01:00
serxa
7b99d9a176 wait for currently running jobs to finish #2 2024-01-22 23:50:53 +00:00
serxa
1d6d1182fd Fix server shutdown due to exception while loading metadata 2024-01-22 19:38:26 +00:00
Robert Schulze
90a0ea36b3
Add statement 'SYSTEM RELOAD ASYNCHRONOUS METRICS' 2024-01-22 17:14:58 +00:00
Vitaly Baranov
01aafda27b
Merge pull request #58804 from vitlibar/allow-kill-query-for-backups 
Allow KILL QUERY for backups
 2024-01-20 23:48:18 +01:00
serxa
824deab761 split resource scheduler off IO/ into Common/Scheduler/ 2024-01-20 12:06:26 +00:00
Vitaly Baranov
4c6d3e7efa Add server settings "shutdown_wait_backups_and_restores" (default=true) 
to set whether shutdown should wait for running backups to finish or just cancel them.
 2024-01-18 13:27:14 +01:00
Mikhail Koviazin
6e6197019c
Reintroduce commit 3067ca6 
Update `query_masking_rules` when reloading the config

This reverts commit 7e55eab1c3.
 2024-01-17 08:10:49 +02:00
Yakov Olkhovskiy
2e124b19b8
Merge branch 'master' into revert-58450-revert-56064-feature-server-iface-metrics 2024-01-10 09:05:18 -05:00
Bharat Nallan
ee37f551a2
independent registration of interpreters (#58443) 
* make interpreter factory an instance

* add registerInterpreter

* register interpreters individually

* try wire everything up

* fix style

* fix test
 2024-01-08 22:33:48 -08:00
Yakov Olkhovskiy
9ca4be5d91 Merge remote-tracking branch 'origin/master' into revert-58450-revert-56064-feature-server-iface-metrics 2024-01-06 00:24:31 +00:00
Michael Kolupaev
52058211e7
Fix some thread pool settings not updating at runtime (#58485) 2024-01-04 17:21:46 +01:00
Yakov Olkhovskiy
85f03478ef
Revert "Revert "Use CH Buffer for HTTP out stream, add metrics for interfaces"" 2024-01-03 11:47:15 -05:00
Raúl Marín
d491758939
Revert "Use CH Buffer for HTTP out stream, add metrics for interfaces" 2024-01-03 10:42:15 +01:00
Yakov Olkhovskiy
5633fb8145
Merge pull request #56064 from ClickHouse/feature-server-iface-metrics 
Use CH Buffer for HTTP out stream, add metrics for interfaces
 2024-01-02 10:17:52 -05:00
Bharat Nallan Chakravarthy
ab28da85d3 try wire everything up 2023-12-29 16:38:17 -08:00
Yakov Olkhovskiy
5dc88542cb
Merge branch 'master' into feature-server-iface-metrics 2023-12-20 16:46:29 -05:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
pufit
36d1905881
Merge pull request #57752 from chizhonkova/refactor_sql_udf_loader 
Refactoring for UserDefinedSQLObjectsLoader
 2023-12-18 16:08:46 -05:00
Yakov Olkhovskiy
f99b36143f Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-12-18 14:14:03 +00:00
Yakov Olkhovskiy
001a38048f use ProfileEvents instead of CurrentMetrics 2023-12-15 19:17:42 +00:00
Alexey Milovidov
7e55eab1c3 Revert "Merge pull request #56573 from mkmkme/mkmkme/reload-config" 
This reverts commit 3067ca64df, reversing
changes made to 1e46460910.
 2023-12-15 19:28:25 +01:00
Yakov Olkhovskiy
febdd542f8 Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-12-14 18:59:44 +00:00
Kseniia Sumarokova
79db3c66df
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-13 23:33:59 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Alexey Milovidov
54698cd2de
Merge pull request #55876 from ClickHouse/keeper-http-readiness-endpoint 
Keeper HTTP readiness endpoint
 2023-12-13 17:07:57 +01:00
Han Fei
a96593c8b1 Merge branch 'master' into feature/more_warnings 2023-12-12 15:25:01 +01:00
Natalya Chizhonkova
3d51192bad Refactoring for UserDefinedSQLObjectsLoader 2023-12-12 17:03:49 +03:00
Konstantin Bogdanov
771d5c58ea
Fix Keeper standalone build 2023-12-11 10:19:49 +01:00
Konstantin Bogdanov
89ba4a8452
Add HTTP readiness endpoint 2023-12-11 10:19:42 +01:00
Azat Khuzhin
8f5a60eb5a More descriptive comment for destroying context prio global thread pool 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 15:39:43 +01:00
Azat Khuzhin
fcdead023c Proper server termination in case of intermediate failures during initialization 
For instance on CI it fails because KeeperDispatch::initialize() failed,
but there is already proper KeeperDispatch::shutdown() in dtor, the
problem is that you first need to shutdown global context, that will
terminate all thread pools, and only after you can join global thread
pool.

CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/6ccbc2ea75c39234dff748cfb147bd44d65d3e75/integration_tests__asan__[3_4].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 08:01:40 +01:00
Azat Khuzhin
b7edde3621 Shutdown embedded keeper even if it was not started successfully 
Since even when it does not, there are still few threads left that
should be joined (request, response, snapshot, snapshot s3):

There are some problems with keeper and invalid config, that got stuck in:

    libc.so.6`___lldb_unnamed_symbol3457 + 231
    libc.so.6`pthread_cond_timedwait + 571
    clickhouse`std::__1::__libcpp_condvar_timedwait[abi:v15000](__cv=0x00007f475c41a708, __m=0x00007f475c41a6b0, __ts=0x00007f45a99f1440) at __threading_support:341:10
    clickhouse`std::__1::condition_variable::__do_timed_wait(this=0x00007f475c41a708, lk=0x00007f45a99f1738, tp=time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1L, 1000000000L> > > @ 0x00007f45a99f1488) at condition_variable.cpp:76:14
    clickhouse`std::__1::cv_status std::__1::condition_variable::wait_for<long long, std::__1::ratio<1l, 1000000000l> >(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __d=0x00007f45a99f1590) at __mutex_base:457:9
    clickhouse`void std::__1::condition_variable::__do_timed_wait<std::__1::chrono::steady_clock>(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __tp=time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1L, 1000000000L> > > @ 0x00007f45a99f15a8) at __mutex_base:512:5
    clickhouse`std::__1::cv_status std::__1::condition_variable::wait_until<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __t=0x00007f45a99f1698) at __mutex_base:415:5
    clickhouse`bool std::__1::condition_variable::wait_until<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> >, bool ConcurrentBoundedQueue<DB::KeeperStorage::RequestForSession>::popImpl<true>(DB::KeeperStorage::RequestForSession&, std::__1::optional<unsigned long>)::'lambda'()>(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __t=0x00007f45a99f1698, __pred=(unnamed class) @ 0x00007f45a99f1650)::'lambda'()) at __mutex_base:427:13
    clickhouse`bool std::__1::condition_variable::wait_for[abi:v15000]<long long, std::__1::ratio<1l, 1000l>, bool ConcurrentBoundedQueue<DB::KeeperStorage::RequestForSession>::popImpl<true>(DB::KeeperStorage::RequestForSession&, std::__1::optional<unsigned long>)::'lambda'()>(this=0x00007f475c41a708, __lk=0x00007f45a99f1738, __d=0x00007f45a99f1720, __pred=(unnamed class) @ 0x00007f45a99f16b8)::'lambda'()) at __mutex_base:471:12
    clickhouse`bool ConcurrentBoundedQueue<DB::KeeperStorage::RequestForSession>::popImpl<true>(this=0x00007f475c41a680, x=0x00007f45a99f1ba0, timeout_milliseconds= Has Value=true ) at ConcurrentBoundedQueue.h:75:50
    clickhouse`ConcurrentBoundedQueue<DB::KeeperStorage::RequestForSession>::tryPop(this=0x00007f475c41a680, x=0x00007f45a99f1ba0, milliseconds=5000) at ConcurrentBoundedQueue.h:161:16
    clickhouse`DB::KeeperDispatcher::requestThread(this=0x00007f475c8fac18) at KeeperDispatcher.cpp:88:33
    clickhouse`DB::KeeperDispatcher::initialize(this=0x00007f45a99f2020)::$_0::operator()() const at KeeperDispatcher.cpp:349:52
    clickhouse`decltype(__f=0x00007f45a99f2020)::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&) at invoke.h:394:23
    clickhouse`decltype(__f=0x00007f45a99f2020, __t=size=0, (null)=__tuple_indices<> @ 0x00007f45a99f1d28) std::__1::__apply_tuple_impl[abi:v15000]<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&, std::__1::tuple<>&>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) at tuple:1789:1
    clickhouse`decltype(__f=0x00007f45a99f2020, __t=size=0) std::__1::apply[abi:v15000]<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&, std::__1::tuple<>&>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&, std::__1::tuple<>&) at tuple:1798:1
    clickhouse`ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(this=0x00007f475c9703c0)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'()::operator()() at ThreadPool.h:245:13
    clickhouse`decltype(__f=0x00007f475c9703c0)::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'()&>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&) at invoke.h:394:23
    clickhouse`void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(__args=0x00007f475c9703c0)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'()&) at invoke.h:479:9
    clickhouse`std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'(), void ()>::operator(this=0x00007f475c9703c0)[abi:v15000]() at function.h:235:12
    clickhouse`void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::KeeperDispatcher::initialize(__buf=0x00007f45a99f22d8)::$_0>(DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr<DB::Macros const> const&)::$_0&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) at function.h:716:16
    clickhouse`std::__1::__function::__policy_func<void ()>::operator(this=0x00007f45a99f22d8)[abi:v15000]() const at function.h:848:16
    clickhouse`std::__1::function<void ()>::operator(this=0x00007f45a99f22d8)() const at function.h:1187:12
    clickhouse`ThreadPoolImpl<std::__1::thread>::worker(this=0x00007f475c8229c0, thread_it=std::__1::list<std::__1::thread, std::__1::allocator<std::__1::thread> >::iterator @ 0x00007f45a99f23a8) at ThreadPool.cpp:427:13
    clickhouse`void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(this=0x00007f45cea0c8a8)>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const at ThreadPool.cpp:180:73
    clickhouse`decltype(__f=0x00007f45cea0c8a8)()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) at invoke.h:394:23
    clickhouse`void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(__t=size=2, (null)=__tuple_indices<> @ 0x00007f45a99f2408)>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) at thread:284:5
    clickhouse`void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()> >(__vp=0x00007f45cea0c8a0) at thread:295:5
    libc.so.6`___lldb_unnamed_symbol3481 + 755
    libc.so.6`___lldb_unnamed_symbol3865 + 11

This should fix the problem with
test_keeper_incorrect_config/test.py::test_invalid_configs test.

CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/72b1052f7c2d453308262924e767ab8dc2206933/integration_tests__asan__[1_6].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 17:54:16 +01:00
Azat Khuzhin
55d5a3affa Correctly wait background threads 
There are some places that make it possible to trigger use-after-free
from threads because some global variable had been destroyed before, for
example some logger.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 17:54:16 +01:00
kssenii
199015576c Add a file which I forgot to commit 2023-12-07 18:30:43 +01:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Alexey Milovidov
cbea3545aa
Revert "add function getClientHTTPHeader" 2023-12-05 13:34:34 +03:00
凌涛
9e05e79d66
Merge branch 'master' into feature/more_warnings 2023-12-05 11:50:05 +08:00
凌涛
6794bbe196 rebase master 2023-12-05 11:42:25 +08:00
Alexey Milovidov
02439eee58
Merge pull request #49351 from ClickHouse/async-loader-integration 
Asynchronous loading of tables
 2023-12-04 17:16:12 +01:00
Yakov Olkhovskiy
416d28687e Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-11-30 23:00:04 +00:00
凌涛
d9f4b4d2c0 support parts, tables, databases 2023-11-30 10:56:31 +08:00
János Benjamin Antal
4c2451f837
Merge pull request #56741 from ClickHouse/well-known-protos 2023-11-28 15:21:15 +01:00
Antonio Andelic
a61f32867e
Merge pull request #56488 from lingtaolf/feature/getHTTPHeader 
add function getClientHTTPHeader
 2023-11-28 11:41:56 +01:00
Yakov Olkhovskiy
b92030b485
Merge branch 'master' into feature-server-iface-metrics 2023-11-27 15:44:25 -05:00
Vitaly Baranov
5769a88b92
Merge pull request #57133 from vitlibar/change-default-for-wait_dictionaries_load_at_startup 
Change the default for wait_dictionaries_load_at_startup to true
 2023-11-24 17:09:05 +01:00
Kseniia Sumarokova
650c1e959d
Merge pull request #57141 from ClickHouse/remove-test-case 
Follow up to #56541
 2023-11-24 11:45:29 +01:00
Alexey Milovidov
b62e9d3a1a Address review comments 2023-11-23 18:13:12 +01:00
Alexey Milovidov
5abed8f8a8 Better exit codes from the server 2023-11-23 17:00:02 +01:00
Vitaly Baranov
5f461ff780 Change the default for "wait_dictionaries_load_at_startup" to true, 
and use this setting only if "dictionaries_lazy_load" is false.
 2023-11-23 14:45:42 +01:00
kssenii
ff3224e7f7 Load named collections in advance 2023-11-23 13:22:06 +01:00
János Benjamin Antal
ab935e3dd7 Use the google proto files when importing protobuf schemas 2023-11-22 12:39:41 +00:00
凌涛
e86aa433c5
Merge branch 'master' into feature/getHTTPHeader 2023-11-21 19:48:47 +08:00
凌涛
c5f16725ec add function getClientHTTPHeader 2023-11-21 11:12:01 +08:00
serxa
9023f61349 Merge branch 'master' into async-loader-integration 2023-11-19 13:05:58 +00:00
Alexey Milovidov
482d8ca9d8
Merge pull request #56102 from ClickHouse/coverage 
Granular code coverage with introspection
 2023-11-16 23:23:40 +01:00
Vitaly Baranov
342e6c7546
Merge pull request #56782 from vitlibar/add-config-setting-wait_dictionaries_load_at_startup 
Add config setting "wait_dictionaries_load_at_startup"
 2023-11-16 11:51:29 +01:00
Alexey Milovidov
3067ca64df
Merge pull request #56573 from mkmkme/mkmkme/reload-config 
Update `query_masking_rules` when reloading the config
 2023-11-16 00:27:01 +01:00
Vitaly Baranov
308654195c Add config setting "wait_dictionaries_load_at_startup". 2023-11-15 11:48:50 +01:00
Sema Checherinda
b9cc1580e0
Merge pull request #56367 from canhld94/ch_table_reinit_new_disk 
Adding new disk to storage configuration without restart
 2023-11-14 15:54:22 +01:00
Mikhail Koviazin
9e1357dd7e
Update query_masking_rules when reloading the config 
Fixes #56449
 2023-11-14 14:03:28 +02:00
Alexey Milovidov
3ef14f6098 Merge branch 'master' of github.com:ClickHouse/ClickHouse into coverage 2023-11-14 06:08:32 +01:00
Alexey Milovidov
82c461031e Fix build 2023-11-13 10:09:23 +01:00
serxa
19aa1375fc Merge branch 'master' into async-loader-integration 2023-11-07 10:25:38 +00:00
serxa
d50747847d rework AsyncLoaderPoolId.h to match settings and metrics 2023-11-06 13:08:27 +00:00
Duc Canh Le
4c21ba7b6f tables auto initialize new disks without restart 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-11-06 10:26:48 +00:00
Alexey Milovidov
cd94d02455 Merge branch 'master' into coverage 2023-11-04 21:49:59 +01:00
Azat Khuzhin
cea78cd093
Add a warning if delayed accounting is not enabled (breaks OSIOWaitMicroseconds) (#56227) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-11-02 13:21:27 +01:00
Alexey Milovidov
56de2333f9 Add warning 2023-10-29 16:55:47 +01:00
Yakov Olkhovskiy
0cf851316c use CH Buffer for HTTP out stream, add metrics for interfaces 2023-10-27 02:38:36 +00:00
serxa
08e63f2a24 fix build 2023-10-24 21:49:15 +00:00
serxa
d3f54a29e3 add startup waits in all DDL queries and global DDL worker 2023-10-24 10:09:10 +00:00
serxa
0e17da22eb Merge branch 'master' into async-loader-integration 2023-10-16 11:08:29 +00:00