thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-03 04:52:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
146,365 Commits 9,181 Branches 1,782 Tags 2.2 GiB
8a202d91ad
Commit Graph

1360 Commits

Author SHA1 Message Date
Antonio Andelic
9e3355acef Remove snapshot_version 2024-02-29 10:11:38 +01:00
Antonio Andelic
0cb839af0b Merge branch 'master' into keeper-snapshot-detached 2024-02-29 09:47:32 +01:00
Antonio Andelic
ad265f978d
Merge pull request #60442 from ClickHouse/keeper-disk-move-fix 
Keeper: fix moving changelog files between disks
 2024-02-29 09:24:07 +01:00
Alexey Milovidov
282c3b55f2 Synchronize small pieces 2024-02-28 23:47:33 +01:00
Alexey Milovidov
b0ab0b37db
Merge pull request #60434 from ClickHouse/revert-60216-revert-59697-check-stack-size-in-parser 
Revert "Revert "Check stack size in Parser""
 2024-02-28 19:20:49 +03:00
Antonio Andelic
f7abe7601e Abort on invalid snapshots 2024-02-28 08:54:54 +01:00
Antonio Andelic
32daeced94 Abort on failure to read from snapshot 2024-02-27 16:38:25 +01:00
Antonio Andelic
2c8759442d Move broken Keeper snapshots to detached folder 2024-02-27 15:16:10 +01:00
Alexey Milovidov
c192a448d0 Update to clang-19 2024-02-27 14:37:21 +01:00
Antonio Andelic
178c60a8e5 Fix removing from source disk 2024-02-27 09:53:49 +01:00
Alexey Milovidov
0b90076a36
Revert "Revert "Check stack size in Parser"" 2024-02-27 01:34:50 +03:00
Nikita Taranov
835b47519a impl 2024-02-25 00:33:50 +01:00
Antonio Andelic
7ea44f95e3
Merge pull request #60226 from ClickHouse/keeper-better-snapshot 
Slightly better Keeper loading from snapshot
 2024-02-23 08:49:15 +01:00
Antonio Andelic
1b238d1180
Merge pull request #59460 from ClickHouse/keeper-logs-cache 
Keeper log cache
 2024-02-22 09:03:41 +01:00
kssenii
4e3f2aae40 Fix keeper build 2024-02-21 18:47:17 +01:00
Antonio Andelic
0b58aa6ec3 better 2024-02-21 17:08:44 +01:00
Antonio Andelic
033cbab8f3 Merge branch 'master' into keeper-better-snapshot 2024-02-21 14:01:37 +01:00
Antonio Andelic
fee1565780
Revert "Check stack size in Parser" 2024-02-21 11:33:08 +01:00
kssenii
d88f8646b1 Fix after merge with master 2024-02-21 11:03:12 +01:00
Antonio Andelic
2571401063 Merge branch 'master' into keeper-logs-cache 2024-02-21 10:19:50 +01:00
Antonio Andelic
bb0b1fa9bc Fix 2024-02-20 14:46:16 +01:00
Antonio Andelic
11f3b06099 Fix 2024-02-20 14:35:19 +01:00
Antonio Andelic
a0bfb47e95 Merge branch 'master' into keeper-more-resilient-disks 2024-02-20 13:51:23 +01:00
Antonio Andelic
cfda64e1a4 Use Keeper only events 2024-02-20 10:33:08 +01:00
Antonio Andelic
008938093d Merge branch 'master' into keeper-logs-cache 2024-02-20 10:17:28 +01:00
Antonio Andelic
0dc73bacc5 Fix cleanup 2024-02-19 15:09:10 +01:00
Antonio Andelic
299a0bd996 Merge branch 'master' into keeper-logs-cache 2024-02-19 10:01:31 +01:00
Alexey Milovidov
6fd563df52 Miscellaneous 2024-02-19 02:58:51 +01:00
Antonio Andelic
b01ccbbc88 Fix 2024-02-16 16:58:11 +01:00
Igor Nikonov
cb0ce2aaa9 Fix build 2024-02-16 15:26:08 +00:00
Antonio Andelic
e12fa6b35c Better snapshot reading 2024-02-16 14:40:49 +01:00
Antonio Andelic
ddfa3dea87 Merge branch 'master' into keeper-logs-cache 2024-02-16 10:07:06 +01:00
Antonio Andelic
0f489de46e
Merge pull request #59945 from ClickHouse/keeper-prometheus-filtered-events 
Send only Keeper related metrics/events for Prometheus
 2024-02-16 09:43:43 +01:00
Antonio Andelic
daf7505e5e Improve 2024-02-15 16:11:20 +01:00
Antonio Andelic
1ade535a83 Merge branch 'master' into keeper-logs-cache 2024-02-15 13:06:37 +01:00
Antonio Andelic
40d4fc1c24 More resilient Disk related operations 2024-02-14 14:53:22 +01:00
Konstantin Bogdanov
f678bdc546
Update tests 2024-02-14 10:51:42 +01:00
Antonio Andelic
0d151e3f0d Send only specific events/metrics for Keeper prometheus 2024-02-13 16:01:50 +01:00
Antonio Andelic
a0507c2044 Whitespace 2024-02-12 10:36:35 +01:00
Antonio Andelic
c9c0e12b4c Address PR reviews 2024-02-12 09:15:40 +01:00
Antonio Andelic
d2eac53c86 Fix 2024-02-09 12:06:01 +01:00
Antonio Andelic
42f2fefeab Fix data race and leak 2024-02-09 10:40:43 +01:00
Antonio Andelic
dfb3a8f7be Another fix 2024-02-08 17:17:12 +01:00
Antonio Andelic
99be5b5cb1 Fix 2024-02-07 15:57:52 +01:00
Antonio Andelic
1279964866
fix build 2024-02-06 10:18:02 +01:00
Antonio Andelic
754b73a38b
remove unused 2024-02-06 09:35:48 +01:00
Antonio Andelic
a0170bcbd5 Fix utils 2024-02-05 17:58:33 +01:00
Antonio Andelic
7ad48c2aa2 Fix 2024-02-05 17:02:16 +01:00
Antonio Andelic
2116043e2c Merge branch 'master' into keeper-more-reduce 2024-02-05 14:58:26 +01:00
Antonio Andelic
b1b564a835 Fix build again 2024-02-01 16:42:11 +00:00
Antonio Andelic
533f7d0314 Build fix 2024-02-01 16:09:00 +00:00
Antonio Andelic
c197fb27d6 Better 2024-02-01 10:42:16 +00:00
Antonio Andelic
b53abf8186 Finalize 2024-02-01 10:04:34 +00:00
Antonio Andelic
82196cac99 Merge branch 'master' into keeper-logs-cache 2024-02-01 08:56:12 +00:00
Antonio Andelic
08f91907dc Fix digest calculation 2024-01-31 16:08:22 +00:00
Antonio Andelic
a91101a21f Working implementation 2024-01-31 15:25:24 +00:00
Antonio Andelic
bd776b4c16 Merge branch 'master' into keeper-logs-cache 2024-01-29 10:26:23 +00:00
Maksim Kita
2a327107b6 Updated implementation 2024-01-25 14:31:49 +03:00
Antonio Andelic
8c7218bac2 Store latest logs inmemory 2024-01-25 08:46:06 +00:00
Antonio Andelic
3e3ae52aca Reduce even more memory 2024-01-25 08:40:19 +00:00
Antonio Andelic
34463fd5e5
Merge pull request #59002 from ClickHouse/keeper-reduce-memory 
Reduce Keeper memory usage
 2024-01-24 11:54:16 +01:00
Antonio Andelic
74dd68afbc
Merge pull request #59102 from ClickHouse/keeper-startup-fix 
Avoid election timeouts during startup in Keeper
 2024-01-23 13:55:52 +01:00
Robert Schulze
e23e7e6b73
Merge pull request #53710 from rschu1ze/system-reload-asynchronous-metrics 
Add statement `SYSTEM RELOAD ASYNCHRONOUS METRICS`
 2024-01-23 11:56:25 +01:00
Antonio Andelic
9ce3f8a7aa
typo 2024-01-23 09:18:51 +01:00
Antonio Andelic
ff644831e2 Avoid election timeouts during startup 2024-01-23 08:02:30 +00:00
Robert Schulze
90a0ea36b3
Add statement 'SYSTEM RELOAD ASYNCHRONOUS METRICS' 2024-01-22 17:14:58 +00:00
Robert Schulze
1262945df7
Cosmetics 2024-01-22 16:51:02 +00:00
alesapin
478b91cc37
Merge pull request #59010 from ClickHouse/alesapin-patch-1 
Change log level for super imporant message in Keeper
 2024-01-22 13:16:56 +01:00
Alexander Sapin
f7b21b6016 Fix build 2024-01-22 11:53:21 +01:00
Antonio Andelic
274c128bd7 Free memory 2024-01-22 10:48:39 +00:00
Antonio Andelic
59f9abcf58 Merge branch 'master' into keeper-reduce-memory 2024-01-22 08:04:17 +00:00
Antonio Andelic
ecaef80ea0
Merge pull request #58938 from ClickHouse/keeper-fetch-max-file-desc-once 
Call `getMaxFileDescriptorCount` once in Keeper
 2024-01-22 08:57:58 +01:00
alesapin
72f5dce9f4
Update src/Coordination/KeeperDispatcher.cpp 
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2024-01-21 11:16:27 +01:00
serxa
824deab761 split resource scheduler off IO/ into Common/Scheduler/ 2024-01-20 12:06:26 +00:00
alesapin
d275d79947
Update KeeperDispatcher.cpp 2024-01-19 19:25:27 +01:00
Antonio Andelic
0132455b39 Reduce Keeper memory usage 2024-01-19 17:00:53 +00:00
Antonio Andelic
19ca00f233 Call getMaxFileDescriptorCount once 2024-01-19 08:19:20 +00:00
Antonio Andelic
30a04471ac
Merge pull request #58806 from ClickHouse/fix-keeper-data-race 
Fix data race in Keeper
 2024-01-18 08:48:16 +01:00
Antonio Andelic
ded830ba33 Correctly handle shutdown 2024-01-17 08:56:36 +00:00
Pradeep Chhetri
09f29b4502 Update nomenclature of variables 2024-01-17 13:52:10 +08:00
Alexey Milovidov
d85abfafa4
Update KeeperAsynchronousMetrics.cpp 2024-01-17 08:05:48 +03:00
Alexey Milovidov
90e35be861
Update KeeperAsynchronousMetrics.cpp 2024-01-17 08:04:32 +03:00
Pradeep Chhetri
eafaab3795 Add metric for keeper memory soft limit 2024-01-17 12:56:18 +08:00
Antonio Andelic
7ecf146072 Fix data race in Keeper 2024-01-15 12:23:59 +00:00
Antonio Andelic
e7375d6785 Don't process request if not preprocessed 2024-01-12 15:49:30 +00:00
Antonio Andelic
2bfed6b8eb Add system commands and 4lw for jemalloc 2024-01-11 08:24:56 +00:00
Sema Checherinda
7c7e72c4b7
Merge pull request #58573 from ClickHouse/chesema-s3-client-creation 
fix and test that S3Clients are reused
 2024-01-09 13:28:20 +01:00
Raúl Marín
803d8d6320
Merge pull request #58594 from Algunenano/missing_includes 
Add some missing LLVM includes
 2024-01-09 11:01:44 +01:00
Antonio Andelic
a891750f53 Fix race 2024-01-08 12:08:17 +00:00
Antonio Andelic
6aee18ced8
Merge pull request #58487 from ClickHouse/keeper-lower-log-levels-test 
Lower log levels for some Raft logs to new test level
 2024-01-08 12:19:02 +01:00
Raúl Marín
c1b95560db Add some missing LLVM includes 2024-01-08 11:25:57 +01:00
Sema Checherinda
d5f86f671d fix and test that S3Clients reused 2024-01-07 02:19:06 +01:00
Antonio Andelic
13749f550f Lower log levels for some Raft logs 2024-01-04 07:59:44 +00:00
Antonio Andelic
8f26e2af67 Even better Keeper startup 2024-01-04 07:54:14 +00:00
Antonio Andelic
d86e59eb08
Merge pull request #58308 from ClickHouse/keeper-preprocess-fix 
Handle another case for preprocessing in Keeper
 2023-12-30 14:13:04 +01:00
Alexey Milovidov
8365722c95
Merge pull request #58336 from amosbird/macro2constexpr 
Organize symbols from src/* into DB namespace
 2023-12-30 12:40:36 +01:00
Amos Bird
0d801c4f6b
Fix unitest 2023-12-29 20:40:52 +08:00
Azat Khuzhin
8c54380d80 Avoid sending ComposeObject requests after upload to GCS 
This should not be required anymore, but leave it as an option, since
likely this is required for old files.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
f4a7789cd4 Convert various S3::Client settings into separate ClientSettings struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Antonio Andelic
8ace9978b3 Handle another case for preprocessing 2023-12-28 13:20:47 +00:00
Azat Khuzhin
3be3b0a280 Fix incorrect Exceptions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 21:26:32 +01:00
Antonio Andelic
b9466023ce
Merge pull request #58069 from ClickHouse/dont-preprocess-if-not-ready 
Fix invalid preprocessing on Keeper
 2023-12-21 09:08:56 +01:00
Antonio Andelic
3cc8358b89 Avoid preprocessing if Keeper leader is not ready 2023-12-20 10:22:00 +00:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials 
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
 2023-12-17 10:29:15 +02:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
alesapin
cbae29d26a
Merge pull request #57699 from hanfei1991/hanfei/keeper_soft_limit_ratio 
support keeper memory soft limit ratio
 2023-12-11 21:43:24 +01:00
Azat Khuzhin
450e8da7ce Fix data race on vptr (ctor/dtor vs virtual call) in AsynchronousMetrics 
TSan report [1]:

     Exception: Sanitizer assert found for instance ==================
     WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=1)
       Write of size 8 at 0x7ffce68f5680 by main thread:
         0 DB::AsynchronousMetrics::~AsynchronousMetrics() build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 (clickhouse+0xf38fa6e) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         1 DB::ServerAsynchronousMetrics::~ServerAsynchronousMetrics() build_docker/./src/Interpreters/ServerAsynchronousMetrics.h:10:7 (clickhouse+0xf2b2f0f) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         2 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./programs/server/Server.cpp:1996:1 (clickhouse+0xf2b2f0f)
         3 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d6c95fe) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         4 DB::Server::run() build_docker/./programs/server/Server.cpp:397:25 (clickhouse+0xf29e041) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         5 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d6e9354) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         6 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:203:20 (clickhouse+0xf29b1a3) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         7 main build_docker/./programs/main.cpp:505:12 (clickhouse+0x72d7fa0) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)

       Previous read of size 8 at 0x7ffce68f5680 by thread T649:
         0 DB::AsynchronousMetrics::update(std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Common/AsynchronousMetrics.cpp:1559:5 (clickhouse+0xf38ebc5) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         1 DB::AsynchronousMetrics::run() build_docker/./src/Common/AsynchronousMetrics.cpp:354:13 (clickhouse+0xf390696) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         2 DB::AsynchronousMetrics::start()::$_0::operator()() const build_docker/./src/Common/AsynchronousMetrics.cpp:273:62 (clickhouse+0xf392eca) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         3 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&>(DB::AsynchronousMetrics::start()::$_0&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf392eca)
         4 decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1 (clickhouse+0xf392eca)
         5 decltype(auto) std::__1::apply[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1 (clickhouse+0xf392eca)
         6 ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:253:13 (clickhouse+0xf392eca)
         7 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(DB::AsynchronousMetrics::start()::$_0&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf392eca)
         8 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf392eca)
         9 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf392eca)
         10 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf392eca)
         11 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xf30682e) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         12 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0xf30682e)
         13 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:421:13 (clickhouse+0xf30682e)
         14 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:183:73 (clickhouse+0xf30d3d1) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         15 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf30d3d1)
         16 void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0xf30d3d1)
         17 void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0xf30d3d1)

     SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 in DB::AsynchronousMetrics::~AsynchronousMetrics()

And the problem is real, see [2].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/52717/a6cf2297ef07ca0b065d944c9e30a3986285331d/integration_tests__tsan__[2_6]/integration_run_parallel0_0.log
  [2]: https://github.com/google/sanitizers/wiki/ThreadSanitizerPopularDataRaces#data-race-on-vptr

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 18:32:40 +01:00
Azat Khuzhin
6ccbc2ea75 Move io_uring reader into the Context from static to make it's thread joinable 
v2: fix for standalone keeper build
CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/72b1052f7c2d453308262924e767ab8dc2206933/stateless_tests__debug__[4_5].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 22:50:48 +01:00
Han Fei
881e58dfcb support soft limit ratio 2023-12-08 21:48:27 +01:00
alesapin
dc7347e0ba
Merge pull request #57271 from hanfei1991/hanfei/keeper_soft_limit 
support memory soft limit for keeper
 2023-12-08 16:12:47 +01:00
jsc0218
cdd5280272
Merge pull request #57387 from evillique/better-disks 
Initialize only required disks in clickhouse-disks
 2023-12-05 13:59:04 -05:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
Han Fei
70f436483c address part of comments 2023-12-05 00:40:15 +01:00
alesapin
acc53431c7
Update src/Coordination/KeeperDispatcher.cpp 2023-12-03 12:43:20 +01:00
alesapin
9a68461cdb
Update src/Coordination/KeeperDispatcher.cpp 2023-12-03 12:43:16 +01:00
Han Fei
e4da317f05 fix test 2023-12-01 18:07:45 +01:00
Nikolay Degterinsky
e8203f8a76 Initialize only required disks 2023-11-30 03:09:55 +00:00
Antonio Andelic
61b112e2c4
Merge pull request #57268 from ClickHouse/ignore-memory-exception-asio 
Ignore memory exception in Keeper asio workers
 2023-11-29 08:33:00 +01:00
Antonio Andelic
852b4d42b6
Update src/Coordination/KeeperServer.cpp 2023-11-28 16:00:06 +01:00
Antonio Andelic
d10aeef23e
Add comment 2023-11-28 15:32:20 +01:00
Han Fei
3aa4a6ae2c check integration test 2023-11-28 14:47:31 +01:00
Antonio Andelic
889c53eca0 Correctly handle unknown changelog versions and snapshot deleting 2023-11-28 09:47:49 +00:00
Han Fei
347efc669e support memory soft limit for keeper 2023-11-27 18:28:10 +01:00
Antonio Andelic
3941520cb0 Avoid memory exceptions in asio workers 2023-11-27 10:08:06 +00:00
Azat Khuzhin
4a02de4674 Add ability to disable checksums for S3 to avoid excessive input file read 
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
  PayloadSigningPolicy::Never)

So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.

By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.

And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.

Here is an example stacktrace of this excessive read:

<details>

<summary>stacktrace</summary>

    (lldb) bt
    * thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
      * frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
        frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
        frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
        frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
        frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
        frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
        frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
        frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
        frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
        frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
        frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
        frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
        frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
        frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
        frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
        frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
        frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
        frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()

</details>

This new behaviour could be enabled with `s3_disable_checksum=true`.

Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
 2023-11-26 19:20:19 +01:00
Antonio Andelic
b18cb1e5cd
Merge pull request #57047 from ClickHouse/keeper-small-fixes 
Small Keeper fixes
 2023-11-22 08:47:58 +01:00
Antonio Andelic
90bb796d63 Small Keeper fixes 2023-11-21 13:11:33 +00:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
Sema Checherinda
f999337dae
Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
Alexander Tokmakov
5031f239c3
Revert "s3 adaptive timeouts" 2023-11-20 14:28:59 +01:00
Sema Checherinda
a950595c24
Merge pull request #56314 from CheSema/s3-aggressive-timeouts 
s3 adaptive timeouts
 2023-11-19 14:12:14 +01:00
Alexey Milovidov
edc3b2fe48
Merge pull request #56958 from ClickHouse/metric-queued-jobs 
Add metrics for the number of queued jobs, which is useful for the IO thread pool
 2023-11-19 10:37:18 +01:00
Alexey Milovidov
75cebb3c29 Fix build 2023-11-18 20:20:21 +01:00
Alexey Milovidov
d56cbda185 Add metrics for the number of queued jobs, which is useful for the IO thread pool 2023-11-18 19:07:59 +01:00
Jianfei Hu
ef79bf6467 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into keeper-az-fix 2023-11-16 21:58:02 +00:00
Jianfei Hu
9df2775f08 reduce timeout and setTimeout earlier. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-14 17:58:16 +00:00
Antonio Andelic
03c3e96859 Small test changes 2023-11-14 17:35:17 +00:00
Sema Checherinda
8d36fd6e54 get rid off of client_with_long_timeout_ptr 2023-11-14 11:34:12 +01:00
Antonio Andelic
2f505786e2 Merge branch 'more-reliable-log-handling-keeper' into disable-compress-logs-default 2023-11-14 09:46:19 +00:00
Antonio Andelic
1ab7e5693d Add unit test 2023-11-14 09:43:09 +00:00
Antonio Andelic
3f5ccabba6 Don't append different log types in same file 2023-11-14 09:07:42 +00:00
Antonio Andelic
5849e06595 Fix 2023-11-14 09:07:17 +00:00
Antonio Andelic
8d27920987 Merge branch 'master' into more-reliable-log-handling-keeper 2023-11-14 08:38:07 +00:00
Jianfei Hu
554d907189 Fix the keeper_server availability zone configuration. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-13 23:42:51 +00:00
Jianfei Hu
d3517140d9 Revert "Revert "Add /keeper/availability-zone node to allow server load balancing within AZ."" 
This reverts commit e5aad4a6fd.
 2023-11-13 19:03:27 +00:00
Antonio Andelic
d9fbab554f
remove useless include 2023-11-13 14:52:32 +01:00
Antonio Andelic
e3c66f8a63 Merge branch 'master' into more-reliable-log-handling-keeper 2023-11-13 12:14:57 +00:00
Antonio Andelic
99d521becf More reliable log handling in Keeper 2023-11-13 12:09:13 +00:00
Alexey Milovidov
82c461031e Fix build 2023-11-13 10:09:23 +01:00
Pradeep Chhetri
d0922782ed Revert "Revert "Add 4-letter command for yielding/resigning leadership"" 
This reverts commit e1766b96d9.
 2023-11-11 15:37:49 +08:00