thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-12 09:22:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
134,406 Commits 9,297 Branches 1,782 Tags 2.1 GiB
a0507c2044
Commit Graph

1213 Commits

Author SHA1 Message Date
Azat Khuzhin
8c54380d80 Avoid sending ComposeObject requests after upload to GCS 
This should not be required anymore, but leave it as an option, since
likely this is required for old files.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
f4a7789cd4 Convert various S3::Client settings into separate ClientSettings struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Antonio Andelic
8ace9978b3 Handle another case for preprocessing 2023-12-28 13:20:47 +00:00
Azat Khuzhin
3be3b0a280 Fix incorrect Exceptions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 21:26:32 +01:00
Antonio Andelic
b9466023ce
Merge pull request #58069 from ClickHouse/dont-preprocess-if-not-ready 
Fix invalid preprocessing on Keeper
 2023-12-21 09:08:56 +01:00
Antonio Andelic
3cc8358b89 Avoid preprocessing if Keeper leader is not ready 2023-12-20 10:22:00 +00:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials 
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
 2023-12-17 10:29:15 +02:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
alesapin
cbae29d26a
Merge pull request #57699 from hanfei1991/hanfei/keeper_soft_limit_ratio 
support keeper memory soft limit ratio
 2023-12-11 21:43:24 +01:00
Azat Khuzhin
450e8da7ce Fix data race on vptr (ctor/dtor vs virtual call) in AsynchronousMetrics 
TSan report [1]:

     Exception: Sanitizer assert found for instance ==================
     WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=1)
       Write of size 8 at 0x7ffce68f5680 by main thread:
         0 DB::AsynchronousMetrics::~AsynchronousMetrics() build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 (clickhouse+0xf38fa6e) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         1 DB::ServerAsynchronousMetrics::~ServerAsynchronousMetrics() build_docker/./src/Interpreters/ServerAsynchronousMetrics.h:10:7 (clickhouse+0xf2b2f0f) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         2 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./programs/server/Server.cpp:1996:1 (clickhouse+0xf2b2f0f)
         3 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d6c95fe) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         4 DB::Server::run() build_docker/./programs/server/Server.cpp:397:25 (clickhouse+0xf29e041) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         5 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d6e9354) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         6 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:203:20 (clickhouse+0xf29b1a3) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         7 main build_docker/./programs/main.cpp:505:12 (clickhouse+0x72d7fa0) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)

       Previous read of size 8 at 0x7ffce68f5680 by thread T649:
         0 DB::AsynchronousMetrics::update(std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Common/AsynchronousMetrics.cpp:1559:5 (clickhouse+0xf38ebc5) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         1 DB::AsynchronousMetrics::run() build_docker/./src/Common/AsynchronousMetrics.cpp:354:13 (clickhouse+0xf390696) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         2 DB::AsynchronousMetrics::start()::$_0::operator()() const build_docker/./src/Common/AsynchronousMetrics.cpp:273:62 (clickhouse+0xf392eca) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         3 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&>(DB::AsynchronousMetrics::start()::$_0&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf392eca)
         4 decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1 (clickhouse+0xf392eca)
         5 decltype(auto) std::__1::apply[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1 (clickhouse+0xf392eca)
         6 ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:253:13 (clickhouse+0xf392eca)
         7 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(DB::AsynchronousMetrics::start()::$_0&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf392eca)
         8 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf392eca)
         9 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf392eca)
         10 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf392eca)
         11 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xf30682e) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         12 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0xf30682e)
         13 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:421:13 (clickhouse+0xf30682e)
         14 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:183:73 (clickhouse+0xf30d3d1) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         15 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf30d3d1)
         16 void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0xf30d3d1)
         17 void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0xf30d3d1)

     SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 in DB::AsynchronousMetrics::~AsynchronousMetrics()

And the problem is real, see [2].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/52717/a6cf2297ef07ca0b065d944c9e30a3986285331d/integration_tests__tsan__[2_6]/integration_run_parallel0_0.log
  [2]: https://github.com/google/sanitizers/wiki/ThreadSanitizerPopularDataRaces#data-race-on-vptr

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 18:32:40 +01:00
Azat Khuzhin
6ccbc2ea75 Move io_uring reader into the Context from static to make it's thread joinable 
v2: fix for standalone keeper build
CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/72b1052f7c2d453308262924e767ab8dc2206933/stateless_tests__debug__[4_5].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 22:50:48 +01:00
Han Fei
881e58dfcb support soft limit ratio 2023-12-08 21:48:27 +01:00
alesapin
dc7347e0ba
Merge pull request #57271 from hanfei1991/hanfei/keeper_soft_limit 
support memory soft limit for keeper
 2023-12-08 16:12:47 +01:00
jsc0218
cdd5280272
Merge pull request #57387 from evillique/better-disks 
Initialize only required disks in clickhouse-disks
 2023-12-05 13:59:04 -05:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
Han Fei
70f436483c address part of comments 2023-12-05 00:40:15 +01:00
alesapin
acc53431c7
Update src/Coordination/KeeperDispatcher.cpp 2023-12-03 12:43:20 +01:00
alesapin
9a68461cdb
Update src/Coordination/KeeperDispatcher.cpp 2023-12-03 12:43:16 +01:00
Han Fei
e4da317f05 fix test 2023-12-01 18:07:45 +01:00
Nikolay Degterinsky
e8203f8a76 Initialize only required disks 2023-11-30 03:09:55 +00:00
Antonio Andelic
61b112e2c4
Merge pull request #57268 from ClickHouse/ignore-memory-exception-asio 
Ignore memory exception in Keeper asio workers
 2023-11-29 08:33:00 +01:00
Antonio Andelic
852b4d42b6
Update src/Coordination/KeeperServer.cpp 2023-11-28 16:00:06 +01:00
Antonio Andelic
d10aeef23e
Add comment 2023-11-28 15:32:20 +01:00
Han Fei
3aa4a6ae2c check integration test 2023-11-28 14:47:31 +01:00
Antonio Andelic
889c53eca0 Correctly handle unknown changelog versions and snapshot deleting 2023-11-28 09:47:49 +00:00
Han Fei
347efc669e support memory soft limit for keeper 2023-11-27 18:28:10 +01:00
Antonio Andelic
3941520cb0 Avoid memory exceptions in asio workers 2023-11-27 10:08:06 +00:00
Azat Khuzhin
4a02de4674 Add ability to disable checksums for S3 to avoid excessive input file read 
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
  PayloadSigningPolicy::Never)

So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.

By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.

And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.

Here is an example stacktrace of this excessive read:

<details>

<summary>stacktrace</summary>

    (lldb) bt
    * thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
      * frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
        frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
        frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
        frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
        frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
        frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
        frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
        frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
        frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
        frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
        frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
        frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
        frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
        frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
        frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
        frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
        frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
        frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()

</details>

This new behaviour could be enabled with `s3_disable_checksum=true`.

Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
 2023-11-26 19:20:19 +01:00
Antonio Andelic
b18cb1e5cd
Merge pull request #57047 from ClickHouse/keeper-small-fixes 
Small Keeper fixes
 2023-11-22 08:47:58 +01:00
Antonio Andelic
90bb796d63 Small Keeper fixes 2023-11-21 13:11:33 +00:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
Sema Checherinda
f999337dae
Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
Alexander Tokmakov
5031f239c3
Revert "s3 adaptive timeouts" 2023-11-20 14:28:59 +01:00
Sema Checherinda
a950595c24
Merge pull request #56314 from CheSema/s3-aggressive-timeouts 
s3 adaptive timeouts
 2023-11-19 14:12:14 +01:00
Alexey Milovidov
edc3b2fe48
Merge pull request #56958 from ClickHouse/metric-queued-jobs 
Add metrics for the number of queued jobs, which is useful for the IO thread pool
 2023-11-19 10:37:18 +01:00
Alexey Milovidov
75cebb3c29 Fix build 2023-11-18 20:20:21 +01:00
Alexey Milovidov
d56cbda185 Add metrics for the number of queued jobs, which is useful for the IO thread pool 2023-11-18 19:07:59 +01:00
Jianfei Hu
ef79bf6467 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into keeper-az-fix 2023-11-16 21:58:02 +00:00
Jianfei Hu
9df2775f08 reduce timeout and setTimeout earlier. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-14 17:58:16 +00:00
Antonio Andelic
03c3e96859 Small test changes 2023-11-14 17:35:17 +00:00
Sema Checherinda
8d36fd6e54 get rid off of client_with_long_timeout_ptr 2023-11-14 11:34:12 +01:00
Antonio Andelic
2f505786e2 Merge branch 'more-reliable-log-handling-keeper' into disable-compress-logs-default 2023-11-14 09:46:19 +00:00
Antonio Andelic
1ab7e5693d Add unit test 2023-11-14 09:43:09 +00:00
Antonio Andelic
3f5ccabba6 Don't append different log types in same file 2023-11-14 09:07:42 +00:00
Antonio Andelic
5849e06595 Fix 2023-11-14 09:07:17 +00:00
Antonio Andelic
8d27920987 Merge branch 'master' into more-reliable-log-handling-keeper 2023-11-14 08:38:07 +00:00
Jianfei Hu
554d907189 Fix the keeper_server availability zone configuration. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-13 23:42:51 +00:00
Jianfei Hu
d3517140d9 Revert "Revert "Add /keeper/availability-zone node to allow server load balancing within AZ."" 
This reverts commit e5aad4a6fd.
 2023-11-13 19:03:27 +00:00
Antonio Andelic
d9fbab554f
remove useless include 2023-11-13 14:52:32 +01:00
Antonio Andelic
e3c66f8a63 Merge branch 'master' into more-reliable-log-handling-keeper 2023-11-13 12:14:57 +00:00
Antonio Andelic
99d521becf More reliable log handling in Keeper 2023-11-13 12:09:13 +00:00
Alexey Milovidov
82c461031e Fix build 2023-11-13 10:09:23 +01:00
Pradeep Chhetri
d0922782ed Revert "Revert "Add 4-letter command for yielding/resigning leadership"" 
This reverts commit e1766b96d9.
 2023-11-11 15:37:49 +08:00
Alexey Milovidov
e1766b96d9
Revert "Add 4-letter command for yielding/resigning leadership" 2023-11-11 05:45:02 +03:00
Alexey Milovidov
e5aad4a6fd
Revert "Add /keeper/availability-zone node to allow server load balancing within AZ." 2023-11-11 05:43:58 +03:00
Mike Kot
785183bb3c Keeper reconfig -- sleep before removing leader to avoid losing accepted 
but uncommitted commands
 2023-11-09 16:22:44 +00:00
Antonio Andelic
788c66027c
Merge pull request #56104 from incfly/keeper-az 
Add /keeper/availability-zone node to allow server load balancing within AZ.
 2023-11-08 09:46:24 +01:00
Pradeep Chhetri
4e8ff997c0 fix 2023-11-07 14:20:40 +08:00
Pradeep Chhetri
201a7b8d5f fix 2023-11-07 14:06:55 +08:00
Pradeep Chhetri
74b53218b8 Rename yield_leader with yield_leadship and its variants 2023-11-07 13:58:45 +08:00
Pradeep Chhetri
361472b29e Add 4-letter command for yielding/resigning leadership 2023-11-07 13:58:45 +08:00
Jianfei Hu
b426cfe462 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into keeper-az 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-03 16:42:29 +00:00
Jianfei Hu
06ff714e2e add configuration option and integration test(wip). 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-02 16:21:24 +00:00
Antonio Andelic
1a75970486 PR review 2023-11-02 11:57:26 +00:00
Antonio Andelic
28edb77e72 Merge branch 'master' into keeper-better-preprocessing 2023-11-02 09:26:35 +00:00
Jianfei Hu
e29b7640c4 gcp metadata use dns. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-01 17:53:34 +00:00
Jianfei Hu
b22773ccd1
Apply suggestions from code review 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-11-01 10:30:29 -07:00
Alexander Gololobov
78b9da692c Do less work when there a lots of watches 2023-11-01 12:27:10 +01:00
Jianfei Hu
6f94770a8e more fix -DUSE_AWS_S3=0 case. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-01 00:51:45 +00:00
Jianfei Hu
3fff813659 style fix and revert src/Coordination/KeeperSnapshotManagerS3.h 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 18:48:59 +00:00
Jianfei Hu
cfa1b96d23 remove comments. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 17:43:24 +00:00
Jianfei Hu
573054d755 add new code and rework. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 04:43:44 +00:00
Jianfei Hu
6d72873bdb use poco better works. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 02:34:50 +00:00
Jianfei Hu
dec8efc39e make az as attribute of keepersnapshotmangers3. 
empty config not working.
https://pastila.nl/?00048e4f/d56af90d13f2d871e30dfb1db9660266#yff6d8HVQm0blQUUiZg8SQ==

Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-29 02:04:38 +00:00
Jianfei Hu
0b301f73c1 initial add keeper az info in /keeper/availabilty-zone 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-28 21:54:59 +00:00
Alexander Gololobov
9a95b82cae Avoid traversing potentially big list of watches 2023-10-27 14:54:58 +02:00
Antonio Andelic
674ffa1d25 Fix install snapshot 2023-10-17 12:56:55 +00:00
Antonio Andelic
17eaa5a35e Fix edge cases 2023-10-16 13:19:25 +00:00
Antonio Andelic
9d218a8c7b Better 2023-10-16 08:50:53 +00:00
Antonio Andelic
2d851ec163 Better 2023-10-16 08:22:39 +00:00
Antonio Andelic
58120b3f47 Polish 2023-10-16 08:17:23 +00:00
Antonio Andelic
606f742254 Merge branch 'master' into keeper-better-preprocessing 2023-10-16 06:57:19 +00:00
Antonio Andelic
c69838e052
Merge pull request #55591 from ClickHouse/fix-keeper-context 
Apply Context changes to standalone Keeper
 2023-10-16 08:41:17 +02:00
alesapin
3b02748cb6 Fix some typos 2023-10-15 15:43:02 +02:00
Antonio Andelic
67068849da
Fix comment 2023-10-13 15:59:37 +02:00
Azat Khuzhin
720ea8441c Fix global context for tests with --gtest_filter 
If you run tests that requires context, but do not initialize it, then
it will SIGSEGV, because Context is not initialized.

Fix this by using google test envrionment, and instead of gtest_main
implement own main function that will initialize it.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-13 13:44:30 +02:00
Antonio Andelic
48933f52fe Fix Keeper Context 2023-10-13 11:30:37 +00:00
Antonio Andelic
34d8bb18cf Better preprocessing in Keeper 2023-09-29 14:34:17 +00:00
Antonio Andelic
01f64be558 Set path in response 2023-09-26 07:41:05 +00:00
Antonio Andelic
176baa6695 Merge branch 'master' into keeper-operation-create-if-not-exists 2023-09-22 13:47:30 +00:00
Antonio Andelic
b65b1ee97b Merge branch 'master' into keeper-operation-create-if-not-exists 2023-09-20 13:56:36 +00:00
Antonio Andelic
88ab5514a0 Merge branch 'master' into keeper-reduce-batch-size 2023-09-20 13:56:06 +00:00
Sema Checherinda
e7550523c8
Merge pull request #54651 from CheSema/limit_backoff_timeout 
limit the delay before next try in S3
 2023-09-18 19:41:26 +02:00
Antonio Andelic
7f3279fa21 Revert batch size to 100 for Keeper 2023-09-18 11:18:39 +00:00
Antonio Andelic
69333912d5 Merge branch 'master' into keeper-operation-create-if-not-exists 2023-09-18 10:57:11 +00:00
Robert Schulze
7d04cf6b58
Merge pull request #54646 from rschu1ze/bump-abseil 
Bump abseil
 2023-09-18 09:44:35 +02:00
Robert Schulze
5f46106984
Try to fix sizes in unit tests, pt. III 2023-09-17 14:41:51 +00:00
Robert Schulze
066bdf57c4
Try to fix sizes in unit test, pt. II 2023-09-15 14:53:05 +00:00
Robert Schulze
603b3c4b26
Try to fix sizes in unit test 2023-09-15 09:32:43 +00:00
Sema Checherinda
d9e15c00c9 limit the delay before next try in S3 2023-09-14 19:45:07 +02:00