thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-13 09:52:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
140,106 Commits 9,324 Branches 1,783 Tags 2.1 GiB
642da7934a
Commit Graph

1285 Commits

Author SHA1 Message Date
Antonio Andelic
0dc73bacc5 Fix cleanup 2024-02-19 15:09:10 +01:00
Antonio Andelic
299a0bd996 Merge branch 'master' into keeper-logs-cache 2024-02-19 10:01:31 +01:00
Alexey Milovidov
6fd563df52 Miscellaneous 2024-02-19 02:58:51 +01:00
Antonio Andelic
b01ccbbc88 Fix 2024-02-16 16:58:11 +01:00
Igor Nikonov
cb0ce2aaa9 Fix build 2024-02-16 15:26:08 +00:00
Antonio Andelic
e12fa6b35c Better snapshot reading 2024-02-16 14:40:49 +01:00
Antonio Andelic
ddfa3dea87 Merge branch 'master' into keeper-logs-cache 2024-02-16 10:07:06 +01:00
Antonio Andelic
0f489de46e
Merge pull request #59945 from ClickHouse/keeper-prometheus-filtered-events 
Send only Keeper related metrics/events for Prometheus
 2024-02-16 09:43:43 +01:00
Antonio Andelic
daf7505e5e Improve 2024-02-15 16:11:20 +01:00
Antonio Andelic
1ade535a83 Merge branch 'master' into keeper-logs-cache 2024-02-15 13:06:37 +01:00
Antonio Andelic
40d4fc1c24 More resilient Disk related operations 2024-02-14 14:53:22 +01:00
Konstantin Bogdanov
f678bdc546
Update tests 2024-02-14 10:51:42 +01:00
Antonio Andelic
0d151e3f0d Send only specific events/metrics for Keeper prometheus 2024-02-13 16:01:50 +01:00
Antonio Andelic
a0507c2044 Whitespace 2024-02-12 10:36:35 +01:00
Antonio Andelic
c9c0e12b4c Address PR reviews 2024-02-12 09:15:40 +01:00
Antonio Andelic
d2eac53c86 Fix 2024-02-09 12:06:01 +01:00
Antonio Andelic
42f2fefeab Fix data race and leak 2024-02-09 10:40:43 +01:00
Antonio Andelic
dfb3a8f7be Another fix 2024-02-08 17:17:12 +01:00
Antonio Andelic
99be5b5cb1 Fix 2024-02-07 15:57:52 +01:00
Antonio Andelic
1279964866
fix build 2024-02-06 10:18:02 +01:00
Antonio Andelic
754b73a38b
remove unused 2024-02-06 09:35:48 +01:00
Antonio Andelic
a0170bcbd5 Fix utils 2024-02-05 17:58:33 +01:00
Antonio Andelic
7ad48c2aa2 Fix 2024-02-05 17:02:16 +01:00
Antonio Andelic
2116043e2c Merge branch 'master' into keeper-more-reduce 2024-02-05 14:58:26 +01:00
Antonio Andelic
b1b564a835 Fix build again 2024-02-01 16:42:11 +00:00
Antonio Andelic
533f7d0314 Build fix 2024-02-01 16:09:00 +00:00
Antonio Andelic
c197fb27d6 Better 2024-02-01 10:42:16 +00:00
Antonio Andelic
b53abf8186 Finalize 2024-02-01 10:04:34 +00:00
Antonio Andelic
82196cac99 Merge branch 'master' into keeper-logs-cache 2024-02-01 08:56:12 +00:00
Antonio Andelic
08f91907dc Fix digest calculation 2024-01-31 16:08:22 +00:00
Antonio Andelic
a91101a21f Working implementation 2024-01-31 15:25:24 +00:00
Antonio Andelic
bd776b4c16 Merge branch 'master' into keeper-logs-cache 2024-01-29 10:26:23 +00:00
Maksim Kita
2a327107b6 Updated implementation 2024-01-25 14:31:49 +03:00
Antonio Andelic
8c7218bac2 Store latest logs inmemory 2024-01-25 08:46:06 +00:00
Antonio Andelic
3e3ae52aca Reduce even more memory 2024-01-25 08:40:19 +00:00
Antonio Andelic
34463fd5e5
Merge pull request #59002 from ClickHouse/keeper-reduce-memory 
Reduce Keeper memory usage
 2024-01-24 11:54:16 +01:00
Antonio Andelic
74dd68afbc
Merge pull request #59102 from ClickHouse/keeper-startup-fix 
Avoid election timeouts during startup in Keeper
 2024-01-23 13:55:52 +01:00
Robert Schulze
e23e7e6b73
Merge pull request #53710 from rschu1ze/system-reload-asynchronous-metrics 
Add statement `SYSTEM RELOAD ASYNCHRONOUS METRICS`
 2024-01-23 11:56:25 +01:00
Antonio Andelic
9ce3f8a7aa
typo 2024-01-23 09:18:51 +01:00
Antonio Andelic
ff644831e2 Avoid election timeouts during startup 2024-01-23 08:02:30 +00:00
Robert Schulze
90a0ea36b3
Add statement 'SYSTEM RELOAD ASYNCHRONOUS METRICS' 2024-01-22 17:14:58 +00:00
Robert Schulze
1262945df7
Cosmetics 2024-01-22 16:51:02 +00:00
alesapin
478b91cc37
Merge pull request #59010 from ClickHouse/alesapin-patch-1 
Change log level for super imporant message in Keeper
 2024-01-22 13:16:56 +01:00
Alexander Sapin
f7b21b6016 Fix build 2024-01-22 11:53:21 +01:00
Antonio Andelic
274c128bd7 Free memory 2024-01-22 10:48:39 +00:00
Antonio Andelic
59f9abcf58 Merge branch 'master' into keeper-reduce-memory 2024-01-22 08:04:17 +00:00
Antonio Andelic
ecaef80ea0
Merge pull request #58938 from ClickHouse/keeper-fetch-max-file-desc-once 
Call `getMaxFileDescriptorCount` once in Keeper
 2024-01-22 08:57:58 +01:00
alesapin
72f5dce9f4
Update src/Coordination/KeeperDispatcher.cpp 
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2024-01-21 11:16:27 +01:00
serxa
824deab761 split resource scheduler off IO/ into Common/Scheduler/ 2024-01-20 12:06:26 +00:00
alesapin
d275d79947
Update KeeperDispatcher.cpp 2024-01-19 19:25:27 +01:00
Antonio Andelic
0132455b39 Reduce Keeper memory usage 2024-01-19 17:00:53 +00:00
Antonio Andelic
19ca00f233 Call getMaxFileDescriptorCount once 2024-01-19 08:19:20 +00:00
Antonio Andelic
30a04471ac
Merge pull request #58806 from ClickHouse/fix-keeper-data-race 
Fix data race in Keeper
 2024-01-18 08:48:16 +01:00
Antonio Andelic
ded830ba33 Correctly handle shutdown 2024-01-17 08:56:36 +00:00
Pradeep Chhetri
09f29b4502 Update nomenclature of variables 2024-01-17 13:52:10 +08:00
Alexey Milovidov
d85abfafa4
Update KeeperAsynchronousMetrics.cpp 2024-01-17 08:05:48 +03:00
Alexey Milovidov
90e35be861
Update KeeperAsynchronousMetrics.cpp 2024-01-17 08:04:32 +03:00
Pradeep Chhetri
eafaab3795 Add metric for keeper memory soft limit 2024-01-17 12:56:18 +08:00
Antonio Andelic
7ecf146072 Fix data race in Keeper 2024-01-15 12:23:59 +00:00
Antonio Andelic
e7375d6785 Don't process request if not preprocessed 2024-01-12 15:49:30 +00:00
Antonio Andelic
2bfed6b8eb Add system commands and 4lw for jemalloc 2024-01-11 08:24:56 +00:00
Sema Checherinda
7c7e72c4b7
Merge pull request #58573 from ClickHouse/chesema-s3-client-creation 
fix and test that S3Clients are reused
 2024-01-09 13:28:20 +01:00
Raúl Marín
803d8d6320
Merge pull request #58594 from Algunenano/missing_includes 
Add some missing LLVM includes
 2024-01-09 11:01:44 +01:00
Antonio Andelic
a891750f53 Fix race 2024-01-08 12:08:17 +00:00
Antonio Andelic
6aee18ced8
Merge pull request #58487 from ClickHouse/keeper-lower-log-levels-test 
Lower log levels for some Raft logs to new test level
 2024-01-08 12:19:02 +01:00
Raúl Marín
c1b95560db Add some missing LLVM includes 2024-01-08 11:25:57 +01:00
Sema Checherinda
d5f86f671d fix and test that S3Clients reused 2024-01-07 02:19:06 +01:00
Antonio Andelic
13749f550f Lower log levels for some Raft logs 2024-01-04 07:59:44 +00:00
Antonio Andelic
8f26e2af67 Even better Keeper startup 2024-01-04 07:54:14 +00:00
Antonio Andelic
d86e59eb08
Merge pull request #58308 from ClickHouse/keeper-preprocess-fix 
Handle another case for preprocessing in Keeper
 2023-12-30 14:13:04 +01:00
Alexey Milovidov
8365722c95
Merge pull request #58336 from amosbird/macro2constexpr 
Organize symbols from src/* into DB namespace
 2023-12-30 12:40:36 +01:00
Amos Bird
0d801c4f6b
Fix unitest 2023-12-29 20:40:52 +08:00
Azat Khuzhin
8c54380d80 Avoid sending ComposeObject requests after upload to GCS 
This should not be required anymore, but leave it as an option, since
likely this is required for old files.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
f4a7789cd4 Convert various S3::Client settings into separate ClientSettings struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Antonio Andelic
8ace9978b3 Handle another case for preprocessing 2023-12-28 13:20:47 +00:00
Azat Khuzhin
3be3b0a280 Fix incorrect Exceptions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 21:26:32 +01:00
Antonio Andelic
b9466023ce
Merge pull request #58069 from ClickHouse/dont-preprocess-if-not-ready 
Fix invalid preprocessing on Keeper
 2023-12-21 09:08:56 +01:00
Antonio Andelic
3cc8358b89 Avoid preprocessing if Keeper leader is not ready 2023-12-20 10:22:00 +00:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials 
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
 2023-12-17 10:29:15 +02:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
alesapin
cbae29d26a
Merge pull request #57699 from hanfei1991/hanfei/keeper_soft_limit_ratio 
support keeper memory soft limit ratio
 2023-12-11 21:43:24 +01:00
Azat Khuzhin
450e8da7ce Fix data race on vptr (ctor/dtor vs virtual call) in AsynchronousMetrics 
TSan report [1]:

     Exception: Sanitizer assert found for instance ==================
     WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=1)
       Write of size 8 at 0x7ffce68f5680 by main thread:
         0 DB::AsynchronousMetrics::~AsynchronousMetrics() build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 (clickhouse+0xf38fa6e) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         1 DB::ServerAsynchronousMetrics::~ServerAsynchronousMetrics() build_docker/./src/Interpreters/ServerAsynchronousMetrics.h:10:7 (clickhouse+0xf2b2f0f) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         2 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./programs/server/Server.cpp:1996:1 (clickhouse+0xf2b2f0f)
         3 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d6c95fe) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         4 DB::Server::run() build_docker/./programs/server/Server.cpp:397:25 (clickhouse+0xf29e041) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         5 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d6e9354) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         6 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:203:20 (clickhouse+0xf29b1a3) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         7 main build_docker/./programs/main.cpp:505:12 (clickhouse+0x72d7fa0) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)

       Previous read of size 8 at 0x7ffce68f5680 by thread T649:
         0 DB::AsynchronousMetrics::update(std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Common/AsynchronousMetrics.cpp:1559:5 (clickhouse+0xf38ebc5) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         1 DB::AsynchronousMetrics::run() build_docker/./src/Common/AsynchronousMetrics.cpp:354:13 (clickhouse+0xf390696) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         2 DB::AsynchronousMetrics::start()::$_0::operator()() const build_docker/./src/Common/AsynchronousMetrics.cpp:273:62 (clickhouse+0xf392eca) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         3 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&>(DB::AsynchronousMetrics::start()::$_0&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf392eca)
         4 decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1 (clickhouse+0xf392eca)
         5 decltype(auto) std::__1::apply[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1 (clickhouse+0xf392eca)
         6 ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:253:13 (clickhouse+0xf392eca)
         7 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(DB::AsynchronousMetrics::start()::$_0&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf392eca)
         8 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf392eca)
         9 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf392eca)
         10 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf392eca)
         11 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xf30682e) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         12 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0xf30682e)
         13 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:421:13 (clickhouse+0xf30682e)
         14 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:183:73 (clickhouse+0xf30d3d1) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         15 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf30d3d1)
         16 void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0xf30d3d1)
         17 void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0xf30d3d1)

     SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 in DB::AsynchronousMetrics::~AsynchronousMetrics()

And the problem is real, see [2].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/52717/a6cf2297ef07ca0b065d944c9e30a3986285331d/integration_tests__tsan__[2_6]/integration_run_parallel0_0.log
  [2]: https://github.com/google/sanitizers/wiki/ThreadSanitizerPopularDataRaces#data-race-on-vptr

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 18:32:40 +01:00
Azat Khuzhin
6ccbc2ea75 Move io_uring reader into the Context from static to make it's thread joinable 
v2: fix for standalone keeper build
CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/72b1052f7c2d453308262924e767ab8dc2206933/stateless_tests__debug__[4_5].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 22:50:48 +01:00
Han Fei
881e58dfcb support soft limit ratio 2023-12-08 21:48:27 +01:00
alesapin
dc7347e0ba
Merge pull request #57271 from hanfei1991/hanfei/keeper_soft_limit 
support memory soft limit for keeper
 2023-12-08 16:12:47 +01:00
jsc0218
cdd5280272
Merge pull request #57387 from evillique/better-disks 
Initialize only required disks in clickhouse-disks
 2023-12-05 13:59:04 -05:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
Han Fei
70f436483c address part of comments 2023-12-05 00:40:15 +01:00
alesapin
acc53431c7
Update src/Coordination/KeeperDispatcher.cpp 2023-12-03 12:43:20 +01:00
alesapin
9a68461cdb
Update src/Coordination/KeeperDispatcher.cpp 2023-12-03 12:43:16 +01:00
Han Fei
e4da317f05 fix test 2023-12-01 18:07:45 +01:00
Nikolay Degterinsky
e8203f8a76 Initialize only required disks 2023-11-30 03:09:55 +00:00
Antonio Andelic
61b112e2c4
Merge pull request #57268 from ClickHouse/ignore-memory-exception-asio 
Ignore memory exception in Keeper asio workers
 2023-11-29 08:33:00 +01:00
Antonio Andelic
852b4d42b6
Update src/Coordination/KeeperServer.cpp 2023-11-28 16:00:06 +01:00
Antonio Andelic
d10aeef23e
Add comment 2023-11-28 15:32:20 +01:00
Han Fei
3aa4a6ae2c check integration test 2023-11-28 14:47:31 +01:00
Antonio Andelic
889c53eca0 Correctly handle unknown changelog versions and snapshot deleting 2023-11-28 09:47:49 +00:00
Han Fei
347efc669e support memory soft limit for keeper 2023-11-27 18:28:10 +01:00
Antonio Andelic
3941520cb0 Avoid memory exceptions in asio workers 2023-11-27 10:08:06 +00:00
Azat Khuzhin
4a02de4674 Add ability to disable checksums for S3 to avoid excessive input file read 
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
  PayloadSigningPolicy::Never)

So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.

By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.

And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.

Here is an example stacktrace of this excessive read:

<details>

<summary>stacktrace</summary>

    (lldb) bt
    * thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
      * frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
        frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
        frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
        frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
        frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
        frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
        frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
        frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
        frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
        frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
        frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
        frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
        frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
        frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
        frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
        frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
        frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
        frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()

</details>

This new behaviour could be enabled with `s3_disable_checksum=true`.

Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
 2023-11-26 19:20:19 +01:00