thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-04 05:22:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
138,929 Commits 9,182 Branches 1,782 Tags 2.3 GiB
05db73f518
Commit Graph

3535 Commits

Author SHA1 Message Date
Sema Checherinda
1bbb5f7f51
Merge pull request #59015 from ClickHouse/chesema-adaptive-timeouts-gcp 
preserve 'amz-sdk-invocation-id' and 'amz-sdk-request' headers with gcp
 2024-01-20 17:48:20 +01:00
serxa
172d68ceca Merge branch 'master' into cpu-scheduler 2024-01-20 12:21:21 +00:00
serxa
824deab761 split resource scheduler off IO/ into Common/Scheduler/ 2024-01-20 12:06:26 +00:00
Alexey Milovidov
8acc981e8c
Merge pull request #59008 from ClickHouse/fix-log-level-for-http 
Update log level for http buffer
 2024-01-20 04:02:42 +01:00
Kruglov Pavel
20379d34d5
Merge branch 'master' into variant-data-type 2024-01-19 23:17:42 +01:00
Sema Checherinda
539451bbe3 preserve 'amz-sdk-invocation-id' and 'amz-sdk-request' headers with gcp 2024-01-19 21:07:47 +01:00
kssenii
4c8ced6af8 Fix log level 2024-01-19 18:04:58 +01:00
Alexey Milovidov
6a75641048
Merge pull request #58966 from ClickHouse/arseek 
Fix seek in ReadBufferFromZipArchive
 2024-01-19 18:04:03 +01:00
Vitaly Baranov
19bc0b744f Add comments and const after review. 2024-01-19 12:54:55 +01:00
Alexey Milovidov
dcdbaf9150
Merge pull request #58900 from MikhailBurdukov/restriction_for_s3_access_key_id 
Restriction for the access key id for s3.
 2024-01-19 04:13:17 +01:00
Alexey Milovidov
370815803e
Update Client.cpp 2024-01-19 06:12:44 +03:00
Alexey Milovidov
7c0b167e2d Merge branch 'master' of github.com:ClickHouse/ClickHouse into enable-coverage-for-debug-build 2024-01-19 03:59:27 +01:00
Michael Kolupaev
3a19ea6e6c Fix seek in ReadBufferFromZipArchive 2024-01-19 01:46:07 +00:00
Vitaly Baranov
24ca8fbbec Improve logs. 2024-01-18 21:15:39 +01:00
Vitaly Baranov
0caafe4810 Fix memory deallocation in copyDataToS3File() 
- ReadBufferFromS3 is created now only when it's necessary and destroyed after that.
 2024-01-18 20:48:56 +01:00
Nikita Taranov
b5e333df96 remove test 2024-01-18 19:56:02 +01:00
Kruglov Pavel
5444cde408
Merge branch 'master' into variant-data-type 2024-01-18 18:31:27 +01:00
MikhailBurdukov
b3d74fc907 style 2024-01-18 15:39:25 +00:00
MikhailBurdukov
eac310fcb2 Review 2024-01-18 15:19:11 +00:00
Smita Kulkarni
e71977b724 Merge branch 'master' into Azure_backup 2024-01-18 08:29:01 +01:00
Yakov Olkhovskiy
490bb809c3
Merge pull request #58846 from ClickHouse/fix-compress-destructor 
Fix HTTP compressors finalization
 2024-01-17 17:49:12 -05:00
Alexey Milovidov
c098f41dbc Merge branch 'master' into enable-coverage-for-debug-build 2024-01-17 22:35:51 +01:00
Alexey Milovidov
f499cf3fd5
Update Client.cpp 2024-01-17 22:30:18 +03:00
Alexander Gololobov
0a42dbafe8 Add attributes 2024-01-17 17:46:15 +01:00
Nikita Taranov
1db805dfa4 impl 2024-01-17 16:24:24 +01:00
Alexander Gololobov
7dda18c209 Spans in mergeBlock and finalizeImpl 2024-01-17 16:07:53 +01:00
Anton Popov
245e1cbd5c Merge remote-tracking branch 'upstream/master' into better-read-buffers 2024-01-17 13:46:48 +00:00
Nikita Taranov
5485d2a2ec impl 2024-01-17 14:38:06 +01:00
Nikita Taranov
bbf070caa6 impl 2024-01-17 14:08:02 +01:00
Yakov Olkhovskiy
d5d060a17e fix memory leak in bzip2 2024-01-17 12:53:37 +00:00
Kruglov Pavel
6d064512e1
Merge pull request #58614 from Blargian/58363_disable_ansi_pretty_automatically 
58363 Automatically disable ANSI escape sequences in Pretty formats if the output is not a terminal
 2024-01-17 13:45:41 +01:00
Nikita Taranov
bb124b544c
Merge pull request #58077 from nickitat/revert_rest 
Remove unneeded parameter `use_external_buffer` from `AsynchronousReadBuffer*`
 2024-01-17 13:45:07 +01:00
MikhailBurdukov
e2729ed4c9 Restriction for the access key id for s3. 2024-01-17 10:43:58 +00:00
Yakov Olkhovskiy
e6b3f69137 style fix 2024-01-17 02:58:18 +00:00
Yakov Olkhovskiy
e06e4695e1 refactoring bzip2 buffer and fix possible data loss in internal buffer 2024-01-17 01:55:38 +00:00
Anton Popov
299293c088 better interface in read buffers 2024-01-16 23:26:20 +00:00
Yakov Olkhovskiy
92d42567fa separate explicit finalization for buffers 2024-01-16 22:15:35 +00:00
Yakov Olkhovskiy
fcd4297d20 fix compressors finalization 2024-01-16 00:22:17 +00:00
xleoken
6f7e7c95ba Ecloud EOS Storage support 2024-01-15 11:07:43 -05:00
Alexey Milovidov
e4cd02ea39 Fix typo 2024-01-15 00:36:24 +01:00
Alexey Milovidov
d112492c56 Remove some code 2024-01-13 03:48:04 +01:00
Blargian
aa8876a611 #58363 - Changes based on review of draft PR - changed output_format_pretty_color to use UInt64Auto. Added isWritingToTerminal function to IO/WriteHelpers.h and updated test 2024-01-12 12:31:57 +02:00
Ilya Yatsishin
904c02d955
Fix 2024-01-12 12:43:11 +04:00
Alexey Milovidov
df04476037
Update src/IO/TimeoutSetter.cpp 
Co-authored-by: Konstantin Bogdanov <thevar1able@users.noreply.github.com>
 2024-01-12 07:41:24 +03:00
Alexey Milovidov
6c604ea1a7 Fix Mac OS X 2024-01-12 04:00:36 +01:00
Yakov Olkhovskiy
4fc11e189c
Merge branch 'master' into revert-58450-revert-56064-feature-server-iface-metrics 2024-01-10 12:03:47 -05:00
Yakov Olkhovskiy
45d5e29ba0 master merge adjustments 2024-01-10 16:55:09 +00:00
Robert Schulze
52d75ab68f
Merge pull request #58458 from rschu1ze/replace-std_regexp-by-re2 
Replace `std::regex` by re2
 2024-01-10 17:46:54 +01:00
Vitaly Baranov
dbb19e72b3
Merge pull request #58629 from vitlibar/remove-finalize-from-destructor-of-write-buffer-from-encrypted-file 
Remove finalize() from ~WriteBufferFromEncryptedFile
 2024-01-10 16:36:11 +01:00
Yakov Olkhovskiy
2e124b19b8
Merge branch 'master' into revert-58450-revert-56064-feature-server-iface-metrics 2024-01-10 09:05:18 -05:00
Robert Schulze
499227b9cf
Merge remote-tracking branch 'rschu1ze/master' into replace-std_regexp-by-re2 2024-01-10 10:02:53 +00:00
pufit
6cf55b82f4
Merge pull request #58539 from canhld94/file_custom_compress_level 
Allow explicitly set compression level in output format
 2024-01-09 13:43:38 -05:00
Vitaly Baranov
01dfe282ea Remove finalize() from ~WriteBufferFromEncryptedFile. 2024-01-09 17:22:17 +01:00
Sema Checherinda
7c7e72c4b7
Merge pull request #58573 from ClickHouse/chesema-s3-client-creation 
fix and test that S3Clients are reused
 2024-01-09 13:28:20 +01:00
Kruglov Pavel
b947609b8e
Merge branch 'master' into variant-data-type 2024-01-08 15:04:51 +01:00
Robert Schulze
f553b55e3a
Merge remote-tracking branch 'rschu1ze/master' into regex-std-re2 2024-01-07 22:31:35 +00:00
Robert Schulze
8e804487f3
Some fixups 2024-01-07 22:28:08 +00:00
Sema Checherinda
6f626d8294 fix auth_settings.hasUpdates function 2024-01-07 23:00:26 +01:00
Smita Kulkarni
fd92c1961e Fix clang tidy build 2024-01-07 16:33:48 +01:00
Yakov Olkhovskiy
45ac4d2b67 Merge remote-tracking branch 'origin/master' into revert-58450-revert-56064-feature-server-iface-metrics 2024-01-07 15:23:46 +00:00
Sema Checherinda
d5f86f671d fix and test that S3Clients reused 2024-01-07 02:19:06 +01:00
avogar
7e5ba62017 Allow to read Bool values into String in JSON input formats 2024-01-05 20:33:30 +00:00
Smita Kulkarni
b250acff78 Fixed style check 2024-01-05 11:26:32 +01:00
Duc Canh Le
29c4d96b48 fix build and add a tests 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-05 06:54:56 +00:00
Smita Kulkarni
c14605caa7 Added flag use_native_copy and updated to use StartCopyFromUri for native copy with large files 2024-01-04 18:27:54 +01:00
Smita Kulkarni
91bad5bc39 Updated to use MultiVersion for BlobContainerClient in Backups and updated to get client from disk 2024-01-04 16:06:36 +01:00
Duc Canh Le
2e14cfb526 add settings for output compression level and window size 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-04 08:16:00 +00:00
Yakov Olkhovskiy
85f03478ef
Revert "Revert "Use CH Buffer for HTTP out stream, add metrics for interfaces"" 2024-01-03 11:47:15 -05:00
Kruglov Pavel
4d8cf71ba7
Merge branch 'master' into variant-data-type 2024-01-03 15:21:23 +01:00
Kruglov Pavel
570d1c013b
Merge pull request #57053 from Avogar/better-parsing-exceptions 
Better exception messages in input formats
 2024-01-03 14:47:51 +01:00
Smita Kulkarni
df221f7db6 Renamed Bucket-Key to Container-Blob 2024-01-03 11:35:06 +01:00
Raúl Marín
d491758939
Revert "Use CH Buffer for HTTP out stream, add metrics for interfaces" 2024-01-03 10:42:15 +01:00
Smita Kulkarni
4122de9721 Updated tests and added settings 2024-01-02 20:19:01 +01:00
Yakov Olkhovskiy
5633fb8145
Merge pull request #56064 from ClickHouse/feature-server-iface-metrics 
Use CH Buffer for HTTP out stream, add metrics for interfaces
 2024-01-02 10:17:52 -05:00
Kruglov Pavel
e69bda7f08
Merge pull request #57438 from Avogar/fix-resetting-buffers 
Fix working with read buffers in StreamingFormatExecutor
 2024-01-02 15:34:35 +01:00
Alexey Milovidov
2f3e17c737 Remove commented code 2023-12-30 18:39:12 +01:00
Alexey Milovidov
a1bd76388e Remove commented code 2023-12-30 18:38:41 +01:00
Alexey Milovidov
aad63c8273 Merge branch 'master' into feature-server-iface-metrics 2023-12-30 15:44:42 +01:00
Alexey Milovidov
8365722c95
Merge pull request #58336 from amosbird/macro2constexpr 
Organize symbols from src/* into DB namespace
 2023-12-30 12:40:36 +01:00
Amos Bird
907e844635
Fix examples 2023-12-30 10:48:19 +08:00
Alexey Milovidov
aa462fa7cd
Merge branch 'master' into feature-server-iface-metrics 2023-12-29 22:37:52 +01:00
Amos Bird
18395a387c
Fix another unitest 2023-12-30 01:14:21 +08:00
Kruglov Pavel
c03e36e012
Merge branch 'master' into better-parsing-exceptions 2023-12-29 18:07:32 +01:00
Azat Khuzhin
8c54380d80 Avoid sending ComposeObject requests after upload to GCS 
This should not be required anymore, but leave it as an option, since
likely this is required for old files.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
f4a7789cd4 Convert various S3::Client settings into separate ClientSettings struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
2b2d10689b Make S3::Client::ComposeObject private 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
avogar
c6085b3eb9 Fix working with read buffers in StreamingFormatExecutor 2023-12-28 18:47:57 +00:00
Smita Kulkarni
bf60a06a11 Merge branch 'master' into Azure_backup 2023-12-28 11:51:49 +01:00
Azat Khuzhin
c6774cfe42 Add missing includes for ZipArchiveWriter 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 17:06:17 +01:00
Azat Khuzhin
cf1dae5321 Add missing includes into BitHelpers.h 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 17:06:17 +01:00
Azat Khuzhin
b9233f6d4f Move Allocator code into module part 
This should reduce amount of code that should be recompiled on
Exception.h changes (and everything else that had been included there).

This will actually not help a lot, because it is also included into
PODArray.h and ThreadPool.h at least... Sigh.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 15:42:08 +01:00
Smita Kulkarni
d46d914521 Updated thread name 2023-12-27 10:28:52 +01:00
Yarik Briukhovetskyi
41c275274b
Merge pull request #58148 from yariks5s/fix_s3_regions 
S3-links region independency
 2023-12-26 12:59:00 +01:00
Alexey Milovidov
817d0c6553
Merge pull request #58202 from vitlibar/explicit-finalize-in-zip-archive-writer 
Explicit finalize in ZipArchiveWriter
 2023-12-24 03:30:21 +01:00
Vitaly Baranov
e1a136b791 Explicit finalize() function in ZipArchiveWriter. 
Simplify too complicated code in ZipArchiveWriter.
 2023-12-24 00:33:59 +01:00
Alexey Milovidov
a400657155
Merge pull request #58165 from azat/s3-passthrough-exception 
Pass through exceptions for reading from S3
 2023-12-23 11:35:23 +01:00
Azat Khuzhin
896cbb6a75 Retry on poco exceptions while reading from S3 as well 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-23 09:07:43 +01:00
Alexey Milovidov
ffca9bc564 Fix #58171 2023-12-23 02:04:35 +01:00
Azat Khuzhin
17e21aa21e Pass throught exceptions for reading from S3 
Reading from S3 has retries on almost any exception, except for
CANNOT_ALLOCATE_MEMORY, but the problem is that this exception may not
be even delivered to ReadBufferFromS3::processException(), due to
std::istream will hide it (see [1]).

  [1]: 1834e42289/libcxx/include/istream (L1059)

So let's pass throught them to see at least some information instead of
just:

    Message: Cannot read from istream at offset 0

And here as an example of exception:

<details>

    (lldb) bt
    * thread 244, name = 'MergeMutate', stop reason = breakpoint 6.1
      * frame 0: 0x000000001aa9c791 clickhouse`::__cxa_throw(thrown_object=0x00007fc3b68fdb00, tinfo=0x000000000548af50, dest=(clickhouse`Poco::Net::NetException::~NetException() at NetException.cpp:26))(void *)) at cxa_exception.cpp:258:33
        frame 1: 0x0000000017c55b2b clickhouse`Poco::Net::SocketImpl::error(code=-1232086272, arg=0x00007fc3be4f4d30) at SocketImpl.cpp:985:3
        frame 2: 0x0000000017c1f7ff clickhouse`Poco::Net::SecureSocketImpl::handleError(int) [inlined] Poco::Net::SocketImpl::error(code=<unavailable>) at SocketImpl.cpp:923:2
        frame 3: 0x0000000017c1f7e4 clickhouse`Poco::Net::SecureSocketImpl::handleError(this=0x00007fc35ae42b30, rc=-1) at SecureSocketImpl.cpp:531
        frame 4: 0x0000000017c208f9 clickhouse`Poco::Net::SecureSocketImpl::receiveBytes(this=0x00007fc35ae42b30, buffer=0x00007fc346e402c4, length=131068, flags=<unavailable>) at SecureSocketImpl.cpp:353:10
        frame 5: 0x0000000017c3b388 clickhouse`Poco::Net::HTTPSession::read(char*, long) [inlined] Poco::Net::StreamSocket::receiveBytes(buffer=<unavailable>, length=<unavailable>, flags=0) at StreamSocket.cpp:130:17
        frame 6: 0x0000000017c3b379 clickhouse`Poco::Net::HTTPSession::read(char*, long) [inlined] Poco::Net::HTTPSession::receive(this=<unavailable>, buffer=<unavailable>, length=<unavailable>) at HTTPSession.cpp:166
        frame 7: 0x0000000017c3b379 clickhouse`Poco::Net::HTTPSession::read(this=0x00007fc3973e9098, buffer=<unavailable>, length=<unavailable>) at HTTPSession.cpp:144
        frame 8: 0x0000000017c0f706 clickhouse`Poco::Net::HTTPSClientSession::read(this=<unavailable>, buffer=<unavailable>, length=<unavailable>) at HTTPSClientSession.cpp:195:23
        frame 9: 0x0000000017c32eb4 clickhouse`Poco::Net::HTTPFixedLengthStreamBuf::readFromDevice(this=0x00007fc361bd6708, buffer=<unavailable>, length=<unavailable>) at HTTPFixedLengthStream.cpp:53:16
        frame 10: 0x0000000017c2a368 clickhouse`Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::underflow(this=0x00007fc361bd6708) at BufferedStreamBuf.h:97:17
        frame 11: 0x0000000008b2abca clickhouse`std::__1::basic_streambuf<char, std::__1::char_traits<char> >::uflow(this=0x00007fc361bd6708) at streambuf:445:9
        frame 12: 0x0000000008b2ab5a clickhouse`std::__1::basic_streambuf<char, std::__1::char_traits<char> >::xsgetn(this=0x00007fc361bd6708, __s="P, __n=1048576) at streambuf:422:25
        frame 13: 0x0000000008b2ca0e clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char> >::read(char*, long) [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char> >::sgetn[abi:v15000](this=<unavailable>, __s="!P, __n=1048576) at streambuf:204:14
        frame 14: 0x0000000008b2ca02 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char> >::read(this=0x00007fc35ae42d40, __s="!P, __n=1048576) at istream:1052
        frame 15: 0x0000000010de0b9f clickhouse`DB::ReadBufferFromIStream::nextImpl(this=0x00007fc3663d2380) at ReadBufferFromIStream.cpp:15:10

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-22 23:29:28 +01:00
yariks5s
50724f1b5d Merge branch 'fix_s3_regions' of github.com:yariks5s/ClickHouse into fix_s3_regions 2023-12-22 17:28:22 +00:00
yariks5s
a543048b5f review fix 2023-12-22 17:28:09 +00:00
Yarik Briukhovetskyi
ab6cc0c9f0
Update Client.cpp 2023-12-22 14:59:21 +01:00
yariks5s
823abaa73e init 2023-12-22 13:27:03 +00:00
Yakov Olkhovskiy
5dc88542cb
Merge branch 'master' into feature-server-iface-metrics 2023-12-20 16:46:29 -05:00
Kruglov Pavel
ca7001d70f
Merge pull request #57850 from shanielh/feature/s3-session-token 
Feature/s3 session token
 2023-12-20 17:22:09 +01:00
Nikita Taranov
354b689800 impl 2023-12-20 16:07:00 +01:00
Kruglov Pavel
087946f385
Merge branch 'master' into variant-data-type 2023-12-20 00:40:18 +01:00
Julia Kartseva
2c32589949
Merge pull request #58019 from jkartseva/fix-build-errno-exception 
fix freebsd build
 2023-12-19 11:16:45 -08:00
Sema Checherinda
802ad4ec47
Merge pull request #57523 from ClickHouse/remove-the-limit-for-connections-per-endpoint 
Remove the limit for the number of connections per endpoint for background fetches + small refactoring
 2023-12-19 19:16:01 +01:00
avogar
319ae440b6 Implement Variant data type 2023-12-19 16:45:15 +00:00
vdimir
252cb8a507
Merge pull request #54602 from fionera/issue-49028 
Add setting http_make_head_request
 2023-12-19 13:33:59 +01:00
Julia Kartseva
31f04b66c5 fix freebsd build 
https://github.com/ClickHouse/ClickHouse/actions/runs/7256640256/job/19769624432?pr=58015#step:8:5466
```
Dec 19 03:09:41 /build/src/IO/AIO.cpp:140:15: error: use of undeclared identifier 'ErrnoException'; did you mean 'DB::ErrnoException'?
Dec 19 03:09:41   140 |         throw ErrnoException(DB::ErrorCodes::CANNOT_IOSETUP, "io_setup failed");
```
 2023-12-19 03:47:28 +00:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
Raúl Marín
65728f14ce Review error codes 2023-12-18 10:40:02 +01:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials 
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
 2023-12-17 10:29:15 +02:00
Yakov Olkhovskiy
42a770f209 optimize event update 2023-12-15 19:51:52 +00:00
Yakov Olkhovskiy
001a38048f use ProfileEvents instead of CurrentMetrics 2023-12-15 19:17:42 +00:00
Raúl Marín
b269f87f4c Better text_log with ErrnoException 2023-12-15 19:27:56 +01:00
Yakov Olkhovskiy
febdd542f8 Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-12-14 18:59:44 +00:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Kruglov Pavel
ee39dca8c7
Merge branch 'master' into better-parsing-exceptions 2023-12-12 17:10:38 +01:00
Kruglov Pavel
909b1fb36b
Merge pull request #57004 from Avogar/template-deadlock 
Fix possible deadlock in Template format during sync after error
 2023-12-11 19:02:40 +01:00
Igor Nikonov
87e47bdbee
Merge pull request #57467 from ClickHouse/rm-select 
Remove select() usage
 2023-12-09 22:27:28 +01:00
Alexey Milovidov
0ee13fde16 Fix "logs and exception messages formatting", part 3 2023-12-07 22:01:53 +01:00
Igor Nikonov
d196973ab3 Merge remote-tracking branch 'origin/master' into rm-select 2023-12-05 12:59:18 +00:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
kssenii
4a28f10c3d Minor cache changes 2023-12-04 19:02:37 +01:00
vdimir
a4ae90de0d
Merge pull request #57275 from ClickHouse/vdimir/merge_task_tmp_data 
Background merges correctly use temporary data storage in the cache
 2023-12-04 14:52:20 +01:00
Igor Nikonov
434c2113d2 Fix style 2023-12-04 10:51:17 +00:00
Igor Nikonov
73ff984755 Remove select() usage 2023-12-04 09:33:34 +00:00
robot-ch-test-poll
1b49463bd2
Merge pull request #55841 from nickitat/optimize_reading3 
Optimize reading from cache
 2023-12-01 17:36:57 +01:00
Yakov Olkhovskiy
416d28687e Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-11-30 23:00:04 +00:00
Nikolai Kochetov
823ba2db46
Merge pull request #57075 from yariks5s/s3_links_fix 
S3-style links bug fix
 2023-11-29 17:41:08 +01:00
vdimir
b5babe1692
MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
Nikita Taranov
52f644c0df Merge branch 'master' into optimize_reading3 2023-11-28 16:36:38 +01:00
Antonio Andelic
6e8e4a6ca5 Lower level for annoying log 2023-11-28 12:41:35 +00:00
Yakov Olkhovskiy
d8e41d8886 fix after merge 2023-11-28 03:28:32 +00:00
Yakov Olkhovskiy
c0a9b82c3c fix 2023-11-27 22:06:01 +00:00
Yakov Olkhovskiy
52ef3fe34b style fix 2023-11-27 21:32:50 +00:00
Yakov Olkhovskiy
4d40ed851f fix after merge 2023-11-27 21:05:35 +00:00
Yakov Olkhovskiy
b92030b485
Merge branch 'master' into feature-server-iface-metrics 2023-11-27 15:44:25 -05:00
Azat Khuzhin
4a02de4674 Add ability to disable checksums for S3 to avoid excessive input file read 
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
  PayloadSigningPolicy::Never)

So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.

By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.

And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.

Here is an example stacktrace of this excessive read:

<details>

<summary>stacktrace</summary>

    (lldb) bt
    * thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
      * frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
        frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
        frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
        frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
        frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
        frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
        frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
        frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
        frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
        frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
        frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
        frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
        frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
        frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
        frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
        frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
        frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
        frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()

</details>

This new behaviour could be enabled with `s3_disable_checksum=true`.

Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
 2023-11-26 19:20:19 +01:00
Alexey Milovidov
289df618f4
Merge pull request #57001 from arthurpassos/aws-s3-sign-any-x-amz-header-clean 
Sign all aws headers
 2023-11-24 21:12:10 +01:00
vdimir
1044e3366b
Update http_make_head_request doc 2023-11-24 09:32:20 +00:00
Tim Windelschmidt
51e3899f73
Add setting http_make_head_request 
Clickhouse always does a HEAD request before actually executing the GET request,
this adds a settings to skip that request.
 2023-11-24 09:32:16 +00:00
Yarik Briukhovetskyi
360d1b075a
Update URI.cpp 2023-11-22 13:43:33 +01:00
Yarik Briukhovetskyi
155cc84941
Update URI.cpp 2023-11-22 13:41:01 +01:00
Kseniia Sumarokova
e4f66b8469
Merge pull request #55158 from kssenii/fs-cache-improvement 
fs cache improvement for big reads
 2023-11-21 21:50:00 +01:00
yariks5s
be3b1f8188 initial fix 2023-11-21 18:49:16 +00:00
vdimir
a139ae97eb
Merge pull request #52918 from ClickHouse/vdimir/s3_blob_log 
Add system table with blob storage operations log
 2023-11-21 17:40:42 +01:00
avogar
7e392eec50 Better exception messages in input formats 2023-11-21 13:13:42 +00:00
Kseniia Sumarokova
d384762123
Merge branch 'master' into fs-cache-improvement 2023-11-21 11:24:52 +01:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
Sema Checherinda
485f1834d8
Merge pull request #56938 from CheSema/lz4-buffering 
Lz4 compression: buffer block in a rare case
 2023-11-20 20:33:30 +01:00
Arthur Passos
3544ee1e5f fix build by removing some const specifiers 2023-11-20 13:52:18 -03:00
avogar
68c72d7e65 Fix possible deadlock in Template format during sync after error 2023-11-20 16:47:52 +00:00
Arthur Passos
e5129990ed sign all aws headers 2023-11-20 13:38:32 -03:00
Sema Checherinda
f999337dae
Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
Alexander Tokmakov
5031f239c3
Revert "s3 adaptive timeouts" 2023-11-20 14:28:59 +01:00
Sema Checherinda
fafd169e7b
Update src/IO/Lz4DeflatingWriteBuffer.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-11-20 14:12:52 +01:00
Sema Checherinda
ebb66c1a9e add comments 2023-11-20 12:13:24 +01:00
Smita Kulkarni
961bf074da Initial draft version of adding backup support to AzureBlobStorage 2023-11-20 10:56:10 +01:00
Sema Checherinda
a950595c24
Merge pull request #56314 from CheSema/s3-aggressive-timeouts 
s3 adaptive timeouts
 2023-11-19 14:12:14 +01:00
Sema Checherinda
cacc23b8b7 safe SinkToOut d-tor 2023-11-19 12:25:42 +01:00
Sema Checherinda
24fbe620d3 fix build 2023-11-19 12:14:53 +01:00
Alexey Milovidov
edc3b2fe48
Merge pull request #56958 from ClickHouse/metric-queued-jobs 
Add metrics for the number of queued jobs, which is useful for the IO thread pool
 2023-11-19 10:37:18 +01:00
Sema Checherinda
053b20a255 fix in_data pointer 2023-11-19 00:44:39 +01:00
Alexey Milovidov
593f04a6b5 Fix style 2023-11-18 20:19:24 +01:00
Alexey Milovidov
d56cbda185 Add metrics for the number of queued jobs, which is useful for the IO thread pool 2023-11-18 19:07:59 +01:00
Sema Checherinda
773715a562 finalize tmp_out 2023-11-18 17:30:49 +01:00
Sema Checherinda
6d5a5f9fcd buffer result if out copacity is not enough 2023-11-17 17:31:00 +01:00
Jianfei Hu
ef79bf6467 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into keeper-az-fix 2023-11-16 21:58:02 +00:00
Kseniia Sumarokova
a2ed756eec
Merge branch 'master' into fs-cache-improvement 2023-11-16 17:49:26 +01:00
kssenii
472cfdc86d Review fix 2023-11-16 17:47:51 +01:00
Sema Checherinda
4a1e207e7a review notes 2023-11-16 12:31:00 +01:00
Jianfei Hu
69f214cdbc fix comments. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-16 08:04:57 +00:00
Jianfei Hu
ea92dbb1c7 fix build for non USE_S3 case 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-15 20:53:35 +00:00
Jianfei Hu
d0398e3c1d remove variant header 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-15 18:47:28 +00:00
Jianfei Hu
d862dfdf9c fix comments 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-15 18:38:23 +00:00
Sema Checherinda
6e3e6383ba perf check 2 2023-11-15 19:00:27 +01:00
avogar
38f200d969 Fix Date text parsing in optimistic path 
1
 2023-11-14 18:58:00 +00:00
Jianfei Hu
9df2775f08 reduce timeout and setTimeout earlier. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-14 17:58:16 +00:00
Sema Checherinda
3075bd9745 track clickhouse high level retries 2023-11-14 11:34:12 +01:00
Sema Checherinda
8d36fd6e54 get rid off of client_with_long_timeout_ptr 2023-11-14 11:34:12 +01:00
Sema Checherinda
27fb25d056 alter the naming, fix client_with_long_timeout in s3 storage 2023-11-14 11:34:12 +01:00
Sema Checherinda
be01a5cd3e turn off agressive timeouts for heavy requests 2023-11-14 11:34:12 +01:00
Sema Checherinda
770a762317 aggressive timeout 2023-11-14 11:34:11 +01:00
Jianfei Hu
554d907189 Fix the keeper_server availability zone configuration. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-13 23:42:51 +00:00
Jianfei Hu
d3517140d9 Revert "Revert "Add /keeper/availability-zone node to allow server load balancing within AZ."" 
This reverts commit e5aad4a6fd.
 2023-11-13 19:03:27 +00:00
Alexey Milovidov
c81cc3f953
Merge pull request #56660 from ClickHouse/better-ccache-2 
Better use of build cache
 2023-11-13 19:28:33 +01:00
Alexey Milovidov
82c461031e Fix build 2023-11-13 10:09:23 +01:00
Alexey Milovidov
58319f08be Merge branch 'master' into system-symbols 2023-11-11 04:12:37 +01:00
Alexey Milovidov
e5aad4a6fd
Revert "Add /keeper/availability-zone node to allow server load balancing within AZ." 2023-11-11 05:43:58 +03:00
Alexey Milovidov
cc39484f7f Merge branch 'master' into system-symbols 2023-11-09 23:05:55 +01:00
Sema Checherinda
7f34e0b24e recheck 404 response for complete multipart upload 2023-11-08 20:51:49 +01:00
Alexey Milovidov
68d3f60ef7 Merge branch 'master' into system-symbols 2023-11-08 17:07:11 +01:00
kssenii
dbea50738b Merge remote-tracking branch 'origin/master' into fs-cache-improvement 2023-11-08 11:53:54 +01:00
Antonio Andelic
788c66027c
Merge pull request #56104 from incfly/keeper-az 
Add /keeper/availability-zone node to allow server load balancing within AZ.
 2023-11-08 09:46:24 +01:00
Nikita Taranov
c52571223e better 2023-11-07 22:04:13 +01:00
Alexey Milovidov
ca83da14f2 Merge branch 'master' of github.com:ClickHouse/ClickHouse into system-symbols 2023-11-07 19:43:19 +01:00
Sema Checherinda
02df5ac59c
Merge pull request #54340 from KevinyhZou/bug_fix_csv_parse_crlf 
Fix CSV read while the end of line is CR(\r)
 2023-11-07 12:51:05 +01:00
Jianfei Hu
033bb19198 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into keeper-az 2023-11-07 02:50:54 +00:00
kevinyhzou
d1743e08f7 ci fix 2023-11-07 09:45:46 +08:00
Alexander Tokmakov
fbdfa9a44d
Merge pull request #56316 from ClickHouse/change_some_exception_codes 
Change some exception codes
 2023-11-06 19:17:02 +01:00
Raúl Marín
653a51a862
Merge pull request #56286 from bigo-sg/ch_56285 
Fix inconsistency of "cast('0' as DateTime64(3))" and "cast('0' as Nullable(DateTime64(3)))"
 2023-11-06 10:30:51 +01:00
Alexey Milovidov
184e6f840e Remove useless header files 2023-11-06 03:49:55 +01:00
Arthur Passos
b6e205dcdf
Add ClickHouse setting to disable tunneling for HTTPS requests over HTTP proxy (#55033) 
* initial commit. integ tests passing, need to re-run unit & my own personal tests

* partial refactoring to remove Protocol::ANY

* improve naming

* remove all usages of ProxyConfiguration::Protocol::ANY

* fix ut

* blabla

* support url functions as well

* support for HTTPS requests over HTTP proxy with tunneling off

* remove gtestabc

* fix silly mistake

* ...

* remove usages of httpclientsession::proxyconfig in src/

* got you

* remove stale comment

* it seems like I need reasonable defaults

* fix ut

* add some comments

* remove no longer needed header

* matrix out

* add https over http proxy with no tunneling

* soem docs

* partial refactoring

* rename to use_tunneling_for_https_requests_over_http_proxy

* improve docs

* use shorter version

* remove useless test

* rename the setting

* update

* fix typo

* fix setting docs typo

* move ); up

* move ) up
 2023-11-04 13:47:52 -04:00
Yakov Olkhovskiy
d9fe33ae7b fix clang tidy 2023-11-04 03:06:45 +00:00
Nikita Taranov
7046832056 change signatures 2023-11-03 22:50:03 +01:00
Alexander Tokmakov
289fcccbfc change some exception codes 2023-11-03 20:19:24 +01:00
Jianfei Hu
b426cfe462 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into keeper-az 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-03 16:42:29 +00:00
Jianfei Hu
bdf49a69f4 throw exception 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-03 16:35:06 +00:00
taiyang-li
3c70b9f65d add uts 2023-11-03 15:32:37 +08:00
taiyang-li
93be383117 fix issue https://github.com/ClickHouse/ClickHouse/issues/56285 2023-11-03 12:41:00 +08:00
Jianfei Hu
136efba321 integration test working and std::exception_ptr. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-03 02:24:42 +00:00
Jianfei Hu
e29b7640c4 gcp metadata use dns. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-01 17:53:34 +00:00
Jianfei Hu
b22773ccd1
Apply suggestions from code review 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-11-01 10:30:29 -07:00
Kruglov Pavel
08971270aa
Merge pull request #56135 from Avogar/fix-json-exception-in-http 
Fix wrong query result when http_write_exception_in_output_format=1
 2023-11-01 14:36:28 +01:00
Kseniia Sumarokova
e5efc42633
Merge branch 'master' into fs-cache-improvement 2023-11-01 10:47:36 +01:00
Jianfei Hu
6f94770a8e more fix -DUSE_AWS_S3=0 case. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-01 00:51:45 +00:00
Nikita Taranov
1d58cedce5 better 2023-10-31 22:48:20 +01:00
Jianfei Hu
045d67ffc1 more style fix. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>

space before catch.

Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 21:41:39 +00:00
Jianfei Hu
3fff813659 style fix and revert src/Coordination/KeeperSnapshotManagerS3.h 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 18:48:59 +00:00
Jianfei Hu
f902243c05 comments use static. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 17:43:40 +00:00
Jianfei Hu
cfa1b96d23 remove comments. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 17:43:24 +00:00
Jianfei Hu
e231d063fb fix the exception handlig. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 05:16:14 +00:00
Jianfei Hu
573054d755 add new code and rework. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 04:43:44 +00:00
Jianfei Hu
6d72873bdb use poco better works. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 02:34:50 +00:00
Yakov Olkhovskiy
9f4be7e847 fix 2023-10-30 23:31:15 +00:00
Kruglov Pavel
4effc676f9
Merge pull request #56046 from Avogar/cr-in-unquoted-csv-string 
Allow unquoted strings with CR in CSV format
 2023-10-30 20:46:20 +01:00
avogar
c7857646bc Fix wrong query result when http_write_exception_in_output_format=1 2023-10-30 18:09:29 +00:00
Yakov Olkhovskiy
e1d0994c0a fix 2023-10-30 05:37:47 +00:00
Yakov Olkhovskiy
d72069dea4 compressors to not produce output for no data, HTTP buffer not to add Content-Encoding if no data 2023-10-29 18:21:21 +00:00
Jianfei Hu
dec8efc39e make az as attribute of keepersnapshotmangers3. 
empty config not working.
https://pastila.nl/?00048e4f/d56af90d13f2d871e30dfb1db9660266#yff6d8HVQm0blQUUiZg8SQ==

Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-29 02:04:38 +00:00
Yakov Olkhovskiy
7887e1e14e fix 2023-10-28 23:26:38 +00:00
Jianfei Hu
0b301f73c1 initial add keeper az info in /keeper/availabilty-zone 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-28 21:54:59 +00:00
Nikita Taranov
213ce63ffb
Merge branch 'master' into optimize_reading3 2023-10-28 00:04:20 +02:00
Kruglov Pavel
f1f1846572
Merge pull request #55861 from bigo-sg/ch_55858 
Throw exception when parsing illegal string as float if precise_float_parsing is true
 2023-10-27 14:24:55 +02:00
李扬
308b2942d7
Update readFloatText.h 2023-10-27 11:42:33 +08:00
Yakov Olkhovskiy
0cf851316c use CH Buffer for HTTP out stream, add metrics for interfaces 2023-10-27 02:38:36 +00:00
avogar
9d207bf027 Allow unquoted strings with CR in CSV format 2023-10-26 13:50:54 +00:00
taiyang-li
d5feaa72c1 improve thrown exception message 2023-10-25 20:16:18 +08:00
Nikita Taranov
a86bda5208 use local thread pool reader for cache segments 2023-10-24 14:06:13 +02:00
taiyang-li
2a894335b7 commit again 2023-10-24 16:16:05 +08:00
zvonand
7d54c1fb00 check EOF before reading buffer 2023-10-20 22:27:02 +02:00
zvonand
e5bfc7d3fc Improved DateTime64 parsing 
Allow to parse unambigous short DateTime64
Throw error on invalid DT64 representation (as for DateTime)
 2023-10-20 22:27:02 +02:00
Nikita Taranov
9b13c796e3 set ignore metric in all readers 2023-10-20 22:24:56 +02:00
Nikita Taranov
2a6a97c1fa read sync from cache 2023-10-20 22:24:56 +02:00