thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-13 01:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
136,808 Commits 9,322 Branches 1,783 Tags 2.1 GiB
bfc4f5e1d3
Commit Graph

3471 Commits

Author SHA1 Message Date
Smita Kulkarni
c14605caa7 Added flag use_native_copy and updated to use StartCopyFromUri for native copy with large files 2024-01-04 18:27:54 +01:00
Smita Kulkarni
91bad5bc39 Updated to use MultiVersion for BlobContainerClient in Backups and updated to get client from disk 2024-01-04 16:06:36 +01:00
Duc Canh Le
2e14cfb526 add settings for output compression level and window size 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-04 08:16:00 +00:00
Yakov Olkhovskiy
85f03478ef
Revert "Revert "Use CH Buffer for HTTP out stream, add metrics for interfaces"" 2024-01-03 11:47:15 -05:00
Kruglov Pavel
4d8cf71ba7
Merge branch 'master' into variant-data-type 2024-01-03 15:21:23 +01:00
Kruglov Pavel
570d1c013b
Merge pull request #57053 from Avogar/better-parsing-exceptions 
Better exception messages in input formats
 2024-01-03 14:47:51 +01:00
Smita Kulkarni
df221f7db6 Renamed Bucket-Key to Container-Blob 2024-01-03 11:35:06 +01:00
Raúl Marín
d491758939
Revert "Use CH Buffer for HTTP out stream, add metrics for interfaces" 2024-01-03 10:42:15 +01:00
Smita Kulkarni
4122de9721 Updated tests and added settings 2024-01-02 20:19:01 +01:00
Yakov Olkhovskiy
5633fb8145
Merge pull request #56064 from ClickHouse/feature-server-iface-metrics 
Use CH Buffer for HTTP out stream, add metrics for interfaces
 2024-01-02 10:17:52 -05:00
Kruglov Pavel
e69bda7f08
Merge pull request #57438 from Avogar/fix-resetting-buffers 
Fix working with read buffers in StreamingFormatExecutor
 2024-01-02 15:34:35 +01:00
Alexey Milovidov
2f3e17c737 Remove commented code 2023-12-30 18:39:12 +01:00
Alexey Milovidov
a1bd76388e Remove commented code 2023-12-30 18:38:41 +01:00
Alexey Milovidov
aad63c8273 Merge branch 'master' into feature-server-iface-metrics 2023-12-30 15:44:42 +01:00
Alexey Milovidov
8365722c95
Merge pull request #58336 from amosbird/macro2constexpr 
Organize symbols from src/* into DB namespace
 2023-12-30 12:40:36 +01:00
Amos Bird
907e844635
Fix examples 2023-12-30 10:48:19 +08:00
Alexey Milovidov
aa462fa7cd
Merge branch 'master' into feature-server-iface-metrics 2023-12-29 22:37:52 +01:00
Amos Bird
18395a387c
Fix another unitest 2023-12-30 01:14:21 +08:00
Kruglov Pavel
c03e36e012
Merge branch 'master' into better-parsing-exceptions 2023-12-29 18:07:32 +01:00
Azat Khuzhin
8c54380d80 Avoid sending ComposeObject requests after upload to GCS 
This should not be required anymore, but leave it as an option, since
likely this is required for old files.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
f4a7789cd4 Convert various S3::Client settings into separate ClientSettings struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
2b2d10689b Make S3::Client::ComposeObject private 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
avogar
c6085b3eb9 Fix working with read buffers in StreamingFormatExecutor 2023-12-28 18:47:57 +00:00
Smita Kulkarni
bf60a06a11 Merge branch 'master' into Azure_backup 2023-12-28 11:51:49 +01:00
Azat Khuzhin
c6774cfe42 Add missing includes for ZipArchiveWriter 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 17:06:17 +01:00
Azat Khuzhin
cf1dae5321 Add missing includes into BitHelpers.h 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 17:06:17 +01:00
Azat Khuzhin
b9233f6d4f Move Allocator code into module part 
This should reduce amount of code that should be recompiled on
Exception.h changes (and everything else that had been included there).

This will actually not help a lot, because it is also included into
PODArray.h and ThreadPool.h at least... Sigh.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 15:42:08 +01:00
Smita Kulkarni
d46d914521 Updated thread name 2023-12-27 10:28:52 +01:00
Yarik Briukhovetskyi
41c275274b
Merge pull request #58148 from yariks5s/fix_s3_regions 
S3-links region independency
 2023-12-26 12:59:00 +01:00
Alexey Milovidov
817d0c6553
Merge pull request #58202 from vitlibar/explicit-finalize-in-zip-archive-writer 
Explicit finalize in ZipArchiveWriter
 2023-12-24 03:30:21 +01:00
Vitaly Baranov
e1a136b791 Explicit finalize() function in ZipArchiveWriter. 
Simplify too complicated code in ZipArchiveWriter.
 2023-12-24 00:33:59 +01:00
Alexey Milovidov
a400657155
Merge pull request #58165 from azat/s3-passthrough-exception 
Pass through exceptions for reading from S3
 2023-12-23 11:35:23 +01:00
Azat Khuzhin
896cbb6a75 Retry on poco exceptions while reading from S3 as well 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-23 09:07:43 +01:00
Alexey Milovidov
ffca9bc564 Fix #58171 2023-12-23 02:04:35 +01:00
Azat Khuzhin
17e21aa21e Pass throught exceptions for reading from S3 
Reading from S3 has retries on almost any exception, except for
CANNOT_ALLOCATE_MEMORY, but the problem is that this exception may not
be even delivered to ReadBufferFromS3::processException(), due to
std::istream will hide it (see [1]).

  [1]: 1834e42289/libcxx/include/istream (L1059)

So let's pass throught them to see at least some information instead of
just:

    Message: Cannot read from istream at offset 0

And here as an example of exception:

<details>

    (lldb) bt
    * thread 244, name = 'MergeMutate', stop reason = breakpoint 6.1
      * frame 0: 0x000000001aa9c791 clickhouse`::__cxa_throw(thrown_object=0x00007fc3b68fdb00, tinfo=0x000000000548af50, dest=(clickhouse`Poco::Net::NetException::~NetException() at NetException.cpp:26))(void *)) at cxa_exception.cpp:258:33
        frame 1: 0x0000000017c55b2b clickhouse`Poco::Net::SocketImpl::error(code=-1232086272, arg=0x00007fc3be4f4d30) at SocketImpl.cpp:985:3
        frame 2: 0x0000000017c1f7ff clickhouse`Poco::Net::SecureSocketImpl::handleError(int) [inlined] Poco::Net::SocketImpl::error(code=<unavailable>) at SocketImpl.cpp:923:2
        frame 3: 0x0000000017c1f7e4 clickhouse`Poco::Net::SecureSocketImpl::handleError(this=0x00007fc35ae42b30, rc=-1) at SecureSocketImpl.cpp:531
        frame 4: 0x0000000017c208f9 clickhouse`Poco::Net::SecureSocketImpl::receiveBytes(this=0x00007fc35ae42b30, buffer=0x00007fc346e402c4, length=131068, flags=<unavailable>) at SecureSocketImpl.cpp:353:10
        frame 5: 0x0000000017c3b388 clickhouse`Poco::Net::HTTPSession::read(char*, long) [inlined] Poco::Net::StreamSocket::receiveBytes(buffer=<unavailable>, length=<unavailable>, flags=0) at StreamSocket.cpp:130:17
        frame 6: 0x0000000017c3b379 clickhouse`Poco::Net::HTTPSession::read(char*, long) [inlined] Poco::Net::HTTPSession::receive(this=<unavailable>, buffer=<unavailable>, length=<unavailable>) at HTTPSession.cpp:166
        frame 7: 0x0000000017c3b379 clickhouse`Poco::Net::HTTPSession::read(this=0x00007fc3973e9098, buffer=<unavailable>, length=<unavailable>) at HTTPSession.cpp:144
        frame 8: 0x0000000017c0f706 clickhouse`Poco::Net::HTTPSClientSession::read(this=<unavailable>, buffer=<unavailable>, length=<unavailable>) at HTTPSClientSession.cpp:195:23
        frame 9: 0x0000000017c32eb4 clickhouse`Poco::Net::HTTPFixedLengthStreamBuf::readFromDevice(this=0x00007fc361bd6708, buffer=<unavailable>, length=<unavailable>) at HTTPFixedLengthStream.cpp:53:16
        frame 10: 0x0000000017c2a368 clickhouse`Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::underflow(this=0x00007fc361bd6708) at BufferedStreamBuf.h:97:17
        frame 11: 0x0000000008b2abca clickhouse`std::__1::basic_streambuf<char, std::__1::char_traits<char> >::uflow(this=0x00007fc361bd6708) at streambuf:445:9
        frame 12: 0x0000000008b2ab5a clickhouse`std::__1::basic_streambuf<char, std::__1::char_traits<char> >::xsgetn(this=0x00007fc361bd6708, __s="P, __n=1048576) at streambuf:422:25
        frame 13: 0x0000000008b2ca0e clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char> >::read(char*, long) [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char> >::sgetn[abi:v15000](this=<unavailable>, __s="!P, __n=1048576) at streambuf:204:14
        frame 14: 0x0000000008b2ca02 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char> >::read(this=0x00007fc35ae42d40, __s="!P, __n=1048576) at istream:1052
        frame 15: 0x0000000010de0b9f clickhouse`DB::ReadBufferFromIStream::nextImpl(this=0x00007fc3663d2380) at ReadBufferFromIStream.cpp:15:10

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-22 23:29:28 +01:00
yariks5s
50724f1b5d Merge branch 'fix_s3_regions' of github.com:yariks5s/ClickHouse into fix_s3_regions 2023-12-22 17:28:22 +00:00
yariks5s
a543048b5f review fix 2023-12-22 17:28:09 +00:00
Yarik Briukhovetskyi
ab6cc0c9f0
Update Client.cpp 2023-12-22 14:59:21 +01:00
yariks5s
823abaa73e init 2023-12-22 13:27:03 +00:00
Yakov Olkhovskiy
5dc88542cb
Merge branch 'master' into feature-server-iface-metrics 2023-12-20 16:46:29 -05:00
Kruglov Pavel
ca7001d70f
Merge pull request #57850 from shanielh/feature/s3-session-token 
Feature/s3 session token
 2023-12-20 17:22:09 +01:00
Nikita Taranov
354b689800 impl 2023-12-20 16:07:00 +01:00
Kruglov Pavel
087946f385
Merge branch 'master' into variant-data-type 2023-12-20 00:40:18 +01:00
Julia Kartseva
2c32589949
Merge pull request #58019 from jkartseva/fix-build-errno-exception 
fix freebsd build
 2023-12-19 11:16:45 -08:00
Sema Checherinda
802ad4ec47
Merge pull request #57523 from ClickHouse/remove-the-limit-for-connections-per-endpoint 
Remove the limit for the number of connections per endpoint for background fetches + small refactoring
 2023-12-19 19:16:01 +01:00
avogar
319ae440b6 Implement Variant data type 2023-12-19 16:45:15 +00:00
vdimir
252cb8a507
Merge pull request #54602 from fionera/issue-49028 
Add setting http_make_head_request
 2023-12-19 13:33:59 +01:00
Julia Kartseva
31f04b66c5 fix freebsd build 
https://github.com/ClickHouse/ClickHouse/actions/runs/7256640256/job/19769624432?pr=58015#step:8:5466
```
Dec 19 03:09:41 /build/src/IO/AIO.cpp:140:15: error: use of undeclared identifier 'ErrnoException'; did you mean 'DB::ErrnoException'?
Dec 19 03:09:41   140 |         throw ErrnoException(DB::ErrorCodes::CANNOT_IOSETUP, "io_setup failed");
```
 2023-12-19 03:47:28 +00:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
Raúl Marín
65728f14ce Review error codes 2023-12-18 10:40:02 +01:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials 
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
 2023-12-17 10:29:15 +02:00
Yakov Olkhovskiy
42a770f209 optimize event update 2023-12-15 19:51:52 +00:00
Yakov Olkhovskiy
001a38048f use ProfileEvents instead of CurrentMetrics 2023-12-15 19:17:42 +00:00
Raúl Marín
b269f87f4c Better text_log with ErrnoException 2023-12-15 19:27:56 +01:00
Yakov Olkhovskiy
febdd542f8 Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-12-14 18:59:44 +00:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Kruglov Pavel
ee39dca8c7
Merge branch 'master' into better-parsing-exceptions 2023-12-12 17:10:38 +01:00
Kruglov Pavel
909b1fb36b
Merge pull request #57004 from Avogar/template-deadlock 
Fix possible deadlock in Template format during sync after error
 2023-12-11 19:02:40 +01:00
Igor Nikonov
87e47bdbee
Merge pull request #57467 from ClickHouse/rm-select 
Remove select() usage
 2023-12-09 22:27:28 +01:00
Alexey Milovidov
0ee13fde16 Fix "logs and exception messages formatting", part 3 2023-12-07 22:01:53 +01:00
Igor Nikonov
d196973ab3 Merge remote-tracking branch 'origin/master' into rm-select 2023-12-05 12:59:18 +00:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
kssenii
4a28f10c3d Minor cache changes 2023-12-04 19:02:37 +01:00
vdimir
a4ae90de0d
Merge pull request #57275 from ClickHouse/vdimir/merge_task_tmp_data 
Background merges correctly use temporary data storage in the cache
 2023-12-04 14:52:20 +01:00
Igor Nikonov
434c2113d2 Fix style 2023-12-04 10:51:17 +00:00
Igor Nikonov
73ff984755 Remove select() usage 2023-12-04 09:33:34 +00:00
robot-ch-test-poll
1b49463bd2
Merge pull request #55841 from nickitat/optimize_reading3 
Optimize reading from cache
 2023-12-01 17:36:57 +01:00
Yakov Olkhovskiy
416d28687e Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-11-30 23:00:04 +00:00
Nikolai Kochetov
823ba2db46
Merge pull request #57075 from yariks5s/s3_links_fix 
S3-style links bug fix
 2023-11-29 17:41:08 +01:00
vdimir
b5babe1692
MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
Nikita Taranov
52f644c0df Merge branch 'master' into optimize_reading3 2023-11-28 16:36:38 +01:00
Antonio Andelic
6e8e4a6ca5 Lower level for annoying log 2023-11-28 12:41:35 +00:00
Yakov Olkhovskiy
d8e41d8886 fix after merge 2023-11-28 03:28:32 +00:00
Yakov Olkhovskiy
c0a9b82c3c fix 2023-11-27 22:06:01 +00:00
Yakov Olkhovskiy
52ef3fe34b style fix 2023-11-27 21:32:50 +00:00
Yakov Olkhovskiy
4d40ed851f fix after merge 2023-11-27 21:05:35 +00:00
Yakov Olkhovskiy
b92030b485
Merge branch 'master' into feature-server-iface-metrics 2023-11-27 15:44:25 -05:00
Azat Khuzhin
4a02de4674 Add ability to disable checksums for S3 to avoid excessive input file read 
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
  PayloadSigningPolicy::Never)

So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.

By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.

And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.

Here is an example stacktrace of this excessive read:

<details>

<summary>stacktrace</summary>

    (lldb) bt
    * thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
      * frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
        frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
        frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
        frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
        frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
        frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
        frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
        frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
        frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
        frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
        frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
        frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
        frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
        frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
        frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
        frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
        frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
        frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()

</details>

This new behaviour could be enabled with `s3_disable_checksum=true`.

Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
 2023-11-26 19:20:19 +01:00
Alexey Milovidov
289df618f4
Merge pull request #57001 from arthurpassos/aws-s3-sign-any-x-amz-header-clean 
Sign all aws headers
 2023-11-24 21:12:10 +01:00
vdimir
1044e3366b
Update http_make_head_request doc 2023-11-24 09:32:20 +00:00
Tim Windelschmidt
51e3899f73
Add setting http_make_head_request 
Clickhouse always does a HEAD request before actually executing the GET request,
this adds a settings to skip that request.
 2023-11-24 09:32:16 +00:00
Yarik Briukhovetskyi
360d1b075a
Update URI.cpp 2023-11-22 13:43:33 +01:00
Yarik Briukhovetskyi
155cc84941
Update URI.cpp 2023-11-22 13:41:01 +01:00
Kseniia Sumarokova
e4f66b8469
Merge pull request #55158 from kssenii/fs-cache-improvement 
fs cache improvement for big reads
 2023-11-21 21:50:00 +01:00
yariks5s
be3b1f8188 initial fix 2023-11-21 18:49:16 +00:00
vdimir
a139ae97eb
Merge pull request #52918 from ClickHouse/vdimir/s3_blob_log 
Add system table with blob storage operations log
 2023-11-21 17:40:42 +01:00
avogar
7e392eec50 Better exception messages in input formats 2023-11-21 13:13:42 +00:00
Kseniia Sumarokova
d384762123
Merge branch 'master' into fs-cache-improvement 2023-11-21 11:24:52 +01:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
Sema Checherinda
485f1834d8
Merge pull request #56938 from CheSema/lz4-buffering 
Lz4 compression: buffer block in a rare case
 2023-11-20 20:33:30 +01:00
Arthur Passos
3544ee1e5f fix build by removing some const specifiers 2023-11-20 13:52:18 -03:00
avogar
68c72d7e65 Fix possible deadlock in Template format during sync after error 2023-11-20 16:47:52 +00:00
Arthur Passos
e5129990ed sign all aws headers 2023-11-20 13:38:32 -03:00
Sema Checherinda
f999337dae
Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
Alexander Tokmakov
5031f239c3
Revert "s3 adaptive timeouts" 2023-11-20 14:28:59 +01:00
Sema Checherinda
fafd169e7b
Update src/IO/Lz4DeflatingWriteBuffer.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-11-20 14:12:52 +01:00
Sema Checherinda
ebb66c1a9e add comments 2023-11-20 12:13:24 +01:00
Smita Kulkarni
961bf074da Initial draft version of adding backup support to AzureBlobStorage 2023-11-20 10:56:10 +01:00
Sema Checherinda
a950595c24
Merge pull request #56314 from CheSema/s3-aggressive-timeouts 
s3 adaptive timeouts
 2023-11-19 14:12:14 +01:00
Sema Checherinda
cacc23b8b7 safe SinkToOut d-tor 2023-11-19 12:25:42 +01:00
Sema Checherinda
24fbe620d3 fix build 2023-11-19 12:14:53 +01:00
Alexey Milovidov
edc3b2fe48
Merge pull request #56958 from ClickHouse/metric-queued-jobs 
Add metrics for the number of queued jobs, which is useful for the IO thread pool
 2023-11-19 10:37:18 +01:00
Sema Checherinda
053b20a255 fix in_data pointer 2023-11-19 00:44:39 +01:00
Alexey Milovidov
593f04a6b5 Fix style 2023-11-18 20:19:24 +01:00
Alexey Milovidov
d56cbda185 Add metrics for the number of queued jobs, which is useful for the IO thread pool 2023-11-18 19:07:59 +01:00
Sema Checherinda
773715a562 finalize tmp_out 2023-11-18 17:30:49 +01:00
Sema Checherinda
6d5a5f9fcd buffer result if out copacity is not enough 2023-11-17 17:31:00 +01:00
Jianfei Hu
ef79bf6467 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into keeper-az-fix 2023-11-16 21:58:02 +00:00
Kseniia Sumarokova
a2ed756eec
Merge branch 'master' into fs-cache-improvement 2023-11-16 17:49:26 +01:00
kssenii
472cfdc86d Review fix 2023-11-16 17:47:51 +01:00
Sema Checherinda
4a1e207e7a review notes 2023-11-16 12:31:00 +01:00
Jianfei Hu
69f214cdbc fix comments. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-16 08:04:57 +00:00
Jianfei Hu
ea92dbb1c7 fix build for non USE_S3 case 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-15 20:53:35 +00:00
Jianfei Hu
d0398e3c1d remove variant header 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-15 18:47:28 +00:00
Jianfei Hu
d862dfdf9c fix comments 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-15 18:38:23 +00:00
Sema Checherinda
6e3e6383ba perf check 2 2023-11-15 19:00:27 +01:00
avogar
38f200d969 Fix Date text parsing in optimistic path 
1
 2023-11-14 18:58:00 +00:00
Jianfei Hu
9df2775f08 reduce timeout and setTimeout earlier. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-14 17:58:16 +00:00
Sema Checherinda
3075bd9745 track clickhouse high level retries 2023-11-14 11:34:12 +01:00
Sema Checherinda
8d36fd6e54 get rid off of client_with_long_timeout_ptr 2023-11-14 11:34:12 +01:00
Sema Checherinda
27fb25d056 alter the naming, fix client_with_long_timeout in s3 storage 2023-11-14 11:34:12 +01:00
Sema Checherinda
be01a5cd3e turn off agressive timeouts for heavy requests 2023-11-14 11:34:12 +01:00
Sema Checherinda
770a762317 aggressive timeout 2023-11-14 11:34:11 +01:00
Jianfei Hu
554d907189 Fix the keeper_server availability zone configuration. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-13 23:42:51 +00:00
Jianfei Hu
d3517140d9 Revert "Revert "Add /keeper/availability-zone node to allow server load balancing within AZ."" 
This reverts commit e5aad4a6fd.
 2023-11-13 19:03:27 +00:00
Alexey Milovidov
c81cc3f953
Merge pull request #56660 from ClickHouse/better-ccache-2 
Better use of build cache
 2023-11-13 19:28:33 +01:00
Alexey Milovidov
82c461031e Fix build 2023-11-13 10:09:23 +01:00
Alexey Milovidov
58319f08be Merge branch 'master' into system-symbols 2023-11-11 04:12:37 +01:00
Alexey Milovidov
e5aad4a6fd
Revert "Add /keeper/availability-zone node to allow server load balancing within AZ." 2023-11-11 05:43:58 +03:00
Alexey Milovidov
cc39484f7f Merge branch 'master' into system-symbols 2023-11-09 23:05:55 +01:00
Sema Checherinda
7f34e0b24e recheck 404 response for complete multipart upload 2023-11-08 20:51:49 +01:00
Alexey Milovidov
68d3f60ef7 Merge branch 'master' into system-symbols 2023-11-08 17:07:11 +01:00
kssenii
dbea50738b Merge remote-tracking branch 'origin/master' into fs-cache-improvement 2023-11-08 11:53:54 +01:00
Antonio Andelic
788c66027c
Merge pull request #56104 from incfly/keeper-az 
Add /keeper/availability-zone node to allow server load balancing within AZ.
 2023-11-08 09:46:24 +01:00
Nikita Taranov
c52571223e better 2023-11-07 22:04:13 +01:00
Alexey Milovidov
ca83da14f2 Merge branch 'master' of github.com:ClickHouse/ClickHouse into system-symbols 2023-11-07 19:43:19 +01:00
Sema Checherinda
02df5ac59c
Merge pull request #54340 from KevinyhZou/bug_fix_csv_parse_crlf 
Fix CSV read while the end of line is CR(\r)
 2023-11-07 12:51:05 +01:00
Jianfei Hu
033bb19198 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into keeper-az 2023-11-07 02:50:54 +00:00
kevinyhzou
d1743e08f7 ci fix 2023-11-07 09:45:46 +08:00
Alexander Tokmakov
fbdfa9a44d
Merge pull request #56316 from ClickHouse/change_some_exception_codes 
Change some exception codes
 2023-11-06 19:17:02 +01:00
Raúl Marín
653a51a862
Merge pull request #56286 from bigo-sg/ch_56285 
Fix inconsistency of "cast('0' as DateTime64(3))" and "cast('0' as Nullable(DateTime64(3)))"
 2023-11-06 10:30:51 +01:00
Alexey Milovidov
184e6f840e Remove useless header files 2023-11-06 03:49:55 +01:00
Arthur Passos
b6e205dcdf
Add ClickHouse setting to disable tunneling for HTTPS requests over HTTP proxy (#55033) 
* initial commit. integ tests passing, need to re-run unit & my own personal tests

* partial refactoring to remove Protocol::ANY

* improve naming

* remove all usages of ProxyConfiguration::Protocol::ANY

* fix ut

* blabla

* support url functions as well

* support for HTTPS requests over HTTP proxy with tunneling off

* remove gtestabc

* fix silly mistake

* ...

* remove usages of httpclientsession::proxyconfig in src/

* got you

* remove stale comment

* it seems like I need reasonable defaults

* fix ut

* add some comments

* remove no longer needed header

* matrix out

* add https over http proxy with no tunneling

* soem docs

* partial refactoring

* rename to use_tunneling_for_https_requests_over_http_proxy

* improve docs

* use shorter version

* remove useless test

* rename the setting

* update

* fix typo

* fix setting docs typo

* move ); up

* move ) up
 2023-11-04 13:47:52 -04:00
Yakov Olkhovskiy
d9fe33ae7b fix clang tidy 2023-11-04 03:06:45 +00:00
Nikita Taranov
7046832056 change signatures 2023-11-03 22:50:03 +01:00
Alexander Tokmakov
289fcccbfc change some exception codes 2023-11-03 20:19:24 +01:00
Jianfei Hu
b426cfe462 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into keeper-az 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-03 16:42:29 +00:00
Jianfei Hu
bdf49a69f4 throw exception 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-03 16:35:06 +00:00
taiyang-li
3c70b9f65d add uts 2023-11-03 15:32:37 +08:00
taiyang-li
93be383117 fix issue https://github.com/ClickHouse/ClickHouse/issues/56285 2023-11-03 12:41:00 +08:00
Jianfei Hu
136efba321 integration test working and std::exception_ptr. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-03 02:24:42 +00:00
Jianfei Hu
e29b7640c4 gcp metadata use dns. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-01 17:53:34 +00:00
Jianfei Hu
b22773ccd1
Apply suggestions from code review 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-11-01 10:30:29 -07:00
Kruglov Pavel
08971270aa
Merge pull request #56135 from Avogar/fix-json-exception-in-http 
Fix wrong query result when http_write_exception_in_output_format=1
 2023-11-01 14:36:28 +01:00
Kseniia Sumarokova
e5efc42633
Merge branch 'master' into fs-cache-improvement 2023-11-01 10:47:36 +01:00
Jianfei Hu
6f94770a8e more fix -DUSE_AWS_S3=0 case. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-11-01 00:51:45 +00:00
Nikita Taranov
1d58cedce5 better 2023-10-31 22:48:20 +01:00
Jianfei Hu
045d67ffc1 more style fix. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>

space before catch.

Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 21:41:39 +00:00
Jianfei Hu
3fff813659 style fix and revert src/Coordination/KeeperSnapshotManagerS3.h 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 18:48:59 +00:00
Jianfei Hu
f902243c05 comments use static. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 17:43:40 +00:00
Jianfei Hu
cfa1b96d23 remove comments. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 17:43:24 +00:00
Jianfei Hu
e231d063fb fix the exception handlig. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 05:16:14 +00:00
Jianfei Hu
573054d755 add new code and rework. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 04:43:44 +00:00
Jianfei Hu
6d72873bdb use poco better works. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-31 02:34:50 +00:00
Yakov Olkhovskiy
9f4be7e847 fix 2023-10-30 23:31:15 +00:00
Kruglov Pavel
4effc676f9
Merge pull request #56046 from Avogar/cr-in-unquoted-csv-string 
Allow unquoted strings with CR in CSV format
 2023-10-30 20:46:20 +01:00
avogar
c7857646bc Fix wrong query result when http_write_exception_in_output_format=1 2023-10-30 18:09:29 +00:00
Yakov Olkhovskiy
e1d0994c0a fix 2023-10-30 05:37:47 +00:00
Yakov Olkhovskiy
d72069dea4 compressors to not produce output for no data, HTTP buffer not to add Content-Encoding if no data 2023-10-29 18:21:21 +00:00
Jianfei Hu
dec8efc39e make az as attribute of keepersnapshotmangers3. 
empty config not working.
https://pastila.nl/?00048e4f/d56af90d13f2d871e30dfb1db9660266#yff6d8HVQm0blQUUiZg8SQ==

Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-29 02:04:38 +00:00
Yakov Olkhovskiy
7887e1e14e fix 2023-10-28 23:26:38 +00:00
Jianfei Hu
0b301f73c1 initial add keeper az info in /keeper/availabilty-zone 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-10-28 21:54:59 +00:00
Nikita Taranov
213ce63ffb
Merge branch 'master' into optimize_reading3 2023-10-28 00:04:20 +02:00
Kruglov Pavel
f1f1846572
Merge pull request #55861 from bigo-sg/ch_55858 
Throw exception when parsing illegal string as float if precise_float_parsing is true
 2023-10-27 14:24:55 +02:00
李扬
308b2942d7
Update readFloatText.h 2023-10-27 11:42:33 +08:00
Yakov Olkhovskiy
0cf851316c use CH Buffer for HTTP out stream, add metrics for interfaces 2023-10-27 02:38:36 +00:00
avogar
9d207bf027 Allow unquoted strings with CR in CSV format 2023-10-26 13:50:54 +00:00
taiyang-li
d5feaa72c1 improve thrown exception message 2023-10-25 20:16:18 +08:00
Nikita Taranov
a86bda5208 use local thread pool reader for cache segments 2023-10-24 14:06:13 +02:00
taiyang-li
2a894335b7 commit again 2023-10-24 16:16:05 +08:00
zvonand
7d54c1fb00 check EOF before reading buffer 2023-10-20 22:27:02 +02:00
zvonand
e5bfc7d3fc Improved DateTime64 parsing 
Allow to parse unambigous short DateTime64
Throw error on invalid DT64 representation (as for DateTime)
 2023-10-20 22:27:02 +02:00
Nikita Taranov
9b13c796e3 set ignore metric in all readers 2023-10-20 22:24:56 +02:00
Nikita Taranov
2a6a97c1fa read sync from cache 2023-10-20 22:24:56 +02:00
taiyang-li
6e3f35fc94 fix issue https://github.com/ClickHouse/ClickHouse/issues/55858 2023-10-20 16:18:39 +08:00
Aleksei Filatov
f56804e6e7 Use pool for proxified S3 http sessions 2023-10-17 12:15:43 +03:00
Michael Kolupaev
ce7eca0615
DWARF input format (#55450) 
* Add ReadBufferFromFileBase::isRegularLocalFile()

* DWARF input format

* Review comments

* Changed things around ENABLE_EMBEDDED_COMPILER build setting

* Added 'ranges' column

* no-msan no-ubsan
 2023-10-16 17:00:07 -07:00
kssenii
43c5e173de Merge remote-tracking branch 'upstream/master' into fs-cache-improvement 2023-10-16 13:06:28 +02:00
alesapin
3b02748cb6 Fix some typos 2023-10-15 15:43:02 +02:00
Alexey Milovidov
9892990f8e
Merge pull request #55262 from azat/compressed-streams-memory-leak 
Avoid possible memory leaks in compressors in case of missing buffer finalization
 2023-10-14 00:22:42 +02:00
Alexey Milovidov
5ad055df98
Merge pull request #55372 from azat/web-disk-fix-race 
Fix data-race in web disk
 2023-10-11 01:03:48 +02:00
Azat Khuzhin
59569b62b9 Add a comment about thread-safety of getFileOffsetOfBufferEnd() 2023-10-10 21:12:31 +02:00
Alexey Milovidov
972f3678fa Review #51946 and partially revert it 2023-10-08 18:00:59 +02:00
Alexey Milovidov
c7ae1afb8f Review #51946 and partially revert it 2023-10-08 15:17:41 +02:00
Azat Khuzhin
28f17bec96 Avoid possible memory leaks in case of missing buffer finalization 
Sometimes it is still possible, the pattern could looks like this:

    try
    {
        writer->finalize();
        writer->flush();
        write_buf->finalize();
    }
    catch (...)
    {
        /// Stop ParallelFormattingOutputFormat correctly.
        release();
        throw;
    }

Here, write_buf will not be finalized, in case of exception during
writer finalize()/flush().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-08 08:51:45 +02:00
kssenii
a72541d93f Improvement for big reads 2023-09-29 19:03:00 +02:00
Alexey Milovidov
8f9a227de1
Merge pull request #54931 from yariks5s/s3_style_url 
s3-style URL fix
 2023-09-29 07:12:01 +03:00
avogar
e8ea41e2d4 Add setting to print pretty deep nested Tuples 2023-09-28 22:17:44 +00:00
yariks5s
61e7054f1f fixed map issue 2023-09-28 17:16:37 +00:00
yariks5s
e4e23d73ea fixed tests + added map as a parameter 2023-09-28 17:04:01 +00:00
Sema Checherinda
f73eef9ed8
Merge pull request #54697 from CheSema/s3_retry_cancelation_point 
add cancelation point to s3 retries
 2023-09-28 19:01:09 +02:00
yariks5s
ad94e08b0b made using macroses 2023-09-28 15:17:15 +00:00
Nikolay Degterinsky
9ebecb5499
Merge pull request #54639 from evillique/timestamp-function 
Add `timestamp` function for compatibility with MySQL
 2023-09-28 16:44:16 +02:00
Sema Checherinda
75e5c80d58 fix typo 2023-09-28 13:37:51 +02:00
Sema Checherinda
29f9673a5f
Update src/IO/S3/Client.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-09-28 12:41:16 +02:00
yariks5s
1dd305107d added default values 2023-09-28 09:55:42 +00:00
yariks5s
d7a2d2fa5b test fix 2023-09-27 23:50:53 +00:00
yariks5s
b3e965b724 small fix 2023-09-27 22:41:35 +00:00
yariks5s
583d4ea8b7 removed chassert 2023-09-27 22:41:02 +00:00
Sema Checherinda
31ea4052d0 cut the build dependency between commin_io and other libraries 2023-09-28 00:15:36 +02:00
Sema Checherinda
5fdca302e6 fix build, resolve process_list_elem race 2023-09-28 00:15:36 +02:00
Sema Checherinda
b9a2c82910 do not share http_client, make code simple 2023-09-28 00:15:26 +02:00
Sema Checherinda
e629a17218 add cancelation point to s3 retries 2023-09-28 00:15:06 +02:00
Robert Schulze
675819be6b
Minor fixes 2023-09-27 21:15:20 +00:00
Nikolay Degterinsky
c1f59eccd5 Add fraction part to the time argument, add docs, better test 2023-09-27 20:17:32 +00:00
Yarik Briukhovetskyi
9c337c7687
Update URI.cpp 2023-09-27 20:55:29 +02:00
yariks5s
adf1af63a5 added tests, moved domains to config 2023-09-27 18:53:05 +00:00
robot-clickhouse-ci-1
7a825c1417
Merge pull request #54427 from Avogar/json-object-as-tuple-inference 
Add new features to schema inference for JSON formats
 2023-09-27 20:12:45 +02:00
Robert Schulze
cde10fe7b5
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-reenable-checks 2023-09-26 18:59:41 +00:00
avogar
6b3dbc4403 Apply suggestions 2023-09-26 16:41:35 +00:00
Vitaly Baranov
fe008c23c4
Merge pull request #54900 from vitlibar/retry-backup-s3-operations-after-conection-reset 
Retry backup S3 operations after connection reset failure
 2023-09-26 18:36:10 +02:00
Antonio Andelic
535b9337f4
Merge pull request #55012 from ClickHouse/lower-s3-credential-logs 
Lower log levels for `SSOCredentialsProvider`
 2023-09-26 16:08:00 +02:00
yariks5s
4939e59cfe Merge branch 'master' into s3_style_url 2023-09-26 13:24:00 +00:00
Kruglov Pavel
bea80ab5b7
Merge branch 'master' into json-object-as-tuple-inference 2023-09-26 15:23:08 +02:00
Kruglov Pavel
69a17bbef6
Merge pull request #52853 from Avogar/http-valid-json-on-exception 
Output valid JSON/XML on excetpion during HTTP query execution
 2023-09-26 14:25:55 +02:00
Robert Schulze
9fff447716
Re-enable clang-tidy checks 2023-09-26 09:34:12 +00:00
Antonio Andelic
ef6c7fe45d Lower log levels for SSOCredentialsProvider 2023-09-26 08:30:34 +00:00
avogar
9e75825515 Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-object-as-tuple-inference 2023-09-25 17:24:36 +00:00
avogar
42ca897f2d Better schema inference for JSON formats 2023-09-25 15:42:59 +00:00
serxa
44acb53ad3 Merge branch 'master' into io-scheduling-throttler 2023-09-23 10:51:18 +00:00
Vitaly Baranov
1e567d5008 Retry backup s3 operations after ConnectionResetException. 2023-09-23 01:56:28 +02:00
yariks5s
e1f40adfe2 fix typo 2023-09-22 16:21:08 +00:00
yariks5s
dce8908641 fixed url convers. 2023-09-22 16:13:29 +00:00
Robert Schulze
877e4f3aab
Merge remote-tracking branch 'rschu1ze/master' into clang-17 2023-09-21 20:21:12 +00:00
Michael Kolupaev
9af9b4a085
Enable connection pooling for s3 table function (#54812) 
Enable connection pooling for s3 table function
 2023-09-21 09:27:20 -07:00
Robert Schulze
5209bd2d51
Merge remote-tracking branch 'rschu1ze/master' into clang-17 2023-09-21 14:45:55 +00:00
Robert Schulze
f5137dd0b4
More clang-tidy fixes 2023-09-21 14:40:57 +00:00
Vitaly Baranov
bf4662aca7
Merge pull request #53336 from azat/backups/s3-throttling 
Fix throttling of BACKUPs from/to S3 (in case native copy was not used) and in some other places as well
 2023-09-21 10:42:28 +02:00
Azat Khuzhin
c706101891 Fix throttling of BACKUPs from/to S3 (in case native copy was not used) 
In some cases native copy is not possible, and such requests should be
throttled.

v0: copyS3FileNativeWithFallback
v2: revert v0 and pass write_settings
v3: pass read_settings to copyFile()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-20 18:28:43 +02:00
Duc Canh Le
2a5b4ee29a Merge branch 'master' into fix_http_timeout 
To trigger CI pipeline
 2023-09-20 07:22:02 +00:00
Kruglov Pavel
e163670357
Merge branch 'master' into http-valid-json-on-exception 2023-09-19 13:42:53 +02:00
serxa
14569b0f7b fix start time in test 2023-09-19 10:13:05 +00:00
serxa
b154f71a57 Merge branch 'io-scheduling-throttler' of github.com:ClickHouse/ClickHouse into io-scheduling-throttler 2023-09-19 10:10:35 +00:00
serxa
311db94640 default max_burst equals to max_speed as in Throttler 2023-09-19 10:10:21 +00:00
serxa
d97b4f0685 better semantics for locking 2023-09-19 10:09:16 +00:00
serxa
71655cda8e add more tests 2023-09-19 10:08:09 +00:00
Sergei Trifonov
b2ed38742c
Update src/IO/ISchedulerNode.h 
Co-authored-by: vdimir <vdimir@clickhouse.com>
 2023-09-19 11:32:05 +02:00
serxa
c5f3170a82 update actual amount of tokens in bucket 2023-09-18 18:12:55 +00:00