Commit Graph

3241 Commits

Author SHA1 Message Date
Alexey Milovidov
aa462fa7cd
Merge branch 'master' into feature-server-iface-metrics 2023-12-29 22:37:52 +01:00
Azat Khuzhin
c6774cfe42 Add missing includes for ZipArchiveWriter
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-27 17:06:17 +01:00
Azat Khuzhin
cf1dae5321 Add missing includes into BitHelpers.h
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-27 17:06:17 +01:00
Azat Khuzhin
b9233f6d4f Move Allocator code into module part
This should reduce amount of code that should be recompiled on
Exception.h changes (and everything else that had been included there).

This will actually not help a lot, because it is also included into
PODArray.h and ThreadPool.h at least... Sigh.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-27 15:42:08 +01:00
Yarik Briukhovetskyi
41c275274b
Merge pull request #58148 from yariks5s/fix_s3_regions
S3-links region independency
2023-12-26 12:59:00 +01:00
Alexey Milovidov
817d0c6553
Merge pull request #58202 from vitlibar/explicit-finalize-in-zip-archive-writer
Explicit finalize in ZipArchiveWriter
2023-12-24 03:30:21 +01:00
Vitaly Baranov
e1a136b791 Explicit finalize() function in ZipArchiveWriter.
Simplify too complicated code in ZipArchiveWriter.
2023-12-24 00:33:59 +01:00
Alexey Milovidov
a400657155
Merge pull request #58165 from azat/s3-passthrough-exception
Pass through exceptions for reading from S3
2023-12-23 11:35:23 +01:00
Azat Khuzhin
896cbb6a75 Retry on poco exceptions while reading from S3 as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-23 09:07:43 +01:00
Alexey Milovidov
ffca9bc564 Fix #58171 2023-12-23 02:04:35 +01:00
Azat Khuzhin
17e21aa21e Pass throught exceptions for reading from S3
Reading from S3 has retries on almost any exception, except for
CANNOT_ALLOCATE_MEMORY, but the problem is that this exception may not
be even delivered to ReadBufferFromS3::processException(), due to
std::istream will hide it (see [1]).

  [1]: 1834e42289/libcxx/include/istream (L1059)

So let's pass throught them to see at least some information instead of
just:

    Message: Cannot read from istream at offset 0

And here as an example of exception:

<details>

    (lldb) bt
    * thread 244, name = 'MergeMutate', stop reason = breakpoint 6.1
      * frame 0: 0x000000001aa9c791 clickhouse`::__cxa_throw(thrown_object=0x00007fc3b68fdb00, tinfo=0x000000000548af50, dest=(clickhouse`Poco::Net::NetException::~NetException() at NetException.cpp:26))(void *)) at cxa_exception.cpp:258:33
        frame 1: 0x0000000017c55b2b clickhouse`Poco::Net::SocketImpl::error(code=-1232086272, arg=0x00007fc3be4f4d30) at SocketImpl.cpp:985:3
        frame 2: 0x0000000017c1f7ff clickhouse`Poco::Net::SecureSocketImpl::handleError(int) [inlined] Poco::Net::SocketImpl::error(code=<unavailable>) at SocketImpl.cpp:923:2
        frame 3: 0x0000000017c1f7e4 clickhouse`Poco::Net::SecureSocketImpl::handleError(this=0x00007fc35ae42b30, rc=-1) at SecureSocketImpl.cpp:531
        frame 4: 0x0000000017c208f9 clickhouse`Poco::Net::SecureSocketImpl::receiveBytes(this=0x00007fc35ae42b30, buffer=0x00007fc346e402c4, length=131068, flags=<unavailable>) at SecureSocketImpl.cpp:353:10
        frame 5: 0x0000000017c3b388 clickhouse`Poco::Net::HTTPSession::read(char*, long) [inlined] Poco::Net::StreamSocket::receiveBytes(buffer=<unavailable>, length=<unavailable>, flags=0) at StreamSocket.cpp:130:17
        frame 6: 0x0000000017c3b379 clickhouse`Poco::Net::HTTPSession::read(char*, long) [inlined] Poco::Net::HTTPSession::receive(this=<unavailable>, buffer=<unavailable>, length=<unavailable>) at HTTPSession.cpp:166
        frame 7: 0x0000000017c3b379 clickhouse`Poco::Net::HTTPSession::read(this=0x00007fc3973e9098, buffer=<unavailable>, length=<unavailable>) at HTTPSession.cpp:144
        frame 8: 0x0000000017c0f706 clickhouse`Poco::Net::HTTPSClientSession::read(this=<unavailable>, buffer=<unavailable>, length=<unavailable>) at HTTPSClientSession.cpp:195:23
        frame 9: 0x0000000017c32eb4 clickhouse`Poco::Net::HTTPFixedLengthStreamBuf::readFromDevice(this=0x00007fc361bd6708, buffer=<unavailable>, length=<unavailable>) at HTTPFixedLengthStream.cpp:53:16
        frame 10: 0x0000000017c2a368 clickhouse`Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::underflow(this=0x00007fc361bd6708) at BufferedStreamBuf.h:97:17
        frame 11: 0x0000000008b2abca clickhouse`std::__1::basic_streambuf<char, std::__1::char_traits<char> >::uflow(this=0x00007fc361bd6708) at streambuf:445:9
        frame 12: 0x0000000008b2ab5a clickhouse`std::__1::basic_streambuf<char, std::__1::char_traits<char> >::xsgetn(this=0x00007fc361bd6708, __s="P, __n=1048576) at streambuf:422:25
        frame 13: 0x0000000008b2ca0e clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char> >::read(char*, long) [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char> >::sgetn[abi:v15000](this=<unavailable>, __s="!P, __n=1048576) at streambuf:204:14
        frame 14: 0x0000000008b2ca02 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char> >::read(this=0x00007fc35ae42d40, __s="!P, __n=1048576) at istream:1052
        frame 15: 0x0000000010de0b9f clickhouse`DB::ReadBufferFromIStream::nextImpl(this=0x00007fc3663d2380) at ReadBufferFromIStream.cpp:15:10

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-22 23:29:28 +01:00
yariks5s
50724f1b5d Merge branch 'fix_s3_regions' of github.com:yariks5s/ClickHouse into fix_s3_regions 2023-12-22 17:28:22 +00:00
yariks5s
a543048b5f review fix 2023-12-22 17:28:09 +00:00
Yarik Briukhovetskyi
ab6cc0c9f0
Update Client.cpp 2023-12-22 14:59:21 +01:00
yariks5s
823abaa73e init 2023-12-22 13:27:03 +00:00
Yakov Olkhovskiy
5dc88542cb
Merge branch 'master' into feature-server-iface-metrics 2023-12-20 16:46:29 -05:00
Kruglov Pavel
ca7001d70f
Merge pull request #57850 from shanielh/feature/s3-session-token
Feature/s3 session token
2023-12-20 17:22:09 +01:00
Julia Kartseva
2c32589949
Merge pull request #58019 from jkartseva/fix-build-errno-exception
fix freebsd build
2023-12-19 11:16:45 -08:00
Sema Checherinda
802ad4ec47
Merge pull request #57523 from ClickHouse/remove-the-limit-for-connections-per-endpoint
Remove the limit for the number of connections per endpoint for background fetches + small refactoring
2023-12-19 19:16:01 +01:00
vdimir
252cb8a507
Merge pull request #54602 from fionera/issue-49028
Add setting http_make_head_request
2023-12-19 13:33:59 +01:00
Julia Kartseva
31f04b66c5 fix freebsd build
https://github.com/ClickHouse/ClickHouse/actions/runs/7256640256/job/19769624432?pr=58015#step:8:5466
```
Dec 19 03:09:41 /build/src/IO/AIO.cpp:140:15: error: use of undeclared identifier 'ErrnoException'; did you mean 'DB::ErrnoException'?
Dec 19 03:09:41   140 |         throw ErrnoException(DB::ErrorCodes::CANNOT_IOSETUP, "io_setup failed");
```
2023-12-19 03:47:28 +00:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
Raúl Marín
65728f14ce Review error codes 2023-12-18 10:40:02 +01:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
2023-12-17 10:29:15 +02:00
Yakov Olkhovskiy
42a770f209 optimize event update 2023-12-15 19:51:52 +00:00
Yakov Olkhovskiy
001a38048f use ProfileEvents instead of CurrentMetrics 2023-12-15 19:17:42 +00:00
Raúl Marín
b269f87f4c Better text_log with ErrnoException 2023-12-15 19:27:56 +01:00
Yakov Olkhovskiy
febdd542f8 Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-12-14 18:59:44 +00:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Kruglov Pavel
909b1fb36b
Merge pull request #57004 from Avogar/template-deadlock
Fix possible deadlock in Template format during sync after error
2023-12-11 19:02:40 +01:00
Igor Nikonov
87e47bdbee
Merge pull request #57467 from ClickHouse/rm-select
Remove select() usage
2023-12-09 22:27:28 +01:00
Alexey Milovidov
0ee13fde16 Fix "logs and exception messages formatting", part 3 2023-12-07 22:01:53 +01:00
Igor Nikonov
d196973ab3 Merge remote-tracking branch 'origin/master' into rm-select 2023-12-05 12:59:18 +00:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads
Add ability to disable checksums for S3 to avoid excessive input file read
2023-12-05 06:34:21 +01:00
kssenii
4a28f10c3d Minor cache changes 2023-12-04 19:02:37 +01:00
vdimir
a4ae90de0d
Merge pull request #57275 from ClickHouse/vdimir/merge_task_tmp_data
Background merges correctly use temporary data storage in the cache
2023-12-04 14:52:20 +01:00
Igor Nikonov
434c2113d2 Fix style 2023-12-04 10:51:17 +00:00
Igor Nikonov
73ff984755 Remove select() usage 2023-12-04 09:33:34 +00:00
robot-ch-test-poll
1b49463bd2
Merge pull request #55841 from nickitat/optimize_reading3
Optimize reading from cache
2023-12-01 17:36:57 +01:00
Yakov Olkhovskiy
416d28687e Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-11-30 23:00:04 +00:00
Nikolai Kochetov
823ba2db46
Merge pull request #57075 from yariks5s/s3_links_fix
S3-style links bug fix
2023-11-29 17:41:08 +01:00
vdimir
b5babe1692
MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
Nikita Taranov
52f644c0df Merge branch 'master' into optimize_reading3 2023-11-28 16:36:38 +01:00
Antonio Andelic
6e8e4a6ca5 Lower level for annoying log 2023-11-28 12:41:35 +00:00
Yakov Olkhovskiy
d8e41d8886 fix after merge 2023-11-28 03:28:32 +00:00
Yakov Olkhovskiy
c0a9b82c3c fix 2023-11-27 22:06:01 +00:00
Yakov Olkhovskiy
52ef3fe34b style fix 2023-11-27 21:32:50 +00:00
Yakov Olkhovskiy
4d40ed851f fix after merge 2023-11-27 21:05:35 +00:00