Vladimir Chebotarev
6e379fe778
Final style fix.
2021-04-15 10:01:47 +03:00
Vladimir Chebotarev
71621eb676
Style fix.
2021-04-15 10:01:47 +03:00
Vladimir Chebotarev
f85e521e47
Style fix.
2021-04-15 10:01:47 +03:00
Vladimir Chebotarev
342d907b65
Review fixes.
2021-04-15 10:01:47 +03:00
Vladimir Chebotarev
116423b30c
Some refactoring.
2021-04-15 10:01:47 +03:00
Vladimir Chebotarev
c0ea3ce08a
Removed STS client.
2021-04-15 10:01:43 +03:00
Vladimir Chebotarev
c61cfd0d97
Minor build fixes.
2021-04-15 10:01:16 +03:00
Vladimir Chebotarev
aae8a1f9d9
Moved AWSSTSAssumeRoleCredentialsProvider
to the top of the provider chain.
2021-04-15 10:01:16 +03:00
Vladimir Chebotarev
3a467ffc50
Attempt to make credentials provider which uses internal credentials provider chain.
2021-04-15 10:01:13 +03:00
Vladimir Chebotarev
037a4de854
Added STS GetCallerIdentity credentials provider.
2021-04-15 10:00:18 +03:00
Vladimir Chebotarev
817c5cec25
Correct S3 read retries.
2021-04-15 09:47:22 +03:00
Vladimir Chebotarev
cf903d526c
Better logging.
2021-04-15 09:47:22 +03:00
Vladimir Chebotarev
10bf56e540
Attempt to add retries to S3 reads.
2021-04-15 09:47:19 +03:00
Azat Khuzhin
9b9e0a9bbc
Block all memory tracking limits in destructors
2021-04-14 23:38:42 +03:00
Nikolai Kochetov
9b86c19836
Use JSONBuilder for explain json result.
2021-04-14 17:07:56 +03:00
Nikita Mikhaylov
b3094412b1
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
4e4b383214
added hash of itiator address
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
2549468c14
better
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
0be3fa178b
save
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
44ca65a9a4
save
2021-04-13 22:39:41 +03:00
alexey-milovidov
ee9aaab4bc
Merge pull request #22937 from Enmk/formatDateTime_toDateTime64_fixes
...
Fixes: formatDateTime and toDateTime64
2021-04-13 01:40:25 +03:00
alexey-milovidov
9e88f5fbd3
Update ReadHelpers.h
2021-04-13 01:39:15 +03:00
alexey-milovidov
a780e8e7e1
Update ReadHelpers.h
2021-04-13 01:38:59 +03:00
Vasily Nemkov
0f7c83b5ac
Fixed parsing DateTime64 as unix timestamp with subsecond precision scaled to integer.
...
e.g. toDateTime64('1111111111222', 3)
2021-04-11 17:28:58 +03:00
kreuzerkrieg
228a0a0647
Fix an error handling in Poco HTTP Client for AWS.
2021-04-11 17:22:19 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Vasily Nemkov
c882d290bd
Fixed parsing string to DateTime64 for large values.
...
Before fix:
SELECT toDateTime64('2201-01-12 12:12:12', 3, 'UTC')
1970-03-26 09:10:48.237
After fix:
SELECT toDateTime64('2201-01-12 12:12:12', 3, 'UTC')
2201-01-12 12:12:12.000
2021-04-09 19:39:54 +03:00
changvvb
b070090ca7
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into support-apple-m1
2021-04-08 18:11:16 +08:00
Ivan
aa47eb7d5e
Fix checkpoint position after putting it on buffer end ( #22518 )
2021-04-07 16:22:38 +03:00
changvvb
1f8bd034a3
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into support-apple-m1
2021-04-06 15:55:03 +08:00
changvvb
3a9d063f12
Fix unit test
2021-04-06 15:54:35 +08:00
Vladimir Chebotarev
0eb051961c
Consistent AWS timeouts.
2021-04-04 14:06:10 +03:00
Ivan Lezhankin
d9d143675a
Merge remote-tracking branch 'upstream/master' into issue-21907
2021-04-01 16:29:26 +03:00
Pavel Kruglov
5f3cb525ee
Remove setting same timeouts in secure socket
2021-03-30 10:51:44 +03:00
Ivan Lezhankin
ae741c7411
Do not limit HTTP chunk size
2021-03-29 20:49:28 +03:00
alexey-milovidov
7ab2890c4d
Merge pull request #22206 from ClickHouse/mmap-cache
...
Add cache for mmap IO
2021-03-29 05:17:55 +03:00
Maksim Kita
459d00f999
Fixed tests
2021-03-28 22:42:34 +03:00
Alexey Milovidov
2a8ac01cdb
Rename as suggested by Kita
2021-03-28 22:24:28 +03:00
Alexey Milovidov
15b41fd110
Review fix
2021-03-28 22:15:13 +03:00
Maksim Kita
608d37deed
CachedCompressedReadBuffer fix cache usage
2021-03-28 21:32:38 +03:00
Alexey Milovidov
50f712e198
Integrate mmap cache to the infrastructure
2021-03-28 04:10:30 +03:00
Alexey Milovidov
bdb69b8f08
Slightly better
2021-03-28 02:11:46 +03:00
Alexey Milovidov
cbf5913109
Review fixes
2021-03-28 01:57:59 +03:00
Alexey Milovidov
bcdf3dce36
Add cache for mmapped files
2021-03-27 02:22:51 +03:00
Alexey Milovidov
1d9e23f925
Fix error
2021-03-27 01:28:57 +03:00
Alexey Milovidov
7aac5e1849
Preparation
2021-03-26 23:51:46 +03:00
Alexey Milovidov
ff891c5002
Preparation
2021-03-26 23:46:04 +03:00
alexey-milovidov
607d36dd9d
Merge pull request #22147 from azat/writers-fixes
...
Use finalize() over next() for nested writers
2021-03-26 22:05:18 +03:00
alexey-milovidov
ac53d30e53
Merge pull request #22146 from azat/writers-uncaught-exceptions
...
Fix uncaught exception in InterserverIOHTTPHandler
2021-03-26 17:23:09 +03:00
tavplubix
d9ec4090d2
Merge pull request #21936 from abyss7/html-form-fix
...
Fix issues with HTMLForm::MultipartReadBuffer and PeekableReadBuffer
2021-03-26 15:35:34 +03:00
tavplubix
fa930d49c4
Update gtest_peekable_read_buffer.cpp
2021-03-26 10:29:58 +03:00
Azat Khuzhin
f1907acbcd
Use finalize() over next() for nested writers
...
Refs: https://github.com/ClickHouse/ClickHouse/pull/21325#discussion_r585348309
2021-03-25 23:42:58 +03:00
Azat Khuzhin
50003e496a
Use existing logger for logging from WriteBufferFromS3 dtor
2021-03-25 23:41:03 +03:00
Kruglov Pavel
66b8431e71
Merge pull request #21429 from Avogar/secure-socket
...
Fix receive and send timeouts and non-blocking read in secure socket
2021-03-25 17:30:17 +03:00
Ivan Lezhankin
be76defcfa
Remove check for absent checkpoint on rollback
2021-03-23 14:59:36 +03:00
Pavel Kovalenko
d55af1fa45
Merge branch 'write-buffer-s3-better-error-handling' of github.com:Jokser/ClickHouse into write-buffer-s3-better-error-handling
2021-03-23 13:02:57 +03:00
Pavel Kovalenko
393631a753
Merge remote-tracking branch 'origin/master' into write-buffer-s3-better-error-handling
...
# Conflicts:
# src/IO/WriteBufferFromS3.cpp
2021-03-23 13:00:09 +03:00
Kruglov Pavel
a215bf7812
Restart CI
2021-03-22 22:12:42 +03:00
Ivan Lezhankin
6d4d669f96
Move checkpoint to sub-buffer only from internal memory
2021-03-22 20:20:42 +03:00
feng lv
d40ffa96e2
Fix misleading log in WriteBufferFromS3
2021-03-22 05:49:41 +00:00
Ivan Lezhankin
957c053f7e
Fix segfault
2021-03-19 23:29:01 +03:00
Kruglov Pavel
0cb88fb20e
Restart tests
2021-03-19 15:00:21 +03:00
Pavel Kruglov
d380454044
Change read_data_timeout from sec to ms, add profile event for hedged requests and better process receiving packets
2021-03-18 20:16:17 +03:00
Pavel Kovalenko
71b36a7029
Better error handling and logging in WriteBufferFromS3
2021-03-17 17:20:55 +03:00
alexey-milovidov
d02726bcac
Merge pull request #9404 from Enmk/DateTime64_extended_range
...
Date time64 extended range
2021-03-17 11:06:03 +03:00
Nikita Mikhaylov
1d15ed34ce
Merge pull request #21737 from excitoon-favorites/reverts3connectionpools
...
Reverted S3 connection pools
2021-03-16 14:40:54 +03:00
Nikita Mikhaylov
7d7a2bc724
Merge pull request #21630 from nikitamikhaylov/pf-local
...
Parallel formatting for clickhouse-local
2021-03-16 14:37:40 +03:00
alexey-milovidov
f52e29cfec
Merge pull request #21624 from vdimir/storage-s3-fail
...
Try to fix test_storage_s3: crash in WriteBufferFromS3
2021-03-16 13:38:48 +03:00
Alexey Milovidov
671395e8c8
Most likely improve performance
2021-03-15 22:23:27 +03:00
Vladimir Chebotarev
35e0d0f672
Reverted S3 connection pools.
2021-03-15 14:17:16 +03:00
alexey-milovidov
53e21443eb
Merge branch 'master' into DateTime64_extended_range
2021-03-14 12:27:17 +03:00
Nikita Mikhaylov
38e1cb41cc
done
2021-03-11 17:04:59 +03:00
vdimir
1194d50e48
Try to fix test_storage_s3: crash in WriteBufferFromS3
2021-03-11 15:21:13 +03:00
Nikita Mikhailov
a6c8d50cc2
review fix
2021-03-11 15:08:19 +03:00
Nikita Mikhailov
6a51ad425e
add tests + improvements
2021-03-11 15:08:18 +03:00
Nikita Mikhailov
adfb2885c9
better
2021-03-11 15:08:18 +03:00
Nikita Mikhailov
41d006d493
try fix stress
2021-03-11 15:08:17 +03:00
Pavel Kruglov
c113af39f4
Merge branch 'master' of github.com:ClickHouse/ClickHouse into secure-socket
2021-03-05 16:18:45 +03:00
Alexey Milovidov
3e77c9b1c5
Merge branch 'master' into DateTime64_extended_range
2021-03-05 03:07:19 +03:00
Kseniia Sumarokova
08cb02c872
Merge pull request #21416 from excitoon-favorites/caseinsensitivecompression
...
Case-insensitive compression methods for table functions
2021-03-04 17:37:28 +03:00
Alexey Milovidov
1096fc328a
Make some tests automated
2021-03-04 02:11:27 +03:00
Alexey Milovidov
4e8239e098
Merge branch 'master' into DateTime64_extended_range
2021-03-03 23:43:20 +03:00
Pavel Kruglov
1c7f16e0ff
Add test and move TimeoutSetter in IO/
2021-03-03 20:47:27 +03:00
Pavel Kruglov
eef7d8c0bc
Fix blocking mode and timeouts in SecureStreamSocket
2021-03-03 16:10:15 +03:00
Vladimir Chebotarev
4022418ff3
Case-insensitive compression methods.
2021-03-03 14:51:09 +03:00
Pavel Kruglov
153bfbfc28
Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests
2021-03-02 11:59:32 +03:00
alexey-milovidov
81be2d1ed9
Merge pull request #21305 from azat/terminate-nested-writers
...
Fix abnormal server termination for nested writers
2021-02-28 21:54:13 +03:00
Azat Khuzhin
eb0387c5a9
Fix abnormal server termination for nested writers
...
Writers with nested writer can call next() from the dtor for nested
writer and this will cause exception again, so the buffer position
should be updated on exceptions.
Found by stress test (thread) here [1] and here [2]:
2021.02.27 19:27:53.498977 [ 302 ] {} <Fatal> BaseDaemon: (version 21.3.1.6130, build id: 2DAEC5DEBF03C5A1C3BF66B7779C886F16239345) (from thread 1614) Terminate called for uncaught exception:
Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 2097498, Stack trace (when copying this message, always include the lines below):
0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15bef2ab in /usr/bin/clickhouse
1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8aea92e in /usr/bin/clickhouse
2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8bbbc45 in /usr/bin/clickhouse
3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8bbc077 in /usr/bin/clickhouse
4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8bbc18a in /usr/bin/clickhouse
5. ./obj-x86_64-linux-gnu/../src/IO/BufferWithOwnMemory.h:137: DB::ZstdDeflatingWriteBuffer::~ZstdDeflatingWriteBuffer() @ 0x118bdc29 in /usr/bin/clickhouse
6. ./obj-x86_64-linux-gnu/../src/IO/ZstdDeflatingWriteBuffer.cpp:32: DB::ZstdDeflatingWriteBuffer::~ZstdDeflatingWriteBuffer() @ 0x118be3ea in /usr/bin/clickhouse
7. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:0: DB::WriteBufferFromHTTPServerResponse::finalize() @ 0x12f1dceb in /usr/bin/clickhouse
8. ./obj-x86_64-linux-gnu/../src/Server/HTTPHandler.cpp:703: DB::HTTPHandler::trySendExceptionToClient(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x12e9fecc in /usr/bin/clickhouse
9. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/string:1444: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x12ea0d60 in /usr/bin/clickhouse
10. ./obj-x86_64-linux-gnu/../src/Server/HTTP/HTTPServerConnection.cpp:0: DB::HTTPServerConnection::run() @ 0x12f16db1 in /usr/bin/clickhouse
11. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x15b184f3 in /usr/bin/clickhouse
12. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:0: Poco::Net::TCPServerDispatcher::run() @ 0x15b18c1f in /usr/bin/clickhouse
13. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/ScopedLock.h:36: Poco::PooledThread::run() @ 0x15c7fdb2 in /usr/bin/clickhouse
14. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x15c7e350 in /usr/bin/clickhouse
15. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/SharedPtr.h:277: Poco::ThreadImpl::runnableEntry(void*) @ 0x15c7cb58 in /usr/bin/clickhouse
16. __tsan_thread_start_func @ 0x8a04ced in /usr/bin/clickhouse
17. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
18. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
(version 21.3.1.6130)
[1]: https://clickhouse-test-reports.s3.yandex.net/21279/4f61ef3099f42f17b496a0b0424773978d9a32dc/stress_test_(thread).html#fail1
[2]: https://clickhouse-test-reports.s3.yandex.net/21292/ae9fea1d0af118a8f87b224d194d61da1567188b/stress_test_(thread).html#fail1
v2: https://clickhouse-test-reports.s3.yandex.net/21305/e969daa6e86c5e09cfef08cfde19712982b64e59/stress_test_(thread).html#fail1
2021-02-28 15:13:42 +03:00
Azat Khuzhin
568a49dad5
Remove unused AsynchronousWriteBuffer
2021-02-28 11:19:56 +03:00
Azat Khuzhin
8b58dba094
Drop unused HexWriteBuffer
2021-02-28 10:52:09 +03:00
Pavel Kruglov
197058dc12
Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests
2021-02-27 00:36:33 +03:00
kssenii
991718d459
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into brotli
2021-02-26 14:31:24 +00:00
Vasily Nemkov
2d03d330bc
Extended range of DateTime64 to years 1925 - 2238
...
The Year 1925 is a starting point because most of the timezones
switched to saner (mostly 15-minutes based) offsets somewhere
during 1924 or before. And that significantly simplifies implementation.
2238 is to simplify arithmetics for sanitizing LUT index access;
there are less than 0x1ffff days from 1925.
* Extended DateLUTImpl internal LUT to 0x1ffff items, some of which
represent negative (pre-1970) time values.
As a collateral benefit, Date now correctly supports dates up to 2149
(instead of 2106).
* Added a new strong typedef ExtendedDayNum, which represents dates
pre-1970 and post 2149.
* Functions that used to return DayNum now return ExtendedDayNum.
* Refactored DateLUTImpl to untie DayNum from the dual role of being
a value and an index (due to negative time). Index is now a different
type LUTIndex with explicit conversion functions from DatNum, time_t,
and ExtendedDayNum.
* Updated DateLUTImpl to properly support values close to epoch start
(1970-01-01 00:00), including negative ones.
* Reduced resolution of DateLUTImpl::Values::time_at_offset_change
to multiple of 15-minutes to allow storing 64-bits of time_t in
DateLUTImpl::Value while keeping same size.
* Minor performance updates to DateLUTImpl when building month LUT
by skipping non-start-of-month days.
* Fixed extractTimeZoneFromFunctionArguments to work correctly
with DateTime64.
* New unit-tests and stateless integration tests for both DateTime
and DateTime64.
2021-02-24 17:08:35 +02:00
Pavel Kruglov
10f54f1714
remove include
2021-02-21 17:18:50 +03:00
Pavel Kruglov
abb5993bd5
return const
2021-02-21 17:13:33 +03:00
Pavel Kruglov
7c76b4a98a
Remove debug logging
2021-02-21 17:12:26 +03:00
Kruglov Pavel
815b4ff0d9
Merge branch 'master' into hedged-requests
2021-02-21 17:10:24 +03:00
Pavel Kruglov
dd5185c779
Refactor 4.0
2021-02-21 17:03:24 +03:00
Alexey Milovidov
af2f4ceb0e
Add disambiguation while parsing already scaled DateTime64 #13194
2021-02-21 15:34:04 +03:00
Alexey Milovidov
b736f4fdee
Suppression for PVS-Studio (looks like a false positive)
2021-02-21 10:21:59 +03:00
Alexey Milovidov
64e76a4a8d
Minor changes in Decimal
2021-02-20 21:13:36 +03:00
kssenii
21ee685ef0
Fix brotly
2021-02-19 22:18:14 +00:00
Ivan
414f470c79
Make Poco HTTP Server zero-copy again ( #19516 )
...
* Refactoring: part 1
* Refactoring: part 2
* Handle request using ReadBuffer interface
* Struggles with ReadBuffer's
* Fix URI parsing
* Implement parsing of multipart/form-data
* Check HTTP_LENGTH_REQUIRED before eof() or will hang
* Fix HTTPChunkedReadBuffer
* Fix build and style
* Fix test
* Resist double-eof
* Fix arcadian build
2021-02-19 15:51:26 +03:00
Kruglov Pavel
794c3449ca
Merge branch 'master' into hedged-requests
2021-02-17 20:36:36 +03:00
Pavel Kruglov
1b78de2142
Use fibers in HedgedRequests
2021-02-17 20:34:52 +03:00
Nikita Mikhailov
6ee98ead05
Merge branch 'master' of github.com:ClickHouse/ClickHouse into keen-wolf-filefuncwithreadbuf
2021-02-16 15:38:03 +03:00
Nikita Mikhailov
7c5d845866
refactor function
2021-02-16 15:37:49 +03:00
Nikita Mikhaylov
cb2beb475e
Merge pull request #20516 from nikitamikhaylov/more-checks-filesegmentation-engine
...
More checks in fileSegmentationEngine
2021-02-16 12:39:27 +03:00
Kruglov Pavel
598576ce70
Merge branch 'master' into hedged-requests
2021-02-15 16:51:45 +03:00
Nikita Mikhailov
d615b8e516
more checks
...
(cherry picked from commit b45168ecaf37d0061edfd12c67a8c5300d45d2e3)
2021-02-15 16:11:16 +03:00
alexey-milovidov
a299790522
Merge pull request #19748 from excitoon-favorites/s3ssec
...
Added Server Side Encryption Customer Keys support in S3 client
2021-02-15 15:23:14 +03:00
Nikita Mikhaylov
a77b740a7c
Merge pull request #20286 from nikitamikhaylov/json-import-bugfix
...
Error from allocator on JSON import
2021-02-15 12:40:09 +03:00
alesapin
9557bca4af
Merge pull request #20464 from azat/abnormal-server-termination-fix
...
Fix abnormal server termination when http client goes away
2021-02-15 11:10:29 +03:00
Azat Khuzhin
33f54cdb3c
Fix abnormal server termination when http client goes away
...
In [1] stress tests found:
2021.02.12 14:20:58.800988 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Debug> executeQuery: (from [::1]:45792, using production parser) (comment: /usr/share/clickhouse-test/queries/0_stateless/01520_client_print_query_id.expect) SELECT * FROM numbers(34599)
2021.02.12 14:20:58.916484 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Trace> ContextAccess (default): Access granted: CREATE TEMPORARY TABLE ON *.*
2021.02.12 14:20:59.071980 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2021.02.12 14:21:10.708202 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Information> executeQuery: Read 34599 rows, 270.30 KiB in 11.876294055 sec., 2913 rows/sec., 22.76 KiB/sec.
2021.02.12 14:22:10.506261 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Debug> DynamicQueryHandler: Done processing query
2021.02.12 14:22:18.238037 [ 375 ] {} <Fatal> BaseDaemon: (version 21.3.1.5996, build id: 8DBCED54529C989F7AD4D991F51410774D55DE6C) (from thread 17728) Terminate called for uncaught exception:
Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 262994, Stack trace (when copying this message, always include the lines below):
0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15c976cb in /usr/bin/clickhouse
1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8c9320e in /usr/bin/clickhouse
2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8d54da5 in /usr/bin/clickhouse
3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8d551d7 in /usr/bin/clickhouse
4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::Write
2021.02.12 14:22:18.811071 [ 18134 ] {} <Fatal> BaseDaemon: ########################################
2021.02.12 14:22:18.878935 [ 18134 ] {} <Fatal> BaseDaemon: (version 21.3.1.5996, build id: 8DBCED54529C989F7AD4D991F51410774D55DE6C) (from thread 17728) (query_id: de3e7894-b401-4f7d-8530-90cd5ab06682) Received signal Aborted (6)
2021.02.12 14:22:18.943148 [ 18134 ] {} <Fatal> BaseDaemon:
2021.02.12 14:22:19.007073 [ 18134 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f109932018b 0x7f10992ff859 0x8bb33ae 0x8e301dd 0x17dac8c4 0x17dac7c7 0x8c3fe0b 0x8d552c5 0x8d552ea 0x11a29914 0x11a2a2ca 0x12f96092 0x12f8c65e 0x12f84300 0x15b84110 0x15bc0913 0x15bc103f 0x15d29a12 0x15d27fb0 0x15d267b8 0x8badbad 0x7f10994d5609 0x7f10993fc293
2021.02.12 14:22:19.255998 [ 18134 ] {} <Fatal> BaseDaemon: 5. raise @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
2021.02.12 14:22:19.270203 [ 18134 ] {} <Fatal> BaseDaemon: 6. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
2021.02.12 14:22:50.108918 [ 370 ] {} <Fatal> Application: Child process was terminated by signal 6.
[1]: https://clickhouse-test-reports.s3.yandex.net/19580/6aecb62416ece880cbb8ee3a803e14d841388dde/stress_test_(thread).html#fail1
Verified locally by commenting out->next() call in
WriteBufferFromHTTPServerResponse::nextImpl(), adding a sleep(1) and
canceling HTTP request before it finished, the stacktrace as follow:
[ 6351 ] {} <Fatal> BaseDaemon: (version 21.3.1.1, build id: 9B40466BF3D2F5AED78A52A995A4A2FD3116787C) (from thread 6677) Terminate called for uncaught exception:
Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 4, Stack trace (when copying this message, always include the lines below):
0. /src/ch/clickhouse/.cmake/../src/Common/StackTrace.cpp:298: StackTrace::tryCapture() @ 0x30a52a in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
1. /src/ch/clickhouse/.cmake/../src/Common/StackTrace.cpp:260: StackTrace::StackTrace() @ 0x30a4e5 in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
2. /src/ch/clickhouse/.cmake/../src/Common/Exception.cpp:53: DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) @ 0x2a61ae in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
3. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:22: DB::WriteBufferFromOStream::nextImpl() @ 0x3b468a in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
4. /src/ch/clickhouse/.cmake/../src/IO/WriteBuffer.h:47
<snip>
[ 8966 ] {} <Fatal> BaseDaemon: 7. __cxxabiv1::__terminate(void (*)()) @ 0x1784ca in /src/ch/clickhouse/.cmake/contrib/replxx-cmake/libreplxxd.so
<snip>
[ 8966 ] {} <Fatal> BaseDaemon: 10. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x3b48c1 in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
[ 8966 ] {} <Fatal> BaseDaemon: 11. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:44: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x3b48ec in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
<snip>
[ 8966 ] {} <Fatal> BaseDaemon: 14. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromHTTPServerResponse.cpp:218: DB::WriteBufferFromHTTPServerResponse::~WriteBufferFromHTTPServerResponse() @ 0x3b33cd in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
<snip>
[ 8966 ] {} <Fatal> BaseDaemon: 22. /src/ch/clickhouse/.cmake/../src/Server/HTTPHandler.h:43: DB::HTTPHandler::Output::~Output() @ 0x260421 in /src/ch/clickhouse/.cmake/src/libclickhouse_serverd.so
[ 8966 ] {} <Fatal> BaseDaemon: 23. /src/ch/clickhouse/.cmake/../src/Server/HTTPHandler.cpp:778: DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) @ 0x253fd4 in /src/ch/clickhouse/.cmake/src/libclickhouse_serverd.so
2021-02-13 13:23:03 +03:00
Azat Khuzhin
fa329808e5
Call next() from sync()/finalize() in WriteBuffer
2021-02-13 13:14:11 +03:00
alexey-milovidov
dc3ffd3fe2
Merge pull request #19451 from azat/safe-writes
...
Do not silently ignore write errors
2021-02-11 21:19:11 +03:00
Nikita Mikhailov
6c9f5e4991
try
2021-02-10 17:16:27 +03:00
alesapin
cb1194d37a
Add final to some classes
2021-02-09 12:05:54 +03:00
alexey-milovidov
ed5a81ab59
Merge pull request #20078 from abyss7/better-read-buffers-4
...
LimitReadBuffer: check that position always advances
2021-02-07 07:33:08 +03:00
alexey-milovidov
f2ab4f2ce9
Merge pull request #20037 from abyss7/better-read-buffers-2
...
ReadBuffer: check for unread data on next()
2021-02-07 01:33:06 +03:00
alexey-milovidov
417cfcd698
Update ReadBuffer.h
2021-02-07 01:31:20 +03:00
Ivan Lezhankin
8953fe1eb2
Fix seekable buffer
2021-02-06 19:30:46 +03:00
Pavel Kruglov
f946aab759
Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests
2021-02-06 17:38:56 +03:00
Pavel Kruglov
0704d3cf27
Refactor
2021-02-06 03:54:27 +03:00
alesapin
011109c82a
Merge pull request #17348 from xjewer/alex/CLICKHOUSE-606_deduplication_UUID
...
CLICKHOUSE-606: query deduplication based on parts' UUID
2021-02-05 22:47:34 +03:00
Ivan Lezhankin
16d399aa35
Another build fix
2021-02-05 13:31:18 +03:00
Ivan
d135a727e2
Merge branch 'master' into better-read-buffers-2
2021-02-05 02:17:05 +03:00
Ivan Lezhankin
d59b45e496
Fix build
2021-02-05 02:14:17 +03:00
Azat Khuzhin
98e3a99a88
Do not catch exceptions during final flush in writers destructors
...
Since this hides real problems, since destructor does final flush and if
it fails, then data will be lost.
One of such examples if MEMORY_LIMIT_EXCEEDED exception, so lock
exceptions from destructors, by using
MemoryTracker::LockExceptionInThread to block these exception, and allow
others (so std::terminate will be called, since this is c++11 with
noexcept for destructors by default).
Here is an example, that leads to empty block in the distributed batch:
2021.01.21 12:43:18.619739 [ 46468 ] {7bd60d75-ebcb-45d2-874d-260df9a4ddac} <Error> virtual DB::CompressedWriteBuffer::~CompressedWriteBuffer(): Code: 241, e.displayText() = DB::Exception: Memory limit (for user) exceeded: would use 332.07 GiB (attempt to allocate chunk of 4355342 bytes), maximum: 256.00 GiB, Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception<>() @ 0x86f7b88 in /usr/bin/clickhouse
...
4. void DB::PODArrayBase<>::resize<>(unsigned long) @ 0xe9e878d in /usr/bin/clickhouse
5. DB::CompressedWriteBuffer::nextImpl() @ 0xe9f0296 in /usr/bin/clickhouse
6. DB::CompressedWriteBuffer::~CompressedWriteBuffer() @ 0xe9f0415 in /usr/bin/clickhouse
7. DB::DistributedBlockOutputStream::writeToShard() @ 0xf6bed4a in /usr/bin/clickhouse
2021-02-05 01:31:45 +03:00
Ivan
0db2aa3c2e
Check that position doesn't go beyond end ( #20039 )
2021-02-05 01:21:24 +03:00
Ivan Lezhankin
84b88c6802
CC
2021-02-04 22:06:43 +03:00
Ivan Lezhankin
2b9909d396
Merge remote-tracking branch 'upstream/master' into better-read-buffers-2
2021-02-04 21:51:23 +03:00
Ivan Lezhankin
bcbe6f209c
Merge branch 'better-read-buffers-1' into better-read-buffers-2
2021-02-04 21:50:31 +03:00
Ivan Lezhankin
610d7b755b
Don't forget to update position of inner buffer
2021-02-04 21:50:15 +03:00
Ivan Lezhankin
ee0ff755e2
Check that position always advances
2021-02-04 17:46:46 +03:00
Azat Khuzhin
ca071381e1
Merge remote-tracking branch 'upstream/master' into writer-partial-buffer
2021-02-03 23:46:39 +03:00
Ivan Lezhankin
72b0a18503
Check for unread data on next()
2021-02-03 15:37:32 +03:00
Ivan Lezhankin
b0f400ca1b
Reset internal buffer position on next()
2021-02-03 15:31:45 +03:00
tavplubix
b24f73fd88
remove outdated comment
2021-02-03 13:27:41 +03:00
Aleksei Semiglazov
921518db0a
CLICKHOUSE-606: query deduplication based on parts' UUID
...
* add the query data deduplication excluding duplicated parts in MergeTree family engines.
query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1
allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.
data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.
NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.
* add _part_uuid virtual column, allowing to use UUIDs in predicates.
Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>
address comments
2021-02-02 16:53:39 +00:00
Azat Khuzhin
f3a56b965c
Add deebug assertion for empty working buffer in WriteBuffer
2021-02-02 01:56:00 +03:00
Azat Khuzhin
63a8f463f9
Allow writing partial buffer
...
In #6012 the behavior of the WriteBuffer has been changed.
Before, WriteBuffer write the data when next() is called.
After, WriteBuffer will write the data if, and only if, the buffer is
full.
See the discussion here [1].
[1]: https://github.com/ClickHouse/ClickHouse/pull/6012#discussion_r567320267
2021-02-02 00:23:55 +03:00
Alexey Milovidov
49979afbc1
Fix bad translation
2021-02-01 10:12:14 +03:00
Alexander Kuzmenkov
f42aefc4a4
Merge pull request #19750 from excitoon-favorites/s3betterglobs
...
Used global region for accessing S3 if can't determine exactly
2021-01-29 23:44:14 +03:00
Pavel Kruglov
25e85d71ee
Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests
2021-01-29 21:08:47 +03:00
Maksim Kita
03eec39719
Merge pull request #19712 from kitaisreal/added-functions-parse-date-time-best-effort-us-or-zero-or-null
...
Added functions parseDateTimeBestEffortUSOrZero, parseDateTimeBestEffortUSOrNull
2021-01-28 11:13:45 +03:00
Vladimir Chebotarev
579f8da573
Added SSE-C support in S3 client.
2021-01-28 09:32:41 +03:00
alexey-milovidov
7c240d0ca7
Merge pull request #17870 from excitoon-favorites/s3testiostream
...
Update of AWS C++ SDK
2021-01-28 07:12:52 +03:00
Vladimir Chebotarev
76e62739ed
Fixed build.
2021-01-27 23:05:41 +03:00
Maksim Kita
9db9ce11dc
Added functions parseDateTimeBestEffortUSOrZero, parseDateTimeBestEffortUSOrNull
2021-01-27 17:10:06 +03:00