thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-17 21:24:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
57,552 Commits 9,139 Branches 1,774 Tags 2.1 GiB
b726845be2
Commit Graph

470 Commits

Author SHA1 Message Date
Ivan Lezhankin
b726845be2 [WIP] 2021-03-04 14:10:21 +03:00
Ivan
414f470c79
Make Poco HTTP Server zero-copy again (#19516) 
* Refactoring: part 1

* Refactoring: part 2

* Handle request using ReadBuffer interface

* Struggles with ReadBuffer's

* Fix URI parsing

* Implement parsing of multipart/form-data

* Check HTTP_LENGTH_REQUIRED before eof() or will hang

* Fix HTTPChunkedReadBuffer

* Fix build and style

* Fix test

* Resist double-eof

* Fix arcadian build
 2021-02-19 15:51:26 +03:00
Nikita Mikhailov
6ee98ead05 Merge branch 'master' of github.com:ClickHouse/ClickHouse into keen-wolf-filefuncwithreadbuf 2021-02-16 15:38:03 +03:00
Nikita Mikhailov
7c5d845866 refactor function 2021-02-16 15:37:49 +03:00
Nikita Mikhaylov
cb2beb475e
Merge pull request #20516 from nikitamikhaylov/more-checks-filesegmentation-engine 
More checks in fileSegmentationEngine
 2021-02-16 12:39:27 +03:00
Nikita Mikhailov
d615b8e516 more checks 
(cherry picked from commit b45168ecaf37d0061edfd12c67a8c5300d45d2e3)
 2021-02-15 16:11:16 +03:00
alexey-milovidov
a299790522
Merge pull request #19748 from excitoon-favorites/s3ssec 
Added Server Side Encryption Customer Keys support in S3 client
 2021-02-15 15:23:14 +03:00
Nikita Mikhaylov
a77b740a7c
Merge pull request #20286 from nikitamikhaylov/json-import-bugfix 
Error from allocator on JSON import
 2021-02-15 12:40:09 +03:00
alesapin
9557bca4af
Merge pull request #20464 from azat/abnormal-server-termination-fix 
Fix abnormal server termination when http client goes away
 2021-02-15 11:10:29 +03:00
Azat Khuzhin
33f54cdb3c Fix abnormal server termination when http client goes away 
In [1] stress tests found:

    2021.02.12 14:20:58.800988 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Debug> executeQuery: (from [::1]:45792, using production parser) (comment: /usr/share/clickhouse-test/queries/0_stateless/01520_client_print_query_id.expect) SELECT * FROM numbers(34599)
    2021.02.12 14:20:58.916484 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Trace> ContextAccess (default): Access granted: CREATE TEMPORARY TABLE ON *.*
    2021.02.12 14:20:59.071980 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
    2021.02.12 14:21:10.708202 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Information> executeQuery: Read 34599 rows, 270.30 KiB in 11.876294055 sec., 2913 rows/sec., 22.76 KiB/sec.
    2021.02.12 14:22:10.506261 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Debug> DynamicQueryHandler: Done processing query
    2021.02.12 14:22:18.238037 [ 375 ] {} <Fatal> BaseDaemon: (version 21.3.1.5996, build id: 8DBCED54529C989F7AD4D991F51410774D55DE6C) (from thread 17728) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 262994, Stack trace (when copying this message, always include the lines below):

    0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15c976cb in /usr/bin/clickhouse
    1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8c9320e in /usr/bin/clickhouse
    2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8d54da5 in /usr/bin/clickhouse
    3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8d551d7 in /usr/bin/clickhouse
    4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::Write

    2021.02.12 14:22:18.811071 [ 18134 ] {} <Fatal> BaseDaemon: ########################################
    2021.02.12 14:22:18.878935 [ 18134 ] {} <Fatal> BaseDaemon: (version 21.3.1.5996, build id: 8DBCED54529C989F7AD4D991F51410774D55DE6C) (from thread 17728) (query_id: de3e7894-b401-4f7d-8530-90cd5ab06682) Received signal Aborted (6)
    2021.02.12 14:22:18.943148 [ 18134 ] {} <Fatal> BaseDaemon:
    2021.02.12 14:22:19.007073 [ 18134 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f109932018b 0x7f10992ff859 0x8bb33ae 0x8e301dd 0x17dac8c4 0x17dac7c7 0x8c3fe0b 0x8d552c5 0x8d552ea 0x11a29914 0x11a2a2ca 0x12f96092 0x12f8c65e 0x12f84300 0x15b84110 0x15bc0913 0x15bc103f 0x15d29a12 0x15d27fb0 0x15d267b8 0x8badbad 0x7f10994d5609 0x7f10993fc293
    2021.02.12 14:22:19.255998 [ 18134 ] {} <Fatal> BaseDaemon: 5. raise @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.02.12 14:22:19.270203 [ 18134 ] {} <Fatal> BaseDaemon: 6. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.02.12 14:22:50.108918 [ 370 ] {} <Fatal> Application: Child process was terminated by signal 6.

  [1]: https://clickhouse-test-reports.s3.yandex.net/19580/6aecb62416ece880cbb8ee3a803e14d841388dde/stress_test_(thread).html#fail1

Verified locally by commenting out->next() call in
WriteBufferFromHTTPServerResponse::nextImpl(), adding a sleep(1) and
canceling HTTP request before it finished, the stacktrace as follow:

    [ 6351 ] {} <Fatal> BaseDaemon: (version 21.3.1.1, build id: 9B40466BF3D2F5AED78A52A995A4A2FD3116787C) (from thread 6677) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 4, Stack trace (when copying this message, always include the lines below):

    0. /src/ch/clickhouse/.cmake/../src/Common/StackTrace.cpp:298: StackTrace::tryCapture() @ 0x30a52a in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    1. /src/ch/clickhouse/.cmake/../src/Common/StackTrace.cpp:260: StackTrace::StackTrace() @ 0x30a4e5 in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    2. /src/ch/clickhouse/.cmake/../src/Common/Exception.cpp:53: DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) @ 0x2a61ae in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    3. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:22: DB::WriteBufferFromOStream::nextImpl() @ 0x3b468a in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    4. /src/ch/clickhouse/.cmake/../src/IO/WriteBuffer.h:47

    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 7. __cxxabiv1::__terminate(void (*)()) @ 0x1784ca in /src/ch/clickhouse/.cmake/contrib/replxx-cmake/libreplxxd.so
    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 10. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x3b48c1 in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    [ 8966 ] {} <Fatal> BaseDaemon: 11. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:44: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x3b48ec in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 14. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromHTTPServerResponse.cpp:218: DB::WriteBufferFromHTTPServerResponse::~WriteBufferFromHTTPServerResponse() @ 0x3b33cd in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 22. /src/ch/clickhouse/.cmake/../src/Server/HTTPHandler.h:43: DB::HTTPHandler::Output::~Output() @ 0x260421 in /src/ch/clickhouse/.cmake/src/libclickhouse_serverd.so
    [ 8966 ] {} <Fatal> BaseDaemon: 23. /src/ch/clickhouse/.cmake/../src/Server/HTTPHandler.cpp:778: DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) @ 0x253fd4 in /src/ch/clickhouse/.cmake/src/libclickhouse_serverd.so
 2021-02-13 13:23:03 +03:00
Azat Khuzhin
fa329808e5 Call next() from sync()/finalize() in WriteBuffer 2021-02-13 13:14:11 +03:00
alexey-milovidov
dc3ffd3fe2
Merge pull request #19451 from azat/safe-writes 
Do not silently ignore write errors
 2021-02-11 21:19:11 +03:00
Nikita Mikhailov
6c9f5e4991 try 2021-02-10 17:16:27 +03:00
alesapin
cb1194d37a Add final to some classes 2021-02-09 12:05:54 +03:00
alexey-milovidov
ed5a81ab59
Merge pull request #20078 from abyss7/better-read-buffers-4 
LimitReadBuffer: check that position always advances
 2021-02-07 07:33:08 +03:00
alexey-milovidov
f2ab4f2ce9
Merge pull request #20037 from abyss7/better-read-buffers-2 
ReadBuffer: check for unread data on next()
 2021-02-07 01:33:06 +03:00
alexey-milovidov
417cfcd698
Update ReadBuffer.h 2021-02-07 01:31:20 +03:00
Ivan Lezhankin
8953fe1eb2 Fix seekable buffer 2021-02-06 19:30:46 +03:00
alesapin
011109c82a
Merge pull request #17348 from xjewer/alex/CLICKHOUSE-606_deduplication_UUID 
CLICKHOUSE-606: query deduplication based on parts' UUID
 2021-02-05 22:47:34 +03:00
Ivan Lezhankin
16d399aa35 Another build fix 2021-02-05 13:31:18 +03:00
Ivan
d135a727e2
Merge branch 'master' into better-read-buffers-2 2021-02-05 02:17:05 +03:00
Ivan Lezhankin
d59b45e496 Fix build 2021-02-05 02:14:17 +03:00
Azat Khuzhin
98e3a99a88 Do not catch exceptions during final flush in writers destructors 
Since this hides real problems, since destructor does final flush and if
it fails, then data will be lost.

One of such examples if MEMORY_LIMIT_EXCEEDED exception, so lock
exceptions from destructors, by using
MemoryTracker::LockExceptionInThread to block these exception, and allow
others (so std::terminate will be called, since this is c++11 with
noexcept for destructors by default).

Here is an example, that leads to empty block in the distributed batch:

    2021.01.21 12:43:18.619739 [ 46468 ] {7bd60d75-ebcb-45d2-874d-260df9a4ddac} <Error> virtual DB::CompressedWriteBuffer::~CompressedWriteBuffer(): Code: 241, e.displayText() = DB::Exception: Memory limit (for user) exceeded: would use 332.07 GiB (attempt to allocate chunk of 4355342 bytes), maximum: 256.00 GiB, Stack trace (when copying this message, always include the lines below):

    0. DB::Exception::Exception<>() @ 0x86f7b88 in /usr/bin/clickhouse
    ...
    4. void DB::PODArrayBase<>::resize<>(unsigned long) @ 0xe9e878d in /usr/bin/clickhouse
    5. DB::CompressedWriteBuffer::nextImpl() @ 0xe9f0296 in /usr/bin/clickhouse
    6. DB::CompressedWriteBuffer::~CompressedWriteBuffer() @ 0xe9f0415 in /usr/bin/clickhouse
    7. DB::DistributedBlockOutputStream::writeToShard() @ 0xf6bed4a in /usr/bin/clickhouse
 2021-02-05 01:31:45 +03:00
Ivan
0db2aa3c2e
Check that position doesn't go beyond end (#20039) 2021-02-05 01:21:24 +03:00
Ivan Lezhankin
84b88c6802 CC 2021-02-04 22:06:43 +03:00
Ivan Lezhankin
2b9909d396 Merge remote-tracking branch 'upstream/master' into better-read-buffers-2 2021-02-04 21:51:23 +03:00
Ivan Lezhankin
bcbe6f209c Merge branch 'better-read-buffers-1' into better-read-buffers-2 2021-02-04 21:50:31 +03:00
Ivan Lezhankin
610d7b755b Don't forget to update position of inner buffer 2021-02-04 21:50:15 +03:00
Ivan Lezhankin
ee0ff755e2 Check that position always advances 2021-02-04 17:46:46 +03:00
Azat Khuzhin
ca071381e1 Merge remote-tracking branch 'upstream/master' into writer-partial-buffer 2021-02-03 23:46:39 +03:00
Ivan Lezhankin
72b0a18503 Check for unread data on next() 2021-02-03 15:37:32 +03:00
Ivan Lezhankin
b0f400ca1b Reset internal buffer position on next() 2021-02-03 15:31:45 +03:00
tavplubix
b24f73fd88
remove outdated comment 2021-02-03 13:27:41 +03:00
Aleksei Semiglazov
921518db0a CLICKHOUSE-606: query deduplication based on parts' UUID 
* add the query data deduplication excluding duplicated parts in MergeTree family engines.

query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1

allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.

data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.

NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.

* add _part_uuid virtual column, allowing to use UUIDs in predicates.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>

address comments
 2021-02-02 16:53:39 +00:00
Azat Khuzhin
f3a56b965c Add deebug assertion for empty working buffer in WriteBuffer 2021-02-02 01:56:00 +03:00
Azat Khuzhin
63a8f463f9 Allow writing partial buffer 
In #6012 the behavior of the WriteBuffer has been changed.

Before, WriteBuffer write the data when next() is called.
After, WriteBuffer will write the data if, and only if, the buffer is
full.

See the discussion here [1].

  [1]: https://github.com/ClickHouse/ClickHouse/pull/6012#discussion_r567320267
 2021-02-02 00:23:55 +03:00
Alexey Milovidov
49979afbc1 Fix bad translation 2021-02-01 10:12:14 +03:00
Alexander Kuzmenkov
f42aefc4a4
Merge pull request #19750 from excitoon-favorites/s3betterglobs 
Used global region for accessing S3 if can't determine exactly
 2021-01-29 23:44:14 +03:00
Maksim Kita
03eec39719
Merge pull request #19712 from kitaisreal/added-functions-parse-date-time-best-effort-us-or-zero-or-null 
Added functions parseDateTimeBestEffortUSOrZero, parseDateTimeBestEffortUSOrNull
 2021-01-28 11:13:45 +03:00
Vladimir Chebotarev
579f8da573 Added SSE-C support in S3 client. 2021-01-28 09:32:41 +03:00
alexey-milovidov
7c240d0ca7
Merge pull request #17870 from excitoon-favorites/s3testiostream 
Update of AWS C++ SDK
 2021-01-28 07:12:52 +03:00
Vladimir Chebotarev
76e62739ed Fixed build. 2021-01-27 23:05:41 +03:00
Maksim Kita
9db9ce11dc Added functions parseDateTimeBestEffortUSOrZero, parseDateTimeBestEffortUSOrNull 2021-01-27 17:10:06 +03:00
Vladimir Chebotarev
afdea5eff0 Changed handling of PocoHTTPClientConfiguration in order to ensure AWS SDK is initialized. 2021-01-27 09:38:35 +03:00
Alexey Milovidov
877be11398 Remove useless code, part 2 2021-01-26 21:32:46 +03:00
Vladimir Chebotarev
3322cc812d Updated AWS C++ SDK. 2021-01-26 20:58:02 +03:00
alexey-milovidov
3fffa16aab
Update src/IO/HTTPCommon.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2021-01-23 00:43:02 +03:00
Azat Khuzhin
d1d82bc897 Fix build with unbundled poco 
Refs: #13405
Cc: @excitoon
 2021-01-22 21:35:48 +03:00
tavplubix
49e1321df4
Merge pull request #19072 from ClickHouse/fix_19021 
Fix checkpoint in PeekableReadBuffer over ConcatReadBuffer
 2021-01-22 21:00:04 +03:00
Azat Khuzhin
e4350e078c Add ability to distinguish remote exceptions from local 2021-01-20 01:10:17 +03:00