thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-18 05:32:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
63,047 Commits 9,136 Branches 1,774 Tags 2 GiB
ee06936596
Commit Graph

400 Commits

Author SHA1 Message Date
Ivan Lezhankin
d9d143675a Merge remote-tracking branch 'upstream/master' into issue-21907 2021-04-01 16:29:26 +03:00
Ivan Lezhankin
19550d5173 Initialize pointer with nullptr 2021-04-01 16:26:13 +03:00
Ivan Lezhankin
cc1c439ad8 Merge remote-tracking branch 'upstream/master' into poco-http2 2021-04-01 16:20:15 +03:00
Alexey Milovidov
9d65d83c83 Better exception message in client in case of exception while writing blocks 2021-04-01 07:12:07 +03:00
Ivan Lezhankin
ae741c7411 Do not limit HTTP chunk size 2021-03-29 20:49:28 +03:00
alesapin
e4a028c4d4 Missed ya.make 2021-03-29 12:25:40 +03:00
alesapin
be132a32a2 More renames 2021-03-29 11:24:56 +03:00
alesapin
ced6d8e6bd
Merge pull request #21677 from ClickHouse/jepsen_for_nukeeper 
Jepsen for nukeeper
 2021-03-27 10:18:42 +03:00
alexey-milovidov
ac53d30e53
Merge pull request #22146 from azat/writers-uncaught-exceptions 
Fix uncaught exception in InterserverIOHTTPHandler
 2021-03-26 17:23:09 +03:00
tavplubix
d9ec4090d2
Merge pull request #21936 from abyss7/html-form-fix 
Fix issues with HTMLForm::MultipartReadBuffer and PeekableReadBuffer
 2021-03-26 15:35:34 +03:00
Azat Khuzhin
e1de960025 Fix uncaught exception in InterserverIOHTTPHandler 
There was one more uncaught exception case [1]:

    2021.03.19 18:11:00.845632 [ 17469 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
    ...
    2021.03.19 18:11:31.698961 [ 80145 ] {} <Fatal> BaseDaemon: ########################################
    2021.03.19 18:11:31.699903 [ 80145 ] {} <Fatal> BaseDaemon: (version 21.4.1.6293 (official build), build id: 859E400E1C65C4702FE491420741DD8B58190002) (from thread 17469) (no query) Received signal Aborted (6)
    2021.03.19 18:11:32.614075 [ 80145 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../contrib/libcxxabi/src/cxa_handlers.cpp:89: std::terminate() @ 0x21e9b3a2 in /usr/bin/clickhouse
    2021.03.19 18:11:43.831215 [ 80145 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/Server/HTTP/WriteBufferFromHTTPServerResponse.cpp:201: ? @ 0x1be77038 in /usr/bin/clickhouse
    2021.03.19 18:11:44.743193 [ 80145 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:892: std::__1::allocator<DB::WriteBufferFromHTTPServerResponse>::destroy(DB::WriteBufferFromHTTPServerResponse*) @ 0x1bddd7c9 in /usr/bin/clickhouse
    2021.03.19 18:11:45.283905 [ 80145 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:541: void std::__1::allocator_traits<std::__1::allocator<DB::WriteBufferFromHTTPServerResponse> >::__destroy<DB::WriteBufferFromHTTPServerResponse>(std::__1::integral_constant<bool, true>, std::__1::allocator<DB::WriteBufferFromHTTPServerResponse>&, DB::WriteBufferFromHTTPServerResponse*) @ 0x1bddd79d in /usr/bin/clickhouse
    2021.03.19 18:11:45.805233 [ 80145 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:487: void std::__1::allocator_traits<std::__1::allocator<DB::WriteBufferFromHTTPServerResponse> >::destroy<DB::WriteBufferFromHTTPServerResponse>(std::__1::allocator<DB::WriteBufferFromHTTPServerResponse>&, DB::WriteBufferFromHTTPServerResponse*) @ 0x1bddd76d in /usr/bin/clickhouse
    2021.03.19 18:11:46.351371 [ 80145 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2611: std::__1::__shared_ptr_emplace<DB::WriteBufferFromHTTPServerResponse, std::__1::allocator<DB::WriteBufferFromHTTPServerResponse> >::__on_zero_shared() @ 0x1bddd525 in /usr/bin/clickhouse
    2021.03.19 18:11:46.579263 [ 80145 ] {} <Fatal> BaseDaemon: 15. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2476: std::__1::__shared_count::__release_shared() @ 0x119490ed in /usr/bin/clickhouse
    2021.03.19 18:11:46.790912 [ 80145 ] {} <Fatal> BaseDaemon: 16. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2517: std::__1::__shared_weak_count::__release_shared() @ 0x1194908f in /usr/bin/clickhouse
    2021.03.19 18:11:47.277990 [ 80145 ] {} <Fatal> BaseDaemon: 17. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3213: std::__1::shared_ptr<DB::WriteBufferFromHTTPServerResponse>::~shared_ptr() @ 0x1bdd75fc in /usr/bin/clickhouse
    2021.03.19 18:11:47.649213 [ 80145 ] {} <Fatal> BaseDaemon: 18. ./obj-x86_64-linux-gnu/../src/Server/InterserverIOHTTPHandler.h:34: DB::InterserverIOHTTPHandler::Output::~Output() @ 0x1bdf6bd5 in /usr/bin/clickhouse
    2021.03.19 18:11:47.921556 [ 80145 ] {} <Fatal> BaseDaemon: 19. ./obj-x86_64-linux-gnu/../src/Server/InterserverIOHTTPHandler.cpp:154: DB::InterserverIOHTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x1bdf653f in /usr/bin/clickhouse

  [1]: https://clickhouse-test-reports.s3.yandex.net/0/78c56b891383288cf3a893139e796fc87476412e/stress_test_(debug).html

Since in case of no errors during processing we should call finalize, to
ensure that it will not be called from dtor.

Fixes: #22046
Fixes: #22067
 2021-03-25 23:41:03 +03:00
Azat Khuzhin
48fe30e5a2 Add missing logging for exception in InterserverIOHTTPHandler 2021-03-25 23:41:03 +03:00
Kruglov Pavel
66b8431e71
Merge pull request #21429 from Avogar/secure-socket 
Fix receive and send timeouts and non-blocking read in secure socket
 2021-03-25 17:30:17 +03:00
alesapin
4c28c25a75 Merge branch 'master' into jepsen_for_nukeeper 2021-03-25 14:34:00 +03:00
Pavel Kruglov
6278a7efac Fix build 2021-03-23 00:21:52 +03:00
Pavel Kruglov
dbc5018000 Update tests for hedged requests 2021-03-22 22:18:06 +03:00
alesapin
27b90007aa Merge branch 'master' into jepsen_for_nukeeper 2021-03-22 11:20:00 +03:00
Ivan Lezhankin
957c053f7e Fix segfault 2021-03-19 23:29:01 +03:00
Nikita Mikhaylov
2e13c767a0
Merge pull request #21149 from nikitamikhaylov/pp-for-csv-with-names 
Support parallel parsing for *WithNames types
 2021-03-16 14:38:03 +03:00
Nikita Mikhaylov
c92d8c776c try to avoid race in grpc server 2021-03-15 15:14:58 +03:00
alesapin
42ccee286a Merge branch 'master' into jepsen_for_nukeeper 2021-03-15 14:17:13 +03:00
alesapin
ba9e1e5a8d Some initial code 
Add some java magic

Allow to connect with old session id

More angry nemesis and fixes

Angry

Fix style

Split to files

Better wrappers

Better structure

Add set test and split to separate files (I think something broken now)

Better

Missed files
 2021-03-12 22:07:57 +03:00
Denis Glazachev
290a6d273e
Add Kerberos support for authenticating existing users when accessing over HTTP 2021-03-11 23:41:10 +03:00
alesapin
bcd073278d
Merge pull request #21425 from ClickHouse/persistent_nukeeper_snapshot_storage 
Persistent nukeeper snapshot storage
 2021-03-09 18:37:23 +03:00
zhangxiao871
846f5a93a9 Fix incorrect fd_ready assignment in NuKeeperTCPHandler. 2021-03-09 17:42:27 +08:00
alesapin
a8b7469857 Fix session timeout update 2021-03-06 17:14:38 +03:00
Ivan Lezhankin
6a07ee1506 CC 2021-03-04 17:01:37 +03:00
Ivan Lezhankin
91ff08452e Remove unused code 2021-03-04 16:58:18 +03:00
Ivan Lezhankin
a3509065b2 Add trivial README about Poco-based code 2021-03-04 15:43:00 +03:00
Ivan Lezhankin
5cfeedb1e0 Add setting http_max_uri_size 2021-03-04 15:21:23 +03:00
Pavel Kruglov
1c7f16e0ff Add test and move TimeoutSetter in IO/ 2021-03-03 20:47:27 +03:00
Nikolai Kochetov
cb122160ae
Merge pull request #19291 from Avogar/hedged-requests 
Implementation of HedgedRequests
 2021-03-03 12:58:37 +03:00
Pavel Kruglov
4166ae1f45 Restart timeout when we make progress 2021-03-02 15:40:24 +03:00
Amos Bird
ad4fd75fb4
mycli compatibility 2021-03-02 18:53:06 +08:00
Pavel Kruglov
153bfbfc28 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-03-02 11:59:32 +03:00
Azat Khuzhin
eb0387c5a9 Fix abnormal server termination for nested writers 
Writers with nested writer can call next() from the dtor for nested
writer and this will cause exception again, so the buffer position
should be updated on exceptions.

Found by stress test (thread) here [1] and here [2]:

    2021.02.27 19:27:53.498977 [ 302 ] {} <Fatal> BaseDaemon: (version 21.3.1.6130, build id: 2DAEC5DEBF03C5A1C3BF66B7779C886F16239345) (from thread 1614) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 2097498, Stack trace (when copying this message, always include the lines below):

    0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15bef2ab in /usr/bin/clickhouse
    1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8aea92e in /usr/bin/clickhouse
    2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8bbbc45 in /usr/bin/clickhouse
    3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8bbc077 in /usr/bin/clickhouse
    4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8bbc18a in /usr/bin/clickhouse
    5. ./obj-x86_64-linux-gnu/../src/IO/BufferWithOwnMemory.h:137: DB::ZstdDeflatingWriteBuffer::~ZstdDeflatingWriteBuffer() @ 0x118bdc29 in /usr/bin/clickhouse
    6. ./obj-x86_64-linux-gnu/../src/IO/ZstdDeflatingWriteBuffer.cpp:32: DB::ZstdDeflatingWriteBuffer::~ZstdDeflatingWriteBuffer() @ 0x118be3ea in /usr/bin/clickhouse
    7. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:0: DB::WriteBufferFromHTTPServerResponse::finalize() @ 0x12f1dceb in /usr/bin/clickhouse
    8. ./obj-x86_64-linux-gnu/../src/Server/HTTPHandler.cpp:703: DB::HTTPHandler::trySendExceptionToClient(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x12e9fecc in /usr/bin/clickhouse
    9. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/string:1444: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x12ea0d60 in /usr/bin/clickhouse
    10. ./obj-x86_64-linux-gnu/../src/Server/HTTP/HTTPServerConnection.cpp:0: DB::HTTPServerConnection::run() @ 0x12f16db1 in /usr/bin/clickhouse
    11. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x15b184f3 in /usr/bin/clickhouse
    12. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:0: Poco::Net::TCPServerDispatcher::run() @ 0x15b18c1f in /usr/bin/clickhouse
    13. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/ScopedLock.h:36: Poco::PooledThread::run() @ 0x15c7fdb2 in /usr/bin/clickhouse
    14. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x15c7e350 in /usr/bin/clickhouse
    15. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/SharedPtr.h:277: Poco::ThreadImpl::runnableEntry(void*) @ 0x15c7cb58 in /usr/bin/clickhouse
    16. __tsan_thread_start_func @ 0x8a04ced in /usr/bin/clickhouse
    17. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    18. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
     (version 21.3.1.6130)

  [1]: https://clickhouse-test-reports.s3.yandex.net/21279/4f61ef3099f42f17b496a0b0424773978d9a32dc/stress_test_(thread).html#fail1
  [2]: https://clickhouse-test-reports.s3.yandex.net/21292/ae9fea1d0af118a8f87b224d194d61da1567188b/stress_test_(thread).html#fail1

v2: https://clickhouse-test-reports.s3.yandex.net/21305/e969daa6e86c5e09cfef08cfde19712982b64e59/stress_test_(thread).html#fail1
 2021-02-28 15:13:42 +03:00
Pavel Kruglov
197058dc12 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-02-27 00:36:33 +03:00
alesapin
a1901383ae Improve performance and fix condition variable wait 2021-02-24 14:54:14 +03:00
alesapin
5b888e1193 Merge branch 'master' into persistent_nukeeper_log_storage 2021-02-22 23:52:26 +03:00
Kruglov Pavel
815b4ff0d9
Merge branch 'master' into hedged-requests 2021-02-21 17:10:24 +03:00
alexey-milovidov
20a7ccf5b1
Merge pull request #20174 from amosbird/alias 
Normalized function names (lower-case case-insensitive names)
 2021-02-21 16:23:51 +03:00
Amos Bird
f37631830f
Comments 2021-02-20 16:45:25 +08:00
alesapin
c61769311e Merge branch 'master' into persistent_nukeeper_log_storage 2021-02-20 10:22:22 +03:00
Azat Khuzhin
1ccb333ac5 Fix WriteBufferFromHTTPServerResponse usage in other places (add missing finalize()) 
Since I saw the following:

    0. DB::WriteBufferFromOStream::nextImpl()
    1. DB::WriteBufferFromHTTPServerResponse::nextImpl()
    2. DB::WriteBufferFromHTTPServerResponse::finalize()
    3. DB::WriteBufferFromHTTPServerResponse::~WriteBufferFromHTTPServerResponse()
    4. DB::StaticRequestHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)
    5. Poco::Net::HTTPServerConnection::run()
    6. Poco::Net::TCPServerConnection::start()
 2021-02-20 10:15:38 +03:00
Azat Khuzhin
fba1c7fcc1 Fix uncaught exception when HTTP client goes away 
Even after #20464 it was still possible, for example [1].

    2021.02.19 11:40:21.886191 [ 68373 ] {} <Trace> DynamicQueryHandler: Request URI: /?database=test_ds2d6y&log_comment=/usr/share/clickhouse-test/queries/0_stateless/01302_aggregate_state_exception_memory_leak.sh&enable_http_compression=1&http_zlib_compression_level=1

    <snip>

    2021.02.19 11:41:35.289940 [ 365 ] {} <Fatal> BaseDaemon: (version 21.3.1.6058, build id: 8D46D65205E2C8B7FE408A0B4EC76CA0483F9E92) (from thread 68373) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 262568, Stack trace (when copying this message, always include the lines below):

    0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15b3c7db in /usr/bin/clickhouse
    1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8aba66e in /usr/bin/clickhouse
    2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8b8c105 in /usr/bin/clickhouse
    3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8b8c537 in /usr/bin/clickhouse
    4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::Write

  [1]: https://clickhouse-test-reports.s3.yandex.net/16481/5d150cce4778dd14f58dcff67435bdec1efa155b/stress_test_(thread).html#fail1

And according to this partial stacktrace it seems that the dtor of
WriteBufferFromOStream was called from
WriteBufferFromHTTPServerResponse, since the class name starts from
DB::Write*

The problem is that if first time WriteBufferFromOStream::next() fails,
it will reset position to make next write no-op, however
WriteBufferFromHTTPServerResponse::next() will set position to available
buffer back, and next() will throw again, but this time it can be from
dtor.
 2021-02-19 22:00:57 +03:00
tavplubix
3509fe88d8
Merge pull request #20878 from ClickHouse/fix_18926 
Fix COM_FIELD_LIST response in MySQL protocol
 2021-02-19 19:25:54 +03:00
alesapin
5c9420c077 More correct epoll usage 2021-02-19 15:56:51 +03:00
Ivan
414f470c79
Make Poco HTTP Server zero-copy again (#19516) 
* Refactoring: part 1

* Refactoring: part 2

* Handle request using ReadBuffer interface

* Struggles with ReadBuffer's

* Fix URI parsing

* Implement parsing of multipart/form-data

* Check HTTP_LENGTH_REQUIRED before eof() or will hang

* Fix HTTPChunkedReadBuffer

* Fix build and style

* Fix test

* Resist double-eof

* Fix arcadian build
 2021-02-19 15:51:26 +03:00
alesapin
3d954c4314 Better request/response logic 2021-02-19 14:06:17 +03:00
alesapin
6c9322bb2e Sane constant while reading requests 2021-02-19 12:57:39 +03:00
Amos Bird
fc185e5fb7
Another try 2021-02-19 11:56:24 +08:00
Alexander Tokmakov
97f4c457ec fix MySQL COMM_FIELD_LIST response 2021-02-18 16:27:51 +03:00
Amos Bird
2c4bc43014
Backward compatible 2021-02-18 20:20:29 +08:00
Pavel Kruglov
1b78de2142 Use fibers in HedgedRequests 2021-02-17 20:34:52 +03:00
Kruglov Pavel
598576ce70
Merge branch 'master' into hedged-requests 2021-02-15 16:51:45 +03:00
Azat Khuzhin
33f54cdb3c Fix abnormal server termination when http client goes away 
In [1] stress tests found:

    2021.02.12 14:20:58.800988 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Debug> executeQuery: (from [::1]:45792, using production parser) (comment: /usr/share/clickhouse-test/queries/0_stateless/01520_client_print_query_id.expect) SELECT * FROM numbers(34599)
    2021.02.12 14:20:58.916484 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Trace> ContextAccess (default): Access granted: CREATE TEMPORARY TABLE ON *.*
    2021.02.12 14:20:59.071980 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
    2021.02.12 14:21:10.708202 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Information> executeQuery: Read 34599 rows, 270.30 KiB in 11.876294055 sec., 2913 rows/sec., 22.76 KiB/sec.
    2021.02.12 14:22:10.506261 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Debug> DynamicQueryHandler: Done processing query
    2021.02.12 14:22:18.238037 [ 375 ] {} <Fatal> BaseDaemon: (version 21.3.1.5996, build id: 8DBCED54529C989F7AD4D991F51410774D55DE6C) (from thread 17728) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 262994, Stack trace (when copying this message, always include the lines below):

    0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15c976cb in /usr/bin/clickhouse
    1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8c9320e in /usr/bin/clickhouse
    2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8d54da5 in /usr/bin/clickhouse
    3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8d551d7 in /usr/bin/clickhouse
    4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::Write

    2021.02.12 14:22:18.811071 [ 18134 ] {} <Fatal> BaseDaemon: ########################################
    2021.02.12 14:22:18.878935 [ 18134 ] {} <Fatal> BaseDaemon: (version 21.3.1.5996, build id: 8DBCED54529C989F7AD4D991F51410774D55DE6C) (from thread 17728) (query_id: de3e7894-b401-4f7d-8530-90cd5ab06682) Received signal Aborted (6)
    2021.02.12 14:22:18.943148 [ 18134 ] {} <Fatal> BaseDaemon:
    2021.02.12 14:22:19.007073 [ 18134 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f109932018b 0x7f10992ff859 0x8bb33ae 0x8e301dd 0x17dac8c4 0x17dac7c7 0x8c3fe0b 0x8d552c5 0x8d552ea 0x11a29914 0x11a2a2ca 0x12f96092 0x12f8c65e 0x12f84300 0x15b84110 0x15bc0913 0x15bc103f 0x15d29a12 0x15d27fb0 0x15d267b8 0x8badbad 0x7f10994d5609 0x7f10993fc293
    2021.02.12 14:22:19.255998 [ 18134 ] {} <Fatal> BaseDaemon: 5. raise @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.02.12 14:22:19.270203 [ 18134 ] {} <Fatal> BaseDaemon: 6. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.02.12 14:22:50.108918 [ 370 ] {} <Fatal> Application: Child process was terminated by signal 6.

  [1]: https://clickhouse-test-reports.s3.yandex.net/19580/6aecb62416ece880cbb8ee3a803e14d841388dde/stress_test_(thread).html#fail1

Verified locally by commenting out->next() call in
WriteBufferFromHTTPServerResponse::nextImpl(), adding a sleep(1) and
canceling HTTP request before it finished, the stacktrace as follow:

    [ 6351 ] {} <Fatal> BaseDaemon: (version 21.3.1.1, build id: 9B40466BF3D2F5AED78A52A995A4A2FD3116787C) (from thread 6677) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 4, Stack trace (when copying this message, always include the lines below):

    0. /src/ch/clickhouse/.cmake/../src/Common/StackTrace.cpp:298: StackTrace::tryCapture() @ 0x30a52a in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    1. /src/ch/clickhouse/.cmake/../src/Common/StackTrace.cpp:260: StackTrace::StackTrace() @ 0x30a4e5 in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    2. /src/ch/clickhouse/.cmake/../src/Common/Exception.cpp:53: DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) @ 0x2a61ae in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    3. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:22: DB::WriteBufferFromOStream::nextImpl() @ 0x3b468a in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    4. /src/ch/clickhouse/.cmake/../src/IO/WriteBuffer.h:47

    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 7. __cxxabiv1::__terminate(void (*)()) @ 0x1784ca in /src/ch/clickhouse/.cmake/contrib/replxx-cmake/libreplxxd.so
    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 10. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x3b48c1 in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    [ 8966 ] {} <Fatal> BaseDaemon: 11. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:44: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x3b48ec in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 14. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromHTTPServerResponse.cpp:218: DB::WriteBufferFromHTTPServerResponse::~WriteBufferFromHTTPServerResponse() @ 0x3b33cd in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 22. /src/ch/clickhouse/.cmake/../src/Server/HTTPHandler.h:43: DB::HTTPHandler::Output::~Output() @ 0x260421 in /src/ch/clickhouse/.cmake/src/libclickhouse_serverd.so
    [ 8966 ] {} <Fatal> BaseDaemon: 23. /src/ch/clickhouse/.cmake/../src/Server/HTTPHandler.cpp:778: DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) @ 0x253fd4 in /src/ch/clickhouse/.cmake/src/libclickhouse_serverd.so
 2021-02-13 13:23:03 +03:00
Azat Khuzhin
592f62d0af Remove superfluous out->next() call in HTTPHandler 2021-02-13 13:22:37 +03:00
alesapin
5f32f65211 Merge branch 'master' into in_memory_raft 2021-02-11 23:09:29 +03:00
alexey-milovidov
dc3ffd3fe2
Merge pull request #19451 from azat/safe-writes 
Do not silently ignore write errors
 2021-02-11 21:19:11 +03:00
Alexey Milovidov
dbb41ce4e2 Remove debug output 2021-02-11 02:55:52 +03:00
Alexey Milovidov
f442b30f30 Fix test 2021-02-11 01:23:27 +03:00
Alexey Milovidov
d3dba0e52a Merge branch 'master' into no-excessive-squash-memory 2021-02-10 23:45:12 +03:00
Alexey Milovidov
30b2554fa3 Fix error 2021-02-10 20:48:39 +03:00
alesapin
9667bdcbd2 Merge branch 'master' into in_memory_raft 2021-02-09 16:06:31 +03:00
alesapin
b8baf3a443 Fix some warnings 2021-02-08 22:40:44 +03:00
tavplubix
ac477d9850
Merge pull request #19771 from ClickHouse/thread_state_improvements 
Minor code improvements around ThreadStatus
 2021-02-08 22:34:55 +03:00
alesapin
add89c17f2 Less serde in responses, watches on followers 2021-02-08 16:06:55 +03:00
tavplubix
060be5b2db
rerun CI checks 2021-02-08 15:04:40 +03:00
alesapin
d5ee7d33e9 Merge branch 'master' into in_memory_raft 2021-02-08 10:30:31 +03:00
Alexey Milovidov
869bca74a7 Fix some tests 2021-02-07 23:37:55 +03:00
alexey-milovidov
f2ab4f2ce9
Merge pull request #20037 from abyss7/better-read-buffers-2 
ReadBuffer: check for unread data on next()
 2021-02-07 01:33:06 +03:00
Pavel Kruglov
f946aab759 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-02-06 17:38:56 +03:00
alesapin
449e8e3fd9 More checks for setting 2021-02-05 13:15:02 +03:00
Azat Khuzhin
64c0bf9829 TCPHandler: catch exceptions from the WriteBuffer in destructor 
For TCPHandler it is safe thing todo.

Otherwise *San will report [1]:

    2021.01.24 15:33:40.103996 [ 270 ] {} <Trace> BaseDaemon: Received signal -1
    2021.01.24 15:33:40.110693 [ 270 ] {} <Fatal> BaseDaemon: (version 21.2.1.5789, build id: FF421B087D1E2EAA19FA17B5AB3AE413832744E0) (from thread 48318) Terminate called for uncaught exception:
    2021.01.24 15:33:40.114845 [ 270 ] {} <Trace> BaseDaemon: Received signal 6
    2021.01.24 15:33:40.138738 [ 218027 ] {} <Fatal> BaseDaemon: ########################################
    2021.01.24 15:33:40.138838 [ 218027 ] {} <Fatal> BaseDaemon: (version 21.2.1.5789, build id: FF421B087D1E2EAA19FA17B5AB3AE413832744E0) (from thread 48318) (no query) Received signal Aborted (6)
    2021.01.24 15:33:40.138912 [ 218027 ] {} <Fatal> BaseDaemon:
    2021.01.24 15:33:40.139277 [ 218027 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f185474118b 0x7f1854720859 0xaddc0cc 0x2af9fab8 0x2af9fa04 0xa91758b 0x1e418bb5 0x20725b4f 0x20725d9e 0x266b47a3 0x269772f5 0x26971847 0x7f18548f6609 0x7f185481d293
    2021.01.24 15:33:40.139637 [ 218027 ] {} <Fatal> BaseDaemon: 3. raise @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.01.24 15:33:40.140113 [ 218027 ] {} <Fatal> BaseDaemon: 4. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.01.24 15:33:40.144121 [ 218027 ] {} <Fatal> BaseDaemon: 5. ./obj-x86_64-linux-gnu/../base/daemon/BaseDaemon.cpp:0: terminate_handler() @ 0xaddc0cc in /usr/bin/clickhouse
    2021.01.24 15:33:40.151208 [ 218027 ] {} <Fatal> BaseDaemon: 6. ./obj-x86_64-linux-gnu/../contrib/libcxxabi/src/cxa_handlers.cpp:61: std::__terminate(void (*)()) @ 0x2af9fab8 in /usr/bin/clickhouse
    2021.01.24 15:33:40.153085 [ 218027 ] {} <Fatal> BaseDaemon: 7. ./obj-x86_64-linux-gnu/../contrib/libcxxabi/src/cxa_handlers.cpp:0: std::terminate() @ 0x2af9fa04 in /usr/bin/clickhouse
    2021.01.24 15:33:40.155209 [ 218027 ] {} <Fatal> BaseDaemon: 8. ? @ 0xa91758b in /usr/bin/clickhouse
    2021.01.24 15:33:40.156621 [ 218027 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::~WriteBufferFromPocoSocket() @ 0x1e418bb5 in /usr/bin/clickhouse
    2021.01.24 15:33:40.161041 [ 218027 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2518: DB::TCPHandler::~TCPHandler() @ 0x20725b4f in /usr/bin/clickhouse
    2021.01.24 15:33:40.164557 [ 218027 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.h:101: DB::TCPHandler::~TCPHandler() @ 0x20725d9e in /usr/bin/clickhouse
    2021.01.24 15:33:40.165921 [ 218027 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/AtomicCounter.h:314: Poco::Net::TCPServerDispatcher::run() @ 0x266b47a3 in /usr/bin/clickhouse
    2021.01.24 15:33:40.167347 [ 218027 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x269772f5 in /usr/bin/clickhouse
    2021.01.24 15:33:40.169401 [ 218027 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x26971847 in /usr/bin/clickhouse
    2021.01.24 15:33:40.169498 [ 218027 ] {} <Fatal> BaseDaemon: 15. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    2021.01.24 15:33:40.169566 [ 218027 ] {} <Fatal> BaseDaemon: 16. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.01.24 15:33:41.027601 [ 218027 ] {} <Fatal> BaseDaemon: Calculated checksum of the binary: 63D7491B39260494BA0D785E1860B427. There is no information about the reference checksum.

  [1]: https://clickhouse-test-reports.s3.yandex.net/19451/1e16bd6f337985a82fbdf4eded695dc6e663af58/stress_test_(address).html#fail1

v2: Fix catching errors in WriteBufferFromPocoSocket destructor
 2021-02-05 01:32:36 +03:00
Azat Khuzhin
4beb5c1b8a TCPHandler: Move constructor into the module and add missing headers 2021-02-05 01:31:42 +03:00
alesapin
c5312bf362 Trying to disable suspicious parameter 2021-02-04 22:29:46 +03:00
Ivan Lezhankin
ac4c3a6b27 Don't pass empty vector to ConcatReadBuffer 2021-02-04 18:59:54 +03:00
alesapin
da51ea1794 Simplify shutdown and requests processing 2021-02-04 15:07:41 +03:00
alesapin
1ff87ac6f9 Add background session lifetime control 2021-02-03 23:32:15 +03:00
Aleksei Semiglazov
921518db0a CLICKHOUSE-606: query deduplication based on parts' UUID 
* add the query data deduplication excluding duplicated parts in MergeTree family engines.

query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1

allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.

data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.

NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.

* add _part_uuid virtual column, allowing to use UUIDs in predicates.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>

address comments
 2021-02-02 16:53:39 +00:00
alesapin
9979181c02 Merge branch 'master' into in_memory_raft 2021-02-02 17:05:20 +03:00
alexey-milovidov
f5dc38a33c
Merge pull request #19584 from azat/client-complete-error-codes 
client: more suggestions
 2021-02-02 11:05:17 +03:00
Maksim Kita
d0151de4bb
Merge pull request #19608 from kreuzerkrieg/Add_IStoragePolicy_interface 
Add IStoragePolicy interface
 2021-02-02 11:03:20 +03:00
Azat Khuzhin
37797fdf5b Merge remote-tracking branch 'upstream/master' into client-complete-error-codes 2021-02-02 00:14:53 +03:00
alesapin
365bf65f5a Fix install script 2021-02-01 17:14:59 +03:00
alesapin
57c9b6c864 Fix build without nuraft 2021-02-01 16:18:17 +03:00
alesapin
0aca40d8cb Merge branch 'master' into in_memory_raft 2021-02-01 14:30:11 +03:00
alesapin
eb5c77f558 Fix some races and better reaction to leader change 2021-02-01 14:27:26 +03:00
alesapin
67412bd529 Don't accept connections if we don't see leader + race fix from upstream 2021-02-01 10:51:10 +03:00
Azat Khuzhin
978f23e343 Fix SIGSEGV due to accessing GRPCServer::currentConnections() before initialization 
This PR makes asynchronous metrics available just after start, and this
breaks GRPCServer, since it requires the server be started before
accessing currentConnections().

However it is possible to trigger the same SIGSEGV even without that
patch, with some timeouts during previous server initilizations and
small asynchronous_metrics_update_period_s.

Fix this by creating GRPCServer::Runner in the ctor.

Stacktrace:
    26.842505 [ 7 ] {} <Information> Application: Listening for MySQL compatibility protocol: 0.0.0.0:9004
    26.842562 [ 7 ] {} <Information> Application: Listening for gRPC protocol: 0.0.0.0:9100
    26.842600 [ 7 ] {} <Debug> AsynchronousMetrics: MemoryTracking: was 350.72 KiB, peak 5.25 MiB, will set to 232.63 MiB (RSS), difference: 232.29 MiB
    26.842834 [ 8 ] {} <Trace> BaseDaemon: Received signal 11
    26.843014 [ 39 ] {} <Fatal> BaseDaemon: ########################################
    26.843055 [ 39 ] {} <Fatal> BaseDaemon: (version 21.2.1.5858, build id: B27D5550AC34F9091BC4437D8021B752EDB34FBB) (from thread 7) (no query) Received signal Segmentation fault (11)
    26.843085 [ 39 ] {} <Fatal> BaseDaemon: Address: 0x78 Access: read. Address not mapped to object.
    26.843104 [ 39 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fe8c4e6afc4 0x13683cf1 0xf52efe5 0xe7c9137 0x85596e1 0x8553635 0x11b7a313 0x8545ebc 0x8544b25 0x84e1cbe 0x7fe8c4c940b3 0x84ac22e
    26.843136 [ 39 ] {} <Fatal> BaseDaemon: 2. pthread_mutex_lock @ 0xbfc4 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    26.843162 [ 39 ] {} <Fatal> BaseDaemon: 3. std::__1::mutex::lock() @ 0x13683cf1 in ?
    26.843188 [ 39 ] {} <Fatal> BaseDaemon: 4. DB::GRPCServer::currentConnections() const @ 0xf52efe5 in /usr/bin/clickhouse
    26.843204 [ 39 ] {} <Fatal> BaseDaemon: 5. DB::AsynchronousMetrics::update() @ 0xe7c9137 in /usr/bin/clickhouse
    26.843228 [ 39 ] {} <Fatal> BaseDaemon: 6. DB::AsynchronousMetrics::start() @ 0x85596e1 in /usr/bin/clickhouse
    26.843248 [ 39 ] {} <Fatal> BaseDaemon: 7. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x8553635 in /usr/bin/clickhouse
    26.843270 [ 39 ] {} <Fatal> BaseDaemon: 8. Poco::Util::Application::run() @ 0x11b7a313 in /usr/bin/clickhouse
    26.843284 [ 39 ] {} <Fatal> BaseDaemon: 9. DB::Server::run() @ 0x8545ebc in /usr/bin/clickhouse
    26.843299 [ 39 ] {} <Fatal> BaseDaemon: 10. mainEntryClickHouseServer(int, char**) @ 0x8544b25 in /usr/bin/clickhouse
    26.843313 [ 39 ] {} <Fatal> BaseDaemon: 11. main @ 0x84e1cbe in /usr/bin/clickhouse
    26.843331 [ 39 ] {} <Fatal> BaseDaemon: 12. __libc_start_main @ 0x270b3 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    26.843346 [ 39 ] {} <Fatal> BaseDaemon: 13. _start @ 0x84ac22e in /usr/bin/clickhouse
 2021-01-31 11:29:46 +03:00
Alexey Milovidov
d604cbc986 Maybe improve server latency 2021-01-31 10:41:24 +03:00
Pavel Kruglov
25e85d71ee Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-01-29 21:08:47 +03:00
Alexander Tokmakov
004517009e fix 2021-01-29 18:13:27 +03:00
alesapin
bac8cc55d2 Now we answer from follower nodes 2021-01-29 15:39:04 +03:00
Alexander Tokmakov
ffaa8e34a6 minor code improvements around ThreadStatus 2021-01-28 16:57:36 +03:00
alesapin
a33963e211 Better raft server startup 2021-01-27 20:54:25 +03:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
alesapin
61d006cbab Fix typo 2021-01-26 11:18:00 +03:00
alesapin
dea4b5009b Some server initialization 2021-01-25 15:29:12 +03:00
alesapin
173b6fefdf Merge branch 'master' into in_memory_raft 2021-01-25 12:03:28 +03:00