thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 10:22:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
81,269 Commits 9,328 Branches 1,783 Tags 2.1 GiB
b2271cc2d9
Commit Graph

688 Commits

Author SHA1 Message Date
Vitaly Baranov
40a3c8281a Fix synchronization in GRPCServer 2021-08-03 09:44:47 +03:00
Azat Khuzhin
1feee18455 Remove test_keeper_server usage (for {operation/session}_timeout_ms) 2021-07-29 21:24:32 +03:00
Nikolai Kochetov
9b5a816b43 Merge branch 'master' into output-streams-to-processors 2021-07-26 18:03:11 +03:00
Nikolai Kochetov
9c92f43359 Update storages. 2021-07-23 22:33:59 +03:00
Nikolai Kochetov
3c17a62686
Merge pull request #26590 from ClickHouse/remove-some-more-streams 
Remove some streams.
 2021-07-22 21:28:50 +03:00
Nikolai Kochetov
3ed3f7a9f7 Fix integration tests. 2021-07-22 13:38:22 +03:00
Nikolai Kochetov
5ffd99dfd4
Merge pull request #25674 from amosbird/distributedreturnconnection 
Drain connection asynchronously
 2021-07-22 11:04:49 +03:00
Nikolai Kochetov
f38de35b14 Rename some constants. 2021-07-21 19:13:17 +03:00
Nikolai Kochetov
179ec05a72 Remove some streams. 2021-07-20 21:18:43 +03:00
anneji-dev
d8f45fbe13
Set client query kind for mysql and postgresql handler (#26498) 
* client query kind is not set for mysql and postgresql

* Update MySQLHandler.cpp

* Update PostgreSQLHandler.cpp

Co-authored-by: anneji <anneji@tencent.com>
 2021-07-20 09:23:58 +03:00
Amos Bird
dbfb699690
Asynchronously drain connections. 2021-07-19 21:53:29 +08:00
Vitaly Baranov
4f1926550b
Merge pull request #26429 from vitlibar/remove-mysql-wire-context 
Remove MySQLWireContext
 2021-07-19 12:21:24 +03:00
Alexey Milovidov
261a220227 Remove some code 2021-07-17 21:06:46 +03:00
alexey-milovidov
38dfe1fc0a
Merge pull request #26172 from ClickHouse/exception_id 
Add error id to exceptions
 2021-07-17 02:07:17 +03:00
Vitaly Baranov
0f8b196682 Remove MySQLWireContext. 2021-07-16 22:21:20 +03:00
Ivan Lezhankin
69e7f082fe Merge remote-tracking branch 'upstream/master' into async-insert 2021-07-13 17:15:32 +03:00
Nikita Mikhaylov
c255f152aa
Merge pull request #26051 from ClickHouse/fix_21184 
Fix sequence_id in MySQL protocol
 2021-07-13 16:03:25 +03:00
mergify[bot]
b7ae4e4ced
Merge branch 'master' into async-insert 2021-07-12 09:21:37 +00:00
Alexey Milovidov
3887e82dac Merge branch 'exception_id' of github.com:yandex/ClickHouse into exception_id 2021-07-12 03:18:39 +03:00
Alexey Milovidov
c4f34689ec Merge branch 'master' into exception_id 2021-07-11 22:12:06 +03:00
Vladimir
f120bd3d09
Directly pass getExceptionMessage() to LOG_ERROR in TCPHandler.cpp 2021-07-10 13:30:38 +03:00
Alexey Milovidov
588c31d9f5 Better messages 2021-07-10 11:18:06 +03:00
Alexey Milovidov
1f91c65aca Add error id to exceptions 2021-07-10 05:43:53 +03:00
vdimir
c01f4588b4
Validate prometheus metric name with regex 2021-07-09 19:12:26 +03:00
Alexander Tokmakov
1a470fb777 fix sequence_id in MySQL protocol 2021-07-07 20:03:28 +03:00
kssenii
0dda5b67c4 Fix set SQL_SELECT_LIMIT 2021-07-04 16:17:59 +03:00
Ivan Lezhankin
37439a8af6 Fix typos 2021-06-29 01:51:01 +03:00
Ivan Lezhankin
09b3db38f0 Don't create input stream for native data 2021-06-28 23:43:37 +03:00
Ivan Lezhankin
1bb41593d1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-28 22:55:56 +03:00
sundy-li
d423d07a87 [mysql] fix mysql select user() return empty 2021-06-25 10:20:22 +08:00
Ivan Lezhankin
37365589ed Merge branch 'master' into async-insert 2021-06-17 16:57:45 +03:00
alexey-milovidov
04be5437d9
Merge pull request #25296 from abyss7/http-issues 
Add settings for HTTP header limitations
 2021-06-17 01:50:48 +03:00
alexey-milovidov
16931a661c
Update HTTPServerRequest.h 2021-06-17 01:50:06 +03:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Ivan Lezhankin
ba08a580f8 Add test 2021-06-16 17:33:14 +03:00
Ivan Lezhankin
b182d87d9c Add settings for HTTP header limitations 2021-06-15 17:33:46 +03:00
Amos Bird
7d9a1106b8
Add initial_query_start_time to query log 2021-06-11 22:24:53 +08:00
Ivan Lezhankin
973cea73b7 Merge branch 'master' into async-insert 2021-06-08 15:10:30 +03:00
Alexey Milovidov
17962459f5 Merge branch 'master' into issue-16775 2021-06-06 02:18:28 +03:00
Ivan Lezhankin
791cbcf0b1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-02 18:50:37 +03:00
Ivan Lezhankin
84c23dc060 Merge branch 'master' into async-insert 2021-06-02 18:06:21 +03:00
Yatsishin Ilya
d51a6a2c1f Merge remote-tracking branch 'origin' into integration-2 2021-06-01 17:23:01 +03:00
Nikolai Kochetov
c673eb2040 Use mutable context for TCPHandler::customizeContext 2021-06-01 12:27:05 +03:00
Yatsishin Ilya
9b3cc2eefb Merge remote-tracking branch 'origin' into integration-2 2021-06-01 09:08:42 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs 
Poco::File to std::filesystem
 2021-05-31 23:17:02 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
Yatsishin Ilya
b683f71ae6 Merge remote-tracking branch 'origin' into integration-2 2021-05-31 07:03:27 +03:00
kssenii
69816e6eff Fix checks 2021-05-30 15:44:58 +03:00
kssenii
db9f4762eb Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-29 01:15:43 +03:00
kssenii
2a631aaf08 Final fixes 2021-05-29 00:34:44 +03:00
Ivan
ce2a809773
Consider last CRLF as part of boundary (#24399) 
* Consider last CRLF as part of boundary
* Update HTMLForm.cpp
* Don't use xxd
 2021-05-28 14:59:11 +03:00
kssenii
31107816bb fs::absolute to fs::canonical 2021-05-24 20:37:54 +03:00
Yatsishin Ilya
c32cea4dd2 Merge remote-tracking branch 'origin' into integration-2 2021-05-24 09:59:16 +03:00
Alexey Milovidov
273226de32 Remove string parameter for Density 2021-05-24 06:43:25 +03:00
Alexey Milovidov
6f70feed2f Merge branch 'master' into issue-16775 2021-05-24 05:56:57 +03:00
kssenii
bbf56e35ac Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-23 10:54:48 +03:00
Alexey Milovidov
40d4f0678f Remove overload (harmful) 2021-05-23 04:25:06 +03:00
kssenii
1c43b333ea Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-22 21:26:48 +03:00
alexey-milovidov
1ed30f7246
Merge branch 'master' into issue-16775 2021-05-22 17:14:23 +03:00
alesapin
e4ce6faab6 Fix build 2021-05-22 15:32:53 +03:00
alesapin
8c63f0f8e2 Fix race condition on keeper shutdown 2021-05-22 10:46:12 +03:00
Yatsishin Ilya
fe630a0139 Merge remote-tracking branch 'origin/master' into integration-2 2021-05-21 17:57:44 +03:00
Ivan Lezhankin
33f816d28d Allow empty HTTP headers 2021-05-19 16:20:23 +03:00
kssenii
5f083b779d A little better, fix checks 2021-05-18 14:25:19 +03:00
kssenii
393b026f2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-11 16:49:23 +03:00
Yatsishin Ilya
9d159072ef mERGe remote-tracking branch 'origin' into integration-2 2021-05-11 13:32:44 +03:00
Alexey Milovidov
ad88819ee4 Fix a bunch of warnings from PVS-Studio 2021-05-08 19:13:10 +03:00
Alexey Milovidov
3025f9e141 More fixes for PVS-Studio 2021-05-08 19:12:49 +03:00
fibersel
cb53bbb7b0 add experimental codecs flag, add integration test for experimental codecs 2021-05-06 14:57:22 +03:00
kssenii
ee06936596 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-01 17:24:31 +03:00
Yatsishin Ilya
744fb93945 Merge remote-tracking branch 'origin' into integration-2 2021-04-30 12:18:24 +03:00
Maksim Kita
318c4bb80d Add examples folder filter to ya.make.in 2021-04-30 11:25:52 +03:00
Maksim Kita
0d0a14a925 Poco::Timespan reference fix 2021-04-29 19:11:20 +03:00
kssenii
deb4903af8 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-04-28 20:57:13 +03:00
kssenii
5e429b1266 Change in Interpreters/* 2021-04-28 20:52:00 +03:00
Yatsishin Ilya
c59a11f798 Merge remote-tracking branch 'origin' into integration-2 2021-04-27 18:34:33 +03:00
Azat Khuzhin
18575c4433 Avoid flushing of BlockOutputStream from destructor 
That may lead to uncaught exception [1].

  [1]: https://github.com/ClickHouse/ClickHouse/pull/21624#issuecomment-808066747

v0: New method BlockOutputStream::finalize() -- bad (forgot some places
    and superfluous)
v2: Flush BlockOutputStream in QueryStatus::releaseQueryStreams()
    Safer query termination (report some uncaugh errors to user)
v3: Flush BlockOutputStream from TCPHandler only
    Other places uses different executeQuery() implementation
v4: Do not call writeSuffix() twice
 2021-04-25 12:49:44 +03:00
Ivan Lezhankin
d9626e73c3 Revert "CC" 
This reverts commit ee194928d2.
 2021-04-22 16:55:18 +03:00
Ivan Lezhankin
ee194928d2 CC 2021-04-21 16:19:28 +03:00
Nikolai Kochetov
44112587d4
Merge pull request #23309 from ClickHouse/try-fix-crash-with-unknown-packet 
Fix crash in case of unknown packet
 2021-04-21 08:47:17 +03:00
Ivan
10c5ba3022
Merge branch 'master' into async-insert 2021-04-20 17:58:15 +03:00
Nikita Mikhaylov
08839bc203
Merge pull request #23197 from nikitamikhaylov/read-task-response 
Fix logical error in stress tests
 2021-04-20 16:09:25 +03:00
alexey-milovidov
c87f846816
Merge pull request #21938 from azat/connection-settings 
Apply idle_connnection_timeout/poll_interval after each query
 2021-04-19 22:58:21 +03:00
Nikolai Kochetov
394cca2434 Fix build. 2021-04-19 21:36:45 +03:00
Nikolai Kochetov
f6bdd35943 Try fix tests. 2021-04-19 20:55:18 +03:00
Ivan Lezhankin
7a88ca31cc Merge branch 'master' into async-insert 2021-04-19 18:33:43 +03:00
Nikolai Kochetov
0c23d44b99 Add test. 2021-04-19 17:12:08 +03:00
Nikita Mikhaylov
648b941922 done 2021-04-19 13:37:02 +00:00
Yatsishin Ilya
4347f6ae43 Merge remote-tracking branch 'origin' into integration-2 2021-04-15 16:49:31 +03:00
Azat Khuzhin
9b9e0a9bbc Block all memory tracking limits in destructors 2021-04-14 23:38:42 +03:00
Ivan Lezhankin
9e5d274467 Merge remote-tracking branch 'upstream/master' into async-insert 2021-04-14 20:16:05 +03:00
Yatsishin Ilya
10b7037d9c Merge remote-tracking branch 'origin' into integration-2 2021-04-14 14:21:40 +03:00
Nikita Mikhaylov
024374a2ec review fixes 2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
09a62e713a rename to s3Cluster 2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
507cb8514a review fixes 2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
7a68820342 style 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
704fb04941 better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
c333c3dedb review fixes 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
36a8419f60 style 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
587fbdd10d better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
4843f86329 use only one connection between initiator and worker 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
ebf1e2fa9c fix build 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
b3094412b1 better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
c1d1313dd8 remote prints 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
851644d8bf fix test 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
cef9e19eb2 better 2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
2549468c14 better 2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
64b4cd0e63 save 2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
31b4f9b17f save 2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
44ca65a9a4 save 2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
081ea84a41 save 2021-04-13 22:39:41 +03:00
alesapin
ef34c95c7e Add SSL to keeper (both client and internal) 2021-04-12 15:25:52 +03:00
Azat Khuzhin
260a77118f Apply idle_connnection_timeout/poll_interval after each query 
Since right now it is impossible to change those settings, only via
user profile.
 2021-04-12 10:39:28 +03:00
Yatsishin Ilya
ec6adb692d Merge remote-tracking branch 'origin' into integration-2 2021-04-12 10:04:11 +03:00
Yatsishin Ilya
7da322f95a better 2021-04-12 10:04:07 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
alesapin
1533f9b9aa
Merge pull request #14113 from johnskopis/dynamic-interserver-creds-v20 
Support interserver credential rotation
 2021-04-08 11:01:40 +03:00
alesapin
98871f8c5a More clear messages 2021-04-07 17:05:40 +03:00
alesapin
2a6560b1c6 Better interface 2021-04-07 16:59:18 +03:00
alesapin
2987bbc948 Small improvements 2021-04-07 16:52:11 +03:00
Ivan
aa47eb7d5e
Fix checkpoint position after putting it on buffer end (#22518) 2021-04-07 16:22:38 +03:00
alesapin
95881bf2dd Fix duplicated headers 2021-04-06 19:31:24 +03:00
alesapin
b4fc13de8e fix style 2021-04-06 17:02:56 +03:00
alesapin
4633afe962 Some fixes 2021-04-06 16:56:14 +03:00
alesapin
d913f327ed Buildable code 2021-04-06 16:42:38 +03:00
alesapin
1c8e33414f Merge branch 'master' into johnskopis/dynamic-interserver-creds-v20 2021-04-06 16:28:46 +03:00
Nicolae Vartolomei
49abdeb967 Try flush write buffer only if it is initialized 
Fixes #22579
 2021-04-04 12:59:51 +03:00
alexey-milovidov
9f5cd35a69
Merge pull request #22323 from abyss7/poco-http2 
Follow up to #19516
 2021-04-03 02:09:10 +03:00
alexey-milovidov
3d860fc6ac
Merge pull request #22322 from abyss7/issue-21907 
Do not limit HTTP chunk size
 2021-04-03 01:58:22 +03:00
Ivan Lezhankin
8a455d347a Fix build 2021-04-02 14:08:32 +03:00
Ivan Lezhankin
d9d143675a Merge remote-tracking branch 'upstream/master' into issue-21907 2021-04-01 16:29:26 +03:00
Ivan Lezhankin
19550d5173 Initialize pointer with nullptr 2021-04-01 16:26:13 +03:00
Ivan Lezhankin
cc1c439ad8 Merge remote-tracking branch 'upstream/master' into poco-http2 2021-04-01 16:20:15 +03:00
Alexey Milovidov
9d65d83c83 Better exception message in client in case of exception while writing blocks 2021-04-01 07:12:07 +03:00
Ivan Lezhankin
de05803177 Merge branch 'master' into async-insert 2021-03-29 20:59:02 +03:00
Ivan Lezhankin
ae741c7411 Do not limit HTTP chunk size 2021-03-29 20:49:28 +03:00
alesapin
e4a028c4d4 Missed ya.make 2021-03-29 12:25:40 +03:00
alesapin
be132a32a2 More renames 2021-03-29 11:24:56 +03:00
alesapin
ced6d8e6bd
Merge pull request #21677 from ClickHouse/jepsen_for_nukeeper 
Jepsen for nukeeper
 2021-03-27 10:18:42 +03:00
alexey-milovidov
ac53d30e53
Merge pull request #22146 from azat/writers-uncaught-exceptions 
Fix uncaught exception in InterserverIOHTTPHandler
 2021-03-26 17:23:09 +03:00
tavplubix
d9ec4090d2
Merge pull request #21936 from abyss7/html-form-fix 
Fix issues with HTMLForm::MultipartReadBuffer and PeekableReadBuffer
 2021-03-26 15:35:34 +03:00
Azat Khuzhin
e1de960025 Fix uncaught exception in InterserverIOHTTPHandler 
There was one more uncaught exception case [1]:

    2021.03.19 18:11:00.845632 [ 17469 ] {} <Debug> InterserverIOHTTPHandler: Done processing query
    ...
    2021.03.19 18:11:31.698961 [ 80145 ] {} <Fatal> BaseDaemon: ########################################
    2021.03.19 18:11:31.699903 [ 80145 ] {} <Fatal> BaseDaemon: (version 21.4.1.6293 (official build), build id: 859E400E1C65C4702FE491420741DD8B58190002) (from thread 17469) (no query) Received signal Aborted (6)
    2021.03.19 18:11:32.614075 [ 80145 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../contrib/libcxxabi/src/cxa_handlers.cpp:89: std::terminate() @ 0x21e9b3a2 in /usr/bin/clickhouse
    2021.03.19 18:11:43.831215 [ 80145 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/Server/HTTP/WriteBufferFromHTTPServerResponse.cpp:201: ? @ 0x1be77038 in /usr/bin/clickhouse
    2021.03.19 18:11:44.743193 [ 80145 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:892: std::__1::allocator<DB::WriteBufferFromHTTPServerResponse>::destroy(DB::WriteBufferFromHTTPServerResponse*) @ 0x1bddd7c9 in /usr/bin/clickhouse
    2021.03.19 18:11:45.283905 [ 80145 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:541: void std::__1::allocator_traits<std::__1::allocator<DB::WriteBufferFromHTTPServerResponse> >::__destroy<DB::WriteBufferFromHTTPServerResponse>(std::__1::integral_constant<bool, true>, std::__1::allocator<DB::WriteBufferFromHTTPServerResponse>&, DB::WriteBufferFromHTTPServerResponse*) @ 0x1bddd79d in /usr/bin/clickhouse
    2021.03.19 18:11:45.805233 [ 80145 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:487: void std::__1::allocator_traits<std::__1::allocator<DB::WriteBufferFromHTTPServerResponse> >::destroy<DB::WriteBufferFromHTTPServerResponse>(std::__1::allocator<DB::WriteBufferFromHTTPServerResponse>&, DB::WriteBufferFromHTTPServerResponse*) @ 0x1bddd76d in /usr/bin/clickhouse
    2021.03.19 18:11:46.351371 [ 80145 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2611: std::__1::__shared_ptr_emplace<DB::WriteBufferFromHTTPServerResponse, std::__1::allocator<DB::WriteBufferFromHTTPServerResponse> >::__on_zero_shared() @ 0x1bddd525 in /usr/bin/clickhouse
    2021.03.19 18:11:46.579263 [ 80145 ] {} <Fatal> BaseDaemon: 15. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2476: std::__1::__shared_count::__release_shared() @ 0x119490ed in /usr/bin/clickhouse
    2021.03.19 18:11:46.790912 [ 80145 ] {} <Fatal> BaseDaemon: 16. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2517: std::__1::__shared_weak_count::__release_shared() @ 0x1194908f in /usr/bin/clickhouse
    2021.03.19 18:11:47.277990 [ 80145 ] {} <Fatal> BaseDaemon: 17. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3213: std::__1::shared_ptr<DB::WriteBufferFromHTTPServerResponse>::~shared_ptr() @ 0x1bdd75fc in /usr/bin/clickhouse
    2021.03.19 18:11:47.649213 [ 80145 ] {} <Fatal> BaseDaemon: 18. ./obj-x86_64-linux-gnu/../src/Server/InterserverIOHTTPHandler.h:34: DB::InterserverIOHTTPHandler::Output::~Output() @ 0x1bdf6bd5 in /usr/bin/clickhouse
    2021.03.19 18:11:47.921556 [ 80145 ] {} <Fatal> BaseDaemon: 19. ./obj-x86_64-linux-gnu/../src/Server/InterserverIOHTTPHandler.cpp:154: DB::InterserverIOHTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x1bdf653f in /usr/bin/clickhouse

  [1]: https://clickhouse-test-reports.s3.yandex.net/0/78c56b891383288cf3a893139e796fc87476412e/stress_test_(debug).html

Since in case of no errors during processing we should call finalize, to
ensure that it will not be called from dtor.

Fixes: #22046
Fixes: #22067
 2021-03-25 23:41:03 +03:00
Azat Khuzhin
48fe30e5a2 Add missing logging for exception in InterserverIOHTTPHandler 2021-03-25 23:41:03 +03:00
Kruglov Pavel
66b8431e71
Merge pull request #21429 from Avogar/secure-socket 
Fix receive and send timeouts and non-blocking read in secure socket
 2021-03-25 17:30:17 +03:00
alesapin
4c28c25a75 Merge branch 'master' into jepsen_for_nukeeper 2021-03-25 14:34:00 +03:00
Pavel Kruglov
6278a7efac Fix build 2021-03-23 00:21:52 +03:00
Pavel Kruglov
dbc5018000 Update tests for hedged requests 2021-03-22 22:18:06 +03:00
alesapin
27b90007aa Merge branch 'master' into jepsen_for_nukeeper 2021-03-22 11:20:00 +03:00
Ivan Lezhankin
4f1fad97ab Merge remote-tracking branch 'upstream/master' into async-insert 2021-03-19 23:41:01 +03:00
Ivan Lezhankin
957c053f7e Fix segfault 2021-03-19 23:29:01 +03:00
Nikita Mikhaylov
2e13c767a0
Merge pull request #21149 from nikitamikhaylov/pp-for-csv-with-names 
Support parallel parsing for *WithNames types
 2021-03-16 14:38:03 +03:00
Nikita Mikhaylov
c92d8c776c try to avoid race in grpc server 2021-03-15 15:14:58 +03:00
alesapin
42ccee286a Merge branch 'master' into jepsen_for_nukeeper 2021-03-15 14:17:13 +03:00
alesapin
ba9e1e5a8d Some initial code 
Add some java magic

Allow to connect with old session id

More angry nemesis and fixes

Angry

Fix style

Split to files

Better wrappers

Better structure

Add set test and split to separate files (I think something broken now)

Better

Missed files
 2021-03-12 22:07:57 +03:00
Denis Glazachev
290a6d273e
Add Kerberos support for authenticating existing users when accessing over HTTP 2021-03-11 23:41:10 +03:00
alesapin
bcd073278d
Merge pull request #21425 from ClickHouse/persistent_nukeeper_snapshot_storage 
Persistent nukeeper snapshot storage
 2021-03-09 18:37:23 +03:00
zhangxiao871
846f5a93a9 Fix incorrect fd_ready assignment in NuKeeperTCPHandler. 2021-03-09 17:42:27 +08:00
alesapin
a8b7469857 Fix session timeout update 2021-03-06 17:14:38 +03:00
Ivan Lezhankin
6a07ee1506 CC 2021-03-04 17:01:37 +03:00
Ivan Lezhankin
91ff08452e Remove unused code 2021-03-04 16:58:18 +03:00
Ivan Lezhankin
a3509065b2 Add trivial README about Poco-based code 2021-03-04 15:43:00 +03:00
Ivan Lezhankin
5cfeedb1e0 Add setting http_max_uri_size 2021-03-04 15:21:23 +03:00
Ivan Lezhankin
74214865eb Merge remote-tracking branch 'upstream/master' into async-insert 2021-03-04 14:14:28 +03:00
Ivan Lezhankin
b726845be2 [WIP] 2021-03-04 14:10:21 +03:00
Pavel Kruglov
1c7f16e0ff Add test and move TimeoutSetter in IO/ 2021-03-03 20:47:27 +03:00
Nikolai Kochetov
cb122160ae
Merge pull request #19291 from Avogar/hedged-requests 
Implementation of HedgedRequests
 2021-03-03 12:58:37 +03:00
Pavel Kruglov
4166ae1f45 Restart timeout when we make progress 2021-03-02 15:40:24 +03:00
Amos Bird
ad4fd75fb4
mycli compatibility 2021-03-02 18:53:06 +08:00
Pavel Kruglov
153bfbfc28 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-03-02 11:59:32 +03:00
Azat Khuzhin
eb0387c5a9 Fix abnormal server termination for nested writers 
Writers with nested writer can call next() from the dtor for nested
writer and this will cause exception again, so the buffer position
should be updated on exceptions.

Found by stress test (thread) here [1] and here [2]:

    2021.02.27 19:27:53.498977 [ 302 ] {} <Fatal> BaseDaemon: (version 21.3.1.6130, build id: 2DAEC5DEBF03C5A1C3BF66B7779C886F16239345) (from thread 1614) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 2097498, Stack trace (when copying this message, always include the lines below):

    0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15bef2ab in /usr/bin/clickhouse
    1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8aea92e in /usr/bin/clickhouse
    2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8bbbc45 in /usr/bin/clickhouse
    3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8bbc077 in /usr/bin/clickhouse
    4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8bbc18a in /usr/bin/clickhouse
    5. ./obj-x86_64-linux-gnu/../src/IO/BufferWithOwnMemory.h:137: DB::ZstdDeflatingWriteBuffer::~ZstdDeflatingWriteBuffer() @ 0x118bdc29 in /usr/bin/clickhouse
    6. ./obj-x86_64-linux-gnu/../src/IO/ZstdDeflatingWriteBuffer.cpp:32: DB::ZstdDeflatingWriteBuffer::~ZstdDeflatingWriteBuffer() @ 0x118be3ea in /usr/bin/clickhouse
    7. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:0: DB::WriteBufferFromHTTPServerResponse::finalize() @ 0x12f1dceb in /usr/bin/clickhouse
    8. ./obj-x86_64-linux-gnu/../src/Server/HTTPHandler.cpp:703: DB::HTTPHandler::trySendExceptionToClient(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x12e9fecc in /usr/bin/clickhouse
    9. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/string:1444: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x12ea0d60 in /usr/bin/clickhouse
    10. ./obj-x86_64-linux-gnu/../src/Server/HTTP/HTTPServerConnection.cpp:0: DB::HTTPServerConnection::run() @ 0x12f16db1 in /usr/bin/clickhouse
    11. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x15b184f3 in /usr/bin/clickhouse
    12. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:0: Poco::Net::TCPServerDispatcher::run() @ 0x15b18c1f in /usr/bin/clickhouse
    13. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/ScopedLock.h:36: Poco::PooledThread::run() @ 0x15c7fdb2 in /usr/bin/clickhouse
    14. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x15c7e350 in /usr/bin/clickhouse
    15. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/SharedPtr.h:277: Poco::ThreadImpl::runnableEntry(void*) @ 0x15c7cb58 in /usr/bin/clickhouse
    16. __tsan_thread_start_func @ 0x8a04ced in /usr/bin/clickhouse
    17. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    18. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
     (version 21.3.1.6130)

  [1]: https://clickhouse-test-reports.s3.yandex.net/21279/4f61ef3099f42f17b496a0b0424773978d9a32dc/stress_test_(thread).html#fail1
  [2]: https://clickhouse-test-reports.s3.yandex.net/21292/ae9fea1d0af118a8f87b224d194d61da1567188b/stress_test_(thread).html#fail1

v2: https://clickhouse-test-reports.s3.yandex.net/21305/e969daa6e86c5e09cfef08cfde19712982b64e59/stress_test_(thread).html#fail1
 2021-02-28 15:13:42 +03:00
Pavel Kruglov
197058dc12 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-02-27 00:36:33 +03:00
Ivan Lezhankin
c9dc07fddd Another fix 2021-02-26 20:37:34 +03:00
alesapin
a1901383ae Improve performance and fix condition variable wait 2021-02-24 14:54:14 +03:00
alesapin
5b888e1193 Merge branch 'master' into persistent_nukeeper_log_storage 2021-02-22 23:52:26 +03:00
Kruglov Pavel
815b4ff0d9
Merge branch 'master' into hedged-requests 2021-02-21 17:10:24 +03:00
alexey-milovidov
20a7ccf5b1
Merge pull request #20174 from amosbird/alias 
Normalized function names (lower-case case-insensitive names)
 2021-02-21 16:23:51 +03:00
Amos Bird
f37631830f
Comments 2021-02-20 16:45:25 +08:00
alesapin
c61769311e Merge branch 'master' into persistent_nukeeper_log_storage 2021-02-20 10:22:22 +03:00
Azat Khuzhin
1ccb333ac5 Fix WriteBufferFromHTTPServerResponse usage in other places (add missing finalize()) 
Since I saw the following:

    0. DB::WriteBufferFromOStream::nextImpl()
    1. DB::WriteBufferFromHTTPServerResponse::nextImpl()
    2. DB::WriteBufferFromHTTPServerResponse::finalize()
    3. DB::WriteBufferFromHTTPServerResponse::~WriteBufferFromHTTPServerResponse()
    4. DB::StaticRequestHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)
    5. Poco::Net::HTTPServerConnection::run()
    6. Poco::Net::TCPServerConnection::start()
 2021-02-20 10:15:38 +03:00
Azat Khuzhin
fba1c7fcc1 Fix uncaught exception when HTTP client goes away 
Even after #20464 it was still possible, for example [1].

    2021.02.19 11:40:21.886191 [ 68373 ] {} <Trace> DynamicQueryHandler: Request URI: /?database=test_ds2d6y&log_comment=/usr/share/clickhouse-test/queries/0_stateless/01302_aggregate_state_exception_memory_leak.sh&enable_http_compression=1&http_zlib_compression_level=1

    <snip>

    2021.02.19 11:41:35.289940 [ 365 ] {} <Fatal> BaseDaemon: (version 21.3.1.6058, build id: 8D46D65205E2C8B7FE408A0B4EC76CA0483F9E92) (from thread 68373) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 262568, Stack trace (when copying this message, always include the lines below):

    0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15b3c7db in /usr/bin/clickhouse
    1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8aba66e in /usr/bin/clickhouse
    2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8b8c105 in /usr/bin/clickhouse
    3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8b8c537 in /usr/bin/clickhouse
    4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::Write

  [1]: https://clickhouse-test-reports.s3.yandex.net/16481/5d150cce4778dd14f58dcff67435bdec1efa155b/stress_test_(thread).html#fail1

And according to this partial stacktrace it seems that the dtor of
WriteBufferFromOStream was called from
WriteBufferFromHTTPServerResponse, since the class name starts from
DB::Write*

The problem is that if first time WriteBufferFromOStream::next() fails,
it will reset position to make next write no-op, however
WriteBufferFromHTTPServerResponse::next() will set position to available
buffer back, and next() will throw again, but this time it can be from
dtor.
 2021-02-19 22:00:57 +03:00
tavplubix
3509fe88d8
Merge pull request #20878 from ClickHouse/fix_18926 
Fix COM_FIELD_LIST response in MySQL protocol
 2021-02-19 19:25:54 +03:00
Ivan Lezhankin
3c4d1c89dd Merge remote-tracking branch 'upstream/master' into async-insert 2021-02-19 16:58:58 +03:00
alesapin
5c9420c077 More correct epoll usage 2021-02-19 15:56:51 +03:00
Ivan
414f470c79
Make Poco HTTP Server zero-copy again (#19516) 
* Refactoring: part 1

* Refactoring: part 2

* Handle request using ReadBuffer interface

* Struggles with ReadBuffer's

* Fix URI parsing

* Implement parsing of multipart/form-data

* Check HTTP_LENGTH_REQUIRED before eof() or will hang

* Fix HTTPChunkedReadBuffer

* Fix build and style

* Fix test

* Resist double-eof

* Fix arcadian build
 2021-02-19 15:51:26 +03:00
alesapin
3d954c4314 Better request/response logic 2021-02-19 14:06:17 +03:00
alesapin
6c9322bb2e Sane constant while reading requests 2021-02-19 12:57:39 +03:00
Amos Bird
fc185e5fb7
Another try 2021-02-19 11:56:24 +08:00
Alexander Tokmakov
97f4c457ec fix MySQL COMM_FIELD_LIST response 2021-02-18 16:27:51 +03:00
Amos Bird
2c4bc43014
Backward compatible 2021-02-18 20:20:29 +08:00
Ivan Lezhankin
dee051dc8e Merge remote-tracking branch 'upstream/master' into async-insert 2021-02-17 21:48:48 +03:00
Pavel Kruglov
1b78de2142 Use fibers in HedgedRequests 2021-02-17 20:34:52 +03:00
Ivan Lezhankin
f2b1708164 Refactor inserts and its tails 2021-02-15 21:57:35 +03:00
Kruglov Pavel
598576ce70
Merge branch 'master' into hedged-requests 2021-02-15 16:51:45 +03:00
Azat Khuzhin
33f54cdb3c Fix abnormal server termination when http client goes away 
In [1] stress tests found:

    2021.02.12 14:20:58.800988 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Debug> executeQuery: (from [::1]:45792, using production parser) (comment: /usr/share/clickhouse-test/queries/0_stateless/01520_client_print_query_id.expect) SELECT * FROM numbers(34599)
    2021.02.12 14:20:58.916484 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Trace> ContextAccess (default): Access granted: CREATE TEMPORARY TABLE ON *.*
    2021.02.12 14:20:59.071980 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
    2021.02.12 14:21:10.708202 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Information> executeQuery: Read 34599 rows, 270.30 KiB in 11.876294055 sec., 2913 rows/sec., 22.76 KiB/sec.
    2021.02.12 14:22:10.506261 [ 17728 ] {de3e7894-b401-4f7d-8530-90cd5ab06682} <Debug> DynamicQueryHandler: Done processing query
    2021.02.12 14:22:18.238037 [ 375 ] {} <Fatal> BaseDaemon: (version 21.3.1.5996, build id: 8DBCED54529C989F7AD4D991F51410774D55DE6C) (from thread 17728) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 262994, Stack trace (when copying this message, always include the lines below):

    0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15c976cb in /usr/bin/clickhouse
    1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8c9320e in /usr/bin/clickhouse
    2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8d54da5 in /usr/bin/clickhouse
    3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8d551d7 in /usr/bin/clickhouse
    4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::Write

    2021.02.12 14:22:18.811071 [ 18134 ] {} <Fatal> BaseDaemon: ########################################
    2021.02.12 14:22:18.878935 [ 18134 ] {} <Fatal> BaseDaemon: (version 21.3.1.5996, build id: 8DBCED54529C989F7AD4D991F51410774D55DE6C) (from thread 17728) (query_id: de3e7894-b401-4f7d-8530-90cd5ab06682) Received signal Aborted (6)
    2021.02.12 14:22:18.943148 [ 18134 ] {} <Fatal> BaseDaemon:
    2021.02.12 14:22:19.007073 [ 18134 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f109932018b 0x7f10992ff859 0x8bb33ae 0x8e301dd 0x17dac8c4 0x17dac7c7 0x8c3fe0b 0x8d552c5 0x8d552ea 0x11a29914 0x11a2a2ca 0x12f96092 0x12f8c65e 0x12f84300 0x15b84110 0x15bc0913 0x15bc103f 0x15d29a12 0x15d27fb0 0x15d267b8 0x8badbad 0x7f10994d5609 0x7f10993fc293
    2021.02.12 14:22:19.255998 [ 18134 ] {} <Fatal> BaseDaemon: 5. raise @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.02.12 14:22:19.270203 [ 18134 ] {} <Fatal> BaseDaemon: 6. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.02.12 14:22:50.108918 [ 370 ] {} <Fatal> Application: Child process was terminated by signal 6.

  [1]: https://clickhouse-test-reports.s3.yandex.net/19580/6aecb62416ece880cbb8ee3a803e14d841388dde/stress_test_(thread).html#fail1

Verified locally by commenting out->next() call in
WriteBufferFromHTTPServerResponse::nextImpl(), adding a sleep(1) and
canceling HTTP request before it finished, the stacktrace as follow:

    [ 6351 ] {} <Fatal> BaseDaemon: (version 21.3.1.1, build id: 9B40466BF3D2F5AED78A52A995A4A2FD3116787C) (from thread 6677) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 4, Stack trace (when copying this message, always include the lines below):

    0. /src/ch/clickhouse/.cmake/../src/Common/StackTrace.cpp:298: StackTrace::tryCapture() @ 0x30a52a in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    1. /src/ch/clickhouse/.cmake/../src/Common/StackTrace.cpp:260: StackTrace::StackTrace() @ 0x30a4e5 in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    2. /src/ch/clickhouse/.cmake/../src/Common/Exception.cpp:53: DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) @ 0x2a61ae in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    3. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:22: DB::WriteBufferFromOStream::nextImpl() @ 0x3b468a in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    4. /src/ch/clickhouse/.cmake/../src/IO/WriteBuffer.h:47

    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 7. __cxxabiv1::__terminate(void (*)()) @ 0x1784ca in /src/ch/clickhouse/.cmake/contrib/replxx-cmake/libreplxxd.so
    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 10. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x3b48c1 in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    [ 8966 ] {} <Fatal> BaseDaemon: 11. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromOStream.cpp:44: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x3b48ec in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 14. /src/ch/clickhouse/.cmake/../src/IO/WriteBufferFromHTTPServerResponse.cpp:218: DB::WriteBufferFromHTTPServerResponse::~WriteBufferFromHTTPServerResponse() @ 0x3b33cd in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    <snip>
    [ 8966 ] {} <Fatal> BaseDaemon: 22. /src/ch/clickhouse/.cmake/../src/Server/HTTPHandler.h:43: DB::HTTPHandler::Output::~Output() @ 0x260421 in /src/ch/clickhouse/.cmake/src/libclickhouse_serverd.so
    [ 8966 ] {} <Fatal> BaseDaemon: 23. /src/ch/clickhouse/.cmake/../src/Server/HTTPHandler.cpp:778: DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) @ 0x253fd4 in /src/ch/clickhouse/.cmake/src/libclickhouse_serverd.so
 2021-02-13 13:23:03 +03:00
Azat Khuzhin
592f62d0af Remove superfluous out->next() call in HTTPHandler 2021-02-13 13:22:37 +03:00
alesapin
5f32f65211 Merge branch 'master' into in_memory_raft 2021-02-11 23:09:29 +03:00
alexey-milovidov
dc3ffd3fe2
Merge pull request #19451 from azat/safe-writes 
Do not silently ignore write errors
 2021-02-11 21:19:11 +03:00
Alexey Milovidov
dbb41ce4e2 Remove debug output 2021-02-11 02:55:52 +03:00
Alexey Milovidov
f442b30f30 Fix test 2021-02-11 01:23:27 +03:00
Alexey Milovidov
d3dba0e52a Merge branch 'master' into no-excessive-squash-memory 2021-02-10 23:45:12 +03:00
Alexey Milovidov
30b2554fa3 Fix error 2021-02-10 20:48:39 +03:00
alesapin
9667bdcbd2 Merge branch 'master' into in_memory_raft 2021-02-09 16:06:31 +03:00
alesapin
b8baf3a443 Fix some warnings 2021-02-08 22:40:44 +03:00
tavplubix
ac477d9850
Merge pull request #19771 from ClickHouse/thread_state_improvements 
Minor code improvements around ThreadStatus
 2021-02-08 22:34:55 +03:00
alesapin
add89c17f2 Less serde in responses, watches on followers 2021-02-08 16:06:55 +03:00
tavplubix
060be5b2db
rerun CI checks 2021-02-08 15:04:40 +03:00
alesapin
d5ee7d33e9 Merge branch 'master' into in_memory_raft 2021-02-08 10:30:31 +03:00
Alexey Milovidov
869bca74a7 Fix some tests 2021-02-07 23:37:55 +03:00
alexey-milovidov
f2ab4f2ce9
Merge pull request #20037 from abyss7/better-read-buffers-2 
ReadBuffer: check for unread data on next()
 2021-02-07 01:33:06 +03:00
Pavel Kruglov
f946aab759 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-02-06 17:38:56 +03:00
alesapin
449e8e3fd9 More checks for setting 2021-02-05 13:15:02 +03:00
Azat Khuzhin
64c0bf9829 TCPHandler: catch exceptions from the WriteBuffer in destructor 
For TCPHandler it is safe thing todo.

Otherwise *San will report [1]:

    2021.01.24 15:33:40.103996 [ 270 ] {} <Trace> BaseDaemon: Received signal -1
    2021.01.24 15:33:40.110693 [ 270 ] {} <Fatal> BaseDaemon: (version 21.2.1.5789, build id: FF421B087D1E2EAA19FA17B5AB3AE413832744E0) (from thread 48318) Terminate called for uncaught exception:
    2021.01.24 15:33:40.114845 [ 270 ] {} <Trace> BaseDaemon: Received signal 6
    2021.01.24 15:33:40.138738 [ 218027 ] {} <Fatal> BaseDaemon: ########################################
    2021.01.24 15:33:40.138838 [ 218027 ] {} <Fatal> BaseDaemon: (version 21.2.1.5789, build id: FF421B087D1E2EAA19FA17B5AB3AE413832744E0) (from thread 48318) (no query) Received signal Aborted (6)
    2021.01.24 15:33:40.138912 [ 218027 ] {} <Fatal> BaseDaemon:
    2021.01.24 15:33:40.139277 [ 218027 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f185474118b 0x7f1854720859 0xaddc0cc 0x2af9fab8 0x2af9fa04 0xa91758b 0x1e418bb5 0x20725b4f 0x20725d9e 0x266b47a3 0x269772f5 0x26971847 0x7f18548f6609 0x7f185481d293
    2021.01.24 15:33:40.139637 [ 218027 ] {} <Fatal> BaseDaemon: 3. raise @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.01.24 15:33:40.140113 [ 218027 ] {} <Fatal> BaseDaemon: 4. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.01.24 15:33:40.144121 [ 218027 ] {} <Fatal> BaseDaemon: 5. ./obj-x86_64-linux-gnu/../base/daemon/BaseDaemon.cpp:0: terminate_handler() @ 0xaddc0cc in /usr/bin/clickhouse
    2021.01.24 15:33:40.151208 [ 218027 ] {} <Fatal> BaseDaemon: 6. ./obj-x86_64-linux-gnu/../contrib/libcxxabi/src/cxa_handlers.cpp:61: std::__terminate(void (*)()) @ 0x2af9fab8 in /usr/bin/clickhouse
    2021.01.24 15:33:40.153085 [ 218027 ] {} <Fatal> BaseDaemon: 7. ./obj-x86_64-linux-gnu/../contrib/libcxxabi/src/cxa_handlers.cpp:0: std::terminate() @ 0x2af9fa04 in /usr/bin/clickhouse
    2021.01.24 15:33:40.155209 [ 218027 ] {} <Fatal> BaseDaemon: 8. ? @ 0xa91758b in /usr/bin/clickhouse
    2021.01.24 15:33:40.156621 [ 218027 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::~WriteBufferFromPocoSocket() @ 0x1e418bb5 in /usr/bin/clickhouse
    2021.01.24 15:33:40.161041 [ 218027 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2518: DB::TCPHandler::~TCPHandler() @ 0x20725b4f in /usr/bin/clickhouse
    2021.01.24 15:33:40.164557 [ 218027 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.h:101: DB::TCPHandler::~TCPHandler() @ 0x20725d9e in /usr/bin/clickhouse
    2021.01.24 15:33:40.165921 [ 218027 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/AtomicCounter.h:314: Poco::Net::TCPServerDispatcher::run() @ 0x266b47a3 in /usr/bin/clickhouse
    2021.01.24 15:33:40.167347 [ 218027 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x269772f5 in /usr/bin/clickhouse
    2021.01.24 15:33:40.169401 [ 218027 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x26971847 in /usr/bin/clickhouse
    2021.01.24 15:33:40.169498 [ 218027 ] {} <Fatal> BaseDaemon: 15. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    2021.01.24 15:33:40.169566 [ 218027 ] {} <Fatal> BaseDaemon: 16. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.01.24 15:33:41.027601 [ 218027 ] {} <Fatal> BaseDaemon: Calculated checksum of the binary: 63D7491B39260494BA0D785E1860B427. There is no information about the reference checksum.

  [1]: https://clickhouse-test-reports.s3.yandex.net/19451/1e16bd6f337985a82fbdf4eded695dc6e663af58/stress_test_(address).html#fail1

v2: Fix catching errors in WriteBufferFromPocoSocket destructor
 2021-02-05 01:32:36 +03:00
Azat Khuzhin
4beb5c1b8a TCPHandler: Move constructor into the module and add missing headers 2021-02-05 01:31:42 +03:00
alesapin
c5312bf362 Trying to disable suspicious parameter 2021-02-04 22:29:46 +03:00
Ivan Lezhankin
ac4c3a6b27 Don't pass empty vector to ConcatReadBuffer 2021-02-04 18:59:54 +03:00
alesapin
da51ea1794 Simplify shutdown and requests processing 2021-02-04 15:07:41 +03:00
alesapin
1ff87ac6f9 Add background session lifetime control 2021-02-03 23:32:15 +03:00
Aleksei Semiglazov
921518db0a CLICKHOUSE-606: query deduplication based on parts' UUID 
* add the query data deduplication excluding duplicated parts in MergeTree family engines.

query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1

allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.

data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.

NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.

* add _part_uuid virtual column, allowing to use UUIDs in predicates.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>

address comments
 2021-02-02 16:53:39 +00:00
alesapin
9979181c02 Merge branch 'master' into in_memory_raft 2021-02-02 17:05:20 +03:00
alexey-milovidov
f5dc38a33c
Merge pull request #19584 from azat/client-complete-error-codes 
client: more suggestions
 2021-02-02 11:05:17 +03:00
Maksim Kita
d0151de4bb
Merge pull request #19608 from kreuzerkrieg/Add_IStoragePolicy_interface 
Add IStoragePolicy interface
 2021-02-02 11:03:20 +03:00
Azat Khuzhin
37797fdf5b Merge remote-tracking branch 'upstream/master' into client-complete-error-codes 2021-02-02 00:14:53 +03:00
alesapin
365bf65f5a Fix install script 2021-02-01 17:14:59 +03:00
alesapin
57c9b6c864 Fix build without nuraft 2021-02-01 16:18:17 +03:00
alesapin
0aca40d8cb Merge branch 'master' into in_memory_raft 2021-02-01 14:30:11 +03:00
alesapin
eb5c77f558 Fix some races and better reaction to leader change 2021-02-01 14:27:26 +03:00
alesapin
67412bd529 Don't accept connections if we don't see leader + race fix from upstream 2021-02-01 10:51:10 +03:00
Azat Khuzhin
978f23e343 Fix SIGSEGV due to accessing GRPCServer::currentConnections() before initialization 
This PR makes asynchronous metrics available just after start, and this
breaks GRPCServer, since it requires the server be started before
accessing currentConnections().

However it is possible to trigger the same SIGSEGV even without that
patch, with some timeouts during previous server initilizations and
small asynchronous_metrics_update_period_s.

Fix this by creating GRPCServer::Runner in the ctor.

Stacktrace:
    26.842505 [ 7 ] {} <Information> Application: Listening for MySQL compatibility protocol: 0.0.0.0:9004
    26.842562 [ 7 ] {} <Information> Application: Listening for gRPC protocol: 0.0.0.0:9100
    26.842600 [ 7 ] {} <Debug> AsynchronousMetrics: MemoryTracking: was 350.72 KiB, peak 5.25 MiB, will set to 232.63 MiB (RSS), difference: 232.29 MiB
    26.842834 [ 8 ] {} <Trace> BaseDaemon: Received signal 11
    26.843014 [ 39 ] {} <Fatal> BaseDaemon: ########################################
    26.843055 [ 39 ] {} <Fatal> BaseDaemon: (version 21.2.1.5858, build id: B27D5550AC34F9091BC4437D8021B752EDB34FBB) (from thread 7) (no query) Received signal Segmentation fault (11)
    26.843085 [ 39 ] {} <Fatal> BaseDaemon: Address: 0x78 Access: read. Address not mapped to object.
    26.843104 [ 39 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fe8c4e6afc4 0x13683cf1 0xf52efe5 0xe7c9137 0x85596e1 0x8553635 0x11b7a313 0x8545ebc 0x8544b25 0x84e1cbe 0x7fe8c4c940b3 0x84ac22e
    26.843136 [ 39 ] {} <Fatal> BaseDaemon: 2. pthread_mutex_lock @ 0xbfc4 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    26.843162 [ 39 ] {} <Fatal> BaseDaemon: 3. std::__1::mutex::lock() @ 0x13683cf1 in ?
    26.843188 [ 39 ] {} <Fatal> BaseDaemon: 4. DB::GRPCServer::currentConnections() const @ 0xf52efe5 in /usr/bin/clickhouse
    26.843204 [ 39 ] {} <Fatal> BaseDaemon: 5. DB::AsynchronousMetrics::update() @ 0xe7c9137 in /usr/bin/clickhouse
    26.843228 [ 39 ] {} <Fatal> BaseDaemon: 6. DB::AsynchronousMetrics::start() @ 0x85596e1 in /usr/bin/clickhouse
    26.843248 [ 39 ] {} <Fatal> BaseDaemon: 7. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x8553635 in /usr/bin/clickhouse
    26.843270 [ 39 ] {} <Fatal> BaseDaemon: 8. Poco::Util::Application::run() @ 0x11b7a313 in /usr/bin/clickhouse
    26.843284 [ 39 ] {} <Fatal> BaseDaemon: 9. DB::Server::run() @ 0x8545ebc in /usr/bin/clickhouse
    26.843299 [ 39 ] {} <Fatal> BaseDaemon: 10. mainEntryClickHouseServer(int, char**) @ 0x8544b25 in /usr/bin/clickhouse
    26.843313 [ 39 ] {} <Fatal> BaseDaemon: 11. main @ 0x84e1cbe in /usr/bin/clickhouse
    26.843331 [ 39 ] {} <Fatal> BaseDaemon: 12. __libc_start_main @ 0x270b3 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    26.843346 [ 39 ] {} <Fatal> BaseDaemon: 13. _start @ 0x84ac22e in /usr/bin/clickhouse
 2021-01-31 11:29:46 +03:00
Alexey Milovidov
d604cbc986 Maybe improve server latency 2021-01-31 10:41:24 +03:00
Pavel Kruglov
25e85d71ee Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-01-29 21:08:47 +03:00
Alexander Tokmakov
004517009e fix 2021-01-29 18:13:27 +03:00
alesapin
bac8cc55d2 Now we answer from follower nodes 2021-01-29 15:39:04 +03:00
Alexander Tokmakov
ffaa8e34a6 minor code improvements around ThreadStatus 2021-01-28 16:57:36 +03:00
alesapin
a33963e211 Better raft server startup 2021-01-27 20:54:25 +03:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
alesapin
61d006cbab Fix typo 2021-01-26 11:18:00 +03:00
alesapin
dea4b5009b Some server initialization 2021-01-25 15:29:12 +03:00
alesapin
173b6fefdf Merge branch 'master' into in_memory_raft 2021-01-25 12:03:28 +03:00
alesapin
7d4945a073
Merge pull request #19412 from ClickHouse/fix_one_more_tiny_race 
Fix one more race in TestKeeper
 2021-01-22 23:04:01 +03:00
alexey-milovidov
b6822bfd04
Merge pull request #19365 from ClickHouse/tavplubix-patch-3 
Better logging in MySQLHandler
 2021-01-22 17:29:59 +03:00
alesapin
4a06bf1d79 Merge branch 'fix_one_more_tiny_race' into in_memory_raft 2021-01-22 15:56:43 +03:00
alesapin
cc2788af88 Fix typo 2021-01-22 13:48:40 +03:00
alesapin
b2a4921558 Fix style 2021-01-22 13:14:53 +03:00
alesapin
83862799cd Fix one more race in TestKeeper 2021-01-22 13:02:48 +03:00
Alexey Milovidov
afc6f2fd8e Add http_referer to client info 2021-01-22 01:55:45 +03:00
alesapin
4aa11b3494 Remove zkutil namespace from TestKeeperStorage 2021-01-21 18:09:48 +03:00
tavplubix
600862f4a2
Better logging in MySQLHandler 2021-01-21 18:00:58 +03:00
alesapin
b8cdd8e375 Merge branch 'fix_race_test_keeper' into in_memory_raft 2021-01-21 15:45:41 +03:00
alesapin
7f32926a21 Fix race condition in TestKeeperHandler on session finish 2021-01-21 14:37:20 +03:00
alesapin
a241793a14 Merge branch 'decompose_test_keeper_storage' into in_memory_raft 2021-01-20 15:51:54 +03:00