thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 02:12:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
87,585 Commits 9,330 Branches 1,783 Tags 2.1 GiB
2257e57aa5
Commit Graph

793 Commits

Author SHA1 Message Date
Azat Khuzhin
08ced87880 Rework MultipartReadBuffer::skipToNextBoundary() 
As suggested by @vitlibar
 2021-12-21 08:38:41 +03:00
Azat Khuzhin
d7c7a91f66 Check that at least one boundary exist in HTMLForm::readMultipart() 2021-12-21 08:38:41 +03:00
Azat Khuzhin
4dfa9324d1 Check for EOF in MultipartReadBuffer::skipToNextBoundary() 
Otherwise you may hit internal assertion when client goes away:

    clickhouse-server: ./src/Server/HTTP/HTMLForm.cpp:245: bool DB::HTMLForm::MultipartReadBuffer::skipToNextBoundary(): Assertion `boundary_hit' failed.

Fixes: 02151_http_s_structure_set_eof
 2021-12-21 08:38:41 +03:00
JackyWoo
7a19570853 keeper handler should remove operation when response sent 2021-12-21 11:43:14 +08:00
Azat Khuzhin
a9f80dd705 Fix Context leak in case of cancel_http_readonly_queries_on_client_close 
To handle cancel_http_readonly_queries_on_client_close=true context
attaches progress callback with holding a copy of the current context,
and this creates recursive context reference and so shared_ptr will be
never released, [1] contains simplest example.

One example of a memory leak, external tables passed with HTTP query
(s_structure + file upload), since they are stored in the context, and
can occupate enough memory to make it visible to user.

Note, that the lifetime of the Context should be fine, since callback
can be called only when query is executed.

  [1]: Example of recursive reference

       ```c
       #include <memory>
       #include <functional>
       #include <cassert>

       bool released = true;

       class Context : public std::enable_shared_from_this<Context>
       {
           using Func = std::function<void()>;
           Func func;

       public:
           Context()  { released = false; }
           ~Context() { released = true; }
           Context(const Context & context) = delete;

           void addCallback(Func func_)
           {
               func = func_;
           }
       };

       int main()
       {
           {
               auto context = std::make_shared<Context>();
               context->addCallback([context]() {});
           }
           assert(released == true);
       }
       ```
 2021-12-20 21:05:38 +03:00
Azat Khuzhin
dd8c64e380 Set QueryScope earlier for HTTP queries 2021-12-20 21:05:38 +03:00
Vitaly Baranov
e61d3eef0c
Merge pull request #32747 from vitlibar/improve-grpc-compression 
Improve gRPC compression support
 2021-12-20 19:44:58 +03:00
Azat Khuzhin
6aebc3e94c Do not loose ProfileEvents in case of thread destroyed before 
v2: drop std::move and add copy ctor for ProfileEvents::Counter::Snapshot
v2: remove std::move
 2021-12-19 23:43:05 +03:00
Vitaly Baranov
708439b036 Support customized compression for input/output data in gRPC protocol. 2021-12-19 16:29:44 +03:00
Alexey Milovidov
2c68192bb6 Fix strange code in TCPHandler 2021-12-12 05:35:33 +03:00
Azat Khuzhin
57c027be0e Remove arcadia build support 2021-12-11 21:25:23 +03:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
alexey-milovidov
96ec92c7cd
Merge pull request #31155 from ClickHouse/incremental-profile-events 
Send incremental profile events to client
 2021-12-01 04:04:41 +03:00
alesapin
c1f38752c4
Merge pull request #31859 from JackyWoo/keeper_timeout 
keeper session timeout doesn't work
 2021-11-28 16:28:17 +03:00
Raúl Marín
051dddd8df Reduce dependencies on ASTIdentifier.h 
Goes from rebuilding 483 objects to 165 when it's modified
 2021-11-26 16:49:40 +01:00
JackyWoo
4c6ad94109 fix ch keeper session timeout doesn't work 2021-11-26 18:17:49 +08:00
Kruglov Pavel
690b460c3d
Fix style 2021-11-24 22:36:36 +03:00
Kruglov Pavel
ad7768e42a
Fix 2021-11-24 22:25:41 +03:00
Kruglov Pavel
b20f4439df
Make better 2021-11-24 22:25:03 +03:00
avogar
1bd8c57908 Fix 2021-11-24 20:21:21 +03:00
avogar
7c84d95908 Fix possible Logical error: Cannot write to finalized buffer 2021-11-24 18:32:41 +03:00
alesapin
646cf38213
Merge pull request #28981 from JackyWoo/add_4_letter_words_commands 
Add four letter commands to keeper
 2021-11-24 13:32:36 +03:00
Kruglov Pavel
b63b47f0f0
Merge pull request #31265 from Avogar/fix-write-buffers 
Fix and refactor WriteBiffer-s a little
 2021-11-23 16:46:09 +03:00
Kruglov Pavel
6d942401bb
Merge pull request #31510 from azat/fix-progress-for-insert 
Fix progress for short INSERT SELECT queries
 2021-11-23 16:37:06 +03:00
alesapin
16f034fe4c Flush bytes after 4 letter commands 2021-11-23 13:31:10 +03:00
alesapin
5825242a7d Merge branch 'add_4_letter_words_commands' of github.com:JackyWoo/ClickHouse into JackyWoo_add_4_letter_words_commands 2021-11-22 18:02:22 +03:00
alesapin
35de9e42d9
Merge branch 'master' into add_4_letter_words_commands 2021-11-22 18:02:02 +03:00
alesapin
1cf204c581 Small changes 2021-11-21 15:28:45 +03:00
Alexander Tokmakov
856502fa81 log long operations in Keeper 2021-11-20 20:39:31 +03:00
Kruglov Pavel
d9c1a0c8ec
Merge branch 'master' into fix-write-buffers 2021-11-20 17:48:24 +03:00
Azat Khuzhin
dd7f780978 Fix Progress for short INSERT SELECT queries 
Note, that we cannot send Progress for regular INSERT (w/ VALUES) w/o
breaking protocol compatibility, but it can be done by increasing
revision.
 2021-11-20 16:41:35 +03:00
Azat Khuzhin
0b4d855c08 Reduce copy-paste in TCPHandler::processInsertQuery() 2021-11-20 16:41:35 +03:00
alesapin
3f8f08e816 Slightly better 2021-11-19 12:30:58 +03:00
alesapin
dcec086573 Small refactoring 2021-11-18 23:17:22 +03:00
JackyWoo
27d50edc59 adjust log output 2021-11-18 15:49:43 +08:00
JackyWoo
007366b506 remove sizeOf method from SnapshotableHashTable 2021-11-18 12:35:32 +08:00
Kruglov Pavel
f27fcf8372
Merge pull request #31004 from Avogar/output-formats 
Small refactoring in formats
 2021-11-17 20:44:46 +03:00
alesapin
5f3eca4cd1 Merge branch 'master' into JackyWoo_add_4_letter_words_commands 2021-11-17 12:59:20 +03:00
Nikolay Degterinsky
ba50e8387d Merge branch 'master' into query_parameters 2021-11-15 19:55:54 +00:00
JackyWoo
a2f3337ca1 code style change 2021-11-12 20:48:42 +08:00
avogar
e5661b9b62 Minor change 2021-11-11 21:09:21 +03:00
avogar
51831afff8 Fix tests 2021-11-11 20:27:23 +03:00
mergify[bot]
58c5981cab
Merge branch 'master' into refactor-pipeline-executor 2021-11-11 16:16:52 +00:00
Nikolay Degterinsky
7b5954b736 Merge branch 'master' into query_parameters 2021-11-11 11:43:00 +00:00
Nikolai Kochetov
90cf835277 A little bit more refactoring. 2021-11-11 14:41:15 +03:00
avogar
c521a9131a Small refactoring of WriteBiffer-s 2021-11-11 02:11:18 +03:00
Artur
4aaa034c8e Correct behaviour if certificate or key was not set 2021-11-10 16:56:07 +00:00
Artur Filatenkov
744d50f328 merge with master 2021-11-10 18:35:35 +03:00
Artur Filatenkov
68bc8d63a1 Working version 2021-11-10 15:55:57 +03:00
JackyWoo
1cbe9f6024 fix build error 2021-11-10 10:50:12 +08:00
JackyWoo
c71fb3337f add more 4lwd commands 2021-11-09 17:39:28 +08:00
Dmitry Novik
0cd858263f Update protocol version 2021-11-08 16:43:34 +03:00
Dmitry Novik
a093395b9b Process incremental profile events on client 2021-11-08 16:38:31 +03:00
Dmitry Novik
8708e00b3a Send incremental profile events to client 2021-11-08 12:40:31 +03:00
Artur Filatenkov
d8510583d0 resolve conflict 2021-11-05 17:07:29 +03:00
avogar
2dd4393ca1 Small refactoring in formats 2021-11-03 20:07:05 +03:00
Vitaly Baranov
ab01b9afc8 Split Authentication.h to common and main parts. 2021-11-01 19:13:49 +03:00
Vitaly Baranov
3ed7f8f0b3 Move access-rights' source files needed for parser to a separate target. 2021-11-01 19:13:49 +03:00
Alexey Milovidov
993f3faefd Miscellaneous 2021-10-31 18:11:46 +03:00
alesapin
30ab4b3a7f
Update KeeperTCPHandler.cpp 2021-10-30 16:02:05 +03:00
alesapin
40bffcec3a Fix error message in Keeper 2021-10-30 15:32:29 +03:00
JackyWoo
694306ee62 add 4lw white list to keeper 2021-10-29 17:54:25 +08:00
Alexey Milovidov
8b4a6a2416 Remove cruft 2021-10-28 02:10:39 +03:00
JackyWoo
e5b0eedd31 adjust code style for keeper 4lw cmd 2021-10-27 22:26:53 +08:00
JackyWoo
a60663e33d add 4lw commands to keeper 2021-10-27 20:26:42 +08:00
Nikolay Degterinsky
677eb5bbda Fix tests 2021-10-21 15:53:02 +03:00
mergify[bot]
69667aab1a
Merge branch 'master' into fix-backward-compatibility-after-21196 2021-10-19 10:11:16 +00:00
Ilya Yatsishin
4396daeb1f
Merge pull request #30064 from azat/client-print-profile-events 2021-10-19 12:42:09 +03:00
Azat Khuzhin
f7b76373ce Move ProfileEvents packet type from TCPHandler into ProfileEventsExt 2021-10-19 00:54:38 +03:00
alesapin
f1fe96e194 Merge branch 'master' into debug_keeper 2021-10-18 10:16:07 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
Nikolai Kochetov
067eaadadd Merge branch 'master' into removing-data-streams-folder 2021-10-16 09:46:05 +03:00
Nikolai Kochetov
c668696047
Merge pull request #30171 from ClickHouse/remove-stream-interfaces 
Remove stream interfaces
 2021-10-16 09:34:01 +03:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
Nikolai Kochetov
ab28c6c855 Remove BlockInputStream interfaces. 2021-10-14 13:25:43 +03:00
Maksim Kita
8ed4e73705
Merge pull request #30000 from ClickHouse/profile-snapshot 
Make read of Counters snapshot non-atomic
 2021-10-14 13:22:44 +03:00
alesapin
90ff7f05fd Start keeper asynchronously if has connection to other nodes 2021-10-14 13:21:41 +03:00
Nikolai Kochetov
3d3e143c29
Merge pull request #30001 from ClickHouse/remove-streams-from-formats 
Remove streams from formats.
 2021-10-14 12:39:07 +03:00
Maksim Kita
3d2e1a24d9 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
04047f76c7 Fixed tests 2021-10-14 00:33:18 +03:00
Nikolai Kochetov
49ac602ee6 Fix GRPC 2021-10-13 15:06:56 +03:00
Nikolai Kochetov
06d9fc602e Merge branch 'master' into fix-backward-compatibility-after-21196 2021-10-12 14:41:23 +03:00
Dmitry Novik
8be70bc417 Update profile event forwarding 2021-10-12 14:15:07 +03:00
Vitaly Baranov
4a4d913cfd
Merge pull request #29954 from vitlibar/fix-flaky-test_grpc_protocol 
gRPC: Fix releasing query ID and session ID at the end of query processing
 2021-10-12 13:32:27 +03:00
Nikolai Kochetov
ec18340351 Remove streams from formats. 2021-10-11 19:11:50 +03:00
Dmitry Novik
bfdd34c13d code cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
9590c97a19 Create ProfileEventsQueue in GRPCServer 2021-10-11 17:39:24 +03:00
Dmitry Novik
3a0764634e Update InternalProfileEventsQueue usage 2021-10-11 17:39:24 +03:00
Dmitry Novik
0bdabf46f2 Send ProfileEvents only to supported clients 2021-10-11 17:39:24 +03:00
Dmitry Novik
9f9af28b5e Output memory usage with progress 2021-10-11 17:39:24 +03:00
Dmitry Novik
73df6190df Cleanup code 2021-10-11 17:39:24 +03:00
Dmitry Novik
1d2e2d7305 cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
7e3caf96be Fix cores approximation 2021-10-11 17:39:24 +03:00
Dmitry Novik
9071a7151e Fix communication & race conditions 2021-10-11 17:39:23 +03:00
Dmitry Novik
356723427d WIP on ProfileEvents forwarding 2021-10-11 17:39:23 +03:00
Dmitry Novik
74cdaba7fa WIP on profile events forwarding 2021-10-11 17:29:50 +03:00
Dmitry Novik
803b8623c1 Fix TCPHandler::sendProfileEvents 2021-10-11 17:29:50 +03:00
Dmitry Novik
e9b1e05461 Send profile events from all threads of current group 2021-10-11 17:29:50 +03:00
Dmitry Novik
362bcb2f66 Introduce ProfileEvents packet 2021-10-11 17:29:50 +03:00
Nikolai Kochetov
a95c28ec4b
Merge pull request #29898 from ClickHouse/remove-native-stream 
Remove some more streams.
 2021-10-10 21:01:16 +03:00
Vitaly Baranov
1dda596689 Fix releasing query ID and session ID at the end of query processing. 2021-10-10 17:34:45 +03:00
alexey-milovidov
ee577e1ab4
Merge pull request #29155 from FArthur-cmd/add_CORS 
Add Cors support
 2021-10-09 20:18:17 +03:00
Artur
a853cd57ca refactoring 2021-10-09 12:56:00 +00:00
Artur
12f59f2dca refactor 2021-10-09 12:52:20 +00:00
Artur
473f7bee3a refactor 2021-10-09 12:16:37 +00:00
Nikolai Kochetov
c6bce1a4cf Update Native. 2021-10-08 20:21:19 +03:00
Nikolai Kochetov
78e1db209f
Remove more data streams (#29491) 
* Remove more streams.

* Fixing build.

* Fixing build.

* Rename files.

* Fix fast test.

* Fix StorageKafka.

* Try fix kafka test.

* Move createBuffer to KafkaSource ctor.

* Revert "Move createBuffer to KafkaSource ctor."

This reverts commit 81fa94d27e.

* Revert "Try fix kafka test."

This reverts commit 2107e54969.

* Comment some rows in test.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-10-07 11:26:08 +03:00
alexey-milovidov
16ad5953d6
Merge pull request #29783 from azat/tcphandler-progress-race 
Fix data-race between fatal error handler and progress packets
 2021-10-07 01:47:29 +03:00
Filatenkov Artur
545673248a
Merge branch 'master' into add_CORS 2021-10-06 16:09:34 +03:00
Filatenkov Artur
a9e8ba0c00
Correct behaviour with unknown methods (#29057) 
* correct behaviour with unknown methods

* move check on earlier stage

* add test

* correct filter for handler

* Update HTTPServerConnection.cpp

* remove trying to get params

* correct filter for head requests

* Update HTTPHandlerFactory.h

* Update HTTPHandlerFactory.h

* Trigger Build
 2021-10-06 14:43:05 +03:00
Azat Khuzhin
4f5118b85f Fix data-race between fatal error handler and progress packets 
This is not that important, but because of packets overlaps the client
may not render the fatal error.

You will find TSan report in `details`.

<details>

```
WARNING: ThreadSanitizer: data race (pid=13384)
  Write of size 8 at 0x7b24000ad818 by thread T47:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x16124d61)
    1 DB::TCPHandler::sendProgress() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1602:10 (clickhouse-tsan+0x16124d61)
    2 DB::TCPHandler::runImpl()::$_6::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:318:25 (clickhouse-tsan+0x1612cf54)

  Previous write of size 8 at 0x7b24000ad818 by thread T229:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x1612b793)
    1 DB::TCPHandler::sendLogData(DB::Block const&) obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1561:10 (clickhouse-tsan+0x1612b793)
    2 DB::TCPHandler::sendLogs() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1632:9 (clickhouse-tsan+0x161228ab)
    3 DB::TCPHandler::runImpl()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:229:62 (clickhouse-tsan+0x1612c4e1)
```

</details>
 2021-10-05 21:21:32 +03:00
Artur Filatenkov
9b1a39fdb9 refactor after move common to base 2021-10-05 17:43:33 +03:00
Filatenkov Artur
60d84df56a
Merge branch 'ClickHouse:master' into add_CORS 2021-10-05 17:27:13 +03:00
Filatenkov Artur
82e6ac8fa2
Update HTTPHandler.cpp 2021-10-05 13:39:18 +03:00
Nikolai Kochetov
e2d07fbdf5
Merge branch 'master' into fix-backward-compatibility-after-21196 2021-10-04 17:35:32 +03:00
Nikolai Kochetov
6169b180cb Fix backward compatibility after #21196. 2021-10-04 17:27:44 +03:00
Maksim Kita
8eda58acfb
Merge pull request #29516 from lingtaolf/feature/support-HSTS 
Support HSTS in Clickhouse HTTP server
 2021-10-04 12:49:51 +03:00
Filatenkov Artur
8df228e6b8
Merge branch 'master' into add_CORS 2021-10-04 10:55:09 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Filatenkov Artur
8959b348a1
Merge branch 'ClickHouse:master' into add_CORS 2021-10-01 14:42:46 +03:00
Azat Khuzhin
e8a90b8ff2 Do not allow to reuse previous credentials in case of inter-server secret 
Before this patch INSERT via Buffer/Kafka may re-use previously set user
for that connection, while this is not correct, it should reset the
user, and use global context.

Note, before [1] there was a fallback to default user, but that code had
been removed, and now it got back.

  [1]: 0159c74f21 ("Secure inter-cluster query execution (with initial_user as current query user) [v3]")

Also note, that context for Buffer table (and others) cannot be changed,
since they don't have any user only profile.

I've tested this patch manually using the following:

    create table dist (key Int) engine=Distributed(test_cluster_two_shards_secure, default, data, key);
    create table buffer (key Int) engine=Buffer(default, dist, 1, 0, 0, 0, 0, 0, 0);
    create table data (key Int) engine=Memory();

    # to start the connection with readonly user
    $ clickhouse-client --user readonly -q 'select * from dist'
    $ clickhouse-client -q 'insert into buffer values (1)'
    # before this patch this produces errors like:
    # 2021.09.27 23:46:48.384920 [ 19474 ] {} <Error> default.dist.DirectoryMonitor: Code: 164. DB::Exception: Received from 127.0.0.2:9000. DB::Exception: readonly: Cannot execute query in readonly mode. Stack trace:

v2: reset the authentication instead of using default user (as suggested by @vitlibar)
v3: reset Session::user and introduce ClientInfo::resetAuthentication (as suggested by @vitlibar)
v4: reset the session every time in interserver mode (suggested by @vitlibar)
 2021-10-01 01:13:15 +03:00
Maksim Kita
7bc6b8fd70
Merge pull request #28604 from azat/safer-http-handler 
Proper buffer flush while handling HTTP queries (avoids std::terminate)
 2021-09-30 16:44:09 +03:00
凌涛
a13d6b0aea Check all requests 2021-09-30 13:40:48 +08:00
凌涛
bd78af435d update code and add document 2021-09-30 04:51:02 +08:00
Artur
fcebf7b985 correct tests 2021-09-29 11:29:24 +00:00
凌涛
c298fba774 Support HSTS in Clickhouse HTTP server 2021-09-29 16:37:48 +08:00
Nikolai Kochetov
78a7665f43 Merge branch 'master' into rewrite-pushing-to-views 2021-09-27 10:56:50 +03:00
Nikolai Kochetov
09232788c0 Fix GRPC tests. 2021-09-27 10:55:33 +03:00
Azat Khuzhin
b3d1bfc67a Send UNKNOWN_DATABASE to the client (via TCP) 
Before this patch:

    $ clickhouse-client --database foo -q 'select 1'
    Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from localhost:9000. (ATTEMPT_TO_READ_AFTER_EOF)

After:

    $ clickhouse-client --database foo -q 'select 1'
    Received exception from server (version 21.11.1):
    Code: 81. DB::Exception: Received from localhost:9000. DB::Exception: Database foo doesn't exist. (UNKNOWN_DATABASE)
    (query: select 1)

Fixes: #26864 (cc @vitlibar)
 2021-09-26 23:45:29 +03:00
Nikolai Kochetov
efbd8e4911 Remove ExceptionHandlingSink. 2021-09-26 17:54:59 +03:00
Alexey Milovidov
cd7f9d981c Remove ya.make 2021-09-25 04:22:54 +03:00
Filatenkov Artur
7bbd08cb5d
Update HTTPHandler.cpp 2021-09-24 15:40:27 +03:00
Artur
e5c49c2e86 Merge branch 'add_CORS' of https://github.com/FArthur-cmd/ClickHouse into add_CORS 2021-09-22 17:15:54 +00:00
Artur
ce4193fe95 small refactoring 2021-09-22 17:10:15 +00:00
Filatenkov Artur
1fe4555dec
Merge branch 'ClickHouse:master' into add_CORS 2021-09-22 19:55:15 +03:00
Artur
2ad7641e7f activate add_http_cors_header by default. 2021-09-22 16:54:36 +00:00
Nikolai Kochetov
eed4e8c754 Fix progress for insert select. 2021-09-22 16:29:58 +03:00
Artur
36223b5f23 update comments for filter 2021-09-22 12:30:55 +00:00
Artur
2cffa98a60 add test and comments in config 2021-09-22 12:22:21 +00:00
Nikolai Kochetov
81bf13a247 Refactor pushing to views. 2021-09-22 13:57:00 +03:00
Artur
c8892ec7a7 add options support 2021-09-22 10:34:48 +00:00
Nikolai Kochetov
db6f1e198c Add cancell callback to completed executor. 2021-09-21 19:37:32 +03:00
Artur
82a849ba8e add options method 2021-09-18 09:36:02 +00:00
Nikolai Kochetov
a8c3b02598 Merge branch 'master' into rewrite-pushing-to-views 2021-09-17 15:38:11 +03:00
Nikolai Kochetov
58bb5fe462 Fix some tests. 2021-09-17 14:40:03 +03:00
Azat Khuzhin
60013cff1a Do not call finalize() for HTTP buffer multiple times 
Before only non-compressed was flushed, and so after compressed will
write to non-compressed and this will trigger multiple finalize() for
raw buffer, fix this, but adding Output::finalize() that take care of
all buffers.

This should fix CI failure [1] and also not only CI failures but also an
assert() that had been added in the previous commit.

    2021-09-05 15:08:43 00429_long_http_bufferization:                                          [ FAIL ] 6.68 sec. - return code 1
    2021-09-05 15:08:43 cmp: EOF on /tmp/clickhouse-test/0_stateless/test_e4xs61/res3 after byte 1048576, in line 4127
    2021-09-05 15:08:43 , result:

    The problem was with the following query:

        curl -q -sS 'http://localhost:8123/?database=default&log_comment=/src/ch/clickhouse/tests/queries/0_stateless/00429_long_http_bufferization.sh&max_block_size=500000&compress=1&buffer_size=1048576&wait_end_of_query=1' -d 'SELECT greatest(toUInt8(1), toUInt8(intHash64(number))) FROM system.numbers LIMIT 1500000 FORMAT RowBinary' | clickhouse-compressor --decompress | less

    It returns less rows.

  [1]: https://clickhouse-test-reports.s3.yandex.net/28604/1f34fd9a9eaff19fde0199f78b45656ab74f0590/functional_stateless_tests_(release).html#fail1
 2021-09-16 22:36:09 +03:00
Azat Khuzhin
9bb9d5952b Catch write-after-finalize bugs in WriteBufferFromHTTPServerResponse 2021-09-16 22:35:51 +03:00
Azat Khuzhin
7b741c28e4 Do not try to write HTTP response after exception 
CI reports [1]:

    [ 25285 ] {} <Trace> DynamicQueryHandler: Request URI: /?database=test_27&log_comment=/usr/share/clickhouse-test/queries/0_stateless/00429_long_http_bufferization.sh&max_block_size=500000&compress=1&buffer_size=1500000&wait_end_of_query=1
    ...
    [ 25285 ] {ee96eb9c-44cc-4b76-9f30-3cc545a5506f} <Debug> executeQuery: (from [::1]:41578) (comment: /usr/share/clickhouse-test/queries/0_stateless/00429_long_http_bufferization.sh) SELECT greatest(toUInt8(1), toUInt8(intHash64(number))) FROM system.numbers LIMIT 1048576 FORMAT RowBinary
    [ 25285 ] {ee96eb9c-44cc-4b76-9f30-3cc545a5506f} <Information> executeQuery: Read 1500000 rows, 11.44 MiB in 63.924507821 sec., 23465 rows/sec., 183.32 KiB/sec.
    [ 25285 ] {ee96eb9c-44cc-4b76-9f30-3cc545a5506f} <Debug> DynamicQueryHandler: Done processing query
    ...
    [ 455 ] {} <Fatal> BaseDaemon: Code: 24. DB::Exception: Cannot write to ostream at offset 2097152. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):
    [ 455 ] {} <Fatal> BaseDaemon:
    [ 455 ] {} <Fatal> BaseDaemon: 0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x18fa949b in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:59: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x99f38dc in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x9adf185 in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromHTTPServerResponse::nextImpl() @ 0x15b71ec0 in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 4. ./obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:0: DB::WriteBuffer::write(char const*, unsigned long) @ 0x99be96a in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 5. ./obj-x86_64-linux-gnu/../src/Compression/CompressedWriteBuffer.cpp:36: DB::CompressedWriteBuffer::nextImpl() @ 0x1436ca4d in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 6. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::CompressedWriteBuffer::~CompressedWriteBuffer() @ 0x1436cd8f in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 7. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2615: std::__1::__shared_ptr_emplace<DB::CompressedWriteBuffer, std::__1::allocator<DB::CompressedWriteBuffer> >::__on_zero_shared() @ 0x15a88ee8 in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2518: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x15b0566b in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Server/HTTP/HTTPServerConnection.cpp:0: DB::HTTPServerConnection::run() @ 0x15b6ba61 in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x18ec71a3 in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:114: Poco::Net::TCPServerDispatcher::run() @ 0x18ec7930 in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/ScopedLock.h:36: Poco::PooledThread::run() @ 0x1903e042 in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x1903c4d0 in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/SharedPtr.h:277: Poco::ThreadImpl::runnableEntry(void*) @ 0x1903ac48 in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 15. __tsan_thread_start_func @ 0x990a76d in /usr/bin/clickhouse
    [ 455 ] {} <Fatal> BaseDaemon: 16. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    [ 455 ] {} <Fatal> BaseDaemon: 17. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    [ 455 ] {} <Fatal> BaseDaemon:  (version 21.10.1.7997)
    [ 25878 ] {} <Fatal> BaseDaemon: ########################################
    [ 25878 ] {} <Fatal> BaseDaemon: (version 21.10.1.7997, build id: DB9812D7EA89EB06D2C7243F5C8BD293AA9BA37D) (from thread 25285) (query_id: ee96eb9c-44cc-4b76-9f30-3cc545a5506f) Received signal Aborted (6)
    [ 25878 ] {} <Fatal> BaseDaemon:
    [ 25878 ] {} <Fatal> BaseDaemon: Stack trace: 0x7efe2a35818b 0x7efe2a337859 0x990ff6e 0x13fa707d 0x1bb17614 0x1bb17507 0x999c97b 0x1436cf11 0x15a88ee8 0x15b0566b 0x15b6ba61 0x18ec71a3 0x18ec7930 0x1903e042 0x1903c4d0 0x1903ac48 0x990a76d 0x7efe2a50d609 0x7efe2a434293
    [ 25878 ] {} <Fatal> BaseDaemon: 5. raise @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    [ 25878 ] {} <Fatal> BaseDaemon: 6. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    [ 450 ] {} <Fatal> Application: Child process was terminated by signal 6.

  [1]: https://clickhouse-test-reports.s3.yandex.net/28604/d6e86dab2754dbea3ed19d9ae72276e7d161aef1/stress_test_(thread).html#fail1
 2021-09-16 22:35:51 +03:00
Azat Khuzhin
230f29ac4e Always flush buffer while handling HTTP queries 
This will avoid possible std::terminate, like in [1]:

  [1]: https://github.com/ClickHouse/ClickHouse/pull/24023#issuecomment-842979875
 2021-09-16 22:35:50 +03:00
Azat Khuzhin
47ad10061b Reduce scope for HTTPHandler::trySendExceptionToClient() 2021-09-16 22:35:50 +03:00