Commit Graph

182 Commits

Author SHA1 Message Date
Vitaly Baranov
f80a3e3247
Merge pull request #30549 from aiven/kmichel-server-reload
Start/stop servers when `listen_host`/`*_port` changes
2021-12-27 14:51:00 +03:00
Kevin Michel
ffc1fca296
Start/stop servers when listen_host/*_port changes
This allows starting and stopping separately each protocol server
without restarting ClickHouse.

This also allows adding or removing `listen_host` entries, which
start and stops servers for all enabled ports.

When stopping a server, the listening socket is immediately closed
(and available for another server).

Protocols with persistent connections try to wait for any currently
running query to finish before closing the connection, but idle
connection are closed quickly (depending on how often the protocol
is polled).

An extra ProfileEvent is added, `MainConfigLoads`, it is
incremented every time the configuration is reloaded. This helps
when trying to assess whether the new configuration was applied.
2021-12-24 08:26:02 +01:00
Azat Khuzhin
6aebc3e94c Do not loose ProfileEvents in case of thread destroyed before
v2: drop std::move and add copy ctor for ProfileEvents::Counter::Snapshot
v2: remove std::move
2021-12-19 23:43:05 +03:00
Alexey Milovidov
2c68192bb6 Fix strange code in TCPHandler 2021-12-12 05:35:33 +03:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
alexey-milovidov
96ec92c7cd
Merge pull request #31155 from ClickHouse/incremental-profile-events
Send incremental profile events to client
2021-12-01 04:04:41 +03:00
Azat Khuzhin
dd7f780978 Fix Progress for short INSERT SELECT queries
Note, that we cannot send Progress for regular INSERT (w/ VALUES) w/o
breaking protocol compatibility, but it can be done by increasing
revision.
2021-11-20 16:41:35 +03:00
Azat Khuzhin
0b4d855c08 Reduce copy-paste in TCPHandler::processInsertQuery() 2021-11-20 16:41:35 +03:00
Dmitry Novik
0cd858263f Update protocol version 2021-11-08 16:43:34 +03:00
Dmitry Novik
a093395b9b Process incremental profile events on client 2021-11-08 16:38:31 +03:00
Dmitry Novik
8708e00b3a Send incremental profile events to client 2021-11-08 12:40:31 +03:00
Alexey Milovidov
993f3faefd Miscellaneous 2021-10-31 18:11:46 +03:00
Alexey Milovidov
8b4a6a2416 Remove cruft 2021-10-28 02:10:39 +03:00
mergify[bot]
69667aab1a
Merge branch 'master' into fix-backward-compatibility-after-21196 2021-10-19 10:11:16 +00:00
Azat Khuzhin
f7b76373ce Move ProfileEvents packet type from TCPHandler into ProfileEventsExt 2021-10-19 00:54:38 +03:00
Nikolai Kochetov
067eaadadd Merge branch 'master' into removing-data-streams-folder 2021-10-16 09:46:05 +03:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
Nikolai Kochetov
06d9fc602e Merge branch 'master' into fix-backward-compatibility-after-21196 2021-10-12 14:41:23 +03:00
Dmitry Novik
8be70bc417 Update profile event forwarding 2021-10-12 14:15:07 +03:00
Dmitry Novik
bfdd34c13d code cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
3a0764634e Update InternalProfileEventsQueue usage 2021-10-11 17:39:24 +03:00
Dmitry Novik
0bdabf46f2 Send ProfileEvents only to supported clients 2021-10-11 17:39:24 +03:00
Dmitry Novik
9f9af28b5e Output memory usage with progress 2021-10-11 17:39:24 +03:00
Dmitry Novik
73df6190df Cleanup code 2021-10-11 17:39:24 +03:00
Dmitry Novik
1d2e2d7305 cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
7e3caf96be Fix cores approximation 2021-10-11 17:39:24 +03:00
Dmitry Novik
9071a7151e Fix communication & race conditions 2021-10-11 17:39:23 +03:00
Dmitry Novik
356723427d WIP on ProfileEvents forwarding 2021-10-11 17:39:23 +03:00
Dmitry Novik
803b8623c1 Fix TCPHandler::sendProfileEvents 2021-10-11 17:29:50 +03:00
Dmitry Novik
e9b1e05461 Send profile events from all threads of current group 2021-10-11 17:29:50 +03:00
Dmitry Novik
362bcb2f66 Introduce ProfileEvents packet 2021-10-11 17:29:50 +03:00
Nikolai Kochetov
c6bce1a4cf Update Native. 2021-10-08 20:21:19 +03:00
Nikolai Kochetov
78e1db209f
Remove more data streams (#29491)
* Remove more streams.

* Fixing build.

* Fixing build.

* Rename files.

* Fix fast test.

* Fix StorageKafka.

* Try fix kafka test.

* Move createBuffer to KafkaSource ctor.

* Revert "Move createBuffer to KafkaSource ctor."

This reverts commit 81fa94d27e.

* Revert "Try fix kafka test."

This reverts commit 2107e54969.

* Comment some rows in test.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2021-10-07 11:26:08 +03:00
Azat Khuzhin
4f5118b85f Fix data-race between fatal error handler and progress packets
This is not that important, but because of packets overlaps the client
may not render the fatal error.

You will find TSan report in `details`.

<details>

```
WARNING: ThreadSanitizer: data race (pid=13384)
  Write of size 8 at 0x7b24000ad818 by thread T47:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x16124d61)
    1 DB::TCPHandler::sendProgress() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1602:10 (clickhouse-tsan+0x16124d61)
    2 DB::TCPHandler::runImpl()::$_6::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:318:25 (clickhouse-tsan+0x1612cf54)

  Previous write of size 8 at 0x7b24000ad818 by thread T229:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x1612b793)
    1 DB::TCPHandler::sendLogData(DB::Block const&) obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1561:10 (clickhouse-tsan+0x1612b793)
    2 DB::TCPHandler::sendLogs() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1632:9 (clickhouse-tsan+0x161228ab)
    3 DB::TCPHandler::runImpl()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:229:62 (clickhouse-tsan+0x1612c4e1)
```

</details>
2021-10-05 21:21:32 +03:00
Nikolai Kochetov
e2d07fbdf5
Merge branch 'master' into fix-backward-compatibility-after-21196 2021-10-04 17:35:32 +03:00
Nikolai Kochetov
6169b180cb Fix backward compatibility after #21196. 2021-10-04 17:27:44 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Azat Khuzhin
e8a90b8ff2 Do not allow to reuse previous credentials in case of inter-server secret
Before this patch INSERT via Buffer/Kafka may re-use previously set user
for that connection, while this is not correct, it should reset the
user, and use global context.

Note, before [1] there was a fallback to default user, but that code had
been removed, and now it got back.

  [1]: 0159c74f21 ("Secure inter-cluster query execution (with initial_user as current query user) [v3]")

Also note, that context for Buffer table (and others) cannot be changed,
since they don't have any user only profile.

I've tested this patch manually using the following:

    create table dist (key Int) engine=Distributed(test_cluster_two_shards_secure, default, data, key);
    create table buffer (key Int) engine=Buffer(default, dist, 1, 0, 0, 0, 0, 0, 0);
    create table data (key Int) engine=Memory();

    # to start the connection with readonly user
    $ clickhouse-client --user readonly -q 'select * from dist'
    $ clickhouse-client -q 'insert into buffer values (1)'
    # before this patch this produces errors like:
    # 2021.09.27 23:46:48.384920 [ 19474 ] {} <Error> default.dist.DirectoryMonitor: Code: 164. DB::Exception: Received from 127.0.0.2:9000. DB::Exception: readonly: Cannot execute query in readonly mode. Stack trace:

v2: reset the authentication instead of using default user (as suggested by @vitlibar)
v3: reset Session::user and introduce ClientInfo::resetAuthentication (as suggested by @vitlibar)
v4: reset the session every time in interserver mode (suggested by @vitlibar)
2021-10-01 01:13:15 +03:00
Nikolai Kochetov
78a7665f43 Merge branch 'master' into rewrite-pushing-to-views 2021-09-27 10:56:50 +03:00
Azat Khuzhin
b3d1bfc67a Send UNKNOWN_DATABASE to the client (via TCP)
Before this patch:

    $ clickhouse-client --database foo -q 'select 1'
    Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from localhost:9000. (ATTEMPT_TO_READ_AFTER_EOF)

After:

    $ clickhouse-client --database foo -q 'select 1'
    Received exception from server (version 21.11.1):
    Code: 81. DB::Exception: Received from localhost:9000. DB::Exception: Database foo doesn't exist. (UNKNOWN_DATABASE)
    (query: select 1)

Fixes: #26864 (cc @vitlibar)
2021-09-26 23:45:29 +03:00
Nikolai Kochetov
eed4e8c754 Fix progress for insert select. 2021-09-22 16:29:58 +03:00
Nikolai Kochetov
81bf13a247 Refactor pushing to views. 2021-09-22 13:57:00 +03:00
Nikolai Kochetov
db6f1e198c Add cancell callback to completed executor. 2021-09-21 19:37:32 +03:00
Nikolai Kochetov
a8c3b02598 Merge branch 'master' into rewrite-pushing-to-views 2021-09-17 15:38:11 +03:00
Nikolai Kochetov
58bb5fe462 Fix some tests. 2021-09-17 14:40:03 +03:00
Nikolai Kochetov
341553febd Fix build. 2021-09-16 20:40:42 +03:00
Nikolai Kochetov
e616732743 Small refactoring. 2021-09-15 22:35:48 +03:00
Nikolai Kochetov
3a9d88fbc9 Fix more tests. 2021-09-10 17:52:24 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
Nikolai Kochetov
66a76ab70f Rewrite PushingToViewsBlockOutputStream part 6 2021-09-03 20:29:36 +03:00