thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
157,971 Commits 9,338 Branches 1,783 Tags 2.1 GiB
4b46065bcd
Commit Graph

574 Commits

Author SHA1 Message Date
Robert Schulze
cde10fe7b5
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-reenable-checks 2023-09-26 18:59:41 +00:00
George Gamezardashvili
0ce30ab6d5
SSH keys authentication (#41109) 
Added new type of authentication based on SSH keys. It works only for Native TCP protocol.

Co-authored-by: Nikita Mikhaylov <nikitamikhaylov@clickhouse.com>
Co-authored-by: Robert Schulze <robert@clickhouse.com>
 2023-09-26 17:50:19 +02:00
Robert Schulze
9fff447716
Re-enable clang-tidy checks 2023-09-26 09:34:12 +00:00
Nikolai Kochetov
9b936c44db
Revert "Revert "Add settings for real-time updates during query execution"" 2023-09-09 12:29:39 +02:00
Alexey Milovidov
03a755732a
Revert "Add settings for real-time updates during query execution" 2023-09-09 03:10:23 +03:00
Nikolai Kochetov
0095124791
Merge pull request #48607 from alexX512/master 
Add settings for real-time updates during query execution
 2023-09-08 09:05:33 +02:00
Raúl Marín
e192d4c624
Fix summary reporting with parallel replicas with LIMIT (#53050) 2023-09-04 15:53:06 +02:00
alexX512
36b8932b37 Merge branch 'master' of github.com:ClickHouse/ClickHouse 2023-08-31 21:40:50 +00:00
Alexey Gerasimchuck
f7d1041e61 minor improvement 2023-08-30 06:14:39 +00:00
Alexey Gerasimchuck
be2f80cc1c minor corrections 2023-08-29 23:59:36 +00:00
Alexey Gerasimchuck
a1687e618b backward compatibility changes + fixes 2023-08-29 23:50:41 +00:00
alexX512
b3ba00f443 Merge branch 'master' of github.com:ClickHouse/ClickHouse 2023-08-29 10:59:29 +00:00
Alexey Gerasimchuck
e9757abea6 Added validate_tcp_client_information setting 2023-08-29 00:15:55 +00:00
Alexey Gerasimchuck
3140958132 Added client_info validation 2023-08-22 03:52:57 +00:00
Alexander Tokmakov
78d8557a56 fix 2023-08-15 15:37:09 +02:00
Alexey Milovidov
fa9abc5038 Better usage of ccache 2023-08-09 05:02:50 +02:00
alexX512
0d84226914 Merge branch 'master' of github.com:ClickHouse/ClickHouse 2023-08-02 19:18:59 +00:00
Alexey Gerasimchuck
e58b3cfd32 Implemented max sessions for user 2023-07-30 22:09:03 +00:00
alexX512
c403f56e09 Merge branch 'master' of github.com:ClickHouse/ClickHouse 2023-07-21 17:56:53 +00:00
Vitaly Baranov
815a3857de Remove non-const function Context::getClientInfo(). 2023-07-17 15:02:07 +02:00
Robert Schulze
271297823a
Allow var-int encoded 64-bit integers with MSB=1 
Resolves: #51486

Until now, it was illegal to encode 64-bit (unsigned) integers with
MSB=1, i.e. values > (1ULL<<63) - 1, as var-int. In more detail, the
var-int code used by ClickHouse server and client spent at most 9 bytes
per value such that 9 * 7 = 63 bits could be encoded. Some 3rd party
clients (e.g. Rust clickhouse-rs) had the same limitation, whereas other
clients understand the full range (Python clickhouse-driver).

PRs #47608 and #48628 added sanity checks as asserts or exceptions
during var-int encoding on the server side. This was considered okay as
such huge integers so far occurred only during testing (usually fuzzing)
but not in practice.

Issue #51486 is a new fuzzing issue where the exception thrown from the
sanity check led to a half-baked progress packet and as a result, a
logical error / server crash.

The only fix which is not another bandaid is to allow the full range in
var-int coding. Clients will have to allow the full range too, a note
will be added to the changelog. (the alternative was to create another
protocol version but as var-int is used all over the place this was
considered infeasible)

Review note: this is the relevant commit.
 2023-07-06 20:23:23 +00:00
Robert Schulze
43c97d76bf
Merge pull request #44149 from zvonand/zvonand-implicit-tz 
Session time zones
 2023-06-23 14:28:49 +02:00
Alexey Milovidov
75ef844f99
Fix long test functions_bad_arguments (#51310) 
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-06-23 13:51:40 +02:00
Andrey Zvonov
b4f2e4020d
Merge branch 'master' into zvonand-implicit-tz 2023-06-21 13:41:35 +02:00
Robert Schulze
1aa158909e
enable_qpl_deflate_codec --> enable_deflate_qpl_codec 2023-06-09 12:43:33 +00:00
jinjunzh
f1192d59af
refine patch according to comments 2023-06-09 12:43:15 +00:00
jinjunzh
056ca4f555
Add extensive testing cases for deflate qpl codec 2023-06-09 12:42:59 +00:00
Andrey Zvonov
d95be4162f
Merge branch 'master' into zvonand-implicit-tz 2023-06-08 18:34:45 +03:00
Antonio Andelic
b11f744252
Correctly disable async insert with deduplication when it's not needed (#50663) 
* Correctly disable async insert when it's not used

* Better

* Add comment

* Better

* Fix tests

---------

Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-06-07 20:33:08 +02:00
zvonand
d4cec1f0e0 fix client thread attachment + make better tests 2023-05-30 01:30:01 +02:00
Andrey Zvonov
9237b904be
Merge branch 'master' into zvonand-implicit-tz 2023-05-10 01:25:33 +02:00
alexX512
f3f9e29509 Merge branch 'master' of github.com:ClickHouse/ClickHouse 2023-05-06 14:35:29 +00:00
Dmitry Novik
2536dfad3a Fix build 2023-05-05 18:21:25 +00:00
Dmitry Novik
951bef4fd2 Fixmemory counting in TCPHandler 2023-05-05 15:10:50 +00:00
zvonand
8338d54c34 Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-implicit-tz 2023-05-05 15:50:26 +02:00
alexX512
2cd1b37f9b Add support for connection of partial result processors with multiple ports 2023-05-02 07:26:36 +00:00
Alexey Milovidov
ded4b17d2c
Merge pull request #48989 from evillique/fix-ssl 
Add fallback to password auth after failed SSL auth
 2023-04-21 21:42:59 +03:00
Nikolay Degterinsky
fb3c0f05df Add test 2023-04-21 14:22:14 +00:00
Andrey Zvonov
984e8fde41
Merge branch 'master' into zvonand-implicit-tz 2023-04-20 23:13:17 +02:00
Nikolay Degterinsky
83f342e94a Fallback to password auth after failed SSL auth 2023-04-20 20:02:05 +00:00
zvonand
22be85d976 renamed setting 2023-04-18 02:42:30 +02:00
Alexander Tokmakov
cad0a5f493 fix a confusing warning about interserver mode 2023-04-14 19:46:44 +02:00
zvonand
bac5fbc3d2 fix error on connection drop after 1st query 2023-04-13 13:26:09 +02:00
zvonand
73675cd8d2 tryfix fasttest 2023-04-12 19:17:48 +02:00
zvonand
3f8956f854 remove additional logging 2023-04-12 17:45:11 +02:00
zvonand
2a9f28b73b resolve 2023-04-12 12:54:39 +02:00
zvonand
5d18343fb8 fixed delay 2023-04-12 00:15:07 +02:00
Vasily Nemkov
e36addb96a Hackish way of setting up timezone on the client 
Warning: lots of debug logging
 2023-04-11 13:03:03 +02:00
Alexander Tokmakov
230adac9f6 fixes 2023-04-01 18:23:59 +02:00
Robert Schulze
eb93ec35f5
Merge pull request #48154 from azat/varuint-v2 
Add sanity checks for writing number in variable length format (resubmit)
 2023-03-31 10:59:21 +02:00
Nikolai Kochetov
f870e6f7a9 Rename setting stop_reading_on_first_cancel to partial_result_on_first_cancel. 2023-03-30 14:00:44 +00:00
Azat Khuzhin
9457b1cc46 Use VAR_UINT_MAX for unknown_packet_in_send_data 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-29 11:02:33 +02:00
Sergei Trifonov
273770519b
Merge branch 'master' into master 2023-03-20 18:28:10 +01:00
Sema Checherinda
0b641fcead
Merge pull request #47564 from CheSema/remove-redundant 
remove counters for threads, fix negative counters
 2023-03-18 02:23:52 +01:00
Nikolay Degterinsky
7fb171f84d
Merge pull request #47596 from evillique/native-client-ssl 
Add SSL User Certificate authentication to the native protocol
 2023-03-17 21:27:58 +01:00
alexX512
023c0ba46e Fix comments 2023-03-17 20:05:10 +00:00
alexX512
6ea522adee Style fix 2023-03-16 14:23:17 +00:00
alexX512
cf75dd2595 Remove sleep_in_receive_cancel 2023-03-16 04:39:11 +00:00
Sema Checherinda
67ac858a52 remove counters for threads, fix negative counters 2023-03-15 22:22:22 +01:00
Andrey Zvonov
4327d707e0
Merge branch 'master' into zvonand-implicit-tz 2023-03-15 22:18:52 +01:00
alexX512
f32df219ef Merge branch 'master' of github.com:alexX512/ClickHouse 2023-03-15 19:54:24 +00:00
alexX512
1f21ac8fcf Fix fast test 2023-03-15 19:53:58 +00:00
Alexey Perevyshin
1e9ef12f51
Merge branch 'master' into master 2023-03-15 22:21:02 +04:00
zvonand
bbb31cf891 added validation on setting modification 2023-03-15 18:37:23 +01:00
alexX512
0088c18192 Fix fasttest error 2023-03-15 13:37:47 +00:00
alexX512
cbbdb5aabc Fix typos 2023-03-15 13:05:38 +00:00
alexX512
d47e10484f Merge branch 'master' of github.com:ClickHouse/ClickHouse 2023-03-15 11:14:34 +00:00
Azat Khuzhin
bcf381c5ae Reimplement interserver mode to avoid replay attacks 
Prevous implementation (DBMS_MIN_REVISION_WITH_INTERSERVER_SECRET)
accepts the salt from the client, which make it useless.

Reimplement the protocol to send the salt by the server and use it in
the client instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-15 08:28:43 +01:00
alexX512
14a7c5a019 Add stop_reading_on_first_cancel setting 2023-03-15 06:06:55 +00:00
Nikolay Degterinsky
398978e0bc Fix build 2023-03-15 00:09:29 +00:00
Nikolay Degterinsky
f3908a9e84 Fix style 2023-03-14 22:49:10 +00:00
Nikolay Degterinsky
eddda2eb73 Add SSL authentication to the native protocol 2023-03-14 22:10:08 +00:00
Alexey Milovidov
a411ae10b5
Merge pull request #47486 from ClickHouse/revert_25674 
Remove "asynchronous connection drain"
 2023-03-13 21:40:39 +03:00
Vladimir C
9cb018545c
Merge pull request #47499 from FrankChen021/span_kind 2023-03-13 11:35:03 +01:00
Frank Chen
0bdd645049 SET SpanKind for HTTP/TCP/GRPC handler 2023-03-12 21:31:08 +08:00
Alexander Tokmakov
7b1b238d0b Revert "Merge pull request #25674 from amosbird/distributedreturnconnection" 
This reverts commit 5ffd99dfd4, reversing
changes made to 2796aa333f.
 2023-03-11 19:09:47 +01:00
zvonand
1ce697d8c0 Revert "revert protocol changes, found better way" 
This reverts commit 3a918ae66a.
 2023-03-07 16:05:23 +01:00
Kruglov Pavel
9e64441353
Merge pull request #47214 from azat/interserver-mode-address 
Add real client (initiator server) address into the logs for interserver mode
 2023-03-07 12:40:59 +01:00
zvonand
5e7a861e68 fix 2023-03-07 02:45:47 +01:00
zvonand
3a918ae66a revert protocol changes, found better way 2023-03-07 02:33:46 +01:00
zvonand
f2fbf2d61e tcp protocol modification (min revision to be updated) 2023-03-06 02:52:05 +01:00
Alexander Tokmakov
082194a755
Merge pull request #47175 from ClickHouse/correct_fatal_handling 
More correct handling of fatal errors
 2023-03-04 00:38:00 +03:00
Azat Khuzhin
5403360924 Add real client (initiator server) address into the logs for interserver mode 
It is useful to understand what is going on, in some obscure cases, for
instance if someone will copy configuration from the production to some
docker env, and then you will see docker's private network addresses
in the logs.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-03 16:38:10 +01:00
Alexander Tokmakov
619b282060 more correct handling of fatal errors 2023-03-03 01:22:04 +01:00
Alexander Tokmakov
edd238273c fix another bug in client 2023-03-02 19:51:05 +01:00
Alexander Tokmakov
a97e15e36f Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-03-01 14:26:41 +01:00
Alexander Tokmakov
cad1e0b768 fix 2023-02-25 01:18:34 +01:00
Alexey Milovidov
f28ab14759 Better handling of fatal errors 2023-02-24 23:29:06 +01:00
Alexander Tokmakov
fe92fd8a61 Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-02-23 22:35:31 +01:00
Alexander Tokmakov
e660c0838c fix multipart requests 2023-02-22 17:54:35 +01:00
Alexander Tokmakov
592af6d652 fix incomplete interst through http 2023-02-22 02:34:03 +01:00
Alexander Tokmakov
f5c232707d cancel insert queries correctly 2023-02-21 22:04:39 +01:00
Alexey Milovidov
d8cda3dbb8 Remove PVS-Studio 2023-02-19 23:30:05 +01:00
Azat Khuzhin
6f7b6e9206 Remove extra try/catch for QueryState reset 
It was possible back when destuctors may throw, in a pre C++11.

I've found one place where it may throw from 2012:

    ~UnionBlockInputStream()
    {
        ...
        if (exception && !std::uncaught_exception())
            exception->rethrow();
        ...
    }

Refs: 8a053aba54
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-18 10:55:15 +01:00
Alexey Milovidov
dd79c5f4f6 Easy debugging of clickhouse-server in Docker 2023-02-14 09:49:05 +01:00
Alexey Milovidov
2e57fab8e3 Instrumentation of callbacks for distributed queries 2023-02-12 10:04:05 +01:00
Nikita Mikhaylov
33877b5e00
Parallel replicas. Part [2] (#43772) 2023-02-03 14:34:18 +01:00
Alexander Tokmakov
6ddd76c5a0 fix 2023-01-24 01:30:26 +01:00
Alexander Tokmakov
3f6594f4c6 forbid old ctor of Exception 2023-01-23 22:18:05 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Alexander Tokmakov
5cd90c1a3e Merge branch 'master' into exception_message_patterns 2023-01-17 20:04:04 +01:00
Nikita Mikhaylov
0fc755806e
One more attempt to fix race in TCPHandler (#45240) 2023-01-17 16:17:14 +01:00
Alexander Tokmakov
870cfcc36a less fmt::runtime usages 2023-01-17 00:11:59 +01:00
Nikolai Kochetov
9893d35476 Fix data race in s3Cluster. 2023-01-10 15:34:52 +00:00
Yakov Olkhovskiy
ce533af162 remove unnecessary includes 2022-12-23 19:29:51 +00:00
Yakov Olkhovskiy
a166dde9c4 add member function getLastForwardedFor to ClientInfo 2022-12-23 18:32:29 +00:00
Yakov Olkhovskiy
8462b724ca
Merge branch 'master' into feature-protocol-proxy 2022-12-20 13:12:07 -05:00
Yakov Olkhovskiy
9092134544 auth_use_forwarded_address config setting added, using forwarded address for proxied TCP and HTTP connection, likely bug is fixed in AccessControl, relevant test is appended 2022-12-20 17:39:40 +00:00
Anton Popov
cce3257f39
Merge branch 'master' into optimize-storage-s3 2022-12-13 21:35:12 +01:00
Nikolay Degterinsky
19e95e8cd8 Better implementation of password rules 2022-12-11 23:59:04 +00:00
Yakov Olkhovskiy
a98ce7d2be propagate PROXYv1 forwarded_for address to http handler in composable protocols; some logging added 2022-12-11 23:39:16 +00:00
Anton Popov
8c95308a08 fix table function s3Cluster 2022-12-07 17:00:10 +00:00
Azat Khuzhin
56bc85746f Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Interpreters/ProcessList.cpp
 2022-10-22 16:49:08 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Alexey Milovidov
ee5f5a4cb4 Unfathomable amount of trash 2022-10-21 05:33:17 +02:00
Yakov Olkhovskiy
3ce1fa09d6
Merge pull request #41198 from ClickHouse/composable-protocol 
Composable protocol
 2022-10-17 11:23:12 -04:00
Kseniia Sumarokova
3819a21033
Merge pull request #42078 from azat/insert-progress 
Fix progress for INSERT SELECT
 2022-10-11 10:56:20 +02:00
Yakov Olkhovskiy
2229087896
Merge branch 'master' into composable-protocol 2022-10-07 02:23:26 -04:00
Azat Khuzhin
3efd941233 Fix final progress for INSERT SELECT over TCP 
Move sending of progress and profile events after calling
BlockIO::onFinish() since it will call on_finish callback, that will do
the final flush of progress (at least WriteProgress).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-05 19:12:18 +02:00
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Yakov Olkhovskiy
fc78af3f69 add 'default_database' endpoint config parameter 2022-09-19 02:01:09 +00:00
Yakov Olkhovskiy
c66f412300 pass session certificate for showCertificate() 2022-09-18 07:11:52 +00:00
Yakov Olkhovskiy
0d6d9e7e44
Merge branch 'master' into composable-protocol 2022-09-11 18:59:57 -04:00
Yakov Olkhovskiy
772bf050da add PROXYv1 handler, add stack exchange data block, tuneup protocols config 2022-09-10 20:21:37 +00:00
Frank Chen
bb00dcc19b Remove using namespace from header 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:20:13 +08:00
Frank Chen
99c37ce6c6
Merge branch 'master' into tracing_context_propagation 2022-08-25 10:07:16 +08:00
Frank Chen
cd19366b44 Move classes into DB::OpenTelemetry namespace 2022-08-24 16:41:40 +08:00
Alexander Tokmakov
77178a18fa fix race on system restart replica 2022-08-23 16:10:44 +02:00
Frank Chen
a3b6ad2a65
Merge branch 'master' into tracing_context_propagation 2022-08-18 20:59:07 +08:00
Alexey Milovidov
8c3d1f2b5b
Merge branch 'master' into server-side-time 2022-08-12 17:40:02 +03:00
Nikita Taranov
17956cb668
Extend protocol with query parameters (#39906) 2022-08-12 14:28:35 +02:00
Alexey Milovidov
33e43cc6f9
Merge branch 'master' into server-side-time 2022-08-10 07:28:15 +03:00
Yakov Olkhovskiy
5304e773cd no session in interserver mode 2022-08-09 12:10:09 -04:00
Yakov Olkhovskiy
d39e9f65de
Merge branch 'master' into fix-quota-key 2022-08-08 11:54:21 -04:00
Alexey Milovidov
855e52a7c9 Send final ProfileEvents just in case 2022-08-08 07:12:08 +02:00
Alexey Milovidov
db85ebfaa7 Add server-side time to Progress 2022-08-08 06:55:41 +02:00
Alexander Tokmakov
c983f14ed1
Merge pull request #39925 from ClickHouse/log_test_on_connection_close 
Add logging to debug flaky tests
 2022-08-05 16:20:17 +03:00
Alexander Tokmakov
e59f217f8d add test logs on connection close 2022-08-05 13:33:27 +02:00
Alexander Tokmakov
20cf4e8d22 slightly more readable if conditions 2022-08-04 18:04:06 +02:00
Yakov Olkhovskiy
1680be19ca some refactoring 2022-08-03 16:36:52 -04:00
Yakov Olkhovskiy
2e34b384c1 update tcp protocol, add quota_key 2022-08-03 15:44:08 -04:00
Frank Chen
40c6e4c0d6 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 10:02:09 +08:00
Anton Kozlov
ae7fd5bf93 Default database resolution in distributed reads 2022-07-29 13:35:57 +00:00
Amos Bird
90fa4863f9
Text log source regexp 2022-07-13 17:52:14 +08:00
Frank Chen
90efbc323d Fix conflicts 2022-07-07 17:44:32 +08:00
Frank Chen
93dc109e36 Fix code 2022-07-07 17:44:19 +08:00
Frank Chen
19f3f3e6ca Initialize thread tracing context for TCPServer 2022-07-07 17:42:22 +08:00
Alexey Milovidov
02adad3f27
Merge pull request #37860 from amosbird/remove-duplicated-log 
Remove duplicate peak mem log
 2022-07-07 08:16:06 +03:00
Azat Khuzhin
9eeb856519 Fix INSERT into Distributed hung due to ProfileEvents 
Right now RemoteInserter does not read ProfileEvents for INSERT, it
handles them only after sending the query or on finish.

But #37391 sends them for each INSERT block, but sometimes they can be
no ProfileEvents packet, since it sends only non-empty blocks.

And this adds too much complexity, and anyway ProfileEvents are useless
for the server, so let's send them only if the query is initial (i.e.
send by user).

Note, that it is okay to change the logic of sending ProfileEvents w/o
changing DBMS_TCP_PROTOCOL_VERSION, because there were no public
releases with the original patch included yet.

Fixes: #37391
Refs: #35075
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-22 15:41:15 +03:00
Azat Khuzhin
4baa7690ae Send profile events for INSERT queries (previously only SELECT was supported) 
Reproducer:

    echo "1" | clickhouse-client --query="insert into function null('foo String') format TSV" --print-profile-events --profile-events-delay-ms=-1

However, clickhouse-local is differnt, it does sent the periodically,
but only if query was long enough, i.e.:

    # yes | head -n100000 | clickhouse-local --query="insert into function null('foo String') format TSV" --print-profile-events --profile-events-delay-ms=-1
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] ContextLock: 10 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] DiskReadElapsedMicroseconds: 29 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] IOBufferAllocBytes: 200000 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] IOBufferAllocs: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] InsertQuery: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] InsertedBytes: 1000000 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] InsertedRows: 100000 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] MemoryTrackerUsage: 1521975 (gauge)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] OSCPUVirtualTimeMicroseconds: 102148 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] OSReadChars: 135700 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] OSWriteChars: 8 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] Query: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] RWLockAcquiredReadLocks: 2 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] ReadBufferFromFileDescriptorRead: 5 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] ReadBufferFromFileDescriptorReadBytes: 134464 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] RealTimeMicroseconds: 293747 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] SoftPageFaults: 382 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] TableFunctionExecute: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] UserTimeMicroseconds: 102148 (increment)

v2: Proper support ProfileEvents in INSERTs (with protocol change)
v3: Receive profile events on INSERT queries
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 11:59:01 +03:00
Anton Popov
df6882d2b9
Revert "Fix errors of CheckTriviallyCopyableMove type" 2022-06-07 13:53:10 +02:00
Vitaly Baranov
d199478169
Merge pull request #37303 from ClickHouse/fix_trash 
Try to fix some trash
 2022-06-07 10:17:39 +02:00
Robert Schulze
2d87af2a15
Merge pull request #37647 from DevTeamBK/Fix-all-CheckTriviallyCopyableMove-Errors 
Fix errors of CheckTriviallyCopyableMove type
 2022-06-05 19:58:47 +02:00
Amos Bird
a5afb73c52
Remove duplicated peak mem log 
Also log "Processed in {} sec" a little bit earlier so that it's also
sent to client and have the log entry related to the query_id.
 2022-06-05 17:02:58 +08:00
HeenaBansal2009
b7eb6bbd38 Fixed clang-tidy-CheckTriviallyCopyableMove-errors 2022-05-30 11:09:03 -07:00
Alexander Tokmakov
3f44f34fe1 review suggestions 2022-05-23 21:55:17 +02:00
Alexander Tokmakov
d0f998fb88 Merge branch 'master' into fix_trash 2022-05-23 21:25:56 +02:00
taiyang-li
f13dcad58c Merge branch 'master' into fix_settings 2022-05-23 17:36:45 +08:00
Alexander Tokmakov
c48410b574 fix trash in my code 2022-05-21 02:05:02 +02:00
Anton Popov
cb0e6c2718 mark all operators bool() as explicit 2022-05-20 15:29:54 +00:00
Alexander Tokmakov
e9568d8cbb fixes 2022-05-19 00:00:05 +02:00
Alexander Tokmakov
3bc39f9929 fixes 2022-05-18 21:57:20 +02:00
Alexander Tokmakov
9772924d06 Merge branch 'master' into fix_trash 2022-05-18 17:27:54 +02:00
Alexander Tokmakov
19afeda4b1 fix style 2022-05-18 14:23:52 +02:00
Alexander Tokmakov
538b5cd536 fix more trash 2022-05-18 14:06:52 +02:00
Alexander Tokmakov
5590adeffa fix more trash 2022-05-17 23:04:24 +02:00
Alexander Tokmakov
dea39d8175 fix some trash 2022-05-17 18:22:52 +02:00
taiyang-li
9f3dadf17c Merge branch 'master' into fix_settings 2022-05-17 16:19:29 +08:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies 2022-05-16 10:12:50 +02:00
kssenii
0feda67ec4 Fix progress for insert select 2022-05-10 15:04:08 +02:00
taiyang-li
fd878f7e7b change as requested 2022-05-10 12:06:50 +08:00
Robert Schulze
1b81bb49b4
Enable clang-tidy modernize-deprecated-headers & hicpp-deprecated-headers 
Official docs:

  Some headers from C library were deprecated in C++ and are no longer
  welcome in C++ codebases. Some have no effect in C++. For more details
  refer to the C++ 14 Standard [depr.c.headers] section. This check
  replaces C standard library headers with their C++ alternatives and
  removes redundant ones.
 2022-05-09 08:23:33 +02:00
Kruglov Pavel
8d647eff06
Merge pull request #36749 from Avogar/fix-timeouts 
Fix bug in receive timeouts in Hedged requests
 2022-05-02 13:16:03 +02:00
Azat Khuzhin
d82f2ff162 Set is_all_data_sent on exceptions too 
This will avoid clickhouse-test complains about left queries after the
test, like in [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/36258/9646487c093a75dc31e3e818417cfad83580b40f/stateful_tests__memory__actions_.html

Follow-up for: #36649
 2022-04-30 13:00:44 +03:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
mergify[bot]
d96c9c5cff
Merge branch 'master' into fix-timeouts 2022-04-28 15:03:19 +00:00
avogar
81f85892eb Fix bug in receive timeouts in Hedged requests 2022-04-28 13:10:27 +00:00
Azat Khuzhin
6339a48923 Add is_all_data_sent into system.processes 
v2: fix SHOW PROCESSLIST (does not have process list entry)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-26 12:15:00 +03:00
Anton Popov
3e361c9759 Merge remote-tracking branch 'upstream/master' into HEAD 2022-04-19 14:18:04 +00:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
Anton Popov
2de6668b3f show names of erroneous files 2022-04-16 00:10:47 +00:00
Azat Khuzhin
097e2f626d Do not overlap profile events snapshots for queries 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-12 20:54:56 +03:00
Antonio Andelic
e15fccd31b Add is_secure info to HTTP 2022-03-30 06:39:40 +00:00
Antonio Andelic
d14ad227b1 Polish TCP is_secure flag 2022-03-30 06:39:40 +00:00
taiyang-li
1b9987ceb1 fix mistake in check-style 2022-03-05 14:22:56 +08:00
taiyang-li
f83132bad2 finish dev 2022-03-01 15:54:23 +08:00
taiyang-li
e6b29167da fix style 2022-02-15 20:26:53 +08:00
taiyang-li
b4440131d0 add metrics for clickhouse-local 2022-02-15 16:25:07 +08:00
Dmitry Novik
d405523598 Fix ProfileEvents sending in INSERT SELECT 2022-02-10 13:27:27 +00:00
Azat Khuzhin
5472aef084 Fix current_user/current_address for interserver mode 
Before this patch current_user/current_address will be preserved from
the previous query.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-02 17:44:19 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Azat Khuzhin
e8c7e5e8f0 Fix typo in sendMergeTreeReadTaskRequestAssumeLocked() 2022-01-07 20:04:57 +03:00
Vitaly Baranov
f80a3e3247
Merge pull request #30549 from aiven/kmichel-server-reload 
Start/stop servers when `listen_host`/`*_port` changes
 2021-12-27 14:51:00 +03:00
Kevin Michel
ffc1fca296
Start/stop servers when listen_host/*_port changes 
This allows starting and stopping separately each protocol server
without restarting ClickHouse.

This also allows adding or removing `listen_host` entries, which
start and stops servers for all enabled ports.

When stopping a server, the listening socket is immediately closed
(and available for another server).

Protocols with persistent connections try to wait for any currently
running query to finish before closing the connection, but idle
connection are closed quickly (depending on how often the protocol
is polled).

An extra ProfileEvent is added, `MainConfigLoads`, it is
incremented every time the configuration is reloaded. This helps
when trying to assess whether the new configuration was applied.
 2021-12-24 08:26:02 +01:00
Azat Khuzhin
6aebc3e94c Do not loose ProfileEvents in case of thread destroyed before 
v2: drop std::move and add copy ctor for ProfileEvents::Counter::Snapshot
v2: remove std::move
 2021-12-19 23:43:05 +03:00
Alexey Milovidov
2c68192bb6 Fix strange code in TCPHandler 2021-12-12 05:35:33 +03:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
alexey-milovidov
96ec92c7cd
Merge pull request #31155 from ClickHouse/incremental-profile-events 
Send incremental profile events to client
 2021-12-01 04:04:41 +03:00
Azat Khuzhin
dd7f780978 Fix Progress for short INSERT SELECT queries 
Note, that we cannot send Progress for regular INSERT (w/ VALUES) w/o
breaking protocol compatibility, but it can be done by increasing
revision.
 2021-11-20 16:41:35 +03:00
Azat Khuzhin
0b4d855c08 Reduce copy-paste in TCPHandler::processInsertQuery() 2021-11-20 16:41:35 +03:00
Dmitry Novik
0cd858263f Update protocol version 2021-11-08 16:43:34 +03:00
Dmitry Novik
a093395b9b Process incremental profile events on client 2021-11-08 16:38:31 +03:00
Dmitry Novik
8708e00b3a Send incremental profile events to client 2021-11-08 12:40:31 +03:00
Alexey Milovidov
993f3faefd Miscellaneous 2021-10-31 18:11:46 +03:00
Alexey Milovidov
8b4a6a2416 Remove cruft 2021-10-28 02:10:39 +03:00
mergify[bot]
69667aab1a
Merge branch 'master' into fix-backward-compatibility-after-21196 2021-10-19 10:11:16 +00:00
Azat Khuzhin
f7b76373ce Move ProfileEvents packet type from TCPHandler into ProfileEventsExt 2021-10-19 00:54:38 +03:00
Nikolai Kochetov
067eaadadd Merge branch 'master' into removing-data-streams-folder 2021-10-16 09:46:05 +03:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
Nikolai Kochetov
06d9fc602e Merge branch 'master' into fix-backward-compatibility-after-21196 2021-10-12 14:41:23 +03:00
Dmitry Novik
8be70bc417 Update profile event forwarding 2021-10-12 14:15:07 +03:00
Dmitry Novik
bfdd34c13d code cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
3a0764634e Update InternalProfileEventsQueue usage 2021-10-11 17:39:24 +03:00
Dmitry Novik
0bdabf46f2 Send ProfileEvents only to supported clients 2021-10-11 17:39:24 +03:00
Dmitry Novik
9f9af28b5e Output memory usage with progress 2021-10-11 17:39:24 +03:00
Dmitry Novik
73df6190df Cleanup code 2021-10-11 17:39:24 +03:00
Dmitry Novik
1d2e2d7305 cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
7e3caf96be Fix cores approximation 2021-10-11 17:39:24 +03:00
Dmitry Novik
9071a7151e Fix communication & race conditions 2021-10-11 17:39:23 +03:00
Dmitry Novik
356723427d WIP on ProfileEvents forwarding 2021-10-11 17:39:23 +03:00
Dmitry Novik
803b8623c1 Fix TCPHandler::sendProfileEvents 2021-10-11 17:29:50 +03:00
Dmitry Novik
e9b1e05461 Send profile events from all threads of current group 2021-10-11 17:29:50 +03:00
Dmitry Novik
362bcb2f66 Introduce ProfileEvents packet 2021-10-11 17:29:50 +03:00
Nikolai Kochetov
c6bce1a4cf Update Native. 2021-10-08 20:21:19 +03:00
Nikolai Kochetov
78e1db209f
Remove more data streams (#29491) 
* Remove more streams.

* Fixing build.

* Fixing build.

* Rename files.

* Fix fast test.

* Fix StorageKafka.

* Try fix kafka test.

* Move createBuffer to KafkaSource ctor.

* Revert "Move createBuffer to KafkaSource ctor."

This reverts commit 81fa94d27e.

* Revert "Try fix kafka test."

This reverts commit 2107e54969.

* Comment some rows in test.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-10-07 11:26:08 +03:00
Azat Khuzhin
4f5118b85f Fix data-race between fatal error handler and progress packets 
This is not that important, but because of packets overlaps the client
may not render the fatal error.

You will find TSan report in `details`.

<details>

```
WARNING: ThreadSanitizer: data race (pid=13384)
  Write of size 8 at 0x7b24000ad818 by thread T47:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x16124d61)
    1 DB::TCPHandler::sendProgress() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1602:10 (clickhouse-tsan+0x16124d61)
    2 DB::TCPHandler::runImpl()::$_6::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:318:25 (clickhouse-tsan+0x1612cf54)

  Previous write of size 8 at 0x7b24000ad818 by thread T229:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x1612b793)
    1 DB::TCPHandler::sendLogData(DB::Block const&) obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1561:10 (clickhouse-tsan+0x1612b793)
    2 DB::TCPHandler::sendLogs() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1632:9 (clickhouse-tsan+0x161228ab)
    3 DB::TCPHandler::runImpl()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:229:62 (clickhouse-tsan+0x1612c4e1)
```

</details>
 2021-10-05 21:21:32 +03:00
Nikolai Kochetov
e2d07fbdf5
Merge branch 'master' into fix-backward-compatibility-after-21196 2021-10-04 17:35:32 +03:00
Nikolai Kochetov
6169b180cb Fix backward compatibility after #21196. 2021-10-04 17:27:44 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Azat Khuzhin
e8a90b8ff2 Do not allow to reuse previous credentials in case of inter-server secret 
Before this patch INSERT via Buffer/Kafka may re-use previously set user
for that connection, while this is not correct, it should reset the
user, and use global context.

Note, before [1] there was a fallback to default user, but that code had
been removed, and now it got back.

  [1]: 0159c74f21 ("Secure inter-cluster query execution (with initial_user as current query user) [v3]")

Also note, that context for Buffer table (and others) cannot be changed,
since they don't have any user only profile.

I've tested this patch manually using the following:

    create table dist (key Int) engine=Distributed(test_cluster_two_shards_secure, default, data, key);
    create table buffer (key Int) engine=Buffer(default, dist, 1, 0, 0, 0, 0, 0, 0);
    create table data (key Int) engine=Memory();

    # to start the connection with readonly user
    $ clickhouse-client --user readonly -q 'select * from dist'
    $ clickhouse-client -q 'insert into buffer values (1)'
    # before this patch this produces errors like:
    # 2021.09.27 23:46:48.384920 [ 19474 ] {} <Error> default.dist.DirectoryMonitor: Code: 164. DB::Exception: Received from 127.0.0.2:9000. DB::Exception: readonly: Cannot execute query in readonly mode. Stack trace:

v2: reset the authentication instead of using default user (as suggested by @vitlibar)
v3: reset Session::user and introduce ClientInfo::resetAuthentication (as suggested by @vitlibar)
v4: reset the session every time in interserver mode (suggested by @vitlibar)
 2021-10-01 01:13:15 +03:00
Nikolai Kochetov
78a7665f43 Merge branch 'master' into rewrite-pushing-to-views 2021-09-27 10:56:50 +03:00
Azat Khuzhin
b3d1bfc67a Send UNKNOWN_DATABASE to the client (via TCP) 
Before this patch:

    $ clickhouse-client --database foo -q 'select 1'
    Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from localhost:9000. (ATTEMPT_TO_READ_AFTER_EOF)

After:

    $ clickhouse-client --database foo -q 'select 1'
    Received exception from server (version 21.11.1):
    Code: 81. DB::Exception: Received from localhost:9000. DB::Exception: Database foo doesn't exist. (UNKNOWN_DATABASE)
    (query: select 1)

Fixes: #26864 (cc @vitlibar)
 2021-09-26 23:45:29 +03:00
Nikolai Kochetov
eed4e8c754 Fix progress for insert select. 2021-09-22 16:29:58 +03:00
Nikolai Kochetov
81bf13a247 Refactor pushing to views. 2021-09-22 13:57:00 +03:00
Nikolai Kochetov
db6f1e198c Add cancell callback to completed executor. 2021-09-21 19:37:32 +03:00
Nikolai Kochetov
a8c3b02598 Merge branch 'master' into rewrite-pushing-to-views 2021-09-17 15:38:11 +03:00
Nikolai Kochetov
58bb5fe462 Fix some tests. 2021-09-17 14:40:03 +03:00
Nikolai Kochetov
341553febd Fix build. 2021-09-16 20:40:42 +03:00
Nikolai Kochetov
e616732743 Small refactoring. 2021-09-15 22:35:48 +03:00
Nikolai Kochetov
3a9d88fbc9 Fix more tests. 2021-09-10 17:52:24 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
Nikolai Kochetov
66a76ab70f Rewrite PushingToViewsBlockOutputStream part 6 2021-09-03 20:29:36 +03:00
Nikolai Kochetov
358babc31b Rewrite PushingToViewsBlockOutputStream part 5. 2021-09-02 14:29:20 +03:00
Anton Popov
6d3274c22c Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-26 03:31:46 +03:00
Vitaly Baranov
64f69462c6
Merge pull request #27921 from vitlibar/add-tests-for-field-init 
Allow implicit cast bool to Field
 2021-08-24 17:15:44 +03:00
Nikita Mikhaylov
fb6462d70a
Merge pull request #27886 from vitlibar/add-separate-constants-for-interfaces 
Add separate constants for interfaces LOCAL and TCP_INTERSERVER.
 2021-08-23 16:25:01 +03:00
Vitaly Baranov
ff1fad8521 Use setSetting() with implicit conversion true/false -> Field. 2021-08-23 08:57:00 +03:00
Alexey Milovidov
c3c31e2895 Move function to appropriate place to make code more readable 2021-08-20 12:56:39 +03:00
Vitaly Baranov
3ab1177d7e Add separate constants for interface LOCAL and TCP_INTERSERVER. 2021-08-19 23:04:36 +03:00
Vitaly Baranov
fabd7193bd Code cleanups and improvements. 2021-08-18 14:24:52 +03:00