thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-12 01:12:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
140,432 Commits 9,297 Branches 1,782 Tags 2.1 GiB
b080adb586
Commit Graph

417 Commits

Author SHA1 Message Date
avogar
e63231700f Fix tests 2023-03-28 16:00:00 +00:00
Kruglov Pavel
c838013b69
Fix special build 2023-03-24 21:34:21 +01:00
Sergei Trifonov
a05aa5c1c6
Merge branch 'master' into segmentator-fix 2023-03-24 16:19:57 +01:00
Kruglov Pavel
f3f93dd06c
Merge branch 'master' into non-blocking-connect 2023-03-24 15:59:40 +01:00
avogar
bf7e62ff56 Refactor RemoteQueryExecutor, make it more thread safe 2023-03-23 19:52:37 +00:00
Amos Bird
a41ba1e6a7
Fix more cases 2023-03-23 14:52:33 +08:00
Amos Bird
02c5d1f364
Correct exact_rows_before_limit in all senarios 2023-03-22 23:26:31 +08:00
serxa
f3c12b8543 fix per-thread perf counters update 2023-03-22 13:51:07 +00:00
avogar
783e3ab789 Try fix 2023-03-22 12:25:16 +00:00
avogar
38e44861ae Fix possible race conditions 2023-03-21 16:01:54 +00:00
avogar
ba389c2a3d Fix 2023-03-20 19:06:02 +00:00
avogar
15f5a468e9 Clean 2023-03-20 17:56:01 +00:00
Kruglov Pavel
1d12e85d55
Merge branch 'master' into non-blocking-connect 2023-03-20 13:23:55 +01:00
avogar
4b46b5b748 Fix, make better 2023-03-17 13:02:20 +00:00
Alexander Tokmakov
a72f0cae07
Merge pull request #46071 from arenadata/ADQM-639 
Add support of arbitrary engines for temporary tables
 2023-03-16 20:56:25 +03:00
avogar
17f5356178 Improve async reading from socket 2023-03-15 12:18:08 +00:00
Alexey Milovidov
a411ae10b5
Merge pull request #47486 from ClickHouse/revert_25674 
Remove "asynchronous connection drain"
 2023-03-13 21:40:39 +03:00
flynn
e56eeac05f fix 2023-03-11 18:48:36 +00:00
Alexander Tokmakov
7b1b238d0b Revert "Merge pull request #25674 from amosbird/distributedreturnconnection" 
This reverts commit 5ffd99dfd4, reversing
changes made to 2796aa333f.
 2023-03-11 19:09:47 +01:00
Alexander Tokmakov
13aa2ce89b Revert "Merge pull request #26807 from amosbird/improvement2" 
This reverts commit 671c2194b3, reversing
changes made to ee13f2e6bc.
 2023-03-11 17:26:59 +01:00
Alexander Tokmakov
bd124a24ea Revert "Merge pull request #31965 from azat/connection-drain-pull" 
This reverts commit 6cb54b4092, reversing
changes made to 42d938ffa0.
 2023-03-11 17:24:38 +01:00
flynn
191bc9717e fix style 2023-03-11 15:58:16 +00:00
flynn
530c9ae490 revert 2023-03-11 15:43:19 +00:00
flynn
0084a61c76 Merge branch 'master' of github.com:ClickHouse/ClickHouse into explain-graph 2023-03-11 15:35:47 +00:00
flynn
3687698b15 Fix explain graph with projection 2023-03-11 15:03:10 +00:00
Alexey Milovidov
a5c8f23f69 Remove a feature 2023-03-10 23:33:56 +01:00
Roman Vasin
95a7e11a47 Merge branch 'master' of github.com:ClickHouse/ClickHouse into ADQM-639 2023-03-06 09:11:20 +00:00
Alexander Tokmakov
c518f3690e Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-03-02 18:24:08 +01:00
Amos Bird
c996dfe562
do-while loop trivia 2023-02-28 22:43:49 +08:00
Roman Vasin
76dfb04511 Send only temporary tables with StorageMemory 2023-02-28 07:56:10 +00:00
Alexander Tokmakov
cad1e0b768 fix 2023-02-25 01:18:34 +01:00
Alexey Milovidov
d8cda3dbb8 Remove PVS-Studio 2023-02-19 23:30:05 +01:00
Alexey Milovidov
2e57fab8e3 Instrumentation of callbacks for distributed queries 2023-02-12 10:04:05 +01:00
Robert Schulze
e490ec91d9
Merge branch 'master' into rs/fix-fragile-linking 2023-02-09 11:33:59 +01:00
Azat Khuzhin
8cc41b7f41 Check return value of ::close() 
Note, that according close(2), EINTR should not be retriable for close()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-07 11:28:22 +01:00
Robert Schulze
84b9ff450f
Fix terribly broken, fragile and potentially cyclic linking 
Sorry for the clickbaity title. This is about static method
ConnectionTimeouts::getHTTPTimeouts(). It was be declared in header
IO/ConnectionTimeouts.h, and defined in header
IO/ConnectionTimeoutsContext.h (!). This is weird and caused issues with
linking on s390x (##45520). There was an attempt to fix some
inconsistencies (#45848) but neither did @Algunenano nor me at first
really understand why the definition is in the header.

Turns out that ConnectionTimeoutsContext.h is only #include'd from
source files which are part of the normal server build BUT NOT part of
the keeper standalone build (which must be enabled via CMake
-DBUILD_STANDALONE_KEEPER=1). This dependency was not documented and as
a result, some misguided workarounds were introduced earlier, e.g.
0341c6c54b

The deeper cause was that getHTTPTimeouts() is passed a "Context". This
class is part of the "dbms" libary which is deliberately not linked by
the standalone build of clickhouse-keeper. The context is only used to
read the settings and the "Settings" class is part of the
clickhouse_common library which is linked by clickhouse-keeper already.

To resolve this mess, this PR

- creates source file IO/ConnectionTimeouts.cpp and moves all
  ConnectionTimeouts definitions into it, including getHTTPTimeouts().

- breaks the wrong dependency by passing "Settings" instead of "Context"
  into getHTTPTimeouts().

- resolves the previous hacks
 2023-02-05 20:49:34 +00:00
Nikita Mikhaylov
33877b5e00
Parallel replicas. Part [2] (#43772) 2023-02-03 14:34:18 +01:00
Robert Schulze
325c6bdf3d
Renaming: "Query Result Cache" --> "Query Cache" 
Reasons:

- The cache will at some point store intermediate results as opposed to
  only query results. We should change the terminology now without
  having to worry about backward compat.

- Equivalent caches in MySQL (1) and Starrocks (2) are called "query
  cache".

- The new name is ca. 13.8% more catchy.

(1) https://dev.mysql.com/doc/refman/5.6/en/query-cache.html
(2) https://docs.starrocks.io/en-us/2.5/using_starrocks/query_cache
 2023-01-31 09:54:34 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Robert Schulze
340f406553
Merge branch 'master' into query-result-cache 2023-01-22 13:21:36 +01:00
Alexander Tokmakov
e7899825e6 save format strings for DB::Exceptions 2023-01-16 23:20:33 +01:00
Robert Schulze
15e11741cb
Cosmetics 2023-01-13 00:00:23 +00:00
Robert Schulze
45dbcf88e5
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-06 20:42:48 +00:00
Robert Schulze
1309e780ca
Fix style 2023-01-05 11:11:20 +00:00
Robert Schulze
4fcead15c3
Fix handling of exceptions during query runtime 2023-01-04 22:33:04 +00:00
Nikolai Kochetov
f421b4bf13
Merge branch 'master' into fix-crash-in-totals-structure 2023-01-03 14:24:22 +01:00
Nikolai Kochetov
078e205154 Fixing a test. 2023-01-02 12:15:31 +00:00
Robert Schulze
e9e04166d9
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-02 08:20:27 +00:00
Nikolai Kochetov
e590d02abd Remove obvious comment. 2022-12-30 17:19:38 +00:00
Nikolai Kochetov
ccb2269641 Apply type convertions for totals and extrems in RemoteQueryExecutor. 2022-12-30 16:55:56 +00:00
Nikolai Kochetov
2d84ef2891 Fix tests and update IntersectOrExceptStep. 2022-12-28 14:44:45 +00:00
Nikolai Kochetov
d0a8afec06 Fixing style. 2022-12-28 13:04:20 +00:00
Nikolai Kochetov
21e93cd41d Fix LOGICAL_ERROR for JOIN with subquery, UNION and totals. 2022-12-28 13:00:32 +00:00
Robert Schulze
537c64aad2
Merge remote-tracking branch 'origin/master' into query-result-cache 
Conflicts:
- tests/queries/0_stateless/02117_show_create_table_system.reference
 2022-12-15 12:17:58 +00:00
Robert Schulze
b78652e138
Query result cache 2022-12-12 10:37:10 +00:00
Robert Schulze
8bcafe4544
Renaming PipelineResourcesHolder.h/cpp to QueryPlanResourceHolder.h/cpp 2022-12-10 18:08:10 +00:00
Igor Nikonov
20e67b7140 Merge remote-tracking branch 'origin/master' into HEAD 2022-11-24 13:10:37 +00:00
Alexey Milovidov
9a0a21db94 Miscellaneous changes 2022-11-14 06:22:05 +01:00
Alexey Milovidov
bfea3deed9 Miscellaneous changes 2022-11-14 06:09:03 +01:00
vdimir
2eef8c7639
upd gracge hash 2022-10-27 15:25:50 +00:00
vdimir
b605af2014
fix build 2022-10-26 13:50:14 +00:00
vdimir
506bf2d225
Merge branch 'master' into grace_hash_join 2022-10-26 12:25:50 +00:00
Azat Khuzhin
56bc85746f Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Interpreters/ProcessList.cpp
 2022-10-22 16:49:08 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Alexey Milovidov
ee5f5a4cb4 Unfathomable amount of trash 2022-10-21 05:33:17 +02:00
vdimir
424971d9ae
wip grace hash 2022-10-18 12:20:52 +00:00
vdimir
bf3f66b5b6
fix delayed blocks for grace hash join 2022-10-18 11:43:01 +00:00
Alexey Milovidov
e93b8d1618 Fix trash 2022-10-17 04:44:58 +02:00
Alexey Milovidov
f88ed8195b Fix trash 2022-10-17 04:21:08 +02:00
Alexey Milovidov
17d4862336 Fix data race in query finish/cancel 2022-10-17 02:29:54 +02:00
Alexey Milovidov
1113fe4a8d Fix data race in query finish/cancel 2022-10-17 02:27:10 +02:00
Alexey Milovidov
c6b2ee47df Fix data race in query finish/cancel 2022-10-17 02:10:36 +02:00
vdimir
59eb763024
Add delayded join blocks to pipleine only if needed 2022-10-06 17:47:40 +00:00
vdimir
a0349cf8c5
wip grace hash 2022-10-05 12:40:32 +00:00
vdimir
f3781be762
wip grace hash 2022-10-04 08:20:13 +00:00
vdimir
7ebc297f4c
Merge branch 'master' into pr/BigRedEye/38191 2022-09-30 09:40:47 +00:00
Raúl Marín
133345eb37 Fix missing read bytes/rows in X-ClickHouse-Summary 2022-09-20 15:12:44 +02:00
vdimir
6d4b6c452a
Merge branch 'master' into grace_hash_join 2022-09-07 08:00:14 +00:00
Vladimir C
963c0111bf
Merge pull request #39418 from vdimir/join_and_sets 
Filter joined streams for `full_sorting_join` by each other before sorting
 2022-09-02 13:57:06 +02:00
vdimir
c778bba13f
Create sets for joins: wip 2022-08-29 09:47:00 +00:00
vdimir
71708d595f
Create sets for joins: wip 2022-08-29 09:46:59 +00:00
vdimir
8f06430ebd
Create sets for joins: upd 2022-08-29 09:46:58 +00:00
vdimir
c5bc7b0a0c
Resize pipeline after full sort join 2022-08-29 09:46:56 +00:00
Frank Chen
a3b6ad2a65
Merge branch 'master' into tracing_context_propagation 2022-08-18 20:59:07 +08:00
Alexey Milovidov
1a8ddf2956 Addition to prev. revision 2022-08-14 09:35:22 +02:00
Alexey Milovidov
001aca3b47 ProfileEvents for incomplete data due to query complexity settings 2022-08-14 09:17:02 +02:00
Nikita Taranov
17956cb668
Extend protocol with query parameters (#39906) 2022-08-12 14:28:35 +02:00
vdimir
708747ca0b
Merge branch 'master' into refactor-prepared-sets 2022-08-08 14:27:18 +02:00
Nikolai Kochetov
60599197b2 Review fixes. 2022-08-04 15:23:10 +00:00
Frank Chen
40c6e4c0d6 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 10:02:09 +08:00
Nikolai Kochetov
59a11b32ad
Merge branch 'master' into use-dag-in-key-condition 2022-07-29 17:01:33 +02:00
Vladimir C
115506356c
Merge branch 'master' into refactor-prepared-sets 2022-07-27 19:57:23 +02:00
vdimir
d9928ac93d
Add methods to SubqueryForSet, do not use refernce to SetPtr 2022-07-26 18:39:09 +00:00
vdimir
1e3fa2e01f
Refactor PreparedSets/SubqueryForSet 2022-07-26 18:39:02 +00:00
Nikolai Kochetov
908c1d8cba
Merge pull request #39601 from ClickHouse/fix-chain-add-sink 
Fix Chain::addSink
 2022-07-26 14:54:44 +02:00
Nikolai Kochetov
683a8866ef Fix Chain::addSink 2022-07-26 09:36:39 +00:00
Nikolai Kochetov
be9c7ed52c Add ReadFromMerge step. 2022-07-25 19:41:43 +00:00
Alexander Tokmakov
bed2206ae9
Merge pull request #39460 from ClickHouse/remove_some_dead_and_commented_code 
Remove some dead and commented code
 2022-07-22 13:24:34 +03:00
Alexander Tokmakov
a8da5d96fc remove some dead and commented code 2022-07-21 15:05:48 +02:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Amos Bird
982e1a73d3
Better 2022-07-12 22:21:46 +08:00
Amos Bird
d3709c6c26
Avoid redundant join block transformation. 2022-07-12 22:20:10 +08:00
Frank Chen
57a7e4a7c9 Remove old API reference 2022-07-07 17:42:35 +08:00
vdimir
7c586a9e7c
Minor updates for full soring merge join 2022-07-06 14:28:05 +00:00
vdimir
881d352e05
upd full sorting join 2022-07-06 14:28:05 +00:00
vdimir
0b994bb258
fix build 2022-07-06 14:27:32 +00:00
vdimir
d184e184b4
full sort join: check key types, more tests 2022-07-06 14:26:19 +00:00
vdimir
4e88e8f5ec
full sort join: move block list to all join state 2022-07-06 14:26:17 +00:00
vdimir
1b429fc1af
wip: any left/right sorting join 2022-07-06 14:23:46 +00:00
vdimir
8dce97123c
wip: any inner full sorting join 2022-07-06 14:23:46 +00:00
vdimir
4a16195964
Calculate output header for full sorting merge join 2022-07-06 14:23:45 +00:00
vdimir
fa8eb35599
Pipeline for full sorting merge join 2022-07-06 14:23:44 +00:00
Maksim Kita
bdc21737d5 MergeTree merge disable batch optimization 2022-07-05 16:15:00 +02:00
Kseniia Sumarokova
cce381a3ae
Merge pull request #38307 from azat/fix-insert-profile-events 
Fix INSERT into Distributed hung due to ProfileEvents
 2022-06-23 10:09:45 +02:00
Nikita Taranov
41ba0118b5
Bring back #36396 (#38110) 
* Revert "Revert "More parallel execution for queries with `FINAL` (#36396)""

This reverts commit 5bfb15262c.

* fix tests

* fix review suggestions

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-06-22 15:05:07 +02:00
Azat Khuzhin
9eeb856519 Fix INSERT into Distributed hung due to ProfileEvents 
Right now RemoteInserter does not read ProfileEvents for INSERT, it
handles them only after sending the query or on finish.

But #37391 sends them for each INSERT block, but sometimes they can be
no ProfileEvents packet, since it sends only non-empty blocks.

And this adds too much complexity, and anyway ProfileEvents are useless
for the server, so let's send them only if the query is initial (i.e.
send by user).

Note, that it is okay to change the logic of sending ProfileEvents w/o
changing DBMS_TCP_PROTOCOL_VERSION, because there were no public
releases with the original patch included yet.

Fixes: #37391
Refs: #35075
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-22 15:41:15 +03:00
Maksim Kita
cb018348cf
Merge pull request #38022 from kitaisreal/sorting-added-batch-queue-variants 
Sorting added batch queue variants
 2022-06-20 22:35:44 +02:00
Sergey Skvortsov
52164efd64
fix: Use correct header for the totals 2022-06-20 22:30:56 +03:00
Sergey Skvortsov
6423e5f2f9
fix: Remove unnecessary include 2022-06-20 21:39:30 +03:00
Sergey Skvortsov
be8fc84653
fix: Use DelayedPortsProcessor, fix stuck pipelines 2022-06-20 21:38:12 +03:00
Kseniia Sumarokova
a756b4be27
Merge pull request #37391 from azat/insert-profile-events-fix 
Send profile events for INSERT queries (previously only SELECT was supported)
 2022-06-20 12:16:29 +02:00
Sergey Skvortsov
6b25a36816
fix: Add signal ports in JoiningTransform conditionally 2022-06-20 03:13:01 +03:00
Sergey Skvortsov
4120ba3ace
feat: Join delayed blocks in multiple threads 2022-06-19 22:17:24 +03:00
Maksim Kita
dbbf499005 Fixed unit tests 2022-06-18 18:20:01 +02:00
Azat Khuzhin
4baa7690ae Send profile events for INSERT queries (previously only SELECT was supported) 
Reproducer:

    echo "1" | clickhouse-client --query="insert into function null('foo String') format TSV" --print-profile-events --profile-events-delay-ms=-1

However, clickhouse-local is differnt, it does sent the periodically,
but only if query was long enough, i.e.:

    # yes | head -n100000 | clickhouse-local --query="insert into function null('foo String') format TSV" --print-profile-events --profile-events-delay-ms=-1
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] ContextLock: 10 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] DiskReadElapsedMicroseconds: 29 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] IOBufferAllocBytes: 200000 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] IOBufferAllocs: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] InsertQuery: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] InsertedBytes: 1000000 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] InsertedRows: 100000 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] MemoryTrackerUsage: 1521975 (gauge)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] OSCPUVirtualTimeMicroseconds: 102148 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] OSReadChars: 135700 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] OSWriteChars: 8 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] Query: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] RWLockAcquiredReadLocks: 2 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] ReadBufferFromFileDescriptorRead: 5 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] ReadBufferFromFileDescriptorReadBytes: 134464 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] RealTimeMicroseconds: 293747 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] SoftPageFaults: 382 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] TableFunctionExecute: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] UserTimeMicroseconds: 102148 (increment)

v2: Proper support ProfileEvents in INSERTs (with protocol change)
v3: Receive profile events on INSERT queries
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 11:59:01 +03:00
Alexander Tokmakov
5bfb15262c Revert "More parallel execution for queries with FINAL (#36396)" 
This reverts commit c8afeafe0e.
 2022-06-15 17:25:38 +03:00
Nikita Taranov
c8afeafe0e
More parallel execution for queries with FINAL (#36396) 2022-06-15 12:44:20 +02:00
Azat Khuzhin
7210be1534 Disable send_logs_level for INSERT into Distributed to avoid possible hung 
In case of INSERT into Distributed table with send_logs_level!=none it
is possible to receive tons of Log packets, and w/o consuming it
properly the socket buffer will be full, and eventually the query will
hung.

This happens because receiver will not read data until it will send logs
packets, but sender does not reads those Log packets and so receiver
hung, and hence the sender will hung too, because receiver do not
consume Data packets anymore.

In the initial version of this patch I tried to properly consume Log
packets, but it is not possible to ensure that before writing Data
blocks all Log packets had been consumed, that said that with current
protocol implementation it is not possible to fix Log packets consuming
properly, to avoid deadlock, so send_logs_level had been simply
disabled.

But note, that this does not differs to the user, in what ClickHouse did
before, since before it simply does not consume those packets, so the
client does not saw those messages anyway.

<details>

The receiver:

    Poco::Net::SocketImpl::poll(Poco::Timespan const&, int)
    Poco::Net::SocketImpl::sendBytes(void const*, int, int)
    Poco::Net::StreamSocketImpl::sendBytes(void const*, int, int)
    DB::WriteBufferFromPocoSocket::nextImpl()
    DB::TCPHandler::sendLogData(DB::Block const&)
    DB::TCPHandler::sendLogs()
    DB::TCPHandler::readDataNext()
    DB::TCPHandler::processInsertQuery()

    State      Recv-Q  Send-Q          Local Address:Port         Peer Address:Port Process
    ESTAB      4331792 211637           127.0.0.1:9000            127.0.0.1:24446 users:(("clickhouse-serv",pid=46874,fd=3850))

The sender:

    Poco::Net::SocketImpl::poll(Poco::Timespan const&, int)
    Poco::Net::SocketImpl::sendBytes(void const*, int, int)
    Poco::Net::StreamSocketImpl::sendBytes(void const*, int, int)
    DB::WriteBufferFromPocoSocket::nextImpl()
    DB::WriteBuffer::write(char const*, unsigned long)
    DB::CompressedWriteBuffer::nextImpl()
    DB::WriteBuffer::write(char const*, unsigned long)
    DB::SerializationString::serializeBinaryBulk(DB::IColumn const&, DB::WriteBuffer&, unsigned long, unsigned long) const
    DB::NativeWriter::write(DB::Block const&)
    DB::Connection::sendData(DB::Block const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool)
    DB::RemoteInserter::write(DB::Block)
    DB::RemoteSink::consume(DB::Chunk)
    DB::SinkToStorage::onConsume(DB::Chunk)

    State      Recv-Q  Send-Q         Local Address:Port         Peer Address:Port Process
    ESTAB      67883   3008240           127.0.0.1:24446           127.0.0.1:9000  users:(("clickhouse-serv",pid=41610,fd=25))

</details>

v2: rebase to use clickhouse_client_timeout and add clickhouse_test_wait_queries
v3: use KILL QUERY
v4: adjust the test
v5: disable send_logs_level for INSERT into Distributed
v6: add no-backward-compatibility-check tag
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-13 13:44:33 +03:00
Anton Popov
df6882d2b9
Revert "Fix errors of CheckTriviallyCopyableMove type" 2022-06-07 13:53:10 +02:00
Azat Khuzhin
078678237e Fix possible "No more packets are available" for distributed queries 
CI founds the following case:

<details>

    2022.05.25 22:36:06.778808 [ 3037 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Fatal> : Logical error: 'No more packets are available.'.
    2022.05.25 22:42:24.960075 [ 17397 ] {} <Fatal> BaseDaemon: ########################################
    2022.05.25 22:42:24.971173 [ 17397 ] {} <Fatal> BaseDaemon: (version 22.6.1.1, build id: 9A1F9489854CED36) (from thread 3037) (query_id: 77743723-1fcd-4b3d-babc-d0615e3ff40e) (query: SELECT * FROM
    2022.05.25 22:42:25.046871 [ 17397 ] {} <Fatal> BaseDaemon: 5. ./build_docker/../src/Common/Exception.cpp:47: DB::abortOnFailedAssertion()
    2022.05.25 22:42:25.181449 [ 17397 ] {} <Fatal> BaseDaemon: 6. ./build_docker/../src/Common/Exception.cpp:70: DB::Exception::Exception()
    2022.05.25 22:42:25.367710 [ 17397 ] {} <Fatal> BaseDaemon: 7. ./build_docker/../src/Client/MultiplexedConnections.cpp:0: DB::MultiplexedConnections::receivePacketUnlocked()
    2022.05.25 22:42:25.414201 [ 17397 ] {} <Fatal> BaseDaemon: 8. ./build_docker/../src/Client/MultiplexedConnections.cpp:0: DB::MultiplexedConnections::receivePacket()
    2022.05.25 22:42:25.493066 [ 17397 ] {} <Fatal> BaseDaemon: 9. ./build_docker/../src/QueryPipeline/RemoteQueryExecutor.cpp:279: DB::RemoteQueryExecutor::read()
    2022.05.25 22:42:25.612679 [ 17397 ] {} <Fatal> BaseDaemon: 10. ./build_docker/../src/Processors/Sources/RemoteSource.cpp:0: DB::RemoteSource::tryGenerate()

Here are additional logs for this query:
    $ pigz -cd clickhouse-server.stress.log.gz | fgrep -a 77743723-1fcd-4b3d-babc-d0615e3ff40e | fgrep -e Connection -e Distributed -e Fatal
    2022.05.25 22:36:04.698671 [ 6613 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Trace> Connection (127.0.0.2:9000): Connecting. Database: (not specified). User: default
    2022.05.25 22:36:04.722568 [ 3419 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Trace> Connection (127.0.0.2:9000): Connecting. Database: (not specified). User: default
    2022.05.25 22:36:05.014432 [ 6613 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Trace> Connection (127.0.0.2:9000): Connected to ClickHouse server version 22.6.1.
    2022.05.25 22:36:05.091397 [ 6613 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Debug> Connection (127.0.0.2:9000): Sent data for 2 scalars, total 2 rows in 0.000125814 sec., 15602 rows/sec., 68.00 B (517.81 KiB/sec.), compressed 0.4594594594594595 times to 148.00 B (1.10 MiB/sec.)
    2022.05.25 22:36:05.301301 [ 3419 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Trace> Connection (127.0.0.2:9000): Connected to ClickHouse server version 22.6.1.
    2022.05.25 22:36:05.343140 [ 3419 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Debug> Connection (127.0.0.2:9000): Sent data for 2 scalars, total 2 rows in 0.000116304 sec., 16889 rows/sec., 68.00 B (559.80 KiB/sec.), compressed 0.4594594594594595 times to 148.00 B (1.19 MiB/sec.)
    2022.05.25 22:36:06.682535 [ 6613 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Trace> StorageDistributed (remote): (127.0.0.2:9000) Cancelling query because enough data has been read
    2022.05.25 22:36:06.778808 [ 3037 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Fatal> : Logical error: 'No more packets are available.'.
    2022.05.25 22:36:06.789505 [ 3419 ] {77743723-1fcd-4b3d-babc-d0615e3ff40e} <Trace> StorageDistributed (remote): (127.0.0.2:9000) Cancelling query because enough data has been read
    2022.05.25 22:42:24.971173 [ 17397 ] {} <Fatal> BaseDaemon: (version 22.6.1.1, build id: 9A1F9489854CED36) (from thread 3037) (query_id: 77743723-1fcd-4b3d-babc-d0615e3ff40e) (query: SELECT * FROM

</details>

So between cancelling different sources the LOGICAL_ERROR occured, I
believe that this is because of the race:

    T1:                                        T2:
    RemoteQueryExecutor::read()
    checks was_cancelled
                                               RemoteQueryExecutor::tryCancel()
                                               connections->cancel()
    calls connections->receivePacket()

Note, for this problem async_socket_for_remote/use_hedged_requests
should be disabled, and original settings was:
- --max_parallel_replicas=3
- --use_hedged_requests=false
- --allow_experimental_parallel_reading_from_replicas=3

CI: https://s3.amazonaws.com/clickhouse-test-reports/37469/41cb029ed23e77f3a108e07e6b1b1bcb03dc7fcf/stress_test__undefined__actions_/fatal_messages.txt
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-07 08:20:32 +03:00
Robert Schulze
2d87af2a15
Merge pull request #37647 from DevTeamBK/Fix-all-CheckTriviallyCopyableMove-Errors 
Fix errors of CheckTriviallyCopyableMove type
 2022-06-05 19:58:47 +02:00
Nikolai Kochetov
00395e752e Cleanup 2022-06-02 16:59:14 +00:00
Nikolai Kochetov
506e102c17 Fix quotas. 2022-06-02 09:27:08 +00:00
Nikolai Kochetov
e401ab8169 Fix more tests. 2022-06-01 11:51:56 +00:00
Nikolai Kochetov
04c14e9c5d Fix tests and add comment. 2022-05-31 20:59:50 +00:00
Nikolai Kochetov
9954c59dc1 Update test. 2022-05-31 19:40:50 +00:00
Nikolai Kochetov
147a819221 Refactor a little bit more. 2022-05-31 14:43:38 +00:00
Nikolai Kochetov
df0d580a8c Fix another one test. 2022-05-30 19:29:57 +00:00
HeenaBansal2009
b7eb6bbd38 Fixed clang-tidy-CheckTriviallyCopyableMove-errors 2022-05-30 11:09:03 -07:00
Nikolai Kochetov
5ef51ed27b Fix more tests. 2022-05-30 13:10:30 +00:00
Nikolai Kochetov
b80b1940ce Fix some tests. 2022-05-27 20:47:35 +00:00
Nikolai Kochetov
6d4a26afac Update ReadProgressCallback. 2022-05-25 19:45:48 +00:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
Nikolai Kochetov
3d84aae0ab Better. 2022-05-24 20:06:08 +00:00
Nikolai Kochetov
fd97a9d885 Move some resources 2022-05-23 19:47:32 +00:00
Nikolai Kochetov
9756b759c6 Move some resources 2022-05-23 13:46:57 +00:00
Nikolai Kochetov
56feef01e7 Move some resources 2022-05-20 19:49:31 +00:00
Nikolai Kochetov
0a715b26db Move some resources. 2022-05-13 20:02:28 +00:00
Vladimir C
4cb6ca2311
Merge pull request #36970 from ClickHouse/concurrent_join 
Follow up for ConcurrentHashJoin
 2022-05-12 15:09:29 +02:00
Kseniia Sumarokova
2a4f763a5e
Merge pull request #37075 from kssenii/fix-progress-for-insert-select 
Fix progress indication for `INSERT SELECT` in clickhouse-local and for file progress in client
 2022-05-12 12:05:53 +02:00
Maksim Kita
4e7d10297b Fixed style 2022-05-11 21:59:51 +02:00
Maksim Kita
ce92f6aab1 Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
8ceb63ee6c Added JIT compilation of SortDescription 2022-05-11 21:59:51 +02:00
Nikolai Kochetov
ec34761d9f
Merge pull request #33631 from ClickHouse/grouping-sets-fix 
Support GROUPING SETS
 2022-05-11 21:28:46 +02:00
Nikolai Kochetov
2d99f0ce13 Simplify code a little bit. 2022-05-11 12:16:15 +00:00
kssenii
61f2737e17 Merge master 2022-05-10 19:31:22 +02:00
kssenii
0feda67ec4 Fix progress for insert select 2022-05-10 15:04:08 +02:00
Anton Popov
e911900054 remove last mentions of data streams 2022-05-09 19:15:24 +00:00
mergify[bot]
55a6d22ad3
Merge branch 'master' into grouping-sets-fix 2022-05-09 14:02:10 +00:00
Nikolai Kochetov
ebfeca2c86 Try to move some code from SourceWithProgress to PipelineExecutor. 2022-05-09 10:28:05 +00:00
vdimir
d712985575
Minor style changes for ConcurrentHashJoin 2022-05-06 15:17:46 +00:00
Vladimir C
bd5fab97d9
Merge pull request #36415 from bigo-sg/concurrent_join 2022-05-06 17:11:10 +02:00
Alexey Milovidov
996d838ca3
Merge pull request #36817 from DevTeamBK/Clang-tidy-Fixes 
Clang -Tidy Fixes
 2022-05-06 02:01:00 +03:00
Dmitry Novik
4cc26aa38b Merge remote-tracking branch 'origin/master' into grouping-sets-fix 
And fix execution of the query with only one grouping set
 2022-05-05 17:14:52 +00:00
lgbo-ustc
35ad33ef75 update codes 2022-05-05 09:23:49 +08:00
mergify[bot]
e0fa563092
Merge branch 'master' into fix_exception_message 2022-05-04 15:56:03 +00:00
Alexander Tokmakov
dbbe2062af fix tests 2022-05-03 21:30:03 +02:00
Alexander Tokmakov
b96dbdf711 fix test 2022-05-03 20:20:27 +02:00
Larry Luo
ee131eefd8 Removed move for trivially-copyable type and added noexcept for move constructor 2022-05-02 18:46:34 -07:00
Kruglov Pavel
188aa3b694
Merge pull request #36781 from Avogar/fix-timeout-message 
Better exception messages while socket timeouts
 2022-05-02 13:12:35 +02:00
Azat Khuzhin
d82f2ff162 Set is_all_data_sent on exceptions too 
This will avoid clickhouse-test complains about left queries after the
test, like in [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/36258/9646487c093a75dc31e3e818417cfad83580b40f/stateful_tests__memory__actions_.html

Follow-up for: #36649
 2022-04-30 13:00:44 +03:00
avogar
04846a39a1 Better exception messages while socket timeouts 2022-04-29 09:04:24 +00:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
lgbo-ustc
5a6b802aa8 update 2022-04-29 09:29:04 +08:00
alesapin
94733a27ec
Merge pull request #36636 from ucasfl/pipe 
remove unneeded if statement
 2022-04-28 11:28:48 +02:00
fenglv
f13f9fa7d7 fix typo 2022-04-27 12:35:56 +00:00
fenglv
1b84d59047 fix typo 
modify comment
 2022-04-27 12:24:49 +00:00
lgbo-ustc
5738871a8b update QueryPipelineBuilder::joinPipelines 2022-04-27 10:24:19 +08:00
lgbo-ustc
520b05b9f1 update test case tests/queries/0_stateless/02236_explain_pipeline_join.sql 2022-04-27 10:08:22 +08:00
lgbo-ustc
0b0fa8453b fixed bug: resize on left pipeline cause the order by result wrong 2022-04-26 18:06:16 +08:00
lgbo-ustc
74ccc233d2 Merge remote-tracking branch 'ck/master' into concurrent_join 2022-04-26 09:21:02 +08:00
fenglv
7b2422ccef remove unneeded if statement 2022-04-25 16:31:37 +00:00
lgbo-ustc
981d560553 Merge remote-tracking branch 'ck/master' into concurrent_join 2022-04-25 13:00:04 +08:00
Amos Bird
bff54af46e
Better exception for ConnectionCollector 2022-04-24 19:44:05 +08:00
Amos Bird
a25bb50096
Refactor many exception messages 
1. Always use fmt variant
2. Remove redundant period at the end of message
3. Remove useless parenthesis
 2022-04-24 19:44:00 +08:00
Amos Bird
e73d7fea31
Rename narrowBlockInputStream to narrowPipe 2022-04-24 18:33:48 +08:00
Maksim Kita
57444fc7d3
Merge pull request #36444 from rschu1ze/clang-tidy-fixes 
Clang tidy fixes
 2022-04-21 16:11:27 +02:00
Dmitry Novik
61deae7105
Merge branch 'master' into grouping-sets-fix 2022-04-21 03:34:42 +02:00
Dmitry Novik
6e73cd6929 Implement parallel grouping sets processing 2022-04-21 01:18:40 +00:00
lgbo-ustc
c844415578 add new join_algorithm parallel_hash 2022-04-20 19:47:16 +08:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings 
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
 2022-04-20 10:29:05 +02:00
lgbo-ustc
3f2fe3ff66 fixed code style 2022-04-19 18:47:29 +08:00
lgbo-ustc
3d7338581b Improve join 
now adding joined blocks from right table can be run parallelly, speedup the join process
 2022-04-19 16:07:30 +08:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
Dmitry Novik
a16710c750 Merge remote-tracking branch 'origin/master' into grouping-sets-fix 2022-04-14 17:29:51 +00:00
Alexander Tokmakov
da00beaf7f Merge branch 'master' into mvcc_prototype 2022-04-05 11:14:42 +02:00
何李夫
09c04e4993
Improve the pipeline description for JOIN (#35612) 
Improve the pipeline description for JOIN
 2022-04-04 13:56:41 +02:00
Alexander Tokmakov
6591d1ceb7 Merge branch 'master' into mvcc_prototype 2022-04-01 15:38:46 +02:00
Amos Bird
12449caf89
Refactoring QueryPipeline 2022-03-31 19:26:35 +08:00