thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 03:42:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
109,051 Commits 9,315 Branches 1,783 Tags 2.1 GiB
2e921e3d6b
Commit Graph

4133 Commits

Author SHA1 Message Date
Azat Khuzhin
a64f6b5f3e Fix possible (likely distributed) query hung 
Recently I saw the following, the client executed long distributed query
and terminated the connection, and in this case query cancellation will
be done from PullingAsyncPipelineExecutor dtor, but during cancellation
one of nodes sent ECONNRESET, and this leads to an exception from
PullingAsyncPipelineExecutor::cancel(), and this leads to a deadlock
when multiple threads waits each others, because cancel() for
LazyOutputFormat wasn't called.

Here is as relevant portion of logs:

    2023.01.04 08:26:09.236208 [ 37968 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Debug> executeQuery: (from 10.61.13.253:44266, user: default)  TooLongDistributedQueryToPost
    ...
    2023.01.04 08:26:09.262424 [ 37968 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Trace> MergeTreeInOrderSelectProcessor: Reading 1 ranges in order from part 9_330_538_18, approx. 61440 rows starting from 0
    2023.01.04 08:26:09.266399 [ 26788 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Trace> Connection (s4.ch:9000): Connecting. Database: (not specified). User: default
    2023.01.04 08:26:09.266849 [ 26788 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Trace> Connection (s4.ch:9000): Connected to ClickHouse server version 22.10.1.
    2023.01.04 08:26:09.267165 [ 26788 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Debug> Connection (s4.ch:9000): Sent data for 2 scalars, total 2 rows in 3.1587e-05 sec., 62635 rows/sec., 68.00 B (2.03 MiB/sec.), compressed 0.4594594594594595 times to 148.00 B (4.41 MiB/sec.)
    2023.01.04 08:39:13.047170 [ 37968 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Error> PullingAsyncPipelineExecutor: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (10.7.142.115:9000). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

    0. ./.build/./contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x1818234c in /usr/lib/debug/usr/bin/clickhouse.debug
    1. ./.build/./src/Common/Exception.cpp:69: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x1004fbda in /usr/lib/debug/usr/bin/clickhouse.debug
    2. ./.build/./src/Common/NetException.h:12: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x14e352f3 in /usr/lib/debug/usr/bin/clickhouse.debug
    3. ./.build/./src/IO/BufferBase.h:39: DB::Connection::sendCancel() @ 0x15c21e6b in /usr/lib/debug/usr/bin/clickhouse.debug
    4. ./.build/./src/Client/MultiplexedConnections.cpp:0: DB::MultiplexedConnections::sendCancel() @ 0x15c4d5b7 in /usr/lib/debug/usr/bin/clickhouse.debug
    5. ./.build/./src/QueryPipeline/RemoteQueryExecutor.cpp:627: DB::RemoteQueryExecutor::tryCancel(char const*, std::__1::unique_ptr<DB::RemoteQueryExecutorReadContext, std::__1::default_delete<DB::RemoteQueryExecutorReadContext> >*) @ 0x14446c09 in /usr/lib/debug/usr/bin/clickhouse.debug
    6. ./.build/./contrib/libcxx/include/__iterator/wrap_iter.h💯 DB::ExecutingGraph::cancel() @ 0x15d2c0de in /usr/lib/debug/usr/bin/clickhouse.debug
    7. ./.build/./contrib/libcxx/include/__memory/unique_ptr.h:300: DB::PullingAsyncPipelineExecutor::cancel() @ 0x15d32055 in /usr/lib/debug/usr/bin/clickhouse.debug
    8. ./.build/./contrib/libcxx/include/__memory/unique_ptr.h:312: DB::PullingAsyncPipelineExecutor::~PullingAsyncPipelineExecutor() @ 0x15d31f4f in /usr/lib/debug/usr/bin/clickhouse.debug
    9. ./.build/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x15cde919 in /usr/lib/debug/usr/bin/clickhouse.debug
    10. ./.build/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x15cd8554 in /usr/lib/debug/usr/bin/clickhouse.debug
    11. ./.build/./src/Server/TCPHandler.cpp:1904: DB::TCPHandler::run() @ 0x15ce6479 in /usr/lib/debug/usr/bin/clickhouse.debug
    12. ./.build/./contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x18074f07 in /usr/lib/debug/usr/bin/clickhouse.debug
    13. ./.build/./contrib/libcxx/include/__memory/unique_ptr.h:54: Poco::Net::TCPServerDispatcher::run() @ 0x180753ed in /usr/lib/debug/usr/bin/clickhouse.debug
    14. ./.build/./contrib/poco/Foundation/src/ThreadPool.cpp:213: Poco::PooledThread::run() @ 0x181e3807 in /usr/lib/debug/usr/bin/clickhouse.debug
    15. ./.build/./contrib/poco/Foundation/include/Poco/SharedPtr.h:156: Poco::ThreadImpl::runnableEntry(void*) @ 0x181e1483 in /usr/lib/debug/usr/bin/clickhouse.debug
    16. ? @ 0x7ffff7e55fd4 in ?
    17. ? @ 0x7ffff7ed666c in ?
     (version 22.10.1.1)

And here is the state of the threads:

<details>

<summary>system.stack_trace</summary>

```sql
SELECT
    arrayStringConcat(arrayMap(x -> demangle(addressToSymbol(x)), trace), '\n') AS sym
FROM system.stack_trace
WHERE query_id = 'f2ed6149-146d-4a3d-874a-b0b751c7b567'
SETTINGS allow_introspection_functions=1

Row 1:
──────
sym:
pthread_cond_wait
std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)
bool ConcurrentBoundedQueue<DB::Chunk>::emplaceImpl<DB::Chunk>(std::__1::optional<unsigned long>, DB::Chunk&&)
DB::IOutputFormat::work()
DB::ExecutionThreadContext::executeTask()
DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*)

Row 2:
──────
sym:
pthread_cond_wait
Poco::EventImpl::waitImpl()
DB::PipelineExecutor::joinThreads()
DB::PipelineExecutor::executeImpl(unsigned long)
DB::PipelineExecutor::execute(unsigned long)

Row 3:
──────
sym:
pthread_cond_wait
Poco::EventImpl::waitImpl()
DB::PullingAsyncPipelineExecutor::Data::~Data()
DB::PullingAsyncPipelineExecutor::~PullingAsyncPipelineExecutor()
DB::TCPHandler::processOrdinaryQueryWithProcessors()
DB::TCPHandler::runImpl()
DB::TCPHandler::run()
Poco::Net::TCPServerConnection::start()
Poco::Net::TCPServerDispatcher::run()
Poco::PooledThread::run()
Poco::ThreadImpl::runnableEntry(void*)
```

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-21 08:05:56 +01:00
Azat Khuzhin
e2fcf0f072 Catch exception on query cancellation 
Since we still want to join the thread, yes it will be done in dtor, but
this looks better.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-21 08:05:56 +01:00
Azat Khuzhin
0566f72d36 Cleanup PullingAsyncPipelineExecutor::cancel() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-21 08:05:56 +01:00
avogar
86336940f8 Better comment 2023-01-20 16:41:59 +00:00
avogar
4432ee9927 Fix aborts in arrow lib 2023-01-20 16:40:33 +00:00
avogar
550a703fbc Make a bit better 2023-01-20 14:58:39 +00:00
Kruglov Pavel
28ddcc2432
Merge branch 'master' into tsv-csv-detect-header 2023-01-20 15:08:38 +01:00
avogar
c34c0aa22e Fix comments 2023-01-19 16:03:46 +00:00
Kruglov Pavel
9820beae68
Apply suggestions from code review 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-01-19 16:11:13 +01:00
Igor Nikonov
d0ce804bfc Fix: dynamic_cast -> typeid_cast for SortingStep 2023-01-19 13:40:21 +00:00
Igor Nikonov
df3776d24b Make test stable 
+ disable debug logging
 2023-01-19 11:43:40 +00:00
Ilya Yatsishin
00962b7ad5
Merge pull request #45424 from Avogar/fix-json-import-nested 2023-01-19 10:31:40 +01:00
Igor Nikonov
57d2fd300a Fix: correct update of data stream sorting properties after removing 
sorting
 2023-01-19 00:11:58 +00:00
avogar
a8f20363f4 Fix JSON/BSONEachRow parsing with HTTP 2023-01-18 22:49:03 +00:00
Igor Nikonov
1866f990de
Revert "Revert "Remove redundant sorting"" 2023-01-18 20:12:34 +01:00
Igor Nikonov
7ed8fec94f
Revert "Remove redundant sorting" 2023-01-18 18:38:25 +01:00
flynn
4dcd3cc4f6 fix style 2023-01-18 08:46:06 +00:00
flynn
a8e1363151 implement storage iceberg 2023-01-18 03:43:56 +00:00
Robert Schulze
4f90824347
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-17 22:49:53 +00:00
Igor Nikonov
0db9bf38a2
Merge branch 'master' into igor/remove_redundant_order_by 2023-01-17 22:26:24 +01:00
Alexander Tokmakov
8b13b85ea0
Merge pull request #44543 from ClickHouse/text_log_add_pattern 
Add a column with a message pattern to text_log
 2023-01-17 20:19:32 +03:00
Igor Nikonov
0cfa08df7a Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-17 16:28:17 +00:00
Igor Nikonov
9855504403 Rename source file according to implementation 2023-01-17 16:24:51 +00:00
Kruglov Pavel
96bb99f864
Merge branch 'master' into tsv-csv-detect-header 2023-01-17 15:33:02 +01:00
Igor Nikonov
6328e02f22 Fix: update input/output stream properties 
After removing sorting step we need to update sorting properties of
input/ouput streams
 2023-01-17 13:39:18 +00:00
Sema Checherinda
35431e91e3
Merge pull request #45276 from ucasfl/avro-fix 
Fix some avro reading bugs
 2023-01-17 12:48:44 +01:00
avogar
5bf4704e7a Support FixedSizeBinary type in Parquet/Arrow 2023-01-16 21:01:31 +00:00
avogar
1c0941d72a Add docs and examples 2023-01-16 16:46:41 +00:00
avogar
3ea80b0f54 Merge branch 'master' of github.com:ClickHouse/ClickHouse into tsv-csv-detect-header 2023-01-16 15:14:25 +00:00
Igor Nikonov
a34991cb65 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-16 12:14:02 +00:00
Robert Schulze
099e30ef2a
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-16 08:04:49 +00:00
Robert Schulze
27fe7ebd93
Cosmetics 2023-01-15 16:12:48 +00:00
flynn
29eb30b49f Fix some reading avro format bugs 
fix
 2023-01-14 18:05:26 +00:00
Alexander Tokmakov
2d7773fccc Merge branch 'master' into text_log_add_pattern 2023-01-13 20:33:46 +01:00
avogar
e2470dd670 Fix tests 2023-01-13 17:03:53 +00:00
Robert Schulze
d7d3f61c73
Cleanup SourceFromChunks a bit 2023-01-13 10:57:31 +00:00
Robert Schulze
15e11741cb
Cosmetics 2023-01-13 00:00:23 +00:00
Robert Schulze
1b53307375
Refcount cache entries to improve lookup performance 2023-01-12 23:38:39 +00:00
Robert Schulze
0c528fca29
Introduce SourceFromChunks to avoid unnecessary partial result chunk concatenation 2023-01-12 23:01:00 +00:00
avogar
b461935374 Better 2023-01-12 13:11:04 +00:00
Kruglov Pavel
05a11ff4a4
Merge branch 'master' into tsv-csv-detect-header 2023-01-12 12:35:18 +01:00
Maksim Kita
a140d6c5b1 Fixed code review issues 2023-01-12 12:07:58 +01:00
Maksim Kita
47f4159909 Analyzer support distributed queries processing 2023-01-12 12:07:58 +01:00
avogar
e4d774d906 Better naming 2023-01-11 22:57:14 +00:00
avogar
26cd56d113 Fix tests, make better 2023-01-11 22:52:15 +00:00
Alexander Gololobov
6adf1c025f
Merge pull request #45165 from ClickHouse/more_logging_for_replicated_ttl 
More logging to facilitate debugging of flaky test_ttl_replicated
 2023-01-11 23:24:48 +01:00
avogar
3b45863d15 Make better implementation, fix tests 2023-01-11 17:12:56 +00:00
Alexander Gololobov
659fa96365 More logging to facilitate debugging 2023-01-11 13:06:38 +01:00
Nikolai Kochetov
5e7a6ac619
Merge pull request #45122 from ClickHouse/revert-45121-revert-44653-custom-reading-for-mutation 
Revert "Revert "Custom reading for mutation""
 2023-01-11 12:37:32 +01:00
avogar
6312b75f44 Fix style 2023-01-10 16:28:52 +00:00
avogar
615fe4cecb Fix tests 2023-01-10 16:27:23 +00:00
Maksim Kita
fbba28b31e Analyzer aggregation without column fix 2023-01-10 16:49:55 +01:00
Nikolai Kochetov
4673b3fe1d
Revert "Revert "Custom reading for mutation"" 2023-01-10 16:31:01 +01:00
Alexander Tokmakov
c8ec130be4
Revert "Custom reading for mutation" 2023-01-10 17:51:30 +03:00
Nikolai Kochetov
11418963c0
Merge pull request #44653 from ClickHouse/custom-reading-for-mutation 
Custom reading for mutation
 2023-01-10 12:16:24 +01:00
Robert Schulze
923fa2c15a
Fix review comments, pt. II 2023-01-10 10:21:08 +00:00
Robert Schulze
0c3b034887
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-10 09:02:41 +00:00
Alexey Milovidov
1229a20fb3
Merge pull request #45047 from ClickHouse/fix-buffer-overflow 
Fix buffer overflow in parser
 2023-01-10 05:06:03 +03:00
Yakov Olkhovskiy
4f32f3b8cb
Merge pull request #44484 from bigo-sg/arrow_struct_field 
Optimization for reading struct fields in parquet/orc files
 2023-01-09 15:36:26 -05:00
Nikolai Kochetov
3e430b1b6e Add comments. Remove unused flag from Context. 2023-01-09 19:52:11 +00:00
Kseniia Sumarokova
119501f1d9
Merge pull request #44698 from Avogar/parquet-bool 
Support Bool type in Arrow/Parquet/ORC
 2023-01-09 12:56:28 +01:00
lgbo-ustc
b639bcabc0 some fixes 2023-01-09 18:13:00 +08:00
lgbo-ustc
f127b3a60a update ArrowFieldIndexUtil 2023-01-09 18:13:00 +08:00
lgbo-ustc
a3bdfddc9d support nested table 2023-01-09 18:13:00 +08:00
lgbo-ustc
4f3f781b85 fixed test case 2023-01-09 18:13:00 +08:00
lgbo-ustc
755f03db4e fixed 2023-01-09 18:13:00 +08:00
lgbo-ustc
f6850d96cb fixed missing columns 2023-01-09 18:13:00 +08:00
lgbo-ustc
4cf6beee27 fixed 2023-01-09 18:13:00 +08:00
lgbo-ustc
81e2832133 fixed 2023-01-09 18:13:00 +08:00
lgbo-ustc
77cea49cec fixed including header failure 2023-01-09 18:13:00 +08:00
lgbo-ustc
8f8f6f966b Optimization for reading struct fields in parquet/orc files 2023-01-09 18:13:00 +08:00
Igor Nikonov
7298b39a18 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-09 10:08:48 +00:00
Igor Nikonov
cec7dc5a5a Renaming according to setting 2023-01-09 10:06:47 +00:00
Igor Nikonov
e29cd5d640 Rename settings to remove_redundant_sorting 2023-01-09 09:50:00 +00:00
Alexey Milovidov
eea5a2c521
Merge branch 'master' into fix-buffer-overflow 2023-01-09 09:15:29 +03:00
Alexey Milovidov
0d39d26a34 Don't fix parallel formatting 2023-01-09 06:15:20 +01:00
Alexey Milovidov
d331f0ce82 Fix buffer overflow in parser 2023-01-09 03:31:12 +01:00
Alexey Milovidov
0d8d6eca67 Non significant changes 2023-01-09 01:51:05 +01:00
Alexey Milovidov
797dfdd42e Non significant changes 2023-01-09 01:50:19 +01:00
Alexey Milovidov
3430df2f8c Non significant changes 2023-01-09 01:48:57 +01:00
avogar
ee72799121 Fix tests, make better 2023-01-06 20:46:43 +00:00
Robert Schulze
45dbcf88e5
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-06 20:42:48 +00:00
Igor Nikonov
2187bdd4cc Disable diagnostics 
+ cleanup
+ disable optimization in sort performance test since it removes sorting
  at all
 2023-01-06 17:00:05 +00:00
Igor Nikonov
d2d5a09802 Fix typo 2023-01-06 15:01:31 +00:00
Igor Nikonov
48b58a9fed Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-06 13:55:26 +00:00
Igor Nikonov
d886b56338 Fix review comments 2023-01-06 13:54:28 +00:00
avogar
7fcdb08ec6 Detect header in CSV/TSV/CustomSeparated files automatically 2023-01-05 22:57:25 +00:00
Robert Schulze
4fcead15c3
Fix handling of exceptions during query runtime 2023-01-04 22:33:04 +00:00
Yakov Olkhovskiy
7a5a36cbed
Merge branch 'master' into refactoring-ip-types 2023-01-04 11:11:06 -05:00
Kruglov Pavel
b9bdf62bf3
Merge branch 'master' into parquet-bool 2023-01-04 14:49:41 +01:00
Kruglov Pavel
59263f3ae1
Merge pull request #44501 from Avogar/validate-types 
Validate data types according to settings.
 2023-01-04 14:48:09 +01:00
Kruglov Pavel
90ae405033
Merge pull request #44876 from Avogar/fix-perf-tests 
Revert some changes from #42777 to fix performance tests
 2023-01-04 14:27:17 +01:00
Kruglov Pavel
0c7d39ac7f
Merge pull request #44832 from ucasfl/row-number 
Fix output_format_pretty_row_numbers does not preserve the counter across the blocks
 2023-01-04 14:15:47 +01:00
Igor Nikonov
1edfdfda4d Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-04 12:34:40 +00:00
Kruglov Pavel
4e261ab230
Fix JSONCompactEachRow 2023-01-03 21:16:38 +01:00
Kruglov Pavel
314d95fd71
Fix special build 2023-01-03 20:34:30 +01:00
avogar
28eb2dbd4c Revert some changes from #42777 2023-01-03 18:53:03 +00:00
Alexey Milovidov
e855d3519a
Merge branch 'master' into refactoring-ip-types 2023-01-02 21:58:53 +03:00
Kruglov Pavel
4a7c399076
Merge branch 'master' into parquet-bool 2023-01-02 16:33:42 +01:00
Kruglov Pavel
0a43976977
Merge branch 'master' into validate-types 2023-01-02 16:10:14 +01:00
Kruglov Pavel
1c2dc05d6e
Merge pull request #44446 from Avogar/arrow-nullables 
Respect setting settings.schema_inference_make_columns_nullable in Parquet/ORC/Arrow formats
 2023-01-02 16:05:57 +01:00
Kruglov Pavel
966f57ef68
Merge pull request #42777 from Avogar/improve-streaming-engines 
Refactor and Improve streaming engines Kafka/RabbitMQ/NATS and data formats
 2023-01-02 15:59:06 +01:00
flynn
7780fae9db fix 2023-01-02 12:38:51 +00:00
flynn
7b487dd923 fix 2023-01-02 12:37:43 +00:00
flynn
3a1dd045dd Fix output_format_pretty_row_numbers does not preserve the counter across the blocks 2023-01-02 09:27:37 +00:00
Robert Schulze
e9e04166d9
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-02 08:20:27 +00:00
Kruglov Pavel
8479615c48
Merge pull request #44684 from Avogar/avro-bool 
Input/ouptut avro bool type as ClickHouse bool type
 2022-12-30 17:56:36 +01:00
Kruglov Pavel
4982d132fb
Merge branch 'master' into validate-types 2022-12-30 17:52:13 +01:00
Nikolay Degterinsky
dfe93b5d82
Merge pull request #42284 from Algunenano/perf_experiment 
Performance experiment
 2022-12-30 03:14:22 +01:00
Kruglov Pavel
894726bd8f
Merge branch 'master' into improve-streaming-engines 2022-12-29 22:59:45 +01:00
Nikolai Kochetov
3c02e208c8 Fix more tests. 2022-12-29 17:52:31 +00:00
Nikita Mikhaylov
ffaff7167b
Fix flaky tests (#44690) 2022-12-29 14:32:10 +01:00
Alexey Milovidov
875797ee63
Merge pull request #44685 from ClickHouse/remove-old-code-4 
Remove old code
 2022-12-29 15:43:59 +03:00
Alexey Milovidov
9378194320
Merge pull request #44688 from ClickHouse/improve-odbc-test 
Improve odbc test
 2022-12-29 15:40:47 +03:00
avogar
a0db1dd1ea Support Bool type in Arrow/Parquet/ORC 2022-12-28 22:58:28 +00:00
Alexey Milovidov
e524d4bcb1
Merge pull request #44673 from ClickHouse/logical-error-with-join-subquery-union-and-totals 
Fix LOGICAL_ERROR for JOIN with subquery, UNION and totals.
 2022-12-29 01:56:15 +03:00
Raúl Marín
5de11979ce
Unify query elapsed time measurements (#43455) 
* Unify query elapsed time reporting

* add-test: Make shell tests executable

* Add some tests around query elapsed time

* Style and ubsan
 2022-12-28 21:01:41 +01:00
Raúl Marín
e915ce1e95 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-12-28 20:15:43 +01:00
Raúl Marín
f6428964cc Better and common error handling 2022-12-28 20:15:27 +01:00
Alexey Milovidov
c3eff78f20 Don't be kind with third-party code 2022-12-28 20:07:26 +01:00
Alexey Milovidov
cf594a659d
Update MongoDBSource.cpp 2022-12-28 21:26:42 +03:00
Alexey Milovidov
33bcd07be5 Remove old code 2022-12-28 19:02:06 +01:00
avogar
f1191bbbc6 Input/ouptut avro bool type as ClickHouse bool type 2022-12-28 17:38:58 +00:00
Nikolai Kochetov
2d84ef2891 Fix tests and update IntersectOrExceptStep. 2022-12-28 14:44:45 +00:00
Nikolai Kochetov
21e93cd41d Fix LOGICAL_ERROR for JOIN with subquery, UNION and totals. 2022-12-28 13:00:32 +00:00
avogar
411f98306a Merge branch 'master' of github.com:ClickHouse/ClickHouse into validate-types 2022-12-27 19:24:15 +00:00
Kruglov Pavel
6dea7336f7
Merge pull request #44405 from Avogar/fix-parquet-orc 
Fix reading columns that are not presented in input data in Parquet/ORC formats
 2022-12-27 16:58:35 +01:00
Kruglov Pavel
e981b2e78b
Merge pull request #43813 from Avogar/fix-sum-if 
Better handling of NULL in aggregate combinators, fix possible segfault/logical error
 2022-12-27 16:50:48 +01:00
Alexey Milovidov
1b21cc018e
Merge pull request #44342 from ClickHouse/add-test-29883 
Add a test for #29883
 2022-12-27 14:55:14 +03:00
Raúl Marín
fc1fa82a39
Merge branch 'master' into perf_experiment 2022-12-27 10:51:58 +01:00
Alexander Tokmakov
ca989e9212 less runtime format strings 2022-12-23 19:50:34 +01:00
Nikolai Kochetov
5030c92373
Merge branch 'master' into fix-wrong-condition-for-async-reading-from-mt 2022-12-23 14:18:08 +01:00
Nikolai Kochetov
4f84becfa0 Fix wrong condition for enabling asyn creading from MergeTree. 2022-12-23 13:00:23 +00:00
Igor Nikonov
e2e898b707 Remove incorrect assert 2022-12-23 12:12:25 +00:00
Vladimir C
7482ea54ab
Merge pull request #43972 from ClickHouse/vdimir/tmp-data-in-fs-cache-2 2022-12-23 11:59:27 +01:00
Dmitry Novik
cff882d506 Merge remote-tracking branch 'origin/master' into refector-function-node 2022-12-22 21:34:29 +00:00
Igor Nikonov
95c3703bbf Remove incorrect assert 
ExpressionStep is not always under SortingStep
It can be ReadFromRemote in distributed query
 2022-12-22 19:32:41 +00:00
Kruglov Pavel
6a017a6586
Merge pull request #43379 from Avogar/better-capn-proto 
Add small improvements in CapnProto format
 2022-12-22 14:50:10 +01:00
vdimir
182b34c11e
Fixes 2022-12-22 10:22:57 +00:00
Yakov Olkhovskiy
a8cb29da4b
Merge branch 'master' into refactoring-ip-types 2022-12-21 23:56:24 -05:00
avogar
4ab3e90382 Validate types in table function arguments/CAST function arguments/JSONAsObject schema inference 2022-12-21 21:21:30 +00:00
Igor Nikonov
c7650850fe Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-21 15:56:08 +00:00
Igor Nikonov
919fa9b84e If aggregation step is in order, - do not remove underneath sorting step 2022-12-21 12:55:42 +00:00
Igor Nikonov
fdb277245c Remove current sorting only if next is full sorting 2022-12-21 12:21:35 +00:00
Kruglov Pavel
5e01a3d74e
Merge branch 'master' into improve-streaming-engines 2022-12-21 10:51:50 +01:00
Kruglov Pavel
09ab5832b1
Merge pull request #44382 from Avogar/fix-bson-object-id 
Fix reading ObjectId in BSON schema inference
 2022-12-21 10:48:50 +01:00
Igor Nikonov
1624647f07 Fix: distributed query with cluster() table function 2022-12-20 23:27:01 +00:00
Alexey Milovidov
a6f50edde3
Merge branch 'master' into add-test-29883 2022-12-21 01:45:18 +03:00
Igor Nikonov
5cd4767838 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-20 17:56:59 +00:00
avogar
c49638e3a9 Respect setting settings.schema_inference_make_columns_nullable in Parquet/ORC/Arrow formats 2022-12-20 17:46:42 +00:00
Dmitry Novik
4793412887
Merge branch 'master' into refector-function-node 2022-12-20 18:26:19 +01:00
Kruglov Pavel
643a35bed1
Merge pull request #44019 from Avogar/refactor-schema-inference 
Refactor and improve schema inference for text formats
 2022-12-20 17:29:03 +01:00
Kruglov Pavel
c0b17ca0af
Merge branch 'master' into fix-bson-object-id 2022-12-20 17:18:10 +01:00
Kruglov Pavel
a0ee1c1227
Merge branch 'master' into fix-sum-if 2022-12-20 15:20:23 +01:00
Kruglov Pavel
fe28faa32d
Fix style 2022-12-20 14:49:39 +01:00
Kruglov Pavel
3f1e40aacd
Merge branch 'master' into fix-orc 2022-12-20 13:32:46 +01:00
Igor Nikonov
15bb4cb65c Fix distributed query check 2022-12-20 10:21:32 +00:00
Raúl Marín
45d27f461b
Merge branch 'master' into perf_experiment 2022-12-20 09:07:48 +00:00
Igor Nikonov
2164b18485 Disable the optimization for distributed queries 
+ based on ReadFromRemote
- improve by disabling optimization only between Union and ReadFromRemote
 2022-12-19 23:24:35 +00:00
avogar
e262e375dc Fix reading columns that are not presented in input data in Parquet/ORC formats 2022-12-19 20:30:54 +00:00
avogar
0c406adce2 Fix reading Map type in ORC format 2022-12-19 18:23:07 +00:00
Igor Nikonov
1e53b3be38 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-19 16:44:22 +00:00
Nikolai Kochetov
860afd1832 Add a very bad fix. 2022-12-19 16:33:10 +00:00
avogar
21cdf6e6ae Fix reading ObjectId in BSON schema inference 2022-12-19 14:13:42 +00:00
Robert Schulze
b93c56323c
Merge remote-tracking branch 'origin/master' into query-result-cache 2022-12-19 09:18:36 +00:00
Alexey Milovidov
3e8dcdc155 Add earlier check 2022-12-18 08:01:43 +01:00
avogar
22e70fa71e Try fix tests 2022-12-16 18:09:42 +00:00
Dmitry Novik
875a24a650 Merge remote-tracking branch 'origin/master' into refector-function-node 2022-12-16 16:07:30 +00:00
avogar
291e51c533 Merge branch 'better-capn-proto' of github.com:Avogar/ClickHouse into better-capn-proto 2022-12-16 14:43:06 +00:00
avogar
4a51bdce86 Fix comments 2022-12-16 13:58:54 +00:00
Kruglov Pavel
3fad5c7f1f
Merge branch 'master' into refactor-schema-inference 2022-12-16 14:24:51 +01:00
avogar
34274be8df Fix use after free 2022-12-16 13:20:36 +00:00
Igor Nikonov
319158f32f Merge remote-tracking branch 'origin/master' into igor/remove_redundant_distinct 2022-12-16 11:46:27 +00:00
Robert Schulze
4db33f16e3
Make DoS attack on cache less likely 2022-12-15 21:42:51 +00:00
Igor Nikonov
ea0bd38ab1 Avoid to remove sorting expression for now 2022-12-15 21:11:02 +00:00
Robert Schulze
bcc54ab9b4
Add cfg for min query duration for caching 2022-12-15 20:10:37 +00:00
avogar
cfcb444699 Merge branch 'master' of github.com:ClickHouse/ClickHouse into better-capn-proto 2022-12-15 20:04:43 +00:00
Kruglov Pavel
25f199dd89
Merge pull request #43332 from Avogar/csv-custom-delimiter 
Improve reading CSV field in CustomSeparated/Template format
 2022-12-15 21:03:29 +01:00
Igor Nikonov
629f317d7d Fix for full sorting merge join 2022-12-15 20:01:50 +00:00
avogar
f26b5b57ad Better comment 2022-12-15 19:52:57 +00:00
avogar
5b4ea53fed Fix fields order 2022-12-15 19:52:25 +00:00
avogar
755b08a49e Fix comments 2022-12-15 19:47:10 +00:00
Kruglov Pavel
c5b2e4cc23
Merge branch 'master' into improve-streaming-engines 2022-12-15 18:44:35 +01:00
Robert Schulze
537c64aad2
Merge remote-tracking branch 'origin/master' into query-result-cache 
Conflicts:
- tests/queries/0_stateless/02117_show_create_table_system.reference
 2022-12-15 12:17:58 +00:00
Robert Schulze
f1e367179e
Make maximum record count per item configurable 2022-12-15 11:16:44 +00:00
Robert Schulze
e52dfaf45c
Make number of cached items configurable 2022-12-15 10:08:28 +00:00
Igor Nikonov
bde3e43d3b Merge remote-tracking branch 'origin/master' into igor/remove_redundant_distinct 2022-12-15 00:36:06 +00:00
Igor Nikonov
6dd2849ae6 Fix for distributed queries 
+ 01952_optimize_distributed_group_by_sharding_key
 2022-12-14 21:54:20 +00:00
avogar
a94a0d9c85 Fix tests, fix bugs 2022-12-14 21:17:00 +00:00
Igor Nikonov
b5de352861 Move checks to functions 2022-12-14 17:51:02 +00:00
Igor Nikonov
6ca0773516 Fix for distributed query 
- do not apply optimization in case of UnionStep on top of Sorting

01104_distributed_one_test.sql
 2022-12-14 16:10:52 +00:00
Dmitry Novik
1df038e39c Merge remote-tracking branch 'origin/master' into refector-function-node 2022-12-14 14:55:30 +00:00
Igor Nikonov
03f7bd1192 Fix for distributed queries 2022-12-14 13:54:42 +00:00
Igor Nikonov
e25053dec0 Query plan visitor with debug logs 2022-12-13 22:46:39 +00:00
Igor Nikonov
1cbdce8eea Fix window function case 2022-12-13 16:21:14 +00:00
Nikolay Degterinsky
9b6d31b95d
Merge branch 'master' into perf_experiment 2022-12-13 17:15:07 +01:00
Igor Nikonov
1bed5064e7 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-13 10:55:50 +00:00
Igor Nikonov
6f5fc6645b Fix: remove ORDER BY before "mergeExpressions" optimization 
When removing ORDER BY we remove corresponding expression,
so to remove only ORDER BY expression, do it before "mergeExpressions"
optimization is applied
 2022-12-13 10:52:18 +00:00
Robert Schulze
8fc23262c5
Merge pull request #44125 from ClickHouse/minor-renaming 
Minor file renaming
 2022-12-13 08:35:26 +01:00
Igor Nikonov
db6e1acc14 Remove redundant DISTINCT 
- it doesn't consider prefixes yet
 2022-12-12 22:40:44 +00:00
avogar
739ad23b1f Make better, fix bugs, improve error messages 2022-12-12 22:00:45 +00:00
Dmitry Novik
3d2fccab87
Merge branch 'master' into refector-function-node 2022-12-12 21:36:39 +01:00
Igor Nikonov
69d1b39ef8 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-12 13:57:48 +00:00
Kruglov Pavel
d8d7385e68
Merge pull request #43161 from AlfVII/fix-race-condition-between-inserts-and-dropping-mvs 
Fixed race condition between inserts and dropping MVs
 2022-12-12 13:53:36 +01:00
Robert Schulze
b78652e138
Query result cache 2022-12-12 10:37:10 +00:00
Robert Schulze
8bcafe4544
Renaming PipelineResourcesHolder.h/cpp to QueryPlanResourceHolder.h/cpp 2022-12-10 18:08:10 +00:00
avogar
cd4fa00d2c Merge branch 'master' of github.com:ClickHouse/ClickHouse into refactor-schema-inference 2022-12-09 14:45:10 +00:00
Maksim Kita
3a3c6eb458
Merge pull request #43341 from kitaisreal/analyzer-support-indexes 
Analyzer added indexes support
 2022-12-09 16:05:40 +03:00
Igor Nikonov
55da7a0d33 Polishing 2022-12-09 12:06:40 +00:00
Igor Nikonov
1658276c41 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-09 11:59:14 +00:00
Igor Nikonov
c2b10e8dba Respect stateful functions in the optimization 2022-12-09 11:57:58 +00:00
Han Fei
6740ae9fd7
Merge pull request #43304 from hanfei1991/hanfei/aysnc-insert 
Support dedup for async insert.
 2022-12-09 12:50:48 +01:00
Alfonso Martinez
9654e06ec8 Fixed merge issue 2022-12-09 09:30:20 +01:00
avogar
c224e397ac Check if delimiters are empty, add comments 2022-12-08 20:00:10 +00:00
avogar
1ec5f8451b Merge branch 'master' of github.com:ClickHouse/ClickHouse into csv-custom-delimiter 2022-12-08 19:17:42 +00:00
Kruglov Pavel
de5ffc96e9
Fix style 2022-12-08 19:02:36 +01:00
avogar
556746692b Fix build 2022-12-08 17:20:43 +00:00
Maksim Kita
4d39a7d103 Fixed code review issues 2022-12-08 17:30:32 +01:00
Maksim Kita
99de697ea2 Analyzer added indexes support 2022-12-08 17:30:32 +01:00
Yakov Olkhovskiy
0641066183
Merge branch 'master' into refactoring-ip-types 2022-12-08 11:12:05 -05:00
avogar
7375a7d429 Refactor and improve schema inference for text formats 2022-12-07 21:19:27 +00:00
Igor Nikonov
49694578ba GROUP BY with check if aggr func depends on order 
+ tests
 2022-12-07 20:16:58 +00:00
Dmitry Novik
25ecb75ca8 Merge remote-tracking branch 'origin/master' into refector-function-node 2022-12-07 18:36:50 +00:00
Igor Nikonov
5684453f09 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-07 11:44:43 +00:00
Alfonso Martinez
a6a37f21a4 Merge remote-tracking branch 'upstream/master' into fix-race-condition-between-inserts-and-dropping-mvs 2022-12-07 09:36:47 +01:00
Nikolai Kochetov
0ed82f3cc0
Merge branch 'master' into aggregating-in-order-from-query-plan 2022-12-06 16:36:49 +01:00
Nikolai Kochetov
9de53389ae Review fixes. 2022-12-06 12:34:07 +00:00
Igor Nikonov
611de4d520 Fix + CROSS JOIN tests 2022-12-05 12:31:12 +00:00
Alfonso Martinez
da965c5705 Merge remote-tracking branch 'upstream/master' into fix-race-condition-between-inserts-and-dropping-mvs 2022-12-05 09:30:52 +01:00
Alfonso Martinez
e9e971ffad Fixed style 2022-12-05 09:27:31 +01:00
Igor Nikonov
63df878e88 Fix type 2022-12-02 21:46:27 +00:00
Igor Nikonov
5a3abaad2a Add GROUP BY w/o aggrecation functions analyze + test 2022-12-02 21:21:15 +00:00
Igor Nikonov
e59703abc6 Some tests + setting 2022-12-02 20:53:22 +00:00
Igor Nikonov
07f8ef1030 Something working 2022-12-02 19:27:45 +00:00
Igor Nikonov
47ca855816 save 2022-12-02 19:27:45 +00:00
Dmitry Novik
48e8aeac01 Clean up code 2022-12-02 19:15:26 +00:00
Dmitry Novik
2c70dbc76a Refactor FunctionNode 2022-12-02 19:15:26 +00:00
Vitaly Baranov
e1f7f04752
Referential dependencies for RESTORE (#43834) 
* Rename DDLDependencyVisitor -> DDLLoadingDependencyVisitor.

* Move building a loading graph to TablesLoader.

* Implement referential dependencies for tables and use them
when restoring tables from a backup.

* Remove StorageID::operator < (because of its inconsistency with ==).

* Add new tests.

* Fix test.

* Fix memory leak.

Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2022-12-02 15:05:46 +01:00
Kruglov Pavel
c35b2a6495
Add a limit for string size in RowBinary format (#43842) 2022-12-02 13:57:11 +01:00
Alfonso Martinez
795a8b00fb Added assert and comment for materialized_view->is_dropped 2022-12-02 09:52:10 +01:00
Alfonso Martinez
d6bd3ad691 Merge remote-tracking branch 'upstream/master' into fix-race-condition-between-inserts-and-dropping-mvs 2022-12-01 15:05:01 +01:00
Nikolai Kochetov
5edc9953b0 Fixing tests. 2022-12-01 12:21:21 +00:00
Alexander Tokmakov
431f6551cb
Merge branch 'master' into fix_assertion_in_thread_status 2022-11-30 23:05:15 +03:00
Nikolai Kochetov
7db145bece Fix style. 2022-11-30 19:19:46 +00:00
Vladimir C
53dc70a2d0
Merge pull request #38191 from BigRedEye/grace_hash_join 
Closes https://github.com/ClickHouse/ClickHouse/issues/11596
 2022-11-30 17:01:00 +01:00
Nikolai Kochetov
43b3c10e9a Add a separate optimisation to enable memory bound aggregation. 2022-11-30 14:31:39 +00:00
Anton Popov
fe5fff0347
Merge pull request #43329 from xiedeyantu/support_nested_column 
s3 table function can support select nested column using {column_name}.{subcolumn_name}
 2022-11-29 22:27:19 +01:00
vdimir
21f93e9f4c
Add IJoin::initialize 2022-11-29 17:11:00 +00:00
Nikolai Kochetov
51439e2c19
Merge pull request #43260 from ClickHouse/read-from-mt-in-io-pool 
Read from MergeTree in I/O pool
 2022-11-29 12:09:03 +01:00
Alexander Tokmakov
e45105bf44 detach threads from thread group 2022-11-28 21:31:55 +01:00