thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 20:02:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
112,750 Commits 9,330 Branches 1,783 Tags 2.1 GiB
8710c15c85
Commit Graph

4479 Commits

Author SHA1 Message Date
Nikita Taranov
1d45cce03c support for aggr in order 2023-01-30 17:10:45 +00:00
Nikita Taranov
a2c9aeb7c9 stash 2023-01-30 17:10:45 +00:00
flynn
801d6db486 Arrow support duration type 2023-01-29 10:55:31 +00:00
flynn
9b517cdc76 fix conflict 2023-01-29 08:34:56 +00:00
Alexey Milovidov
9ad87f9917 Fix style and typo 2023-01-29 03:51:42 +01:00
Alexey Milovidov
54bb8b1ceb
Merge branch 'master' into fix-kafka-protobuf 2023-01-29 00:59:40 +03:00
Igor Nikonov
101a0cc90b Merge remote-tracking branch 'origin/master' into igor/remove_redundant_distinct 2023-01-27 19:08:38 +00:00
Nikolai Kochetov
286a58801e Aggregate Projections analysis using query plan [In progress] 2023-01-27 18:38:14 +00:00
Igor Nikonov
300f78df96
Merge pull request #45567 from ClickHouse/enable-remove-redundant-sorting 
Enable query_plan_remove_redundant_sorting optimization by default
 2023-01-27 19:14:36 +01:00
avogar
0a99d421f5 Fix formats parser resetting, test processing bad messages in kafka 2023-01-27 13:22:50 +00:00
Igor Nikonov
4480d8ae60 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_distinct 2023-01-26 22:16:24 +00:00
Igor Nikonov
60a69d4689 Fix distinct after JOIN or UNION 2023-01-26 21:53:08 +00:00
exX512
1793730df8 Change behaviour of cancel in PipelineExecutor 2023-01-26 21:30:36 +00:00
Igor Nikonov
211edc054e Fix check for sum*(Float) 2023-01-26 19:26:53 +00:00
Alexander Tokmakov
067b1f5f13 Merge branch 'master' into exception_message_patterns4 2023-01-26 15:20:58 +01:00
Igor Nikonov
105d72e299 Fix: sum*() with Nullable(Float) 2023-01-26 13:17:06 +00:00
Igor Nikonov
e1302e1a32 Fix: sum*() with Floats 
sum(Float) depends on order, do not remove sorting in such case
 2023-01-25 22:39:28 +00:00
Igor Nikonov
eea076cee0 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_distinct 2023-01-25 20:16:41 +00:00
Igor Nikonov
dd5acf9caf Update 2023-01-25 17:09:26 +00:00
Azat Khuzhin
71286e779e Extend assertion in buildPushingToViewsChain() to respect is_detached 
Follow-up for: #45566 (@tavplubix)
Refs: #43161 (@AlfVII)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-25 15:42:06 +01:00
Alexander Tokmakov
d1baa7300c reformat ParsingException 2023-01-24 23:21:29 +01:00
Alexander Tokmakov
dd57215934 Merge branch 'master' into exception_message_patterns4 2023-01-24 17:03:12 +01:00
Kruglov Pavel
23c12ac8ee
Merge branch 'master' into parquet-fixed-binary 2023-01-24 16:51:05 +01:00
Kruglov Pavel
689fbea759
Merge pull request #45478 from Avogar/fix-arrow-abort 
Fix possible aborts in arrow lib
 2023-01-24 16:37:44 +01:00
Kruglov Pavel
4bd3f0e5ef
Merge pull request #44953 from Avogar/tsv-csv-detect-header 
Detect header in CSV/TSV/CustomSeparated files automatically
 2023-01-24 15:13:52 +01:00
Alexander Tokmakov
6ecae8388e Merge branch 'master' into exception_message_patterns4 2023-01-24 14:42:36 +01:00
Igor Nikonov
7108189b45 Merge remote-tracking branch 'origin/master' into revert-45414-revert-43905-igor/remove_redundant_order_by 2023-01-24 11:46:24 +00:00
Alexey Milovidov
851f6bf910
Merge pull request #45448 from azat/fix-query-hang 
Fix possible (likely distributed) query hung
 2023-01-24 13:00:37 +03:00
Alexander Tokmakov
c6910f39b9 fix 2023-01-24 01:11:58 +01:00
Kruglov Pavel
cd1cd904a7
Merge branch 'master' into tsv-csv-detect-header 2023-01-23 23:49:56 +01:00
Alexander Tokmakov
bb4c8e169f check number of parameters in format string 2023-01-23 23:16:16 +01:00
Alexander Tokmakov
3f6594f4c6 forbid old ctor of Exception 2023-01-23 22:18:05 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Kruglov Pavel
478a552a0a
Merge branch 'master' into tsv-csv-detect-header 2023-01-23 21:47:17 +01:00
Robert Schulze
0ab7ae6c17
Merge pull request #43797 from ClickHouse/query-result-cache 
Query result cache [experimental]
 2023-01-23 19:54:35 +01:00
Maksim Kita
7b48c75e82
Merge pull request #45485 from kitaisreal/remove-unnecessary-get-total-row-count-function-calls 
Remove unnecessary getTotalRowCount function calls
 2023-01-23 21:02:51 +03:00
Kruglov Pavel
af2c1bac6a
Fix typo 2023-01-23 17:13:16 +01:00
Kruglov Pavel
01ddf326ac
Merge branch 'master' into parquet-fixed-binary 2023-01-23 15:31:45 +01:00
Kruglov Pavel
84200be7d2
Better comment 2023-01-23 15:31:07 +01:00
Anton Popov
f181254fb0 fix race in destructor of ParallelParsingInputFormat 2023-01-23 01:18:58 +00:00
Igor Nikonov
f807b4b315 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_distinct 2023-01-22 18:30:30 +00:00
Robert Schulze
340f406553
Merge branch 'master' into query-result-cache 2023-01-22 13:21:36 +01:00
Maksim Kita
47385a19e7 Remove unnecessary getTotalRowCount function calls 2023-01-21 11:27:07 +01:00
Azat Khuzhin
a64f6b5f3e Fix possible (likely distributed) query hung 
Recently I saw the following, the client executed long distributed query
and terminated the connection, and in this case query cancellation will
be done from PullingAsyncPipelineExecutor dtor, but during cancellation
one of nodes sent ECONNRESET, and this leads to an exception from
PullingAsyncPipelineExecutor::cancel(), and this leads to a deadlock
when multiple threads waits each others, because cancel() for
LazyOutputFormat wasn't called.

Here is as relevant portion of logs:

    2023.01.04 08:26:09.236208 [ 37968 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Debug> executeQuery: (from 10.61.13.253:44266, user: default)  TooLongDistributedQueryToPost
    ...
    2023.01.04 08:26:09.262424 [ 37968 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Trace> MergeTreeInOrderSelectProcessor: Reading 1 ranges in order from part 9_330_538_18, approx. 61440 rows starting from 0
    2023.01.04 08:26:09.266399 [ 26788 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Trace> Connection (s4.ch:9000): Connecting. Database: (not specified). User: default
    2023.01.04 08:26:09.266849 [ 26788 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Trace> Connection (s4.ch:9000): Connected to ClickHouse server version 22.10.1.
    2023.01.04 08:26:09.267165 [ 26788 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Debug> Connection (s4.ch:9000): Sent data for 2 scalars, total 2 rows in 3.1587e-05 sec., 62635 rows/sec., 68.00 B (2.03 MiB/sec.), compressed 0.4594594594594595 times to 148.00 B (4.41 MiB/sec.)
    2023.01.04 08:39:13.047170 [ 37968 ] {f2ed6149-146d-4a3d-874a-b0b751c7b567} <Error> PullingAsyncPipelineExecutor: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (10.7.142.115:9000). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

    0. ./.build/./contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x1818234c in /usr/lib/debug/usr/bin/clickhouse.debug
    1. ./.build/./src/Common/Exception.cpp:69: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x1004fbda in /usr/lib/debug/usr/bin/clickhouse.debug
    2. ./.build/./src/Common/NetException.h:12: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x14e352f3 in /usr/lib/debug/usr/bin/clickhouse.debug
    3. ./.build/./src/IO/BufferBase.h:39: DB::Connection::sendCancel() @ 0x15c21e6b in /usr/lib/debug/usr/bin/clickhouse.debug
    4. ./.build/./src/Client/MultiplexedConnections.cpp:0: DB::MultiplexedConnections::sendCancel() @ 0x15c4d5b7 in /usr/lib/debug/usr/bin/clickhouse.debug
    5. ./.build/./src/QueryPipeline/RemoteQueryExecutor.cpp:627: DB::RemoteQueryExecutor::tryCancel(char const*, std::__1::unique_ptr<DB::RemoteQueryExecutorReadContext, std::__1::default_delete<DB::RemoteQueryExecutorReadContext> >*) @ 0x14446c09 in /usr/lib/debug/usr/bin/clickhouse.debug
    6. ./.build/./contrib/libcxx/include/__iterator/wrap_iter.h💯 DB::ExecutingGraph::cancel() @ 0x15d2c0de in /usr/lib/debug/usr/bin/clickhouse.debug
    7. ./.build/./contrib/libcxx/include/__memory/unique_ptr.h:300: DB::PullingAsyncPipelineExecutor::cancel() @ 0x15d32055 in /usr/lib/debug/usr/bin/clickhouse.debug
    8. ./.build/./contrib/libcxx/include/__memory/unique_ptr.h:312: DB::PullingAsyncPipelineExecutor::~PullingAsyncPipelineExecutor() @ 0x15d31f4f in /usr/lib/debug/usr/bin/clickhouse.debug
    9. ./.build/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x15cde919 in /usr/lib/debug/usr/bin/clickhouse.debug
    10. ./.build/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x15cd8554 in /usr/lib/debug/usr/bin/clickhouse.debug
    11. ./.build/./src/Server/TCPHandler.cpp:1904: DB::TCPHandler::run() @ 0x15ce6479 in /usr/lib/debug/usr/bin/clickhouse.debug
    12. ./.build/./contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x18074f07 in /usr/lib/debug/usr/bin/clickhouse.debug
    13. ./.build/./contrib/libcxx/include/__memory/unique_ptr.h:54: Poco::Net::TCPServerDispatcher::run() @ 0x180753ed in /usr/lib/debug/usr/bin/clickhouse.debug
    14. ./.build/./contrib/poco/Foundation/src/ThreadPool.cpp:213: Poco::PooledThread::run() @ 0x181e3807 in /usr/lib/debug/usr/bin/clickhouse.debug
    15. ./.build/./contrib/poco/Foundation/include/Poco/SharedPtr.h:156: Poco::ThreadImpl::runnableEntry(void*) @ 0x181e1483 in /usr/lib/debug/usr/bin/clickhouse.debug
    16. ? @ 0x7ffff7e55fd4 in ?
    17. ? @ 0x7ffff7ed666c in ?
     (version 22.10.1.1)

And here is the state of the threads:

<details>

<summary>system.stack_trace</summary>

```sql
SELECT
    arrayStringConcat(arrayMap(x -> demangle(addressToSymbol(x)), trace), '\n') AS sym
FROM system.stack_trace
WHERE query_id = 'f2ed6149-146d-4a3d-874a-b0b751c7b567'
SETTINGS allow_introspection_functions=1

Row 1:
──────
sym:
pthread_cond_wait
std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)
bool ConcurrentBoundedQueue<DB::Chunk>::emplaceImpl<DB::Chunk>(std::__1::optional<unsigned long>, DB::Chunk&&)
DB::IOutputFormat::work()
DB::ExecutionThreadContext::executeTask()
DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*)

Row 2:
──────
sym:
pthread_cond_wait
Poco::EventImpl::waitImpl()
DB::PipelineExecutor::joinThreads()
DB::PipelineExecutor::executeImpl(unsigned long)
DB::PipelineExecutor::execute(unsigned long)

Row 3:
──────
sym:
pthread_cond_wait
Poco::EventImpl::waitImpl()
DB::PullingAsyncPipelineExecutor::Data::~Data()
DB::PullingAsyncPipelineExecutor::~PullingAsyncPipelineExecutor()
DB::TCPHandler::processOrdinaryQueryWithProcessors()
DB::TCPHandler::runImpl()
DB::TCPHandler::run()
Poco::Net::TCPServerConnection::start()
Poco::Net::TCPServerDispatcher::run()
Poco::PooledThread::run()
Poco::ThreadImpl::runnableEntry(void*)
```

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-21 08:05:56 +01:00
Azat Khuzhin
e2fcf0f072 Catch exception on query cancellation 
Since we still want to join the thread, yes it will be done in dtor, but
this looks better.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-21 08:05:56 +01:00
Azat Khuzhin
0566f72d36 Cleanup PullingAsyncPipelineExecutor::cancel() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-21 08:05:56 +01:00
avogar
86336940f8 Better comment 2023-01-20 16:41:59 +00:00
avogar
4432ee9927 Fix aborts in arrow lib 2023-01-20 16:40:33 +00:00
avogar
550a703fbc Make a bit better 2023-01-20 14:58:39 +00:00
Kruglov Pavel
28ddcc2432
Merge branch 'master' into tsv-csv-detect-header 2023-01-20 15:08:38 +01:00
Nikolai Kochetov
134ac9b2dd Aggregate Projections analysis using query plan [In progress] 2023-01-19 19:53:22 +00:00
avogar
c34c0aa22e Fix comments 2023-01-19 16:03:46 +00:00
Kruglov Pavel
9820beae68
Apply suggestions from code review 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-01-19 16:11:13 +01:00
Igor Nikonov
d0ce804bfc Fix: dynamic_cast -> typeid_cast for SortingStep 2023-01-19 13:40:21 +00:00
Igor Nikonov
df3776d24b Make test stable 
+ disable debug logging
 2023-01-19 11:43:40 +00:00
Ilya Yatsishin
00962b7ad5
Merge pull request #45424 from Avogar/fix-json-import-nested 2023-01-19 10:31:40 +01:00
Igor Nikonov
57d2fd300a Fix: correct update of data stream sorting properties after removing 
sorting
 2023-01-19 00:11:58 +00:00
avogar
a8f20363f4 Fix JSON/BSONEachRow parsing with HTTP 2023-01-18 22:49:03 +00:00
Igor Nikonov
1866f990de
Revert "Revert "Remove redundant sorting"" 2023-01-18 20:12:34 +01:00
Igor Nikonov
7ed8fec94f
Revert "Remove redundant sorting" 2023-01-18 18:38:25 +01:00
flynn
4dcd3cc4f6 fix style 2023-01-18 08:46:06 +00:00
flynn
a8e1363151 implement storage iceberg 2023-01-18 03:43:56 +00:00
Robert Schulze
4f90824347
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-17 22:49:53 +00:00
Igor Nikonov
0db9bf38a2
Merge branch 'master' into igor/remove_redundant_order_by 2023-01-17 22:26:24 +01:00
Nikolai Kochetov
9c6c6d9844 Aggregate Projections analysis using query plan [In progress] 2023-01-17 20:12:34 +00:00
Alexander Tokmakov
8b13b85ea0
Merge pull request #44543 from ClickHouse/text_log_add_pattern 
Add a column with a message pattern to text_log
 2023-01-17 20:19:32 +03:00
Igor Nikonov
0cfa08df7a Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-17 16:28:17 +00:00
Igor Nikonov
9855504403 Rename source file according to implementation 2023-01-17 16:24:51 +00:00
Kruglov Pavel
96bb99f864
Merge branch 'master' into tsv-csv-detect-header 2023-01-17 15:33:02 +01:00
Igor Nikonov
6328e02f22 Fix: update input/output stream properties 
After removing sorting step we need to update sorting properties of
input/ouput streams
 2023-01-17 13:39:18 +00:00
Sema Checherinda
35431e91e3
Merge pull request #45276 from ucasfl/avro-fix 
Fix some avro reading bugs
 2023-01-17 12:48:44 +01:00
avogar
5bf4704e7a Support FixedSizeBinary type in Parquet/Arrow 2023-01-16 21:01:31 +00:00
avogar
1c0941d72a Add docs and examples 2023-01-16 16:46:41 +00:00
avogar
3ea80b0f54 Merge branch 'master' of github.com:ClickHouse/ClickHouse into tsv-csv-detect-header 2023-01-16 15:14:25 +00:00
Igor Nikonov
a34991cb65 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-16 12:14:02 +00:00
Robert Schulze
099e30ef2a
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-16 08:04:49 +00:00
Robert Schulze
27fe7ebd93
Cosmetics 2023-01-15 16:12:48 +00:00
flynn
29eb30b49f Fix some reading avro format bugs 
fix
 2023-01-14 18:05:26 +00:00
Alexander Tokmakov
2d7773fccc Merge branch 'master' into text_log_add_pattern 2023-01-13 20:33:46 +01:00
avogar
e2470dd670 Fix tests 2023-01-13 17:03:53 +00:00
Robert Schulze
d7d3f61c73
Cleanup SourceFromChunks a bit 2023-01-13 10:57:31 +00:00
Robert Schulze
15e11741cb
Cosmetics 2023-01-13 00:00:23 +00:00
Robert Schulze
1b53307375
Refcount cache entries to improve lookup performance 2023-01-12 23:38:39 +00:00
Robert Schulze
0c528fca29
Introduce SourceFromChunks to avoid unnecessary partial result chunk concatenation 2023-01-12 23:01:00 +00:00
avogar
b461935374 Better 2023-01-12 13:11:04 +00:00
Kruglov Pavel
05a11ff4a4
Merge branch 'master' into tsv-csv-detect-header 2023-01-12 12:35:18 +01:00
Maksim Kita
a140d6c5b1 Fixed code review issues 2023-01-12 12:07:58 +01:00
Maksim Kita
47f4159909 Analyzer support distributed queries processing 2023-01-12 12:07:58 +01:00
avogar
e4d774d906 Better naming 2023-01-11 22:57:14 +00:00
avogar
26cd56d113 Fix tests, make better 2023-01-11 22:52:15 +00:00
Alexander Gololobov
6adf1c025f
Merge pull request #45165 from ClickHouse/more_logging_for_replicated_ttl 
More logging to facilitate debugging of flaky test_ttl_replicated
 2023-01-11 23:24:48 +01:00
avogar
3b45863d15 Make better implementation, fix tests 2023-01-11 17:12:56 +00:00
Alexander Gololobov
659fa96365 More logging to facilitate debugging 2023-01-11 13:06:38 +01:00
Nikolai Kochetov
5e7a6ac619
Merge pull request #45122 from ClickHouse/revert-45121-revert-44653-custom-reading-for-mutation 
Revert "Revert "Custom reading for mutation""
 2023-01-11 12:37:32 +01:00
avogar
6312b75f44 Fix style 2023-01-10 16:28:52 +00:00
avogar
615fe4cecb Fix tests 2023-01-10 16:27:23 +00:00
Maksim Kita
fbba28b31e Analyzer aggregation without column fix 2023-01-10 16:49:55 +01:00
Nikolai Kochetov
4673b3fe1d
Revert "Revert "Custom reading for mutation"" 2023-01-10 16:31:01 +01:00
Alexander Tokmakov
c8ec130be4
Revert "Custom reading for mutation" 2023-01-10 17:51:30 +03:00
Nikolai Kochetov
11418963c0
Merge pull request #44653 from ClickHouse/custom-reading-for-mutation 
Custom reading for mutation
 2023-01-10 12:16:24 +01:00
Robert Schulze
923fa2c15a
Fix review comments, pt. II 2023-01-10 10:21:08 +00:00
Robert Schulze
0c3b034887
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-10 09:02:41 +00:00
Alexey Milovidov
1229a20fb3
Merge pull request #45047 from ClickHouse/fix-buffer-overflow 
Fix buffer overflow in parser
 2023-01-10 05:06:03 +03:00
Yakov Olkhovskiy
4f32f3b8cb
Merge pull request #44484 from bigo-sg/arrow_struct_field 
Optimization for reading struct fields in parquet/orc files
 2023-01-09 15:36:26 -05:00
Nikolai Kochetov
3e430b1b6e Add comments. Remove unused flag from Context. 2023-01-09 19:52:11 +00:00
Kseniia Sumarokova
119501f1d9
Merge pull request #44698 from Avogar/parquet-bool 
Support Bool type in Arrow/Parquet/ORC
 2023-01-09 12:56:28 +01:00
lgbo-ustc
b639bcabc0 some fixes 2023-01-09 18:13:00 +08:00
lgbo-ustc
f127b3a60a update ArrowFieldIndexUtil 2023-01-09 18:13:00 +08:00
lgbo-ustc
a3bdfddc9d support nested table 2023-01-09 18:13:00 +08:00
lgbo-ustc
4f3f781b85 fixed test case 2023-01-09 18:13:00 +08:00
lgbo-ustc
755f03db4e fixed 2023-01-09 18:13:00 +08:00
lgbo-ustc
f6850d96cb fixed missing columns 2023-01-09 18:13:00 +08:00
lgbo-ustc
4cf6beee27 fixed 2023-01-09 18:13:00 +08:00
lgbo-ustc
81e2832133 fixed 2023-01-09 18:13:00 +08:00
lgbo-ustc
77cea49cec fixed including header failure 2023-01-09 18:13:00 +08:00
lgbo-ustc
8f8f6f966b Optimization for reading struct fields in parquet/orc files 2023-01-09 18:13:00 +08:00
Igor Nikonov
7298b39a18 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-09 10:08:48 +00:00
Igor Nikonov
cec7dc5a5a Renaming according to setting 2023-01-09 10:06:47 +00:00
Igor Nikonov
e29cd5d640 Rename settings to remove_redundant_sorting 2023-01-09 09:50:00 +00:00
Alexey Milovidov
eea5a2c521
Merge branch 'master' into fix-buffer-overflow 2023-01-09 09:15:29 +03:00
Alexey Milovidov
0d39d26a34 Don't fix parallel formatting 2023-01-09 06:15:20 +01:00
Alexey Milovidov
d331f0ce82 Fix buffer overflow in parser 2023-01-09 03:31:12 +01:00
Alexey Milovidov
0d8d6eca67 Non significant changes 2023-01-09 01:51:05 +01:00
Alexey Milovidov
797dfdd42e Non significant changes 2023-01-09 01:50:19 +01:00
Alexey Milovidov
3430df2f8c Non significant changes 2023-01-09 01:48:57 +01:00
avogar
ee72799121 Fix tests, make better 2023-01-06 20:46:43 +00:00
Robert Schulze
45dbcf88e5
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-06 20:42:48 +00:00
Igor Nikonov
2187bdd4cc Disable diagnostics 
+ cleanup
+ disable optimization in sort performance test since it removes sorting
  at all
 2023-01-06 17:00:05 +00:00
Igor Nikonov
d2d5a09802 Fix typo 2023-01-06 15:01:31 +00:00
Igor Nikonov
48b58a9fed Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-06 13:55:26 +00:00
Igor Nikonov
d886b56338 Fix review comments 2023-01-06 13:54:28 +00:00
avogar
7fcdb08ec6 Detect header in CSV/TSV/CustomSeparated files automatically 2023-01-05 22:57:25 +00:00
Robert Schulze
4fcead15c3
Fix handling of exceptions during query runtime 2023-01-04 22:33:04 +00:00
Yakov Olkhovskiy
7a5a36cbed
Merge branch 'master' into refactoring-ip-types 2023-01-04 11:11:06 -05:00
Kruglov Pavel
b9bdf62bf3
Merge branch 'master' into parquet-bool 2023-01-04 14:49:41 +01:00
Kruglov Pavel
59263f3ae1
Merge pull request #44501 from Avogar/validate-types 
Validate data types according to settings.
 2023-01-04 14:48:09 +01:00
Kruglov Pavel
90ae405033
Merge pull request #44876 from Avogar/fix-perf-tests 
Revert some changes from #42777 to fix performance tests
 2023-01-04 14:27:17 +01:00
Kruglov Pavel
0c7d39ac7f
Merge pull request #44832 from ucasfl/row-number 
Fix output_format_pretty_row_numbers does not preserve the counter across the blocks
 2023-01-04 14:15:47 +01:00
Igor Nikonov
1edfdfda4d Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-04 12:34:40 +00:00
Kruglov Pavel
4e261ab230
Fix JSONCompactEachRow 2023-01-03 21:16:38 +01:00
Kruglov Pavel
314d95fd71
Fix special build 2023-01-03 20:34:30 +01:00
avogar
28eb2dbd4c Revert some changes from #42777 2023-01-03 18:53:03 +00:00
Alexey Milovidov
e855d3519a
Merge branch 'master' into refactoring-ip-types 2023-01-02 21:58:53 +03:00
Kruglov Pavel
4a7c399076
Merge branch 'master' into parquet-bool 2023-01-02 16:33:42 +01:00
Kruglov Pavel
0a43976977
Merge branch 'master' into validate-types 2023-01-02 16:10:14 +01:00
Kruglov Pavel
1c2dc05d6e
Merge pull request #44446 from Avogar/arrow-nullables 
Respect setting settings.schema_inference_make_columns_nullable in Parquet/ORC/Arrow formats
 2023-01-02 16:05:57 +01:00
Kruglov Pavel
966f57ef68
Merge pull request #42777 from Avogar/improve-streaming-engines 
Refactor and Improve streaming engines Kafka/RabbitMQ/NATS and data formats
 2023-01-02 15:59:06 +01:00
flynn
7780fae9db fix 2023-01-02 12:38:51 +00:00
flynn
7b487dd923 fix 2023-01-02 12:37:43 +00:00
flynn
3a1dd045dd Fix output_format_pretty_row_numbers does not preserve the counter across the blocks 2023-01-02 09:27:37 +00:00
Robert Schulze
e9e04166d9
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-02 08:20:27 +00:00
Kruglov Pavel
8479615c48
Merge pull request #44684 from Avogar/avro-bool 
Input/ouptut avro bool type as ClickHouse bool type
 2022-12-30 17:56:36 +01:00
Kruglov Pavel
4982d132fb
Merge branch 'master' into validate-types 2022-12-30 17:52:13 +01:00
Nikolay Degterinsky
dfe93b5d82
Merge pull request #42284 from Algunenano/perf_experiment 
Performance experiment
 2022-12-30 03:14:22 +01:00
Kruglov Pavel
894726bd8f
Merge branch 'master' into improve-streaming-engines 2022-12-29 22:59:45 +01:00
Nikolai Kochetov
3c02e208c8 Fix more tests. 2022-12-29 17:52:31 +00:00
Nikita Mikhaylov
ffaff7167b
Fix flaky tests (#44690) 2022-12-29 14:32:10 +01:00
Alexey Milovidov
875797ee63
Merge pull request #44685 from ClickHouse/remove-old-code-4 
Remove old code
 2022-12-29 15:43:59 +03:00
Alexey Milovidov
9378194320
Merge pull request #44688 from ClickHouse/improve-odbc-test 
Improve odbc test
 2022-12-29 15:40:47 +03:00
avogar
a0db1dd1ea Support Bool type in Arrow/Parquet/ORC 2022-12-28 22:58:28 +00:00
Alexey Milovidov
e524d4bcb1
Merge pull request #44673 from ClickHouse/logical-error-with-join-subquery-union-and-totals 
Fix LOGICAL_ERROR for JOIN with subquery, UNION and totals.
 2022-12-29 01:56:15 +03:00
Raúl Marín
5de11979ce
Unify query elapsed time measurements (#43455) 
* Unify query elapsed time reporting

* add-test: Make shell tests executable

* Add some tests around query elapsed time

* Style and ubsan
 2022-12-28 21:01:41 +01:00
Raúl Marín
e915ce1e95 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-12-28 20:15:43 +01:00
Raúl Marín
f6428964cc Better and common error handling 2022-12-28 20:15:27 +01:00
Alexey Milovidov
c3eff78f20 Don't be kind with third-party code 2022-12-28 20:07:26 +01:00
Alexey Milovidov
cf594a659d
Update MongoDBSource.cpp 2022-12-28 21:26:42 +03:00
Alexey Milovidov
33bcd07be5 Remove old code 2022-12-28 19:02:06 +01:00
avogar
f1191bbbc6 Input/ouptut avro bool type as ClickHouse bool type 2022-12-28 17:38:58 +00:00
Nikolai Kochetov
2d84ef2891 Fix tests and update IntersectOrExceptStep. 2022-12-28 14:44:45 +00:00
Nikolai Kochetov
21e93cd41d Fix LOGICAL_ERROR for JOIN with subquery, UNION and totals. 2022-12-28 13:00:32 +00:00
avogar
411f98306a Merge branch 'master' of github.com:ClickHouse/ClickHouse into validate-types 2022-12-27 19:24:15 +00:00
Kruglov Pavel
6dea7336f7
Merge pull request #44405 from Avogar/fix-parquet-orc 
Fix reading columns that are not presented in input data in Parquet/ORC formats
 2022-12-27 16:58:35 +01:00
Kruglov Pavel
e981b2e78b
Merge pull request #43813 from Avogar/fix-sum-if 
Better handling of NULL in aggregate combinators, fix possible segfault/logical error
 2022-12-27 16:50:48 +01:00
Alexey Milovidov
1b21cc018e
Merge pull request #44342 from ClickHouse/add-test-29883 
Add a test for #29883
 2022-12-27 14:55:14 +03:00
Raúl Marín
fc1fa82a39
Merge branch 'master' into perf_experiment 2022-12-27 10:51:58 +01:00
Alexander Tokmakov
ca989e9212 less runtime format strings 2022-12-23 19:50:34 +01:00
Nikolai Kochetov
5030c92373
Merge branch 'master' into fix-wrong-condition-for-async-reading-from-mt 2022-12-23 14:18:08 +01:00
Nikolai Kochetov
4f84becfa0 Fix wrong condition for enabling asyn creading from MergeTree. 2022-12-23 13:00:23 +00:00
Igor Nikonov
e2e898b707 Remove incorrect assert 2022-12-23 12:12:25 +00:00
Vladimir C
7482ea54ab
Merge pull request #43972 from ClickHouse/vdimir/tmp-data-in-fs-cache-2 2022-12-23 11:59:27 +01:00
Dmitry Novik
cff882d506 Merge remote-tracking branch 'origin/master' into refector-function-node 2022-12-22 21:34:29 +00:00
Igor Nikonov
95c3703bbf Remove incorrect assert 
ExpressionStep is not always under SortingStep
It can be ReadFromRemote in distributed query
 2022-12-22 19:32:41 +00:00
Kruglov Pavel
6a017a6586
Merge pull request #43379 from Avogar/better-capn-proto 
Add small improvements in CapnProto format
 2022-12-22 14:50:10 +01:00
vdimir
182b34c11e
Fixes 2022-12-22 10:22:57 +00:00
Yakov Olkhovskiy
a8cb29da4b
Merge branch 'master' into refactoring-ip-types 2022-12-21 23:56:24 -05:00
avogar
4ab3e90382 Validate types in table function arguments/CAST function arguments/JSONAsObject schema inference 2022-12-21 21:21:30 +00:00
Igor Nikonov
c7650850fe Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-21 15:56:08 +00:00
Igor Nikonov
919fa9b84e If aggregation step is in order, - do not remove underneath sorting step 2022-12-21 12:55:42 +00:00
Igor Nikonov
fdb277245c Remove current sorting only if next is full sorting 2022-12-21 12:21:35 +00:00
Kruglov Pavel
5e01a3d74e
Merge branch 'master' into improve-streaming-engines 2022-12-21 10:51:50 +01:00
Kruglov Pavel
09ab5832b1
Merge pull request #44382 from Avogar/fix-bson-object-id 
Fix reading ObjectId in BSON schema inference
 2022-12-21 10:48:50 +01:00
Igor Nikonov
1624647f07 Fix: distributed query with cluster() table function 2022-12-20 23:27:01 +00:00
Alexey Milovidov
a6f50edde3
Merge branch 'master' into add-test-29883 2022-12-21 01:45:18 +03:00
Igor Nikonov
5cd4767838 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-20 17:56:59 +00:00
avogar
c49638e3a9 Respect setting settings.schema_inference_make_columns_nullable in Parquet/ORC/Arrow formats 2022-12-20 17:46:42 +00:00
Dmitry Novik
4793412887
Merge branch 'master' into refector-function-node 2022-12-20 18:26:19 +01:00
Kruglov Pavel
643a35bed1
Merge pull request #44019 from Avogar/refactor-schema-inference 
Refactor and improve schema inference for text formats
 2022-12-20 17:29:03 +01:00
Kruglov Pavel
c0b17ca0af
Merge branch 'master' into fix-bson-object-id 2022-12-20 17:18:10 +01:00
Kruglov Pavel
a0ee1c1227
Merge branch 'master' into fix-sum-if 2022-12-20 15:20:23 +01:00
Kruglov Pavel
fe28faa32d
Fix style 2022-12-20 14:49:39 +01:00
Kruglov Pavel
3f1e40aacd
Merge branch 'master' into fix-orc 2022-12-20 13:32:46 +01:00
Igor Nikonov
15bb4cb65c Fix distributed query check 2022-12-20 10:21:32 +00:00
Raúl Marín
45d27f461b
Merge branch 'master' into perf_experiment 2022-12-20 09:07:48 +00:00
Igor Nikonov
2164b18485 Disable the optimization for distributed queries 
+ based on ReadFromRemote
- improve by disabling optimization only between Union and ReadFromRemote
 2022-12-19 23:24:35 +00:00
avogar
e262e375dc Fix reading columns that are not presented in input data in Parquet/ORC formats 2022-12-19 20:30:54 +00:00
avogar
0c406adce2 Fix reading Map type in ORC format 2022-12-19 18:23:07 +00:00
Igor Nikonov
1e53b3be38 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-19 16:44:22 +00:00
Nikolai Kochetov
860afd1832 Add a very bad fix. 2022-12-19 16:33:10 +00:00
avogar
21cdf6e6ae Fix reading ObjectId in BSON schema inference 2022-12-19 14:13:42 +00:00
Robert Schulze
b93c56323c
Merge remote-tracking branch 'origin/master' into query-result-cache 2022-12-19 09:18:36 +00:00
Alexey Milovidov
3e8dcdc155 Add earlier check 2022-12-18 08:01:43 +01:00
avogar
22e70fa71e Try fix tests 2022-12-16 18:09:42 +00:00
Dmitry Novik
875a24a650 Merge remote-tracking branch 'origin/master' into refector-function-node 2022-12-16 16:07:30 +00:00
avogar
291e51c533 Merge branch 'better-capn-proto' of github.com:Avogar/ClickHouse into better-capn-proto 2022-12-16 14:43:06 +00:00
avogar
4a51bdce86 Fix comments 2022-12-16 13:58:54 +00:00
Kruglov Pavel
3fad5c7f1f
Merge branch 'master' into refactor-schema-inference 2022-12-16 14:24:51 +01:00
avogar
34274be8df Fix use after free 2022-12-16 13:20:36 +00:00
Igor Nikonov
319158f32f Merge remote-tracking branch 'origin/master' into igor/remove_redundant_distinct 2022-12-16 11:46:27 +00:00
Robert Schulze
4db33f16e3
Make DoS attack on cache less likely 2022-12-15 21:42:51 +00:00
Igor Nikonov
ea0bd38ab1 Avoid to remove sorting expression for now 2022-12-15 21:11:02 +00:00
Robert Schulze
bcc54ab9b4
Add cfg for min query duration for caching 2022-12-15 20:10:37 +00:00
avogar
cfcb444699 Merge branch 'master' of github.com:ClickHouse/ClickHouse into better-capn-proto 2022-12-15 20:04:43 +00:00
Kruglov Pavel
25f199dd89
Merge pull request #43332 from Avogar/csv-custom-delimiter 
Improve reading CSV field in CustomSeparated/Template format
 2022-12-15 21:03:29 +01:00
Igor Nikonov
629f317d7d Fix for full sorting merge join 2022-12-15 20:01:50 +00:00
avogar
f26b5b57ad Better comment 2022-12-15 19:52:57 +00:00
avogar
5b4ea53fed Fix fields order 2022-12-15 19:52:25 +00:00
avogar
755b08a49e Fix comments 2022-12-15 19:47:10 +00:00
Kruglov Pavel
c5b2e4cc23
Merge branch 'master' into improve-streaming-engines 2022-12-15 18:44:35 +01:00
Robert Schulze
537c64aad2
Merge remote-tracking branch 'origin/master' into query-result-cache 
Conflicts:
- tests/queries/0_stateless/02117_show_create_table_system.reference
 2022-12-15 12:17:58 +00:00
Robert Schulze
f1e367179e
Make maximum record count per item configurable 2022-12-15 11:16:44 +00:00
Robert Schulze
e52dfaf45c
Make number of cached items configurable 2022-12-15 10:08:28 +00:00
Igor Nikonov
bde3e43d3b Merge remote-tracking branch 'origin/master' into igor/remove_redundant_distinct 2022-12-15 00:36:06 +00:00
Igor Nikonov
6dd2849ae6 Fix for distributed queries 
+ 01952_optimize_distributed_group_by_sharding_key
 2022-12-14 21:54:20 +00:00
avogar
a94a0d9c85 Fix tests, fix bugs 2022-12-14 21:17:00 +00:00
Igor Nikonov
b5de352861 Move checks to functions 2022-12-14 17:51:02 +00:00
Igor Nikonov
6ca0773516 Fix for distributed query 
- do not apply optimization in case of UnionStep on top of Sorting

01104_distributed_one_test.sql
 2022-12-14 16:10:52 +00:00
Dmitry Novik
1df038e39c Merge remote-tracking branch 'origin/master' into refector-function-node 2022-12-14 14:55:30 +00:00
Igor Nikonov
03f7bd1192 Fix for distributed queries 2022-12-14 13:54:42 +00:00
Igor Nikonov
e25053dec0 Query plan visitor with debug logs 2022-12-13 22:46:39 +00:00
Igor Nikonov
1cbdce8eea Fix window function case 2022-12-13 16:21:14 +00:00
Nikolay Degterinsky
9b6d31b95d
Merge branch 'master' into perf_experiment 2022-12-13 17:15:07 +01:00
Igor Nikonov
1bed5064e7 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-13 10:55:50 +00:00
Igor Nikonov
6f5fc6645b Fix: remove ORDER BY before "mergeExpressions" optimization 
When removing ORDER BY we remove corresponding expression,
so to remove only ORDER BY expression, do it before "mergeExpressions"
optimization is applied
 2022-12-13 10:52:18 +00:00
Robert Schulze
8fc23262c5
Merge pull request #44125 from ClickHouse/minor-renaming 
Minor file renaming
 2022-12-13 08:35:26 +01:00
Igor Nikonov
db6e1acc14 Remove redundant DISTINCT 
- it doesn't consider prefixes yet
 2022-12-12 22:40:44 +00:00
avogar
739ad23b1f Make better, fix bugs, improve error messages 2022-12-12 22:00:45 +00:00
Dmitry Novik
3d2fccab87
Merge branch 'master' into refector-function-node 2022-12-12 21:36:39 +01:00
Igor Nikonov
69d1b39ef8 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-12 13:57:48 +00:00
Kruglov Pavel
d8d7385e68
Merge pull request #43161 from AlfVII/fix-race-condition-between-inserts-and-dropping-mvs 
Fixed race condition between inserts and dropping MVs
 2022-12-12 13:53:36 +01:00
Robert Schulze
b78652e138
Query result cache 2022-12-12 10:37:10 +00:00
Robert Schulze
8bcafe4544
Renaming PipelineResourcesHolder.h/cpp to QueryPlanResourceHolder.h/cpp 2022-12-10 18:08:10 +00:00
avogar
cd4fa00d2c Merge branch 'master' of github.com:ClickHouse/ClickHouse into refactor-schema-inference 2022-12-09 14:45:10 +00:00
Maksim Kita
3a3c6eb458
Merge pull request #43341 from kitaisreal/analyzer-support-indexes 
Analyzer added indexes support
 2022-12-09 16:05:40 +03:00
Igor Nikonov
55da7a0d33 Polishing 2022-12-09 12:06:40 +00:00
Igor Nikonov
1658276c41 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-09 11:59:14 +00:00
Igor Nikonov
c2b10e8dba Respect stateful functions in the optimization 2022-12-09 11:57:58 +00:00
Han Fei
6740ae9fd7
Merge pull request #43304 from hanfei1991/hanfei/aysnc-insert 
Support dedup for async insert.
 2022-12-09 12:50:48 +01:00
Alfonso Martinez
9654e06ec8 Fixed merge issue 2022-12-09 09:30:20 +01:00
avogar
c224e397ac Check if delimiters are empty, add comments 2022-12-08 20:00:10 +00:00
avogar
1ec5f8451b Merge branch 'master' of github.com:ClickHouse/ClickHouse into csv-custom-delimiter 2022-12-08 19:17:42 +00:00
Kruglov Pavel
de5ffc96e9
Fix style 2022-12-08 19:02:36 +01:00
avogar
556746692b Fix build 2022-12-08 17:20:43 +00:00
Maksim Kita
4d39a7d103 Fixed code review issues 2022-12-08 17:30:32 +01:00
Maksim Kita
99de697ea2 Analyzer added indexes support 2022-12-08 17:30:32 +01:00
Yakov Olkhovskiy
0641066183
Merge branch 'master' into refactoring-ip-types 2022-12-08 11:12:05 -05:00
avogar
7375a7d429 Refactor and improve schema inference for text formats 2022-12-07 21:19:27 +00:00
Igor Nikonov
49694578ba GROUP BY with check if aggr func depends on order 
+ tests
 2022-12-07 20:16:58 +00:00
Dmitry Novik
25ecb75ca8 Merge remote-tracking branch 'origin/master' into refector-function-node 2022-12-07 18:36:50 +00:00
Igor Nikonov
5684453f09 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2022-12-07 11:44:43 +00:00
Alfonso Martinez
a6a37f21a4 Merge remote-tracking branch 'upstream/master' into fix-race-condition-between-inserts-and-dropping-mvs 2022-12-07 09:36:47 +01:00
Nikolai Kochetov
0ed82f3cc0
Merge branch 'master' into aggregating-in-order-from-query-plan 2022-12-06 16:36:49 +01:00
Nikolai Kochetov
9de53389ae Review fixes. 2022-12-06 12:34:07 +00:00
Igor Nikonov
611de4d520 Fix + CROSS JOIN tests 2022-12-05 12:31:12 +00:00
Alfonso Martinez
da965c5705 Merge remote-tracking branch 'upstream/master' into fix-race-condition-between-inserts-and-dropping-mvs 2022-12-05 09:30:52 +01:00
Alfonso Martinez
e9e971ffad Fixed style 2022-12-05 09:27:31 +01:00
Igor Nikonov
63df878e88 Fix type 2022-12-02 21:46:27 +00:00
Igor Nikonov
5a3abaad2a Add GROUP BY w/o aggrecation functions analyze + test 2022-12-02 21:21:15 +00:00
Igor Nikonov
e59703abc6 Some tests + setting 2022-12-02 20:53:22 +00:00
Igor Nikonov
07f8ef1030 Something working 2022-12-02 19:27:45 +00:00
Igor Nikonov
47ca855816 save 2022-12-02 19:27:45 +00:00
Dmitry Novik
48e8aeac01 Clean up code 2022-12-02 19:15:26 +00:00
Dmitry Novik
2c70dbc76a Refactor FunctionNode 2022-12-02 19:15:26 +00:00
Vitaly Baranov
e1f7f04752
Referential dependencies for RESTORE (#43834) 
* Rename DDLDependencyVisitor -> DDLLoadingDependencyVisitor.

* Move building a loading graph to TablesLoader.

* Implement referential dependencies for tables and use them
when restoring tables from a backup.

* Remove StorageID::operator < (because of its inconsistency with ==).

* Add new tests.

* Fix test.

* Fix memory leak.

Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2022-12-02 15:05:46 +01:00
Kruglov Pavel
c35b2a6495
Add a limit for string size in RowBinary format (#43842) 2022-12-02 13:57:11 +01:00
Alfonso Martinez
795a8b00fb Added assert and comment for materialized_view->is_dropped 2022-12-02 09:52:10 +01:00
Alfonso Martinez
d6bd3ad691 Merge remote-tracking branch 'upstream/master' into fix-race-condition-between-inserts-and-dropping-mvs 2022-12-01 15:05:01 +01:00
Nikolai Kochetov
5edc9953b0 Fixing tests. 2022-12-01 12:21:21 +00:00
Alexander Tokmakov
431f6551cb
Merge branch 'master' into fix_assertion_in_thread_status 2022-11-30 23:05:15 +03:00
Nikolai Kochetov
7db145bece Fix style. 2022-11-30 19:19:46 +00:00
Vladimir C
53dc70a2d0
Merge pull request #38191 from BigRedEye/grace_hash_join 
Closes https://github.com/ClickHouse/ClickHouse/issues/11596
 2022-11-30 17:01:00 +01:00
Nikolai Kochetov
43b3c10e9a Add a separate optimisation to enable memory bound aggregation. 2022-11-30 14:31:39 +00:00
Anton Popov
fe5fff0347
Merge pull request #43329 from xiedeyantu/support_nested_column 
s3 table function can support select nested column using {column_name}.{subcolumn_name}
 2022-11-29 22:27:19 +01:00
vdimir
21f93e9f4c
Add IJoin::initialize 2022-11-29 17:11:00 +00:00
Nikolai Kochetov
51439e2c19
Merge pull request #43260 from ClickHouse/read-from-mt-in-io-pool 
Read from MergeTree in I/O pool
 2022-11-29 12:09:03 +01:00
Alexander Tokmakov
e45105bf44 detach threads from thread group 2022-11-28 21:31:55 +01:00
Nikolai Kochetov
500d54847e Fixing style. 2022-11-28 19:11:05 +00:00
Han Fei
f23ee45d84
Merge branch 'master' into hanfei/aysnc-insert 2022-11-28 16:58:45 +01:00
Nikolai Kochetov
6d0646ed8f
Merge branch 'master' into aggregating-in-order-from-query-plan 2022-11-28 16:53:29 +01:00
Nikolai Kochetov
1dfa188c7a Add order info for aggregating step in plan. Added test. 2022-11-28 15:15:36 +00:00
Yakov Olkhovskiy
770b520ded
Merge branch 'master' into refactoring-ip-types 2022-11-28 08:50:19 -05:00
Kruglov Pavel
dd7ac8bb96
Update src/Processors/Formats/Impl/CapnProtoRowOutputFormat.cpp 
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
 2022-11-28 14:17:52 +01:00
Nikita Taranov
8ed5cfc265
Memory bound merging for distributed aggregation in order (#40879) 
* impl

* fix style

* make executeQueryWithParallelReplicas similar to executeQuery

* impl for parallel replicas

* cleaner code for remote sorting properties

* update test

* fix

* handle when nodes of old versions participate

* small fixes

* temporary enable for testing

* fix after merge

* Revert "temporary enable for testing"

This reverts commit cce7f8884c.

* review fixes

* add bc test

* Update src/Core/Settings.h
 2022-11-28 00:41:31 +01:00
Nikolai Kochetov
4632e7c644 Add max_streams_for_merge_tree_reading setting. 2022-11-25 17:14:22 +00:00
Kruglov Pavel
2818ecf7f0
Merge pull request #43297 from arthurpassos/fix_arrow_list_column_parsing 
Flatten list type arrow chunks on parsing
 2022-11-25 18:13:27 +01:00
Han Fei
ee5be80d9f Merge branch 'master' into hanfei/aysnc-insert 2022-11-25 17:27:56 +01:00
Anton Popov
c811f34a41
Simplify AsynchronousInsertQueue and fix race (#43233) 2022-11-25 15:02:22 +01:00
Nikolai Kochetov
660c942258 Merge branch 'master' into read-from-mt-in-io-pool 2022-11-25 13:56:48 +00:00
Igor Nikonov
236e7e3989 Small fixes 2022-11-25 12:04:12 +00:00
Igor Nikonov
20e67b7140 Merge remote-tracking branch 'origin/master' into HEAD 2022-11-24 13:10:37 +00:00
Raúl Marín
4d3217cb18
Merge branch 'master' into perf_experiment 2022-11-24 10:32:12 +01:00
Han Fei
28564df6f1
Merge branch 'master' into hanfei/aysnc-insert 2022-11-24 09:12:43 +01:00
Nikolai Kochetov
62debf42b4 Aggregating in order [in progress]. 2022-11-23 18:38:12 +00:00
xiedeyantu
304b6ebf3a s3 table function can support select nested column using {column_name}.{subcolumn_name} 2022-11-23 23:36:12 +08:00
Alexey Milovidov
c23b8c69db
Merge pull request #42791 from azat/fix-pe-overflow 
Fix incorrect UserTimeMicroseconds/SystemTimeMicroseconds accounting
 2022-11-22 19:22:51 +01:00
Raúl Marín
4aa29b6a63 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-11-22 19:09:00 +01:00
Raúl Marín
e63ba06048 Better cache management 2022-11-22 19:03:17 +01:00
Kruglov Pavel
98d6b96c82
Merge pull request #42033 from mark-polokhov/BSONEachRow 
Add BSONEachRow input/output format
 2022-11-22 14:45:21 +01:00
Nikolai Kochetov
9e84a351c7 Fixing special build. Disable setting by default. 2022-11-22 11:24:24 +00:00
Nikolai Kochetov
aeb1de5b59 Aggregating in order [in progress]. 2022-11-21 18:58:44 +00:00
avogar
ecdeff622b Add small improvements in CapnProto format 2022-11-18 20:13:00 +00:00
Nikolai Kochetov
5da1d893fd
Merge branch 'master' into read-from-mt-in-io-pool 2022-11-18 21:10:45 +01:00
Azat Khuzhin
ccde92f855 Fix incorrect UserTimeMicroseconds/SystemTimeMicroseconds accounting 
After #40732 it became possible that getrusage() (from detachQuery(),
from buildPushingToViewsChain()) will be called for incorrect thread,
and so when the difference will be calculated it will be simply garbage.

But actually the root of this problem is #25714, after which it became
possible to have multiple ThreadStatus for one thread, and this is very
tricky (sigh).

Here are some other thoughts about it:
- Make ThreadStatus nested - decided that complexity does not worth it,
  at least only for this case
- Move some members into ThreadGroupStatus - will break per-thread
  statistics (and hence query_thread_log, BTW does somebody uses it?)
- Move some members into a separate structure

But decided to fix the issue w/o any refactoring, to make easy for
backport.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-18 21:10:43 +01:00
Yakov Olkhovskiy
dbaeabcf38 fixed some bugs, some functions corrected, some tests corrected 2022-11-18 20:10:27 +00:00
Nikolai Kochetov
03c35e146b Refactor a bit. 2022-11-18 20:09:20 +00:00
Han Fei
b80a2d6c89 address comments 2022-11-18 17:22:05 +01:00
Alfonso Martinez
4b4486e139 Improved stability of test 2022-11-18 12:18:01 +01:00
Arthur Passos
414fd07bba add docs 2022-11-17 17:28:51 -03:00
Arthur Passos
dd37ca7767 add docs 2022-11-17 17:25:27 -03:00
Arthur Passos
12d3f799a5 small change 2022-11-17 17:18:54 -03:00
Arthur Passos
fcc032a31e handle both zero based and non zero based arrow offsets 2022-11-17 17:15:24 -03:00
avogar
fcfdd73d17 Improve reading CSV field in CustomSeparated/Template format 2022-11-17 15:36:56 +00:00
Raúl Marín
80403015e7 Fix assert reached with lines without data 2022-11-17 16:19:53 +01:00
Alfonso Martinez
6a8101b8b6 Test redone on bash 2022-11-17 10:14:59 +01:00
Han Fei
14701670b5 Merge branch 'master' into hanfei/aysnc-insert 2022-11-16 17:24:36 +01:00
Arthur Passos
ed080b8ba5 fix style 2022-11-16 13:22:23 -03:00
Arthur Passos
e1236340b5 Flatten list type arrow chunks on parsing 2022-11-16 12:27:01 -03:00
Nikolai Kochetov
29098045d8 Try read from MergeTree in background pool. 2022-11-15 17:38:37 +00:00
avogar
2af60f34eb Restrict document size in parallel parsing, allow to read ObjectId/JS code into String column 2022-11-15 13:35:17 +00:00
avogar
842d25c358 Minor improvements, better docs 2022-11-14 20:05:01 +00:00
Alfonso Martinez
e62ecf1a59 Merge remote-tracking branch 'upstream/master' into fix-race-condition-between-inserts-and-dropping-mvs 2022-11-14 18:23:31 +01:00
avogar
098dfcff56 Fix tests 2022-11-14 15:48:23 +00:00
Han Fei
11f93dec3f tmp commit 2022-11-14 15:53:45 +01:00
Yakov Olkhovskiy
9aeebf3bdf
Merge branch 'master' into refactoring-ip-types 2022-11-14 09:21:54 -05:00
Yakov Olkhovskiy
4d144be39c replace domain IP types (IPv4, IPv6) with native 2022-11-14 14:17:17 +00:00
Alexey Milovidov
127631ee47
Merge branch 'master' into perf_experiment 2022-11-12 18:58:25 +01:00
Nikolai Kochetov
9106910de8 Try read from MergeTree in background pool. 2022-11-11 20:47:11 +00:00
Alfonso Martinez
9bce4b8178 Added warning to fixed case 2022-11-11 21:02:37 +01:00
Alfonso Martinez
ed3a576e7d Added warning to fixed case 2022-11-11 21:02:14 +01:00
avogar
564d83bbc7 Better handle uint64 2022-11-11 13:24:12 +00:00
avogar
94c6dc42eb Use better types 2022-11-11 13:17:48 +00:00
Alfonso Martinez
b4769811fc Fixed race condition between inserts and dropping MVs 2022-11-11 12:07:50 +01:00
avogar
cd36caf013 Fix style 2022-11-10 20:37:24 +00:00
avogar
e0b3b9efae Remove old test, clean up a bit 2022-11-10 20:21:29 +00:00
avogar
4d787f3953 Remove unneded method 2022-11-10 20:18:52 +00:00
avogar
9e89af28c6 Refactor BSONEachRow format, fix bugs, support more data types, support parallel parsing and schema inference 2022-11-10 20:15:14 +00:00
Nikolai Kochetov
77c0728e77 Fix aarch build. 2022-11-10 11:59:25 +00:00
Nikolai Kochetov
f2f5c170e4
Merge branch 'master' into read-in-order-from-query-plan 2022-11-09 22:36:36 +01:00
Nikolai Kochetov
ff65ca42ad Fix limit. 2022-11-09 18:12:01 +00:00
Nikolai Kochetov
3c3771a8d7 Fix typo. 2022-11-09 16:17:36 +00:00
Nikolai Kochetov
997881c7f7 Review fixes. 2022-11-09 16:07:38 +00:00
Nikolai Kochetov
5a3d4cd72e Fix typos. Add comments. 2022-11-08 19:57:29 +00:00
Raúl Marín
0c5159d26c Merge remote-tracking branch 'blessed/master' into agg_memory_rails 2022-11-08 10:59:07 +01:00
Vladimir C
88033562cd
Merge pull request #42874 from azat/query-hung-on-cancel-fix 2022-11-08 10:52:58 +01:00
Raúl Marín
a04b9fa947 Add some guard rails around aggregation memory management 2022-11-07 17:25:04 +01:00
Nikolai Kochetov
9043df56a8 Add comments, fix tests. 2022-11-04 17:31:20 +00:00
Nikolai Kochetov
2db1638b41 Disable optimize_in_window_order in case if read-in-order for query plan is enabled. 2022-11-04 16:18:35 +00:00
Nikolai Kochetov
51ec95eb82 Try to fix #39157 in a different way. 2022-11-03 20:33:48 +00:00
Nikolai Kochetov
d01ef8ca59
Merge branch 'master' into read-in-order-from-query-plan 2022-11-03 18:59:28 +01:00
Nikolai Kochetov
280e609744 More fixes. 2022-11-03 16:42:26 +00:00
Kruglov Pavel
b124875257
Merge branch 'master' into improve-streaming-engines 2022-11-03 13:22:06 +01:00
Nikolay Degterinsky
30ad1a6826
Merge branch 'master' into perf_experiment 2022-11-03 02:18:21 +03:00
avogar
7cc87679e4 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BSONEachRow 2022-11-02 19:47:42 +00:00
Vladimir C
512abfe511 Fix style, remove commented code 2022-11-02 19:42:57 +00:00
vdimir
ef3dbf8192 clang-format BSONUtils 2022-11-02 19:42:49 +00:00
vdimir
223614ee1d Fix typos 2022-11-02 19:42:32 +00:00
vdimir
ab61932223 Apply clang-format for BSONEachRow 2022-11-02 19:42:13 +00:00
Mark Polokhov
2fff4887ac Add BSON input/output format 2022-11-02 19:39:14 +00:00
Nikolai Kochetov
1f11c73dab Another try. 2022-11-02 18:46:56 +00:00
Nikolai Kochetov
4641f12964 Another try. 2022-11-02 18:33:48 +00:00
Nikolai Kochetov
d551161a29 Another one try. 2022-11-02 17:11:17 +00:00
avogar
9f39a6a049 Fix possible heap-use-after-free 2022-11-02 14:17:48 +00:00
Kruglov Pavel
38124b6533
Merge pull request #42780 from Avogar/parallel-parsing 
Support parallel parsing for LineAsString input format
 2022-11-02 13:21:53 +01:00
Nikolai Kochetov
2766c550f6 Fix typos. 2022-11-02 10:38:32 +00:00
Nikolai Kochetov
478d3079c8 Merge branch 'master' into read-in-order-from-query-plan 2022-11-01 21:02:22 +01:00
Nikolai Kochetov
30f7c04896 Remove some debug output. 2022-11-01 19:58:20 +00:00
Nikolai Kochetov
9ffebf4a0c Fixing read-in-order for special storages. 2022-11-01 19:51:52 +00:00
Azat Khuzhin
e58008b711 Fix rare possible hung on query cancellation 
Some queries can hung after cancelling (because enough rows had been
read for Distributed queries) or after KILL. Example of such query is a
Distributed query that uses DelayedSource that uses ExpandPipeline to
add new processors, and in this case it is possible that all already
existing processes was cancelled but new had been added and now
PipelineExecutor will wait for them undefinitelly since nobody will read
from them and nobody will cancel them either.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-01 18:17:47 +01:00
Nikolai Kochetov
52204233e4 Fix some more tests. 2022-11-01 15:11:26 +00:00
Anton Popov
876dca48da
Merge pull request #36969 from CurtizJ/dynamic-columns-14 
Support `Object` type inside other types
 2022-11-01 15:20:02 +01:00
avogar
e39e61fc71 Fix heap-use-after-free in PeekableReadBuffer 2022-11-01 12:58:20 +00:00
Maksim Kita
fe48a1ce7e
Merge pull request #42681 from kitaisreal/merge-tree-indexes-use-actions-dag 
MergeTree indexes use ActionsDAG
 2022-11-01 13:55:17 +03:00
Nikolai Kochetov
e99fd4ed25 Add test 2022-10-31 20:50:10 +00:00
Anton Popov
2ae3cfa9e0
Merge branch 'master' into dynamic-columns-14 2022-10-31 16:15:19 +01:00
Nikolai Kochetov
068ae90448 Comment debug code. 2022-10-31 14:01:51 +00:00
Nikolai Kochetov
375db5bd15 Use read-in-order from query plan by default. 2022-10-31 13:54:03 +00:00
vdimir
6a4247ca32
Merge branch 'master' into grace_hash_join 2022-10-31 09:54:37 +00:00
Gabriel
94866979c5
Merge branch 'master' into comment_typo 2022-10-31 13:29:53 +08:00
Gabriel
620caeb07c Fix typo in comments 2022-10-31 13:20:58 +08:00
avogar
fe0aea2e3a Support parallel parsing for LineAsString input format 2022-10-28 21:56:09 +00:00
Nikolai Kochetov
5d41e7a6d5 Read-in-order over query plan (continuation) 2022-10-28 20:03:23 +00:00
avogar
d5f68e013d Fix style 2022-10-28 17:09:08 +00:00
avogar
8e13d1f1ec Improve and refactor Kafka/StorageMQ/NATS and data formats 2022-10-28 16:41:10 +00:00
Nikolai Kochetov
5106c24e88 Merge branch 'master' into read-in-order-from-query-plan 2022-10-27 18:12:20 +00:00
Raúl Marín
e77fcb0a99 More style 2022-10-27 13:22:44 +02:00
Raúl Marín
891484b462 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-10-27 13:17:07 +02:00
Robert Schulze
31ef6c66cb
Merge pull request #42513 from ClickHouse/update-libcxx-to-15 
Build with libcxx(abi) 15
 2022-10-27 10:30:36 +02:00
Raúl Marín
56a802188b Fix bugs introduced when changing the logic 2022-10-26 18:05:05 +02:00
Maksim Kita
1c17e9d454 Fixed tests 2022-10-26 15:54:37 +02:00
vdimir
506bf2d225
Merge branch 'master' into grace_hash_join 2022-10-26 12:25:50 +00:00
Kruglov Pavel
219553df8d
Merge pull request #42580 from hanfei1991/fei/limit_max_rows 
fix behaviour of max_rows_to_read for trival limit queries
 2022-10-26 14:09:59 +02:00
Maksim Kita
1b6293f6db MergeTree indexes use ActionsDAG 2022-10-26 12:44:37 +02:00
Raúl Marín
9395f77421 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-10-26 11:46:17 +02:00
Robert Schulze
728342ec7c
Un-inline stuff 2022-10-25 21:41:08 +00:00
Raúl Marín
6e0a9452e7 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-10-25 15:25:06 +02:00
Raúl Marín
2fa3c54caa ValuesBlockInputFormat: Adapt to the full tokenizer 2022-10-25 15:22:22 +02:00
Maksim Kita
e6bfff3de2 Analyzer change setting into allow_experimental_analyzer 2022-10-25 12:35:25 +02:00
Han Fei
8f00d0d1cc fix tidy 2022-10-24 12:56:36 +02:00
Maksim Kita
8cadb1b318 Added WINDOW functions support 2022-10-24 10:22:20 +02:00
Maksim Kita
4f68305dbe Fixed tests 2022-10-24 10:22:20 +02:00
Maksim Kita
ca93ee7479 Fixed tests 2022-10-24 10:22:20 +02:00
Maksim Kita
fed146e198 Added SAMPLE BY support. Added SAMPLE BY, FINAL support for JOINS. 2022-10-24 10:22:20 +02:00
Maksim Kita
85193ede42 Added UNION support 2022-10-24 10:22:19 +02:00
Robert Schulze
c119cd2f00
Merge branch 'master' into update-libcxx-to-15 2022-10-24 08:29:37 +02:00
Han Fei
2fc91fd338 fix behaviour of max_rows_to_read for trival limit queries 2022-10-22 18:27:24 +02:00
Azat Khuzhin
56bc85746f Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Interpreters/ProcessList.cpp
 2022-10-22 16:49:08 +02:00
Azat Khuzhin
5094c0dd6d Fix clang-tidy performance-inefficient-vector-operation 
By some reason it appears only after static_cast<> was added [1]:

    /build/src/Processors/Formats/Impl/AvroRowInputFormat.cpp
    Oct 18 01:03:56 /build/src/Processors/Formats/Impl/AvroRowInputFormat.cpp:351:21: error: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation,-warnings-as-errors]
    Oct 18 01:03:56                     symbols.push_back(root_node->nameAt(i));
    Oct 18 01:03:56                     ^
    Oct 18 01:03:56 /build/src/Processors/Formats/Impl/AvroRowInputFormat.cpp:511:17: error: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation,-warnings-as-errors]
    Oct 18 01:03:56                 union_skip_fns.push_back(createSkipFn(root_node->leafAt(i)));
    Oct 18 01:03:56                 ^
    Oct 18 01:03:56 /build/src/Processors/Formats/Impl/AvroRowInputFormat.cpp:552:17: error: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation,-warnings-as-errors]
    Oct 18 01:03:56                 field_skip_fns.push_back(createSkipFn(root_node->leafAt(i)));
    Oct 18 01:03:56                 ^
    Oct 18 01:03:56 197965 warnings generated.

  [1]: https://s3.amazonaws.com/clickhouse-builds/42190/453d91fa3539882dcef1d5ecd5097747499572d8/clickhouse_special_build_check/report.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:43 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Alexey Milovidov
ee5f5a4cb4 Unfathomable amount of trash 2022-10-21 05:33:17 +02:00
Alexey Milovidov
ff26251477 Merge branch 'master' into fix-race-condition-finish-cancel 2022-10-21 04:14:21 +02:00
Alexander Tokmakov
68c18abfbb
Merge pull request #42406 from ClickHouse/template_format_better_error 
Better error message for unsupported delimiters in custom formats
 2022-10-20 15:52:08 +03:00
Robert Schulze
820e6b4276
Build with libcxx(abi) 15 2022-10-20 10:52:43 +00:00
vdimir
5118c5d74f
wip grace hash 2022-10-19 10:20:41 +00:00
Alexey Milovidov
dfa202a15d Merge branch 'master' into fix-race-condition-finish-cancel 2022-10-19 02:35:42 +02:00
Kruglov Pavel
29513f6a1f
Merge pull request #41885 from Avogar/with-names-error-message 
Better exception message for duplicate column names in schema inference
 2022-10-18 15:26:46 +02:00
Kruglov Pavel
25e13bdd2f
Merge pull request #41107 from Avogar/improve-combinators 
Support all combinators combination in WindowTransform/arratReduce*/initializeAggregation/aggregate functions versioning
 2022-10-18 15:24:49 +02:00
vdimir
bf3f66b5b6
fix delayed blocks for grace hash join 2022-10-18 11:43:01 +00:00
Vladimir C
a298a5238b
Merge pull request #42217 from ClickHouse/vdimir/style-check-code-used-in-comment 2022-10-18 13:26:39 +02:00
Alexander Tokmakov
fffecbb9ad better error message for unsupported delimiters in custom formats 2022-10-17 18:08:52 +02:00
vdimir
adb63a5583
Merge branch 'master' into grace_hash_join 2022-10-17 12:32:56 +00:00
Alexey Milovidov
f88ed8195b Fix trash 2022-10-17 04:21:08 +02:00
Alexey Milovidov
c6b2ee47df Fix data race in query finish/cancel 2022-10-17 02:10:36 +02:00
Kruglov Pavel
7980920bd7
Merge branch 'master' into fix-format-row 2022-10-14 20:49:21 +02:00
Kruglov Pavel
6fc12dd922
Merge pull request #41703 from Avogar/json-object-each-row 
Add setting to obtain object name as column value in JSONObjectEachRow format
 2022-10-14 20:11:04 +02:00
Kruglov Pavel
ff11904850
Merge branch 'master' into improve-combinators 2022-10-14 17:19:31 +02:00
vdimir
ad255206f4
Exclude comments from style-check defined extern 2022-10-14 11:42:18 +00:00
vdimir
0178307c27 Followup for TemporaryDataOnDisk 2022-10-12 15:25:23 +02:00
Nikolai Kochetov
dac71f445b
Merge pull request #42186 from ClickHouse/igor/distinct_in_order_fix 
Fix: DISTINCT in order fails with LOGICAL_ERROR
 2022-10-10 09:51:29 +02:00
Igor Nikonov
f4792c37ef Fix #42185: DISTINCT in order fails with LOGICAL_ERROR 
... if first column in sorting key contains function
 2022-10-08 10:13:12 +00:00
Alexander Tokmakov
4175f8cde6 abort instead of __builtin_unreachable in debug builds 2022-10-07 21:49:08 +02:00
vdimir
5a7c493559
Merge branch 'master' into pr/BigRedEye/38191 2022-10-06 14:43:35 +00:00
Robert Schulze
78be400ac0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup 2022-10-06 12:22:43 +00:00
Robert Schulze
da5a2e2db0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup 
Physical merge conflicts:
- src/Common/ZooKeeper/ZooKeeperImpl.cpp
- src/Core/config_core.h.in
- src/Functions/FunctionsAES.h
- src/Functions/config_functions.h.in
- src/configure_config.cmake

Logical merge conflicts:
- Functions/tryDecrypt.cpp
 2022-10-06 08:43:25 +00:00
vdimir
ff55c369bc
Merge branch 'tmp-data-followup' 2022-10-05 18:10:05 +00:00
vdimir
ccd29e172f
Followup for TemporaryDataOnDisk 2022-10-05 16:42:27 +00:00
vdimir
a0349cf8c5
wip grace hash 2022-10-05 12:40:32 +00:00
Anton Popov
fe90919f10 fix crash in SummingMergeTree with LowCardinality 2022-10-04 15:33:38 +00:00
vdimir
e5678afc2d
wip grace hash 2022-10-04 08:21:02 +00:00
vdimir
f3781be762
wip grace hash 2022-10-04 08:20:13 +00:00
Anton Popov
6e61cf92f5 Merge remote-tracking branch 'upstream/master' into HEAD 2022-10-03 13:16:57 +00:00
Nikolai Kochetov
dcfb8a1131
Merge pull request #41949 from ClickHouse/improvement-from-38083 
Earlier throw exception in PullingAsyncPipelineExecutor.
 2022-10-03 13:05:58 +02:00
Robert Schulze
db5ef7b3cb
Merge branch 'master' into generated-file-cleanup 2022-10-02 23:13:18 +02:00
Vitaly Baranov
f65d3ff95a Fix parallel parsing: segmentator now checks max_block_size. 2022-09-30 22:34:03 +02:00
vdimir
7ebc297f4c
Merge branch 'master' into pr/BigRedEye/38191 2022-09-30 09:40:47 +00:00
Vladimir C
895afdec45
Merge pull request #40893 from ClickHouse/vdimir/track-tmp-disk 2022-09-30 11:27:24 +02:00
Robert Schulze
cc92a2d174
Merge branch 'master' into generated-file-cleanup 2022-09-30 09:56:31 +02:00
Nikolai Kochetov
8f9e6dd03a Earlier throw exception in PullingAsyncPipelineExecutor. 2022-09-29 18:07:43 +00:00
Nikolai Kochetov
8e23bee03c
Merge pull request #41650 from ClickHouse/keep-destruction-order-of-view-thread-statuses 
Fix a destruction order for views ThreadStatus
 2022-09-29 14:28:14 +02:00
vdimir
f495361e28
fixes for TemporaryDataOnDisk 2022-09-29 10:09:29 +00:00
vdimir
14cd2b632c
fix Context::shutdown 2022-09-29 09:51:49 +00:00
vdimir
0f1a7c252d
better TemporaryDataOnDisk 2022-09-29 09:51:46 +00:00
vdimir
9f3f34548c
Allow to create temporaty streams on leaf TemporaryDataOnDisk 2022-09-29 09:51:45 +00:00
vdimir
858769584f
Fix MergeSortingTransform 2022-09-29 09:51:43 +00:00
vdimir
15c7a3be34
Temp data on disk: build 2022-09-29 09:51:41 +00:00
vdimir
c0898ce289
Use abstraction for temporary data on disk in Sort and Aggregation 2022-09-29 09:51:41 +00:00
vdimir
ac39bbb3f1
[wip] Common interface for temporary data on disk 2022-09-29 09:51:40 +00:00
Robert Schulze
f24fab7747
Fix some #include atrocities 2022-09-28 13:49:28 +00:00
Robert Schulze
fd86829824
Consolidate config_core.h into config.h 
Less duplication, less confusion ...
 2022-09-28 13:31:57 +00:00
avogar
c353928eb5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-format-row 2022-09-28 13:15:51 +00:00
avogar
2583e6d3ce Use string_view 2022-09-28 13:14:54 +00:00
Robert Schulze
09c62f6728
Consolidate config_formats.h into config.h 
Less duplication, less confusion ...
 2022-09-28 12:59:05 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Kruglov Pavel
f1ac2d66be
Merge branch 'master' into json-object-each-row 2022-09-28 14:15:02 +02:00
avogar
1bd7e531db Better exception message for duplicate column names in schema inference 2022-09-28 12:07:25 +00:00
Kruglov Pavel
3dc54272ed
Merge branch 'master' into improve-combinators 2022-09-26 13:03:32 +02:00
Igor Nikonov
5aa92e470a
Merge branch 'master' into distinct_in_order_wo_order_by 2022-09-23 11:34:54 +02:00
Igor Nikonov
6551966dc7 Change the way reading in order is requested from plan optimizations 2022-09-22 20:47:00 +00:00
Igor Nikonov
9ea277c047 Remove unnecessary method from ReadFromMergeTree 2022-09-22 19:52:02 +00:00
Igor Nikonov
6f7d0fec52 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-22 19:38:34 +00:00
Nikolai Kochetov
00965ce17a Fixing race. 2022-09-22 17:57:04 +00:00
avogar
6a1cb604c4 Style 2022-09-22 17:06:56 +00:00
avogar
4f32ef9bb7 Add docs 2022-09-22 17:04:42 +00:00
avogar
d3d06251a3 Add setting to obtain object name as column value in JSONObjectEachRow format 2022-09-22 16:48:54 +00:00
avogar
f23a77156f Check file path for path traversal attacks in errors logger for input formats 2022-09-22 13:56:51 +00:00
Kruglov Pavel
55d7addcfe
Merge branch 'master' into fix-format-row 2022-09-22 12:32:58 +02:00
Kruglov Pavel
2c83abaaba
Merge pull request #41614 from ClickHouse/Avogar-patch-1 
Fix typos in JSON formats after #40910
 2022-09-22 10:58:47 +02:00
Igor Nikonov
8c93a9adda Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-22 07:40:14 +00:00
Alexey Milovidov
1d1898bc88
Merge pull request #41586 from Algunenano/fix_missing_read_bytes_in_mv 
Fix read bytes/rows in X-ClickHouse-Summary with materialized views
 2022-09-22 07:32:16 +03:00
Alexey Milovidov
2a75e025f2
Merge pull request #40715 from ClickHouse/compress-marks 
Merging #37693: Compress marks and primary key
 2022-09-22 07:30:34 +03:00
Alexey Milovidov
45afacdae4
Merge pull request #41186 from ClickHouse/fix-three-fourth-of-trash 
Fix more than half of the trash
 2022-09-22 07:28:26 +03:00
Nikolai Kochetov
446453bdf5 Fix a destruction order for views ThreadStatus 2022-09-21 18:37:40 +00:00
Nikita Taranov
100c055510
Prefetching in aggregation (#39304) 
* impl

* stash

* clean up

* do not apply when HT is small

* make branch static

* also in merge

* do not hardcode look ahead value

* fix

* apply to methods with cheap key calculation

* more tests

* silence tidy

* fix build

* support HashMethodKeysFixed

* apply during merge only for cheap

* stash

* fixes

* rename method

* add feature flag

* cache prefetch threshold value

* fix

* fix

* Update HashMap.h

* fix typo

* 256KB as default l2 size

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2022-09-21 18:59:07 +02:00
Nikolai Kochetov
1204f643f1
Merge pull request #41576 from ClickHouse/fix-partial-sort-optimisation-bug-from-41182 
Fix a bug with missing rows after partial sort optimization
 2022-09-21 16:47:18 +02:00
Nikolai Kochetov
7e0914c3ce
Merge pull request #41588 from ClickHouse/pipeline-stuck-and-offset 
Fix possible pipeline stuck exception for queries with OFFSET
 2022-09-21 16:42:42 +02:00
Igor Nikonov
c849dd4c76
Merge pull request #41574 from ClickHouse/igor/use_only_input_order_info 
ReadFromMergeTree: reading in order only if input order info is provided
 2022-09-21 15:36:51 +02:00
Arthur Passos
cf1ed58710 Use separate functions for parquet time32 and time64 2022-09-21 14:56:11 +02:00
Vladimir C
efa34b4013 Fix style 2022-09-21 14:56:11 +02:00
Arthur Passos
c0914a39a7 Add Parquet Time32/64 conversion to CH DateTime32/64 2022-09-21 14:56:11 +02:00
Kruglov Pavel
dcb8fbc3f8
Fix JSONEachRow 2022-09-21 14:25:34 +02:00
Kruglov Pavel
95135e1e31
Fix typos in JSON formats after #40910 2022-09-21 14:24:26 +02:00
Kruglov Pavel
22e11aef2d
Merge pull request #40910 from Avogar/new-json-formats 
Add new JSON formats, add improvements and refactoring
 2022-09-21 14:19:08 +02:00
avogar
6239a1a235 Fix build 2022-09-21 11:29:00 +00:00
avogar
f956e7915e Fix tests 2022-09-20 20:37:30 +00:00
Alexey Milovidov
45bd3cfc30 Merge branch 'master' into fix-three-fourth-of-trash 2022-09-20 21:27:41 +02:00
Igor Nikonov
f579da5b30 Merge remote-tracking branch 'origin/master' into igor/use_only_input_order_info 2022-09-20 15:01:33 +00:00
Nikolai Kochetov
0df2a31911 Fix possible pipeline stuck exception for queries with OFFSET 2022-09-20 14:58:53 +00:00
Igor Nikonov
757736679f Fix build 2022-09-20 14:53:45 +00:00
Nikolai Kochetov
2b46735c42 Fix a bug with missing rows after partial sort optimisation #41182 2022-09-20 14:08:39 +00:00
avogar
90e0f98d45 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-format-row 2022-09-20 13:55:44 +00:00
avogar
b86aec41d4 Remove unused file after renaming 2022-09-20 13:54:54 +00:00
avogar
868ce8bc16 Fix comments, make better naming, add docs, add setting output_format_json_quote_64bit_floats 2022-09-20 13:49:17 +00:00
Igor Nikonov
394c2f708d ReadFromMergeTree: reading in order only if input order info is provided 
- remove checking optimize*in*order settings, input order info should be
  set only if a corresponding optimization is enabled and applicable
 2022-09-20 13:30:27 +00:00
Raúl Marín
133345eb37 Fix missing read bytes/rows in X-ClickHouse-Summary 2022-09-20 15:12:44 +02:00
Igor Nikonov
88b60f861f Do not overwrite direction by distinct in order optimization 
+ tests
 2022-09-20 11:27:37 +00:00
Igor Nikonov
aca810ba62 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-19 18:34:38 +00:00
Igor Nikonov
78bb598d64 Handling DISTINCT in order on top of other read in order optimization 2022-09-19 18:33:05 +00:00
avogar
a7de3daa13 Fix tests 2022-09-19 14:13:46 +00:00
Kruglov Pavel
57f0dc1f89
Merge branch 'master' into fix-format-row 2022-09-19 14:37:58 +02:00
Kruglov Pavel
47f6f09ce0
Merge branch 'master' into improve-combinators 2022-09-19 14:31:12 +02:00
Igor Nikonov
81a46b6346 Merge remote-tracking branch 'origin/master' into fix_sort_desc_when_read_in_order 2022-09-19 09:40:23 +00:00
Igor Nikonov
54c2fed9cc Update: use type explicitly 2022-09-19 09:31:45 +00:00
Alexey Milovidov
45b6052b20 Merge branch 'master' into compress-marks 2022-09-19 10:57:05 +02:00
Igor Nikonov
b4a3ac926f Fix typo 2022-09-19 08:26:44 +00:00
Alexey Milovidov
730655d4fd Fix 8/9 of trash 2022-09-19 08:53:20 +02:00
Alexey Milovidov
91baedf03a Fix 6/7 of trash 2022-09-19 08:53:20 +02:00
Alexey Milovidov
84f42e0874 Fix 3/4 of trash 2022-09-19 08:50:53 +02:00
Alexey Milovidov
81e8cb4be6
Merge branch 'master' into fix-bug-orc 2022-09-19 06:38:17 +03:00
Alexey Milovidov
d4b9fe41be
Merge pull request #41457 from ClickHouse/remove-trash-5 
Remove trash from Field
 2022-09-19 06:36:48 +03:00
Igor Nikonov
785c33bf7d Updating sort description for ReadFromMergeTree correctly 2022-09-18 23:35:21 +00:00
Igor Nikonov
e888859396 Fix: correct sort description for ReadFromMergeTree with read in order 
optimization
 2022-09-18 22:21:13 +00:00
Kruglov Pavel
519bcbb3be
Merge pull request #41236 from Avogar/fix-totals-extremes 
Don't output totals/extremes in all row formats, update docs
 2022-09-18 14:55:07 +02:00
Alexey Milovidov
8764fa4439 Fix very strange behavior of Apache ORC 2022-09-18 08:25:25 +02:00
Alexey Milovidov
791de6592b Remove trash from Field 2022-09-18 05:16:08 +02:00
Alexey Milovidov
ada7a44ae4 Remove -WithTerminatingZero methods 2022-09-17 05:34:18 +02:00
Alexey Milovidov
c09604bef2
Merge pull request #41363 from kitaisreal/small-style-fixes 
Small style fixes
 2022-09-17 00:19:00 +03:00
Nikita Taranov
6f186d3dd2
Do not return empty blocks from ConvertingAggregatedToChunksTransform (#41152) 
* impl

* add test

* update test
 2022-09-16 21:54:36 +02:00
avogar
0101cc2e56 Support complex combinators in window transform, arrayReduce*, initializeAggregation and Aggregate functons versionning 2022-09-16 19:07:36 +00:00
Igor Nikonov
24de5530d6 Remove unnecessary headers 2022-09-16 17:16:32 +00:00
Igor Nikonov
b34ec6453b Fix review comments 
+ one more test
 2022-09-16 17:11:08 +00:00
avogar
1de7b65b97 Fix tests 2022-09-16 14:05:08 +00:00
Kruglov Pavel
2d4a6b38af
Merge branch 'master' into fix-totals-extremes 2022-09-16 15:03:59 +02:00