Igor Nikonov
366ead3828
Consider aliases when checking if sorting order is preserved by
...
expression
2022-08-09 11:27:17 +00:00
Igor Nikonov
1439664df6
EXPLAIN tests
2022-08-08 20:46:43 +00:00
Maksim Kita
c030fd05e7
ActionsDAG rename index to outputs
2022-08-08 18:01:32 +02:00
Igor Nikonov
15bdeba074
Fix review comments
2022-08-08 14:43:03 +00:00
avogar
1304e3487c
Add comments, remove unneded stuff
2022-08-08 13:43:14 +00:00
Igor Nikonov
86c5280db3
Merge remote-tracking branch 'origin/master' into skipping_sorting_step
2022-08-08 12:43:06 +00:00
Igor Nikonov
6572a447fa
Check if sorting order is preserved for FilterStep
2022-08-08 12:42:10 +00:00
avogar
2f95726b06
Fix comments
2022-08-08 12:41:00 +00:00
vdimir
708747ca0b
Merge branch 'master' into refactor-prepared-sets
2022-08-08 14:27:18 +02:00
Vladimir C
6bd4821656
Merge pull request #39575 from vdimir/join_pushdown_column_not_found
...
Fix column not found for push down with join
2022-08-08 14:04:42 +02:00
alexX512
6bf29cb610
Change class LRUCache to class CachBase. Check running CacheBase with default pcahce policy SLRU
2022-08-07 19:59:30 +00:00
Igor Nikonov
254d14b1d3
Remove debug trace from DistinctStep
...
+ this information can be seen later via EXPLAIN PLAN
2022-08-07 13:12:11 +00:00
Igor Nikonov
f52720fedc
Try to fix style errors
2022-08-06 23:33:19 +00:00
Igor Nikonov
988ce29e9e
Check if expression step violates sorting correctly
2022-08-06 22:37:14 +00:00
Igor Nikonov
8278da6475
Fix: read row counts before move columns out of chunk
2022-08-05 21:29:57 +00:00
Igor Nikonov
9fddf6efde
Merge remote-tracking branch 'origin/master' into ordinary_distinct_small_refact
2022-08-05 19:23:44 +00:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
Igor Nikonov
f5f6ada69b
ActionsDAG check is not dependent on sort description
2022-08-05 11:00:18 +00:00
Nikolai Kochetov
cf34232675
Output header is now empty for every MV chain.
...
Instead of checking that number of processors different for different
threads, simply always return empty header from buildChainImpl(), by
adding explicit conversion.
v2: ignore UNKNOWN_TABLE errors in test
2022-08-05 13:16:32 +03:00
Igor Nikonov
246f7b71b6
Merge branch 'master' into skipping_sorting_step
2022-08-05 00:03:54 +02:00
Igor Nikonov
a70c47f780
EXPLAIN PLAN sortmode options
...
+ ActionsDAG fixes
2022-08-04 22:02:53 +00:00
Igor Nikonov
0a659f5ab8
Analyze ActionsDAG in ExpressionStep
...
+ check if sorting order can be kept from prevous step
+ EXPLAIN PLAN header=1 contains Sort Mode and Sort Description for each step
+ some tests
+ fix some review comments
2022-08-04 19:30:25 +00:00
Arthur Passos
62d48053c0
Use insertDefault instead of insert(0)
2022-08-04 15:53:44 -03:00
Arthur Passos
c307e9a228
Fix ArrowColumn dictionary to CH low cardinality conversion
2022-08-04 15:34:44 -03:00
Nikolai Kochetov
658a269d56
Merge branch 'master' into use-dag-in-key-condition
2022-08-04 16:18:40 +02:00
Vladimir C
a627b00c43
Merge branch 'master' into refactor-prepared-sets
2022-08-04 13:27:38 +02:00
Kruglov Pavel
235649cb98
Merge pull request #39458 from Avogar/fix-cancel-insert-into-function
...
Fix WriteBuffer finalize when cancel insert into function
2022-08-04 13:02:08 +02:00
vdimir
298fb2431d
Fix column not found for push down with join
2022-08-04 10:58:21 +00:00
Nikita Taranov
4943202921
Improve memory usage during memory efficient merging of aggregation results ( #39429 )
2022-08-03 17:56:59 +02:00
Vladimir Chebotaryov
484d8f7028
Merge branch 'master' into betterorderbyoptimization
2022-08-03 08:14:50 +03:00
Igor Nikonov
8eafca65c9
Fix style
2022-08-02 17:58:31 +00:00
Igor Nikonov
30782a2b05
Test: distinct sorted is not used on const column
2022-08-02 17:44:43 +00:00
Kruglov Pavel
6b2186bfeb
Merge branch 'master' into numbers-schema-inference
2022-08-02 19:34:53 +02:00
Igor Nikonov
56697125d3
Merge remote-tracking branch 'origin/master' into distinct_sorted_simplify
2022-08-02 11:18:46 +00:00
Igor Nikonov
4bf42ce867
Merge remote-tracking branch 'origin/master' into skipping_sorting_step
2022-08-02 10:50:25 +00:00
Igor Nikonov
0cb59aa38c
ExpressionStep: simple check if expression changes sorting order
2022-08-02 10:45:05 +00:00
Kruglov Pavel
42136b7630
Merge pull request #39647 from Avogar/fix-arrow-strings
...
Fix strings in dictionary in Arrow format
2022-08-01 12:46:07 +02:00
Vladimir Chebotaryov
aaa6049678
Merge branch 'master' into betterorderbyoptimization
2022-08-01 11:42:48 +03:00
Vladimir Chebotaryov
9fc3addea9
Fixed using column_after_join for handling WINDOW expressions in ExpressionAnalyzer, shall be aggregated_columns.
2022-07-31 23:36:20 +03:00
Nikolai Kochetov
22fbfe19a4
Merge branch 'master' into use-dag-in-key-condition
2022-07-31 21:54:12 +02:00
Igor Nikonov
7f0adb5eb0
Merge remote-tracking branch 'origin/master' into skipping_sorting_step
2022-07-31 07:07:36 +00:00
Igor Nikonov
83e1dd1172
Merge branch 'master' into ordinary_distinct_small_refact
2022-07-31 00:23:21 +02:00
Alexey Milovidov
4828be7fc4
Fix double escaping in the metadata of FORMAT JSON
2022-07-30 23:56:41 +02:00
Igor Nikonov
7245ddcc20
Simple refactoring: ordinary DISTINCT implementation
2022-07-30 20:25:56 +00:00
Igor Nikonov
a7cfad105e
Merge branch 'master' into distinct_sorted_simplify
2022-07-30 21:57:53 +02:00
Igor Nikonov
3be51a6dea
Construct DistinctSortedTransform only when it makes sense
...
otherwise fallback to DistinctTransform (i.e. ordinary distinct)
2022-07-30 19:41:03 +00:00
Maksim Kita
acb0137dbb
Merge pull request #39718 from kitaisreal/join-enums-refactoring
...
Join enums refactoring
2022-07-30 13:53:08 +02:00
Igor Nikonov
d951154ef4
Proved NULLs direction when compare rows
2022-07-29 22:12:03 +00:00
Igor Nikonov
13dc1697fb
Remove unnecessary initialization
2022-07-29 20:34:23 +00:00
Igor Nikonov
b44373ba8f
Merge remote-tracking branch 'origin/master' into distinct_sorted_simplify
2022-07-29 20:33:26 +00:00
Igor Nikonov
7b0b38e997
DistinctSortedTransform works only if columns contains sort prefix of
...
sort description
2022-07-29 20:01:07 +00:00
Kruglov Pavel
ccd1e1bdb8
Merge branch 'master' into fix-cancel-insert-into-function
2022-07-29 20:27:32 +02:00
Maksim Kita
8fc6bad4f4
Join enums refactoring
2022-07-29 18:35:05 +02:00
Nikolai Kochetov
59a11b32ad
Merge branch 'master' into use-dag-in-key-condition
2022-07-29 17:01:33 +02:00
Igor Nikonov
fb5799fe7f
More generic getInputOrderInfo() in query info
2022-07-29 14:36:36 +00:00
Igor Nikonov
4af435bdda
Fix: handle all const columns case correctly
2022-07-28 21:22:06 +00:00
Igor Nikonov
f414af2348
Merge pull request #39538 from ClickHouse/fix_distinct_sorted
...
Fix: DistinctSortedTransform doesn't take advantage of sorting
2022-07-28 21:47:10 +02:00
Sergei Trifonov
edf6b02723
Merge pull request #37558 from ClickHouse/concurrency-control
...
add concurrency control for pipeline executors
2022-07-28 08:56:24 +02:00
Vladimir C
115506356c
Merge branch 'master' into refactor-prepared-sets
2022-07-27 19:57:23 +02:00
Sergei Trifonov
05e4bd6c8b
review fix
2022-07-27 15:36:50 +02:00
Igor Nikonov
377c04fbf1
Merge remote-tracking branch 'origin/master' into fix_distinct_sorted
2022-07-27 13:01:18 +00:00
Anton Popov
1547c010b9
Merge pull request #39432 from ClickHouse/distinct_sorted_chunk_perf_impr
...
DISTINCT in order: perf improvement
2022-07-27 14:17:58 +02:00
avogar
01a309d4e3
Fix strings in dictionary in Arrow format
2022-07-27 12:02:27 +00:00
avogar
f925046dc4
Add more Pretty formats
2022-07-27 11:37:02 +00:00
vdimir
8eecb9ef82
upd PreparedSets: rename/change signature of methods, add comments
2022-07-27 11:22:16 +00:00
Nikolai Kochetov
873432fb53
Merge pull request #37849 from ClickHouse/bug-with-fill-date
...
Enforce equality of WITH FILL type with ORDER BY column's type for date/time types.
2022-07-27 12:27:53 +02:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache
2022-07-27 11:35:36 +02:00
Igor Nikonov
64e51e56e7
Allocate memory for column arrays once
2022-07-27 08:22:07 +00:00
Igor Nikonov
12a7567402
Some polishing
2022-07-27 07:58:54 +00:00
Igor Nikonov
589104fa6e
Make building column arrays for chunk processing straightforward
2022-07-27 07:44:42 +00:00
Alexander Gololobov
a64aa00869
Merge pull request #37893 from zhangjmruc/feature/sql-standard-delete
...
Support SQL standard "delete from ... where ..." syntax and lightweight implementation on merge tree tables
2022-07-26 23:39:07 +02:00
Igor Nikonov
cac4d77d0b
Merge remote-tracking branch 'origin/master' into distinct_sorted_chunk_perf_impr
2022-07-26 20:36:38 +00:00
Yakov Olkhovskiy
0055c9307d
style fix
2022-07-26 16:08:03 -04:00
Igor Nikonov
24f3a6905f
Merge branch 'master' into fix_distinct_sorted
2022-07-26 21:57:44 +02:00
Igor Nikonov
d196ab24d4
Calculate DISTINCT columns positions which form sort prefix in sort
...
description once
2022-07-26 19:55:29 +00:00
Yakov Olkhovskiy
d93c67e303
comment and test added
2022-07-26 15:28:11 -04:00
vdimir
d9928ac93d
Add methods to SubqueryForSet, do not use refernce to SetPtr
2022-07-26 18:39:09 +00:00
vdimir
1e3fa2e01f
Refactor PreparedSets/SubqueryForSet
2022-07-26 18:39:02 +00:00
Nikolai Kochetov
d7cc3831ea
Push predicate over StorageMerge.
2022-07-26 15:01:39 +00:00
Kruglov Pavel
c362551642
Merge pull request #39357 from Avogar/fix-protobuf-capnp-empty-message
...
Fix schema inference in case of empty messages in Protobuf/CapnProto formats
2022-07-26 12:33:28 +02:00
Kruglov Pavel
53159db782
Merge branch 'master' into numbers-schema-inference
2022-07-26 12:32:49 +02:00
Sergei Trifonov
24ab5fbb86
fix finish() condition to account only active threads
2022-07-26 11:32:45 +02:00
Alexander Gololobov
0666ec2e1f
Merge branch 'master' into feature/sql-standard-delete
2022-07-26 10:42:39 +02:00
Vladimir Chebotaryov
f32d9c5539
Uppercase ROWS, GROUPS, RANGE in queries with windows. ( #39410 )
2022-07-25 22:53:53 +02:00
Alexander Gololobov
25deba2c1b
Merge branch 'master' into feature/sql-standard-delete
2022-07-25 22:13:20 +02:00
Nikolai Kochetov
b70be40804
Merge branch 'master' into use-dag-in-key-condition
2022-07-25 14:30:22 +02:00
Igor Nikonov
572dd864fe
Merge remote-tracking branch 'origin/master' into skipping_sorting_step
2022-07-25 12:19:14 +00:00
Sergei Trifonov
d9d1381c3f
Merge branch 'master' into concurrency-control
2022-07-25 14:16:46 +02:00
Kruglov Pavel
83c7da6e88
Merge branch 'master' into fix-protobuf-capnp-empty-message
2022-07-25 13:02:41 +02:00
Igor Nikonov
dc96a6270d
Remove optimize_memory_usage flag
...
Memory usage with DistinctSortedTransform fixed in #39538
2022-07-25 10:27:33 +00:00
Alexey Milovidov
388d06fda1
Merge pull request #39535 from ClickHouse/stringref
...
Less usage of StringRef
2022-07-25 04:06:11 +03:00
Igor Nikonov
41e72aac83
Fix: DistinctSortedTransform doesn't take advantage of sorting
...
clearing_columns are set incorrectly, so we never clear HashSet
2022-07-24 21:35:36 +00:00
Alexander Gololobov
460950ecdc
Merge branch 'master' into feature/sql-standard-delete
2022-07-24 21:27:22 +02:00
Robert Schulze
4333750985
Less usage of StringRef
...
... replaced by std::string_view, see #39262
2022-07-24 18:33:52 +00:00
Igor Nikonov
95511428b3
Couple optimizations
...
+ do not apply filter to chunk if there is no data for output
+ checking clear_data flag at compile time
2022-07-23 00:03:26 +00:00
Igor Nikonov
739ff34c6e
Add some tests, still not sure about optimize_memory_usage option
2022-07-22 22:48:26 +00:00
Igor Nikonov
329adc3419
Merge branch 'master' into distinct_sorted_chunk_perf_impr
2022-07-22 12:58:59 +02:00
Igor Nikonov
51f63c3a0c
Add setting optimize_distinct_in_order_memory_usage
2022-07-22 10:57:57 +00:00
Alexander Tokmakov
bed2206ae9
Merge pull request #39460 from ClickHouse/remove_some_dead_and_commented_code
...
Remove some dead and commented code
2022-07-22 13:24:34 +03:00
Igor Nikonov
7db5d54820
Adopt to the case when not all columns in distinct are part of sorting
...
description
2022-07-21 21:04:58 +00:00
avogar
794aa691bc
Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-protobuf-capnp-empty-message
2022-07-21 17:04:37 +00:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache
2022-07-21 18:59:14 +02:00
Robert Schulze
ea0a3bf600
Merge branch 'master' into stringref-to-string_view
2022-07-21 18:33:06 +02:00
avogar
fd534aa3fa
wqMerge branch 'master' of github.com:ClickHouse/ClickHouse into numbers-schema-inference
2022-07-21 15:43:17 +00:00
Igor Nikonov
122a1123b2
- disable the worst case for distinct in order in perf test for now
...
+ functional test for query with the worst perfomance
+ debug logging in DistinctStep
2022-07-21 15:03:19 +00:00
Alexander Tokmakov
a8da5d96fc
remove some dead and commented code
2022-07-21 15:05:48 +02:00
avogar
6b541aa98f
Fix WriteBuffer finalize when cancel insert into function
2022-07-21 12:18:37 +00:00
Alexander Gololobov
1ea9f143ff
Leave only _row_exists-based implementation of lightweight delete
2022-07-21 11:26:13 +02:00
Nikolai Kochetov
e15967e9db
Merge pull request #38475 from ClickHouse/additional-filters
...
Additional filters for a table (from setting)
2022-07-21 07:52:04 +02:00
Alexey Milovidov
844042fc18
Merge pull request #39433 from ClickHouse/revert-39396-try-fix-write-buffer-terminate
...
Revert "Fix WriteBuffer finalize in destructor when cacnel query"
2022-07-21 07:04:07 +03:00
Alexey Milovidov
dcda9d3bd1
Merge pull request #39365 from Avogar/fix-capnproto-abort
...
Avoid possible abort() in CapnProto on exception descruction
2022-07-21 05:20:45 +03:00
Kruglov Pavel
92995a832b
Revert "Fix WriteBuffer finalize in destructor when cacnel query"
2022-07-21 01:45:16 +02:00
Igor Nikonov
635a566bec
Comment change
2022-07-20 21:37:46 +00:00
Igor Nikonov
e50aebb5f0
Merge branch 'master' into distinct_sorted_chunk_perf_impr
2022-07-20 23:17:11 +02:00
Igor Nikonov
965f96bd84
DISTINCT in order: perf improvement
...
+ reduce allocations in DistinctSortedChunkTransform
+ use it for final distinct as well
2022-07-20 20:44:47 +00:00
Nikolai Kochetov
91043351aa
Fixing build.
2022-07-20 20:30:16 +00:00
Nikolai Kochetov
4e8cd70b1d
Merge branch 'master' into use-dag-in-key-condition
2022-07-20 17:38:33 +02:00
Yakov Olkhovskiy
c4d040e02c
Merge branch 'master' into bug-with-fill-date
2022-07-20 09:10:45 -04:00
Dmitry Novik
a09422de7b
Merge pull request #38642 from ClickHouse/group-by-use-nulls
...
Add group_by_use_nulls setting
2022-07-20 14:42:42 +02:00
Kruglov Pavel
46da17ca8c
Merge branch 'master' into numbers-schema-inference
2022-07-20 13:32:39 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache
2022-07-20 13:30:42 +02:00
avogar
784ee11594
Add settings to skip fields with unsupported types in Protobuf/CapnProto schema inference
2022-07-20 11:16:25 +00:00
Kruglov Pavel
a1b63b4a02
Fix style
2022-07-20 12:07:22 +02:00
Kruglov Pavel
7722b647b7
Merge pull request #39396 from Avogar/try-fix-write-buffer-terminate
...
Fix WriteBuffer finalize in destructor when cacnel query
2022-07-20 12:06:20 +02:00
Nikolai Kochetov
f570cde815
Fixing build.
2022-07-19 20:19:57 +00:00
avogar
5c16d6b553
Fix WriteBuffer finalize in destructor when cacnel query
2022-07-19 19:21:30 +00:00
Igor Nikonov
c74600d282
Merge branch 'master' into skipping_sorting_step
2022-07-19 18:59:36 +02:00
Igor Nikonov
1fe83cc8d8
optimize_sorting_for_input_stream setting and perf tests
2022-07-19 16:58:15 +00:00
Nikolai Kochetov
eaeb30a71a
Merge branch 'master' into use-dag-in-key-condition
2022-07-19 18:39:52 +02:00
Dmitry Novik
50989bdb68
Merge branch 'master' into group-by-use-nulls
2022-07-19 14:58:01 +02:00
avogar
4f020654be
Get rid of unneded ifdefs
2022-07-19 12:12:40 +00:00
Robert Schulze
81ef1099cc
Even less usage of StringRef
...
--> see #39300
2022-07-19 07:01:06 +00:00
avogar
6eb234a1cc
Avoid abort() in capnproto on exception descruction
2022-07-18 19:53:24 +00:00
Alexander Gololobov
9de72d995a
POC lightweight delete using __row_exists virtual column and prewhere-like filtering
2022-07-18 20:06:42 +02:00
Robert Schulze
32637cb1b9
Fix build
2022-07-18 07:58:59 +00:00
Robert Schulze
13482af4ee
First try at reducing the use of StringRef
...
- to be replaced by std::string_view
- suggested in #39262
2022-07-17 17:26:02 +00:00
Robert Schulze
fffeab2e86
Merge pull request #39262 from ClickHouse/stringref-by-value
...
Pass const StringRef by value, not by reference
2022-07-17 16:53:15 +02:00
Igor Nikonov
508fd5616d
Merge remote-tracking branch 'origin/master' into skipping_sorting_step
2022-07-16 00:24:20 +00:00
Igor Nikonov
ef0ef9e03b
Merge pull request #39191 from ClickHouse/sort_transform_cleanup
...
Cleanup: done during #38719 (SortingStep: deduce way to sort based on input stream sort)
2022-07-16 01:53:58 +02:00
Igor Nikonov
e607d103cc
Just to trigger CI
2022-07-15 23:16:02 +00:00
Igor Nikonov
6f224b026a
Perf test. Code polishing
2022-07-15 21:54:57 +00:00
Igor Nikonov
d9b312f955
Self-review + revert test
2022-07-15 17:00:25 +00:00
Robert Schulze
deda29b46b
Pass const StringRef by value, not by reference
...
See #39224
2022-07-15 11:34:56 +00:00
Igor Nikonov
a3a1ccc520
Fix: SortMode::Chunk
2022-07-15 10:26:13 +00:00
jianmei zhang
9d27af7ee2
For some columns mutations, skip to apply deleted mask when read some columns. Also add unit test case
2022-07-15 12:32:41 +08:00
Igor Nikonov
8170f4e33a
Merge branch 'master' into skipping_sorting_step
2022-07-14 23:05:45 +02:00
Igor Nikonov
1efdb4e3e5
Disable finish sort with sorted chunks
2022-07-14 21:02:44 +00:00
Igor Nikonov
7cd12393c2
If sorting type is specified then use it. Otherwise rely on sort description
2022-07-14 16:26:25 +00:00
Igor Nikonov
1d49adad20
Introduce Auto mode for sorting step (replace others for now)
2022-07-14 13:29:39 +00:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache
2022-07-14 12:29:54 +02:00
Ilya Yatsishin
5c8b9e4a14
Merge pull request #39170 from qoega/otel-trace-processors-setting
...
Add setting opentelemetry_trace_processors
2022-07-14 11:27:05 +02:00
Vladimir C
66f2ea5ebb
Merge pull request #39123 from vdimir/fjs-pushdown
...
Pushdown filter to the right side of sorting join
2022-07-14 09:32:19 +02:00
Igor Nikonov
b7f46d954e
Cleanup: related to #38719 (SortingStep: deduce way to sort based on input stream sort)
2022-07-13 17:57:37 +00:00
Igor Nikonov
b73aca2a3b
Merge branch 'master' into skipping_sorting_step
2022-07-13 19:06:41 +02:00
Igor Nikonov
159c9428bd
clean up
2022-07-13 17:05:54 +00:00
Igor Nikonov
f0d547993a
Fix: 01655_plan_optimizations_optimize_read_in_window_order_long
...
basically I returned code. Both plans with Finish sorting, need to check
sorting prefix
2022-07-13 16:51:51 +00:00
avogar
7cde9d3b40
Add new features in schema inference
2022-07-13 15:57:55 +00:00
Vladimir C
27ec36ca91
Merge pull request #37648 from vdimir/enable_parallel_hash
...
Bugfixes and performance improvements for parallel_hash
2022-07-13 14:52:57 +02:00
vdimir
4124dc9ac4
Rewrite tryPushDownFilter for join with lambda
2022-07-13 12:06:29 +00:00
vdimir
549a85fee9
Throw logical error on child idx mismatch in tryAddNewFilterStep
2022-07-13 11:53:46 +00:00
vdimir
fa59133463
Do not spam log in MergeJoinAlgorithm
2022-07-13 11:51:11 +00:00
Nikolai Kochetov
4f8cc871eb
Merge pull request #39132 from ClickHouse/fix-data-race-in-completed-pipeline-executor
...
Fix data race in CompletedPipelineExecutor.
2022-07-13 13:39:44 +02:00
vdimir
bddf6c1b32
Pushdown filter to the right side of sorting join
2022-07-13 11:36:25 +00:00
Yatsishin Ilya
d886aea351
Add setting opentelemetry_trace_processors
2022-07-13 11:31:24 +00:00
Vladimir C
d1d1e4d8a1
Merge pull request #38943 from amosbird/better-join-plan1
...
Avoid redundant join block transformation during planning.
2022-07-13 12:39:45 +02:00
Igor Nikonov
1d6f699a12
Use sort mode Port for reading in order
2022-07-12 21:56:00 +00:00
Dmitry Novik
5f65b45269
Merge branch 'master' into group-by-use-nulls
2022-07-12 22:36:04 +02:00
vdimir
63aebd17b2
Remove TabSeparatedSorted
2022-07-12 20:22:35 +02:00
vdimir
46df417c2e
Fix empty line sorting in TabSeparatedSorted
2022-07-12 20:22:35 +02:00
vdimir
f51b25b262
clickhouse test ignore order via special format
2022-07-12 20:22:35 +02:00
Nikolai Kochetov
ec24f730b1
Merge pull request #39125 from ClickHouse/fix-insert-into-mv-with-extremes-enabled
...
Fix insert into MV with enabled extremes.
2022-07-12 20:19:41 +02:00
Nikolai Kochetov
93bf22f585
Fix data race in CompletedPipelineExecutor.
2022-07-12 17:59:31 +00:00
Amos Bird
982e1a73d3
Better
2022-07-12 22:21:46 +08:00
Amos Bird
d3709c6c26
Avoid redundant join block transformation.
2022-07-12 22:20:10 +08:00
Amos Bird
b9d9ca5194
style fix
2022-07-12 22:20:08 +08:00
Nikolai Kochetov
75c3926cbb
Fix insert into MV with enabled extremes.
2022-07-12 13:57:36 +00:00
Dmitry Novik
aabf5123d6
Fixup
2022-07-12 13:46:06 +00:00
Igor Nikonov
a1f36d5536
Merge branch 'master' into skipping_sorting_step
2022-07-12 15:15:14 +02:00
Igor Nikonov
2c8d9080bd
Fix: consider collation in column sort description comparison
2022-07-12 13:14:10 +00:00
Dmitry Novik
cfca3db884
Fix crash with totals
2022-07-12 12:15:43 +00:00
mergify[bot]
48688da835
Merge branch 'master' into concurrency-control
2022-07-12 10:57:13 +00:00
Igor Nikonov
ea5e7793b2
Fix: self-review comments
2022-07-11 21:26:39 +00:00
Igor Nikonov
e0776b1c82
Fix: test for optimize read in window order
...
+ code polishing
2022-07-11 20:59:38 +00:00
Igor Nikonov
0ca8166ab2
Fix: forgot to return sorting type in constructors
2022-07-11 20:59:38 +00:00
Igor Nikonov
47bed7e318
Try to choose sorting transform based on sort description with fallback
2022-07-11 20:59:38 +00:00
Igor Nikonov
53272b2f08
Code polishing
2022-07-11 20:59:38 +00:00
Igor Nikonov
2a7e3bd741
Fix + SortMode::None as default value
2022-07-11 20:59:38 +00:00
Igor Nikonov
16d2319a8d
SortingStep: type of sorting is deduced based on input stream sorting description in during transformation
...
+ perf test
2022-07-11 20:59:38 +00:00
Igor Nikonov
7d4d92bd61
In case full sort was wrong choise during plan interpretation
2022-07-11 20:59:38 +00:00
Igor Nikonov
67ce421e38
Skip sorting step if input stream is globally sorted
2022-07-11 20:59:38 +00:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_"
...
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.
As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.
In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).
(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
2022-07-11 19:36:18 +02:00
vdimir
da523f3288
Fix assertion in full soring merge join
2022-07-08 11:31:15 +00:00
Dmitry Novik
d1df66687b
Merge branch 'master' into group-by-use-nulls
2022-07-07 20:54:38 +02:00
Dmitry Novik
1587385f7a
Cleanup code
2022-07-07 18:53:20 +00:00
Robert Schulze
f15d9ca59c
Merge pull request #38774 from zvonand/zvonand-nnd
...
Reintroduce nonNegativeDerivative()
2022-07-07 20:39:13 +02:00
vdimir
7c586a9e7c
Minor updates for full soring merge join
2022-07-06 14:28:05 +00:00
vdimir
881d352e05
upd full sorting join
2022-07-06 14:28:05 +00:00
vdimir
aff6654d52
minor changes in full sort join
2022-07-06 14:27:33 +00:00
vdimir
f8e66601a7
Fix column remap in MergeJoinTransform
2022-07-06 14:27:32 +00:00
vdimir
0b994bb258
fix build
2022-07-06 14:27:32 +00:00
vdimir
753a567da8
full sorting join with using
2022-07-06 14:27:29 +00:00
vdimir
a90ac59ee5
MergeJoinAlgorithm::createBlockWithDefaults
2022-07-06 14:26:19 +00:00
vdimir
d184e184b4
full sort join: check key types, more tests
2022-07-06 14:26:19 +00:00
vdimir
a2a7abc2e9
add not implemented checks, add using testcase to full sort join
2022-07-06 14:26:18 +00:00
vdimir
92ff43eb7c
tests full sort join
2022-07-06 14:26:18 +00:00
vdimir
a0144e115d
full sorting all join
2022-07-06 14:26:18 +00:00
vdimir
4e88e8f5ec
full sort join: move block list to all join state
2022-07-06 14:26:17 +00:00
vdimir
94192a23fc
enable total compare in MergeJoinAlgorithm
2022-07-06 14:26:16 +00:00
vdimir
a92c60ba06
fix nulls comparsion in full sorting join
2022-07-06 14:26:15 +00:00
vdimir
7c5a5f4b64
full sorted any join tests passed
2022-07-06 14:26:15 +00:00
vdimir
26d812ec72
wip any full sorting merge, rewrite cursor
2022-07-06 14:26:14 +00:00
vdimir
a2d190edb8
wip MergeJoinTransform
2022-07-06 14:25:12 +00:00
vdimir
0b9d4ee640
wip sort join same rows
2022-07-06 14:25:12 +00:00
vdimir
6d198ff3d7
fix style
2022-07-06 14:25:11 +00:00
vdimir
88d8dc5be2
wip full sort any join
2022-07-06 14:25:11 +00:00
vdimir
ba787db0bb
Fix build, small changes
2022-07-06 14:25:10 +00:00
vdimir
d34a66c915
wip sorting merge
2022-07-06 14:25:09 +00:00
vdimir
1b429fc1af
wip: any left/right sorting join
2022-07-06 14:23:46 +00:00
vdimir
8dce97123c
wip: any inner full sorting join
2022-07-06 14:23:46 +00:00
vdimir
4a16195964
Calculate output header for full sorting merge join
2022-07-06 14:23:45 +00:00
vdimir
fa8eb35599
Pipeline for full sorting merge join
2022-07-06 14:23:44 +00:00
Maksim Kita
b94489d52c
Merge pull request #38859 from kitaisreal/merge-tree-merge-disable-batch-optimization
...
MergeTree merge disable batch optimization
2022-07-06 15:59:40 +02:00
Nikolai Kochetov
7de2f229ab
Merge pull request #38584 from ClickHouse/filimonov-AggregatingTransform-expandPipeline
...
Add check for empty proccessors in AggregatingTransform::expandPipeline
2022-07-06 14:38:40 +02:00
Andrey Zvonov
7de39d9b15
Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-nnd
2022-07-06 10:59:35 +03:00
Maksim Kita
bdc21737d5
MergeTree merge disable batch optimization
2022-07-05 16:15:00 +02:00
zvonand
8a270c01e9
fix floating point in intervals
2022-07-04 20:45:05 +03:00
Dmitry Novik
864ab20582
Use correct intermediate header for ROLLUP and CUBE
2022-07-04 16:17:58 +00:00
Kruglov Pavel
4080f055b6
Merge pull request #38477 from Avogar/sql-insert-format
...
Add SQLInsert output format
2022-07-04 15:06:33 +02:00
zvonand
f814985adf
minor improvements
2022-07-04 16:03:59 +03:00
zvonand
eac84351f6
fix behavior
2022-07-04 01:26:07 +03:00
Igor Nikonov
2e2ef08712
Merge pull request #37803 from ClickHouse/dictinct_in_order_optimization
...
DISTINCT in order optimization
2022-07-03 21:59:04 +02:00
Igor Nikonov
2577b59f4c
Merge pull request #38721 from ClickHouse/sort_description_4_read_from_merge_tree
...
Provide sort description for output stream in ReadFromMergeTree step
2022-07-03 18:06:25 +02:00
zvonand
8e99ea84a8
fix LOGICAL_ERROR
2022-07-02 14:09:51 +03:00
mergify[bot]
12f5250e86
Merge branch 'master' into dictinct_in_order_optimization
2022-07-01 22:51:35 +00:00
Igor Nikonov
9ef8ff5a31
Addressing review comments
2022-07-01 22:50:00 +00:00
Igor Nikonov
1d665e9576
Provide sort description for output stream in ReadFromMergeTree step
2022-07-01 21:28:01 +00:00
Anton Popov
ef87e1207c
better support of read_in_order in case of fixed prefix of sorting key
2022-07-01 16:45:01 +00:00
Dmitry Novik
81dd90893e
Merge remote-tracking branch 'origin/master' into group-by-use-nulls
2022-07-01 16:24:05 +00:00
Nikita Taranov
8ba3d405de
impl
2022-07-01 16:05:32 +02:00
zvonand
3b5332d15e
Revert "Revert "Non Negative Derivative window function""
...
This reverts commit dea3b5bfce
2022-07-01 18:59:07 +05:00
Alexey Milovidov
20841f0e1e
Merge pull request #38551 from ClickHouse/revert-37628-non-neg-deriv
...
Revert "Non Negative Derivative window function"
2022-07-01 02:46:28 +03:00
avogar
5b0fd31c64
Put column names in quotes
2022-06-30 16:14:30 +00:00
Dmitry Novik
33f601ec0a
Commit support use_nulls for GS
2022-06-30 15:14:26 +00:00
Igor Nikonov
488ee75fc4
+ use DistinctSorted for final distinct step
...
+ fix performance tests
2022-06-30 13:03:39 +00:00
Maksim Kita
0de66a2712
Merge pull request #38449 from ClickHouse/revert-38361-revert-38324-fix-partial-sort
...
Revert "Revert "Fix optimization in PartialSortingTransform (SIGSEGV and possible incorrect result)""
2022-06-30 13:02:38 +02:00
Dmitry Novik
98e9bc84d5
Refector ROLLUP and CUBE
2022-06-30 10:13:58 +00:00
Antonio Andelic
de264117fd
Merge pull request #38118 from bigo-sg/storagehive_struct_type
...
Add struct type support in `StorageHive`
2022-06-30 09:11:13 +02:00
mergify[bot]
4cbbfb431d
Merge branch 'master' into dictinct_in_order_optimization
2022-06-29 23:32:17 +00:00
Igor Nikonov
d435532c68
Adapt range search algorithm to high cardinality case
...
+ range search done in steps of some number of rows.
Controled by new
setting `distinct_in_order_range_search_step`. By default 0, i.e.
whole chunk
+ before start binary search, linear probing is done on each step (32
rows currently)
2022-06-29 23:30:35 +00:00
mergify[bot]
36139eacd7
Merge branch 'master' into dictinct_in_order_optimization
2022-06-29 13:37:16 +00:00
filimonov
e0acb6e337
Add check for empty proccessors in AggregatingTransform::expandPipeline
2022-06-29 15:23:53 +02:00
Igor Nikonov
3627c6ff36
Perf tests with high cardinality
2022-06-29 13:13:39 +00:00
mergify[bot]
9482c99ab8
Merge branch 'master' into sql-insert-format
2022-06-29 11:03:07 +00:00
mergify[bot]
26258959b1
Merge branch 'master' into distinct_sorted_small_refact
2022-06-29 09:38:34 +00:00
Alexey Milovidov
4cda5491f6
Merge pull request #38542 from ClickHouse/unique_lock-to-lock_guard
...
Don't use std::unique_lock unless we need to
2022-06-29 09:40:56 +03:00
Alexey Milovidov
dea3b5bfce
Revert "Non Negative Derivative window function"
2022-06-29 08:56:15 +03:00
Nikita Taranov
f5d26572df
Quick fix for aggregation pipeline ( #38295 )
2022-06-29 01:16:30 +02:00
Igor Nikonov
4a00e33e6b
Fixes for some review comments
2022-06-28 21:42:46 +00:00
Igor Nikonov
c1840e798c
Fix: wrong header variable was used
2022-06-28 20:15:16 +00:00
Igor Nikonov
d80a21a445
Distinct sorted: calculate column positions once in constructor
...
- instead of calculating them on every chunk
2022-06-28 19:59:05 +00:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to
...
Replace where possible by std::lock_guard which is more light-weight.
2022-06-28 19:19:06 +00:00
Igor Nikonov
59295724ac
Mark condition for empty chunk as unlikely
2022-06-27 20:44:39 +00:00
mergify[bot]
a9c1b68034
Merge branch 'master' into dictinct_in_order_optimization
2022-06-27 20:16:00 +00:00
Igor Nikonov
5a26349695
Fix: input chunk can have empty columns (no rows)
2022-06-27 19:51:06 +00:00
Dmitry Novik
1d15d72211
Support NULLs in ROLLUP
2022-06-27 18:42:26 +00:00
avogar
9bb68bc6de
Add SQLInsert output format
2022-06-27 18:31:57 +00:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
Nikita Taranov
2487ba7f00
Move updateInputStream to ITransformingStep ( #37393 )
2022-06-27 13:16:52 +02:00
Maksim Kita
3ebe6a03b1
Revert "Revert "Fix optimization in PartialSortingTransform (SIGSEGV and possible incorrect result)""
2022-06-27 10:37:19 +02:00
Igor Nikonov
edd29707ca
Some polishing
2022-06-26 21:44:10 +00:00
Igor Nikonov
68927dd60c
Adapt distinct for sorted chunks to handle sorted stream, so we can use
...
it for final distinct as well
2022-06-26 14:52:36 +00:00
Igor Nikonov
1140cf6fb5
Fixes:
...
+ test warning
+ proper capacity for column positions array in DistinctTransform
2022-06-26 09:43:31 +00:00
Igor Nikonov
04ce070da0
Remove unnecessary include
2022-06-24 23:11:52 +00:00
mergify[bot]
b65cf4e1fe
Merge branch 'master' into dictinct_in_order_optimization
2022-06-24 22:52:14 +00:00
Igor Nikonov
d5c6f5c18f
Fixes
...
+ flaky test with explain pipeline
+ consider sort direction from read order info in sort description
(ReadFromMergeTree step)
2022-06-24 22:49:27 +00:00
Alexander Tokmakov
3f4a09478d
Revert "Fix optimization in PartialSortingTransform (SIGSEGV and possible incorrect result)"
2022-06-23 23:01:11 +03:00
Igor Nikonov
2fd5467f36
Merge remote-tracking branch 'origin/master' into dictinct_in_order_optimization
2022-06-23 16:04:08 +00:00
mergify[bot]
b5d3fd50d2
Merge branch 'master' into dictinct_in_order_optimization
2022-06-23 09:48:38 +00:00
Igor Nikonov
944c247345
DISTINCT in order optimization
...
+ try use the optimization for final distinct in case of sorted stream
(sorting inside and among chunks)
+ sorting description contains only columns from sorting key which are in
header as well
2022-06-23 09:47:22 +00:00
lgbo-ustc
cd8e5c7c49
update headers
2022-06-23 17:43:54 +08:00
lgbo-ustc
96e6f9a2d0
fixed code style
2022-06-23 16:10:01 +08:00
lgbo-ustc
c1770c22b9
Merge remote-tracking branch 'ck/master' into storagehive_struct_type
2022-06-23 15:54:20 +08:00
Azat Khuzhin
9db64952c0
Fix SIGSEGV in optimization in PartialSortingTransform
...
Fixes : #37992 (cc @kitaisreal)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-22 21:39:10 +03:00
mergify[bot]
9688388be2
Merge branch 'master' into concurrency-control
2022-06-22 15:07:56 +00:00
Nikita Taranov
41ba0118b5
Bring back #36396 ( #38110 )
...
* Revert "Revert "More parallel execution for queries with `FINAL` (#36396 )""
This reverts commit 5bfb15262c
2022-06-22 15:05:07 +02:00
mergify[bot]
f45b4f56d8
Merge branch 'master' into dictinct_in_order_optimization
2022-06-21 21:25:37 +00:00
Igor Nikonov
b0a98bd875
DISTINCT in order optimization
...
+ use SortDescription from input data stream in DistinctStep to decide if the optimization is applicable
2022-06-21 21:23:49 +00:00
Nikolai Kochetov
b8d27aa8dd
Merge pull request #37469 from azat/projections-optimize_aggregation_in_order
...
Implement in order aggregation (optimize_aggregation_in_order) for projections for tables with fully materialized projections
2022-06-21 12:17:35 +02:00
Kseniia Sumarokova
71ee5dbbb2
Merge pull request #38227 from kssenii/diff-reduce
...
buffer's getFileSize small changes
2022-06-21 12:12:22 +02:00
Maksim Kita
cb018348cf
Merge pull request #38022 from kitaisreal/sorting-added-batch-queue-variants
...
Sorting added batch queue variants
2022-06-20 22:35:44 +02:00
Kseniia Sumarokova
e48ce50863
Update ArrowBufferedStreams.cpp
2022-06-20 19:12:51 +02:00
Sergei Trifonov
d6be2be5aa
fix single-thread mode for pulling and pushing executors to work properly
2022-06-20 19:04:24 +02:00
kssenii
5dd1bb2fd8
improvements for getFileSize
2022-06-20 15:22:56 +02:00
Roman Vasin
be565959fa
Fix Fatal errors caused by using spawnThreads()
2022-06-20 14:52:33 +02:00
Igor Nikonov
6ac68e8303
DISTINCT in order optimization
...
+ optimization for DISTINCT containing primary key columns
2022-06-20 10:06:15 +00:00
Vladimir Chebotarev
aef6fe6008
Rebase fix.
2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
92a553fb77
Build fix.
2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
6a363b7429
Build fix.
2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
d41c97ea1d
Review fixes.
2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
4f38e01343
Unused code.
2022-06-20 05:15:08 +03:00
