thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-12 09:22:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
99,804 Commits 9,297 Branches 1,782 Tags 2.1 GiB
85193ede42
Commit Graph

962 Commits

Author SHA1 Message Date
vdimir
4a16195964
Calculate output header for full sorting merge join 2022-07-06 14:23:45 +00:00
vdimir
fa8eb35599
Pipeline for full sorting merge join 2022-07-06 14:23:44 +00:00
Maksim Kita
b94489d52c
Merge pull request #38859 from kitaisreal/merge-tree-merge-disable-batch-optimization 
MergeTree merge disable batch optimization
 2022-07-06 15:59:40 +02:00
Nikolai Kochetov
7de2f229ab
Merge pull request #38584 from ClickHouse/filimonov-AggregatingTransform-expandPipeline 
Add check for empty proccessors in AggregatingTransform::expandPipeline
 2022-07-06 14:38:40 +02:00
Andrey Zvonov
7de39d9b15 Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-nnd 2022-07-06 10:59:35 +03:00
Maksim Kita
bdc21737d5 MergeTree merge disable batch optimization 2022-07-05 16:15:00 +02:00
zvonand
8a270c01e9 fix floating point in intervals 2022-07-04 20:45:05 +03:00
Dmitry Novik
864ab20582 Use correct intermediate header for ROLLUP and CUBE 2022-07-04 16:17:58 +00:00
zvonand
f814985adf minor improvements 2022-07-04 16:03:59 +03:00
zvonand
eac84351f6 fix behavior 2022-07-04 01:26:07 +03:00
Igor Nikonov
2e2ef08712
Merge pull request #37803 from ClickHouse/dictinct_in_order_optimization 
DISTINCT in order optimization
 2022-07-03 21:59:04 +02:00
zvonand
8e99ea84a8 fix LOGICAL_ERROR 2022-07-02 14:09:51 +03:00
mergify[bot]
12f5250e86
Merge branch 'master' into dictinct_in_order_optimization 2022-07-01 22:51:35 +00:00
Anton Popov
ef87e1207c better support of read_in_order in case of fixed prefix of sorting key 2022-07-01 16:45:01 +00:00
Dmitry Novik
81dd90893e Merge remote-tracking branch 'origin/master' into group-by-use-nulls 2022-07-01 16:24:05 +00:00
Nikita Taranov
8ba3d405de impl 2022-07-01 16:05:32 +02:00
zvonand
3b5332d15e Revert "Revert "Non Negative Derivative window function"" 
This reverts commit dea3b5bfce.
 2022-07-01 18:59:07 +05:00
Alexey Milovidov
20841f0e1e
Merge pull request #38551 from ClickHouse/revert-37628-non-neg-deriv 
Revert "Non Negative Derivative window function"
 2022-07-01 02:46:28 +03:00
Igor Nikonov
488ee75fc4 + use DistinctSorted for final distinct step 
+ fix performance tests
 2022-06-30 13:03:39 +00:00
Maksim Kita
0de66a2712
Merge pull request #38449 from ClickHouse/revert-38361-revert-38324-fix-partial-sort 
Revert "Revert "Fix optimization in PartialSortingTransform (SIGSEGV and possible incorrect result)""
 2022-06-30 13:02:38 +02:00
Dmitry Novik
98e9bc84d5 Refector ROLLUP and CUBE 2022-06-30 10:13:58 +00:00
mergify[bot]
4cbbfb431d
Merge branch 'master' into dictinct_in_order_optimization 2022-06-29 23:32:17 +00:00
Igor Nikonov
d435532c68 Adapt range search algorithm to high cardinality case 
+ range search done in steps of some number of rows.
  Controled by new
  setting `distinct_in_order_range_search_step`. By default 0, i.e.
  whole chunk
+ before start binary search, linear probing is done on each step (32
  rows currently)
 2022-06-29 23:30:35 +00:00
filimonov
e0acb6e337
Add check for empty proccessors in AggregatingTransform::expandPipeline 2022-06-29 15:23:53 +02:00
Igor Nikonov
3627c6ff36 Perf tests with high cardinality 2022-06-29 13:13:39 +00:00
mergify[bot]
26258959b1
Merge branch 'master' into distinct_sorted_small_refact 2022-06-29 09:38:34 +00:00
Alexey Milovidov
dea3b5bfce
Revert "Non Negative Derivative window function" 2022-06-29 08:56:15 +03:00
Igor Nikonov
4a00e33e6b Fixes for some review comments 2022-06-28 21:42:46 +00:00
Igor Nikonov
c1840e798c Fix: wrong header variable was used 2022-06-28 20:15:16 +00:00
Igor Nikonov
d80a21a445 Distinct sorted: calculate column positions once in constructor 
- instead of calculating them on every chunk
 2022-06-28 19:59:05 +00:00
Igor Nikonov
59295724ac Mark condition for empty chunk as unlikely 2022-06-27 20:44:39 +00:00
mergify[bot]
a9c1b68034
Merge branch 'master' into dictinct_in_order_optimization 2022-06-27 20:16:00 +00:00
Igor Nikonov
5a26349695 Fix: input chunk can have empty columns (no rows) 2022-06-27 19:51:06 +00:00
Dmitry Novik
1d15d72211 Support NULLs in ROLLUP 2022-06-27 18:42:26 +00:00
Nikita Taranov
2487ba7f00
Move updateInputStream to ITransformingStep (#37393) 2022-06-27 13:16:52 +02:00
Maksim Kita
3ebe6a03b1
Revert "Revert "Fix optimization in PartialSortingTransform (SIGSEGV and possible incorrect result)"" 2022-06-27 10:37:19 +02:00
Igor Nikonov
edd29707ca Some polishing 2022-06-26 21:44:10 +00:00
Igor Nikonov
68927dd60c Adapt distinct for sorted chunks to handle sorted stream, so we can use 
it for final distinct as well
 2022-06-26 14:52:36 +00:00
Igor Nikonov
1140cf6fb5 Fixes: 
+ test warning
+ proper capacity for column positions array in DistinctTransform
 2022-06-26 09:43:31 +00:00
mergify[bot]
b65cf4e1fe
Merge branch 'master' into dictinct_in_order_optimization 2022-06-24 22:52:14 +00:00
Alexander Tokmakov
3f4a09478d
Revert "Fix optimization in PartialSortingTransform (SIGSEGV and possible incorrect result)" 2022-06-23 23:01:11 +03:00
Igor Nikonov
2fd5467f36 Merge remote-tracking branch 'origin/master' into dictinct_in_order_optimization 2022-06-23 16:04:08 +00:00
mergify[bot]
b5d3fd50d2
Merge branch 'master' into dictinct_in_order_optimization 2022-06-23 09:48:38 +00:00
Igor Nikonov
944c247345 DISTINCT in order optimization 
+ try use the optimization for final distinct in case of sorted stream
  (sorting inside and among chunks)
+ sorting description contains only columns from sorting key which are in
  header as well
 2022-06-23 09:47:22 +00:00
Azat Khuzhin
9db64952c0 Fix SIGSEGV in optimization in PartialSortingTransform 
Fixes: #37992 (cc @kitaisreal)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-22 21:39:10 +03:00
Nikita Taranov
41ba0118b5
Bring back #36396 (#38110) 
* Revert "Revert "More parallel execution for queries with `FINAL` (#36396)""

This reverts commit 5bfb15262c.

* fix tests

* fix review suggestions

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-06-22 15:05:07 +02:00
mergify[bot]
f45b4f56d8
Merge branch 'master' into dictinct_in_order_optimization 2022-06-21 21:25:37 +00:00
Igor Nikonov
b0a98bd875 DISTINCT in order optimization 
+ use SortDescription from input data stream in DistinctStep to decide if the optimization is applicable
 2022-06-21 21:23:49 +00:00
Nikolai Kochetov
b8d27aa8dd
Merge pull request #37469 from azat/projections-optimize_aggregation_in_order 
Implement in order aggregation (optimize_aggregation_in_order) for projections for tables with fully materialized projections
 2022-06-21 12:17:35 +02:00
Igor Nikonov
6ac68e8303 DISTINCT in order optimization 
+ optimization for DISTINCT containing primary key columns
 2022-06-20 10:06:15 +00:00
Maksim Kita
c86191a6b9 SortingTransform add cursor order 2022-06-18 18:20:00 +02:00
Maksim Kita
3664f02690 Replaced SortingHeap with SortingQueue 2022-06-18 18:20:00 +02:00
Maksim Kita
9670504781 Fixed tests 2022-06-18 18:20:00 +02:00
Maksim Kita
d7e3e85d2f Sorting added batch queue variants 2022-06-18 18:20:00 +02:00
Alexander Gololobov
54e599161d Check row size to avoid out of bounds access 2022-06-18 00:16:45 +02:00
Maksim Kita
ed03bb5dc6
Merge pull request #38127 from iyupeng/2022061601.fix_merge_sorter 
Fix redundant memory reservation for output block in MergeSorter
 2022-06-17 17:50:26 +02:00
Maksim Kita
cd1bbc6a1b SortDescription compile fix typo 2022-06-16 16:40:02 +02:00
Maksim Kita
f8d5e908d7
Update SortingTransform.cpp 2022-06-16 16:39:34 +02:00
Yu, Peng
518a726c1d Fix redundant memory reservation for output block in MergeSorter 2022-06-16 15:54:30 +08:00
Azat Khuzhin
4694929623 Implement merging only for AggregatingStep 
v2: fill AggregateColumnsConstData only for only_merge
    (fixes 01291_aggregation_in_order and some other tests)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Maksim Kita
7d5af1a79f
Merge pull request #37992 from kitaisreal/partial-sorting-transform-refactoring 
PartialSortingTransform refactoring
 2022-06-15 22:55:04 +02:00
Alexander Tokmakov
5bfb15262c Revert "More parallel execution for queries with FINAL (#36396)" 
This reverts commit c8afeafe0e.
 2022-06-15 17:25:38 +03:00
Nikita Taranov
c8afeafe0e
More parallel execution for queries with FINAL (#36396) 2022-06-15 12:44:20 +02:00
Alexey Milovidov
ab9fc572d5
Merge pull request #37667 from ClickHouse/group-by-enum-fix 
Support types with non-standard defaults in ROLLUP, CUBE, GROUPING SETS
 2022-06-15 05:14:33 +03:00
Maksim Kita
20227e0718 Fixed style check 2022-06-13 18:49:20 +02:00
Maksim Kita
eb861499c7 Fixed tests 2022-06-13 16:43:11 +02:00
Maksim Kita
411e1ad227 Fixed tests 2022-06-13 12:30:38 +02:00
Robert Schulze
9ec56fa362
Merge pull request #37628 from zvonand/non-neg-deriv 
Non Negative Derivative window function
 2022-06-13 11:21:16 +02:00
Maksim Kita
d5b148792b
Merge pull request #37990 from kitaisreal/sorting-added-sort-queue-variants 
Sorting added sort queue variants
 2022-06-11 15:22:29 +02:00
Maksim Kita
f56b88e949 PartialSortingTransform refactoring 2022-06-10 18:34:45 +02:00
Maksim Kita
95a3a10769 Sorting added sort queue variants 2022-06-10 17:45:11 +02:00
zvonand
6d727b60ce merge from master 2022-06-10 17:51:17 +03:00
zvonand
5beb6e0c3b Merge branch 'master' of github.com:ClickHouse/ClickHouse into non-neg-deriv 2022-06-10 17:40:16 +03:00
zvonand
794dbe6b60 new stateless queries + minor fixes 2022-06-10 17:01:49 +03:00
mergify[bot]
cc712a14f3
Merge branch 'master' into destroy_aggregate_states_on_thread_pool 2022-06-08 12:15:28 +00:00
Alexander Tokmakov
51e27c22a7
Merge pull request #37902 from ClickHouse/revert-37647-Fix-all-CheckTriviallyCopyableMove-Errors 
Revert "Fix errors of CheckTriviallyCopyableMove type"
 2022-06-07 19:23:19 +03:00
Dmitry Novik
a3d70f27f4
Merge pull request #36944 from excitoon-favorites/better_exp_smooth 
Fix for exponential time decaying window functions
 2022-06-07 17:20:43 +02:00
Anton Popov
df6882d2b9
Revert "Fix errors of CheckTriviallyCopyableMove type" 2022-06-07 13:53:10 +02:00
mergify[bot]
8ff44bf48d
Merge branch 'master' into destroy_aggregate_states_on_thread_pool 2022-06-06 23:06:28 +00:00
Yakov Olkhovskiy
addefa2020 from or to statement can be absent 2022-06-06 16:36:03 -04:00
Nikita Taranov
edf5f5b234 check variant size also 2022-06-06 18:28:02 +00:00
mergify[bot]
58c3f31bf4
Merge branch 'master' into bug-with-fill-date 2022-06-06 15:10:33 +00:00
Nikita Taranov
2721502a7b do not spawn thread if only one variant 2022-06-06 08:30:03 +00:00
Yakov Olkhovskiy
52ae3f0e92 full refactoring - move type check to transform 2022-06-05 22:39:21 -04:00
Nikita Taranov
20bb8dff96 set thread name 2022-06-05 21:37:45 +00:00
Robert Schulze
2d87af2a15
Merge pull request #37647 from DevTeamBK/Fix-all-CheckTriviallyCopyableMove-Errors 
Fix errors of CheckTriviallyCopyableMove type
 2022-06-05 19:58:47 +02:00
Nikita Taranov
ac23e75da5 fix 2022-06-05 11:48:13 +00:00
Nikita Taranov
0a9d8398d8 impl 2022-06-04 19:14:38 +00:00
Vladimir Chebotarev
31e316dcb2 Review and build fixes. 2022-06-04 03:42:17 +03:00
Vladimir Chebotarev
c8cae1d9dc Added a test and fixed exponentialTimeDecayedMax. 2022-06-03 20:58:35 +03:00
Vladimir Chebotarev
e80c8e15e6 Fixed the bug. 2022-06-03 18:44:24 +03:00
Vladimir Chebotarev
c6736e9463 Draft. 2022-06-03 18:44:24 +03:00
Vladimir Chebotarev
ad1402d1eb Draft. 2022-06-03 18:44:24 +03:00
zvonand
615f37a488 fix identation 2022-06-03 14:16:00 +03:00
zvonand
5e4e349495 small fixes 2022-06-03 14:14:20 +03:00
HeenaBansal2009
e3080f2a97 Merge remote-tracking branch 'origin' into Fix-all-CheckTriviallyCopyableMove-Errors 2022-06-02 07:30:08 -07:00
Nikolai Kochetov
8770060fdf Skip progress for merging sources. 2022-06-02 09:15:17 +00:00
Nikolai Kochetov
86fbb74703 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-31 18:07:47 +00:00
zvonand
869486cc3b fix segfault(2) 2022-05-31 11:40:49 +03:00
Dmitry Novik
0e63583b8f Support types with non-standard defaults in ROLLUP, CUBE, GROUPING SETS 2022-05-31 00:11:10 +00:00
zvonand
8aebaa7194 fix segfault 2022-05-31 01:33:44 +03:00
Nikolai Kochetov
77b07dd0a8
Merge pull request #37163 from ClickHouse/grouping-function 
Add GROUPING function
 2022-05-30 20:45:04 +02:00
HeenaBansal2009
b7eb6bbd38 Fixed clang-tidy-CheckTriviallyCopyableMove-errors 2022-05-30 11:09:03 -07:00
Anton Popov
52d3791eb9
Merge pull request #37600 from CurtizJ/fix-with-fill-interval 
Fix `WITH FILL` with negative intervals in `STEP` clause
 2022-05-30 19:43:12 +02:00
Andrey Zvonov
55a9b99cb4 style fix(2) 2022-05-30 02:48:28 +03:00
Andrey Zvonov
2dbbf14de5
Merge branch 'master' into non-neg-deriv 2022-05-29 10:09:51 +03:00
zvonand
295a0f9ec2 added tests 2022-05-29 03:38:42 +03:00
zvonand
032e54abbf works now 2022-05-29 03:21:07 +03:00
Anton Popov
abc90fad8d fix WITH FILL with negative itervals 2022-05-27 12:42:51 +00:00
zvonand
5c558d0be9 old work upload 2022-05-27 15:07:22 +03:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
Nikolai Kochetov
3d84aae0ab Better. 2022-05-24 20:06:08 +00:00
Maksim Kita
bdc537ead3 Column compareImpl devirtualize compare call 2022-05-24 14:28:33 +02:00
ndchikin
4859108feb WindowTransform::moveRowNumber fix 2022-05-20 20:49:10 +00:00
Nikolai Kochetov
56feef01e7 Move some resources 2022-05-20 19:49:31 +00:00
Dmitry Novik
b3ccf96c81 Merge remote-tracking branch 'origin/master' into grouping-function 2022-05-19 17:58:33 +00:00
Kseniia Sumarokova
24d0899cb7
Merge pull request #36945 from Vxider/wv-populate 
POPULATE support for WindowView
 2022-05-19 13:47:52 +02:00
Azat Khuzhin
dea1706d4e
Fix GROUP BY AggregateFunction (#37093) 
* Fix GROUP BY AggregateFunction

finalizeChunk() was unconditionally converting AggregateFunction to the
underlying type, however this should be done only if the aggregate was
applied.

So pass names of aggregates as an argument to the finalizeChunk()

Fuzzer report [1]:

    Logical error: 'Bad cast from type DB::ColumnArray to DB::ColumnAggregateFunction'. Received signal 6 Received signal Aborted (6)

For the following query:

    SELECT
        arraySort(groupArrayArray(grp_simple)),
        grp_aggreg,
        arraySort(groupArrayArray(grp_simple)),
        b,
        arraySort(groupArrayArray(grp_simple)) AS grs
    FROM data_02294
    GROUP BY
        a,
        grp_aggreg,
        b
    SETTINGS optimize_aggregation_in_order = 1

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37050/323ae98202d80fc4b311be1e7308ef2ac39e6063/fuzzer_astfuzzerdebug,actions//fuzzer.log

v2: fix conflicts in src/Interpreters/InterpreterSelectQuery.cpp
v3: Fix header for GROUP BY AggregateFunction WITH TOTALS
v4: Add sanity check into finalizeBlock()
v5: Use typeid_cast<&> to get more sensible error in case of bad cast (as suggested by @nickitat)
v6: Fix positions passed to finalizeChunk()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Core/ColumnNumbers.h: remove unused <string>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Optimize finalizeChunk()/finalizeBlock()

v2: s/ByPosition/Mask/ s/by_position/mask/
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-18 23:37:43 +02:00
Dmitry Novik
e5b395e054 Support ROLLUP and CUBE in GROUPING function 2022-05-16 17:33:38 +00:00
zvonand
eaca4f2625 add basic logic 2022-05-14 22:17:14 +03:00
Vxider
f2ef27e5d6 rename windowview header 2022-05-14 09:21:54 +00:00
mergify[bot]
7ffad69019
Merge branch 'master' into wv-populate 2022-05-12 11:52:04 +00:00
Maksim Kita
437d70d4da Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
496302cb1d Fix tests 2022-05-11 21:59:51 +02:00
Maksim Kita
75555c436b Fix usage of min_count_to_compile_sort_description setting 2022-05-11 21:59:51 +02:00
Maksim Kita
06b21a5cc4 Fix build 2022-05-11 21:59:51 +02:00
Maksim Kita
4e7d10297b Fixed style 2022-05-11 21:59:51 +02:00
Maksim Kita
cbfb773b50 Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
8ceb63ee6c Added JIT compilation of SortDescription 2022-05-11 21:59:51 +02:00
Vxider
5d0a5d34c8 removing function populate in windowview 2022-05-10 15:06:58 +08:00
Anton Popov
e911900054 remove last mentions of data streams 2022-05-09 19:15:24 +00:00
Vxider
72a634b5c6 removing-replacing-window-column-transform 2022-05-07 08:41:51 +00:00
Robert Schulze
c48d9a4174
Merge pull request #36844 from rschu1ze/shared_ptr_helper3 
Remove inherited create() method + disallow copying of storages
 2022-05-04 19:16:30 +02:00
mergify[bot]
64084b5e32
Merge branch 'master' into shared_ptr_helper3 2022-05-03 20:46:16 +00:00
mergify[bot]
57fcca6dda
Merge branch 'master' into nth_value 2022-05-03 16:37:09 +00:00
Dmitry Novik
9be17ef50c
Merge pull request #35111 from azat/optimize_aggregation_in_order-prefix 
Implement partial GROUP BY key for optimize_aggregation_in_order
 2022-05-02 17:49:48 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Alexey Milovidov
b034146ba4
Merge pull request #36799 from azat/cleanup 
Tiny cleanup
 2022-05-01 14:23:13 +03:00
Azat Khuzhin
7a092e2a8c Remove unused AggregatedArenasChunkInfo 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 17:04:56 +03:00
Azat Khuzhin
b7b7d91bd1 Remove memory reservation for SquashingTransform 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 17:04:56 +03:00
Azat Khuzhin
8845fb0883 Fix outdated comment in buildPushingToViewsChain 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 17:04:56 +03:00
Azat Khuzhin
0ce44f3021 Optimize optimize_aggregation_in_order with a prefix key 
Before it does lots of extra work, now, it will be significantly more
optimal (thousands of rows -> 1-2 million of rows).

v2: s/executeOnBlockSimple/executeOnBlockSmall/
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 06:58:28 +03:00
Azat Khuzhin
3931dbd848 Implement partial GROUP BY key for optimize_aggregation_in_order 
Suppose you have a table with lots of rows, like:

    create table data_02233 (parent_key Int, child_key Int, value Int) engine=MergeTree() order by parent_key

And you want to do GROUP BY (parent_key, child_key) with optimize_aggregation_in_order:

    select parent_key, child_key, count() from data_02233 group by parent_key, child_key with totals order by parent_key, child_key

Right now, it is not possible, because optimize_aggregation_in_order
supports only w/o key aggregation, i.e. GROUP BY cannot be done inside
unique parent_key region.

v2: rebase on top SortDescriptionWithPositions
v3: disable two-level aggregation
v4: fix merging of aggregates
v5: improve tests coverage (add a test with multiple parts, to add merge processor)
v6: add a test for compiled aggregate functions (sum()) explicitly
v7: add missing sortBlock()
v8: remove group_by_description_optimized
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 06:58:07 +03:00
Azat Khuzhin
767acd53fb Add ability to pass range of rows to Aggregator 
v2: fix compiled aggregate functions (seek result to row_start)
v3: fix compiled aggregate functions (seek args to row_start)
v4: change signatures for JIT
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 06:57:55 +03:00
Azat Khuzhin
599a255741 AggregatingInOrderTransform: correctly invalidate variants 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 06:56:32 +03:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
ndchikin
4912e55cc4 nth_value bug fix 2022-04-25 03:34:08 +00:00
ndchikin
7bc0626e55 nth_value bug fix 2022-04-25 02:47:30 +00:00
Maksim Kita
57444fc7d3
Merge pull request #36444 from rschu1ze/clang-tidy-fixes 
Clang tidy fixes
 2022-04-21 16:11:27 +02:00
Kseniia Sumarokova
842879beea
Merge pull request #36272 from kssenii/mongo-fix 
Allow missing columns for mongo storage
 2022-04-20 17:54:35 +02:00
Kseniia Sumarokova
70100dbd8d
Update MongoDBSource.cpp 2022-04-20 13:18:11 +02:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings 
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
 2022-04-20 10:29:05 +02:00
mergify[bot]
d894aed7ae
Merge branch 'master' into mongo-fix 2022-04-19 09:32:51 +00:00
kssenii
f8f66dd23d Better 2022-04-19 10:59:47 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
Alexey Milovidov
f6ab2bd523
Merge pull request #36312 from ClickHouse/remove-arcadia 
Remove remaining parts of Arcadia
 2022-04-18 07:02:54 +03:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
Alexey Milovidov
cbeeb7ec4f Remove Arcadia 2022-04-16 00:20:47 +02:00
kssenii
d8e2d693e5 Allow missing columns for mongo storage 2022-04-14 19:31:07 +02:00
Yakov Olkhovskiy
90c4cd3de7
Merge branch 'master' into interpolate-feature 2022-04-05 14:39:07 -04:00
Nikita Taranov
bd89fcafdb
Make SortDescription::column_name always non-empty (#35805) 2022-04-04 14:17:15 +02:00
Anton Popov
687942ce70 more strict quota for written bytes 2022-04-01 15:02:49 +00:00
Yakov Olkhovskiy
538373a79b style fix 2022-03-31 12:13:49 -04:00
Yakov Olkhovskiy
a15996315e bugfix - columns order tracking 2022-03-31 11:51:13 -04:00
Yakov Olkhovskiy
b5682c1f02 minor refactoring 2022-03-31 08:33:50 -04:00
Yakov Olkhovskiy
6a1e116c46 refactoring 2022-03-30 16:34:19 -04:00
Anton Popov
caacc7d385 add quota for written bytes 2022-03-29 18:21:29 +00:00
Yakov Olkhovskiy
615efa1381 aliases processing fixed 2022-03-28 19:15:53 -04:00
Azat Khuzhin
bf4df5c6bb Fix SIGSEGV for build under gcc-11 (due to auto deduction) 
During building with gcc-11 you will got SIGSEGV for building
InterpretersMySQLDDLQuery (and some others), and it is due to endless
recursion:

    (gdb) bt 5
    0  0x00000000010978f2 in structural_comptypes (t1=0x7fde028c7dc8, t2=0x7fde028d1e70, strict=0) at ../../src/gcc/cp/typeck.c:1274
    1  0x00000000011c3f9d in comp_template_parms (parms1=<optimized out>, parms2=<optimized out>) at ../../src/gcc/cp/pt.c:3369
    2  0x0000000001097bd9 in structural_comptypes (t1=0x7fde028c7dc8, t2=0x7fde028d1e70, strict=<optimized out>) at ../../src/gcc/cp/typeck.c:1361
    3  0x00000000011c3f9d in comp_template_parms (parms1=<optimized out>, parms2=<optimized out>) at ../../src/gcc/cp/pt.c:3369
    4  0x0000000001097bd9 in structural_comptypes (t1=0x7fde028c7dc8, t2=0x7fde028d1e70, strict=<optimized out>) at ../../src/gcc/cp/typeck.c:1361
    (gdb) bt -X
    1397454 0x0000000001097bd9 in structural_comptypes (t1=0x7fde028d1540, t2=0x7fde028d27e0, strict=<optimized out>) at ../../src/gcc/cp/typeck.c:1361
    1397455 0x0000000000f2d8b5 in cp_tree_equal (t1=<optimized out>, t2=<optimized out>) at ../../src/gcc/cp/tree.c:4144
    1397456 0x00000000010909cb in template_args_equal (ot=0x7fde028cf578, nt=0x7fde028cfc58, partial_order=<optimized out>) at ../../src/gcc/cp/pt.c:9256
    1397457 0x0000000001090422 in template_args_equal (partial_order=false, nt=0x7fde028cfc58, ot=0x7fde028cf578) at ../../src/gcc/cp/pt.c:9295
    1397458 comp_template_args (oldargs=0x7fde028cf550, newargs=0x7fde028cfc30, oldarg_ptr=0x0, newarg_ptr=0x0, partial_order=false) at ../../src/gcc/cp/pt.c:9285
    1397459 0x00000000010a08f4 in spec_hasher::equal (e1=0x7fde028c95d0, e2=0x7ffd1194e8c0) at ../../src/gcc/cp/pt.c:1726
    1397460 0x0000000001085965 in hash_table<spec_hasher, false, xcallocator>::find_with_hash (this=0x7fde36b7f450, comparable=@0x7ffd1194e8b8: 0x7ffd1194e8c0, hash=<optimized out>) at ../../src/gcc/hash-table.h:936
    1397461 0x0000000001079698 in lookup_template_class_1 (d1=<optimized out>, arglist=0x7fde028cfc30, in_decl=0x0, context=<optimized out>, entering_scope=<optimized out>, complain=3) at ../../src/gcc/cp/pt.c:9896
    1397462 0x000000000109f8ef in lookup_template_class (complain=3, entering_scope=1, context=0x7fde27558e40, in_decl=0x0, arglist=0x7fde028cfc08, d1=0x7fde269bcd20) at ../../src/gcc/cp/pt.c:10251
    1397463 tsubst_aggr_type (t=0x7fde269bcd20, args=<optimized out>, complain=3, in_decl=0x0, entering_scope=1) at ../../src/gcc/cp/pt.c:13646
    1397464 0x000000000108f797 in tsubst (t=0x7fde269bcdc8, args=0x7fde028cf7a8, complain=3, in_decl=<optimized out>) at ../../src/gcc/cp/pt.c:16108
    1397465 0x0000000000e61bf2 in rewrite_template_parm (level=1, complain=3, tsubst_args=0x7fde028cf7a8, index=5, olddecl=0x7fde269b5600) at ../../src/gcc/cp/pt.c:28556
    1397466 rewrite_tparm_list(tree_node*, unsigned int, unsigned int, tree_node*, unsigned int, int) [clone .constprop.0] (oldelt=0x7fde269bd190, index=5, targs=0x7fde028cf7a8, targs_index=4, complain=3, level=1) at ../../src/gcc/cp/pt.c:28640
    1397467 0x00000000009f3748 in build_deduction_guide (type=type@entry=0x7fde26e13dc8, ctor=0x7fde269ac300, outer_args=outer_args@entry=0x0, complain=complain@entry=3) at ../../src/gcc/cp/pt.c:28769
    1397468 0x00000000009f444f in ctor_deduction_guides_for (complain=3, tmpl=<optimized out>) at ../../src/gcc/cp/cp-tree.h:842
    1397469 deduction_guides_for (tmpl=<optimized out>, any_dguides_p=<optimized out>, complain=3) at ../../src/gcc/cp/pt.c:29282
    1397470 0x00000000008507a8 in do_class_deduction (complain=3, flags=1, init=<optimized out>, tmpl=0x7fde26e0f980, ptype=0x7fde028c7b28) at ../../src/gcc/cp/pt.c:29402
    1397471 do_auto_deduction (type=0x7fde028c7b28, init=<optimized out>, auto_node=<optimized out>, complain=3, context=<optimized out>, outer_targs=<optimized out>, flags=1) at ../../src/gcc/cp/pt.c:29572
    1397472 0x00000000007c9569 in finish_compound_literal (type=<optimized out>, compound_literal=0x7fde028c95b8, complain=3, fcl_context=fcl_functional) at ../../src/gcc/cp/semantics.c:3060
    1397473 0x0000000001123a79 in cp_parser_functional_cast (parser=0x7fde27558da8, type=0x7fde028c7b28) at ../../src/gcc/cp/parser.c:30670
    1397474 0x0000000000fd7873 in cp_parser_postfix_expression (parser=0x7fde27558da8, address_p=<optimized out>, cast_p=<optimized out>, member_access_only_p=<optimized out>, decltype_p=false, pidk_return=0x0) at ../../src/gcc/cp/parser.c:7437
    1397475 0x0000000000fd4ddf in cp_parser_binary_expression (parser=0x7fde27558da8, cast_p=<optimized out>, no_toplevel_fold_p=false, decltype_p=<optimized out>, prec=PREC_NOT_OPERATOR, pidk=<optimized out>) at ../../src/gcc/cp/parser.c:9842
    1397476 0x0000000000fd4595 in cp_parser_assignment_expression (parser=0x7fde27558da8, pidk=<optimized out>, cast_p=<optimized out>, decltype_p=<optimized out>) at ../../src/gcc/cp/parser.c:10146
    1397477 0x0000000000fd3b90 in cp_parser_constant_expression (parser=0x7fde27558da8, allow_non_constant_p=2, non_constant_p=0x7ffd1194f1d7, strict_p=<optimized out>) at ../../src/gcc/cp/parser.c:10449
    1397478 0x0000000000fcfdd5 in cp_parser_initializer_clause (non_constant_p=<optimized out>, parser=0x7fde27558da8) at ../../src/gcc/cp/parser.c:24253
    1397479 cp_parser_initializer (parser=0x7fde27558da8, is_direct_init=<optimized out>, non_constant_p=<optimized out>, subexpression_p=<optimized out>) at ../../src/gcc/cp/parser.c:24193
    1397480 0x000000000062e5d8 in cp_parser_decomposition_declaration (init_loc=0x7ffd1194f1d8, maybe_range_for_decl=0x7ffd1194f498, decl_specifiers=0x7ffd1194f1f0, parser=0x7fde27558da8) at ../../src/gcc/cp/parser.c:14734
    1397481 cp_parser_simple_declaration (parser=0x7fde27558da8, function_definition_allowed_p=<optimized out>, maybe_range_for_decl=0x7ffd1194f498) at ../../src/gcc/cp/parser.c:14393
    1397482 0x000000000109b870 in cp_parser_init_statement (parser=0x7fde27558da8, decl=0x7ffd1194f498) at ../../src/gcc/cp/parser.c:13420
    1397483 0x00000000010996f0 in cp_parser_for (unroll=0, ivdep=false, parser=0x7fde27558da8) at ../../src/gcc/cp/parser.c:12708
    1397484 cp_parser_iteration_statement (parser=0x7fde27558da8, if_p=0x0, ivdep=<optimized out>, unroll=<optimized out>) at ../../src/gcc/cp/parser.c:13343
    1397485 0x0000000000fe5c46 in cp_parser_statement (parser=0x7fde27558da8, in_statement_expr=0x0, in_compound=<optimized out>, if_p=0x0, chain=0x0, loc_after_labels=0x0) at ../../src/gcc/cp/parser.c:11718
    1397486 0x0000000000fe15ac in cp_parser_statement_seq_opt (in_statement_expr=<optimized out>, parser=<optimized out>) at ../../src/gcc/cp/parser.c:12201
    1397487 cp_parser_compound_statement (parser=0x7fde27558da8, in_statement_expr=0x0, bcs_flags=<optimized out>, function_body=<optimized out>) at ../../src/gcc/cp/parser.c:12150

Interesting frame is 1397471, from which we can extract location:

    (gdb) p line_table[0].info_ordinary.maps[line_table[0].info_ordinary.cache]
    $54 = {
      <line_map> = {
        start_location = 1396581280
      },
      members of line_map_ordinary:
      reason = LC_RENAME,
      sysp = 0 '\000',
      m_column_and_range_bits = 8,
      m_range_bits = 0,
      to_file = 0x3eb4bb0 "/ch/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp",
      to_line = 46,
      included_from = 0
    }

By replicating SOURCE_LINE() macro from gcc-11 (libcpp/include/line-map.h):

    /* Converts a map and a location_t to source line.  */
    inline linenum_type
    SOURCE_LINE (const line_map_ordinary *ord_map, location_t loc)
    {
      return ((loc - ord_map->start_location)
          >> ord_map->m_column_and_range_bits) + ord_map->to_line;
    }

We got line 154:

    (gdb) p ((input_location-1396581280) >> 8) + 46
    $61 = 154

And this is:

    auto [column_name_and_type, declare_column_ast] = std::tuple{columns_name_and_type.begin(), columns_definition->children.begin()};

After rewriting it, everything works correctly.

Also here is a reproducer for gcc-11 (and no failures for gcc-12, but
gcc-12 has other issues, like [1] and one more for hash table):

    # cat /tmp/test.cpp
    #include <tuple>

    auto multi()
    {
            return std::tuple{1, 1};
    }
    double foo()
    {
            auto [a, b] = multi();
            return a - b;
    }

    # g++-11 -std=gnu++20 -c -o /dev/null -isystem /ch/contrib/libcxx/include -nostdinc++ /tmp/test.cpp
    g++-11: internal compiler error: Segmentation fault signal terminated program cc1plus
    Please submit a full bug report,
    with preprocessed source if appropriate.
    See <file:///usr/share/doc/gcc-11/README.Bugs> for instructions.
    # g++-12 -std=gnu++20 -c -o /dev/null -isystem /ch/contrib/libcxx/include -nostdinc++ /tmp/test.cpp

  [1]: https://reviews.llvm.org/D122598

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-28 22:21:45 +03:00
Yakov Olkhovskiy
5a4694f340 major refactoring, simplified, optimized, bugs fixed 2022-03-27 14:32:09 -04:00
Yakov Olkhovskiy
adefcfd299
Merge branch 'master' into interpolate-feature 2022-03-24 15:33:09 -04:00
Yakov Olkhovskiy
83f406b722 optimization, INTERPOLATE without expr. list, any column is allowed except WITH FILL 2022-03-24 15:29:29 -04:00
mergify[bot]
e11ef05c2b
Merge branch 'master' into issue_33147 2022-03-21 13:40:17 +00:00
Kruglov Pavel
ed7b40253c
Merge pull request #35039 from zzsmdfj/issue/#34890_fix_input_format_null_as_default_bug 
to #34890_fix_input_format_null_as_default_bug
 2022-03-21 12:42:17 +01:00
Yakov Olkhovskiy
481ee8aad5
Update FillingTransform.cpp 
use range-based for loop
 2022-03-19 12:17:56 -04:00
Yakov Olkhovskiy
eb7474e73a
Merge branch 'master' into interpolate-feature 2022-03-19 03:11:14 -04:00
Nikita Taranov
7d61fa5f05 impl 2022-03-18 22:58:35 +00:00
Yakov Olkhovskiy
a8e1671a76 type match check for INTERPOLATE expressions added, bugfix, printout fixed 2022-03-18 16:44:27 -04:00
mergify[bot]
28734562bb
Merge branch 'master' into issue/#34890_fix_input_format_null_as_default_bug 2022-03-17 13:24:34 +00:00
Yakov Olkhovskiy
00700988ee style fix 2022-03-17 02:31:01 -04:00
Yakov Olkhovskiy
7bb66e6702 added INTERPOLATE extension for ORDER BY WITH FILL 2022-03-17 01:51:35 -04:00
Anton Popov
0ba78c3c3a Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-16 15:28:09 +00:00
Maksim Kita
2665724301 Fix clang-tidy warnings in Parsers, Processors, QueryPipeline folders 2022-03-14 18:17:35 +00:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Maksim Kita
ad6b3693e1
Merge pull request #35123 from zhanghuajieHIT/fix_build_fail_with_gcc 
fix build fail with gcc
 2022-03-14 10:36:15 +01:00
Nikolai Kochetov
47f4bd30cd
Merge pull request #35186 from amosbird/fixwithtotalemptychunk 
Fix empty chunk in with total transform
 2022-03-11 10:24:19 +01:00
metahys
ff934cf0c2
Fix unexpected result when use -state type aggregate function in window frame (#34999) 
* Fix unexpected result when use -state type aggregate function in window frame

* fix style

* fix style

* fix test

* fix flaky test

* fix flaky test

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-03-11 11:54:17 +03:00
Amos Bird
a1b61dabfd
Fix empty chunk in with total transform. 2022-03-10 23:27:36 +08:00
mergify[bot]
c326ebd67f
Merge branch 'master' into issue/#34890_fix_input_format_null_as_default_bug 2022-03-09 15:59:04 +00:00
zhanghuajie
11dde7c127 fix build fail with gcc 2022-03-08 22:34:51 +08:00
zzsmdfj
7252c18ff0 to #34890_fix_input_format_null_as_default_bug 2022-03-04 15:04:43 +08:00
Anton Popov
df3b07fe7c Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-03 22:25:28 +00:00
Maksim Kita
b1a956c5f1 clang-tidy check performance-move-const-arg fix 2022-03-02 18:15:27 +00:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
zvonand
90c857c5e3 merge 2022-02-17 18:23:37 +03:00
zvonand
cf244689a2 fixed filling transform 2022-02-16 15:14:25 +03:00
Anton Popov
72e75fdaf5
Merge pull request #34601 from CurtizJ/filtering-by-sparse-columns 
Support filtering by sparse columns without conversion to full
 2022-02-15 23:26:13 +03:00
Anton Popov
7cddae1351 return back result_size_hint 2022-02-15 15:12:25 +03:00
Anton Popov
5c316ffabe support filtering by sparse columns without convertion to full 2022-02-15 14:30:54 +03:00
Kruglov Pavel
cf454a6539
Merge pull request #34532 from CurtizJ/fix-aggregation-in-order-3 
Fix aggregation in order with distributed_aggregation_memory_efficient=0
 2022-02-15 14:26:15 +03:00