thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-13 18:02:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
100,871 Commits 9,328 Branches 1,783 Tags 2.1 GiB
9106910de8
Commit Graph

3545 Commits

Author SHA1 Message Date
avogar
afc34dca41 Add new JSON formats, add improvements and refactoring 2022-09-01 19:00:24 +00:00
Kruglov Pavel
7a4a65bc36
Make better exception message in schema inference 2022-09-01 20:36:08 +02:00
Kruglov Pavel
f53aa86a20
Merge pull request #40485 from arthurpassos/fix-parquet-chunked-array-deserialization 
Add support for extended (chunked) arrays for Parquet format
 2022-09-01 19:40:40 +02:00
Dmitry Novik
ddadb362cf
Merge pull request #39762 from quickhouse/betterorderbyoptimization 
Fixed `Unknown identifier (aggregate-function)` exception which appears when a user tries to calculate WINDOW ORDER BY/PARTITION BY expressions over aggregate functions
 2022-09-01 18:08:06 +02:00
Frank Chen
9d63cbe811 Merge 'origin/master' into tracing_context_propagation to resolve conflicts 2022-09-01 23:18:59 +08:00
Vladimir C
12e6fc4182
Merge branch 'master' into join_and_sets 2022-09-01 14:56:14 +02:00
Kseniia Sumarokova
c6c67a248d
Merge pull request #40792 from canhld94/ch_canh_intersect_distinct 
Implement intersect + except distinct
 2022-09-01 14:35:26 +02:00
Anton Popov
f7bdf07adc
Merge pull request #38715 from CurtizJ/fix-read-in-order-fixed-prefix 
Better support of `optimize_read_in_order` in case of fixed prefix of sorting key
 2022-09-01 12:59:18 +02:00
Robert Schulze
de64c6b103
Merge branch 'master' into clang-tidy-for-headers 2022-09-01 10:24:56 +02:00
Kruglov Pavel
86516d3bb4
Merge pull request #40740 from amosbird/row-policy-index-fix-1 
Use index when row_policy_filter is always false
 2022-08-31 18:46:14 +02:00
Robert Schulze
cedf75ed5e
Enable clang-tidy for headers 
clang-tidy now also checks code in header files. Because the analyzer
finds tons of issues, activate the check only for directory "base/" (see
file ".clang-tidy"). All other directories, in particular "src/" are
left to future work.

While many findings were fixed, some were not (and suppressed instead).
Reasons for this include: a) the file is 1:1 copypaste of a 3rd-party
lib (e.g. pcg_extras.h) and fixing stuff would make upgrades/fixes more
difficult b) a fix would have broken lots of using code
 2022-08-31 10:48:15 +00:00
Anton Popov
3504781529
Merge branch 'master' into fix-read-in-order-fixed-prefix 2022-08-30 23:32:43 +02:00
Dmitry Novik
0a8378d9cd
Merge branch 'master' into betterorderbyoptimization 2022-08-30 14:23:22 +02:00
vdimir
0f6f3c73b0
Minor fix 2022-08-30 11:57:28 +00:00
Duc Canh Le
8590cc46c4 implement intersect + except distinct 2022-08-30 18:09:01 +08:00
Frank Chen
f17d56b528 Merge branch 'master' into tracing_context_propagation 2022-08-30 14:24:36 +08:00
vdimir
24f62e8486
Throw an error in CreatingSetsOnTheFlyTransform in case of input for finished 2022-08-29 11:27:08 +00:00
vdimir
b0e2616aa9
Style fixes in CreateSetAndFilterOnTheFlyTransform and related 2022-08-29 11:26:21 +00:00
Anton Popov
2a3e012931
Merge branch 'master' into fix-read-in-order-fixed-prefix 2022-08-29 13:17:26 +02:00
vdimir
7915b6948f
Fix build after rebase 2022-08-29 09:49:16 +00:00
vdimir
afb6b7d9cf
Test plan and pipeline for filtering step for join 2022-08-29 09:49:15 +00:00
vdimir
afeff512b5
Aux port for ReadHeadBalancedProcessor is empty Block 2022-08-29 09:49:14 +00:00
vdimir
95f87dc34e
fix sanitizer assert in CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:12 +00:00
vdimir
c67ab33d90
small fix CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:11 +00:00
vdimir
51e02d09f6
set preserves_sorting = true for CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:10 +00:00
vdimir
714c53ab24
fix typos 2022-08-29 09:49:09 +00:00
vdimir
8e1632f824
Create sets for joins: better code 2022-08-29 09:49:08 +00:00
vdimir
7228091ff1
rename CreateSetAndFilterOnTheFlyTransform 2022-08-29 09:49:07 +00:00
vdimir
67a9acc8db
rename CreatingSetOnTheFlyStep -> CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:07 +00:00
vdimir
d82a75ae75
cleanup PingPongProcessor 2022-08-29 09:49:06 +00:00
vdimir
e472e13c70
move PingPongProcessor/ReadHeadBalancedProceesor into separate file 2022-08-29 09:49:05 +00:00
vdimir
51a51694d6
Create sets for joins: better code 2022-08-29 09:49:01 +00:00
vdimir
c778bba13f
Create sets for joins: wip 2022-08-29 09:47:00 +00:00
vdimir
31a167848d
Fix set finish condition in CreatingSetsOnTheFlyTransform 2022-08-29 09:46:59 +00:00
vdimir
71708d595f
Create sets for joins: wip 2022-08-29 09:46:59 +00:00
vdimir
8f06430ebd
Create sets for joins: upd 2022-08-29 09:46:58 +00:00
vdimir
3292566603
Format bytes in CreatingSetsOnTheFlyTransform logs 2022-08-29 09:46:57 +00:00
vdimir
031aaf3a45
Add Creating/FilterBySetsOnTheFlyStep for full sorting join 2022-08-29 09:46:57 +00:00
vdimir
c5bc7b0a0c
Resize pipeline after full sort join 2022-08-29 09:46:56 +00:00
Azat Khuzhin
f9812d9917 Fix memory leak while pushing to MVs w/o query context (from Kafka/...) 
While pushign to MVs, there is a low-level code that create
ThreadGroupStatus/ThreadStatus, it is required to gather some metrics
for system.query_views_log.

But, one should not use ThreadGroupStatus of the MainThreadStatus, since
this structure can hold some state, that may not be cleaned, plus this
may be racy, instead it is better to create new ThreadGroupStatus and
attach it instead.

Also this place misses detachQuery(), and because of this it leaks
ThreadGroupStatus::finished_threads_counters_memory. But it is only the
problem pushing to MVs is done w/o query context (i.e. from Kafka/...),
since when it has query context detachQuery() will be called eventually.

Before this patch series, when I've tried the reproducer with
500 MVs attached to Kafka engine (that @den-crane suggested), jemalloc
report looks like this:

    $ ../jeprof --text ~/ch/tmp/upstream/clickhouse-binary --base jeprof.44384.0.i0.heap jeprof.44384.167.i167.heap
    Using local file /home/azat/ch/tmp/upstream/clickhouse-binary.
    Using local file jeprof.44384.167.i167.heap.
    Total: 915.6 MB
       910.7  99.5%  99.5%    910.7  99.5% Snapshot (inline)
         9.5   1.0% 100.5%      9.5   1.0% std::__1::__libcpp_operator_new (inline)
         0.5   0.1% 100.6%      0.5   0.1% DB::TasksStatsCounters::create

And with focus to this place:

    $ ../jeprof --focus Snapshot --text ~/ch/tmp/upstream/clickhouse-binary --base jeprof.44384.0.i0.heap jeprof.44384.167.i167.heap
    Using local file /home/azat/ch/tmp/upstream/clickhouse-binary.
    Using local file jeprof.44384.167.i167.heap.
    Total: 915.6 MB
       910.7 100.0% 100.0%    910.7 100.0% Snapshot (inline)
         0.0   0.0% 100.0%    910.7 100.0% DB::QueryPipeline::reset
         0.0   0.0% 100.0%    910.7 100.0% DB::StorageKafka::streamToViews
         0.0   0.0% 100.0%    910.7 100.0% DB::StorageKafka::threadFunc
         0.0   0.0% 100.0%    910.7 100.0% ProfileEvents::Counters::getPartiallyAtomicSnapshot
         0.0   0.0% 100.0%    910.7 100.0% ~ThreadStatus
         0.0   0.0% 100.0%    910.7 100.0% ~ViewRuntimeData
         0.0   0.0% 100.0%    910.7 100.0% ~ViewRuntimeStats (inline)

Actually this report does not looks great (you understand it because I
stripped it), because --text does not that smart, but if you will use
--pdf for the report you will see the stacktrace (will attach pdf to the
pull request).

But after this patch series the process RSS does not goes beyond
~700MiB.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-29 11:36:33 +02:00
Amos Bird
15a69bce84
Use index when row_policy_filter is always false 2022-08-29 16:44:32 +08:00
Alexey Milovidov
365a600fdb Merge branch 'force-documentation-3' of github.com:ClickHouse/ClickHouse into force-documentation-3 2022-08-27 22:28:54 +02:00
Alexey Milovidov
6b2e227c8b Fix integration test 2022-08-27 22:28:38 +02:00
Vladimir C
e067629e0d
Merge pull request #40239 from vdimir/vdimir/tmp-file-metrics 
More metrics for on-disk temporary data
 2022-08-26 11:28:01 +02:00
Alexander Gololobov
6a69e08799
Merge pull request #40559 from ClickHouse/lwd_vertical_merge_fix 
Fix vertical merge of parts with lightweight deleted rows
 2022-08-25 20:47:44 +02:00
Frank Chen
bb00dcc19b Remove using namespace from header 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:20:13 +08:00
Frank Chen
99c37ce6c6
Merge branch 'master' into tracing_context_propagation 2022-08-25 10:07:16 +08:00
Nikita Taranov
ac34a17551
Merge branch 'master' into control_block_size_in_aggregator 2022-08-24 20:25:28 +02:00
vdimir
91788f29e8
Upd TemporaryFileOnDisk 2022-08-24 16:15:54 +00:00
vdimir
7194df1184
Move back TemporaryFile -> TemporaryFileOnDisk 2022-08-24 16:14:11 +00:00
vdimir
0349c85017
Use getCompressedBytes in BufferingToFileTransform and TemporaryFileStream 2022-08-24 16:14:10 +00:00
vdimir
51c44424cc
More metrics for temp files 2022-08-24 16:14:09 +00:00
vdimir
1321ac87b5
Minor fixes 2022-08-24 16:14:07 +00:00
vdimir
7e0c9062c7
Add ProfileEvents::ExternalSort(Un)CompressedBytes 2022-08-24 16:14:07 +00:00
Kruglov Pavel
e6e7f5db93
Merge pull request #40491 from mini4/fix-settings-input_format_tsv_skip_first_lines 
Fix bug in settings input_format_tsv_skip_first_lines of format TSV
 2022-08-24 15:57:45 +02:00
Alexander Gololobov
1c2dd50ca5 Fix vertical merge of parts with lightweight deleted rows 2022-08-24 15:18:33 +02:00
Kruglov Pavel
0781e8b4f7
Merge pull request #40534 from Avogar/nested-in-avro 
Support reading Array(Record) into flatten nested table in Avro
 2022-08-24 13:33:12 +02:00
Frank Chen
cd19366b44 Move classes into DB::OpenTelemetry namespace 2022-08-24 16:41:40 +08:00
kgurjev
f62c2c3221 Fix bug in settings input_format_tsv_skip_first_lines of format TSV 2022-08-24 10:02:57 +03:00
Kruglov Pavel
72f02bd6eb
Merge pull request #40414 from Avogar/improve-schema-inference-cache 
Improve schema inference cache, respect format settings that can change the schema
 2022-08-23 17:04:58 +02:00
avogar
29a887578b Fix 2022-08-23 11:42:57 +00:00
avogar
581e569d04 Support reading Array(Record) into flatten nested table in Avro 2022-08-23 11:05:02 +00:00
Arthur Passos
f8e2ab0a20 Use FileReader::GetRecordBatchReader instead of FileReader::ReadRowGroup to parse Parquet 2022-08-22 08:21:32 -03:00
Alexey Milovidov
ab91c99495
Merge branch 'master' into control_block_size_in_aggregator 2022-08-20 21:28:27 +03:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
avogar
612ffaffde Make schema inference cache better, respect format settings that can change the schema 2022-08-19 16:39:13 +00:00
Nikita Taranov
1b6e7b9ca2
Merge branch 'master' into sort_mode_rename 2022-08-19 12:31:59 +02:00
Kruglov Pavel
b67cb9e378
Merge pull request #40173 from Avogar/arrow-dict 
Improve and fix dictionaries in Arrow format
 2022-08-18 20:54:55 +02:00
Kruglov Pavel
09a2ff8843
Merge pull request #40293 from joshuataylor/feature/arrow-large-binary-string 
Add support for LARGE_BINARY/LARGE_STRING with Arrow
 2022-08-18 14:01:58 +02:00
avogar
a6318cecd5 Fix hive test 2022-08-18 11:32:42 +00:00
Nikolai Kochetov
5a85531ef7
Merge pull request #38286 from Avogar/schema-inference-cache 
Add schema inference cache for s3/hdfs/file/url
 2022-08-18 13:07:50 +02:00
Kruglov Pavel
d7056376eb
Merge pull request #40068 from Avogar/schema-inference-hints 
Allow to specify structure hints in schema inference
 2022-08-18 12:19:45 +02:00
Igor Nikonov
6fe8b61345
Merge branch 'master' into sort_mode_rename 2022-08-17 19:19:29 +02:00
Yakov Olkhovskiy
40fd6e189a
call readColumnWithStringData 2022-08-17 09:54:01 -04:00
Kruglov Pavel
19af748737
Fix typo 2022-08-17 14:29:09 +02:00
Kruglov Pavel
00d04456ff
Try reduce code duplication 2022-08-17 14:28:15 +02:00
Vladimir C
b876cc17c9
Merge pull request #39593 from quickhouse/fixexponentialdecaywindowfunctions 
Fixed point of origin for exponential decay window functions to the last value in window
 2022-08-17 14:19:59 +02:00
Igor Nikonov
5ceaeb9e12 Sorting mode renaming 
+ sort mode -> sort scope
+ Stream -> Global
+ Port -> Stream
 2022-08-17 12:19:36 +00:00
avogar
8dd54c043d Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache 2022-08-17 11:47:40 +00:00
Igor Nikonov
46ed4f6cdf
Merge pull request #38719 from ClickHouse/skipping_sorting_step 
SortingStep: deduce way to sort based on input stream sort description
 2022-08-17 12:58:11 +02:00
Josh Taylor
628d2bbff5 Add support for LARGE_BINARY/LARGE_STRING with Arrow 2022-08-17 10:25:06 +08:00
Nikita Taranov
6bdbaccc37 use max_block_size from settings 2022-08-16 18:56:22 +02:00
Nikita Taranov
63bc894a42 more parallelism 2022-08-16 18:56:22 +02:00
Nikita Taranov
f650b23ee3 generate many blocks 2022-08-16 18:56:22 +02:00
Nikita Taranov
db0110fd7a more accurate crutch 2022-08-16 18:56:22 +02:00
Nikita Taranov
e5e0a24ab3 return chunks from prepareBlockAndFillWithoutKey 2022-08-16 18:56:22 +02:00
Igor Nikonov
d4367de7bb Rename setting to optimize_sorting_by_input_stream_properties 2022-08-16 16:27:41 +00:00
Vladimir Chebotaryov
3cc03b141e Fixed tests on Debug build type. 2022-08-16 15:43:37 +02:00
Vladimir Chebotaryov
66f9bfca61 Fixed point of origin for exponential decay window functions to the last value in window. 2022-08-16 15:43:37 +02:00
avogar
99d8727335 Fix tests 2022-08-16 12:56:51 +00:00
avogar
936c457734 Remove unnended field 2022-08-16 09:51:52 +00:00
avogar
e1ff996ec3 Allow to specify structure hints in schema inference 2022-08-16 09:46:57 +00:00
Maksim Kita
110470809b
Merge pull request #40121 from amosbird/profile-processor-1 
Extend processors_profile_log with more info
 2022-08-16 09:49:12 +02:00
Igor Nikonov
aba00952f5 Fix: don't set sort mode in ReadFromMergeTree if sort description empty 2022-08-15 20:58:20 +00:00
Kruglov Pavel
2c5c0d6d47
Fix typo 2022-08-15 19:55:28 +02:00
avogar
ca0d883c0f Fix possible segfault in CapnProto input format 2022-08-15 15:36:18 +00:00
Igor Nikonov
ea10fd65b8 Sorting properties in EXPLAIN PLAN 
~ change formatting for sorting
~ rename sortmode option -> sorting
 2022-08-15 15:14:59 +00:00
avogar
c160033837 Fix 2022-08-15 11:38:28 +00:00
Igor Nikonov
d83bea626c Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-08-13 21:46:34 +00:00
Igor Nikonov
f33a0d8c85 More simple way to check if sorting order is preserved 
- there is a case where it's done wrong
 2022-08-12 23:42:37 +00:00
avogar
78e197063c Better example 2022-08-12 19:08:36 +00:00
avogar
763f84b623 Remove bad comment 2022-08-12 19:05:57 +00:00
avogar
9addded80e Remove logging 2022-08-12 19:01:02 +00:00
avogar
000336622a Remove logging 2022-08-12 18:59:52 +00:00
avogar
398576e9c9 Improve and fix dictionaries in Arrow format 2022-08-12 18:56:21 +00:00
Kseniia Sumarokova
a6cfc7bc3b
Merge pull request #34651 from alexX512/master 
New caching strategies
 2022-08-12 17:23:37 +02:00
Anton Popov
4bd50bb06c
Merge branch 'master' into distinct_sorted_simplify 2022-08-12 17:11:18 +02:00
Kruglov Pavel
4c7222d938
Merge pull request #40020 from canhld94/ch_canh_fix_hash 
fix HashMethodOneNumber with const column
 2022-08-12 14:40:24 +02:00
Amos Bird
99a38e41aa
processor profile 2022-08-11 21:03:34 +08:00
Igor Nikonov
75f6fcfa70 Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-08-11 12:35:55 +00:00
Amos Bird
fa8fab2e8f
Fix KeyCondition with other filters 2022-08-11 19:20:44 +08:00
Maksim Kita
6bec0f5854
Merge pull request #38956 from vdimir/dict-join-refactoring 
Join with dictionary refactoring
 2022-08-11 11:54:11 +02:00
Vladimir C
2d44e6c458
Merge pull request #39343 from vdimir/refactor-prepared-sets 
Refactor PreparedSets/SubqueryForSet
 2022-08-11 11:19:18 +02:00
Vladimir Chebotaryov
748979a9c0
Merge branch 'master' into betterorderbyoptimization 2022-08-11 11:09:52 +03:00
Duc Canh Le
84cd867aa8 materialize column instead of handling column in hash method 2022-08-11 10:46:06 +08:00
Anton Popov
3fdf428834
Merge pull request #39186 from Avogar/numbers-schema-inference 
Add new features in schema inference
 2022-08-11 00:53:54 +02:00
vdimir
ad91c16ba0
Rename join_common -> JoinUtils 2022-08-10 14:20:28 +00:00
vdimir
b7c5c54181
Fix build 2022-08-10 13:43:55 +00:00
vdimir
5eb4cd39e0
Merge branch 'master' into refactor-prepared-sets 2022-08-10 11:47:49 +00:00
Maksim Kita
aff8149f5c
Merge pull request #39998 from kitaisreal/actions-dag-refactoring 
ActionsDAG rename index to outputs
 2022-08-10 11:44:18 +02:00
Igor Nikonov
754a9fb096 Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-08-09 22:20:17 +00:00
Arthur Passos
c4d8ad2222 Add docs 2022-08-09 15:58:46 -03:00
Arthur Passos
e724e7bef6 Update arrow dict to lc comment 2022-08-09 15:52:37 -03:00
Arthur Passos
6eb89fd780 Fix both arrow dict de-serialization and dict of nullable de-serialization 2022-08-09 15:06:22 -03:00
Arthur Passos
be1e32c3f1
Merge branch 'ClickHouse:master' into fix_arrow_column_dictionary_to_ch_lc 2022-08-09 15:04:06 -03:00
Maksim Kita
acbfcf440b
Merge branch 'master' into actions-dag-refactoring 2022-08-09 18:52:08 +02:00
Igor Nikonov
70b52f7cb9 Fix test, review comments 2022-08-09 16:29:56 +00:00
Maksim Kita
a576a55375 Fixed build 2022-08-09 15:03:59 +02:00
Kruglov Pavel
088e8cf9bd
Merge branch 'master' into numbers-schema-inference 2022-08-09 14:00:36 +02:00
Kruglov Pavel
99b9e85a8f
Merge pull request #39646 from Avogar/more-formats 
Add more Pretty formats
 2022-08-09 13:59:47 +02:00
Igor Nikonov
366ead3828 Consider aliases when checking if sorting order is preserved by 
expression
 2022-08-09 11:27:17 +00:00
Igor Nikonov
1439664df6 EXPLAIN tests 2022-08-08 20:46:43 +00:00
Maksim Kita
c030fd05e7 ActionsDAG rename index to outputs 2022-08-08 18:01:32 +02:00
Igor Nikonov
15bdeba074 Fix review comments 2022-08-08 14:43:03 +00:00
avogar
1304e3487c Add comments, remove unneded stuff 2022-08-08 13:43:14 +00:00
Igor Nikonov
86c5280db3 Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-08-08 12:43:06 +00:00
Igor Nikonov
6572a447fa Check if sorting order is preserved for FilterStep 2022-08-08 12:42:10 +00:00
avogar
2f95726b06 Fix comments 2022-08-08 12:41:00 +00:00
vdimir
708747ca0b
Merge branch 'master' into refactor-prepared-sets 2022-08-08 14:27:18 +02:00
Vladimir C
6bd4821656
Merge pull request #39575 from vdimir/join_pushdown_column_not_found 
Fix column not found for push down with join
 2022-08-08 14:04:42 +02:00
alexX512
6bf29cb610 Change class LRUCache to class CachBase. Check running CacheBase with default pcahce policy SLRU 2022-08-07 19:59:30 +00:00
Igor Nikonov
254d14b1d3 Remove debug trace from DistinctStep 
+ this information can be seen later via EXPLAIN PLAN
 2022-08-07 13:12:11 +00:00
Igor Nikonov
f52720fedc Try to fix style errors 2022-08-06 23:33:19 +00:00
Igor Nikonov
988ce29e9e Check if expression step violates sorting correctly 2022-08-06 22:37:14 +00:00
Igor Nikonov
8278da6475 Fix: read row counts before move columns out of chunk 2022-08-05 21:29:57 +00:00
Igor Nikonov
9fddf6efde Merge remote-tracking branch 'origin/master' into ordinary_distinct_small_refact 2022-08-05 19:23:44 +00:00
avogar
9b1a267203 Refactor, remove TTL, add size limit, add system table and system query 2022-08-05 16:20:15 +00:00
Igor Nikonov
f5f6ada69b ActionsDAG check is not dependent on sort description 2022-08-05 11:00:18 +00:00
Nikolai Kochetov
cf34232675 Output header is now empty for every MV chain. 
Instead of checking that number of processors different for different
threads, simply always return empty header from buildChainImpl(), by
adding explicit conversion.

v2: ignore UNKNOWN_TABLE errors in test
 2022-08-05 13:16:32 +03:00
Igor Nikonov
246f7b71b6
Merge branch 'master' into skipping_sorting_step 2022-08-05 00:03:54 +02:00
Igor Nikonov
a70c47f780 EXPLAIN PLAN sortmode options 
+ ActionsDAG fixes
 2022-08-04 22:02:53 +00:00
Igor Nikonov
0a659f5ab8 Analyze ActionsDAG in ExpressionStep 
+ check if sorting order can be kept from prevous step
+ EXPLAIN PLAN header=1 contains Sort Mode and Sort Description for each step
+ some tests
+ fix some review comments
 2022-08-04 19:30:25 +00:00
Arthur Passos
62d48053c0 Use insertDefault instead of insert(0) 2022-08-04 15:53:44 -03:00
Arthur Passos
c307e9a228 Fix ArrowColumn dictionary to CH low cardinality conversion 2022-08-04 15:34:44 -03:00
Nikolai Kochetov
658a269d56
Merge branch 'master' into use-dag-in-key-condition 2022-08-04 16:18:40 +02:00
Vladimir C
a627b00c43
Merge branch 'master' into refactor-prepared-sets 2022-08-04 13:27:38 +02:00
Kruglov Pavel
235649cb98
Merge pull request #39458 from Avogar/fix-cancel-insert-into-function 
Fix WriteBuffer finalize when cancel insert into function
 2022-08-04 13:02:08 +02:00
vdimir
298fb2431d
Fix column not found for push down with join 2022-08-04 10:58:21 +00:00
Nikita Taranov
4943202921
Improve memory usage during memory efficient merging of aggregation results (#39429) 2022-08-03 17:56:59 +02:00
Vladimir Chebotaryov
484d8f7028
Merge branch 'master' into betterorderbyoptimization 2022-08-03 08:14:50 +03:00
Igor Nikonov
8eafca65c9 Fix style 2022-08-02 17:58:31 +00:00
Igor Nikonov
30782a2b05 Test: distinct sorted is not used on const column 2022-08-02 17:44:43 +00:00
Kruglov Pavel
6b2186bfeb
Merge branch 'master' into numbers-schema-inference 2022-08-02 19:34:53 +02:00
Igor Nikonov
56697125d3 Merge remote-tracking branch 'origin/master' into distinct_sorted_simplify 2022-08-02 11:18:46 +00:00
Igor Nikonov
4bf42ce867 Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-08-02 10:50:25 +00:00
Igor Nikonov
0cb59aa38c ExpressionStep: simple check if expression changes sorting order 2022-08-02 10:45:05 +00:00
Kruglov Pavel
42136b7630
Merge pull request #39647 from Avogar/fix-arrow-strings 
Fix strings in dictionary in Arrow format
 2022-08-01 12:46:07 +02:00
Vladimir Chebotaryov
aaa6049678
Merge branch 'master' into betterorderbyoptimization 2022-08-01 11:42:48 +03:00
Vladimir Chebotaryov
9fc3addea9 Fixed using column_after_join for handling WINDOW expressions in ExpressionAnalyzer, shall be aggregated_columns. 2022-07-31 23:36:20 +03:00
Nikolai Kochetov
22fbfe19a4 Merge branch 'master' into use-dag-in-key-condition 2022-07-31 21:54:12 +02:00
Igor Nikonov
7f0adb5eb0 Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-07-31 07:07:36 +00:00
Igor Nikonov
83e1dd1172
Merge branch 'master' into ordinary_distinct_small_refact 2022-07-31 00:23:21 +02:00
Alexey Milovidov
4828be7fc4 Fix double escaping in the metadata of FORMAT JSON 2022-07-30 23:56:41 +02:00
Igor Nikonov
7245ddcc20 Simple refactoring: ordinary DISTINCT implementation 2022-07-30 20:25:56 +00:00
Igor Nikonov
a7cfad105e
Merge branch 'master' into distinct_sorted_simplify 2022-07-30 21:57:53 +02:00
Igor Nikonov
3be51a6dea Construct DistinctSortedTransform only when it makes sense 
otherwise fallback to DistinctTransform (i.e. ordinary distinct)
 2022-07-30 19:41:03 +00:00
Maksim Kita
acb0137dbb
Merge pull request #39718 from kitaisreal/join-enums-refactoring 
Join enums refactoring
 2022-07-30 13:53:08 +02:00
Igor Nikonov
d951154ef4 Proved NULLs direction when compare rows 2022-07-29 22:12:03 +00:00
Igor Nikonov
13dc1697fb Remove unnecessary initialization 2022-07-29 20:34:23 +00:00
Igor Nikonov
b44373ba8f Merge remote-tracking branch 'origin/master' into distinct_sorted_simplify 2022-07-29 20:33:26 +00:00
Igor Nikonov
7b0b38e997 DistinctSortedTransform works only if columns contains sort prefix of 
sort description
 2022-07-29 20:01:07 +00:00
Kruglov Pavel
ccd1e1bdb8
Merge branch 'master' into fix-cancel-insert-into-function 2022-07-29 20:27:32 +02:00
Maksim Kita
8fc6bad4f4 Join enums refactoring 2022-07-29 18:35:05 +02:00
Nikolai Kochetov
59a11b32ad
Merge branch 'master' into use-dag-in-key-condition 2022-07-29 17:01:33 +02:00
Igor Nikonov
fb5799fe7f More generic getInputOrderInfo() in query info 2022-07-29 14:36:36 +00:00
Igor Nikonov
4af435bdda Fix: handle all const columns case correctly 2022-07-28 21:22:06 +00:00
Igor Nikonov
f414af2348
Merge pull request #39538 from ClickHouse/fix_distinct_sorted 
Fix: DistinctSortedTransform doesn't take advantage of sorting
 2022-07-28 21:47:10 +02:00
Sergei Trifonov
edf6b02723
Merge pull request #37558 from ClickHouse/concurrency-control 
add concurrency control for pipeline executors
 2022-07-28 08:56:24 +02:00
Vladimir C
115506356c
Merge branch 'master' into refactor-prepared-sets 2022-07-27 19:57:23 +02:00
Sergei Trifonov
05e4bd6c8b review fix 2022-07-27 15:36:50 +02:00
Igor Nikonov
377c04fbf1 Merge remote-tracking branch 'origin/master' into fix_distinct_sorted 2022-07-27 13:01:18 +00:00
Anton Popov
1547c010b9
Merge pull request #39432 from ClickHouse/distinct_sorted_chunk_perf_impr 
DISTINCT in order: perf improvement
 2022-07-27 14:17:58 +02:00
avogar
01a309d4e3 Fix strings in dictionary in Arrow format 2022-07-27 12:02:27 +00:00
avogar
f925046dc4 Add more Pretty formats 2022-07-27 11:37:02 +00:00
vdimir
8eecb9ef82
upd PreparedSets: rename/change signature of methods, add comments 2022-07-27 11:22:16 +00:00
Nikolai Kochetov
873432fb53
Merge pull request #37849 from ClickHouse/bug-with-fill-date 
Enforce equality of WITH FILL type with ORDER BY column's type for date/time types.
 2022-07-27 12:27:53 +02:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache 2022-07-27 11:35:36 +02:00
Igor Nikonov
64e51e56e7 Allocate memory for column arrays once 2022-07-27 08:22:07 +00:00
Igor Nikonov
12a7567402 Some polishing 2022-07-27 07:58:54 +00:00
Igor Nikonov
589104fa6e Make building column arrays for chunk processing straightforward 2022-07-27 07:44:42 +00:00
Alexander Gololobov
a64aa00869
Merge pull request #37893 from zhangjmruc/feature/sql-standard-delete 
Support SQL standard "delete from ... where ..." syntax and lightweight implementation on merge tree tables
 2022-07-26 23:39:07 +02:00
Igor Nikonov
cac4d77d0b Merge remote-tracking branch 'origin/master' into distinct_sorted_chunk_perf_impr 2022-07-26 20:36:38 +00:00
Yakov Olkhovskiy
0055c9307d
style fix 2022-07-26 16:08:03 -04:00
Igor Nikonov
24f3a6905f
Merge branch 'master' into fix_distinct_sorted 2022-07-26 21:57:44 +02:00
Igor Nikonov
d196ab24d4 Calculate DISTINCT columns positions which form sort prefix in sort 
description once
 2022-07-26 19:55:29 +00:00
Yakov Olkhovskiy
d93c67e303 comment and test added 2022-07-26 15:28:11 -04:00
vdimir
d9928ac93d
Add methods to SubqueryForSet, do not use refernce to SetPtr 2022-07-26 18:39:09 +00:00
vdimir
1e3fa2e01f
Refactor PreparedSets/SubqueryForSet 2022-07-26 18:39:02 +00:00
Nikolai Kochetov
d7cc3831ea Push predicate over StorageMerge. 2022-07-26 15:01:39 +00:00
Kruglov Pavel
c362551642
Merge pull request #39357 from Avogar/fix-protobuf-capnp-empty-message 
Fix schema inference in case of empty messages in Protobuf/CapnProto formats
 2022-07-26 12:33:28 +02:00
Kruglov Pavel
53159db782
Merge branch 'master' into numbers-schema-inference 2022-07-26 12:32:49 +02:00
Sergei Trifonov
24ab5fbb86 fix finish() condition to account only active threads 2022-07-26 11:32:45 +02:00
Alexander Gololobov
0666ec2e1f
Merge branch 'master' into feature/sql-standard-delete 2022-07-26 10:42:39 +02:00
Vladimir Chebotaryov
f32d9c5539
Uppercase ROWS, GROUPS, RANGE in queries with windows. (#39410) 2022-07-25 22:53:53 +02:00
Alexander Gololobov
25deba2c1b
Merge branch 'master' into feature/sql-standard-delete 2022-07-25 22:13:20 +02:00
Nikolai Kochetov
b70be40804
Merge branch 'master' into use-dag-in-key-condition 2022-07-25 14:30:22 +02:00
Igor Nikonov
572dd864fe Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-07-25 12:19:14 +00:00
Sergei Trifonov
d9d1381c3f Merge branch 'master' into concurrency-control 2022-07-25 14:16:46 +02:00
Kruglov Pavel
83c7da6e88
Merge branch 'master' into fix-protobuf-capnp-empty-message 2022-07-25 13:02:41 +02:00
Igor Nikonov
dc96a6270d Remove optimize_memory_usage flag 
Memory usage with DistinctSortedTransform fixed in #39538
 2022-07-25 10:27:33 +00:00
Alexey Milovidov
388d06fda1
Merge pull request #39535 from ClickHouse/stringref 
Less usage of StringRef
 2022-07-25 04:06:11 +03:00
Igor Nikonov
41e72aac83 Fix: DistinctSortedTransform doesn't take advantage of sorting 
clearing_columns are set incorrectly, so we never clear HashSet
 2022-07-24 21:35:36 +00:00
Alexander Gololobov
460950ecdc
Merge branch 'master' into feature/sql-standard-delete 2022-07-24 21:27:22 +02:00
Robert Schulze
4333750985
Less usage of StringRef 
... replaced by std::string_view, see #39262
 2022-07-24 18:33:52 +00:00
Igor Nikonov
95511428b3 Couple optimizations 
+ do not apply filter to chunk if there is no data for output
+ checking clear_data flag at compile time
 2022-07-23 00:03:26 +00:00
Igor Nikonov
739ff34c6e Add some tests, still not sure about optimize_memory_usage option 2022-07-22 22:48:26 +00:00
Igor Nikonov
329adc3419
Merge branch 'master' into distinct_sorted_chunk_perf_impr 2022-07-22 12:58:59 +02:00
Igor Nikonov
51f63c3a0c Add setting optimize_distinct_in_order_memory_usage 2022-07-22 10:57:57 +00:00
Alexander Tokmakov
bed2206ae9
Merge pull request #39460 from ClickHouse/remove_some_dead_and_commented_code 
Remove some dead and commented code
 2022-07-22 13:24:34 +03:00
Igor Nikonov
7db5d54820 Adopt to the case when not all columns in distinct are part of sorting 
description
 2022-07-21 21:04:58 +00:00
avogar
794aa691bc Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-protobuf-capnp-empty-message 2022-07-21 17:04:37 +00:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache 2022-07-21 18:59:14 +02:00
Robert Schulze
ea0a3bf600
Merge branch 'master' into stringref-to-string_view 2022-07-21 18:33:06 +02:00
avogar
fd534aa3fa wqMerge branch 'master' of github.com:ClickHouse/ClickHouse into numbers-schema-inference 2022-07-21 15:43:17 +00:00
Igor Nikonov
122a1123b2 - disable the worst case for distinct in order in perf test for now 
+ functional test for query with the worst perfomance
+ debug logging in DistinctStep
 2022-07-21 15:03:19 +00:00
Alexander Tokmakov
a8da5d96fc remove some dead and commented code 2022-07-21 15:05:48 +02:00
avogar
6b541aa98f Fix WriteBuffer finalize when cancel insert into function 2022-07-21 12:18:37 +00:00
Alexander Gololobov
1ea9f143ff Leave only _row_exists-based implementation of lightweight delete 2022-07-21 11:26:13 +02:00
Nikolai Kochetov
e15967e9db
Merge pull request #38475 from ClickHouse/additional-filters 
Additional filters for a table (from setting)
 2022-07-21 07:52:04 +02:00
Alexey Milovidov
844042fc18
Merge pull request #39433 from ClickHouse/revert-39396-try-fix-write-buffer-terminate 
Revert "Fix WriteBuffer finalize in destructor when cacnel query"
 2022-07-21 07:04:07 +03:00
Alexey Milovidov
dcda9d3bd1
Merge pull request #39365 from Avogar/fix-capnproto-abort 
Avoid possible abort() in CapnProto on exception descruction
 2022-07-21 05:20:45 +03:00
Kruglov Pavel
92995a832b
Revert "Fix WriteBuffer finalize in destructor when cacnel query" 2022-07-21 01:45:16 +02:00
Igor Nikonov
635a566bec Comment change 2022-07-20 21:37:46 +00:00
Igor Nikonov
e50aebb5f0
Merge branch 'master' into distinct_sorted_chunk_perf_impr 2022-07-20 23:17:11 +02:00
Igor Nikonov
965f96bd84 DISTINCT in order: perf improvement 
+ reduce allocations in DistinctSortedChunkTransform
+ use it for final distinct as well
 2022-07-20 20:44:47 +00:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Nikolai Kochetov
4e8cd70b1d
Merge branch 'master' into use-dag-in-key-condition 2022-07-20 17:38:33 +02:00
Yakov Olkhovskiy
c4d040e02c
Merge branch 'master' into bug-with-fill-date 2022-07-20 09:10:45 -04:00
Dmitry Novik
a09422de7b
Merge pull request #38642 from ClickHouse/group-by-use-nulls 
Add group_by_use_nulls setting
 2022-07-20 14:42:42 +02:00
Kruglov Pavel
46da17ca8c
Merge branch 'master' into numbers-schema-inference 2022-07-20 13:32:39 +02:00