thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
92,689 Commits 9,328 Branches 1,783 Tags 2.1 GiB
ca2e27aaa2
Commit Graph

715 Commits

Author SHA1 Message Date
Azat Khuzhin
3be3c503aa Fix some comments 2021-08-08 09:58:07 +03:00
alexey-milovidov
c5207fc237
Merge pull request #26466 from azat/optimize-dist-select 
Rework SELECT from Distributed optimizations
 2021-08-08 03:59:32 +03:00
mergify[bot]
589505e76b
Merge branch 'master' into legacy-tuple-name 2021-08-07 16:42:20 +00:00
Azat Khuzhin
1358a2db92 Fix WithMergeableStateAfterAggregationAndLimit with LIMIT BY and LIMIT OFFSET 
In case of LIMIT BY, pre LIMIT cannot be applied (that is done on the
shard before), and hence shard applies regular LIMIT, but it goes with
applying OFFSET, and also the initiator will do apply OFFSET too.
 2021-08-06 00:16:26 +03:00
Azat Khuzhin
c4e2342a06 Add verbose description for pre-LIMIT into EXPLAIN query 2021-08-06 00:16:26 +03:00
Raúl Marín
479f053f2e Merge remote-tracking branch 'blessed/master' into materialization_log 2021-08-05 17:24:10 +02:00
Alexander Kuzmenkov
d852207b0e
Merge pull request #25721 from CurtizJ/read-in-order-perf-2 
Reduce memory usage in queries with `ORDER BY primary_key
 2021-08-05 16:13:48 +03:00
Anton Popov
915ef50c2d optimize reading in order of key more 2021-08-04 18:18:03 +03:00
Anton Popov
699a3d9031 implement legacy_column_name_of_tuple_literal in less intrusive way 2021-08-03 21:03:24 +03:00
Anton Popov
e36736b50c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-02 22:52:02 +03:00
Azat Khuzhin
ff12f5102a Avoid running LIMIT BY/DISTINCT step on the initiator for optimize_distributed_group_by_sharding_key 
Before the following queries was running LimitBy/Distinct step on the
initator:

  select distinct sharding_key from dist order by k

While this can be omitted.
 2021-08-02 21:04:30 +03:00
Raúl Marín
9f32ecca89 Merge remote-tracking branch 'blessed/master' into materialization_log 2021-08-02 16:21:10 +02:00
Anton Popov
cc3ed12ed6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-02 02:56:00 +03:00
Amos Bird
7b9e81742f
Shard level constness 2021-07-30 18:53:34 +08:00
Alexander Tokmakov
c74bfbf991 Merge branch 'master' into mvcc_prototype 2021-07-28 22:21:48 +03:00
Raúl Marín
f0125262b7 Add views and it's tables to system.query_log 2021-07-27 16:29:33 +02:00
Alexander Tokmakov
63ab38ee09 rename MaterializeMySQL to MaterializedMySQL 2021-07-26 21:17:28 +03:00
Anton Popov
f99374cca6 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-07-20 18:20:21 +03:00
mergify[bot]
39f71f4a08
Merge branch 'master' into fix_no_column_materialized_mysql 2021-07-19 12:27:38 +00:00
Anton Popov
f61ea15d56 fix pushdown of limit to reading stage 2021-07-16 20:40:00 +03:00
Anton Popov
03c785931a fix pushdown of limit to reading stage 2021-07-16 17:27:38 +03:00
Anton Popov
c45a9e3bd6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-16 17:11:34 +03:00
Raúl Marín
c2f0d7c514 Don't build set for indices when analyzing a query 2021-07-15 14:25:22 +02:00
Amos Bird
0558ecdc3f
Aggressive IN index analysis for projections. 2021-07-14 22:56:52 +08:00
Anton Popov
cb042afa9d pushdown limit while reading in order of primary key 2021-07-13 17:24:45 +03:00
mergify[bot]
d5898ecba6
Merge branch 'master' into fix_no_column_materialized_mysql 2021-07-13 14:03:05 +00:00
alexey-milovidov
4b1e38eed9
Merge branch 'master' into projection-fix11 2021-07-13 04:52:15 +03:00
Alexey Milovidov
e0ae04297a Fix fast test 2021-07-12 18:19:49 +03:00
Anton Popov
6ba41dc265 fix projections with storage snapshot 2021-07-12 16:45:35 +03:00
Anton Popov
3ed7f5a6cc dynamic subcolumns: add snapshot for storage 2021-07-09 06:15:41 +03:00
vdimir
d4256a8583
Minor style changes in JoinedTables 2021-07-08 13:49:13 +03:00
vdimir
5d7fc61bd5
Merge branch 'master' into join-materialized-columns 2021-07-08 12:30:00 +03:00
Anton Popov
072e65b728 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-07-07 17:20:38 +03:00
mergify[bot]
446474932b
Merge branch 'master' into projection-fix11 2021-07-07 05:02:44 +00:00
Amos Bird
55981cb0ae
Fix normal projection 2021-07-07 13:01:30 +08:00
alexey-milovidov
b724138541
Merge pull request #25786 from kssenii/inconsisteny 
Fix inconsistent behaviour of GROUP BY constant on empty set
 2021-07-06 03:36:50 +03:00
Alexander Tokmakov
0f377fc4d8 Merge branch 'master' into fix_no_column_materialized_mysql 2021-07-05 22:07:56 +03:00
alexey-milovidov
fef7c557f6
Merge pull request #24789 from kitaisreal/compile-aggregation-functions 
Compile aggregate functions
 2021-07-05 02:57:49 +03:00
kssenii
77879f6826 Better 2021-07-04 01:53:51 +03:00
kssenii
a9a91a4f15 Setting move to prewhere if final 2021-07-03 11:37:50 +00:00
kssenii
02681019f8 Fix 2021-07-03 08:48:42 +00:00
Maksim Kita
da8c957167 Aggregator added CompiledExpressionCache 2021-07-01 22:56:36 +03:00
Maksim Kita
d24d3ae992 Added second variant of compilation 2021-07-01 22:56:36 +03:00
Maksim Kita
3fe559b31f Compile aggregate functions 2021-07-01 22:56:35 +03:00
Alexander Tokmakov
fd914f3d97 fix 'Not found column' error in MaterializeMySQL 2021-06-29 20:47:41 +03:00
Nikolai Kochetov
3544b059ce Merge branch 'master' into remove-PrewhereDAGInfo 2021-06-25 17:52:03 +03:00
Nikolai Kochetov
6bc0a628cd Remove PrewhereDAGInfo. 2021-06-25 17:49:28 +03:00
vdimir
2949cd1e6f
Support ALIASed columns in JOIN ON expression 2021-06-25 14:50:02 +03:00
vdimir
241b64d02c
Support ALIASed columns for right joined table 2021-06-24 17:57:21 +03:00
vdimir
84e02911cf
Materialized columns for joined table, don't add to asterisk select 2021-06-24 17:14:36 +03:00
Anton Popov
c2754c34f4 Merge remote-tracking branch 'upstream/master' into HEAD 2021-06-21 12:42:48 +03:00
Anton Popov
d8b6f15ef4
Merge pull request #23027 from azat/distributed-push-down-limit 
Add ability to push down LIMIT for distributed queries
 2021-06-20 23:08:50 +03:00
Nikolai Kochetov
b2620e1def
Merge pull request #24405 from amosbird/replacemorealias 
Rewrite more alias columns
 2021-06-18 09:10:31 +03:00
Amos Bird
0f7728cd5e
Fix RBAC test again 2021-06-17 23:00:03 +08:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Anton Popov
ea9b890618
Merge branch 'master' into add-compatibility-setting 2021-06-16 21:10:56 +03:00
Anton Popov
8bd7cbd925 add compatibility setting for name of columns 2021-06-16 21:10:07 +03:00
mergify[bot]
7959d92029
Merge branch 'master' into minor-changes-3 2021-06-15 18:07:24 +00:00
Nikita Mikhaylov
a52bba91b7
Merge pull request #16401 from abyss7/ast-table-identifier-2 
ASTTableIdentifier part #2: Introduce ASTTableIdentifier
 2021-06-15 13:51:30 +03:00
Amos Bird
aaac231c6a
Record expanded aliases for RBAC 2021-06-15 18:28:13 +08:00
Amos Bird
e3ae2f6e7a
Rewrite more alias columns 2021-06-15 17:54:28 +08:00
Alexey Milovidov
447d7bb8cd Minor changes 2021-06-14 07:13:35 +03:00
Alexander Tokmakov
6f5cf773ef Merge branch 'master' into mvcc_prototype 2021-06-09 12:50:54 +03:00
Azat Khuzhin
18e8f0eb5e Add ability to push down LIMIT for distributed queries 
This way the remote nodes will not need to send all the rows, so this
will decrease network io and also this will make queries w/
optimize_aggregation_in_order=1/LIMIT X and w/o ORDER BY faster since it
initiator will not need to read all the rows, only first X (but note
that for this you need to your data to be sharded correctly or you may
get inaccurate results).

Note, that having lots of processing stages will increase the complexity
of interpreter (it is already not that clean and simple right now).

Although using separate QueryProcessingStage looks pretty natural.

Another option is to make WithMergeableStateAfterAggregation always, but
in this case you will not be able to disable only this optimization,
i.e. if there will be some issue with it.

v2: fix OFFSET
v3: convert 01814_distributed_push_down_limit test to .sh and add retries
v4: add test with OFFSET
v5: add new query stage into the bash completion
v6/tests: use LIMIT O,L syntax over LIMIT L OFFSET O since it is broken in ANTLR parser
          https://clickhouse-test-reports.s3.yandex.net/23027/a18a06399b7aeacba7c50b5d1e981ada5df19745/functional_stateless_tests_(antlr_debug).html#fail1
v7/tests: set use_hedged_requests to 0, to avoid excessive log entries on retries
          https://clickhouse-test-reports.s3.yandex.net/23027/a18a06399b7aeacba7c50b5d1e981ada5df19745/functional_stateless_tests_flaky_check_(address).html#fail1
 2021-06-09 02:29:50 +03:00
Ivan
579250168f
Merge branch 'master' into ast-table-identifier-2 2021-06-07 20:30:39 +03:00
Anton Popov
205a23282b Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-06-07 19:38:22 +03:00
Alexey Milovidov
885ce194e0 Making fundamentals correct 2021-06-07 00:49:55 +03:00
Amos Bird
9dfa4c06bd
Try reusing built sets during multi-pass analysis 2021-06-03 22:06:05 +08:00
Ivan Lezhankin
ae38b4a3f1 Merge remote-tracking branch 'upstream/master' into ast-table-identifier-2 2021-06-02 16:07:59 +03:00
Anton Popov
9f52362b81 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-05-31 23:08:45 +03:00
Alexander Tokmakov
5692db736c Merge branch 'master' into mvcc_prototype 2021-05-31 21:11:22 +03:00
Alexander Kuzmenkov
6cccb0992a fix ORDER BY after window fuctions over Distributed 2021-05-27 19:06:37 +03:00
Anton Popov
0bdf9d207c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-05-26 00:25:47 +03:00
mergify[bot]
970243850f
Merge branch 'master' into ast-table-identifier-2 2021-05-21 12:56:49 +00:00
Maksim Kita
b0476c1fa2
Merge pull request #24273 from kitaisreal/expression-actions-compile-only-necessary-places 
ExpressionActions compile only necessary places
 2021-05-20 10:53:30 +03:00
Maksim Kita
c50909468f Updated ExpressionActionSettings 2021-05-19 17:32:07 +03:00
Alexander Kuzmenkov
e9b69bbd70
Merge pull request #23906 from azat/fix-distributed_group_by_no_merge 
distributed_group_by_no_merge fixes
 2021-05-19 16:16:08 +03:00
Maksim Kita
150a88d647 ExpressionActions compile only necessary places 2021-05-19 11:43:16 +03:00
Ivan Lezhankin
bd16dd22b9 Merge remote-tracking branch 'upstream/master' into ast-table-identifier-2 2021-05-14 15:40:33 +03:00
Alexander Tokmakov
df2849be90 Merge branch 'master' into mvcc_prototype 2021-05-13 21:48:36 +03:00
Anton Popov
37989cd2ae fix reading of nested 2021-05-11 15:01:41 +03:00
Amos Bird
5836e2e277
Fix issues in normal projection 2021-05-11 18:12:29 +08:00
Amos Bird
dfa5629071
Remove projection type ast 2021-05-11 18:12:29 +08:00
Amos Bird
51d685bb9c
Even more fixes 2021-05-11 18:12:29 +08:00
Amos Bird
718c284437
Fix more tests 2021-05-11 18:12:28 +08:00
Amos Bird
ba17acbd63
Fix tests 2021-05-11 18:12:28 +08:00
Amos Bird
42f161b1dc
Fix more 2021-05-11 18:12:28 +08:00
Amos Bird
9c069ebdbf
support prewhere, row_filter, read_in_order and decent projection selection 
TODO set index analysis in projection
 2021-05-11 18:12:27 +08:00
Amos Bird
35961c0c5d
Use ActionsDAG to rewrite projection queries 2021-05-11 18:12:27 +08:00
Nikolai Kochetov
29d7038e30
Add ProjectionDescription::Type enum. Remove some checks. 2021-05-11 18:12:27 +08:00
Amos Bird
ebaf42a448
Reformat and fix some tests 2021-05-11 18:12:27 +08:00
Nikolai Kochetov
c2396c5144
Disable read-in-order optimization in case of projections. 2021-05-11 18:12:26 +08:00
Nikolai Kochetov
3d36fb57eb
Cleanup code. 2021-05-11 18:12:26 +08:00
Nikolai Kochetov
3296c9292f
Try to merge projectons faster. 2021-05-11 18:12:26 +08:00
Amos Bird
cd6414639e
add metadata_snapshot to getQueryProcessingStage 2021-05-11 18:12:26 +08:00
Amos Bird
264cff6415
Projections 
TODO (suggested by Nikolai)

1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies

3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)

Also need to figure out how prewhere works for projections, and
row_filter_policies.

wip
 2021-05-11 18:12:23 +08:00
Azat Khuzhin
eefd67fce5 Disable optimize_distributed_group_by_sharding_key with window functions 2021-05-06 00:44:22 +03:00
Azat Khuzhin
8703b7863d Fix distributed_group_by_no_merge=2 with GROUP BY 
Before this patch the following query:

  SELECT assumeNotNull(argMax(dummy, 1))
  FROM remote('127.1', system.one)
  SETTINGS distributed_group_by_no_merge = 2

Leads to:

  Code: 10. DB::Exception: Received from localhost:9000. DB::Exception: Not found column argMax(dummy, 1) in block: while executing 'INPUT : 0 -> argMax(dummy, 1) UInt8 : 0'.

Since it tries to execute function one more time, but shards will not
send this column when the query processed with
distributed_group_by_no_merge=2 (i.e. up to
WithMergeableStateAfterAggregation).

v0: no exception
v2: execut window functions
v3: throw exception, since executing window function in this case will
lead to messy output
 2021-05-06 00:21:47 +03:00
Anton Popov
e44706911e dynamic columns: better getting of sample block 2021-05-05 02:02:54 +03:00
Nikolai Kochetov
25fe5f63ee Fix other tests. 2021-04-29 12:08:49 +03:00
Nikolai Kochetov
69ddca1d73 Refactor join step, part 3 2021-04-28 20:32:12 +03:00
Anton Popov
644df6be7d dynamic subcolumns: wip 2021-04-24 07:09:01 +03:00
Anton Popov
6240169bbb dynamic subcolumns: wip 2021-04-24 02:56:26 +03:00
Alexander Kuzmenkov
ab75f86ee9 add a comment 2021-04-23 18:45:45 +03:00
Ivan Lezhankin
cdd825b18b Merge remote-tracking branch 'upstream/master' into ast-table-identifier-2 2021-04-23 18:08:25 +03:00
Alexander Kuzmenkov
6982ab5324 fix extra distinct 2021-04-23 15:44:46 +03:00
Alexander Kuzmenkov
9d95ac3d66 remove debug 2021-04-23 14:00:17 +03:00
Alexander Kuzmenkov
54ad086753 fix for distributed 2021-04-23 13:59:21 +03:00
Azat Khuzhin
86069ffb40 Fix LOGICAL_ERROR for max_distributed_connections=0 2021-04-20 09:05:44 +03:00
Ivan
a73483751f
Merge branch 'master' into ast-table-identifier-2 2021-04-12 20:23:28 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
Ivan Lezhankin
3da3250bfd Merge branch 'master' into ast-table-identifier-2 2021-04-09 17:56:15 +03:00
Nikolai Kochetov
28ca191102
Merge pull request #22087 from ClickHouse/better-filter-push-down 
Better filter push down
 2021-04-09 10:22:17 +03:00
Alexander Tokmakov
3422bd1742 check parts visibility for select 2021-04-08 20:20:45 +03:00
Alexander Kuzmenkov
06bb58cb69
Merge pull request #22704 from ClickHouse/aku/window-multiple 
fix window functions with multiple input streams and no sorting
 2021-04-07 15:15:47 +03:00
Azat Khuzhin
f157278b72 Safer SCOPE_EXIT 
It executes the code in the dtor, that should never throw.
 2021-04-06 21:10:37 +03:00
Alexander Kuzmenkov
585da2c8d0 Merge remote-tracking branch 'origin/master' into HEAD 2021-04-06 14:58:21 +03:00
Alexander Kuzmenkov
06e7a9f8f2 fix window functions with multiple input streams and no sorting 2021-04-06 14:43:27 +03:00
Nikolai Kochetov
6102652c99 Merge branch 'master' into better-filter-push-down 2021-04-06 13:38:03 +03:00
Anton Popov
f58d470dd6
Merge pull request #21830 from foolchi/optimize_select_with_final 
optimize select final with prewhere primary key
 2021-03-25 15:32:13 +03:00
Nikolai Kochetov
0afede382b Use empty header for CreatingSets step. 2021-03-25 11:39:55 +03:00
Nikolai Kochetov
12de01cb48 Remove AddingDelayedSourceStep. 2021-03-24 16:22:51 +03:00
Nikolai Kochetov
730785f4c2
Merge pull request #21459 from ClickHouse/refactor-actions-dag 
Refactor actions dag
 2021-03-22 11:58:16 +03:00
Alexander Kuzmenkov
1cd9f28bd4 another way 2021-03-17 18:31:30 +03:00
fuqi
5e20ea2c33 optimize select final with prewhere primary key 2021-03-17 18:49:24 +08:00
Nikolai Kochetov
4f133ae036 Merge branch 'master' into refactor-actions-dag 2021-03-10 12:40:55 +03:00
Nikolai Kochetov
5ef9d87790 Fix some other tests. 2021-03-10 11:41:24 +03:00
vdimir
36006a7e96
Merge branch 'master' into cross-to-inner-join-rewrite 2021-03-09 09:36:15 +03:00
Vladimir
6ec19b2632
Merge pull request #19885 from ClickHouse/vdimir/join-cast-types-v2 2021-03-09 09:30:50 +03:00
vdimir
354757d94d
Add logging for need_analyze_again in InterpreterSelectQuery 2021-03-08 09:33:30 +03:00
Nikolai Kochetov
a669f7d641 Merge branch 'master' into refactor-actions-dag 2021-03-05 18:21:14 +03:00
vdimir
4a8708fbb2
Clean subqueries_for_sets before second analyze in InterpreterSelectQuery 2021-03-05 17:43:26 +03:00
vdimir
b0f6f30298
Revert "Refactor join: make IJoin implementations independent from TableJoin" 
This reverts commit e6406c3f4c.
 2021-03-05 17:34:43 +03:00
vdimir
e6406c3f4c
Refactor join: make IJoin implementations independent from TableJoin 2021-03-05 16:38:49 +03:00
Nikolai Kochetov
9a39459888 Refactor ActionsDAG 2021-03-04 20:38:12 +03:00
Nikolai Kochetov
15921fbfcb Remove index by name from ActionsDAG 2021-03-03 23:01:07 +03:00
Nikolai Kochetov
ffbea93b10
Merge pull request #19576 from traceon/row-policy-with-prewhere 
Allow row policies with PREWHERE
 2021-03-02 13:51:37 +03:00
Denis Glazachev
eb80c189e4 Merge branch 'master' into row-policy-with-prewhere 2021-02-28 20:14:31 +04:00
Nikolai Kochetov
d91b8a3acb
Merge branch 'master' into filter-push-down 2021-02-26 19:33:27 +03:00
Nikolai Kochetov
d328bfa41f Review fixes. Add setting max_optimizations_to_apply. 2021-02-26 19:29:56 +03:00
vdimir
4c6cde4d53
Merge remote-tracking branch 'upstream/master' into join-cast-types-v2 2021-02-25 16:24:28 +03:00
Alexey Milovidov
0ab14120ef Improve performance of trivial count query in presense of "distributed_aggregation_memory_efficient" 2021-02-21 23:06:31 +03:00
Denis Glazachev
c1158b50f4 Merge branch 'master' into row-policy-with-prewhere 
* master: (155 commits)
  Update version_date.tsv after release 20.8.13.15
  Update version_date.tsv after release 20.12.7.3
  Update version_date.tsv after release 21.1.5.4
  Update version_date.tsv after release 21.2.4.6
  fix
  Add test to skip list
  Fix WriteBufferFromHTTPServerResponse usage in other places (add missing finalize())
  Fix WriteBufferFromHTTPServerResponse usage in odbc-bridge
  Update config.xml
  Suppress signed overflow in AggregateFunctionGroupArrayMoving 2
  Update BaseDaemon.cpp
  review suggestions
  Fix bash syntax in 01731_async_task_queue_wait
  Do not use view() in 01731_async_task_queue_wait to fix ANTLR parser
  Increase buffer for uncaught exception / std::terminate
  Even more better
  Fix uncaught exception when HTTP client goes away
  test for decimal ( p , s) in dictionaries
  Just little better
  Fixed style check
  ...
 2021-02-21 13:25:47 +04:00
Denis Glazachev
dfde9de0e9 Remove commented-out code 2021-02-20 23:30:21 +04:00
Nikolai Kochetov
8a876b9510 Fix aliases for row level actions. 2021-02-20 17:19:11 +03:00
Nikolai Kochetov
673e24d7ef Refactor 2021-02-20 14:00:16 +03:00
Denis Glazachev
8cc72b8fe0 Merge branch 'master' into row-policy-with-prewhere 
* master: (160 commits)
  Make Poco HTTP Server zero-copy again (#19516)
  Fixed documentation
  ccache 4.2+ does not requires any quirks for SOURCE_DATE_EPOCH
  Add a function `htmlOrXmlCoarseParse` to extract content from html or xml format string. (#19600)
  Reinterpret function added Decimal, DateTim64 support
  Add test
  Update InterpreterSelectQuery.cpp
  Improved serialization for data types combined of Arrays and Tuples. Improved matching enum data types to protobuf enum type. Fixed serialization of the Map data type. Omitted values are now set by default.
  Log stdout and stderr when failed to start docker in integration tests.
  Added comment
  Don't backport base commit of branch in the same branch (#20628)
  Fix fasttest retry for failed tests
  Dictionary create source with functions crash fix
  Added error reinterpretation tests
  Update run.sh
  Updated documentation
  fix subquery with limit
  Rename untyped function reinterpretAs into reinterpret
  ignore data store files
  Support vhost
  ...
 2021-02-19 19:49:37 +04:00
Denis Glazachev
5e5b3b80ce Remove debug printouts 2021-02-19 19:46:52 +04:00
vdimir
435f63f42b
Calculate common type for join using in TreeRewriter 
Split before_join and converting_join_columns dags

Add more detailed comments for column type conversion for join using
 2021-02-19 14:28:32 +03:00
vdimir
4c36cd1737
Add converting step for 'join using' 2021-02-19 14:24:28 +03:00
Azat Khuzhin
af660140c3 Do only merging of sorted blocks on initiator with distributed_group_by_no_merge=2 
When distributed_group_by_no_merge=2 is used (or when
optimize_distributed_group_by_sharding_key takes place), remote servers
will do full ORDER BY, so initiator can skip this step and do only merge
of ordered blocks.
 2021-02-18 21:43:27 +03:00
Azat Khuzhin
b854a7b7f8 Add some details into comment for first_stage/second_stage 
Regardless
distributed_group_by_no_merge=2/optimize_distributed_group_by_sharding_key
 2021-02-18 21:41:50 +03:00
alexey-milovidov
3891dd6284
Update InterpreterSelectQuery.cpp 2021-02-17 21:23:27 +03:00
feng lv
3b40099578 fix subquery with limit 2021-02-17 08:26:52 +00:00
Nikolai Kochetov
bcf30d8412 Try fix tests. 2021-02-16 15:57:00 +03:00
Nikolai Kochetov
a14b6c2650 Fix trivial count optimization 2021-02-16 12:37:19 +03:00
Nikolai Kochetov
f139ad8080 Comment debug output. 2021-02-16 09:15:12 +03:00
Nikolai Kochetov
ae73600fb0 Refactor row level security actions. 2021-02-15 22:48:06 +03:00
vdimir
9afa6b5b1b
Add option cross_to_inner_join_rewrite 2021-02-15 22:40:32 +03:00
Denis Glazachev
45e90961f7 Store and process alias_actions in FilterInfo 2021-02-14 21:16:40 +04:00
Denis Glazachev
99a04b6c64 Merge branch 'master' into row-policy-with-prewhere 
* master: (759 commits)
  Suppress UBSan report in Decimal comparison
  Suppress UBSan report in Decimal comparison
  Fix UBSan report in arrayDifference
  Update README.md
  Non significant change in AggregationCommon
  Print stack trace on SIGTRAP
  Fix dependent test
  Fix tests for better parallel run
  Add test for already working code
  Revert "Fix access control manager destruction order"
  Update index.md
  Update index.md
  Update index.md
  Bit more complicated example for isIPv4String - ru
  Bit more complicated example for isIPv4String
  cleanup
  Replace database with ordinary
  Added comments
  Split tests to make them stable
  Fixes
  ...

# Conflicts:
#	src/Storages/MergeTree/MergeTreeRangeReader.cpp
 2021-02-14 02:24:00 +04:00
Denis Glazachev
79592b73f8 Store filter info in prewhere info instead of multiple prewheres 
Some cleanups
 2021-02-14 02:07:13 +04:00
Alexander Kuzmenkov
363007b964 fixes 2021-02-11 11:39:39 +03:00
Alexander Kuzmenkov
38678ff484 Merge remote-tracking branch 'origin/master' into HEAD 2021-02-11 10:40:05 +03:00
Alexander Kuzmenkov
e557d60cdb fixes for sort order 2021-02-10 16:27:22 +03:00
Alexander Kuzmenkov
b57b3244f5 reuse sort order 2021-02-10 15:44:07 +03:00
Amos Bird
46f2b4063f
Fix CTE in INSERT-SELECT 2021-02-09 00:25:24 +08:00
alexey-milovidov
44b44c1fe7
Update InterpreterSelectQuery.cpp 2021-02-07 01:09:55 +03:00
Maxim Akhmedov
0cd3601928 Allow using MergeTreeWhereOptimizer not only with MergeTree-based storages 2021-02-06 18:08:42 +03:00
Nikolai Kochetov
afdc9635cb Update MergeTreeRangeReader 2021-01-29 19:12:53 +03:00
Denis Glazachev
009d1e3fc5 Merge branch 'master' into row-policy-with-prewhere 
* master: (247 commits)
  Update 01014_lazy_database_basic.sh
  Update 00459_group_array_insert_at.sql
  Same for ALTER
  Syntax and links
  Added test.
  Check where and prewhere identifiers exist.
  Remove redundant lsof
  Add lsof to fasttest
  Add new line to status
  Make integration odbc tests idempotent
  Minor code improvement in JOIN
  Arcadia does not have bitmaps
  Make Fuzzer more reliable
  Add missing lsof for fasttest docker image
  Update test
  The test most likely would not work in Arcadia
  Less parser depth
  Fix stack overflow in coroutine
  Fuzzer: better messages.
  Fixed build.
  ...
 2021-01-28 21:48:12 +04:00
Denis Glazachev
6a92295982 Remove TODOs 2021-01-27 19:16:31 +04:00
Amos Bird
a2ceca3fa1
Fix recursive propagation 2021-01-26 10:29:12 +08:00
Denis Glazachev
4870e0af69 Add filter as a (first) prewhere 2021-01-25 22:09:17 +04:00
Denis Glazachev
4f6c880232 Pass and handle a chain of multiple prewhere infos 2021-01-25 18:31:59 +04:00
alexey-milovidov
d2366b5c6c
Merge pull request #18233 from vitlibar/fix-access-rights-for-select-count 
Fix access rights for SELECT count()
 2021-01-17 00:12:08 +03:00
alexey-milovidov
bf522d0eea
Update InterpreterSelectQuery.cpp 2021-01-17 00:11:58 +03:00
sundy-li
ad01534bd5 Merge branch 'master' into alias_column_partition_prune 2021-01-12 18:28:30 +08:00
Nikolai Kochetov
f955ccecf4 Do not materialize block for FetchColumns header. 2021-01-05 21:04:25 +03:00
sundyli
f8815f233c
Merge branch 'master' into alias_column_partition_prune 2021-01-02 11:23:55 +08:00
Vitaly Baranov
235ea220ec SELECT count() FROM table now requires at least one column to be accessible in the table. 2020-12-30 15:00:08 +03:00
Vitaly Baranov
460006b502
Merge pull request #18232 from vitlibar/fix-access-rights-for-select-join 
Fix access rights for SELECT JOIN.
 2020-12-30 14:42:16 +03:00
Ivan Lezhankin
5e89145a42 Merge remote-tracking branch 'upstream/master' into ast-table-identifier-2 2020-12-30 13:04:17 +03:00
Alexander Kuzmenkov
d4e5d5fdb4
Merge pull request #18455 from ClickHouse/aku/window-prototype 
[wip] some window function fixes
 2020-12-29 19:23:19 +03:00
hexiaoting
4adbdf364e Fix bug for orderby withfill with limit clause 2020-12-28 13:05:16 +03:00
Vitaly Baranov
9271b3d627 SELECT JOIN now requires the SELECT privilege on each of the joined tables. 2020-12-26 18:35:32 +03:00
Alexander Kuzmenkov
5bd025a180 fix DISTINCT and ORDER BY 2020-12-25 06:13:30 +03:00
Alexander Kuzmenkov
23d77a1698 remove debug output 2020-12-24 09:59:38 +03:00
Alexander Kuzmenkov
844239b1c3 tmp fixes (split me) 2020-12-24 07:03:33 +03:00
Alexander Kuzmenkov
f2b25c708c review fixes 2020-12-22 22:01:15 +03:00
Alexander Kuzmenkov
3f33e6b3e0 some review fixes 2020-12-22 20:46:31 +03:00
Alexander Kuzmenkov
52ace7a3d1 one plan step per window 2020-12-22 04:37:45 +03:00
Alexander Kuzmenkov
77b816caea cleanup 2020-12-22 02:47:14 +03:00
Alexander Kuzmenkov
85c1bc1253 Merge remote-tracking branch 'origin/master' into tmp 2020-12-21 10:46:21 +03:00
Amos Bird
9924cad96c
identify internal queries 2020-12-18 14:54:38 +08:00
Amos Bird
2f92d57af9
Better code. 2020-12-18 14:13:58 +08:00
Amos Bird
272311f6bd
Extensive information in system.query_log 2020-12-18 14:13:58 +08:00
Alexander Kuzmenkov
a1326414ff Merge remote-tracking branch 'origin/master' into tmp 2020-12-17 00:46:14 +03:00
Ivan Lezhankin
2fe6195577 Merge remote-tracking branch 'upstream/master' into ast-table-identifier-2 2020-12-16 14:50:36 +03:00
sundy-li
5c9f84b2f4 Merge branch 'master' into alias_column_partition_prune 2020-12-16 18:44:00 +08:00
Alexander Kuzmenkov
fabb37d3c9 cleanup 2020-12-15 20:39:58 +03:00
Alexander Kuzmenkov
a23a9fd947 fixes after merge 2020-12-15 20:16:01 +03:00
Nikita Mikhaylov
31bef163ba
Merge pull request #17848 from azat/max_distributed_connections-fix 
Fix max_distributed_connections
 2020-12-15 17:34:38 +03:00
sundy-li
cf087002db Merge branch 'master' into alias_column_partition_prune 2020-12-15 16:37:00 +08:00
Alexander Kuzmenkov
a75ca3cb79 Merge remote-tracking branch 'origin/master' into tmp 2020-12-15 03:36:43 +03:00
Alexander Kuzmenkov
6d5b23de67 something works 2020-12-15 03:36:03 +03:00
sundy-li
d2209a1460 trivial fix 2020-12-13 17:33:02 +08:00
alexey-milovidov
bce02de691
Merge pull request #17539 from azat/remerge_sort_lowered_memory_bytes_ratio 
Add remerge_sort_lowered_memory_bytes_ratio setting
 2020-12-13 04:38:50 +03:00
sundyli
36ca21c24a
Merge branch 'master' into alias_column_partition_prune 2020-12-13 00:45:14 +08:00
sundy-li
9407028540 update 2020-12-13 00:42:15 +08:00
Alexander Kuzmenkov
d757f32300 Remove some redundant includes to speed up build 2020-12-11 02:58:11 +03:00
Alexander Kuzmenkov
dea446be2f Remove some redundant includes to speed up build 2020-12-11 02:56:57 +03:00
Alexander Kuzmenkov
ac8b896f36 returning constant column as window function 2020-12-10 22:06:52 +03:00
Azat Khuzhin
c3a4e5e7b7 Fix max_distributed_connections (one more time) 
With prefer_localhost_replica=1 max_distributed_connections uses
max_threads before this patch.

Fixes: #14936
Previous fix: #9673
 2020-12-07 01:19:53 +03:00
Azat Khuzhin
ccd794a676 Add remerge_sort_lowered_memory_bytes_ratio setting 
Sometimes default ratio is not good enough (2), since it depends on lots
of factors:
- size of LIMIT
- max_memory_usage
- ...

So add a separate setting for it.

But note that it make sense to set
remerge_sort_lowered_memory_bytes_ratio only in range (1, 2]

Since <= 1 will just use more CPU, and > 2 will use more RAM and will
unlikely be faster.
 2020-11-29 21:16:42 +03:00
Nikita Mikhaylov
084c75fa6e
Merge pull request #17309 from nikitamikhaylov/merging-sequential-consistency 
Merging #16309
 2020-11-26 15:26:05 +03:00
nikitamikhaylov
72c7cd6693 replace Context& to Settings& 2020-11-25 16:47:32 +03:00
Alexander Kuzmenkov
1d7dcef4a0 Merge remote-tracking branch 'origin/master' into HEAD 2020-11-24 21:45:10 +03:00
Alexander Kuzmenkov
412c2fe079 Merge remote-tracking branch 'origin/master' into HEAD 2020-11-23 19:51:18 +03:00
nikitamikhaylov
68bef22fda Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-sequential-consistency 2020-11-23 16:28:35 +03:00
Ivan Lezhankin
1002148886 Merge remote-tracking branch 'upstream/master' into ast-table-identifier-2 2020-11-22 22:34:36 +03:00
sundy-li
fe8b11fd5f add replaceAliasColumnsInFilter function 2020-11-22 12:16:27 +08:00
sundy-li
ad99dab3ad add ColumnAliasesVisitor to improve partition prune for mergetree 2020-11-21 20:38:21 +08:00
Alexander Kuzmenkov
6cb378e072 cleanup 2020-11-19 18:52:11 +03:00
Nikolai Kochetov
13eceff55c Remove Converting transform and step. 2020-11-17 20:16:55 +03:00
Ivan Lezhankin
9250d5c4e9 Merge remote-tracking branch 'upstream/master' into ast-table-identifier-2 2020-11-12 16:25:46 +03:00
Nikolai Kochetov
9b13d0e0eb Revert "Add ActionsDAG::updateHeader()" 
This reverts commit b78657656c.
 2020-11-11 20:17:26 +03:00
Nikolai Kochetov
58c4a7b890 Some renames. 2020-11-11 19:52:27 +03:00
Nikolai Kochetov
b78657656c Add ActionsDAG::updateHeader() 2020-11-11 16:50:29 +03:00
Nikolai Kochetov
1846bb3cac Merge branch 'master' into actions-dag-f14 2020-11-11 13:08:57 +03:00
feng lv
fec4080d42 Merge branch 'master' of github.com:ClickHouse/ClickHouse into union-distinct 2020-11-11 03:10:10 +00:00
Nikolai Kochetov
c6575c9032 Update ExpressionActions constructur 2020-11-10 19:27:55 +03:00
Nikolai Kochetov
195c941c4e Merge branch 'master' into storage-read-query-plan 2020-11-10 15:02:22 +03:00
feng lv
9194985d48 Merge branch 'master' of github.com:ClickHouse/ClickHouse into union-distinct 2020-11-10 06:09:39 +00:00
Nikolai Kochetov
6717c7a0af Merge branch 'master' into actions-dag-f14 2020-11-09 14:57:48 +03:00
Alexey Milovidov
5314185e25 Merge branch 'master' into azat-optimize_skip_unused_shards-optimization 2020-11-08 00:17:59 +03:00
feng lv
1ca7dcb0a8 fix conflict 2020-11-07 11:44:29 +00:00
Nikolai Kochetov
c10f733587 Merge branch 'master' into storage-read-query-plan 2020-11-06 15:43:46 +03:00
Nikolai Kochetov
9aeb757da4 Merge branch 'master' into actions-dag-f14 2020-11-06 15:04:20 +03:00
Nikolai Kochetov
40a9648269
Merge pull request #16308 from nikitamikhaylov/totals-having 
Incorrect behaviour of SELECT WITH TOTALS
 2020-11-05 11:27:28 +03:00
Alexander Kuzmenkov
faf1f1e0cd Merge remote-tracking branch 'origin/master' into HEAD 2020-11-05 11:09:11 +03:00
Nikolai Kochetov
7b61f5d641 Refactor more 2020-11-03 22:05:47 +03:00
Azat Khuzhin
e7f15dff55 Do not pass StoragePtr to ReadInOrderOptimizer::getInputOrder() 
Looks like this is not required anymore, since #11745
 2020-11-03 21:22:46 +03:00
Nikolai Kochetov
07a7c46b89 Refactor ExpressionActions [Part 3] 2020-11-03 14:28:28 +03:00
Nikolai Kochetov
ceb83602b8 Refactor 2020-11-02 15:07:01 +03:00
feng lv
e82ce52e2f fix 2020-11-01 14:57:41 +00:00
feng lv
382fff9009 fix 2020-11-01 13:54:07 +00:00
Ivan Lezhankin
4083406f54 Introduce ASTTableIdentifier 2020-10-26 18:49:00 +03:00
hchen9
feaab260d8 Fix conflict 2020-10-23 12:54:33 -07:00
nikitamikhaylov
d908cddb49 done 2020-10-23 22:09:02 +03:00
Nikolai Kochetov
c91b453630 Try fix tests. 2020-10-22 14:08:12 +03:00
Nikolai Kochetov
7fa045cff8 Merge branch 'master' into storage-read-query-plan 2020-10-22 13:31:10 +03:00
alexey-milovidov
adeba6bdd8
Merge pull request #15074 from amosbird/btc 
Extend trivial count optimization.
 2020-10-22 02:50:57 +03:00
alexey-milovidov
be7776608d
Merge pull request #15933 from ClickHouse/fix-global-in-result 
Fix wrong result in case of GLOBAL IN and PREWHERE
 2020-10-19 21:21:57 +03:00
Alexander Tokmakov
7d58992ea8 fix select count() 2020-10-14 16:15:15 +03:00
Nikolai Kochetov
534b60e242 Fix build. 2020-10-13 18:10:30 +03:00
Nikolai Kochetov
23afd74e1c Update InterpreterSelectQuery. 2020-10-13 17:55:22 +03:00
Nikolai Kochetov
1af0792f67 Update InterpreterSelectQuery. 2020-10-13 17:33:34 +03:00
Nikolai Kochetov
2bbdfc77ea Update creating sets. 2020-10-13 16:48:00 +03:00
Nikolai Kochetov
337098367b Try fix perf 2020-10-13 11:36:15 +03:00
Nikolai Kochetov
7e58f99f64 Merge branch 'master' into storage-read-query-plan 2020-10-12 13:12:39 +03:00
Amos Bird
867216103f
Extend trivial count optimization. 2020-10-08 18:08:17 +08:00
Amos Bird
c9ab87ed17
Fix explain syntax 2020-10-08 17:06:04 +08:00
Amos Bird
ac102540d0
Global WITH and WITH propagation. 2020-10-08 15:03:34 +08:00
Nikolai Kochetov
f9bf1e3406 Merge branch 'master' into storage-read-query-plan 2020-10-06 09:50:10 +03:00
Azat Khuzhin
587cde853e Avoid skipping unused shards twice (for query processing stage and read itself) 2020-10-02 22:42:09 +03:00
Nikolai Kochetov
bef96faa1a Tru fix tests. 2020-10-02 15:38:33 +03:00
hchen9
68888408da Pass context ptr to ref 2020-10-01 21:23:41 -07:00
hchen9
b3949db00f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into trivial_count_seq_consistency 2020-10-01 21:15:26 -07:00
hchen9
b4494578d7 Support select_sequential_consistency in StorageReplicatedMergeTree.totalRows and totalBytes 2020-09-30 16:50:58 -07:00
hchen9
a5ac39b564 Pass Context parameter for IStorage.totalRows and IStorage.totalBytes 2020-09-30 16:47:42 -07:00
Artem Zuikov
0f6a6453fe
Remove multiple_joins_rewriter v1 (#15472) 2020-09-30 23:11:49 +03:00
Nikolai Kochetov
576ffadb17 Fix explain for ISourceStep. 2020-09-30 15:22:30 +03:00
hchen9
fda6bde428 improve change 2020-09-28 17:05:22 -07:00
Nikolai Kochetov
dea90009e3 Fix build 2020-09-25 16:03:12 +03:00
Nikolai Kochetov
62ef40394a
Merge pull request #14856 from ClickHouse/query-plan-for-creating-sets 
Query plan for creating sets
 2020-09-22 16:19:20 +03:00
Nikolai Kochetov
54c1f04b38 Review fixes. 2020-09-21 11:36:12 +03:00
Alexey Milovidov
059646743b Remove OpenCL 2020-09-19 17:20:00 +03:00
alexey-milovidov
acbdc51460
Merge pull request #14771 from amosbird/withsub 
First attempt of supporting CTE
 2020-09-19 16:25:34 +03:00
Artem Zuikov
28afbafa08 fix crash in RIGHT or FULL JOIN switch 2020-09-18 19:25:20 +03:00
Nikolai Kochetov
b26f11c00c Support StorageDistributed::read for QueryPlan. 2020-09-18 17:16:53 +03:00
Nikolai Kochetov
2ffc2b6927 Refactor IStorage::read with query plan. 2020-09-18 15:10:59 +03:00
Nikolai Kochetov
50674a320f Refactor IStorage::read with query plan. 2020-09-18 14:39:07 +03:00
Nikolai Kochetov
21bd45fb22
Update InterpreterSelectQuery.cpp 2020-09-17 12:57:48 +03:00
roman
b41421cb1c [settings]: introduce new query complexity settings for leaf-nodes 
The new setting should allow to control query complexity on leaf nodes
excluding the final merging stage on the root-node. For example, distributed
query that reads 1k rows from 5 shards will breach the `max_rows_to_read=5000`,
while effectively every shard reads only 1k rows. With setting `max_rows_to_read_leaf=1500`
this limit won't be reached and query will succeed since every shard reads
not more that ~1k rows.
 2020-09-17 10:37:05 +01:00
Nikolai Kochetov
3cc89d467c Merge branch 'master' into query-plan-for-creating-sets 2020-09-15 20:15:41 +03:00
Nikolai Kochetov
118a8a513e Update mutations interpreter. 2020-09-15 20:13:13 +03:00
Nikolai Kochetov
e29f27e333
Merge pull request #14815 from ClickHouse/storage-read-query-plan 
Return query plan from storage.
 2020-09-15 18:34:11 +03:00
Nikolai Kochetov
4c783f19ee Use QueryPlan for SubqueryForSet. 2020-09-15 16:25:14 +03:00
Nikolai Kochetov
6eac433c55 Review fixes. 2020-09-15 13:40:39 +03:00
Artem Zuikov
51ba12c2c3
Try speedup build (#14809) 2020-09-15 12:55:57 +03:00
Nikolai Kochetov
bcfc83022a Return query plan from storage. 2020-09-14 17:13:58 +03:00
Amos Bird
882b2a3348
CTE 2020-09-13 09:39:17 +08:00
alexey-milovidov
f2e9a09b11
Merge pull request #14585 from ClickHouse/join-processor 
Remove Join from ExpressionActions
 2020-09-08 23:56:09 +03:00
Nikolai Kochetov
39aad9979a Remove Join from ExpressionActions. 2020-09-08 13:40:53 +03:00
Nikolai Kochetov
c761970bc5 Merge branch 'master' into pipes-and-creating-sets 2020-09-04 11:38:35 +03:00
Nikolai Kochetov
b7e53208d1 Fix tests. 2020-09-04 11:36:47 +03:00
Azat Khuzhin
10b4f3b41f Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key 
Previous set of QueryProcessingStage does not allow to do this.
But after WithMergeableStateAfterAggregation had been introduced the
following queries can be optimized too under
optimize_distributed_group_by_sharding_key:
- GROUP BY sharding_key LIMIT
- GROUP BY sharding_key LIMIT BY
- GROUP BY sharding_key ORDER BY

And right now it is still not supports:
- WITH TOTALS (looks like it can be supported)
- WITH ROLLUP (looks like it can be supported)
- WITH CUBE
- SETTINGS extremes=1 (looks like it can be supported)
But will be implemented separatelly.

vX: fixes
v2: fix WITH *
v3: fix extremes
v4: fix LIMIT OFFSET (and make a little bit cleaner)
v5: fix HAVING
v6: fix ORDER BY
v7: rebase against 20.7
v8: move out WithMergeableStateAfterAggregation
v9: add optimize_distributed_group_by_sharding_key into test names
 2020-09-03 00:52:51 +03:00
Azat Khuzhin
4043be3121 Add new query processing stage WithMergeableStateAfterAggregation 
Process query until the stage where the aggregate functions were
calculated and finalized.

It will be used for optimize_distributed_group_by_sharding_key.

v2: fix aliases
v3: Fix protocol ABI breakage due to WithMergeableStateAfterAggregation
    Conditions >= for QueryProcessingStage::Enum has been verified, and they
    are ok (in InterpreterSelectQuery).
 2020-09-03 00:51:08 +03:00
Nikolai Kochetov
acfd35a908 Update QueryPipeline::addCreatingSetsTransform 2020-09-02 16:04:54 +03:00