thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
92,468 Commits 9,326 Branches 1,783 Tags 2.1 GiB
c2ae762a5a
Commit Graph

6805 Commits

Author SHA1 Message Date
Nikolai Kochetov
5b4658aa5e Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-30 09:47:35 +00:00
Amos Bird
6525bfc4cd
Avoid context copy for InterpreterSelects 2022-05-30 17:08:12 +08:00
Azat Khuzhin
78bd47d8df Fix excessive LIST requests to coordinator for transactions 
In [1] there was only few transactions, but lots of List for /test/clickhouse/txn/log:

    $ clickhouse-local --format TSVWithNamesAndTypes --file zookeeper_log.tsv.gz -q "select * except('path|session_id|event_time|thread_id|event_date|xid') apply(x->groupUniqArray(x)), path, session_id, min(event_time), max(event_time), count() from table where has_watch and type = 'Request' group by path, session_id order by count() desc limit 1 format Vertical"
    Row 1:
    ──────
    groupUniqArray(type):             ['Request']
    groupUniqArray(query_id):         ['','62d75128-9031-48a5-87ba-aec3f0b591c6']
    groupUniqArray(address):          ['::1']
    groupUniqArray(port):             [9181]
    groupUniqArray(has_watch):        [1]
    groupUniqArray(op_num):           ['List']
    groupUniqArray(data):             ['']
    groupUniqArray(is_ephemeral):     [0]
    groupUniqArray(is_sequential):    [0]
    groupUniqArray(version):          []
    groupUniqArray(requests_size):    [0]
    groupUniqArray(request_idx):      [0]
    groupUniqArray(error):            []
    groupUniqArray(watch_type):       []
    groupUniqArray(watch_state):      []
    groupUniqArray(stat_version):     [0]
    groupUniqArray(stat_cversion):    [0]
    groupUniqArray(stat_dataLength):  [0]
    groupUniqArray(stat_numChildren): [0]
    groupUniqArray(children):         [[]]
    path:                             /test/clickhouse/txn/log
    session_id:                       1
    min(event_time):                  2022-05-27 12:54:09.025897
    max(event_time):                  2022-05-27 13:37:12.846314 <!-- last transaction was at 12:54, see server log
    count():                          3673675 <-- huge

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37593/2613149f6bf4f242bbbf2c3c8539b5176fd77286/stateless_tests__debug__actions__[1/3].html

Server log:

    $ pigz -cd clickhouse-server.log.gz | fgrep TransactionLog: | head -1
    2022.05.27 12:54:09.026852 [ 5018 ] {62d75128-9031-48a5-87ba-aec3f0b591c6} <Trace> TransactionLog: Loading 33 entries from /test/clickhouse/txn/log: csn-0000000000..csn-0000000032
    $ pigz -cd clickhouse-server.log.gz | fgrep TransactionLog: | tail -1
    2022.05.27 12:54:58.909222 [ 509 ] {} <Test> TransactionLog: Closing readonly transaction (177, 38, 41b51ff1-bcba-43bf-bcea-e97ad05f6040)
    $ pigz -cd clickhouse-server.log.gz | fgrep 62d75128-9031-48a5-87ba-aec3f0b591c6 | tail -1
    2022.05.27 12:54:09.064857 [ 5018 ] {62d75128-9031-48a5-87ba-aec3f0b591c6} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.

Fixes: #37398 (cc @tavplubix)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-30 11:39:34 +03:00
taiyang-li
0b0d38b18c Merge branch 'master' into fix_settings 2022-05-30 10:07:09 +08:00
Alexey Milovidov
73e2e63414
Merge pull request #37612 from ClickHouse/clang-tidy-14 
Fix clang-tidy-14, part 1
 2022-05-29 03:16:32 +03:00
Alexander Tokmakov
4e52f45695 Merge branch 'master' into fix_trash 2022-05-28 19:43:19 +02:00
Anton Popov
1d9b3be7da
Merge pull request #37536 from CurtizJ/profile-events-for-part-types 
Add profile events for introspection of part types
 2022-05-28 14:25:21 +02:00
Alexey Milovidov
c50791dd3b Fix clang-tidy-14, part 1 2022-05-27 22:52:14 +02:00
Nikolai Kochetov
b80b1940ce Fix some tests. 2022-05-27 20:47:35 +00:00
Alexander Tokmakov
7e659036f8 fix 2022-05-27 20:30:06 +02:00
Dmitry Novik
60b9d81773 Remove global_memory_usage_overcommit_max_wait_microseconds 2022-05-27 16:30:29 +00:00
Azat Khuzhin
8a224239c1 Prohibit optimize_aggregation_in_order with GROUPING SETS 
AggregatingStep ignores it anyway, and it leads to the following error
in getSortDescriptionFromGroupBy(), like in [1]:

    2022.05.24 04:29:29.279431 [ 3395 ] {26543564-8bc8-4a3a-b984-70a2adf0245d} <Fatal> : Logical error: 'Trying to get name of not a column: ExpressionList'.

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/36914/67d3ac72d26ab74d69f03c03422349d4faae9e19/stateless_tests__ubsan__actions_.html

v2: revert change to getSortDescriptionFromGroupBy() after
    GroupingSetsRewriterVisitor had been introduced
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-27 17:44:57 +03:00
Azat Khuzhin
1f29b0a901 Rewrite queries GROUPING SETS (foo, bar) to GROUP BY foo, bar 
This is better then introducing separate
SelectQueryExpressionAnalyzer::useGroupingSetKey(), since for
optimize_aggregation_in_order that method will not be enough, because
size of ManyExpressionActions will not match size of SortDescription, in
ReadInOrderOptimizer::ReadInOrderOptimizer()

And plus it is cleaner.

v2: fix clang-tidy
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-27 17:44:51 +03:00
alesapin
6d6779f17a
Merge pull request #37139 from ClickHouse/i_object_storage 
Separate object storage operations from disks
 2022-05-27 13:59:50 +02:00
Azat Khuzhin
2613149f6b Fix converting types for UNION queries (may produce LOGICAL_ERROR) 
CI founds [1]:

    2022.02.20 15:14:23.969247 [ 492 ] {} <Fatal> BaseDaemon: (version 22.3.1.1, build id: 6082C357CFA6FF99) (from thread 472) (query_id: a5187ff9-962a-4e7c-86f6-8d48850a47d6) (query: SELECT 0., round(avgWeighted(x, y)) FROM (SELECT toDate(toDate('214748364.8', '-922337203.6854775808', '-0.1', NULL) - NULL, 10.000100135803223, '-2147483647'), 255 AS x, -2147483647 AS y UNION ALL SELECT y, NULL AS x, 2147483646 AS y)) Received signal Aborted (6)

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/26d0e5438c86e52a145aaaf4cb523c399989a878/fuzzer_astfuzzerdebug,actions//report.html

The problem is that subqueries returns different headers:
- first query  -- x, y
- second query -- y, x

v2: Make order of columns strict only for UNION
    https://s3.amazonaws.com/clickhouse-test-reports/34775/9cc8c01a463d18c471853568b2f0af659a4e643f/stateless_tests__address__actions__[2/2].html
    Fixes: 00597_push_down_predicate_long
v3: add no-backward-compatibility-check for the test
Fixes: #37569
Resubmit: #34775
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit a813f5996e)
 2022-05-27 14:11:57 +03:00
zhanglistar
ca67e67a74 Fix a typo 2022-05-27 15:52:04 +08:00
Vxider
54d6f98122 flush and shutdown temporary table before drop 2022-05-27 04:50:36 +00:00
Dmitry Novik
3a9239b79f
Revert "RFC: Fix converting types for UNION queries (may produce LOGICAL_ERROR)" 2022-05-27 04:05:32 +02:00
Alexander Tokmakov
eb71dd4c78
Merge pull request #37547 from ClickHouse/followup_37398 
Follow-up to #37398
 2022-05-26 20:29:41 +03:00
Dmitry Novik
673a521d0b
Merge pull request #34775 from azat/union-type-cast 
RFC: Fix converting types for UNION queries (may produce LOGICAL_ERROR)
 2022-05-26 17:28:23 +02:00
Alexander Tokmakov
e8f33fb0d9 fix flaky tests 2022-05-26 14:17:05 +02:00
Nikolai Kochetov
bf95541531 Fixing style. 2022-05-26 11:09:36 +00:00
Nikolai Kochetov
84f97b53de Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-26 11:07:45 +00:00
Nikolai Kochetov
fea2401f1f
Merge pull request #37532 from ClickHouse/add-separate-mutex-for-factories-info 
Use a separate mutex for query_factories_info in Context.
 2022-05-26 13:03:28 +02:00
Dmitry Novik
16c6b60703 Introduce AggregationKeysInfo 2022-05-25 23:22:29 +00:00
Alexander Tokmakov
3d259fa0de
Update Context.cpp 2022-05-25 23:18:07 +03:00
alesapin
6f5c86e55e Merge branch 'master' into i_object_storage 2022-05-25 20:49:01 +02:00
Alexander Tokmakov
779e6ea0b9 make it better, fix on cluster queries 2022-05-25 20:17:49 +02:00
Azat Khuzhin
a813f5996e Fix converting types for UNION queries (may produce LOGICAL_ERROR) 
CI founds [1]:

    2022.02.20 15:14:23.969247 [ 492 ] {} <Fatal> BaseDaemon: (version 22.3.1.1, build id: 6082C357CFA6FF99) (from thread 472) (query_id: a5187ff9-962a-4e7c-86f6-8d48850a47d6) (query: SELECT 0., round(avgWeighted(x, y)) FROM (SELECT toDate(toDate('214748364.8', '-922337203.6854775808', '-0.1', NULL) - NULL, 10.000100135803223, '-2147483647'), 255 AS x, -2147483647 AS y UNION ALL SELECT y, NULL AS x, 2147483646 AS y)) Received signal Aborted (6)

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/26d0e5438c86e52a145aaaf4cb523c399989a878/fuzzer_astfuzzerdebug,actions//report.html

The problem is that subqueries returns different headers:
- first query  -- x, y
- second query -- y, x

v2: Make order of columns strict only for UNION
    https://s3.amazonaws.com/clickhouse-test-reports/34775/9cc8c01a463d18c471853568b2f0af659a4e643f/stateless_tests__address__actions__[2/2].html
    Fixes: 00597_push_down_predicate_long
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-25 20:31:47 +03:00
Nikolai Kochetov
1f9b1cf726 Fixing build. 2022-05-25 18:59:46 +02:00
Nikolai Kochetov
7b681fa8ac Fixing build. 2022-05-25 17:15:23 +02:00
Anton Popov
16e839ac71 add profile events for introspection of part types 2022-05-25 14:54:49 +00:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
Nikolai Kochetov
6370c29049 Use a separate mutex for query_factories_info in Context. 2022-05-25 14:16:59 +00:00
mergify[bot]
f49552d48e
Merge branch 'master' into grouping-sets-optimization-fix 2022-05-25 13:03:54 +00:00
Nikolai Kochetov
3d84aae0ab Better. 2022-05-24 20:06:08 +00:00
Igor Nikonov
04e2737a57
Merge pull request #37337 from ClickHouse/column_decl_null_before_default_value 
Column declaration: [NOT] NULL right after type 

+ fixed: data_type_default_nullable=true, it didn't make columns nullable
         if the column declaration contains default expression w/o type

Issue #37229
 2022-05-24 21:16:25 +02:00
Dmitry Novik
c4dc0f7cda Fix ORDER BY optimization in case of GROUPING SETS 2022-05-24 18:56:22 +00:00
Alexander Tokmakov
4618429201 fixes 2022-05-24 18:53:52 +02:00
Vitaly Baranov
497c70d786
Merge pull request #37269 from vitlibar/rework-access-control-notifications 
Rework AccessControl's notifications.
 2022-05-24 17:47:45 +02:00
Alexander Tokmakov
bb5f04efaf Merge branch 'master' into fix_trash 2022-05-24 17:45:51 +02:00
Vladimir C
0eaf22b370
Merge pull request #37453 from vdimir/join_auto_lc_nullable_bug 2022-05-24 16:28:23 +02:00
Vladimir C
bec4ae87c9
Merge pull request #37472 from amosbird/joinpushdown 2022-05-24 16:08:26 +02:00
Alexander Tokmakov
229d35408b
Merge pull request #37398 from ClickHouse/fixes_for_transactions 
Fixes for transactions
 2022-05-24 15:28:01 +03:00
Alexander Tokmakov
dbde63d275 fixes 2022-05-24 14:24:03 +02:00
Amos Bird
76ddb39d02
refactor format 2022-05-24 12:09:00 +08:00
taiyang-li
c7e68d664d fix building 2022-05-24 10:58:45 +08:00
Igor Nikonov
838a6c6f61 Column declaration: [NOT] NULL right after type 
+ fixed: data_type_default_nullable=true, it didn't make columns nullable
         if the column declaration contains default expression w/o type

Issue #37229
 2022-05-23 22:03:45 +00:00
mergify[bot]
51ff49a0ee
Merge branch 'master' into i_object_storage 2022-05-23 20:29:49 +00:00
Alexander Tokmakov
3f44f34fe1 review suggestions 2022-05-23 21:55:17 +02:00
Nikolai Kochetov
fd97a9d885 Move some resources 2022-05-23 19:47:32 +00:00
Alexander Tokmakov
d0f998fb88 Merge branch 'master' into fix_trash 2022-05-23 21:25:56 +02:00
Alexander Tokmakov
fa21121f77 fix 2022-05-23 21:17:52 +02:00
Alexander Tokmakov
a262492cc1 slightly better 2022-05-23 20:53:33 +02:00
Alexander Tokmakov
6bc68c0cbc Merge branch 'master' into fixes_for_transactions 2022-05-23 18:49:21 +02:00
Vladimir C
14a820c2a9
Merge pull request #37336 from vdimir/fix_clash_const_aggegate_join 2022-05-23 17:09:30 +02:00
Nikolai Kochetov
9756b759c6 Move some resources 2022-05-23 13:46:57 +00:00
vdimir
dabb150a95
Fix cast lowcard of nullable in JoinSwitcher 2022-05-23 13:05:59 +00:00
Antonio Andelic
87d445295f
Merge pull request #37342 from azat/projections-optimize_aggregation_in_order-fix 
Fix projections with GROUP/ORDER BY in query and optimize_aggregation_in_order
 2022-05-23 12:29:16 +02:00
taiyang-li
c120b19802 fix building 2022-05-23 17:42:20 +08:00
taiyang-li
f13dcad58c Merge branch 'master' into fix_settings 2022-05-23 17:36:45 +08:00
mergify[bot]
747aa5575c
Merge branch 'master' into remove-useless-code-2 2022-05-22 17:41:57 +00:00
alesapin
06c3dd69c0 Move directories 2022-05-22 13:51:48 +02:00
alesapin
c8d92b87c8 Merge branch 'master' into i_object_storage 2022-05-22 12:16:10 +02:00
Robert Schulze
19d53c14fa
Merge pull request #37382 from ClickHouse/wc++98-compat-extra-semi 
Enable -Wc++98-compat-extra-semi
 2022-05-22 09:40:45 +02:00
Alexey Milovidov
6322d064b8
Merge pull request #37384 from Avogar/short-circuit-process 
Try to improve short circuit functions processing
 2022-05-21 23:43:56 +03:00
Alexey Milovidov
d1678c9662
Update ExpressionActions.cpp 2022-05-21 23:43:26 +03:00
Vitaly Baranov
58f4a86ec7 Rework notifications used in access management. 2022-05-21 10:15:39 +02:00
Alexander Tokmakov
1b453f517d fix 2022-05-21 02:32:35 +02:00
Alexander Tokmakov
c48410b574 fix trash in my code 2022-05-21 02:05:02 +02:00
Alexander Tokmakov
44f2d4529a better waiting, add fault injection 2022-05-20 22:10:01 +02:00
Nikolai Kochetov
56feef01e7 Move some resources 2022-05-20 19:49:31 +00:00
Alexey Milovidov
c77b078c2c
Merge pull request #37367 from ClickHouse/grouping-sets-with-rollup-cube 
Disable WITH ROLLUP/CUBE for GROUPING SETS
 2022-05-20 22:34:56 +03:00
Alexander Tokmakov
1027798517 handle connection loss on commit 2022-05-20 17:35:29 +02:00
Robert Schulze
0f6715bd91
Follow-up to PR #37300: semicolon warnings 
In PR #37300, Alexej asked why we the compiler does not warn about
unnecessary semicolons, e.g.

  f()
  {
  }; // <-- here

The answer is surprising: In C++98, above syntax was disallowed but by
most compilers accepted it regardless. C++>11 introduced "empty
declarations" which made the syntax legal.

The previous behavior can be restored using flag
-Wc++98-compat-extra-semi. This finds many useless semicolons which were
removed in this change. Unfortunately, there are also false positives
which would require #pragma-s and HAS_* logic (--> check_flags.cmake) to
suppress. In the end, -Wc++98-compat-extra-semi comes with extra effort
for little benefit. Therefore, this change only fixes some semicolons
but does not enable the flag.
 2022-05-20 15:06:34 +02:00
avogar
22b072bc7f Make better 2022-05-20 13:04:43 +00:00
avogar
3884819bd1 Fix 2022-05-20 12:48:58 +00:00
vdimir
339980469f
Revert "Swap order of converting_join_columns and before_join steps" 
This reverts commit 2995b69f4a.
 2022-05-20 11:31:59 +00:00
vdimir
23a85d3406
Perserve constness in ExpressionActionsChain::JoinStep 2022-05-20 11:31:58 +00:00
avogar
5d19150906 Try to improve short circuit functions processing 2022-05-20 10:47:28 +00:00
Alexander Tokmakov
12bbb7de87 fix race on TID allocation 2022-05-20 12:41:44 +02:00
alesapin
654b27e307 Merge branch 'master' into i_object_storage 2022-05-20 11:56:13 +02:00
Robert Schulze
b475fbc9a7
Merge pull request #37300 from ClickHouse/cmake-cleanup-pt3 
Various cmake cleanups
 2022-05-20 10:02:36 +02:00
alesapin
92c15ec97c Get rid of IDiskRemote 2022-05-19 20:07:15 +02:00
Dmitry Novik
b3ccf96c81 Merge remote-tracking branch 'origin/master' into grouping-function 2022-05-19 17:58:33 +00:00
Dmitry Novik
d4c66f4a48 Code cleanup & fix GROUPING() with TOTALS 2022-05-19 16:36:51 +00:00
Dmitry Novik
86d48e1c99 Disable WITH ROLLUP/CUBE for GROUPING SETS 2022-05-19 14:10:04 +00:00
Alexander Tokmakov
a53cfa9fca
Merge pull request #37355 from ClickHouse/revert-37266-fix-mutations-with-object 
Revert "Fix mutations in tables with columns of type `Object`"
 2022-05-19 16:27:22 +03:00
Kseniia Sumarokova
24d0899cb7
Merge pull request #36945 from Vxider/wv-populate 
POPULATE support for WindowView
 2022-05-19 13:47:52 +02:00
Kseniia Sumarokova
791cc5ced1
Merge pull request #37290 from azat/query-kind-client 
Add ability to pass QueryKind via clickhouse-client/local (useful for debugging)
 2022-05-19 13:22:42 +02:00
Alexander Tokmakov
f787dc7097
Revert "Fix mutations in tables with columns of type Object" 2022-05-19 13:24:48 +03:00
Azat Khuzhin
e32b695775 Fix projections with GROUP/ORDER BY in query and optimize_aggregation_in_order 
With projections, GROUP BY/ORDER BY in query, optimize_aggregation_in_order,
GROUP BY's InputOrderInfo was used incorrectly for ORDER BY.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-19 13:11:22 +03:00
Vitaly Baranov
16c8e93dd0
Merge pull request #36941 from nvartolomei/nv/show-grants-current-roles 
Allow a user to inspect grants from granted roles
 2022-05-19 09:20:52 +02:00
Azat Khuzhin
54a6aaef62 Log query processing stage in executeQuery() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-19 07:19:26 +03:00
Azat Khuzhin
dea1706d4e
Fix GROUP BY AggregateFunction (#37093) 
* Fix GROUP BY AggregateFunction

finalizeChunk() was unconditionally converting AggregateFunction to the
underlying type, however this should be done only if the aggregate was
applied.

So pass names of aggregates as an argument to the finalizeChunk()

Fuzzer report [1]:

    Logical error: 'Bad cast from type DB::ColumnArray to DB::ColumnAggregateFunction'. Received signal 6 Received signal Aborted (6)

For the following query:

    SELECT
        arraySort(groupArrayArray(grp_simple)),
        grp_aggreg,
        arraySort(groupArrayArray(grp_simple)),
        b,
        arraySort(groupArrayArray(grp_simple)) AS grs
    FROM data_02294
    GROUP BY
        a,
        grp_aggreg,
        b
    SETTINGS optimize_aggregation_in_order = 1

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37050/323ae98202d80fc4b311be1e7308ef2ac39e6063/fuzzer_astfuzzerdebug,actions//fuzzer.log

v2: fix conflicts in src/Interpreters/InterpreterSelectQuery.cpp
v3: Fix header for GROUP BY AggregateFunction WITH TOTALS
v4: Add sanity check into finalizeBlock()
v5: Use typeid_cast<&> to get more sensible error in case of bad cast (as suggested by @nickitat)
v6: Fix positions passed to finalizeChunk()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Core/ColumnNumbers.h: remove unused <string>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Optimize finalizeChunk()/finalizeBlock()

v2: s/ByPosition/Mask/ s/by_position/mask/
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-18 23:37:43 +02:00
Alexander Tokmakov
3bc39f9929 fixes 2022-05-18 21:57:20 +02:00
vdimir
2995b69f4a
Swap order of converting_join_columns and before_join steps 2022-05-18 19:31:14 +00:00
Vitaly Baranov
9ccddc44c6
Merge pull request #35767 from azat/ON_CLUSTER-grant 
RFC: Add separate CLUSTER grant
 2022-05-18 19:19:54 +02:00
Alexander Tokmakov
9772924d06 Merge branch 'master' into fix_trash 2022-05-18 17:27:54 +02:00
Dmitry Novik
6356112a76 Refactor GROUPING function 2022-05-18 15:23:31 +00:00
Alexander Tokmakov
19afeda4b1 fix style 2022-05-18 14:23:52 +02:00
Alexander Tokmakov
538b5cd536 fix more trash 2022-05-18 14:06:52 +02:00
Vladimir C
7d88b81622
Merge pull request #37193 from azat/fix-limit-by-in-dist-queries 
Fix "Cannot create column of type Set" for distributed queries with LIMIT BY
 2022-05-18 13:28:19 +02:00
Anton Popov
4def736ecb
Merge pull request #37275 from ernado/fix/open-telemetry-span-log-fix-alias 
Fix attribute.values alias for OTEL table
 2022-05-18 13:18:07 +02:00
Nicolae Vartolomei
32b9959dec Allow a user to inspect grants from granted roles 2022-05-18 10:44:18 +00:00
Vladimir C
1779fb08ed
Merge pull request #36271 from yuuch/support_collation_in_create_query 2022-05-18 12:43:49 +02:00
Vladimir C
f979a03fbe
Merge pull request #37256 from vdimir/issue_37217 2022-05-18 12:42:47 +02:00
alesapin
e99c90118e Merge branch 'master' into i_object_storage 2022-05-18 12:36:22 +02:00
Kruglov Pavel
aaace46da2
Merge pull request #37266 from CurtizJ/fix-mutations-with-object 
Fix mutations in tables with columns of type `Object`
 2022-05-18 12:19:55 +02:00
Dmitry Novik
a82239984c Remove include 2022-05-18 01:44:51 +00:00
Alexander Tokmakov
5590adeffa fix more trash 2022-05-17 23:04:24 +02:00
Dmitry Novik
a97ba9741e Fix possible deadlock in OvercommitTracker during logging 2022-05-17 18:07:52 +00:00
Alexander Tokmakov
dea39d8175 fix some trash 2022-05-17 18:22:52 +02:00
Vitaly Baranov
f34a5cdee2
Merge branch 'master' into ON_CLUSTER-grant 2022-05-17 13:21:20 +02:00
taiyang-li
8228bbc36f disable normarlize functions when query is secondary 2022-05-17 16:22:31 +08:00
taiyang-li
9f3dadf17c Merge branch 'master' into fix_settings 2022-05-17 16:19:29 +08:00
Vxider
f74c934a61 Merged with master 2022-05-17 02:53:59 +00:00
Aleksandr Razumov
60693afb56
Fix attribute.values alias for OTEL table 2022-05-17 01:28:36 +03:00
Robert Schulze
43945cea1b
Fixing some warnings 2022-05-16 20:59:27 +02:00
Anton Popov
b6c5ab4fcf fix mutations in tables with columns of type Object 2022-05-16 18:26:53 +00:00
Dmitry Novik
e5b395e054 Support ROLLUP and CUBE in GROUPING function 2022-05-16 17:33:38 +00:00
Alexander Tokmakov
712a04d69c Revert "Removed an unnecessary check and extra logging" 
This reverts commit 59ed1f9d8e.
 2022-05-16 18:56:00 +02:00
alesapin
ce6d5f7dcc Merge branch 'master' into i_object_storage 2022-05-16 17:42:22 +02:00
Dmitry Novik
dd1e7b55b8
Merge pull request #37050 from azat/fix-optimize_aggregation_in_order-prefix-Array 
Fix optimize_aggregation_in_order with prefix GROUP BY and *Array aggregate functions
 2022-05-16 17:17:56 +02:00
vdimir
76852ae49c
Pass need_filter, has_null_map to joinRightColumns 2022-05-16 14:27:27 +00:00
Robert Schulze
ed7df7cabd
Merge pull request #36960 from ClickHouse/clangtidies 
Enable clangtidies
 2022-05-16 14:22:50 +02:00
alesapin
1829256ae2
Merge pull request #37178 from ClickHouse/ignore_auto_increment_in_create_table_2 
Ignore AUTO_INCREMENT in CREATE TABLE to simplify migration from MySQL
 2022-05-16 13:29:22 +02:00
Azat Khuzhin
01e1c5345a Add separate CLUSTER grant 
In case you have different roles for the same user on multiple clusters,
ON CLUSTER query can help to overcome some limitations.

Consider the following example:
- cluster_with_data, dev_user (readonly=2)
- stage_cluster, dev_user (readonly=0)

So when you will execute the following query from stage_cluster, it will
be successfully executed, since ON CLUSTER queries has different system
profile:

    DROP DATABASE default ON CLUSTER cluster_with_data

This is not 100% safe, but at least something.

Note, that right now only ON CLUSTER query it self is supported, but
separate clusters are not (i.e. GRANT CLUSTER some_cluster_name TO
default), since right now grants sticked to database+.

v2: on_cluster_queries_require_cluster_grant
v3: fix test and process flags as bit mask
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-16 13:57:45 +03:00
alesapin
b10f26e3ed Fix style 2022-05-16 12:25:25 +02:00
Vladimir C
75008d5903
Merge pull request #37166 from vdimir/issue_37141 2022-05-16 12:21:41 +02:00
Robert Schulze
d66dcdad79
Fix new occurrences of new clang-tidy warnings 2022-05-16 11:31:36 +02:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies 2022-05-16 10:12:50 +02:00
Vasily Nemkov
59ed1f9d8e Removed an unnecessary check and extra logging 2022-05-16 09:29:54 +03:00
Alexey Milovidov
8334329dff
Merge pull request #37023 from ClickHouse/chained-or 
Merging #34932.
 2022-05-15 04:32:31 +03:00
Vxider
f2ef27e5d6 rename windowview header 2022-05-14 09:21:54 +00:00
Dmitry Novik
6fc7dfea80 Support ordinary GROUP BY 2022-05-13 23:04:12 +00:00
alesapin
cc304a3178 Merge branch 'master' into i_object_storage 2022-05-13 23:44:00 +02:00
Nikolai Kochetov
0a715b26db Move some resources. 2022-05-13 20:02:28 +00:00
Dmitry Novik
efb30bdf64 Correctly use __grouping_set_map column 2022-05-13 18:20:12 +00:00
Azat Khuzhin
b29a0f3c84 Fix "Cannot create column of type Set" for distributed queries with LIMIT BY 
The problem was that after #36454 it requires too much columns.

Fixes: #36454
Fixes: #37045
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 20:03:19 +03:00
Azat Khuzhin
751a85521e Fill possible WITH FILL error in SelectQueryExpressionAnalyzer::appendOrderBy() 
If order_by_keys was filled by the code above then WITH FILL will
incorrectly handle constant columns.

Fixes: #16618
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 20:03:19 +03:00
alesapin
eba60ff38f Fix restorable schema 2022-05-13 17:00:47 +02:00
Dmitry Novik
ae81268d4d Try to compute helper column lazy 2022-05-13 14:55:50 +00:00
Igor Nikonov
74be0946ff Test fixes 2022-05-13 13:33:45 +00:00
Vasily Nemkov
fdd42f19ff Less obscure message on destroying Session 2022-05-13 15:27:59 +03:00
Igor Nikonov
8d5cabc8f7 Fix style warnings and review comments 2022-05-13 11:27:17 +00:00
vdimir
976a274f91
Add comment for visiting children of select in ApplyWithGlobalVisitor 2022-05-13 10:53:26 +00:00
Vasily Nemkov
2e669f69f4 Added checks to validate that authentication happens before creating query or session context; 
Updated logging of user_id, removed unnecessary precautions
Explicitly stating in Session d-tor if user was not authenticated
 2022-05-13 13:46:24 +03:00
vdimir
dc2a90a344
ApplyWithGlobalVisitor for WITH in subquery 2022-05-13 10:40:51 +00:00
Kseniia Sumarokova
6d5137879b
Merge pull request #37054 from kssenii/better-read-from-cache 
Better read from cache buffer
 2022-05-13 12:20:06 +02:00
Alexander Tokmakov
432f1d570c
Merge pull request #37153 from azat/fix-optimize_skip_unused_shards_rewrite_in 
Fix implicit cast for optimize_skip_unused_shards_rewrite_in
 2022-05-13 12:58:28 +03:00
Igor Nikonov
f423e9bf40 Ignore AUTO_INCREMENT in CREATE TABLE to simplify migration from MySQL 
Issue #35893
 2022-05-13 09:22:52 +00:00
Kseniia Sumarokova
320efb2221
Merge branch 'master' into better-read-from-cache 2022-05-12 23:35:39 +02:00
Maksim Kita
2bf6d9c028
Merge pull request #37154 from azat/context-const 
Add const qualifier for few methods from Context
 2022-05-12 22:20:39 +02:00
Kseniia Sumarokova
6e0a910fcd
Merge branch 'master' into better-read-from-cache 2022-05-12 19:44:18 +02:00
Dmitry Novik
c5b40a9c91 WIP on GROUPING function 2022-05-12 16:40:26 +00:00
alesapin
e7296a2b28
Merge pull request #36825 from kssenii/cache-fix-1 
Removed forceful drop cache command, fix detached status state
 2022-05-12 17:18:21 +02:00
Vxider
dc33c3a9bb Merged with master 2022-05-12 23:04:07 +08:00
Azat Khuzhin
a0ba113e5e Mark Context::getInterserverIOHandler() as const 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-12 16:54:01 +03:00
Azat Khuzhin
a04b6ad314 Mark Context::getInterserverCredentials() as const 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-12 16:54:01 +03:00
Azat Khuzhin
586eda5242 Fix implicit cast for optimize_skip_unused_shards_rewrite_in 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Fixes: #37073
 2022-05-12 16:17:30 +03:00
Vladimir C
4cb6ca2311
Merge pull request #36970 from ClickHouse/concurrent_join 
Follow up for ConcurrentHashJoin
 2022-05-12 15:09:29 +02:00
kssenii
483e056bcc Merge master 2022-05-12 14:58:24 +02:00
Dmitry Novik
df071bb701
Merge pull request #37129 from ClickHouse/enable-overcommit-for-user 
Update documentation and defaults for memory overcommit
 2022-05-12 14:38:43 +02:00
mergify[bot]
7ffad69019
Merge branch 'master' into wv-populate 2022-05-12 11:52:04 +00:00
Kseniia Sumarokova
59bfcd917b
Merge pull request #36876 from kssenii/extend-cache-log 
Log into filesystem_cache_log when cache not even attempted
 2022-05-12 12:38:36 +02:00
Nikolai Kochetov
55188665f4
Merge pull request #37071 from ClickHouse/join_dup_columns_issue_contin 
Continue fixing #36199.
 2022-05-12 11:36:15 +02:00
mergify[bot]
f4bae3969d
Merge branch 'master' into enable-overcommit-for-user 2022-05-11 20:04:20 +00:00
Maksim Kita
b1c154ccb4 Fixed style check 2022-05-11 21:59:51 +02:00
Maksim Kita
75555c436b Fix usage of min_count_to_compile_sort_description setting 2022-05-11 21:59:51 +02:00
Maksim Kita
ea8ce3140a Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
4e7d10297b Fixed style 2022-05-11 21:59:51 +02:00
Maksim Kita
ce92f6aab1 Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
b3b46d2e5c Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
8ceb63ee6c Added JIT compilation of SortDescription 2022-05-11 21:59:51 +02:00
Nikolai Kochetov
ec34761d9f
Merge pull request #33631 from ClickHouse/grouping-sets-fix 
Support GROUPING SETS
 2022-05-11 21:28:46 +02:00
Dmitry Novik
17608b3d93 Update documentation and defaults for memory overcommit 2022-05-11 16:18:41 +00:00
Vladimir C
641e2be6ef
Merge pull request #37085 from vdimir/force-cross-to-inner-option 
Option to force cross_to_inner_join_rewrite
 2022-05-11 18:03:53 +02:00
Kruglov Pavel
af3db5a992
Merge pull request #36726 from PolyProgrammist/s3clustermacro 
Handle cluster macro in s3Cluster table function
 2022-05-11 18:00:57 +02:00
Nikolai Kochetov
2d99f0ce13 Simplify code a little bit. 2022-05-11 12:16:15 +00:00
vdimir
2f5959aca6
Upd cross_to_inner_join_rewrite setting, add test 2022-05-11 10:56:43 +00:00
Vxider
e4f43d0b01 update getSampleBlock for windowview 2022-05-11 08:47:08 +00:00
kssenii
18dd74be0c Remove unneede changes 2022-05-11 10:26:30 +02:00
Alexey Milovidov
c0acd5a325 Merge branch 'chained-or' of github.com:ClickHouse/ClickHouse into chained-or 2022-05-11 01:29:22 +02:00
Alexey Milovidov
3fcf63830c Merge branch 'master' into chained-or 2022-05-11 01:28:12 +02:00
Nikolai Kochetov
f7dbd48ee5 Simplify code a little bit. 2022-05-10 16:12:03 +00:00
Nikolai Kochetov
a02e1d2f4a Simplify code a little bit. 2022-05-10 16:00:00 +00:00
vdimir
00cd21cacf
Option to force cross_to_inner_join_rewrite 2022-05-10 15:12:17 +00:00
vdimir
9b24e0d260
Apply optimizeCountConstantAndSumOne before JOINs 2022-05-10 14:30:33 +00:00
vdimir
ff001382fb
Handle non unique column names in NotJoinedBlocks 2022-05-10 14:30:33 +00:00
vdimir
837d4bd13f
Revert "Continue fixing #36199." 
This reverts commit 1feba64a4e.
 2022-05-10 14:30:02 +00:00
Nikolai Kochetov
1feba64a4e
Continue fixing #36199. 2022-05-10 14:26:49 +00:00
kssenii
f4b45a5805 Better profile events 2022-05-10 16:00:53 +02:00
Vladimir C
4065b83ba9
Update ConcurrentHashJoin.cpp 2022-05-10 14:14:58 +02:00
Vladimir C
43935f71f7
Update ConcurrentHashJoin.cpp 2022-05-10 14:09:49 +02:00
Anton Popov
f075070fdf
Merge pull request #37046 from azat/fix-optimize_aggregation_in_order-Array 
Fix optimize_aggregation_in_order with *Array aggregate functions
 2022-05-10 14:01:01 +02:00
Anton Popov
aec30c4076
Merge pull request #37053 from CurtizJ/remove-streams-comments 
Remove last mentions of data streams
 2022-05-10 13:38:13 +02:00
Vasily Nemkov
0cbc9c18e2
No debug messge if user not set 2022-05-10 13:39:23 +03:00
Vxider
5d0a5d34c8 removing function populate in windowview 2022-05-10 15:06:58 +08:00
taiyang-li
fd878f7e7b change as requested 2022-05-10 12:06:50 +08:00
Alexey Milovidov
5d8fad8ab6
Update ConvertFunctionOrLikeVisitor.cpp 2022-05-10 02:23:24 +03:00
kssenii
2f0f76369a Better 2022-05-09 21:25:26 +02:00
Anton Popov
e911900054 remove last mentions of data streams 2022-05-09 19:15:24 +00:00
Azat Khuzhin
323ae98202 Fix optimize_aggregation_in_order with prefix GROUP BY and *Array aggregate functions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Fixes: #35111
See-also: #37046
 2022-05-09 21:32:40 +03:00
Azat Khuzhin
6ada8a6337 Fix optimize_aggregation_in_order with *Array aggregate functions 
row_begin was wrong, and before this patch aggregator processing
{row_end, row_end} range, in other words, zero range.

Fixes: #9113 (cc @dimarub2000)
v2: add static_cast to fix UBSan
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-09 20:43:19 +03:00
mergify[bot]
55a6d22ad3
Merge branch 'master' into grouping-sets-fix 2022-05-09 14:02:10 +00:00
Nikolai Kochetov
a34411f51e Disable trivial_insert_select if query has aggregate functions. 
(cherry picked from commit 70ae043455)
 2022-05-09 15:56:39 +02:00
Vasily Nemkov
de9a06df5b
Merge branch 'master' into fix_session_log_crash 2022-05-09 17:00:24 +04:00
Vasily Nemkov
58953b5f28 Fixed issues with concurrent access to User object 
ContextAccess might be updated from another thread, nullifying UserPtr.
Can happen if user was dropped in concurrent query.

This could crash server under certain conditions.
 2022-05-09 15:56:50 +03:00
Alexander Tokmakov
b6be194d3a
Merge pull request #36976 from ClickHouse/replicated_database_fixes 
Fixes for Replicated database
 2022-05-09 14:41:49 +03:00
Nikolai Kochetov
ebfeca2c86 Try to move some code from SourceWithProgress to PipelineExecutor. 2022-05-09 10:28:05 +00:00
Alexey Milovidov
fec7447ab4
Update ConvertFunctionOrLikeVisitor.cpp 2022-05-09 06:16:10 +03:00
Alexey Milovidov
56abce6a75 Merge branch 'chained_or' of github.com:danlark1/ClickHouse into chained-or 2022-05-08 23:37:09 +02:00
Robert Schulze
61cbcbf073
Enable clang-tidy readability-misleading-indentation 
Official docs:

  Correct indentation helps to understand code. Mismatch of the
  syntactical structure and the indentation of the code may hide serious
  problems.
 2022-05-08 19:12:01 +02:00
Vxider
34f1821eb9 function now support for windowview populate 2022-05-08 23:43:08 +08:00
Vxider
e7c4eb2697 Merged with master 2022-05-08 18:15:31 +08:00
Alexey Milovidov
2b88c0898a
Update ConvertFunctionOrLikeVisitor.cpp 2022-05-07 14:11:06 +03:00
Kseniia Sumarokova
0f5f75a784
Merge pull request #36887 from kssenii/fix-current-size-count-in-cache 
Fix current size count in cache
 2022-05-07 11:45:20 +02:00
Dmitry Novik
8d59211f01 Remove redundant ActionType 2022-05-06 18:50:21 +00:00
Alexander Tokmakov
c806869040 fix sync database replica 2022-05-06 18:37:20 +02:00
vdimir
d712985575
Minor style changes for ConcurrentHashJoin 2022-05-06 15:17:46 +00:00
Vladimir C
bd5fab97d9
Merge pull request #36415 from bigo-sg/concurrent_join 2022-05-06 17:11:10 +02:00
Anton Popov
13e8db6299
Merge pull request #36762 from CurtizJ/dynamic-columns-12 
Fix insertion to columns of type `Object` from multiple files
 2022-05-06 14:14:32 +02:00
kssenii
32d004017b Merge origin 2022-05-06 13:54:51 +02:00
kssenii
fd1ab053f8 Merge master 2022-05-06 12:50:33 +02:00
Dmitry Novik
b4e1dbc3c8 Remove whitespaces 2022-05-05 18:35:41 +00:00
Dmitry Novik
9a251e0028 Cleanup code 2022-05-05 18:13:00 +00:00
Dmitry Novik
4cc26aa38b Merge remote-tracking branch 'origin/master' into grouping-sets-fix 
And fix execution of the query with only one grouping set
 2022-05-05 17:14:52 +00:00
Vxider
9eea003d69 Merged with master 2022-05-06 00:12:17 +08:00
mergify[bot]
2a8aad42dc
Merge branch 'master' into add-asynchronous-metrics-for-filesystem-cache 2022-05-05 15:43:38 +00:00
Vxider
a7dc1f48e5 populate support for windowview 2022-05-05 23:31:15 +08:00
Dmitry Novik
161f52292b Support distributed queries 2022-05-05 13:56:16 +00:00
Vitaly Baranov
dd51265218
Merge pull request #36864 from vitlibar/backup-improvements-4 
Backup improvements
 2022-05-05 15:37:51 +02:00
Anton Popov
fa1feb238e
Merge pull request #36853 from qoega/add-other-time-microsecons 
Add Other Query Time Microseconds Profile Event
 2022-05-05 13:58:13 +02:00
lgbo-ustc
595ba6bd9f udpate dispatchBlock() 2022-05-05 12:04:11 +08:00
lgbo-ustc
f9f241e566 fixed fast test. remove unused header 2022-05-05 11:16:33 +08:00
lgbo-ustc
29a8db1345 fixed fast test 2022-05-05 10:32:19 +08:00
lgbo-ustc
2ddccdc21c update codes 2022-05-05 10:02:50 +08:00
alesapin
aba78e5ed0
Merge pull request #36859 from ClickHouse/revert-36858-revert-35637-memory-overcommit-free 
Revert "Revert "Memory overcommit: continue query execution if memory is available""
 2022-05-04 23:43:00 +02:00
Kruglov Pavel
77e55c344c
Merge pull request #36667 from Avogar/mysqldump-format 
Add MySQLDump input format
 2022-05-04 19:49:48 +02:00
Robert Schulze
c48d9a4174
Merge pull request #36844 from rschu1ze/shared_ptr_helper3 
Remove inherited create() method + disallow copying of storages
 2022-05-04 19:16:30 +02:00
Robert Schulze
78cdb3ec64
Merge pull request #36862 from rschu1ze/new-clangtidies 
Activated a bunch of LLVM 12/13/14 clang-tidy warnings
 2022-05-04 15:26:57 +02:00
Kseniia Sumarokova
f11cfdca94
Merge pull request #36874 from kssenii/add-size-to-system-cache-log 
Add column to system.filesystem_cache_log
 2022-05-04 11:53:29 +02:00
Robert Schulze
45c0ec44b0
Fix build, pt. II 2022-05-04 10:15:25 +02:00
mergify[bot]
64084b5e32
Merge branch 'master' into shared_ptr_helper3 2022-05-03 20:46:16 +00:00
mergify[bot]
17aecac7ff
Merge branch 'master' into new-clangtidies 2022-05-03 19:44:01 +00:00
kssenii
8f99046100 Add come asynchronous metrics 2022-05-03 20:02:19 +02:00
kssenii
0266cdf125 Add entries to cache log when cache was not event attempted 2022-05-03 18:29:41 +02:00
Nikolai Kochetov
35095191eb Merge branch 'master' into refactor-something-in-part-volumes 2022-05-03 17:51:47 +02:00
mergify[bot]
ddd1c3e1b3
Merge branch 'master' into add-other-time-microsecons 2022-05-03 15:45:40 +00:00
Dmitry Novik
5ba7a55c18
Merge pull request #36650 from bigo-sg/hive_text_parallel_parsing 
Parallel parsing of hive text format
 2022-05-03 15:56:28 +02:00
kssenii
000f2c9f7a Better 2022-05-03 15:55:36 +02:00
Maksim Kita
36d1c8238a
Merge pull request #36766 from kitaisreal/evaluate-constant-expression-fix 
Fix evaluateConstantExpression for subqueries
 2022-05-03 12:32:14 +02:00
Vitaly Baranov
5257ce31f8 Improved using ThreadPool for making backup or restoring, changed columns in system.backups. 2022-05-03 11:03:13 +02:00
Vitaly Baranov
409edfd3fa Rework RestoreCoordination: make restore deterministic. 2022-05-03 11:01:44 +02:00
Vitaly Baranov
2c754f44fc Make calculation of shard_num & replica_num not dependant on match of the cluster's definitions on nodes. 2022-05-03 11:01:44 +02:00
Vitaly Baranov
2a645bb187 Fix sending 'create_table' and 'create_database' restore settings to cluster. 2022-05-03 11:01:44 +02:00
Vitaly Baranov
bddec55d35 Added ASTBackupQuery::setDatabase(). 2022-05-03 11:01:44 +02:00
Robert Schulze
0a4eccb73e
Activated a bunch of LLVM 12/13/14 clang-tidy warnings 
Omitted new checks which produce too many matches or which are
controversial (e.g. readability-identifier-length).

New checks:

- misc-misleading-bidirectional + misc-misleading-identifier

  Detects potential attack as described in the Trojan Source attack

- modernize-macro-to-enum

  Replaces groups of adjacent macros with an unscoped anonymous enum

- modernize-shrink-to-fit

  Replace copy and swap tricks on shrinkable containers with the
  shrink_to_fit() method call

- modernize-use-transparent-functors

  Prefer transparent functors to non-transparent ones

- modernize-use-uncaught-exceptions

  This check will warn on calls to std::uncaught_exception and replace
  them with calls to std::uncaught_exceptions (uncaught_exception was
  deprecated with C++17)

- performance-no-int-to-ptr

  Diagnoses every integer to pointer cast

- readability-duplicate-include

  Looks for duplicate includes and removes them

- readability-redundant-preprocessor

  Finds potentially redundant preprocessor directives

- bugprone-lambda-function-name

  Checks for attempts to get the name of a function from within a lambda
  expression

- bugprone-redundant-branch-condition

  Finds condition variables in nested if statements that were also
  checked in the outer if statement and were not changed

- bugprone-shared-ptr-array-mismatch

  Finds initializations of C++ shared pointers to non-array type that
  are initialized with an array

- bugprone-stringview-nullptr

  Checks for various ways that the const CharT* constructor of
  std::basic_string_view can be passed a null argument and replaces them
  with the default constructor in most cases

- bugprone-suspicious-memory-comparison

  Finds potentially incorrect calls to memcmp() based on properties of
  the arguments
 2022-05-03 09:22:11 +02:00
Dmitry Novik
2ed5a4013a
Revert "Revert "Memory overcommit: continue query execution if memory is available"" 2022-05-03 00:45:13 +02:00
alesapin
f0b7af0aa2
Revert "Memory overcommit: continue query execution if memory is available" 2022-05-03 00:36:50 +02:00
Dmitry Novik
71b6f89166
Merge pull request #35637 from ClickHouse/memory-overcommit-free 
Memory overcommit: continue query execution if memory is available
 2022-05-02 19:00:18 +02:00
mergify[bot]
93f5aa7488
Merge branch 'master' into aggregator-jit-lock-fix 2022-05-02 16:16:36 +00:00
Dmitry Novik
9be17ef50c
Merge pull request #35111 from azat/optimize_aggregation_in_order-prefix 
Implement partial GROUP BY key for optimize_aggregation_in_order
 2022-05-02 17:49:48 +02:00
Ilya Yatsishin
6c3bbb4933 Add Other Time Microseconds Profile Event 2022-05-02 17:13:57 +02:00
Maksim Kita
c059629f8d Aggregator JIT compilation lock fix 2022-05-02 16:21:10 +02:00
Maksim Kita
2377106370 Fix build 2022-05-02 14:27:00 +02:00
Kruglov Pavel
b98ac57090
Merge pull request #36803 from CurtizJ/fix-unflatten-nested 
Fix creation of tables with `flatten_nested = 0`
 2022-05-02 13:58:16 +02:00
Kruglov Pavel
d613f7eab0
Merge branch 'master' into mysqldump-format 2022-05-02 13:31:57 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Kseniia Sumarokova
e3d2d72b39
Merge pull request #36802 from KinderRiven/local_cache_trace 
system table used for statistical query of remote file access
 2022-05-02 00:16:04 +02:00
Azat Khuzhin
3982638e9a Validate that function had been passed in CREATE FUNCTION 
Otherwise right now it leads to LOGICAL_ERROR, due to bad cast.
 2022-05-01 19:01:03 +03:00
Nikita Mikhaylov
93a65463d0
Added SYSTEM SYNC DATABASE query (#35944) 2022-05-01 15:40:18 +02:00
KinderRiven
9413aa423b fix 2022-05-01 19:45:16 +08:00
KinderRiven
7d0789472c fix 2022-05-01 19:45:16 +08:00
KinderRiven
7db67160fe fix 2022-05-01 19:45:16 +08:00
KinderRiven
e1acacf831 fix 2022-05-01 19:45:16 +08:00
KinderRiven
01210c8d38 fix header 2022-05-01 19:45:16 +08:00
KinderRiven
d80aa0fd17 impl system.filesystem_cache_log 2022-05-01 19:45:16 +08:00
KinderRiven
d595df1604 fix 2022-05-01 19:45:16 +08:00
KinderRiven
6a2e4559c8 impl system.cache_log 2022-05-01 19:45:16 +08:00
KinderRiven
ce9a6965d0 impl system.log_table 2022-05-01 19:45:16 +08:00
KinderRiven
a4b8f9e9cf impl system.cache_log 2022-05-01 19:45:16 +08:00
KinderRiven
9c9870f021 build cache_log table 2022-05-01 19:45:16 +08:00
KinderRiven
463b42b4c3 remove unused var 2022-05-01 19:45:16 +08:00
KinderRiven
5a361a0db5 impl cache_log in filecache 2022-05-01 19:45:16 +08:00
KinderRiven
4db54ff6bc add log record 2022-05-01 19:45:16 +08:00
KinderRiven
29f0471caf add cache log 2022-05-01 19:45:16 +08:00
KinderRiven
0059a660c7 impl create cache_log table 2022-05-01 19:45:16 +08:00
KinderRiven
02080aad1c impl create cache_log table 2022-05-01 19:45:16 +08:00
KinderRiven
44c9996a1e impl create cache_log table 2022-05-01 19:45:16 +08:00
KinderRiven
d5f287369e add include 2022-05-01 19:45:16 +08:00
KinderRiven
ee6ef29538 add include 2022-05-01 19:45:16 +08:00
KinderRiven
ef863a7942 add include 2022-05-01 19:45:16 +08:00
KinderRiven
021a3a788b add cache log table in system database 2022-05-01 19:45:16 +08:00
KinderRiven
6f930be28b impl cache trace 2022-05-01 19:45:16 +08:00
Alexey Milovidov
b034146ba4
Merge pull request #36799 from azat/cleanup 
Tiny cleanup
 2022-05-01 14:23:13 +03:00
Alexey Milovidov
321514a7f3
Merge pull request #36816 from azat/is_all_data_sent-on-exception 
Set is_all_data_sent on exceptions too
 2022-05-01 14:22:06 +03:00
Nikita Taranov
0fd9740c72
Log hash table's cache messages with TRACE level (#36830) 2022-05-01 12:54:54 +02:00
Alexey Milovidov
189938e935
Merge pull request #36431 from rschu1ze/bugprone-branch-clone 
Fixed warnings of clang-tidy check "bugprone-branch-clone"
 2022-05-01 13:49:37 +03:00
Alexey Milovidov
91d1ac913c
Merge pull request #36820 from azat/fix-asynchronous_metric_log 
Avoid recreation of system.asynchronous_metric_log (due to difference in codec)
 2022-05-01 13:42:17 +03:00
Robert Schulze
89aa9ae00f
Fixed clang-tidy check "bugprone-branch-clone" 
The check is currently *not* part of .clang-tidy. It complains about:
(1) "switch has multiple consecutive identical branches"
(2) "repeated branch in conditional chain"

About (1): Lots of findings in switches were about redundant
"[[fallthrough]]" in places where the compiler would not warn anyways. I
have cleaned these up.

About (2): In if-else_if-else chains, fixing the warning would usually
mean concatenating multiple if-conditions. As this would reduce
readability in most cases, I did not fix these places.

Because of (2), I also refrained from adding "bugprone-branch-clone" to
.clang-tidy.
 2022-04-30 19:40:28 +02:00
mergify[bot]
cc08ccb420
Merge branch 'master' into remove-useless-code-2 2022-04-30 12:48:15 +00:00
Nikita Mikhaylov
117ffa90cf
Fix ungrammatical error message (#36760) 2022-04-30 13:57:13 +02:00
Azat Khuzhin
d82f2ff162 Set is_all_data_sent on exceptions too 
This will avoid clickhouse-test complains about left queries after the
test, like in [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/36258/9646487c093a75dc31e3e818417cfad83580b40f/stateful_tests__memory__actions_.html

Follow-up for: #36649
 2022-04-30 13:00:44 +03:00
Azat Khuzhin
a75559aa36 Avoid recreation of system.asynchronous_metric_log (due to difference in codec) 
From the log:

    2022.04.29 12:33:40.397528 [ 486 ] {} <Debug> SystemLog (system.asynchronous_metric_log): Existing table system.asynchronous_metric_log for system log has obsolete or different structure. Renaming it to asynchronous_metric_log_352.
    Old: CREATE TABLE system.asynchronous_metric_log (`event_date` Date CODEC(Delta(2), ZSTD(1)), `event_time` DateTime CODEC(Delta(4), ZSTD(1)), `metric` LowCardinality(String) CODEC(ZSTD(1)), `value` Float64 CODEC(Gorilla, ZSTD(3))) ENGINE = MergeTree PARTITION BY toYYYYMM(event_date) ORDER BY (metric, event_date, event_time) SETTINGS index_granularity = 8192
    New: CREATE TABLE system.asynchronous_metric_log (`event_date` Date CODEC(Delta(2), ZSTD), `event_time` DateTime CODEC(Delta(4), ZSTD), `metric` LowCardinality(String) CODEC(ZSTD), `value` Float64 CODEC(Gorilla, ZSTD(3))) ENGINE = MergeTree PARTITION BY toYYYYMM(event_date) ORDER BY (metric, event_date, event_time) SETTINGS index_granularity = 8192

As you can see the difference in default ZSTD level.

P.S. I decided not to fix the printer, since this may introduce changes
in metadata, that may lead to changes in metadata between ZooKeeper and
clickhouse server.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-30 12:19:29 +03:00
Alexey Milovidov
1ddb04b992
Merge pull request #36715 from amosbird/refactorbase 
Reorganize source files so that base won't depend on Common
 2022-04-30 09:40:58 +03:00
mergify[bot]
e9fde5d067
Merge branch 'master' into memory-overcommit-free 2022-04-29 19:31:59 +00:00
Nikolai Kochetov
5807ca47b0
Merge pull request #36425 from ClickHouse/pool-size-hot-reload 
Allow to increase number of threads and tasks for background executors
 2022-04-29 17:24:12 +02:00
Anton Popov
cef462f83d fix creation of tables with flatten_nested = 0 2022-04-29 14:53:08 +00:00
Azat Khuzhin
16b1d2bee4 Remove outdated comment from ThreadStatusExt 
Refs: https://github.com/ClickHouse/ClickHouse/pull/16121#discussion_r511500850

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 17:04:56 +03:00
Azat Khuzhin
b7b7d91bd1 Remove memory reservation for SquashingTransform 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 17:04:56 +03:00
Vladimir C
e1e7a67670
Merge pull request #36201 from vdimir/join_dup_columns_issue36199 2022-04-29 12:57:10 +02:00
lgbo-ustc
5021cb707c update 2022-04-29 17:41:26 +08:00
vdimir
81c24fc521
Fix assertion in join 2022-04-29 09:15:00 +00:00
Maksim Kita
5b9bb36f80
Merge pull request #36753 from kitaisreal/added-user-defined-path-config-setting 
Added user_defined_path config setting
 2022-04-29 11:10:51 +02:00
Alexey Milovidov
b81da324de
Merge pull request #36765 from ClickHouse/fix_flaky_test_non_leader_replica 
Fix flaky test `test_ddl_worker_non_leader`
 2022-04-29 11:18:49 +03:00
Amos Bird
9d30be2c59
Fix build again 2022-04-29 14:45:27 +08:00
Azat Khuzhin
0ce44f3021 Optimize optimize_aggregation_in_order with a prefix key 
Before it does lots of extra work, now, it will be significantly more
optimal (thousands of rows -> 1-2 million of rows).

v2: s/executeOnBlockSimple/executeOnBlockSmall/
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 06:58:28 +03:00
Azat Khuzhin
190ce217bb Disable GROUP BY statistics for optimize_aggregation_in_order 
This statistics significantly decrease performance of
optimize_aggregation_in_order with a prefix key.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 06:58:27 +03:00
Azat Khuzhin
767acd53fb Add ability to pass range of rows to Aggregator 
v2: fix compiled aggregate functions (seek result to row_start)
v3: fix compiled aggregate functions (seek args to row_start)
v4: change signatures for JIT
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 06:57:55 +03:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
Maksim Kita
04429d85db Added user_defined_path config setting 2022-04-28 22:13:33 +02:00
Maksim Kita
55994afe0c Fix evaluateConstantExpression for subqueries 2022-04-28 22:09:29 +02:00
Alexander Tokmakov
6797e7dc76 fix flaky test 2022-04-28 21:39:45 +02:00
Anton Popov
1fc51e09ff fix insertion to column of type Object from multiple files via table function 2022-04-28 18:51:13 +00:00
Alexey Milovidov
d8fa806fca
Merge pull request #36360 from ClickHouse/optimize-asynchronous-metric-log 
Optimize `system.asynchronous_metric_log`
 2022-04-28 20:16:17 +03:00
Vadim Volodin
233f0b4b52 Handle cluster macro in s3Cluster table function 2022-04-28 18:38:17 +03:00
Kruglov Pavel
4d08587559
Merge branch 'master' into mysqldump-format 2022-04-28 15:58:18 +02:00
zhanglistar
7d798b3655
Count distinct optimization by using subquery of group by (#35993) 2022-04-28 14:55:37 +02:00
Maksim Kita
ec47abe9af
Merge pull request #36482 from Enmk/fix_session_log_error_on_local_and_tcp_interserver_session 
Fixed missing enum values for ClientInfo::Interface
 2022-04-28 13:06:19 +02:00
Vitaly Baranov
641a5f5e35
Merge pull request #36198 from vitlibar/backup-improvements-3 
Backup for replicated tables and other improvements
 2022-04-28 09:23:53 +02:00
Alexey Milovidov
60e512ac4b Fix for table structure comparison 2022-04-28 05:10:57 +02:00
mergify[bot]
b7e63154df
Merge branch 'master' into optimize-asynchronous-metric-log 2022-04-27 18:27:27 +00:00
Kseniia Sumarokova
0d09bb9897
Merge pull request #36699 from amosbird/bettermysqlhandler 
Slightly better mysql handler
 2022-04-27 17:07:35 +02:00
Amos Bird
1cba00478d
Slightly better mysql handler 2022-04-27 18:28:09 +08:00
lgbo-ustc
10b7e08319 update 2022-04-27 16:41:34 +08:00
lgbo-ustc
7505ce426a update 2022-04-27 13:11:24 +08:00
taiyang-li
0341880250 Merge remote-tracking branch 'origin/master' into hive_text_parallel_parsing 2022-04-27 11:15:16 +08:00
Alexey Milovidov
b8a9f1411a
Merge pull request #36649 from azat/system.processes-is_all_data_sent 
clickhouse-test: improve left queries after the test hardening
 2022-04-27 05:21:22 +03:00
lgbo-ustc
520b05b9f1 update test case tests/queries/0_stateless/02236_explain_pipeline_join.sql 2022-04-27 10:08:22 +08:00
Nikolai Kochetov
e44af67fee Merge branch 'master' into refactor-something-in-part-volumes 2022-04-26 21:08:00 +02:00
Vitaly Baranov
a8e924caf6 Make BACKUP & RESTORE synchronous by default. 2022-04-26 18:45:39 +02:00
avogar
33d845dade Add MySQLDump input format 2022-04-26 10:42:56 +00:00
lgbo-ustc
20fc676bff fixed bug: resize on left pipeline cause the order by result wrong 2022-04-26 18:27:42 +08:00
mergify[bot]
7571387a6e
Merge branch 'master' into memory-overcommit-free 2022-04-26 10:15:18 +00:00
lgbo-ustc
d96c29810a fixed bug: resize on left pipeline cause the order by result wrong 2022-04-26 18:12:14 +08:00
lgbo-ustc
0b0fa8453b fixed bug: resize on left pipeline cause the order by result wrong 2022-04-26 18:06:16 +08:00
Kseniia Sumarokova
81db081b88
Merge pull request #36639 from kssenii/better-drop-cache 
Improve drop cache query
 2022-04-26 11:25:46 +02:00
Azat Khuzhin
6339a48923 Add is_all_data_sent into system.processes 
v2: fix SHOW PROCESSLIST (does not have process list entry)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-26 12:15:00 +03:00