Commit Graph

5580 Commits

Author SHA1 Message Date
taiyang-li
2cd469f113 support print ast in dot format 2022-03-10 18:04:35 +08:00
Azat Khuzhin
c4b6342853
Improvements for parallel_distributed_insert_select (and related) (#34728)
* Add a warning if parallel_distributed_insert_select was ignored

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Respect max_distributed_depth for parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Print warning for non applied parallel_distributed_insert_select only for initial query

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Remove Cluster::getHashOfAddresses()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Forbid parallel_distributed_insert_select for remote()/cluster() with different addresses

Before it uses empty cluster name (getClusterName()) which is not
correct, compare all addresses instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix max_distributed_depth check

max_distributed_depth=1 must mean not more then one distributed query,
not two, since max_distributed_depth=0 means no limit, and
distribute_depth is 0 for the first query.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix INSERT INTO remote()/cluster() with parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Add a test for parallel_distributed_insert_select with cluster()/remote()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Return <remote> instead of empty cluster name in Distributed engine

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Make user with sharding_key and w/o in remote()/cluster() identical

Before with sharding_key the user was "default", while w/o it it was
empty.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-08 15:24:39 +01:00
tavplubix
34505be1de
Merge pull request #35081 from azat/fix-shutdown-deadlock
Avoid possible deadlock on server shutdown
2022-03-07 13:47:15 +01:00
Kseniia Sumarokova
cdb9a05229
Merge pull request #34605 from bigo-sg/add_metric_for_local
Add cpu/mem metric for clickhouse-local
2022-03-07 11:40:11 +01:00
Azat Khuzhin
081f9caa04 Avoid possible deadlock on server shutdown
Here is an example for deadlock during shutting down DDLWorker:

Server main thread:

    6  ThreadFromGlobalPool::join () at ../src/Common/ThreadPool.h:217
    7  DB::DDLWorker::shutdown () at ../src/Interpreters/DDLWorker.cpp:123
    8  DB::DDLWorker::~DDLWorker () at ../src/Interpreters/DDLWorker.cpp:131
    9  DB::DDLWorker::~DDLWorker () at ../src/Interpreters/DDLWorker.cpp:130
    10 std::__1::default_delete<DB::DDLWorker>::operator() () at ../contrib/libcxx/include/memory:1397
    11 std::__1::unique_ptr<>::reset (this=0x7f7521d44fd0, __p=0x0) at ../contrib/libcxx/include/memory:1658
    12 DB::ContextSharedPart::shutdown () at ../src/Interpreters/Context.cpp:380 <!-- holds mutex
    13 DB::Context::shutdown () at ../src/Interpreters/Context.cpp:2677

DDLWorker thread:

    7  DB::Context::getLock () at ../src/Interpreters/Context.cpp:472 <-- trying to acquire shared.mutex
    8  DB::Context::getTCPPortSecure () at ../src/Interpreters/Context.cpp:2120
    9  DB::DDLTask::findCurrentHostID () at ../src/Interpreters/DDLTask.cpp:169
    10 DB::DDLWorker::initAndCheckTask () at ../src/Interpreters/DDLWorker.cpp:191
    11 DB::DDLWorker::scheduleTasks () at ../src/Interpreters/DDLWorker.cpp:367
    12 DB::DDLWorker::runMainThread () at ../src/Interpreters/DDLWorker.cpp:1063

v2: replace getTryLockTimed() with getTryLock() since std::recursive_mutex does not have try_lock_for()
v3: stole objects from context
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Fixes: #21318
2022-03-06 14:31:18 +03:00
taiyang-li
1b9987ceb1 fix mistake in check-style 2022-03-05 14:22:56 +08:00
tavplubix
e49521163f
Merge pull request #34860 from zhangjmruc/master
Support non-table DDLs on cross replicated cluster
2022-03-04 19:37:51 +01:00
Kruglov Pavel
d45b6ae03c
Extract schema only once on table creation and add it to metadata (#34684)
* Add columns description to metadata in case of schema inference

* Make better

* Remove unnecessary code

* Fix tests

* More tests

* Add tag no-fasttest

* Fix test

* Fix test

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-04 21:23:19 +03:00
Maksim Kita
c6e0922449
Merge pull request #35018 from kitaisreal/executable-udf-deterministic-in-scope-of-query-fix
ExecutableUDF function deterministic in scope of query fix
2022-03-04 10:46:19 +01:00
Anton Popov
aea7bfb59a
Merge pull request #34992 from azat/fix-asynchronous_inserts-race
Fix race between INSERT async_insert=1 and system.asynchronous_inserts
2022-03-03 20:55:19 +03:00
mergify[bot]
e4fba1dc99
Merge branch 'master' into executable-udf-deterministic-in-scope-of-query-fix 2022-03-03 15:17:16 +00:00
Maksim Kita
a6c510ceb1 ExecutableUDF function deterministic in scope of query fix 2022-03-03 12:40:42 +00:00
Frank Chen
b4829465d9
Improve the opentelemetry span logs for INSERT on distributed table (#34480) 2022-03-03 12:53:29 +01:00
Maksim Kita
b1a956c5f1 clang-tidy check performance-move-const-arg fix 2022-03-02 18:15:27 +00:00
Maksim Kita
1f5837359e clang-tidy check performance-noexcept-move-constructor fix 2022-03-02 18:15:27 +00:00
mreddy017
f893002b69 Fix vulnerable code related to std::move and noexcept
This commit fixes the vulnerable code related to std::move and noexcept identified by clangtidy tool.
2022-03-02 18:15:27 +00:00
Azat Khuzhin
57f636a1e8 Fix race between INSERT async_insert=1 and system.asynchronous_inserts
CI report [1]:

    [c190f600f8c6] 2022.03.02 01:07:34.553012 [ 23552 ] {76b6113b-1479-46c9-90ab-e78a3c9f3dbb}  executeQuery: Code: 60. DB::Exception: Both table name and UUID are empty. (UNKNOWN_TABLE) (version 22.3.1.1) (from [::1]:42040) (comment: '02015_async_inserts_stress_long.sh') (in query: SELECT * FROM system.asynchronous_inserts FORMAT Null), Stack trace (when copying this message, always include the lines below):

    0. ClickHouse/contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int) @ 0xf50e04c in /fasttest-workspace/build/programs/clickhouse
    1. ClickHouse/src/Common/Exception.cpp:58: DB::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int, bool) @ 0x663ebfa in /fasttest-workspace/build/programs/clickhouse
    2. DB::StorageID::assertNotEmpty() const @ 0xbc08591 in /fasttest-workspace/build/programs/clickhouse
    3. ClickHouse/contrib/libcxx/include/string:1444: DB::StorageID::getDatabaseName() const @ 0xe50d2b6 in /fasttest-workspace/build/programs/clickhouse
    4. ClickHouse/contrib/libcxx/include/string:1957: DB::StorageSystemAsynchronousInserts::fillData(std::__1::vector::mutable_ptr, std::__1::allocator::mutable_ptr > >&, std::__1::shared_ptr, DB::SelectQueryInfo const&) const @ 0xdac636c in /fasttest-workspace/build/programs/clickhouse

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/34973/e6fc6a22d5c018961c18247242dd3a40b8c54ff2/fast_test__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-02 15:28:06 +03:00
Azat Khuzhin
a49106e583 Fix non-MergeTree engines for system.*_log
Fixes: https://github.com/ClickHouse/ClickHouse/pull/34949
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-02 13:24:40 +03:00
Anton Popov
82d24f06eb
Merge pull request #34866 from CurtizJ/async-insert-table-function
Fix async inserts to table functions
2022-03-01 19:45:22 +03:00
Kruglov Pavel
97d634d1cc
Merge pull request #34732 from Avogar/insert-select-schema
Support schema inference for INSERT INTO FUNCTION
2022-03-01 13:38:16 +03:00
alesapin
4b61e4795c
Merge pull request #34949 from nikitamikhaylov/system_log_tables_and_settings
Recreate system.{*}_log table on settings changes
2022-03-01 11:15:19 +01:00
taiyang-li
f83132bad2 finish dev 2022-03-01 15:54:23 +08:00
taiyang-li
b31440c77a Merge branch 'master' into add_metric_for_local 2022-03-01 12:29:36 +08:00
Nikita Mikhaylov
d6036f6da3 Better
(cherry picked from commit 4ae445c9e227581ea9f1cbe9aa9d1ba82e1236c9)
2022-02-28 15:27:52 +00:00
Anton Popov
8fa112b841 add comments 2022-02-28 17:09:46 +03:00
Kruglov Pavel
011813957d
Merge pull request #34938 from azat/create-as-ignore-ttl
Ignore per-column TTL in CREATE TABLE AS if new table engine does not support it
2022-02-28 16:58:15 +03:00
Azat Khuzhin
644f9168fa Ignore per-column TTL in CREATE TABLE AS if new table engine does not support it
Follow-up for: #6968
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-28 10:29:26 +03:00
Alexandre Snarskii
93de1b1c99 Implement MemoryStatisticsOS for FreeBSD 2022-02-25 14:05:00 +03:00
Anton Popov
97d57e22ea fix async inserts to table functions 2022-02-24 14:26:47 +03:00
Jianmei Zhang
d148cc30cc Support non-table DDLs on cross replicated cluster 2022-02-24 14:37:35 +08:00
Maksim Kita
174257dad0
Merge pull request #34653 from kitaisreal/executable-udf-support-specify-argument-names
ExecutableUserDefinedFunctions allow to specify argument names
2022-02-22 00:02:14 +01:00
Dmitry Novik
1df43a7f57
Merge pull request #34385 from nvartolomei/nv/move-part-count
Disable optimize_trivial_count when deduplication for part movement feature is enabled
2022-02-21 08:53:09 -08:00
Anton Popov
065305ab65
Merge pull request #34764 from ucasfl/hints-index
Add name hints for data skipping indices
2022-02-21 16:50:59 +03:00
Dmitry Novik
4428e7aa1b
Merge branch 'master' into nv/move-part-count 2022-02-21 02:14:23 -08:00
Mikhail f. Shiryaev
f690aba441
Merge pull request #34078 from godliness/fix-distributed-max-query-size
Fix distributed subquery max_query_size limitation inconsistency
2022-02-21 11:01:02 +01:00
Vitaly Baranov
7b97c986cb
Revert "Allow restrictive row policies without permissive" 2022-02-21 06:54:28 +03:00
feng lv
07280e0ab1 Add name hints for data skipping indices
fix test
2022-02-20 11:48:22 +00:00
alexey-milovidov
26d0e5438c
Merge pull request #33057 from azat/jemalloc-dev-branch
[RFC] Update jemalloc to 5.3RC
2022-02-20 14:19:41 +03:00
Chao Ma
895396e61a Fix distributed subquery max_query_size limitation inconsistency 2022-02-20 10:07:57 +08:00
Vitaly Baranov
874b2c8dcb
Merge pull request #34596 from vitlibar/allow-restrictive-without-permissive
Allow restrictive row policies without permissive
2022-02-19 21:45:28 +07:00
Vladimir C
9b7d011ee7
Merge pull request #34529 from vdimir/join-nullable-on-pipeline
Apply join_use_nulls on types before join
2022-02-18 18:34:44 +01:00
avogar
653d769d34 Support schema inference for INSERT INTO FUNCTION 2022-02-18 16:19:42 +00:00
Maksim Kita
0f9f30cc9e Added tests 2022-02-18 15:21:11 +00:00
vdimir
6ca154858c
upd TableJoin::applyKeyConvertToTable 2022-02-18 13:40:57 +00:00
vdimir
6eda8ef914
use isNullable function in join_common/convertTypeToNullable 2022-02-18 13:09:11 +00:00
Rajkumar
de154c5f9c Adding noexcept for move constructor 2022-02-17 13:18:12 -08:00
Azat Khuzhin
6a2abd3a68 jemalloc: remove jemalloc 4 support
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-17 21:25:58 +03:00
Kruglov Pavel
7411178064
Merge pull request #34650 from azat/fix-projection-optimization
Fix allow_experimental_projection_optimization with enable_global_with_statement
2022-02-17 13:09:57 +03:00
Dmitry Novik
c32313b045
Merge branch 'master' into memory-overcommit-test 2022-02-17 00:50:15 -08:00
Vitaly Baranov
f4da321dbb SHOW CREATE ROW POLICY now always outputs "AS permissive" or "AS restrictive". 2022-02-17 14:18:15 +07:00