thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 03:12:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
88,310 Commits 9,326 Branches 1,783 Tags 2.1 GiB
19afeda4b1
Commit Graph

6249 Commits

Author SHA1 Message Date
Nikolai Kochetov
bcd3c76296
Merge pull request #35214 from ClickHouse/add-database_replicated_allow_only_replicated_engine-setting 
Add database_replicated_allow_only_replicated_engine setting.
 2022-03-16 11:24:32 +01:00
vdimir
695708d57c
[wip] merge join duplicate rows 2022-03-15 18:51:04 +00:00
Alexander Tokmakov
b5b3d09b4a fix 2022-03-15 17:51:53 +01:00
Alexander Tokmakov
1f571b7734 Merge branch 'master' into mvcc_prototype 2022-03-15 14:45:06 +01:00
Alexander Tokmakov
9c3e4cdc6e fix 2022-03-15 14:35:48 +01:00
Anton Popov
b5a04ff938
Merge pull request #35207 from bigo-sg/add_distributed_depth_in_querylog 
Add column  distributed_depth for system.query_log and system.processes
 2022-03-15 13:38:21 +01:00
Vladimir C
af88d3f133
Merge pull request #35173 from bigo-sg/dump_ast_in_dot 2022-03-15 11:18:41 +01:00
taiyang-li
3688ff10e0 fix bug 2022-03-15 12:19:08 +08:00
Alexander Tokmakov
c8d6c13c2d fix 2022-03-15 00:08:25 +01:00
Alexander Tokmakov
9702b5177d Merge branch 'master' into mvcc_prototype 2022-03-14 21:45:38 +01:00
Alexander Tokmakov
278d779a01 log cleanup, more comments 2022-03-14 21:43:34 +01:00
Vitaly Baranov
1eb2e8693e Fix code style and other minor corrections after implementing allow_no_password. 2022-03-14 20:55:34 +01:00
Nikolai Kochetov
7526c36f30 Remove comment 2022-03-14 19:38:03 +00:00
Nikolai Kochetov
ce94b560bb Add test. 2022-03-14 19:30:52 +00:00
Maksim Kita
e30117a3d6 Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
Anton Popov
063917786e minor fixes 2022-03-14 17:29:18 +00:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Vitaly Baranov
4af61fb9d3
Merge pull request #34738 from DevTeamBK/Issue-33953 
New setting in Server Configuration to on/off AuthType Plaintext_password and No_password
 2022-03-14 17:09:46 +01:00
alesapin
b13b415d3f
Merge pull request #35257 from azat/fix-s3-pool-shutdown 
Wait for IDiskRemote thread pool properly
 2022-03-14 13:56:37 +01:00
Nikolai Kochetov
ff09076524 A little bit better. 2022-03-14 11:31:24 +00:00
Kseniia Sumarokova
58a2d2b458
Merge pull request #35118 from zzsmdfj/issue/#31469_MaterializedMysql_mysqlDate2CkDate32 
to #31469_MaterializedMysql_mysqlDate2CkDate32
 2022-03-14 10:32:33 +01:00
Azat Khuzhin
d13fc9ec44 Wait for IDiskRemote thread pool properly 
ASan report [1]:

    WARNING: ThreadSanitizer: heap-use-after-free (pid=611)
      Read of size 4 at 0x7b48006f0250 by thread T273:
        0 MemoryTracker::allocImpl() obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:105:49 (clickhouse+0xa05d0ef)
        1 MemoryTracker::allocImpl() obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:239:22 (clickhouse+0xa05d580)
        2 MemoryTracker::alloc(long) obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:246:5 (clickhouse+0xa05dea7)
        3 DB::ThreadStatus::~ThreadStatus() obj-x86_64-linux-gnu/../src/Common/ThreadStatus.cpp:154:28 (clickhouse+0xa059b01)
        4 ThreadFromGlobalPool::ThreadFromGlobalPool<>()::'lambda'()::operator()() obj-x86_64-linux-gnu/../s>

      Previous write of size 8 at 0x7b48006f0250 by main thread:
        8 std::__1::unordered_map<std::__1::basic_string<>, DB::ProcessListForUser, std::__1::hash<>, std::__1>
        9 DB::ProcessList::~ProcessList() obj-x86_64-linux-gnu/../src/Interpreters/ProcessList.h:275:7 (clickhouse+0x17b71c9c)
        10 DB::ContextSharedPart::~ContextSharedPart() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:304:5 (clickhouse+0x17b70712)
        13 DB::SharedContextHolder::reset() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:453:44 (clickhouse+0x17b3daf9)
        14 DB::Server::main()::$_8::o>

      Thread T273 (tid=946, running) created by thread T223 at:
        0 pthread_create <null> (clickhouse+0x9f7d49d)
        15 DB::WriteBufferFromS3::makeSinglepartUpload() obj-x86_64-linux-gnu/../src/IO/WriteBufferFromS3.cpp:332:9 (clickhouse+0x17a81b6a)

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/35072/19be9c8c6433ffd1bbf8eb08975d8afbaebf3d43/stress_test__thread__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-14 12:07:47 +03:00
Maksim Kita
0dd807d19d
Merge pull request #34750 from kitaisreal/merge-tree-improve-insert-performance 
MergeTree improve insert performance
 2022-03-13 13:39:18 +01:00
Nikolai Kochetov
cdc529fa2a Add database_replicated_allow_only_replicated_engine setting. 2022-03-11 16:03:05 +00:00
Maksim Kita
5f8900cee6
Merge pull request #35079 from qieqieplus/master 
fix #33798
 2022-03-11 11:55:49 +01:00
taiyang-li
1f9e050152 add column distributed_depth for system.query_log and system.processes 2022-03-11 17:57:34 +08:00
Kseniia Sumarokova
818459b9f0
Merge pull request #33717 from kssenii/local-cache-for-remote-fs 
Local cache for remote filesystem
 2022-03-11 07:23:10 +01:00
HeenaBansal2009
3f031df225 Code refactoring 2022-03-10 22:22:51 -08:00
Maksim Kita
cbe059f4bd Updated IColumn interface to support getting stable permutation 2022-03-10 21:45:31 +00:00
Maksim Kita
765cd09d06 MergeTree improve insert performance 2022-03-10 21:45:31 +00:00
Alexander Tokmakov
7f47f20aba add kill transaction query 2022-03-10 22:29:58 +01:00
alesapin
c0d8ccc91b
Merge pull request #35178 from Varinara/master 
Added disk_name to system.part_log
 2022-03-10 22:22:37 +01:00
alexey-milovidov
58e53b06a6
Merge pull request #35134 from azat/optimize_skip_unused_shards_rewrite_in-signed 
Fix `optimize_skip_unused_shards_rewrite_in` for signed columns and negative values
 2022-03-10 23:12:05 +03:00
mergify[bot]
059682a0cd
Merge branch 'master' into asof_ftw 2022-03-10 17:29:38 +00:00
mergify[bot]
9f4ebc313b
Merge branch 'master' into master 2022-03-10 16:55:27 +00:00
Varinara
f5523f7ff0 added disk_name to system.part_log 2022-03-10 18:44:19 +03:00
Rajkumar
3d3b6d1956 clang-tidy report issues with Medium priority 2022-03-10 07:23:49 -08:00
Alexander Tokmakov
061fa6a6f2 Merge branch 'master' into mvcc_prototype 2022-03-10 13:13:04 +01:00
kssenii
787a0805a5 Merge master 2022-03-10 11:42:19 +01:00
taiyang-li
ee8fc2730e remote comment 2022-03-10 18:29:52 +08:00
taiyang-li
382417d893 fix bug 2022-03-10 18:29:00 +08:00
taiyang-li
2cd469f113 support print ast in dot format 2022-03-10 18:04:35 +08:00
HeenaBansal2009
3ce9397246 Added Suggestions from Code review 2022-03-09 20:35:01 -08:00
Alexander Tokmakov
0906b59fba fixes 2022-03-09 21:38:18 +01:00
Vladimir C
ce266b5a3e
Merge pull request #35146 from amosbird/fixpartitionprunerin 2022-03-09 13:23:45 +01:00
Nikolai Kochetov
c364908061
Merge pull request #35094 from amosbird/getridofredundantplan 
Get rid of duplicate query planing.
 2022-03-09 09:10:20 +01:00
Amos Bird
a19224bc9b
Fix partition pruner: non-monotonic function IN 2022-03-09 15:48:42 +08:00
Azat Khuzhin
ec6bae2b2a Fix optimize_skip_unused_shards_rewrite_in for signed columns 
Fixes: #35131
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-09 08:29:28 +03:00
Alexander Tokmakov
d2f838bd91 fix another race condition 2022-03-08 20:11:47 +01:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00
Azat Khuzhin
c4b6342853
Improvements for parallel_distributed_insert_select (and related) (#34728) 
* Add a warning if parallel_distributed_insert_select was ignored

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Respect max_distributed_depth for parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Print warning for non applied parallel_distributed_insert_select only for initial query

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Remove Cluster::getHashOfAddresses()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Forbid parallel_distributed_insert_select for remote()/cluster() with different addresses

Before it uses empty cluster name (getClusterName()) which is not
correct, compare all addresses instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix max_distributed_depth check

max_distributed_depth=1 must mean not more then one distributed query,
not two, since max_distributed_depth=0 means no limit, and
distribute_depth is 0 for the first query.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix INSERT INTO remote()/cluster() with parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Add a test for parallel_distributed_insert_select with cluster()/remote()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Return <remote> instead of empty cluster name in Distributed engine

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Make user with sharding_key and w/o in remote()/cluster() identical

Before with sharding_key the user was "default", while w/o it it was
empty.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 15:24:39 +01:00
zzsmdfj
67b9f81104 to #31469_MaterializedMysql_mysqlDate2CkDate32 2022-03-08 18:17:22 +08:00
taiyang-li
b4174b0bef merge master and fix conflicts 2022-03-08 11:39:25 +08:00
Alexander Tokmakov
8acfb8d27f Merge branch 'master' into mvcc_prototype 2022-03-07 17:40:15 +01:00
Amos Bird
fe4534d464
Get rid of duplicate query planing. 2022-03-08 00:02:58 +08:00
Anton Popov
0bc57da238 Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-07 14:46:08 +00:00
tavplubix
34505be1de
Merge pull request #35081 from azat/fix-shutdown-deadlock 
Avoid possible deadlock on server shutdown
 2022-03-07 13:47:15 +01:00
Kseniia Sumarokova
cdb9a05229
Merge pull request #34605 from bigo-sg/add_metric_for_local 
Add cpu/mem metric for clickhouse-local
 2022-03-07 11:40:11 +01:00
Azat Khuzhin
081f9caa04 Avoid possible deadlock on server shutdown 
Here is an example for deadlock during shutting down DDLWorker:

Server main thread:

    6  ThreadFromGlobalPool::join () at ../src/Common/ThreadPool.h:217
    7  DB::DDLWorker::shutdown () at ../src/Interpreters/DDLWorker.cpp:123
    8  DB::DDLWorker::~DDLWorker () at ../src/Interpreters/DDLWorker.cpp:131
    9  DB::DDLWorker::~DDLWorker () at ../src/Interpreters/DDLWorker.cpp:130
    10 std::__1::default_delete<DB::DDLWorker>::operator() () at ../contrib/libcxx/include/memory:1397
    11 std::__1::unique_ptr<>::reset (this=0x7f7521d44fd0, __p=0x0) at ../contrib/libcxx/include/memory:1658
    12 DB::ContextSharedPart::shutdown () at ../src/Interpreters/Context.cpp:380 <!-- holds mutex
    13 DB::Context::shutdown () at ../src/Interpreters/Context.cpp:2677

DDLWorker thread:

    7  DB::Context::getLock () at ../src/Interpreters/Context.cpp:472 <-- trying to acquire shared.mutex
    8  DB::Context::getTCPPortSecure () at ../src/Interpreters/Context.cpp:2120
    9  DB::DDLTask::findCurrentHostID () at ../src/Interpreters/DDLTask.cpp:169
    10 DB::DDLWorker::initAndCheckTask () at ../src/Interpreters/DDLWorker.cpp:191
    11 DB::DDLWorker::scheduleTasks () at ../src/Interpreters/DDLWorker.cpp:367
    12 DB::DDLWorker::runMainThread () at ../src/Interpreters/DDLWorker.cpp:1063

v2: replace getTryLockTimed() with getTryLock() since std::recursive_mutex does not have try_lock_for()
v3: stole objects from context
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Fixes: #21318
 2022-03-06 14:31:18 +03:00
qieqieplus
5287f78d8b fix #33798 2022-03-06 12:02:17 +08:00
taiyang-li
1b9987ceb1 fix mistake in check-style 2022-03-05 14:22:56 +08:00
tavplubix
e49521163f
Merge pull request #34860 from zhangjmruc/master 
Support non-table DDLs on cross replicated cluster
 2022-03-04 19:37:51 +01:00
Kruglov Pavel
d45b6ae03c
Extract schema only once on table creation and add it to metadata (#34684) 
* Add columns description to metadata in case of schema inference

* Make better

* Remove unnecessary code

* Fix tests

* More tests

* Add tag no-fasttest

* Fix test

* Fix test

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-03-04 21:23:19 +03:00
Maksim Kita
c6e0922449
Merge pull request #35018 from kitaisreal/executable-udf-deterministic-in-scope-of-query-fix 
ExecutableUDF function deterministic in scope of query fix
 2022-03-04 10:46:19 +01:00
Anton Popov
df3b07fe7c Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-03 22:25:28 +00:00
Anton Popov
aea7bfb59a
Merge pull request #34992 from azat/fix-asynchronous_inserts-race 
Fix race between INSERT async_insert=1 and system.asynchronous_inserts
 2022-03-03 20:55:19 +03:00
mergify[bot]
e4fba1dc99
Merge branch 'master' into executable-udf-deterministic-in-scope-of-query-fix 2022-03-03 15:17:16 +00:00
Maksim Kita
a6c510ceb1 ExecutableUDF function deterministic in scope of query fix 2022-03-03 12:40:42 +00:00
Frank Chen
b4829465d9
Improve the opentelemetry span logs for INSERT on distributed table (#34480) 2022-03-03 12:53:29 +01:00
Maksim Kita
b1a956c5f1 clang-tidy check performance-move-const-arg fix 2022-03-02 18:15:27 +00:00
Maksim Kita
1f5837359e clang-tidy check performance-noexcept-move-constructor fix 2022-03-02 18:15:27 +00:00
mreddy017
f893002b69 Fix vulnerable code related to std::move and noexcept 
This commit fixes the vulnerable code related to std::move and noexcept identified by clangtidy tool.
 2022-03-02 18:15:27 +00:00
kssenii
d5952109fb Merge master 2022-03-02 18:15:25 +01:00
mergify[bot]
93b13c0232
Merge branch 'master' into asof_ftw 2022-03-02 13:20:50 +00:00
Azat Khuzhin
57f636a1e8 Fix race between INSERT async_insert=1 and system.asynchronous_inserts 
CI report [1]:

    [c190f600f8c6] 2022.03.02 01:07:34.553012 [ 23552 ] {76b6113b-1479-46c9-90ab-e78a3c9f3dbb}  executeQuery: Code: 60. DB::Exception: Both table name and UUID are empty. (UNKNOWN_TABLE) (version 22.3.1.1) (from [::1]:42040) (comment: '02015_async_inserts_stress_long.sh') (in query: SELECT * FROM system.asynchronous_inserts FORMAT Null), Stack trace (when copying this message, always include the lines below):

    0. ClickHouse/contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int) @ 0xf50e04c in /fasttest-workspace/build/programs/clickhouse
    1. ClickHouse/src/Common/Exception.cpp:58: DB::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int, bool) @ 0x663ebfa in /fasttest-workspace/build/programs/clickhouse
    2. DB::StorageID::assertNotEmpty() const @ 0xbc08591 in /fasttest-workspace/build/programs/clickhouse
    3. ClickHouse/contrib/libcxx/include/string:1444: DB::StorageID::getDatabaseName() const @ 0xe50d2b6 in /fasttest-workspace/build/programs/clickhouse
    4. ClickHouse/contrib/libcxx/include/string:1957: DB::StorageSystemAsynchronousInserts::fillData(std::__1::vector::mutable_ptr, std::__1::allocator::mutable_ptr > >&, std::__1::shared_ptr, DB::SelectQueryInfo const&) const @ 0xdac636c in /fasttest-workspace/build/programs/clickhouse

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/34973/e6fc6a22d5c018961c18247242dd3a40b8c54ff2/fast_test__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-02 15:28:06 +03:00
Azat Khuzhin
a49106e583 Fix non-MergeTree engines for system.*_log 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/34949
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-02 13:24:40 +03:00
kssenii
a594f388a4 Merge master 2022-03-01 19:43:45 +01:00
Anton Popov
c1fdcf7a64 Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-01 20:21:39 +03:00
Anton Popov
82d24f06eb
Merge pull request #34866 from CurtizJ/async-insert-table-function 
Fix async inserts to table functions
 2022-03-01 19:45:22 +03:00
Kruglov Pavel
97d634d1cc
Merge pull request #34732 from Avogar/insert-select-schema 
Support schema inference for INSERT INTO FUNCTION
 2022-03-01 13:38:16 +03:00
alesapin
4b61e4795c
Merge pull request #34949 from nikitamikhaylov/system_log_tables_and_settings 
Recreate system.{*}_log table on settings changes
 2022-03-01 11:15:19 +01:00
taiyang-li
f83132bad2 finish dev 2022-03-01 15:54:23 +08:00
taiyang-li
b31440c77a Merge branch 'master' into add_metric_for_local 2022-03-01 12:29:36 +08:00
HeenaBansal2009
c14c60f1d3 Merge branch 'master' into Issue-33953 2022-02-28 11:12:54 -08:00
HeenaBansal2009
aa8494a808 Fix: System Reload Config Failure 2022-02-28 10:51:49 -08:00
Nikita Mikhaylov
d6036f6da3 Better 
(cherry picked from commit 4ae445c9e227581ea9f1cbe9aa9d1ba82e1236c9)
 2022-02-28 15:27:52 +00:00
Anton Popov
8fa112b841 add comments 2022-02-28 17:09:46 +03:00
Kruglov Pavel
011813957d
Merge pull request #34938 from azat/create-as-ignore-ttl 
Ignore per-column TTL in CREATE TABLE AS if new table engine does not support it
 2022-02-28 16:58:15 +03:00
Azat Khuzhin
644f9168fa Ignore per-column TTL in CREATE TABLE AS if new table engine does not support it 
Follow-up for: #6968
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-28 10:29:26 +03:00
Alexandre Snarskii
93de1b1c99 Implement MemoryStatisticsOS for FreeBSD 2022-02-25 14:05:00 +03:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
Anton Popov
97d57e22ea fix async inserts to table functions 2022-02-24 14:26:47 +03:00
Jianmei Zhang
d148cc30cc Support non-table DDLs on cross replicated cluster 2022-02-24 14:37:35 +08:00
Alexander Tokmakov
aa6b9a2abc Merge branch 'master' into mvcc_prototype 2022-02-23 23:22:03 +03:00
kssenii
c637385dd0 Merge master 2022-02-22 13:17:51 +01:00
Maksim Kita
174257dad0
Merge pull request #34653 from kitaisreal/executable-udf-support-specify-argument-names 
ExecutableUserDefinedFunctions allow to specify argument names
 2022-02-22 00:02:14 +01:00
Dmitry Novik
1df43a7f57
Merge pull request #34385 from nvartolomei/nv/move-part-count 
Disable optimize_trivial_count when deduplication for part movement feature is enabled
 2022-02-21 08:53:09 -08:00
Dmitry Novik
be9082f1c4 Update comments for OvercommitTracker 2022-02-21 15:45:17 +00:00
Anton Popov
065305ab65
Merge pull request #34764 from ucasfl/hints-index 
Add name hints for data skipping indices
 2022-02-21 16:50:59 +03:00
kssenii
4543513a5d Add comments 2022-02-21 13:54:03 +01:00
Dmitry Novik
4428e7aa1b
Merge branch 'master' into nv/move-part-count 2022-02-21 02:14:23 -08:00
Mikhail f. Shiryaev
f690aba441
Merge pull request #34078 from godliness/fix-distributed-max-query-size 
Fix distributed subquery max_query_size limitation inconsistency
 2022-02-21 11:01:02 +01:00
Vitaly Baranov
7b97c986cb
Revert "Allow restrictive row policies without permissive" 2022-02-21 06:54:28 +03:00
Alexey Milovidov
522c1c1397 Allow to use default port in cluster configuration 2022-02-20 19:27:14 +01:00
feng lv
07280e0ab1 Add name hints for data skipping indices 
fix test
 2022-02-20 11:48:22 +00:00
alexey-milovidov
26d0e5438c
Merge pull request #33057 from azat/jemalloc-dev-branch 
[RFC] Update jemalloc to 5.3RC
 2022-02-20 14:19:41 +03:00
Chao Ma
895396e61a Fix distributed subquery max_query_size limitation inconsistency 2022-02-20 10:07:57 +08:00
Raúl Marín
4439b23e2b Attempt manual unpack 2022-02-19 21:01:47 +01:00
Vitaly Baranov
874b2c8dcb
Merge pull request #34596 from vitlibar/allow-restrictive-without-permissive 
Allow restrictive row policies without permissive
 2022-02-19 21:45:28 +07:00
Raúl Marín
07d2b6ba80 Remove struct packing 2022-02-19 11:16:28 +01:00
HeenaBansal2009
1b263f0c15 Added FT testcase 2022-02-18 12:58:46 -08:00
Raúl Marín
606f580922 Fix style 2022-02-18 18:37:46 +01:00
Vladimir C
9b7d011ee7
Merge pull request #34529 from vdimir/join-nullable-on-pipeline 
Apply join_use_nulls on types before join
 2022-02-18 18:34:44 +01:00
Raúl Marín
bb1d773ffd Reduce changes against master 2022-02-18 17:58:23 +01:00
avogar
653d769d34 Support schema inference for INSERT INTO FUNCTION 2022-02-18 16:19:42 +00:00
Raúl Marín
2f4c595e09 Bring back some clarity 2022-02-18 16:50:15 +01:00
Maksim Kita
0f9f30cc9e Added tests 2022-02-18 15:21:11 +00:00
Raúl Marín
f8cc024ba1 Less indirect access 2022-02-18 16:16:29 +01:00
vdimir
6ca154858c
upd TableJoin::applyKeyConvertToTable 2022-02-18 13:40:57 +00:00
vdimir
6eda8ef914
use isNullable function in join_common/convertTypeToNullable 2022-02-18 13:09:11 +00:00
Raúl Marín
57b9639872 Faster binary search 2022-02-18 14:05:42 +01:00
Raúl Marín
4ba5b931a5 AsofRowRefs: Remove extra level of indirection 2022-02-18 11:15:11 +01:00
Raúl Marín
34b110b778 Remove variant from AsofRowRefs 2022-02-18 11:02:14 +01:00
HeenaBansal2009
d16cae53b4 Initial Commit for Plaintext password feature 2022-02-17 21:25:18 -08:00
Alexander Tokmakov
f4a46a13fb fixes 2022-02-18 00:26:37 +03:00
Rajkumar
de154c5f9c Adding noexcept for move constructor 2022-02-17 13:18:12 -08:00
Azat Khuzhin
6a2abd3a68 jemalloc: remove jemalloc 4 support 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-17 21:25:58 +03:00
Raúl Marín
b751218fd6 Help the compiler with the variant dispatch code 2022-02-17 16:39:55 +01:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
Kruglov Pavel
7411178064
Merge pull request #34650 from azat/fix-projection-optimization 
Fix allow_experimental_projection_optimization with enable_global_with_statement
 2022-02-17 13:09:57 +03:00
Dmitry Novik
c32313b045
Merge branch 'master' into memory-overcommit-test 2022-02-17 00:50:15 -08:00
Vitaly Baranov
f4da321dbb SHOW CREATE ROW POLICY now always outputs "AS permissive" or "AS restrictive". 2022-02-17 14:18:15 +07:00
Vitaly Baranov
2de6e8e575 Change type of RowPolicyKind: bool -> enum. 2022-02-17 14:18:10 +07:00
Dmitry Novik
07d0e3c823 cleanup 2022-02-16 20:19:10 +00:00
Dmitry Novik
44d5ddc939 Fix deadlock in OvercommitTracker 2022-02-16 20:02:14 +00:00
Azat Khuzhin
774744a86d Fix allow_experimental_projection_optimization with enable_global_with_statement 
allow_experimental_projection_optimization requires one more
InterpreterSelectQuery, which with enable_global_with_statement will
apply ApplyWithAliasVisitor if the query is not subquery.

But this should not be done for queries from
MergeTreeData::getQueryProcessingStage()/getQueryProcessingStageWithAggregateProjections()
since this will duplicate WITH statements over and over.

This will also fix scalar.xml perf tests, that leads to the following
error now:

    scalar.query0.prewarm0: DB::Exception: Stack size too large.

And since it has very long query in the log, this leads to the following
perf test error:

    _csv.Error: field larger than field limit (131072)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-16 19:14:47 +03:00
Maksim Kita
669b18e037 ExecutableUserDefinedFunctions allow to specify argument names 2022-02-16 15:31:30 +00:00
Anton Popov
edd686e1d9 fix unit test 2022-02-16 17:18:03 +03:00
kssenii
47f94120da Merge master 2022-02-16 14:43:28 +01:00
Nikolai Kochetov
f9d2dae88e
Merge pull request #34424 from yakov-olkhovskiy/ephemeral-column 
Ephemeral column issue #9436
 2022-02-16 12:04:57 +01:00
Raúl Marín
d06ef3e1db AsofRowRefs: Simplify variant code 2022-02-16 11:12:28 +01:00
Raúl Marín
47d325edd1 Improve SortedLookupVector::find 2022-02-16 11:12:28 +01:00
Raúl Marín
0a620b82a3 RowRefs: Use assert_cast instead of typeid_cast 
They couldn't fail and it meant over 1% of the query runtime:
1.23%  typeid_cast<DB::ColumnVector<unsigned int> const&, DB::IColumn const>
    - 0.53% DB::AsofRowRefs::insert
    - 0.56% DB::AsofRowRefs::findAsof
 2022-02-16 11:12:28 +01:00
Raúl Marín
b99d757633 Packed RowRef 
- sizeof(DB::AsofRowRefs::Entry<Int32>): Was 24, now 16
- sizeof(DB::AsofRowRefs::Entry<Int64>): Was 24, now 24 too
 2022-02-16 11:12:28 +01:00
Anton Popov
a661eaf39f better performance of getting storage snapshot 2022-02-16 02:17:22 +03:00
Nikolai Kochetov
1276c50271 Fix build. 2022-02-15 20:15:07 +00:00
alesapin
bc2d0ee7c7
Merge pull request #34215 from ClickHouse/revert-34211-revert-34153-add_func_tests_over_s3 
Add func tests run with s3 and fix several bugs
 2022-02-15 19:07:11 +03:00
Dmitry Novik
bb6dad7d0e Fix lock order 2022-02-15 15:04:13 +00:00
taiyang-li
e6b29167da fix style 2022-02-15 20:26:53 +08:00
Nikolai Kochetov
b2b29b7ee8 Do not use ReduceToPhysicalTransform. 2022-02-15 12:23:22 +00:00
vdimir
510a909d0d
Remove nullable_left/right_side flags from Hash/MergeJoin 2022-02-15 11:15:50 +00:00
vdimir
2de6eaf838
Fix LowCardinality using key for join_use_nulls in pipeline 2022-02-15 11:11:59 +00:00
Nikolai Kochetov
ab288642f6 Merge branch 'master' into ephemeral-column 2022-02-15 10:03:34 +00:00
Nikolai Kochetov
d6cbac1ed3
Merge pull request #34577 from ClickHouse/alwasy-remove-unused-actions-for-add-missing-defaults 
Always remove unused actions from addMissingDefaults
 2022-02-15 11:01:29 +01:00
Nikolai Kochetov
4d2c53c32b Fix clang tidy 2022-02-15 09:59:51 +00:00
taiyang-li
b4440131d0 add metrics for clickhouse-local 2022-02-15 16:25:07 +08:00
李扬
f52b67b939
Merge branch 'master' into rocksdb_metacache 2022-02-15 02:16:29 -06:00
Alexander Tokmakov
1e4e569151 Merge branch 'master' into mvcc_prototype 2022-02-15 02:26:47 +03:00
Alexander Tokmakov
ae5aa8c12d write part version before other files 2022-02-15 02:24:51 +03:00
alexey-milovidov
bc206ec423
Merge pull request #31182 from ClickHouse/memory-overcommit 
Memory overcommit
 2022-02-14 23:55:02 +03:00
Alexander Tokmakov
cbd3b45646 add EXPLAIN CURRENT TRANSACTION 2022-02-14 22:47:17 +03:00
vdimir
99ca89c0ca
Fix StorageJoin and Asof or join_use_nulls in pipeline 2022-02-14 14:14:27 +00:00
Nikolai Kochetov
22a938f83d Simplify a little bit. 2022-02-14 14:53:32 +01:00
Nikolai Kochetov
ebb5d1807d Always remove unused actipns from addMissingDefaults 2022-02-14 12:49:07 +00:00
alesapin
89373155fc Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-13 21:07:54 +03:00
Yakov Olkhovskiy
579fe6c97a major rework, transform added to the insert pipe 2022-02-13 17:42:59 +00:00
Maksim Kita
340614e5ec
Merge pull request #34489 from vitlibar/make-create-row-policy-privilege-table-level 
Privileges CREATE/ALTER/DROP ROW POLICY are table-level now
 2022-02-13 15:15:49 +01:00
李扬
daa27d0bda
Merge branch 'master' into rocksdb_metacache 2022-02-12 07:50:12 -06:00
vdimir
ee09ec4dd1
Apply join_use_nulls on types before join 2022-02-11 15:56:45 +00:00
tavplubix
6d1fae834a
Merge pull request #34187 from qoega/default-table-engine 
Default table engine
 2022-02-11 18:14:29 +03:00
Alexander Tokmakov
07e66e690d Merge branch 'master' into mvcc_prototype 2022-02-11 15:53:32 +03:00
tavplubix
4a9d0c84df
Update InterpreterCreateQuery.cpp 2022-02-11 13:59:13 +03:00
Vitaly Baranov
e148b43a29 Privileges CREATE/ALTER/DROP ROW POLICY now can be granted on a table or on database.* as well as globally *.* 2022-02-11 12:35:52 +07:00
Alexander Tokmakov
78afe31397 fix 2022-02-11 02:15:43 +03:00
Alexander Tokmakov
9e893dfda7 fixes 2022-02-10 23:35:03 +03:00
Kruglov Pavel
a4f5610764
Merge pull request #34476 from CurtizJ/avoid-settings-copy 
Avoid unnecessary copying of `Settings`
 2022-02-10 14:13:46 +03:00
mergify[bot]
cb3e5f8538
Merge branch 'master' into memory-overcommit 2022-02-10 11:01:43 +00:00
Anton Popov
298838f891 avoid unnecessary copying of Settings 2022-02-10 12:13:51 +03:00
Yatsishin Ilya
33b5c2251f Merge branch 'default-table-engine' of github.com:qoega/ClickHouse into default-table-engine 2022-02-10 09:09:20 +00:00
Yatsishin Ilya
9228b10fa6 refactoring 2022-02-10 08:53:50 +00:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
mergify[bot]
d55eadc3c9
Merge branch 'master' into default-table-engine 2022-02-09 17:08:31 +00:00
Nikolai Kochetov
38fb50f736
Merge pull request #33958 from Algunenano/mv_cacheable_scalars 
Scalar cache improvements
 2022-02-09 16:46:53 +03:00
Maksim Kita
9f29c977de
Merge pull request #34234 from azat/agg-leak 
Fix memory leak in AggregatingInOrderTransform
 2022-02-09 12:42:04 +01:00
alesapin
10c3e6e546 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-09 14:14:58 +03:00
mergify[bot]
fd5caa3196
Merge branch 'master' into short-circuit-bug 2022-02-09 11:04:19 +00:00
Kseniia Sumarokova
86956ca08b
Merge pull request #34392 from kssenii/http-buffer-skip-not-found-url-for-globs 
Allow to skip not found urls for globs
 2022-02-09 09:32:09 +01:00
Azat Khuzhin
4fa2ae76bc Fix memory leak in AggregatingInOrderTransform 
Reproducer:

    # NOTE: we need clickhouse from 33957 since right now LSan is broken due to getauxval().
    $ url=https://s3.amazonaws.com/clickhouse-builds/33957/e04b862673644d313712607a0078f5d1c48b5377/package_asan/clickhouse
    $ wget $url -o clickhouse-asan
    $ chmod +x clickhouse-asan
    $ ./clickhouse-asan server &

    $ ./clickhouse-asan client
    :) create table data (key Int, value String) engine=MergeTree() order by key
    :) insert into data select number%5, toString(number) from numbers(10e6)

    # usually it is enough one query, benchmark is just for stability of the results
    # note, that if the exception was not happen from AggregatingInOrderTransform then add --continue_on_errors and wait
    $ ./clickhouse-asan benchmark --query 'select key, uniqCombined64(value), groupArray(value) from data group by key' --optimize_aggregation_in_order=1 --memory_tracker_fault_probability=0.01, max_untracked_memory='2Mi'

LSan report:

    ==24595==ERROR: LeakSanitizer: detected memory leaks

    Direct leak of 3932160 byte(s) in 6 object(s) allocated from:
        0 0xcadba93 in realloc ()
        1 0xcc108d9 in Allocator<false, false>::realloc() obj-x86_64-linux-gnu/../src/Common/Allocator.h:134:30
        2 0xde19eae in void DB::PODArrayBase<>::realloc<DB::Arena*&>(unsigned long, DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h:161:25
        3 0xde5f039 in void DB::PODArrayBase<>::reserveForNextSize<DB::Arena*&>(DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h
        4 0xde5f039 in void DB::PODArray<>::push_back<>(DB::GroupArrayNodeString*&, DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h:432:19
        5 0xde5f039 in DB::GroupArrayGeneralImpl<>::add() const obj-x86_64-linux-gnu/../src/AggregateFunctions/AggregateFunctionGroupArray.h:465:31
        6 0xde5f039 in DB::IAggregateFunctionHelper<>::addBatchSinglePlaceFromInterval() const obj-x86_64-linux-gnu/../src/AggregateFunctions/IAggregateFunction.h:481:53
        7 0x299df134 in DB::Aggregator::executeOnIntervalWithoutKeyImpl() obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:869:31
        8 0x2ca75f7d in DB::AggregatingInOrderTransform::consume() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingInOrderTransform.cpp:124:13

    ...

    SUMMARY: AddressSanitizer: 4523184 byte(s) leaked in 12 allocation(s).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-09 09:23:56 +03:00
taiyang-li
d04ccc0489 Merge branch 'master' into rocksdb_metacache 2022-02-09 11:54:10 +08:00
Yakov Olkhovskiy
b9777be271 insertion with ephemerals requires columns list, insertion without columns list omits ephemeral 2022-02-09 01:11:29 +00:00
Anton Popov
587d7399ba support dynamic subcolumns for Memory engine 2022-02-09 03:18:53 +03:00
alesapin
02a93cb852 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-08 19:42:27 +03:00
avogar
08c009ab50 Fix possible error 'Cannot convert column Function to mask' 2022-02-08 18:23:43 +03:00
Nikolai Kochetov
0f7c0c72bd
Merge pull request #34305 from amosbird/projection-fix27 
Fix various issues when projection is enabled by default
 2022-02-08 17:19:56 +03:00
Kruglov Pavel
720310fcf1
Merge branch 'master' into http-buffer-skip-not-found-url-for-globs 2022-02-08 16:56:22 +03:00
Yatsishin Ilya
3e2e422ddd refactoring 2022-02-08 12:21:16 +00:00
Yatsishin Ilya
bcf4303286 Merge remote-tracking branch 'origin' into default-table-engine 2022-02-08 10:35:29 +00:00
taiyang-li
b6132d490f merge master and solve conflict 2022-02-08 15:24:59 +08:00
Yakov Olkhovskiy
97daa84f6f revert unnecesarry change 2022-02-08 00:14:15 +00:00
Yakov Olkhovskiy
4d5fb56c29 added EPHEMERAL default for column (preliminary) 2022-02-07 23:21:10 +00:00
kssenii
eba3011ada Fix 2022-02-07 20:40:47 +01:00
Nicolae Vartolomei
1cdb50cf13 Disable optimize_trivial_count when deduplication for part movement feature is enabled 
Fixes #34089
 2022-02-07 18:26:49 +00:00
kssenii
2e58733750 Merge master 2022-02-07 15:23:10 +01:00
avogar
975ba9146f Fix tests naming, remove code duplication 2022-02-07 14:12:19 +03:00
mergify[bot]
dd947f964c
Merge branch 'master' into mv_cacheable_scalars 2022-02-07 10:07:26 +00:00
alesapin
ba28c2c013 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-07 12:44:56 +03:00
Yatsishin Ilya
e0803064e6 Merge remote-tracking branch 'origin' into default-table-engine 2022-02-07 08:45:32 +00:00
Alexander Tokmakov
45be75b4db Merge branch 'master' into mvcc_prototype 2022-02-06 23:36:08 +03:00
Amos Bird
a6f0b01e6a
Fix order by after aggregation 2022-02-07 00:42:11 +08:00
Amos Bird
52aabf98fe
Revise and add more comments 2022-02-06 16:53:54 +08:00
Amos Bird
1ab773cc90
Fix aggregation_in_order with normal projection 2022-02-06 16:46:12 +08:00
Amos Bird
7674bc986e
Disable projection when there is JOIN or SAMPLE 2022-02-06 16:46:09 +08:00
Maksim Kita
bc6328e573 Fixed tests 2022-02-04 20:56:49 +00:00
Alexander Tokmakov
3956941aaf fixes 2022-02-04 21:18:20 +03:00
alesapin
c028269e6f Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-04 18:02:25 +03:00
Maksim Kita
629c8ae915
Merge pull request #34284 from CurtizJ/async-insert-in-client 
Minor fixes for #34267
 2022-02-04 13:13:47 +01:00
alesapin
2ed45b2a98 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-04 11:23:46 +03:00
Alexander Tokmakov
6a646b413e fix 2022-02-04 00:40:17 +03:00
Alexander Tokmakov
3e700e854d cancel merges before acquiring lock for truncate 2022-02-03 23:44:53 +03:00
Alexander Tokmakov
fe30e0f162 fixes 2022-02-03 21:57:09 +03:00
Alexander Tokmakov
ca5f951558 Merge branch 'master' into mvcc_prototype 2022-02-03 18:56:44 +03:00
Anton Popov
9b844c6b42
Merge pull request #32748 from CurtizJ/read-in-order-fixed-prefix 
Support `optimize_read_in_order` if prefix of sorting key is already sorted
 2022-02-03 18:17:08 +03:00
alesapin
2a9bc7cba8 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-03 16:20:06 +03:00
alesapin
8e33427e19 Fix check for existing table 2022-02-03 15:17:55 +03:00
Anton Popov
0755cfa584 minor fixes 2022-02-03 15:04:13 +03:00
alexey-milovidov
a6d1fcadcb
Merge pull request #34267 from CurtizJ/async-insert-in-client 
Support async inserts in clickhouse-client for queries with inlined data
 2022-02-03 05:14:47 +03:00
Maksim Kita
02c6cfba6a
Merge pull request #34225 from vitlibar/fix-checking-grants-for-show-grants 
Fix checking grants for SHOW GRANTS
 2022-02-02 22:51:51 +01:00
Yatsishin Ilya
cce0452d19 add more tests, fix style, fix MATERIALIZED VIEW POPULATE without ENGINE 2022-02-02 18:07:03 +00:00
Anton Popov
5bb1b3ce77 support async inserts in clickhouse-client for queries with inlined data 2022-02-02 20:59:37 +03:00
Vitaly Baranov
30557aebfb Add helper class to cache the result of checking access. 2022-02-02 22:22:41 +07:00
Yatsishin Ilya
8c822755cb fix AS 2022-02-02 12:33:29 +00:00
Nikolay Degterinsky
29c15f8c65
Merge pull request #33723 from FrankChen021/trace_id_order 
Parse and store opentelemetry trace-id in big-endian order
 2022-02-02 14:59:46 +03:00
tavplubix
7304fd654a
Merge pull request #34224 from CurtizJ/fix-query-metric 
Fix metric `Query`
 2022-02-02 13:39:07 +03:00
Yatsishin Ilya
759574d6f0 improve 2022-02-01 16:59:41 +00:00
Anton Popov
a4aa1f37ce remove unused code 2022-02-01 17:44:23 +03:00
Yatsishin Ilya
2a698b8da3 Merge remote-tracking branch 'origin/master' into default-table-engine 2022-02-01 13:23:52 +00:00
Vitaly Baranov
1f81e43f1e Fix checking grants for SHOW GRANTS. 2022-02-01 19:55:24 +07:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Anton Popov
c28255850a fix metric Query 2022-02-01 15:06:49 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
Ilya Yatsishin
9c2ef98d23
Apply suggestions from code review 
Co-authored-by: tavplubix <tavplubix@gmail.com>
 2022-02-01 10:03:50 +03:00
Alexander Tokmakov
5fad3fdffc throw exception on non-transactional queries 2022-02-01 01:27:55 +03:00
Amos Bird
ec7d367814
DiskLocal checker 
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
 2022-02-01 05:55:27 +08:00
Yatsishin Ilya
676fc5b20f Initial commit 2022-01-31 17:10:59 +00:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
Maksim Kita
9c91a50050 Sort block refactoring 2022-01-29 15:50:55 +00:00
Maksim Kita
f3453024ff
Merge pull request #34060 from amosbird/optimizetupleorderby 
Make ORDER BY tuple almost as fast as ORDER BY columns
 2022-01-29 15:58:09 +01:00
Amos Bird
04cff632db
Revise 2022-01-29 20:55:52 +08:00
李扬
6d50d36405
Merge branch 'master' into rocksdb_metacache 2022-01-28 22:00:31 -06:00
alexey-milovidov
6535b75322
Merge pull request #34001 from azat/memory-tracker-fix 
Fix memory accounting for queries that uses < max_untracker_memory
 2022-01-29 00:59:53 +03:00
Amos Bird
2b3ce910c6
Fix collation check 2022-01-29 04:49:58 +08:00
Federico Rodriguez
c0a5bd187a ExecuteScalarSubqueriesVisitor missing static const 
Added string_view
 2022-01-28 15:00:04 -05:00
Alexander Tokmakov
fb9b2d5326 Merge branch 'master' into mvcc_prototype 2022-01-28 21:18:36 +03:00
Alexander Tokmakov
e0304c2a58 review fixes, write tid into mutation entry 2022-01-28 20:47:37 +03:00
Amos Bird
09a3ab79b7
Fix all const 2022-01-29 01:37:39 +08:00
Anton Popov
fa883f2ac3 fix tests 2022-01-28 17:23:51 +03:00
Raúl Marín
6c79e147d0 Add a few extra comments around uses_view_source and why it's propagated 2022-01-28 14:47:49 +01:00
Azat Khuzhin
162f96f8e1 Get back detachQueryIfNotDetached() into ExternalLoader 
ExternalLoader from ThreadPool (async loading) is done from the server
context, not from the query context, and the context will already go
away, so we should detachQueryIfNotDetached() to avoid trigger assertion
in ThreadStatus.

CI: https://s3.amazonaws.com/clickhouse-test-reports/34001/8cace291d17fa9553a98b2a1e8bf15b30fe5a1bd/stateless_tests__debug__actions__[3/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Azat Khuzhin
b0c862c297 Fix memory accounting for queries that uses < max_untracker_memory 
MemoryTracker starts accounting memory directly only after per-thread
allocation exceeded max_untracker_memory (or memory_profiler_step).

But even memory under this limit should be accounted too, and there is
code to do this in ThreadStatus dtor, however due to
PullingAsyncPipelineExecutor detached the query from thread group that
memory was not accounted.

So remove CurrentThread::detachQueryIfNotDetached() from threads that
uses ThreadFromGlobalPool since it has ThreadStatus, and the query will
be detached using CurrentThread::defaultThreadDeleter.

Note, that before this patch memory accounting works for HTTP queries
due to it had been accounted from ParallelFormattingOutputFormat, but
not for TCP.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Raúl Marín
64799867a1 Add a comment about IInterpreterUnionOrSelectQuery::usesViewSource() 2022-01-28 12:39:56 +01:00
Nikolai Kochetov
94999e86c2
Merge pull request #33392 from azat/_shard_num 
Replace old _shard_num implementation with shardNum() function
 2022-01-28 11:02:19 +03:00
taiyang-li
3de8bde7ce Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-28 09:58:52 +08:00
alexey-milovidov
63d8d75001
Merge pull request #33957 from azat/fix-getauxval 
Fix getauxval() in glibc-compatibility and fix some leaks (after LSan started to work)
 2022-01-27 23:05:23 +03:00
tavplubix
8c9266b24c
Merge pull request #33978 from amosbird/separatesystemlog 
Separate base parts out of SystemLog
 2022-01-27 20:07:55 +03:00
Amos Bird
faee95b897
Make ORDER BY tuple almost as fast as ORDER BY columns 
We have special optimizations for multiple column ORDER BY: https://github.com/ClickHouse/ClickHouse/pull/10831 . It's beneficial to also apply to tuple columns.

Before:

select * from numbers(300000000) order by (1 - number , number + 1 , number) limit 10;
2.613 sec.

After:

select * from numbers(300000000) order by (1 - number , number + 1 , number) limit 10;
0.755 sec

No tuple:

select * from numbers(300000000) order by 1 - number , number + 1 , number limit 10;
0.755 sec
 2022-01-27 21:42:08 +08:00
Azat Khuzhin
af9910b518 Fix leak of demangle result (for opentelementry) 
LSan found [1]:

    Direct leak of 5170176 byte(s) in 5049 object(s) allocated from:
        0 0xc598edd in malloc (/usr/bin/clickhouse+0xc598edd)
        1 0x39679739 in (anonymous namespace)::itanium_demangle::initializeOutputStream(char*, unsigned long*, (anonymous namespace)::itanium_demangle::OutputStream&, unsigned long) obj-x86_64-linux-gnu/../contrib/libcxxabi/src/demangle/Utility.h:178:31
        2 0x39679739 in __cxa_demangle obj-x86_64-linux-gnu/../contrib/libcxxabi/src/cxa_demangle.cpp:351:13
        3 0x28f6f3ed in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:662:44

  [1]:
https://s3.amazonaws.com/clickhouse-test-reports/33957/08f4f45fd9da923ae3e3fdd8a527c297d35247eb/stress_test__address__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-27 13:46:37 +03:00
kssenii
c91b86e220 Better 2022-01-26 21:56:08 +03:00
Raúl Marín
045c92e2b9 Fix readability issues 2022-01-26 17:36:45 +01:00
Raúl Marín
78d2caee7e Can only pre-save scalars if they are cacheable 2022-01-26 17:36:45 +01:00
Raúl Marín
e77fc9e9be Comment 2022-01-26 17:36:45 +01:00
Raúl Marín
444a8e2519 Minimize changes, improve scalar subquery for MVs 2022-01-26 17:36:45 +01:00
Raúl Marín
4b5ab80e3b Better scalar cache handling 
- Fixes global CTE scalar cache.
- Adds MVs back (views dependent on the source are cached locally and others globally
 2022-01-26 17:36:45 +01:00
tavplubix
9e91a9dfd1
Merge pull request #33429 from zzsmdfj/MaterializeMySQL_support_set_and_other_dataType 
MaterializeMySQL support SET YEAR TIME GEOMETRY dataType
 2022-01-26 11:29:46 +03:00
Maksim Kita
bf768a94e8
Merge pull request #33971 from Avogar/fix-ipv6 
Fix parsing IPv6 from query parameter and fix IPv6 to string conversion
 2022-01-25 22:45:11 +01:00
Vitaly Baranov
cc38fe3fb6 Fix checking access for the SYSTEM command. 2022-01-25 18:54:59 +07:00
Amos Bird
385caa3e0a
Separate base parts out of SystemLog 2022-01-25 18:48:43 +08:00
avogar
ab49472155 Add comment 2022-01-25 10:10:04 +03:00
avogar
ad01d68f36 Fix parsing IPv6 from query parameter and fix IPv6 to string conversion 2022-01-25 10:09:17 +03:00
zhongyuankai
a6254516e0 Fix Alter ttl modification unsupported table engine 2022-01-24 21:48:52 +08:00
Ilya Yatsishin
35c03b317e
Merge pull request #33639 from Algunenano/async_metrics_handle_failures 
AsynchronousMetrics: Ignore inaccessible sensors
 2022-01-24 14:04:07 +03:00
tavplubix
da9a38655b
Merge pull request #33847 from ClickHouse/fix_33806 
Fix logical error on RESTORE REPLICA
 2022-01-24 13:28:37 +03:00
taiyang-li
73def8b483 merge master and solve conflict 2022-01-24 11:01:43 +08:00
Maksim Kita
7fffe5846f
Merge pull request #33924 from kitaisreal/sql-user-defined-functions-invalid-lambda-crash-fix 
SQLUserDefinedFunctions invalid lambda additional cases
 2022-01-23 22:06:49 +01:00
Maksim Kita
243a280936 Updated tests 2022-01-23 17:02:43 +00:00
alexey-milovidov
d7a63dfda6
Merge pull request #33677 from ucasfl/fix-sumif 
Fix sumIf rewrite
 2022-01-22 17:12:13 +03:00
Maksim Kita
1bc3f086b1
Merge pull request #33889 from kitaisreal/sql-user-defined-functions-invalid-lambda-crash-fix 
SQLUserDefinedFunctions invalid lambda additional fixes
 2022-01-22 12:09:42 +01:00
alexey-milovidov
eb6849f7c7
Merge pull request #33842 from azat/cmake-contrib-fixes 
More cmake external modules cleanups
 2022-01-22 10:34:54 +03:00
mergify[bot]
7734f9d629
Merge branch 'master' into fix-sumif 2022-01-22 06:08:47 +00:00
Maksim Kita
3a252cd3a0 Updated tests 2022-01-21 20:49:42 +00:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001 Merge branch 'master' into mvcc_prototype 2022-01-21 18:59:01 +03:00
Maksim Kita
330987a64b
Merge pull request #33868 from kitaisreal/sql-user-defined-functions-invalid-lambda-crash-fix 
SQLUserDefinedFunctions invalid lambda crash fix
 2022-01-21 16:23:12 +01:00
Maksim Kita
3b24c39294 SQLUserDefinedFunctions invalid lambda crash fix 2022-01-21 14:33:57 +00:00
Nikolai Kochetov
012453e212
Merge pull request #33588 from alexeypavlenko/fix_nested_default 
Fix default value generation for columns with dot
 2022-01-21 17:19:42 +03:00
Kruglov Pavel
daba57a933
Merge pull request #33737 from Avogar/bool-type 
Some improvements and fixes for Bool data type
 2022-01-21 15:24:58 +03:00
Alexander Tokmakov
8921725d71 fix logical error on RESTORE REPLICA 2022-01-21 14:32:19 +03:00
Azat Khuzhin
dc7affd813 contrib/libmetrohash: add ALIAS library 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-21 10:11:23 +03:00
zzsmdfj
411f43ec4d to MaterializeMySQL_support_set_and_other_dataType 2022-01-21 12:24:12 +08:00
zzsmdfj
e380e59b16 MaterializeMySQL support SET YEAR TIME GEOMETRY dataType 2022-01-21 12:24:11 +08:00
Maksim Kita
e2fe8dc3a7 Fix build with ENABLE_EXAMPLES 2022-01-20 19:29:21 +00:00
Alexander Tokmakov
13857ceb69 fixes 2022-01-20 21:15:23 +03:00
tavplubix
41ee114abb
Merge pull request #33819 from bigo-sg/optimize_explain_create_function 
Support explain create function query
 2022-01-20 19:07:43 +03:00
Alexander Tokmakov
c021fb2bd5 fix runtime linking error (lol) 2022-01-20 18:23:44 +03:00
Frank Chen
c1f9d6cd6f Fix traceparent propagation out of ClickHouse via URL engine 2022-01-20 23:12:23 +08:00
Alexander Tokmakov
0ea0a8ccc7 Merge branch 'master' into mvcc_prototype 2022-01-20 17:05:55 +03:00
taiyang-li
8886ac61e7 remove useless code 2022-01-20 19:33:19 +08:00
avogar
ba54a5ff80 Some improvements and fixes for Bool data type 2022-01-20 14:16:11 +03:00
taiyang-li
e4f5444073 support explain create function query 2022-01-20 16:23:52 +08:00
Azat Khuzhin
48b5c098dd Remove unbundled farmhash support 2022-01-20 10:01:12 +03:00
Azat Khuzhin
8ede97925e Remove unbundled sparsehash support 2022-01-20 10:01:11 +03:00
Frank Chen
c9f9fd2a38 Merge branch 'master' into trace_id_order 2022-01-20 13:07:50 +08:00
Alexander Tokmakov
ba5844f1d1 fix build after merge 2022-01-19 23:16:05 +03:00
Alexander Tokmakov
e9a5a64a71 Merge branch 'master' into mvcc_prototype 2022-01-19 21:41:23 +03:00
Alexander Tokmakov
7458012103 some fixes, add test with restart 2022-01-19 21:29:31 +03:00
Nikita Mikhaylov
e3de3889e3
Revert "Revert "Ignore parse failure of opentelemetry header (#32116)" (#33594)" (#33595) 2022-01-19 20:53:39 +03:00
Amos Bird
62441f0a0f
Fix mutation when table contains projections (#33679) 2022-01-19 15:27:11 +03:00
alexey-milovidov
d222cb9771
Merge pull request #33695 from amosbird/buildimprove 
Some build improvement
 2022-01-19 03:19:22 +03:00
Dmitry Novik
4e612d0e1a
Merge branch 'master' into memory-overcommit 2022-01-18 23:33:29 +03:00
Dmitry Novik
d7b4a32938 fix build 2022-01-18 20:26:12 +00:00
alesapin
e5ec1aa446
Merge pull request #33642 from ClickHouse/add_integration_test_for_zero_duplication 
Fix bug with background moves in zero copy replication
 2022-01-18 18:32:16 +03:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl 
RFC: Split headers, move SystemLog into module, more forward declarations
 2022-01-18 17:22:49 +03:00
Dmitry Novik
c0970b75ee
Merge branch 'master' into memory-overcommit 2022-01-18 15:30:24 +03:00
Dmitry Novik
83c663e2d6 Cleanup after code review 2022-01-18 12:21:59 +00:00
Maksim Kita
30dab61f97
Merge pull request #33526 from kitaisreal/dictionary-rename-fix 
Dictionary rename fix
 2022-01-18 13:11:34 +01:00
frank chen
da0608f365 Parse trace-id in big-endian order 
Signed-off-by: frank chen <frank.chen021@outlook.com>
 2022-01-18 13:24:56 +08:00
Nikolai Kochetov
96ba5b14c5 Update addMissingDefaults.cpp 2022-01-17 22:35:12 +02:00
Nikolai Kochetov
65519527ef Update addMissingDefaults.cpp 
Fix style
 2022-01-17 22:35:11 +02:00
alexeypavlenko
690590f7c3 Fix default value generation for columns with dot 2022-01-17 22:35:11 +02:00
Amos Bird
6d62060e16
Build improvement 2022-01-17 22:36:27 +08:00
alesapin
4caef03e6a Add ephemeral lock in zookeeper 2022-01-17 14:52:51 +03:00
feng lv
1cdadbd250 Fix sumIf rewrite 2022-01-16 06:25:11 +00:00
Alexey Milovidov
4ab7dfc741 Fix #8340 2022-01-16 08:12:55 +03:00
Maksim Kita
a0ad7a1014 Update IExternalLoadable interface 2022-01-16 00:06:10 +00:00
Raúl Marín
ac1665cdcb AsynchronousMetrics: Ignore inaccessible sensors 2022-01-14 15:52:08 +01:00
Alexander Tokmakov
4dd216ce8f Merge branch 'master' into mvcc_prototype 2022-01-14 17:11:32 +03:00
Alexander Tokmakov
e1a41fc694 add system log for introspection 2022-01-14 17:03:00 +03:00
Azat Khuzhin
4dcb332be9 Add ability to rewrite only JOIN tables in AddDefaultDatabaseVisitor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 11:18:52 +03:00
Azat Khuzhin
249bdce453 Fix AddDefaultDatabaseVisitor for temporary tables 
This will fix rewriting of distributed queries with GLOBAL IN

v2: get external tables only for non-global context

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 11:18:52 +03:00
Azat Khuzhin
0a9b1ee803 Remove RestoreQualifiedNamesMatcher::Data::rename (always true) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 11:18:52 +03:00
Nikita Mikhaylov
340ec0fd20
Revert "Ignore parse failure of opentelemetry header (#32116)" (#33594) 
This reverts commit 4e5f9c8ff7.
 2022-01-13 16:29:18 +04:00
Frank Chen
4e5f9c8ff7
Ignore parse failure of opentelemetry header (#32116) 2022-01-13 16:16:10 +04:00
tavplubix
1075f604ba
Merge pull request #33577 from DevTeamBK/Issue167 
skip MySQLCreateRewritten unit testsuite when enable_mysql is off
 2022-01-13 14:28:48 +03:00
taiyang-li
964f92f2c1 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-13 18:29:38 +08:00
tavplubix
3b7f8a4ae9
Merge pull request #33293 from zhanghuajieHIT/hangup_in_drop_query_log_sync 
fix hang up with command 'drop table system.query_log sync'
 2022-01-13 12:28:31 +03:00
taiyang-li
6567cd7abc fix all the stateless test 2022-01-13 15:27:41 +08:00
alexey-milovidov
6c71a7c40f
Merge pull request #33518 from hexiaoting/dev-limit-offset 
Fix bug for select from VIEW with format and limit settings
 2022-01-12 23:39:31 +03:00
Suzy Wang
ad11f3f05d skip MySQLCreateRewritten unit testsuite when enable_mysql is off 2022-01-12 11:05:56 -08:00
zhanghuajie
fc097173ef fix hang up with command 'drop table system.query_log sync' 2022-01-12 20:23:22 +08:00
Azat Khuzhin
248b879fd5 Split TraceCollector between Common and Interpreters 
This is required to leave TraceCollector::collect() inside main for
MemoryTrackier.
 2022-01-11 22:30:55 +03:00
hexiaoting
5c847264f7 Fix bug for select from VIEW with format and limit settings 2022-01-11 20:19:41 +08:00
Maksim Kita
8aa930b52f
Merge pull request #33294 from ClickHouse/query-kind-concurent_restriction 
Add IAST::QueryKind enum and use it in query limiter
 2022-01-11 11:23:18 +03:00
taiyang-li
6751c8552a merge master and resolve conflict 2022-01-11 15:21:56 +08:00
taiyang-li
90a92dd14a fix stateless test 2022-01-11 11:09:52 +08:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
c1dea66907 Move TraceCollector into Interpreters 
Since now it relies on SystemLog that is in Interpreters, and it cannot
be moved into Common, since it has lots of dependencies.
 2022-01-10 22:35:42 +03:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog 
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
 2022-01-10 22:01:41 +03:00
Azat Khuzhin
30e0df1e8c Forward declaration of IStorage in InterpreterWatchQuery 2022-01-10 22:01:41 +03:00
Azat Khuzhin
1637c41d42 Remove leftovers of old _shard_num via identifier implementation 2022-01-10 21:21:24 +03:00
Azat Khuzhin
97acf190ed Replace old _shard_num implementation with shardNum() function 
_shard_num via constant identifier (from #7624) has too much issues,
take a look at #16947 for example.

shardNum() function (from #27020) should works better.

This changes the behaviour of _shard_num slightly, now it returns nested
shard number in case of nested distributed tables (Distributed over
Distributed), but this should be minor.

v2: Rewrite _shard_num to shardNum() in TreeRewriter
 2022-01-10 21:21:24 +03:00
Mikhail f. Shiryaev
d35ad19135
Fix review points 2022-01-10 18:23:17 +01:00
Alexander Tokmakov
bf2b6c2c37 Merge branch 'master' into mvcc_prototype 2022-01-10 19:53:50 +03:00
Mikhail f. Shiryaev
a8c83dce14
Use map for QueryKindAmounts 2022-01-10 13:49:53 +01:00
Anton Popov
46b9279d81
Merge pull request #33300 from CurtizJ/support-prewhere-storage-merge 
Support `optimize_move_to_prewhere` with storage `Merge`
 2022-01-10 15:15:15 +03:00
taiyang-li
7e227040cb optimize getMergeTreeMetadataCache 2022-01-10 19:52:54 +08:00
Kseniia Sumarokova
e6ea7c361d
Merge pull request #32836 from stigsb/explain-table-override 
Add `EXPLAIN TABLE OVERRIDE` for pre-validating overrides.
 2022-01-10 09:58:16 +03:00
taiyang-li
3d409a4526 add metrics MergeTreeMetadataCacheSize 2022-01-10 12:20:39 +08:00
taiyang-li
5bff268f84 optimize code as advices 2022-01-10 11:45:06 +08:00
taiyang-li
1b4b727d79 fix conflict 2022-01-10 10:43:13 +08:00
Alexey Milovidov
3787bf72c2 Fix potential exponential growth of untuple function 2022-01-07 09:15:10 +03:00
taiyang-li
83d064c24a fix unit test and build error 2022-01-06 11:41:24 +08:00
taiyang-li
2bb7ec8f72 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-06 10:44:17 +08:00
Azat Khuzhin
5ed7440381 Fix possible use-after-free for INSERT into MV with concurrent DROP 
ASan founds [1]:

    ==553==
    ERROR: AddressSanitizer: heap-use-after-free on address 0x61e004694080 at pc 0x000029150af2 bp 0x7f70b3f8ef10 sp 0x7f70b3f8ef08
    READ of size 8 at 0x61e004694080 thread T477 (QueryPipelineEx)
        0 0x29150af1 in DB::MergeTreeDataWriter::writeTempPart() >
        1 0x293b8e43 in DB::MergeTreeSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeSink.cpp:27:65
        2 0x29dac73b in DB::SinkToStorage::onConsume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5
        3 0x29c72dd2 in DB::ExceptionKeepingTransform::work()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Processors/Transforms/ExceptionKeepingTransform.cpp:151:51

    0x61e004694080 is located 2048 bytes inside of 2480-byte region [0x61e004693880,0x61e004694230)
    freed by thread T199 (BgSchPool) here:
        ...
        4 0x26220f20 in DB::DatabaseCatalog::TableMarkedAsDropped::~TableMarkedAsDropped() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.h:248:12
        5 0x26220f20 in DB::DatabaseCatalog::dropTableDataTask() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:908:1

      [1]: https://s3.amazonaws.com/clickhouse-test-reports/33201/4f04d6af61eabf4899eb8188150dc862aaab80fc/stress_test__address__actions_.html

There was a fix in #32572, but it was not complete (yes it reduced the
race window a lot, but not completely), since the inner table still can
go away after the INSERT chain was built, to fix this obtain the
reference earlier.

Follow-up for: #32572 (cc @tavplubix)
 2022-01-04 13:34:01 +03:00
taiyang-li
abefefc719 support multi disk 2022-01-04 18:17:23 +08:00
taiyang-li
37c7c282d7 fix unit test 2022-01-04 13:41:11 +08:00
taiyang-li
e13e1f5d7e add unit test && use enum_name 2022-01-04 11:44:10 +08:00
taiyang-li
3f6d830536 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-04 10:00:34 +08:00
Maksim Kita
a7d0f3e637 UserDefinedExecutableFunction fix exception 2022-01-02 16:21:52 +03:00
alexey-milovidov
3737d83d3e
Merge pull request #33333 from ClickHouse/add-access-control-to-system-commands 
Add missing access control to some system commands
 2021-12-31 14:36:59 +03:00
alexey-milovidov
6941b072f4
Merge pull request #33330 from cmsxbc/support-self-closing-tag-for-cluster-secure 
support `<secure/>` in cluster configuration.
 2021-12-31 14:36:19 +03:00
Kseniia Sumarokova
a703bcb0c5
Merge pull request #33098 from bigo-sg/lrucache 
Add alternative LRUCache version
 2021-12-30 23:10:28 +03:00
Alexey Milovidov
d547fddf22 Add missing access control to some system commands 2021-12-30 21:37:21 +03:00
cmsxbc
c28fcd5a2f
support <secure/> for cluster 2021-12-30 23:51:05 +08:00
Maksim Kita
0593f1498a
Merge pull request #33285 from zbtzbtzbt/opt_hashTable 
rm redundant judge in hashmap iter operation
 2021-12-30 16:34:08 +03:00
Alexander Tokmakov
0d91887cdc save versions of parts 2021-12-30 16:15:28 +03:00
vdimir
ae7e5691b8
Get rid of IAST::getQueryKindString 2021-12-30 12:55:50 +03:00
Stig Bakken
6fbb9f5e87 Implement EXPLAIN TABLE OVERRIDE for pre-validating overrides. 
Example:

```sql
EXPLAIN TABLE OVERRIDE mysql('127.0.0.1:3306', 'db', 'table', 'user', 'pw')
    PARTITION BY tuple(toYYYYMM(created), id % 8)
```

Validations done:

 * check that storage overrides do not reference unknown or nullable columns
 * check that default specifier is not modified for columns
 2021-12-30 09:02:27 +01:00
Anton Popov
abcc48c1f6 Merge remote-tracking branch 'upstream/master' into support-prewhere-storage-merge 2021-12-29 21:07:35 +03:00
Kruglov Pavel
489a30859f
Merge pull request #32455 from Avogar/schema-inference 
Automatic schema inference for input formats
 2021-12-29 21:03:48 +03:00
Anton Popov
7c6f7f6732 support 'optimize_move_to_prewhere' with storage 'Merge' 2021-12-29 20:49:10 +03:00
Mikhail f. Shiryaev
a882e64644
Use IAST::QueryKind instead of strings in QueryKindAmount 2021-12-29 17:32:59 +01:00
Mikhail f. Shiryaev
272ea7fc5b
Merge pull request #32609 from cmsxbc/query-kind-concurent_restriction 
add settings: max_concurrent_select_queries and max_concurrent_insert_queries
 2021-12-29 15:23:45 +01:00
Nikolai Kochetov
c715204e18
Merge pull request #32271 from Algunenano/kill_scalar_github 
Be able to KILL scalar queries
 2021-12-29 16:00:59 +03:00
mergify[bot]
5070784282
Merge branch 'master' into memory-overcommit 2021-12-29 10:35:32 +00:00
Kruglov Pavel
dd7f61b337
Merge branch 'master' into schema-inference 2021-12-29 12:59:05 +03:00
zhoubintao
8690cf082f remove redundant judge to make heavy map iter operation faster 2021-12-29 17:48:39 +08:00
Maksim Kita
a5256e5b6e
Merge pull request #30957 from kitaisreal/executable-refactoring 
Executable refactoring
 2021-12-29 12:43:41 +03:00
Maksim Kita
c3c8af747d
Merge pull request #33261 from kitaisreal/dictionaries-fix-comment 
Dictionaries fix comment
 2021-12-29 12:43:24 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
lgbo-ustc
59cbd76880 Add LRUResourceCache 
1. add LRUResourceCache for managing resource cache in lru policy
2. rollback LRUCache to the original version
3. add remove() in LRUCache
4. add unit tests for LRUResourceCache and LRUCache
 2021-12-29 15:25:33 +08:00
Kseniia Sumarokova
9b63fa6949
Merge pull request #33239 from CurtizJ/async-insert-query-log 
Add asynchronous inserts to query log
 2021-12-29 09:34:08 +03:00
taiyang-li
63dc6821d2 fix ut and some bug 2021-12-29 12:31:54 +08:00
Anton Popov
d25560d37a fix queries without table name 2021-12-29 00:14:22 +03:00
Maksim Kita
9ef359ce2c Dictionaries fix comment 2021-12-28 23:50:48 +03:00
Maksim Kita
d1db3c9f42 Updated ShellCommandSource 2021-12-28 22:55:31 +03:00
Maksim Kita
5590cfa3aa Updated executable function integration tests 2021-12-28 22:55:30 +03:00
Maksim Kita
2ffd83ee83 Fix shell command source start 2021-12-28 22:55:30 +03:00
Maksim Kita
eb4e400c54 Executable refactoring 2021-12-28 22:55:30 +03:00
Anton Popov
a6e90a046d add more info to query log in async inserts 2021-12-28 17:53:25 +03:00
taiyang-li
98f37afc80 fix building 2021-12-28 19:29:01 +08:00
Alexander Tokmakov
8aec3ae94f Merge branch 'master' into mvcc_prototype 2021-12-28 14:24:36 +03:00
Alexander Tokmakov
f0b3c81212 add log in zookeeper 2021-12-28 14:23:35 +03:00
taiyang-li
2a1fe52b9f rename symbols && fix uts && add setting use_metadata_cache 2021-12-28 18:06:13 +08:00
taiyang-li
ae2078a58e Merge branch 'master' into rocksdb_metacache 2021-12-28 15:56:10 +08:00
Vladimir C
bb6fc853e3
Merge pull request #31442 from vdimir/cluster-discovery 2021-12-28 10:53:30 +03:00
taiyang-li
e070cdc247 Merge branch 'rocksdb_metacache' of https://github.com/bigo-sg/ClickHouse into rocksdb_metacache 2021-12-28 15:47:45 +08:00
taiyang-li
22b9f9d3e5 fix ut 2021-12-28 15:45:48 +08:00
李扬
91334ecf50
Merge branch 'master' into rocksdb_metacache 2021-12-27 23:20:41 -06:00
taiyang-li
7dab7caa9d wrap cache related code with USE_ROCKSDB 2021-12-28 11:57:43 +08:00
Anton Popov
0e2d0d70ba add asynchronous inserts to query log 2021-12-27 23:16:53 +03:00
vdimir
5f82190cd3
Log exception on cluster discovery initial update 2021-12-27 18:24:25 +03:00
vdimir
01a9199098
Move stop_flag to clusters_to_update in ClusterDiscovery 2021-12-27 16:10:36 +03:00
vdimir
5bd620fba1
Reset backoff when cluster discovery is up to date 2021-12-27 16:10:35 +03:00
vdimir
3934e99dce
update ClusterDiscovery::ConcurrentFlags::set 2021-12-27 16:09:50 +03:00
vdimir
a76e93209d
Backoff for ClusterDiscovery::runMainThread 2021-12-27 16:09:44 +03:00
vdimir
be7efb14c6
force updating cluster discovery periodically 2021-12-27 15:26:12 +03:00
vdimir
3d3fae126e
check if clusters updated successfully on current iteration of discovery 2021-12-27 15:26:12 +03:00
vdimir
36cd9ecb14
log level debug in cluster discovery 2021-12-27 15:26:11 +03:00
vdimir
72e63b4f12
restart runMainThread in cluster discovery 2021-12-27 15:26:11 +03:00
vdimir
4f8a9cc539
cluster discovery: versioning for format of data stored in zk 2021-12-27 15:26:11 +03:00
vdimir
d3b1058c40
cluster discovery configuration, support shards 2021-12-27 15:26:11 +03:00
vdimir
f47ce534f2
preserve auto clusters on config update 2021-12-27 15:26:10 +03:00
vdimir
d75a6b7abc
more logging for cluster discovery 2021-12-27 15:26:10 +03:00
vdimir
fbdb5c60bd
initial initialization of cluster discovery in main thread 2021-12-27 15:26:10 +03:00
vdimir
7a38716360
add 'secure' option to cluster discovery 2021-12-27 15:26:10 +03:00
vdimir
57cbbd9fcf
global option allow_experimental_cluster_discovery 2021-12-27 15:26:09 +03:00
vdimir
96a0371eca
add option allow_experimental_cluster_discovery 2021-12-27 15:26:09 +03:00
vdimir
ceec643f90
update config structure: remote_servers_discovery -> remote_servers.discovery 2021-12-27 15:26:09 +03:00
vdimir
7404205f37
fix cluster discovery startup race 2021-12-27 15:26:09 +03:00
vdimir
8bf3d94f04
write data for cluster discovery to zk in json 2021-12-27 15:26:09 +03:00
vdimir
76170156ef
log nodes update in cluster discovery 2021-12-27 15:26:08 +03:00
vdimir
a2d37fe2ed
use map and condvar instead of queue in cluster discovery 2021-12-27 15:26:08 +03:00
vdimir
5bbd151aae
style fix ClusterDiscovery 2021-12-27 15:26:08 +03:00
vdimir
e7d3dbeebd
cluster discovery init only with zookeeper 2021-12-27 15:26:08 +03:00
vdimir
4816d1afcc
cluster discovery: check cluster is changed, keep info 2021-12-27 15:26:07 +03:00
vdimir
aa0d79659b
check version in ClusterDiscovery::updateCluster 2021-12-27 15:26:07 +03:00
vdimir
84fb4354bc
cluster discovery worker thread 2021-12-27 15:26:06 +03:00
vdimir
94bb7cba62
cluster discovery wip 2021-12-27 15:26:05 +03:00
Vitaly Baranov
f80a3e3247
Merge pull request #30549 from aiven/kmichel-server-reload 
Start/stop servers when `listen_host`/`*_port` changes
 2021-12-27 14:51:00 +03:00
lgbo-ustc
ef1d7142f5 remove getOrTrySet 2021-12-27 16:12:39 +08:00
taiyang-li
e7401d2a5e wrap rocksdb metacache related code with USE_ROCKSDB 2021-12-27 11:50:59 +08:00
taiyang-li
94a7a09598 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2021-12-27 10:14:24 +08:00
alexey-milovidov
0a55fa3dc2
Revert "Grouping sets dev" 2021-12-25 20:30:31 +03:00
Alexey Milovidov
29d28c531f Move code around to avoid dlsym on Musl 2021-12-24 12:25:27 +03:00
Kevin Michel
ffc1fca296
Start/stop servers when listen_host/*_port changes 
This allows starting and stopping separately each protocol server
without restarting ClickHouse.

This also allows adding or removing `listen_host` entries, which
start and stops servers for all enabled ports.

When stopping a server, the listening socket is immediately closed
(and available for another server).

Protocols with persistent connections try to wait for any currently
running query to finish before closing the connection, but idle
connection are closed quickly (depending on how often the protocol
is polled).

An extra ProfileEvent is added, `MainConfigLoads`, it is
incremented every time the configuration is reloaded. This helps
when trying to assess whether the new configuration was applied.
 2021-12-24 08:26:02 +01:00
mergify[bot]
d4ec469ed1
Merge branch 'master' into query-kind-concurent_restriction 2021-12-24 07:02:00 +00:00
Alexey Milovidov
26e10c1fbb Remove unused header 2021-12-24 04:02:26 +03:00
Raúl Marín
cb22091b33 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-23 13:59:33 +01:00
Alexey Milovidov
d98675b49c Whitespace 2021-12-23 11:50:40 +03:00
Vitaly Baranov
b9090029e6
Merge pull request #32662 from vitlibar/improve-exceptions-usage-in-access-control 
Improve exceptions usage in access control
 2021-12-23 06:48:28 +03:00
Kseniia Sumarokova
764bb1181f
Merge pull request #33017 from stigsb/table-overrides-drop-add-column-support 
Remove the possibility of adding columns with table overrides
 2021-12-22 22:30:21 +03:00
Alexander Tokmakov
b7184732d5 Merge branch 'master' into mvcc_prototype 2021-12-22 19:46:09 +03:00
Raúl Marín
1d9916eb25 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-22 16:42:10 +01:00
tavplubix
d8fb293d57
Merge pull request #32900 from zzsmdfj/issue/#15182_MaterializeMySQL_support_bit_type 
Add support for BIT data type in MaterializedMySQL
 2021-12-22 15:58:29 +03:00
Stig Bakken
1cc70ad524 Remove the possibility of adding columns with table overrides 
(except ALIAS columns)

Supporting this properly requires more comprehensive changes.
 2021-12-22 08:20:14 +01:00
Kseniia Sumarokova
5ae2f0028f
Merge pull request #32961 from kssenii/fix-positional-args 
Fix for positional args
 2021-12-22 09:41:55 +03:00
kssenii
de70a57700 Better 2021-12-21 21:48:25 +03:00
Raúl Marín
eddccb0065 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-21 10:00:17 +01:00
kssenii
30996b2448 Fix 2021-12-21 01:19:51 +03:00
kssenii
175ad8a989 Fix 2021-12-21 00:20:08 +03:00
Alexander Tokmakov
9cd49bc0ec Merge branch 'master' into mvcc_prototype 2021-12-20 22:06:22 +03:00
Alexander Tokmakov
a15444031c add test 2021-12-20 21:53:05 +03:00
Vitaly Baranov
6721060649 Rename function IAccessStorage::login() -> IAccessStorage::authenticate(). 
Remove functions IAccessStorage::hasSubscriptionImpl() and IAccessStorage::existsImpl().
 2021-12-20 21:26:27 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment 
Loops remove postfix increment
 2021-12-20 19:51:07 +03:00
Maksim Kita
ebff389701
Merge pull request #32972 from kitaisreal/containers-iteration-fix-erase 
Containers iteration fix erase
 2021-12-20 16:47:19 +03:00
Maksim Kita
51477adf1b Updated additional cases 2021-12-20 15:55:07 +03:00
Nikolai Kochetov
2e62f086a1
Merge pull request #32751 from ClickHouse/fix-32668 
Apply some more optimizations to NO_QUERY ast.
 2021-12-20 15:47:25 +03:00
Maksim Kita
3feab5a975 Containers iteration fix erase 2021-12-20 13:42:31 +03:00
kssenii
965e2bfb21 Fix 2021-12-20 08:43:07 +03:00
liyang
37ba8004ff Speep up mergetree starting up process 2021-12-18 16:39:59 +08:00
kssenii
e6ebb55c4e Move to Interpreters 2021-12-17 22:19:10 +03:00
kssenii
dd902b6875 Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-17 22:03:58 +03:00
alexey-milovidov
6b97af4c63
Merge pull request #26869 from taylor12805/grouping-sets-dev 
Grouping sets dev
 2021-12-17 20:50:15 +03:00
Nikolai Kochetov
ffdcf52a9e
Merge pull request #32858 from ClickHouse/fix-has-column-in-table-subquery-2 
Always apply const-condition-if optimization.
 2021-12-17 20:39:30 +03:00
Nikolai Kochetov
93a33d52dc Merge branch 'fix-32668' of github.com:yandex/ClickHouse into fix-32668 2021-12-17 20:37:29 +03:00
Nikolai Kochetov
c8a92c046f Another try 2021-12-17 20:36:37 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
Nikolai Kochetov
20b88d9b4c Always apply const if optimixation. 2021-12-17 15:31:30 +03:00
alesapin
6bd7e425c6
Merge pull request #22535 from CurtizJ/sparse-serialization 
Sparse serialization and ColumnSparse
 2021-12-17 15:26:17 +03:00
Nikolai Kochetov
7b56e97cf8 Revert "Add debug logging" 
This reverts commit 49a9bc6b90.
 2021-12-17 13:26:05 +03:00
mergify[bot]
785d16d733
Merge branch 'master' into kill_scalar_github 2021-12-17 10:21:54 +00:00
Alexander Tokmakov
32e62ed5c2 Merge branch 'master' into mvcc_prototype 2021-12-17 10:43:46 +03:00
zzsmdfj
e9eed1f927 to #15182_MaterializeMySQL_support_bit_type 2021-12-17 15:03:24 +08:00
kssenii
97222be3d9 Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-17 01:14:28 +03:00
Nikolai Kochetov
49a9bc6b90 Add debug logging 2021-12-16 20:29:04 +03:00
Nikolai Kochetov
4799f07922
Merge branch 'master' into fix-has-column-in-table-subquery-2 2021-12-16 19:02:58 +03:00
Nikolai Kochetov
1b8ed6276a Allow string for constant if consition optmization. 2021-12-16 19:00:59 +03:00
Dmitry Novik
dce9390ece Init thread_group before QueryStatus creation 2021-12-16 16:46:15 +03:00
mergify[bot]
cc3176d437
Merge branch 'master' into memory-overcommit 2021-12-16 12:44:50 +00:00