Commit Graph

5679 Commits

Author SHA1 Message Date
Azat Khuzhin
dc7affd813 contrib/libmetrohash: add ALIAS library
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-21 10:11:23 +03:00
zzsmdfj
411f43ec4d to MaterializeMySQL_support_set_and_other_dataType 2022-01-21 12:24:12 +08:00
zzsmdfj
e380e59b16 MaterializeMySQL support SET YEAR TIME GEOMETRY dataType 2022-01-21 12:24:11 +08:00
Maksim Kita
e2fe8dc3a7 Fix build with ENABLE_EXAMPLES 2022-01-20 19:29:21 +00:00
tavplubix
41ee114abb
Merge pull request #33819 from bigo-sg/optimize_explain_create_function
Support explain create function query
2022-01-20 19:07:43 +03:00
Frank Chen
c1f9d6cd6f Fix traceparent propagation out of ClickHouse via URL engine 2022-01-20 23:12:23 +08:00
taiyang-li
8886ac61e7 remove useless code 2022-01-20 19:33:19 +08:00
avogar
ba54a5ff80 Some improvements and fixes for Bool data type 2022-01-20 14:16:11 +03:00
taiyang-li
e4f5444073 support explain create function query 2022-01-20 16:23:52 +08:00
Azat Khuzhin
48b5c098dd Remove unbundled farmhash support 2022-01-20 10:01:12 +03:00
Azat Khuzhin
8ede97925e Remove unbundled sparsehash support 2022-01-20 10:01:11 +03:00
Frank Chen
c9f9fd2a38 Merge branch 'master' into trace_id_order 2022-01-20 13:07:50 +08:00
Nikita Mikhaylov
e3de3889e3
Revert "Revert "Ignore parse failure of opentelemetry header (#32116)" (#33594)" (#33595) 2022-01-19 20:53:39 +03:00
Amos Bird
62441f0a0f
Fix mutation when table contains projections (#33679) 2022-01-19 15:27:11 +03:00
alexey-milovidov
d222cb9771
Merge pull request #33695 from amosbird/buildimprove
Some build improvement
2022-01-19 03:19:22 +03:00
Dmitry Novik
4e612d0e1a
Merge branch 'master' into memory-overcommit 2022-01-18 23:33:29 +03:00
Dmitry Novik
d7b4a32938 fix build 2022-01-18 20:26:12 +00:00
alesapin
e5ec1aa446
Merge pull request #33642 from ClickHouse/add_integration_test_for_zero_duplication
Fix bug with background moves in zero copy replication
2022-01-18 18:32:16 +03:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl
RFC: Split headers, move SystemLog into module, more forward declarations
2022-01-18 17:22:49 +03:00
Dmitry Novik
c0970b75ee
Merge branch 'master' into memory-overcommit 2022-01-18 15:30:24 +03:00
Dmitry Novik
83c663e2d6 Cleanup after code review 2022-01-18 12:21:59 +00:00
Maksim Kita
30dab61f97
Merge pull request #33526 from kitaisreal/dictionary-rename-fix
Dictionary rename fix
2022-01-18 13:11:34 +01:00
frank chen
da0608f365 Parse trace-id in big-endian order
Signed-off-by: frank chen <frank.chen021@outlook.com>
2022-01-18 13:24:56 +08:00
Nikolai Kochetov
96ba5b14c5 Update addMissingDefaults.cpp 2022-01-17 22:35:12 +02:00
Nikolai Kochetov
65519527ef Update addMissingDefaults.cpp
Fix style
2022-01-17 22:35:11 +02:00
alexeypavlenko
690590f7c3 Fix default value generation for columns with dot 2022-01-17 22:35:11 +02:00
Amos Bird
6d62060e16
Build improvement 2022-01-17 22:36:27 +08:00
alesapin
4caef03e6a Add ephemeral lock in zookeeper 2022-01-17 14:52:51 +03:00
feng lv
1cdadbd250 Fix sumIf rewrite 2022-01-16 06:25:11 +00:00
Alexey Milovidov
4ab7dfc741 Fix #8340 2022-01-16 08:12:55 +03:00
Maksim Kita
a0ad7a1014 Update IExternalLoadable interface 2022-01-16 00:06:10 +00:00
Raúl Marín
ac1665cdcb AsynchronousMetrics: Ignore inaccessible sensors 2022-01-14 15:52:08 +01:00
Azat Khuzhin
4dcb332be9 Add ability to rewrite only JOIN tables in AddDefaultDatabaseVisitor
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-14 11:18:52 +03:00
Azat Khuzhin
249bdce453 Fix AddDefaultDatabaseVisitor for temporary tables
This will fix rewriting of distributed queries with GLOBAL IN

v2: get external tables only for non-global context

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-14 11:18:52 +03:00
Azat Khuzhin
0a9b1ee803 Remove RestoreQualifiedNamesMatcher::Data::rename (always true)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-14 11:18:52 +03:00
Nikita Mikhaylov
340ec0fd20
Revert "Ignore parse failure of opentelemetry header (#32116)" (#33594)
This reverts commit 4e5f9c8ff7.
2022-01-13 16:29:18 +04:00
Frank Chen
4e5f9c8ff7
Ignore parse failure of opentelemetry header (#32116) 2022-01-13 16:16:10 +04:00
tavplubix
1075f604ba
Merge pull request #33577 from DevTeamBK/Issue167
skip MySQLCreateRewritten unit testsuite when enable_mysql is off
2022-01-13 14:28:48 +03:00
tavplubix
3b7f8a4ae9
Merge pull request #33293 from zhanghuajieHIT/hangup_in_drop_query_log_sync
fix hang up with command 'drop table system.query_log sync'
2022-01-13 12:28:31 +03:00
alexey-milovidov
6c71a7c40f
Merge pull request #33518 from hexiaoting/dev-limit-offset
Fix bug for select from VIEW with format and limit settings
2022-01-12 23:39:31 +03:00
Suzy Wang
ad11f3f05d skip MySQLCreateRewritten unit testsuite when enable_mysql is off 2022-01-12 11:05:56 -08:00
zhanghuajie
fc097173ef fix hang up with command 'drop table system.query_log sync' 2022-01-12 20:23:22 +08:00
Azat Khuzhin
248b879fd5 Split TraceCollector between Common and Interpreters
This is required to leave TraceCollector::collect() inside main for
MemoryTrackier.
2022-01-11 22:30:55 +03:00
hexiaoting
5c847264f7 Fix bug for select from VIEW with format and limit settings 2022-01-11 20:19:41 +08:00
Maksim Kita
8aa930b52f
Merge pull request #33294 from ClickHouse/query-kind-concurent_restriction
Add IAST::QueryKind enum and use it in query limiter
2022-01-11 11:23:18 +03:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
c1dea66907 Move TraceCollector into Interpreters
Since now it relies on SystemLog that is in Interpreters, and it cannot
be moved into Common, since it has lots of dependencies.
2022-01-10 22:35:42 +03:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
2022-01-10 22:01:41 +03:00
Azat Khuzhin
30e0df1e8c Forward declaration of IStorage in InterpreterWatchQuery 2022-01-10 22:01:41 +03:00
Azat Khuzhin
1637c41d42 Remove leftovers of old _shard_num via identifier implementation 2022-01-10 21:21:24 +03:00
Azat Khuzhin
97acf190ed Replace old _shard_num implementation with shardNum() function
_shard_num via constant identifier (from #7624) has too much issues,
take a look at #16947 for example.

shardNum() function (from #27020) should works better.

This changes the behaviour of _shard_num slightly, now it returns nested
shard number in case of nested distributed tables (Distributed over
Distributed), but this should be minor.

v2: Rewrite _shard_num to shardNum() in TreeRewriter
2022-01-10 21:21:24 +03:00
Mikhail f. Shiryaev
d35ad19135
Fix review points 2022-01-10 18:23:17 +01:00
Mikhail f. Shiryaev
a8c83dce14
Use map for QueryKindAmounts 2022-01-10 13:49:53 +01:00
Anton Popov
46b9279d81
Merge pull request #33300 from CurtizJ/support-prewhere-storage-merge
Support `optimize_move_to_prewhere` with storage `Merge`
2022-01-10 15:15:15 +03:00
Kseniia Sumarokova
e6ea7c361d
Merge pull request #32836 from stigsb/explain-table-override
Add `EXPLAIN TABLE OVERRIDE` for pre-validating overrides.
2022-01-10 09:58:16 +03:00
Alexey Milovidov
3787bf72c2 Fix potential exponential growth of untuple function 2022-01-07 09:15:10 +03:00
Azat Khuzhin
5ed7440381 Fix possible use-after-free for INSERT into MV with concurrent DROP
ASan founds [1]:

    ==553==
    ERROR: AddressSanitizer: heap-use-after-free on address 0x61e004694080 at pc 0x000029150af2 bp 0x7f70b3f8ef10 sp 0x7f70b3f8ef08
    READ of size 8 at 0x61e004694080 thread T477 (QueryPipelineEx)
        0 0x29150af1 in DB::MergeTreeDataWriter::writeTempPart() >
        1 0x293b8e43 in DB::MergeTreeSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeSink.cpp:27:65
        2 0x29dac73b in DB::SinkToStorage::onConsume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5
        3 0x29c72dd2 in DB::ExceptionKeepingTransform::work()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Processors/Transforms/ExceptionKeepingTransform.cpp:151:51

    0x61e004694080 is located 2048 bytes inside of 2480-byte region [0x61e004693880,0x61e004694230)
    freed by thread T199 (BgSchPool) here:
        ...
        4 0x26220f20 in DB::DatabaseCatalog::TableMarkedAsDropped::~TableMarkedAsDropped() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.h:248:12
        5 0x26220f20 in DB::DatabaseCatalog::dropTableDataTask() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:908:1

      [1]: https://s3.amazonaws.com/clickhouse-test-reports/33201/4f04d6af61eabf4899eb8188150dc862aaab80fc/stress_test__address__actions_.html

There was a fix in #32572, but it was not complete (yes it reduced the
race window a lot, but not completely), since the inner table still can
go away after the INSERT chain was built, to fix this obtain the
reference earlier.

Follow-up for: #32572 (cc @tavplubix)
2022-01-04 13:34:01 +03:00
Maksim Kita
a7d0f3e637 UserDefinedExecutableFunction fix exception 2022-01-02 16:21:52 +03:00
alexey-milovidov
3737d83d3e
Merge pull request #33333 from ClickHouse/add-access-control-to-system-commands
Add missing access control to some system commands
2021-12-31 14:36:59 +03:00
alexey-milovidov
6941b072f4
Merge pull request #33330 from cmsxbc/support-self-closing-tag-for-cluster-secure
support `<secure/>` in cluster configuration.
2021-12-31 14:36:19 +03:00
Kseniia Sumarokova
a703bcb0c5
Merge pull request #33098 from bigo-sg/lrucache
Add alternative LRUCache version
2021-12-30 23:10:28 +03:00
Alexey Milovidov
d547fddf22 Add missing access control to some system commands 2021-12-30 21:37:21 +03:00
cmsxbc
c28fcd5a2f
support <secure/> for cluster 2021-12-30 23:51:05 +08:00
Maksim Kita
0593f1498a
Merge pull request #33285 from zbtzbtzbt/opt_hashTable
rm redundant judge in hashmap iter operation
2021-12-30 16:34:08 +03:00
vdimir
ae7e5691b8
Get rid of IAST::getQueryKindString 2021-12-30 12:55:50 +03:00
Stig Bakken
6fbb9f5e87 Implement EXPLAIN TABLE OVERRIDE for pre-validating overrides.
Example:

```sql
EXPLAIN TABLE OVERRIDE mysql('127.0.0.1:3306', 'db', 'table', 'user', 'pw')
    PARTITION BY tuple(toYYYYMM(created), id % 8)
```

Validations done:

 * check that storage overrides do not reference unknown or nullable columns
 * check that default specifier is not modified for columns
2021-12-30 09:02:27 +01:00
Anton Popov
abcc48c1f6 Merge remote-tracking branch 'upstream/master' into support-prewhere-storage-merge 2021-12-29 21:07:35 +03:00
Kruglov Pavel
489a30859f
Merge pull request #32455 from Avogar/schema-inference
Automatic schema inference for input formats
2021-12-29 21:03:48 +03:00
Anton Popov
7c6f7f6732 support 'optimize_move_to_prewhere' with storage 'Merge' 2021-12-29 20:49:10 +03:00
Mikhail f. Shiryaev
a882e64644
Use IAST::QueryKind instead of strings in QueryKindAmount 2021-12-29 17:32:59 +01:00
Mikhail f. Shiryaev
272ea7fc5b
Merge pull request #32609 from cmsxbc/query-kind-concurent_restriction
add settings: max_concurrent_select_queries and max_concurrent_insert_queries
2021-12-29 15:23:45 +01:00
Nikolai Kochetov
c715204e18
Merge pull request #32271 from Algunenano/kill_scalar_github
Be able to KILL scalar queries
2021-12-29 16:00:59 +03:00
mergify[bot]
5070784282
Merge branch 'master' into memory-overcommit 2021-12-29 10:35:32 +00:00
Kruglov Pavel
dd7f61b337
Merge branch 'master' into schema-inference 2021-12-29 12:59:05 +03:00
zhoubintao
8690cf082f remove redundant judge to make heavy map iter operation faster 2021-12-29 17:48:39 +08:00
Maksim Kita
a5256e5b6e
Merge pull request #30957 from kitaisreal/executable-refactoring
Executable refactoring
2021-12-29 12:43:41 +03:00
Maksim Kita
c3c8af747d
Merge pull request #33261 from kitaisreal/dictionaries-fix-comment
Dictionaries fix comment
2021-12-29 12:43:24 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
lgbo-ustc
59cbd76880 Add LRUResourceCache
1. add LRUResourceCache for managing resource cache in lru policy
2. rollback LRUCache to the original version
3. add remove() in LRUCache
4. add unit tests for LRUResourceCache and LRUCache
2021-12-29 15:25:33 +08:00
Kseniia Sumarokova
9b63fa6949
Merge pull request #33239 from CurtizJ/async-insert-query-log
Add asynchronous inserts to query log
2021-12-29 09:34:08 +03:00
Anton Popov
d25560d37a fix queries without table name 2021-12-29 00:14:22 +03:00
Maksim Kita
9ef359ce2c Dictionaries fix comment 2021-12-28 23:50:48 +03:00
Maksim Kita
d1db3c9f42 Updated ShellCommandSource 2021-12-28 22:55:31 +03:00
Maksim Kita
5590cfa3aa Updated executable function integration tests 2021-12-28 22:55:30 +03:00
Maksim Kita
2ffd83ee83 Fix shell command source start 2021-12-28 22:55:30 +03:00
Maksim Kita
eb4e400c54 Executable refactoring 2021-12-28 22:55:30 +03:00
Anton Popov
a6e90a046d add more info to query log in async inserts 2021-12-28 17:53:25 +03:00
Vladimir C
bb6fc853e3
Merge pull request #31442 from vdimir/cluster-discovery 2021-12-28 10:53:30 +03:00
Anton Popov
0e2d0d70ba add asynchronous inserts to query log 2021-12-27 23:16:53 +03:00
vdimir
5f82190cd3
Log exception on cluster discovery initial update 2021-12-27 18:24:25 +03:00
vdimir
01a9199098
Move stop_flag to clusters_to_update in ClusterDiscovery 2021-12-27 16:10:36 +03:00
vdimir
5bd620fba1
Reset backoff when cluster discovery is up to date 2021-12-27 16:10:35 +03:00
vdimir
3934e99dce
update ClusterDiscovery::ConcurrentFlags::set 2021-12-27 16:09:50 +03:00
vdimir
a76e93209d
Backoff for ClusterDiscovery::runMainThread 2021-12-27 16:09:44 +03:00
vdimir
be7efb14c6
force updating cluster discovery periodically 2021-12-27 15:26:12 +03:00
vdimir
3d3fae126e
check if clusters updated successfully on current iteration of discovery 2021-12-27 15:26:12 +03:00
vdimir
36cd9ecb14
log level debug in cluster discovery 2021-12-27 15:26:11 +03:00
vdimir
72e63b4f12
restart runMainThread in cluster discovery 2021-12-27 15:26:11 +03:00
vdimir
4f8a9cc539
cluster discovery: versioning for format of data stored in zk 2021-12-27 15:26:11 +03:00
vdimir
d3b1058c40
cluster discovery configuration, support shards 2021-12-27 15:26:11 +03:00
vdimir
f47ce534f2
preserve auto clusters on config update 2021-12-27 15:26:10 +03:00
vdimir
d75a6b7abc
more logging for cluster discovery 2021-12-27 15:26:10 +03:00
vdimir
fbdb5c60bd
initial initialization of cluster discovery in main thread 2021-12-27 15:26:10 +03:00
vdimir
7a38716360
add 'secure' option to cluster discovery 2021-12-27 15:26:10 +03:00
vdimir
57cbbd9fcf
global option allow_experimental_cluster_discovery 2021-12-27 15:26:09 +03:00
vdimir
96a0371eca
add option allow_experimental_cluster_discovery 2021-12-27 15:26:09 +03:00
vdimir
ceec643f90
update config structure: remote_servers_discovery -> remote_servers.discovery 2021-12-27 15:26:09 +03:00
vdimir
7404205f37
fix cluster discovery startup race 2021-12-27 15:26:09 +03:00
vdimir
8bf3d94f04
write data for cluster discovery to zk in json 2021-12-27 15:26:09 +03:00
vdimir
76170156ef
log nodes update in cluster discovery 2021-12-27 15:26:08 +03:00
vdimir
a2d37fe2ed
use map and condvar instead of queue in cluster discovery 2021-12-27 15:26:08 +03:00
vdimir
5bbd151aae
style fix ClusterDiscovery 2021-12-27 15:26:08 +03:00
vdimir
e7d3dbeebd
cluster discovery init only with zookeeper 2021-12-27 15:26:08 +03:00
vdimir
4816d1afcc
cluster discovery: check cluster is changed, keep info 2021-12-27 15:26:07 +03:00
vdimir
aa0d79659b
check version in ClusterDiscovery::updateCluster 2021-12-27 15:26:07 +03:00
vdimir
84fb4354bc
cluster discovery worker thread 2021-12-27 15:26:06 +03:00
vdimir
94bb7cba62
cluster discovery wip 2021-12-27 15:26:05 +03:00
Vitaly Baranov
f80a3e3247
Merge pull request #30549 from aiven/kmichel-server-reload
Start/stop servers when `listen_host`/`*_port` changes
2021-12-27 14:51:00 +03:00
lgbo-ustc
ef1d7142f5 remove getOrTrySet 2021-12-27 16:12:39 +08:00
alexey-milovidov
0a55fa3dc2
Revert "Grouping sets dev" 2021-12-25 20:30:31 +03:00
Alexey Milovidov
29d28c531f Move code around to avoid dlsym on Musl 2021-12-24 12:25:27 +03:00
Kevin Michel
ffc1fca296
Start/stop servers when listen_host/*_port changes
This allows starting and stopping separately each protocol server
without restarting ClickHouse.

This also allows adding or removing `listen_host` entries, which
start and stops servers for all enabled ports.

When stopping a server, the listening socket is immediately closed
(and available for another server).

Protocols with persistent connections try to wait for any currently
running query to finish before closing the connection, but idle
connection are closed quickly (depending on how often the protocol
is polled).

An extra ProfileEvent is added, `MainConfigLoads`, it is
incremented every time the configuration is reloaded. This helps
when trying to assess whether the new configuration was applied.
2021-12-24 08:26:02 +01:00
mergify[bot]
d4ec469ed1
Merge branch 'master' into query-kind-concurent_restriction 2021-12-24 07:02:00 +00:00
Alexey Milovidov
26e10c1fbb Remove unused header 2021-12-24 04:02:26 +03:00
Raúl Marín
cb22091b33 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-23 13:59:33 +01:00
Alexey Milovidov
d98675b49c Whitespace 2021-12-23 11:50:40 +03:00
Vitaly Baranov
b9090029e6
Merge pull request #32662 from vitlibar/improve-exceptions-usage-in-access-control
Improve exceptions usage in access control
2021-12-23 06:48:28 +03:00
Kseniia Sumarokova
764bb1181f
Merge pull request #33017 from stigsb/table-overrides-drop-add-column-support
Remove the possibility of adding columns with table overrides
2021-12-22 22:30:21 +03:00
Raúl Marín
1d9916eb25 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-22 16:42:10 +01:00
tavplubix
d8fb293d57
Merge pull request #32900 from zzsmdfj/issue/#15182_MaterializeMySQL_support_bit_type
Add support for BIT data type in MaterializedMySQL
2021-12-22 15:58:29 +03:00
Stig Bakken
1cc70ad524 Remove the possibility of adding columns with table overrides
(except ALIAS columns)

Supporting this properly requires more comprehensive changes.
2021-12-22 08:20:14 +01:00
Kseniia Sumarokova
5ae2f0028f
Merge pull request #32961 from kssenii/fix-positional-args
Fix for positional args
2021-12-22 09:41:55 +03:00
kssenii
de70a57700 Better 2021-12-21 21:48:25 +03:00
Raúl Marín
eddccb0065 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-21 10:00:17 +01:00
kssenii
30996b2448 Fix 2021-12-21 01:19:51 +03:00
kssenii
175ad8a989 Fix 2021-12-21 00:20:08 +03:00
Vitaly Baranov
6721060649 Rename function IAccessStorage::login() -> IAccessStorage::authenticate().
Remove functions IAccessStorage::hasSubscriptionImpl() and IAccessStorage::existsImpl().
2021-12-20 21:26:27 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment
Loops remove postfix increment
2021-12-20 19:51:07 +03:00
Maksim Kita
ebff389701
Merge pull request #32972 from kitaisreal/containers-iteration-fix-erase
Containers iteration fix erase
2021-12-20 16:47:19 +03:00
Maksim Kita
51477adf1b Updated additional cases 2021-12-20 15:55:07 +03:00
Nikolai Kochetov
2e62f086a1
Merge pull request #32751 from ClickHouse/fix-32668
Apply some more optimizations to NO_QUERY ast.
2021-12-20 15:47:25 +03:00
Maksim Kita
3feab5a975 Containers iteration fix erase 2021-12-20 13:42:31 +03:00
kssenii
965e2bfb21 Fix 2021-12-20 08:43:07 +03:00
kssenii
e6ebb55c4e Move to Interpreters 2021-12-17 22:19:10 +03:00
kssenii
dd902b6875 Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-17 22:03:58 +03:00
alexey-milovidov
6b97af4c63
Merge pull request #26869 from taylor12805/grouping-sets-dev
Grouping sets dev
2021-12-17 20:50:15 +03:00
Nikolai Kochetov
ffdcf52a9e
Merge pull request #32858 from ClickHouse/fix-has-column-in-table-subquery-2
Always apply const-condition-if optimization.
2021-12-17 20:39:30 +03:00
Nikolai Kochetov
93a33d52dc Merge branch 'fix-32668' of github.com:yandex/ClickHouse into fix-32668 2021-12-17 20:37:29 +03:00
Nikolai Kochetov
c8a92c046f Another try 2021-12-17 20:36:37 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
Nikolai Kochetov
20b88d9b4c Always apply const if optimixation. 2021-12-17 15:31:30 +03:00
alesapin
6bd7e425c6
Merge pull request #22535 from CurtizJ/sparse-serialization
Sparse serialization and ColumnSparse
2021-12-17 15:26:17 +03:00
Nikolai Kochetov
7b56e97cf8 Revert "Add debug logging"
This reverts commit 49a9bc6b90.
2021-12-17 13:26:05 +03:00
mergify[bot]
785d16d733
Merge branch 'master' into kill_scalar_github 2021-12-17 10:21:54 +00:00
zzsmdfj
e9eed1f927 to #15182_MaterializeMySQL_support_bit_type 2021-12-17 15:03:24 +08:00
kssenii
97222be3d9 Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-17 01:14:28 +03:00
Nikolai Kochetov
49a9bc6b90 Add debug logging 2021-12-16 20:29:04 +03:00
Nikolai Kochetov
4799f07922
Merge branch 'master' into fix-has-column-in-table-subquery-2 2021-12-16 19:02:58 +03:00
Nikolai Kochetov
1b8ed6276a Allow string for constant if consition optmization. 2021-12-16 19:00:59 +03:00
Dmitry Novik
dce9390ece Init thread_group before QueryStatus creation 2021-12-16 16:46:15 +03:00
mergify[bot]
cc3176d437
Merge branch 'master' into memory-overcommit 2021-12-16 12:44:50 +00:00
Dmitry Novik
0f3e2bb32f Fix removing constant columns 2021-12-16 15:39:25 +03:00
Vladimir C
2f2d98d8be
Merge pull request #32811 from Algunenano/materialized_view_scalar_subqueries 2021-12-16 13:35:22 +03:00
alexey-milovidov
6083869b5d
Merge pull request #32669 from Algunenano/wip_mv_into_join
Fix LOGICAL_ERROR when the target of a materialized view is a JOIN or a SET table
2021-12-16 02:01:31 +03:00
alexey-milovidov
d112b30d78
Merge pull request #32291 from cmsxbc/fix-multiply-row-policies-on-same-column
Fix broken select query with multiply row policies on same column
2021-12-16 01:52:12 +03:00
kssenii
7583c8007e Some review fixes 2021-12-16 00:38:46 +03:00
Raúl Marín
c61277fbac Move the logic to the saving part instead
This way we can reuse scalars from other queries (non MVs) and
leave the door open for future improvements
2021-12-15 17:47:17 +01:00
Dmitry Novik
0d91864ec0 Avoid possible data race when thread_group is initialized 2021-12-15 18:11:17 +03:00
Maksim Kita
655cc20525
Merge pull request #32758 from kitaisreal/sql-user-defined-functions-support-subqueries
SQLUserDefinedFunctions support subqueries
2021-12-15 15:57:20 +03:00
kssenii
dc1f0c58fa Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-15 15:27:58 +03:00
mergify[bot]
bff0424139
Merge branch 'master' into kill_scalar_github 2021-12-15 09:22:58 +00:00
mergify[bot]
3a7179756f
Merge branch 'master' into fix-32668 2021-12-15 08:31:10 +00:00
mergify[bot]
dfdf886976
Merge branch 'master' into memory-overcommit 2021-12-15 00:00:28 +00:00
Dmitry Novik
56a3f4a000 Cleanup code 2021-12-14 22:15:14 +03:00
kssenii
47f549166d Table Override for PostgreSQL 2021-12-14 20:36:43 +03:00
mergify[bot]
c915b26a17
Merge branch 'master' into fix-multiply-row-policies-on-same-column 2021-12-14 16:21:35 +00:00
alesapin
d7663b2179 Merge branch 'master' into fix_special_build_check 2021-12-14 19:08:28 +03:00
alesapin
884801e1bd Fixing 2021-12-14 19:08:08 +03:00
Anton Popov
16312e7e4a Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-14 18:58:17 +03:00
Maksim Kita
c4bc72672e SQLUserDefinedFunctions support subqueries 2021-12-14 18:46:21 +03:00
Nikolai Kochetov
d394f0e753 Allpy some more optimizations to NO_QUERY ast. 2021-12-14 17:19:18 +03:00
Dmitry Novik
de432f9270 Fix possible NPE when thread_group is not set 2021-12-14 16:39:53 +03:00
Anton Popov
96070b414d support read_in_order optimization if prefix of sorting key is fixed 2021-12-14 15:54:20 +03:00
Anton Popov
bbb981655d
Merge pull request #32695 from CurtizJ/fix-add-ttl
Fix `MATERIALIZE TTL` with `TTL WHERE` and `TTL GROUP BY`
2021-12-14 14:01:15 +03:00
cmsxbc
b30e250eed add max_concurrent_select_queries and max_concurrent_insert_queries 2021-12-14 07:37:38 +00:00
mergify[bot]
96f6915233
Merge branch 'master' into wip_mv_into_join 2021-12-14 01:31:47 +00:00
Kseniia Sumarokova
dadaeabda7
Merge pull request #32670 from CurtizJ/fix-reading-in-order
Fix reading in order of sorting key from `Distributed` and `Merge` tables
2021-12-14 00:42:48 +03:00
Kseniia Sumarokova
a9ca84ff7d
Merge pull request #32325 from stigsb/table-overrides
Add TABLE OVERRIDE feature for MaterializedMySQL databases
2021-12-13 20:39:42 +03:00
Anton Popov
0b68980977 fix MATERIALIZE TTL with WHERE and GROUP BY 2021-12-13 17:21:05 +03:00
Kruglov Pavel
52328f6abc
Merge pull request #32490 from ClickHouse/quote-more-identifiers
Fix issue with ambiguous query formatting in distributed queries.
2021-12-13 16:41:01 +03:00
Anton Popov
745fd4fc16 fix reading in order from Distributed and Merge tables 2021-12-13 15:35:25 +03:00
Raúl Marín
b25cfa0b4d InterpreterInsertQuery::buildChain: Make sure to keep the context alive 2021-12-13 11:50:46 +01:00
mergify[bot]
b81d600c1e
Merge branch 'master' into memory-overcommit 2021-12-12 23:01:56 +00:00
tavplubix
be35ab171b
Merge pull request #32572 from azat/DROP-INSERT-UAF-fix
Fix table lifetime in case of parallel DROP TABLE and INSERT
2021-12-12 17:05:07 +03:00
cmsxbc
e4b405716f fix multiply row policies on same column 2021-12-12 09:54:07 +00:00
Alexey Milovidov
453135eb63 Fix indecent error message 2021-12-12 05:24:47 +03:00
Alexey Milovidov
58ffe89643 Merge branch 'master' into quote-more-identifiers 2021-12-12 03:33:20 +03:00
Azat Khuzhin
9a71598973 Fix table lifetime in case of parallel DROP TABLE and INSERT
Stress tests founds [1]:

==527==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x37078ffd in unsigned long std::__1::__cxx_atomic_fetch_add<unsigned long>(std::__1::__cxx_atomic_base_impl<unsigned long>*, unsigned long, std::__1::memory_order) obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:1050:12
    1 0x37078ffd in std::__1::__atomic_base<unsigned long, true>::fetch_add(unsigned long, std::__1::memory_order) obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:1719:17
    2 0x37078ffd in std::__1::__atomic_base<unsigned long, true>::operator++() obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:1756:57
    3 0x37078ffd in SimpleIncrement::get() obj-x86_64-linux-gnu/../src/Common/SimpleIncrement.h:20:16
    4 0x37078ffd in DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::shared_ptr<DB::Context const>) obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeDataWriter.cpp:276:46
    5 0x373c446c in DB::MergeTreeSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeSink.cpp:27:65

  Uninitialized value was created by a heap deallocation
    6 0x32d481e8 in DB::DatabaseCatalog::TableMarkedAsDropped::~TableMarkedAsDropped() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.h:248:12
    7 0x32d3c134 in DB::DatabaseCatalog::dropTableDataTask() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:908:1

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/32534/fa6090f588dbf4cbb5f28bd2210847b070bb8218/stress_test__memory__actions_.html

The query was CREATE MATERIALIZED VIEW ... POPULATE AS SELECT ... from
00040_aggregating_materialized_view test.
2021-12-11 11:00:31 +03:00
alexey-milovidov
c90e588e24
Merge pull request #32506 from ClickHouse/fix-has-column-in-table-subquery
Fix queries with hasColumnInTable constant condition and non existing column
2021-12-11 02:50:06 +03:00
alexey-milovidov
17e5f5ccfe
Merge pull request #32508 from vdimir/issue_32458
Handle const column in JoinCommon::removeColumnNullability
2021-12-11 01:21:10 +03:00
Anton Popov
37837f3881
Merge pull request #32453 from kssenii/disable-optimizations-for-window-functions
Disable some optimizations for window functions
2021-12-11 00:37:24 +03:00
vdimir
ed4ea6fe3a
Handle const column in JoinCommon::removeColumnNullability 2021-12-10 15:57:36 +03:00
Nikolai Kochetov
7bf5c83561 Fix queries with hasColumnInTable constant condition and non existing column. 2021-12-10 15:45:04 +03:00
Alexey Milovidov
5c90ed2ed9 Unambiguous formatting of distributed queries 2021-12-10 00:55:14 +03:00
Raúl Marín
3a7e524f9e Style 2021-12-09 18:24:28 +01:00
Raúl Marín
3fc4167c54 Rework how progress is reported in views 2021-12-09 17:08:29 +01:00
Nikolai Kochetov
e41974fabf
Merge branch 'master' into fix-31538 2021-12-09 19:00:22 +03:00
Nikolai Kochetov
272c0bb35a Fix crash in case of MATERIALIZE COLUMN with no default expression. 2021-12-09 18:58:28 +03:00
kssenii
40c9ffdfff Disable 2021-12-09 14:15:23 +03:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
Raúl Marín
0158133b62 SourceWithProgress: Don't use process_list_elem without checking it 2021-12-09 10:02:32 +01:00
Anton Popov
61a5f8a61a add comments 2021-12-08 18:56:30 +03:00
Anton Popov
d8367334a3 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-08 18:26:19 +03:00
Nikita Mikhaylov
7a8fe9a9ad
Merge pull request #32234 from FrankChen021/span_name
Improve the operation name of an opentelemetry span
2021-12-08 15:46:19 +03:00
Nikita Mikhaylov
9a6a1379fd
Merge pull request #32358 from nikitamikhaylov/mutation-livelock
Fixed empty mutations livelock
2021-12-08 15:43:19 +03:00
Kruglov Pavel
e0a8c5a4ac
Merge pull request #32348 from CurtizJ/fix-materialize-column
Add cast to `MATERIALIZE COLUMN`
2021-12-08 11:13:17 +03:00
Nikita Mikhaylov
76f91c2c6c Done 2021-12-07 16:50:57 +00:00
Anton Popov
ef7ae7cb75 fix MATERIALIZE COLUMN in case when data type of expression is not equal to data type of column 2021-12-07 16:45:30 +03:00
mergify[bot]
79351e9d59
Merge branch 'master' into span_name 2021-12-07 12:05:48 +00:00
Dmitry Novik
8d8222acf2 Fix build with gcc 2021-12-07 13:11:31 +03:00
mergify[bot]
907a042af3
Merge branch 'master' into memory-overcommit 2021-12-07 09:50:37 +00:00
Stig Bakken
bb521cd363 Add TABLE OVERRIDE feature for MaterializedMySQL databases 2021-12-07 09:45:02 +01:00
dongyifeng
9df664e1c9
fix bug when remove unneeded columns in subquery (#32289) 2021-12-07 11:09:39 +03:00
mergify[bot]
600dcb749a
Merge branch 'master' into memory-overcommit 2021-12-07 00:40:20 +00:00
Dmitry Novik
12101d82aa Fix overcommit ratio comparison and race condition 2021-12-06 21:34:52 +03:00
tavplubix
657db07795
Merge pull request #32288 from ClickHouse/fix_xml_dict_cyclic_dependency
Fix cyclic dependency with XML dictionary and StorageDictionary
2021-12-06 20:02:38 +03:00
Alexander Tokmakov
7797a72907 fix false-positive ciclyc dependency with XML dict 2021-12-06 16:35:43 +03:00
Maksim Kita
11df9a6bc4
Merge pull request #32187 from kitaisreal/dictionary-qualify-name-with-database-fix
DictionariesLoader qualify name with database fix
2021-12-06 15:11:09 +03:00
Maksim Kita
1ec9039b1d ExternalDictionariesLoader fix getCurrentDatabase multiple times 2021-12-06 14:01:56 +03:00
Kruglov Pavel
007e323515
Merge pull request #31820 from kitaisreal/fix-recursive-user-defined-functions-crash
Fix recursive user defined functions crash
2021-12-06 13:52:56 +03:00
Nikolai Kochetov
91c4c89c66
Merge pull request #31636 from Algunenano/pull_timeout
Check max_execution_time in the pipeline and pulling executors
2021-12-06 11:33:39 +03:00
frank chen
55d50c0b4d Improve span operation name
Signed-off-by: frank chen <frank.chen021@outlook.com>
2021-12-04 21:56:52 +08:00
Maksim Kita
4c916a0e38 DictionariesLoader qualify name with database fix 2021-12-03 14:31:49 +03:00
mergify[bot]
478c35538d
Merge branch 'master' into trace_finish_time_bug 2021-12-03 10:12:48 +00:00
Kruglov Pavel
17d6f539b3
Merge pull request #32019 from azat/dist-desc
Fix excessive DESC TABLE for remote() over identifier
2021-12-03 12:15:05 +03:00
Maksim Kita
74979a5f01
Merge pull request #31095 from amosbird/addprojection
Function name normalization for ALTER queries
2021-12-03 12:07:07 +03:00
Anton Popov
f6be3d16fd
Merge pull request #24820 from kssenii/versioning
Versioning of aggregate function states
2021-12-03 01:41:44 +03:00
Frank Chen
e895c85f49
Add exception/exception_code to trace span log (#32040) 2021-12-03 01:00:02 +03:00
Raúl Marín
755ba5dc09 Don't forget to check the output of checkTimeLimitSoft 2021-12-02 15:09:53 +01:00
Raúl Marín
5662d0aa59 Use softer checks 2021-12-02 14:53:55 +01:00
Kruglov Pavel
049b2c0c14
Merge pull request #31072 from MaxWk/feature-support-bool-type
Feature support bool type
2021-12-02 14:50:29 +03:00
kssenii
8edf146687 Merge branch 'master' of github.com:ClickHouse/ClickHouse into versioning 2021-12-02 07:57:14 +00:00
Azat Khuzhin
4535232f4a Fix excessive DESC TABLE for remote() over identifier
Fixes: #14228
2021-12-01 22:52:20 +03:00
Anton Popov
6f4d9a53b2 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-12-01 15:54:33 +03:00
Anton Popov
54f51444c0 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-01 15:49:02 +03:00
Maksim Kita
0cba91f63d SQLUserDefinedFunctionVisitor updated exception message 2021-12-01 11:41:44 +03:00
frank chen
3678df7e1e Fix finish time
Signed-off-by: frank chen <frank.chen021@outlook.com>
2021-12-01 11:32:56 +08:00
Anton Popov
63fceca6a8 support nested in json type (wip) 2021-12-01 05:59:56 +03:00
Kseniia Sumarokova
3c348a2998
Merge pull request #8331 from Vxider/window-view
Support for Stream Processing
2021-11-30 18:20:12 +03:00
kssenii
71bfc72e37 Fix 2021-11-30 14:42:37 +00:00
fanzhou
b94f8878ff 1. refactor ExpressionAnalyzer
2. Update formatAST to support grouping sets query with distributed table
3. modify astExpression to support function in grouping sets
2021-11-30 14:58:55 +03:00
Kseniia Sumarokova
a515767c51
Merge pull request #31741 from kssenii/fix-positional-args-setting
Fix aliases with positional args enabled
2021-11-30 13:17:45 +03:00
kssenii
f658a8c181 Small simplification, some comments 2021-11-30 11:16:24 +03:00
alexey-milovidov
bab8ea144b
Merge pull request #31866 from ClickHouse/fix-31687
Support toUInt8/toInt8 for if constant condition optimization.
2021-11-30 05:54:58 +03:00
mergify[bot]
c8533aab07
Merge branch 'master' into fix-positional-args-setting 2021-11-29 20:53:55 +00:00
Kseniia Sumarokova
edf1ec776a
Merge pull request #31824 from azat/system-log-tables-recreate-different-engine
Recreate system.*_log tables in case of different engine/partition_by
2021-11-29 22:56:36 +03:00
tavplubix
be5bb7fe25
Merge pull request #31295 from ClickHouse/tavplubix-patch-4
Add assertions to ZooKeeperLock
2021-11-29 22:00:25 +03:00
Raúl Marín
a7ae715950 Merge remote-tracking branch 'blessed/master' into pull_timeout 2021-11-29 19:51:18 +01:00
fanzhou
e725630fbf refactor 2021-11-29 19:35:33 +03:00
fanzhou
43db4594ba some changes 2021-11-29 19:35:33 +03:00
fanzhou
534160b0d4 grouing sets transformer instead of fork processor in aggregating transformer 2021-11-29 18:59:15 +03:00
fanzhou
2912344896 grouping sets support multiple group 2021-11-29 18:32:55 +03:00
Kseniia Sumarokova
8f0fc225b1
Update ExpressionAnalyzer.cpp 2021-11-29 18:07:11 +03:00
Nikolai Kochetov
7bf7735470
Update OptimizeIfWithConstantConditionVisitor.cpp 2021-11-29 17:31:54 +03:00
Kseniia Sumarokova
c7a8ad69c4
Merge pull request #31825 from azat/query_views_log-test
Cover query_views_log
2021-11-29 16:10:56 +03:00
mergify[bot]
8d5460b469
Merge branch 'master' into feature-support-bool-type 2021-11-29 11:50:18 +00:00
Maksim Kita
05da57f7a6 Fix identity user defined function 2021-11-29 13:19:54 +03:00
kssenii
be3b4ca8fe Merge branch 'master' of github.com:ClickHouse/ClickHouse into versioning 2021-11-27 09:44:31 +00:00
kssenii
515261f5dd Better 2021-11-27 09:40:46 +00:00
alexey-milovidov
83391977f8
Merge pull request #31896 from Algunenano/parser_deps
Reduce the files that depend on parser headers
2021-11-27 00:18:31 +03:00
MaxTheHuman
b39f5bd45f grouping-sets: rearrange result columns so that resize is possible after grouping sets 2021-11-26 23:25:25 +03:00
MaxTheHuman
9953e1bc0a grouping sets: add tests, fix bug 2021-11-26 23:09:23 +03:00
MaxTheHuman
2640f51843 grouping sets: fix 2021-11-26 23:05:07 +03:00
MaxTheHuman
4f07ec2659 grouping sets: make simple aggregation with grouping sets to work 2021-11-26 22:59:56 +03:00
MaxTheHuman
e32d58a36d grouping sets dev: fix errors, something works 2021-11-26 22:46:15 +03:00
MaxTheHuman
e7711d953a grouping sets cleanup 2021-11-26 22:33:14 +03:00
MaxTheHuman
ddd1799743 grouping sets development 2021-11-26 22:11:34 +03:00
MaxTheHuman
d2258decf5 grouping sets development 2021-11-26 21:50:03 +03:00
MaxTheHuman
e60d1dd818 grouping sets development 2021-11-26 21:38:44 +03:00
Raúl Marín
7781fc12ed Reduce dependencies on ASTSelectWithUnionQuery.h
521 -> 77 files requiring changes
2021-11-26 19:27:16 +01:00
MaxTheHuman
3418de337a development 2021-11-26 20:37:51 +03:00
Raúl Marín
f0ee0724ac Reduce dependencies on ASTSelectQuery.h
243 -> 152
2021-11-26 18:35:24 +01:00
MaxTheHuman
2bd07ef338 feat grouping-sets: initial changes 2021-11-26 20:24:35 +03:00
Raúl Marín
b2cfa70541 Reduce dependencies on ASTFunction.h
481 -> 230
2021-11-26 18:21:54 +01:00
Raúl Marín
91bf938075 Reduce dependencies on ASTLiteral.h
590 -> 537
2021-11-26 17:54:57 +01:00
Raúl Marín
0b6ae2e508 Reduce dependencies on IParser.h
533 -> 500 files rebuilt on change
2021-11-26 17:04:21 +01:00
Raúl Marín
051dddd8df Reduce dependencies on ASTIdentifier.h
Goes from rebuilding 483 objects to 165 when it's modified
2021-11-26 16:49:40 +01:00
Anton Popov
982163c54b
Merge pull request #31886 from CurtizJ/fix-typos
Fix typos
2021-11-26 18:18:11 +03:00
Anton Popov
923159d408 fix typos 2021-11-26 17:47:40 +03:00
Nikolai Kochetov
185c20cf88 Support toUInt8/toInt8 for if constant condition optimization. 2021-11-26 15:38:32 +03:00
Raúl Marín
c498b7ba59 Move limits check to ProcessList 2021-11-26 12:44:39 +01:00
Raúl Marín
be60759e68 Merge remote-tracking branch 'blessed/master' into pull_timeout 2021-11-26 11:37:19 +01:00
Azat Khuzhin
5fafeea763 Recreate system.*_log tables in case of different engine/partition_by.
- Mark getCreateTableQuery() non virtual to call it from ctor
- Make check for CREATE TABLE simpler (by caching result)
- Extend error message.

v2: at SystemLog tables has not been initialized yet, so SHOW CREATE
cannot be obtained for the existing table, move the check to the
prepareTable() call.
v3: reset UUID and SETTINGS of MergeTree in CREATE TABLE query
v4: fix re-creating table with the same UUID
2021-11-26 10:05:20 +03:00
Azat Khuzhin
f0628073bb Use comments that can be parsed by clang-tidy in ThreadStatus::initQueryProfiler() 2021-11-26 01:13:55 +03:00
Azat Khuzhin
2840405c8a Fix typo in ThreadStatus::query_profiler_enabled 2021-11-26 01:13:55 +03:00
kssenii
223048d395 Merge branch 'master' of github.com:ClickHouse/ClickHouse into vxider-window-view 2021-11-26 01:06:06 +03:00
Maksim Kita
df3ac768db Fix recursive user defined functions crash 2021-11-25 19:03:56 +03:00
tavplubix
f320a404e0
Fix 'there are no such cluster here' (#31723)
* Update Context.cpp

* Update DDLTask.cpp
2021-11-25 17:28:06 +03:00
Kruglov Pavel
f28130193a
Merge branch 'master' into fix-whole-text-serialization 2021-11-25 14:00:20 +03:00
Kseniia Sumarokova
93cf66df12
Merge pull request #30936 from kssenii/seekable-read-buffers
Reduce memory usage for some formats when reading with s3/url/hdfs
2021-11-25 11:19:24 +03:00