thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-12 01:12:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
81,890 Commits 9,297 Branches 1,782 Tags 2.1 GiB
385caa3e0a
Commit Graph

5390 Commits

Author SHA1 Message Date
Maksim Kita
8aa930b52f
Merge pull request #33294 from ClickHouse/query-kind-concurent_restriction 
Add IAST::QueryKind enum and use it in query limiter
 2022-01-11 11:23:18 +03:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
c1dea66907 Move TraceCollector into Interpreters 
Since now it relies on SystemLog that is in Interpreters, and it cannot
be moved into Common, since it has lots of dependencies.
 2022-01-10 22:35:42 +03:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog 
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
 2022-01-10 22:01:41 +03:00
Azat Khuzhin
30e0df1e8c Forward declaration of IStorage in InterpreterWatchQuery 2022-01-10 22:01:41 +03:00
Mikhail f. Shiryaev
d35ad19135
Fix review points 2022-01-10 18:23:17 +01:00
Mikhail f. Shiryaev
a8c83dce14
Use map for QueryKindAmounts 2022-01-10 13:49:53 +01:00
Anton Popov
46b9279d81
Merge pull request #33300 from CurtizJ/support-prewhere-storage-merge 
Support `optimize_move_to_prewhere` with storage `Merge`
 2022-01-10 15:15:15 +03:00
Kseniia Sumarokova
e6ea7c361d
Merge pull request #32836 from stigsb/explain-table-override 
Add `EXPLAIN TABLE OVERRIDE` for pre-validating overrides.
 2022-01-10 09:58:16 +03:00
Alexey Milovidov
3787bf72c2 Fix potential exponential growth of untuple function 2022-01-07 09:15:10 +03:00
Azat Khuzhin
5ed7440381 Fix possible use-after-free for INSERT into MV with concurrent DROP 
ASan founds [1]:

    ==553==
    ERROR: AddressSanitizer: heap-use-after-free on address 0x61e004694080 at pc 0x000029150af2 bp 0x7f70b3f8ef10 sp 0x7f70b3f8ef08
    READ of size 8 at 0x61e004694080 thread T477 (QueryPipelineEx)
        0 0x29150af1 in DB::MergeTreeDataWriter::writeTempPart() >
        1 0x293b8e43 in DB::MergeTreeSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeSink.cpp:27:65
        2 0x29dac73b in DB::SinkToStorage::onConsume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5
        3 0x29c72dd2 in DB::ExceptionKeepingTransform::work()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Processors/Transforms/ExceptionKeepingTransform.cpp:151:51

    0x61e004694080 is located 2048 bytes inside of 2480-byte region [0x61e004693880,0x61e004694230)
    freed by thread T199 (BgSchPool) here:
        ...
        4 0x26220f20 in DB::DatabaseCatalog::TableMarkedAsDropped::~TableMarkedAsDropped() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.h:248:12
        5 0x26220f20 in DB::DatabaseCatalog::dropTableDataTask() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:908:1

      [1]: https://s3.amazonaws.com/clickhouse-test-reports/33201/4f04d6af61eabf4899eb8188150dc862aaab80fc/stress_test__address__actions_.html

There was a fix in #32572, but it was not complete (yes it reduced the
race window a lot, but not completely), since the inner table still can
go away after the INSERT chain was built, to fix this obtain the
reference earlier.

Follow-up for: #32572 (cc @tavplubix)
 2022-01-04 13:34:01 +03:00
Maksim Kita
a7d0f3e637 UserDefinedExecutableFunction fix exception 2022-01-02 16:21:52 +03:00
alexey-milovidov
3737d83d3e
Merge pull request #33333 from ClickHouse/add-access-control-to-system-commands 
Add missing access control to some system commands
 2021-12-31 14:36:59 +03:00
alexey-milovidov
6941b072f4
Merge pull request #33330 from cmsxbc/support-self-closing-tag-for-cluster-secure 
support `<secure/>` in cluster configuration.
 2021-12-31 14:36:19 +03:00
Kseniia Sumarokova
a703bcb0c5
Merge pull request #33098 from bigo-sg/lrucache 
Add alternative LRUCache version
 2021-12-30 23:10:28 +03:00
Alexey Milovidov
d547fddf22 Add missing access control to some system commands 2021-12-30 21:37:21 +03:00
cmsxbc
c28fcd5a2f
support <secure/> for cluster 2021-12-30 23:51:05 +08:00
Maksim Kita
0593f1498a
Merge pull request #33285 from zbtzbtzbt/opt_hashTable 
rm redundant judge in hashmap iter operation
 2021-12-30 16:34:08 +03:00
vdimir
ae7e5691b8
Get rid of IAST::getQueryKindString 2021-12-30 12:55:50 +03:00
Stig Bakken
6fbb9f5e87 Implement EXPLAIN TABLE OVERRIDE for pre-validating overrides. 
Example:

```sql
EXPLAIN TABLE OVERRIDE mysql('127.0.0.1:3306', 'db', 'table', 'user', 'pw')
    PARTITION BY tuple(toYYYYMM(created), id % 8)
```

Validations done:

 * check that storage overrides do not reference unknown or nullable columns
 * check that default specifier is not modified for columns
 2021-12-30 09:02:27 +01:00
Anton Popov
abcc48c1f6 Merge remote-tracking branch 'upstream/master' into support-prewhere-storage-merge 2021-12-29 21:07:35 +03:00
Kruglov Pavel
489a30859f
Merge pull request #32455 from Avogar/schema-inference 
Automatic schema inference for input formats
 2021-12-29 21:03:48 +03:00
Anton Popov
7c6f7f6732 support 'optimize_move_to_prewhere' with storage 'Merge' 2021-12-29 20:49:10 +03:00
Mikhail f. Shiryaev
a882e64644
Use IAST::QueryKind instead of strings in QueryKindAmount 2021-12-29 17:32:59 +01:00
Mikhail f. Shiryaev
272ea7fc5b
Merge pull request #32609 from cmsxbc/query-kind-concurent_restriction 
add settings: max_concurrent_select_queries and max_concurrent_insert_queries
 2021-12-29 15:23:45 +01:00
Nikolai Kochetov
c715204e18
Merge pull request #32271 from Algunenano/kill_scalar_github 
Be able to KILL scalar queries
 2021-12-29 16:00:59 +03:00
Kruglov Pavel
dd7f61b337
Merge branch 'master' into schema-inference 2021-12-29 12:59:05 +03:00
zhoubintao
8690cf082f remove redundant judge to make heavy map iter operation faster 2021-12-29 17:48:39 +08:00
Maksim Kita
a5256e5b6e
Merge pull request #30957 from kitaisreal/executable-refactoring 
Executable refactoring
 2021-12-29 12:43:41 +03:00
Maksim Kita
c3c8af747d
Merge pull request #33261 from kitaisreal/dictionaries-fix-comment 
Dictionaries fix comment
 2021-12-29 12:43:24 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
lgbo-ustc
59cbd76880 Add LRUResourceCache 
1. add LRUResourceCache for managing resource cache in lru policy
2. rollback LRUCache to the original version
3. add remove() in LRUCache
4. add unit tests for LRUResourceCache and LRUCache
 2021-12-29 15:25:33 +08:00
Kseniia Sumarokova
9b63fa6949
Merge pull request #33239 from CurtizJ/async-insert-query-log 
Add asynchronous inserts to query log
 2021-12-29 09:34:08 +03:00
Anton Popov
d25560d37a fix queries without table name 2021-12-29 00:14:22 +03:00
Maksim Kita
9ef359ce2c Dictionaries fix comment 2021-12-28 23:50:48 +03:00
Maksim Kita
d1db3c9f42 Updated ShellCommandSource 2021-12-28 22:55:31 +03:00
Maksim Kita
5590cfa3aa Updated executable function integration tests 2021-12-28 22:55:30 +03:00
Maksim Kita
2ffd83ee83 Fix shell command source start 2021-12-28 22:55:30 +03:00
Maksim Kita
eb4e400c54 Executable refactoring 2021-12-28 22:55:30 +03:00
Anton Popov
a6e90a046d add more info to query log in async inserts 2021-12-28 17:53:25 +03:00
Vladimir C
bb6fc853e3
Merge pull request #31442 from vdimir/cluster-discovery 2021-12-28 10:53:30 +03:00
Anton Popov
0e2d0d70ba add asynchronous inserts to query log 2021-12-27 23:16:53 +03:00
vdimir
5f82190cd3
Log exception on cluster discovery initial update 2021-12-27 18:24:25 +03:00
vdimir
01a9199098
Move stop_flag to clusters_to_update in ClusterDiscovery 2021-12-27 16:10:36 +03:00
vdimir
5bd620fba1
Reset backoff when cluster discovery is up to date 2021-12-27 16:10:35 +03:00
vdimir
3934e99dce
update ClusterDiscovery::ConcurrentFlags::set 2021-12-27 16:09:50 +03:00
vdimir
a76e93209d
Backoff for ClusterDiscovery::runMainThread 2021-12-27 16:09:44 +03:00
vdimir
be7efb14c6
force updating cluster discovery periodically 2021-12-27 15:26:12 +03:00
vdimir
3d3fae126e
check if clusters updated successfully on current iteration of discovery 2021-12-27 15:26:12 +03:00
vdimir
36cd9ecb14
log level debug in cluster discovery 2021-12-27 15:26:11 +03:00
vdimir
72e63b4f12
restart runMainThread in cluster discovery 2021-12-27 15:26:11 +03:00
vdimir
4f8a9cc539
cluster discovery: versioning for format of data stored in zk 2021-12-27 15:26:11 +03:00
vdimir
d3b1058c40
cluster discovery configuration, support shards 2021-12-27 15:26:11 +03:00
vdimir
f47ce534f2
preserve auto clusters on config update 2021-12-27 15:26:10 +03:00
vdimir
d75a6b7abc
more logging for cluster discovery 2021-12-27 15:26:10 +03:00
vdimir
fbdb5c60bd
initial initialization of cluster discovery in main thread 2021-12-27 15:26:10 +03:00
vdimir
7a38716360
add 'secure' option to cluster discovery 2021-12-27 15:26:10 +03:00
vdimir
57cbbd9fcf
global option allow_experimental_cluster_discovery 2021-12-27 15:26:09 +03:00
vdimir
96a0371eca
add option allow_experimental_cluster_discovery 2021-12-27 15:26:09 +03:00
vdimir
ceec643f90
update config structure: remote_servers_discovery -> remote_servers.discovery 2021-12-27 15:26:09 +03:00
vdimir
7404205f37
fix cluster discovery startup race 2021-12-27 15:26:09 +03:00
vdimir
8bf3d94f04
write data for cluster discovery to zk in json 2021-12-27 15:26:09 +03:00
vdimir
76170156ef
log nodes update in cluster discovery 2021-12-27 15:26:08 +03:00
vdimir
a2d37fe2ed
use map and condvar instead of queue in cluster discovery 2021-12-27 15:26:08 +03:00
vdimir
5bbd151aae
style fix ClusterDiscovery 2021-12-27 15:26:08 +03:00
vdimir
e7d3dbeebd
cluster discovery init only with zookeeper 2021-12-27 15:26:08 +03:00
vdimir
4816d1afcc
cluster discovery: check cluster is changed, keep info 2021-12-27 15:26:07 +03:00
vdimir
aa0d79659b
check version in ClusterDiscovery::updateCluster 2021-12-27 15:26:07 +03:00
vdimir
84fb4354bc
cluster discovery worker thread 2021-12-27 15:26:06 +03:00
vdimir
94bb7cba62
cluster discovery wip 2021-12-27 15:26:05 +03:00
Vitaly Baranov
f80a3e3247
Merge pull request #30549 from aiven/kmichel-server-reload 
Start/stop servers when `listen_host`/`*_port` changes
 2021-12-27 14:51:00 +03:00
lgbo-ustc
ef1d7142f5 remove getOrTrySet 2021-12-27 16:12:39 +08:00
alexey-milovidov
0a55fa3dc2
Revert "Grouping sets dev" 2021-12-25 20:30:31 +03:00
Alexey Milovidov
29d28c531f Move code around to avoid dlsym on Musl 2021-12-24 12:25:27 +03:00
Kevin Michel
ffc1fca296
Start/stop servers when listen_host/*_port changes 
This allows starting and stopping separately each protocol server
without restarting ClickHouse.

This also allows adding or removing `listen_host` entries, which
start and stops servers for all enabled ports.

When stopping a server, the listening socket is immediately closed
(and available for another server).

Protocols with persistent connections try to wait for any currently
running query to finish before closing the connection, but idle
connection are closed quickly (depending on how often the protocol
is polled).

An extra ProfileEvent is added, `MainConfigLoads`, it is
incremented every time the configuration is reloaded. This helps
when trying to assess whether the new configuration was applied.
 2021-12-24 08:26:02 +01:00
mergify[bot]
d4ec469ed1
Merge branch 'master' into query-kind-concurent_restriction 2021-12-24 07:02:00 +00:00
Alexey Milovidov
26e10c1fbb Remove unused header 2021-12-24 04:02:26 +03:00
Raúl Marín
cb22091b33 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-23 13:59:33 +01:00
Alexey Milovidov
d98675b49c Whitespace 2021-12-23 11:50:40 +03:00
Vitaly Baranov
b9090029e6
Merge pull request #32662 from vitlibar/improve-exceptions-usage-in-access-control 
Improve exceptions usage in access control
 2021-12-23 06:48:28 +03:00
Kseniia Sumarokova
764bb1181f
Merge pull request #33017 from stigsb/table-overrides-drop-add-column-support 
Remove the possibility of adding columns with table overrides
 2021-12-22 22:30:21 +03:00
Raúl Marín
1d9916eb25 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-22 16:42:10 +01:00
tavplubix
d8fb293d57
Merge pull request #32900 from zzsmdfj/issue/#15182_MaterializeMySQL_support_bit_type 
Add support for BIT data type in MaterializedMySQL
 2021-12-22 15:58:29 +03:00
Stig Bakken
1cc70ad524 Remove the possibility of adding columns with table overrides 
(except ALIAS columns)

Supporting this properly requires more comprehensive changes.
 2021-12-22 08:20:14 +01:00
Kseniia Sumarokova
5ae2f0028f
Merge pull request #32961 from kssenii/fix-positional-args 
Fix for positional args
 2021-12-22 09:41:55 +03:00
kssenii
de70a57700 Better 2021-12-21 21:48:25 +03:00
Raúl Marín
eddccb0065 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-21 10:00:17 +01:00
kssenii
30996b2448 Fix 2021-12-21 01:19:51 +03:00
kssenii
175ad8a989 Fix 2021-12-21 00:20:08 +03:00
Vitaly Baranov
6721060649 Rename function IAccessStorage::login() -> IAccessStorage::authenticate(). 
Remove functions IAccessStorage::hasSubscriptionImpl() and IAccessStorage::existsImpl().
 2021-12-20 21:26:27 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment 
Loops remove postfix increment
 2021-12-20 19:51:07 +03:00
Maksim Kita
ebff389701
Merge pull request #32972 from kitaisreal/containers-iteration-fix-erase 
Containers iteration fix erase
 2021-12-20 16:47:19 +03:00
Maksim Kita
51477adf1b Updated additional cases 2021-12-20 15:55:07 +03:00
Nikolai Kochetov
2e62f086a1
Merge pull request #32751 from ClickHouse/fix-32668 
Apply some more optimizations to NO_QUERY ast.
 2021-12-20 15:47:25 +03:00
Maksim Kita
3feab5a975 Containers iteration fix erase 2021-12-20 13:42:31 +03:00
kssenii
965e2bfb21 Fix 2021-12-20 08:43:07 +03:00
kssenii
e6ebb55c4e Move to Interpreters 2021-12-17 22:19:10 +03:00
kssenii
dd902b6875 Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-17 22:03:58 +03:00
alexey-milovidov
6b97af4c63
Merge pull request #26869 from taylor12805/grouping-sets-dev 
Grouping sets dev
 2021-12-17 20:50:15 +03:00
Nikolai Kochetov
ffdcf52a9e
Merge pull request #32858 from ClickHouse/fix-has-column-in-table-subquery-2 
Always apply const-condition-if optimization.
 2021-12-17 20:39:30 +03:00