thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
118,431 Commits 9,328 Branches 1,783 Tags 2.1 GiB
ca6930eb11
Commit Graph

10166 Commits

Author SHA1 Message Date
serxa
35e77f8e2a fix comment 2023-05-19 17:53:22 +00:00
mateng915
5237dd0245
New system table zookeeper connection (#45245) 
* Feature: Support new system table to show which zookeeper node be connected

Description:
============
Currently we have no place to check which zk node be connected otherwise using
lsof command. It not convenient

Solution:
=========
Implemented a new system table, system.zookeeper_host when CK Server has zk
this table will show the zk node dir which connected by current CK server

Noted: This table can support multi-zookeeper cluster scenario.

* fixed review comments

* added test case

* update test cases

* remove unused code

* fixed review comments and removed unused code

* updated test cases for print host, port and is_expired

* modify the code comments

* fixed CI Failed

* fixed code style check failure

* updated test cases by added Tags

* update test reference

* update test cases

* added system.zookeeper_connection doc

* Update docs/en/operations/system-tables/zookeeper_connection.md

* Update docs/en/operations/system-tables/zookeeper_connection.md

* Update docs/en/operations/system-tables/zookeeper_connection.md

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-05-19 17:06:43 +03:00
Sergei Trifonov
5db5f6e44b
Merge branch 'master' into fix-throttlers 2023-05-19 14:08:36 +02:00
Alexey Milovidov
f5506210d6 Geo types are production ready 2023-05-19 12:43:55 +02:00
alesapin
e5b001abda
Merge branch 'master' into fix_some_tests4 2023-05-19 12:34:03 +02:00
kssenii
d6df009842 Fix 2023-05-19 12:22:46 +02:00
Azat Khuzhin
422cbe08fe Do not use PackedHashMap for non-POD for the purposes of layout 
In clang-16 the behaviour for POD types had been changed in [1], this
does not allows us to use PackedHashMap for some types.

  [1]: 277123376c

Note, that I tried to come up with a more generic solution then
enumeratic types, but failed. Though now I think that this is good,
since this shows which types are not allowed for PackedHashMap

Another option is to use -fclang-abi-compat=13.0 but I doubt it is a
good idea.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-19 06:07:21 +02:00
Azat Khuzhin
b44497fd4c Introduce PackedHashMap (HashMap with structure without padding) 
In case of you have HashMap with <UInt64, UInt16> as <Key, Value> the
overhead of 38% can be crutial, especially if you have tons of keys.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-19 06:07:21 +02:00
Kseniia Sumarokova
adebac1a92
Merge branch 'master' into fix-assertion-in-do-cleanup 2023-05-18 12:22:02 +02:00
Antonio Andelic
bf6afd27a6 build with disks 2023-05-18 07:39:22 +00:00
SmitaRKulkarni
a91c793684
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-05-18 09:24:25 +02:00
Kseniia Sumarokova
855c95f626
Update src/Interpreters/Cache/Metadata.cpp 
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
 2023-05-17 22:46:09 +02:00
kssenii
f2dbcb5146 Better fix 2023-05-17 16:27:06 +02:00
Nikita Taranov
0dd67bacf2
Merge branch 'master' into optimize_reading2 2023-05-17 15:06:41 +02:00
Kseniia Sumarokova
edceda494d
Merge branch 'master' into add-more-logging-for-cache 2023-05-17 12:24:59 +02:00
Kseniia Sumarokova
3787b7f127
Update Metadata.cpp 2023-05-17 12:16:18 +02:00
Azat Khuzhin
3c80e30f02 Fix per-query IO/BACKUPs throttling settings (when default profile has them) 
When some of this settings was set for default profile (in
users.xml/users.yml), then it will be always used regardless of what
user passed.

Fix this by not inherit per-query throttlers, for this they should be
reset before making query context and they should not be initialized as
before in Context::makeQueryContext(), since makeQueryContext() called
too early, when user settings was not read yet.

But there we had also initialization of per-server throttling, move this
into the ContextSharedPart::configureServerWideThrottling(), and call it
once we have ServerSettings set.

Also note, that this patch makes the following settings - server
settings:
- max_replicated_fetches_network_bandwidth_for_server
- max_replicated_sends_network_bandwidth_for_server
But this change should not affect anybody, since it is done with
compatiblity (i.e. if this setting is set in users profile it will be
read from it as well as a fallback).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-17 12:12:39 +02:00
kssenii
724949927b Add logging 2023-05-16 17:36:48 +02:00
vdimir
1f55c320b4 Fix style 2023-05-16 16:23:53 +02:00
vdimir
ca005ecea1 Update comment about filtering nulls in asof join 2023-05-16 16:23:53 +02:00
vdimir
a7bb8f412f Allow ASOF JOIN over nullable right column 2023-05-16 16:23:53 +02:00
Azat Khuzhin
68138395eb Fix parameterized views when query parameter used multiple times in the query 
Example:

    CREATE VIEW view AS
    SELECT *
    FROM system.one
    WHERE dummy = {k1:Int}+1 OR dummy = {k1:Int}+2
                   ^^                    ^^

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-16 15:13:21 +02:00
Alexander Tokmakov
b6716a8f0f Merge branch 'master' into fix_some_tests4 2023-05-16 14:46:27 +02:00
kssenii
d4ea3ea045 Fix 2023-05-16 13:54:13 +02:00
Sergei Trifonov
cbc15bf35a
Add DynamicResourceManager and FairPolicy into scheduling subsystem (#49671) 
* Add `DynamicResourceManager` and `FairPolicy` into scheduling subsystem

* fix test

* fix tidy build
 2023-05-15 23:13:17 +02:00
Alexander Tokmakov
05ae7b2c2d fix some tests 2023-05-15 18:28:12 +02:00
Igor Nikonov
4d168400ae Merge remote-tracking branch 'origin/master' into fill_with_by_sorting_prefix_2 2023-05-15 14:40:46 +00:00
Kruglov Pavel
558eda4146
Merge pull request #49412 from azat/block-use-dense-hash-map 
Switch Block::NameMap to google::dense_hash_map over HashMap
 2023-05-15 12:22:55 +02:00
Igor Nikonov
e743693fb3 Merge remote-tracking branch 'origin/master' into fill_with_by_sorting_prefix_2 2023-05-14 15:56:12 +00:00
Igor Nikonov
93384d6546 Fix suffix generation (see added test in commit 3355894) 2023-05-14 15:52:49 +00:00
Igor Nikonov
629a1a6668 Minor cleanup 2023-05-13 22:11:23 +00:00
Alexey Milovidov
5a44dc26e7 Fixes for clang-17 2023-05-13 02:57:31 +02:00
Robert Schulze
b9c185af44
Merge pull request #49678 from azat/build/llvm-16 
Switch to LLVM/clang 16 (16.0.3)
 2023-05-12 13:47:36 +02:00
Nikita Taranov
536c5ac261
Merge branch 'master' into optimize_reading2 2023-05-12 12:59:32 +02:00
Han Fei
07341a04fd
Merge pull request #49666 from hanfei1991/hanfei/regexp-dict-read 
Support `dictionary` table function for `RegExpTreeDictionary`
 2023-05-12 11:40:46 +02:00
Azat Khuzhin
2c40dd6a4c Switch Block::NameMap to google::dense_hash_map over HashMap 
Since HashMap creates 2^8 elements by default, while dense_hash_map
should be good here.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-12 05:52:57 +02:00
Azat Khuzhin
e5c4eb35e8 Merge remote-tracking branch 'u/master' into build/llvm-16 
* u/master:
  enable used flags's reinit only when the hash talbe rehash
  Fix build of libfiu on clang-16
  fix flaky test 02504_regexp_dictionary_ua_parser
  fix convertation
  Fix test that expected CH to apply a wrong optimization
  ActionsDAG: remove wrong optimization
 2023-05-11 16:32:25 +02:00
SmitaRKulkarni
9bf6175919
Merge pull request #47584 from aiven-sal/aiven-sal/having_pr 
ActionsDAG: fix wrong optimization
 2023-05-11 14:03:46 +02:00
Han Fei
6b3160f7ce trivial change 2023-05-11 13:49:02 +02:00
lgbo-ustc
a07359fbe8 enable used flags's reinit only when the hash talbe rehash 2023-05-11 11:06:13 +08:00
Igor Nikonov
72f4aed64b Add setting use_with_fill_by_sorting_prefix 2023-05-10 21:08:29 +00:00
Han Fei
2273884ded address comments 2023-05-10 17:42:08 +02:00
Nikita Taranov
ec63a6ee7c
Merge branch 'master' into optimize_reading2 2023-05-10 17:24:37 +02:00
Igor Nikonov
49971276d3
Merge branch 'master' into fill_with_by_sorting_prefix_2 2023-05-10 16:29:28 +02:00
Azat Khuzhin
a0cc5b7e47 Merge remote-tracking branch 'u/master' into build/llvm-16 2023-05-10 16:28:36 +02:00
Han Fei
d17aa828b3
Merge pull request #49361 from hanfei1991/hanfei/fiu 
support named fault injection and add a simple test
 2023-05-10 15:51:51 +02:00
Igor Nikonov
a0ef2e272f Fix suffix generation for empty result set 2023-05-10 12:41:33 +00:00
Kseniia Sumarokova
1ea53b5885
Merge pull request #49690 from kssenii/add-logging-for-try-reserve 
Add some logging
 2023-05-10 12:59:56 +02:00
Igor Nikonov
37bc0becc9 Merge remote-tracking branch 'origin/master' into fill_with_by_sorting_prefix_2 2023-05-10 10:53:10 +00:00
SmitaRKulkarni
5c030c428c
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-05-10 09:51:38 +02:00
robot-clickhouse-ci-2
c698d2af99
Merge pull request #49702 from ClickHouse/nickitat-patch-11 
Enable `do_not_evict_index_and_mark_files` by default
 2023-05-10 02:26:38 +02:00
Anton Popov
862008da0d
Merge pull request #49663 from CurtizJ/fix-race-context-copy 
Fix race in `Context::createCopy`
 2023-05-10 02:11:35 +02:00
kssenii
e35c669892 Fix 2023-05-09 22:51:47 +02:00
Yakov Olkhovskiy
815cc8de9b removed unnecessary check for thread's joinability 2023-05-09 19:43:39 +00:00
Azat Khuzhin
d03ae2abfa Fix modernize-loop-convert clang-tidy warning 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-09 21:19:37 +02:00
kssenii
c20aa0786d More logging 2023-05-09 19:29:57 +02:00
kssenii
13b77f1dac Fix 2023-05-09 19:13:39 +02:00
Anton Popov
d88224dc43 fix build 2023-05-09 16:30:43 +00:00
Nikita Taranov
7cd6db3fb3
Enable do_not_evict_index_and_mark_files by default 2023-05-09 18:04:47 +02:00
kssenii
75aa775d2b Add logging 2023-05-09 15:58:20 +02:00
Kseniia Sumarokova
abde991cc4
Merge pull request #49675 from ClickHouse/cache-cleanup-after-locking-rework 
Some cache cleanup after rework locking
 2023-05-09 11:29:45 +02:00
Alexey Milovidov
3556c26075
Merge pull request #49434 from ClickHouse/ins 
Make INSERT do more things in parallel to avoid getting bottlenecked on one thread
 2023-05-09 08:10:15 +03:00
Alexey Milovidov
a8bdb20fc4
Merge pull request #48787 from ClickHouse/background-memory-tracker 
Add MemoryTracker for the background tasks [Resubmit]
 2023-05-09 07:58:36 +03:00
Alexey Milovidov
dda6e9175c
Merge pull request #49647 from ClickHouse/optimize-opentelemetry-span-log 
Optimize `opentelemetry_span_log`
 2023-05-09 06:40:49 +03:00
Han Fei
64b67b9a37 Merge branch 'master' into hanfei/fiu 2023-05-09 01:14:30 +02:00
Yakov Olkhovskiy
70516f2f45 close pipe in run() if exception happens 2023-05-08 23:00:20 +00:00
Han Fei
82ce47ce56 address comments 2023-05-08 23:02:36 +02:00
Igor Nikonov
903021e896 Fix review comment 2023-05-08 18:29:47 +00:00
Igor Nikonov
e7fff74797 Cleanup 2023-05-08 17:45:42 +00:00
Han Fei
71c96ceb61
Update src/Interpreters/ExternalDictionariesLoader.h 2023-05-08 16:21:42 +02:00
Han Fei
72fc567d4a
Merge branch 'master' into hanfei/regexp-dict-read 2023-05-08 16:20:12 +02:00
Han Fei
92e57817a2 Support dictionary table function for RegExpTreeDictionary 2023-05-08 16:14:08 +02:00
Anton Popov
713ffb3bda fix race in Context::createCopy 2023-05-08 13:22:31 +00:00
Yakov Olkhovskiy
1e3cd4ed6d close pipe before join 2023-05-08 12:39:01 +00:00
Kseniia Sumarokova
336bb41c5d
Merge branch 'master' into remove-dependency-from-context 2023-05-08 12:46:10 +02:00
Alexey Milovidov
baf162b26b Optimize opentelemetry_span_log 2023-05-08 05:15:28 +02:00
Alexey Milovidov
647472ed46
Merge branch 'master' into aiven-sal/having_pr 2023-05-08 05:53:38 +03:00
Alexey Milovidov
e8228cb0d2
Merge pull request #49530 from ClickHouse/optimize-query-log 
Optimize the system.query_log and system.query_thread_log tables by applying LowCardinality when appropriate
 2023-05-07 03:36:30 +03:00
robot-clickhouse-ci-1
d1ad3ea24e
Merge pull request #49593 from ClickHouse/rs/qc-empty-chunk 
Query Cache: Safeguard against empty chunks
 2023-05-07 02:20:54 +02:00
Igor Nikonov
6e9c519195 Merge remote-tracking branch 'origin/master' into fill_with_by_sorting_prefix_2 2023-05-06 19:33:26 +00:00
kssenii
2228107134 Better name 2023-05-06 20:12:26 +02:00
Sema Checherinda
2104baabce
Merge pull request #49527 from ClickHouse/thread-detach-fix 
Fix thread MemoryTracker parent after detach
 2023-05-06 19:11:55 +02:00
Alexey Milovidov
54163b87ec
Merge pull request #47977 from ClickHouse/rs/non-experimental-qc 
Mark query cache as non-experimental
 2023-05-06 19:49:23 +03:00
Alexey Milovidov
62d8677773
Update QueryCache.cpp 2023-05-06 18:25:43 +03:00
Alexey Milovidov
2d39acf2e1
Merge pull request #49554 from kssenii/fix-assertions-3 
Fix stress test (assertion 'key_metadata.lock()')
 2023-05-06 18:16:40 +03:00
Robert Schulze
883e0abea1
Query Cache: Safeguard against empty chunks 
Reading from the query cache is implemented using processor
"SourceFromChunks" which inherits from ISource. The latter has logic
which finishes processing (= calls .finish() on the output port +
returns Status::Finished) when the derived class returns an empty chunk.
If this empty chunk is not the last chunk, i.e. if it is followed by
non-empty chunks, the query result will be incorrect. This situation
should theoretically never occur in practice but who knows ... To be on
the safe side, writing into the query cache now rejects empty chunks and
thereby avoids this scenario.
 2023-05-06 14:22:23 +00:00
Robert Schulze
0598bfd79e
Mark query cache as non-experimental 2023-05-06 13:58:40 +00:00
Robert Schulze
983e37048c
Merge pull request #48853 from ClickHouse/rs/qc-totals-extremes2 
QueryCache: Support queries with totals and extremes
 2023-05-06 15:41:10 +02:00
Robert Schulze
7137c8811a
Merge pull request #49158 from ClickHouse/rs/show-indexes 
Implement SHOW INDEX
 2023-05-06 15:30:29 +02:00
Robert Schulze
39bd320e37
Merge pull request #49573 from ClickHouse/rs/qc-no-internal-queries 
Query cache: disable for internal queries
 2023-05-06 15:27:44 +02:00
Han Fei
d2bd0bcd59
Merge branch 'master' into patch-1 2023-05-05 23:39:08 +02:00
Robert Schulze
3dfc0bd265
Merge pull request #49413 from azat/build/headers 
Slightly reduce inter-header dependencies
 2023-05-05 23:37:58 +02:00
Alexey Milovidov
a8c410c21a Merge branch 'master' into optimize-query-log 2023-05-05 23:21:01 +02:00
Nikolai Kochetov
b2c36fc3e5 Fixing style. 2023-05-05 20:51:38 +00:00
Robert Schulze
9cd81400f3
Query cache: disable for internal queries 2023-05-05 19:22:14 +00:00
Nikolai Kochetov
618486815b Merge branch 'master' into refactor-subqueries-for-in 2023-05-05 20:39:09 +02:00
Nikolai Kochetov
54ef6769f1 Refactor PreparedSets [4] 2023-05-05 18:30:08 +00:00
Robert Schulze
bba85c9c25
Improve docs 2023-05-05 18:11:33 +00:00
Robert Schulze
945a5130cf
Simplify simplify simplify 2023-05-05 16:28:51 +00:00
Robert Schulze
aba5db40a1
Merge remote-tracking branch 'origin/master' into rs/qc-totals-extremes2 2023-05-05 14:11:10 +00:00
Robert Schulze
f7e31a9664
More protection against SQL injection 2023-05-05 13:58:08 +00:00
Nikolay Degterinsky
7a727bbbe3
Merge pull request #49520 from IvanTakarlikov-st/it-grammar-fixes 
Fix some grammar mistakes in documentation, code and tests
 2023-05-05 15:45:58 +02:00
kssenii
6bc302959b Better 2023-05-05 14:55:52 +02:00
Han Fei
9c19bd8179
Merge branch 'master' into patch-1 2023-05-05 14:41:40 +02:00
kssenii
1050754432 Fix 2023-05-05 14:41:35 +02:00
wangxiaobo
f704c0dfec
Implement SYSTEM DROP REPLICA from auxillary ZooKeeper clusters (#48932) 
* multiple zookeeper drop replica bug fix.

* add an integration test.

* format code.

* set stay_alive=True for test.

* style check bug fix.

* add check for is_active

* format code

* remove table_settings and has_metadata_out param.
 2023-05-05 14:11:11 +03:00
Bulat Gaifullin
8556493c9e
Update LRUFileCachePriority.cpp 2023-05-05 12:00:24 +03:00
Bulat Gaifullin
a222c4c5a1
Update LRUFileCachePriority.h 2023-05-05 11:59:59 +03:00
Bulat Gaifullin
82e6ce22fa
Update IFileCachePriority.h 
remove extra semicolumn
 2023-05-05 11:59:37 +03:00
Alexey Milovidov
dfbba48510
Merge pull request #49282 from loneylee/49280 
Fix all key value is null and group use rollup return wrong answer
 2023-05-05 06:43:41 +03:00
Michael Kolupaev
0fa311ee38 Make INSERT do more things in parallel to avoid getting bottlenecked on one thread 2023-05-05 00:49:27 +00:00
Alexey Milovidov
c5184ed329 Optimize the system.query_log and system.query_thread_log tables by applying LowCardinality when appropriate 2023-05-05 01:36:01 +02:00
Dmitry Novik
4ba7530c23 Fix thread MemoryTracker parent after detach 2023-05-05 00:21:13 +02:00
Kseniia Sumarokova
3cbe9be22d
Merge pull request #49491 from kssenii/fix-assertions-2 
Fix logical error in stress test, add some logging
 2023-05-05 00:00:57 +02:00
Nikolai Kochetov
f598a39ea2 Refactor PreparedSets [3] 2023-05-04 17:54:08 +00:00
Ivan Takarlikov
8873856ce5 Fix some grammar mistakes in documentation, code and tests 2023-05-04 13:35:18 -03:00
Dmitry Novik
befcdbcc6d
Merge pull request #47630 from ClickHouse/revert-47476-revert_46622 
Fix user MemoryTracker counter in async inserts
 2023-05-04 16:57:41 +02:00
Yakov Olkhovskiy
3aaea2b91d join thread even exception happened 2023-05-04 13:07:28 +00:00
Igor Nikonov
0f9271a349 Passing sort description apart of fill description 2023-05-04 12:10:06 +00:00
kssenii
1433f5ffc9 Merge remote-tracking branch 'upstream/master' into remove-dependency-from-context 2023-05-04 13:24:02 +02:00
kssenii
cf3a8e9b93 One more fix 2023-05-04 12:49:23 +02:00
kssenii
663a007fa6 Add assertion 2023-05-04 12:11:44 +02:00
alesapin
412b161104
Merge pull request #48791 from kssenii/better-local-object-storage 
Make local object storage work consistently with s3 object storage, fix problem with append, make it configurable as independent storage
 2023-05-04 11:47:43 +02:00
kssenii
7ad4330c14 Fix logical error in stress test 2023-05-04 11:37:19 +02:00
alesapin
89caf40840
Merge pull request #46528 from myrrc/feature/show-secrets-option 
Show secrets in `SHOW` and `SELECT` queries
 2023-05-04 11:21:30 +02:00
SmitaRKulkarni
a28da97e87
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-05-04 10:14:11 +02:00
Li Shuai
279970337a Fix all key value is null and group use rollup return wrong answer 2023-05-04 11:07:17 +08:00
Alexander Gololobov
a90c2ec90d
Merge pull request #49425 from ClickHouse/fix_build_set_with_remote 
Fix fuzz bug when subquery set is not built when reading from remote()
 2023-05-03 21:47:29 +02:00
Alexander Tokmakov
e399903030
Merge pull request #48548 from ClickHouse/clusters_is_active_column 
Add some columns to system.clusters
 2023-05-03 17:42:40 +03:00
Nikolay Degterinsky
048b4ec738
Merge branch 'master' into bcrypt 2023-05-03 16:14:57 +02:00
Anton Popov
7896d30737
Merge pull request #49122 from CurtizJ/add-async-insert-mt-setting 
Add `MergeTree` setting `async_insert`
 2023-05-03 16:01:13 +02:00
Kseniia Sumarokova
59401ce93c
Merge pull request #49411 from ClickHouse/kssenii-patch-7 
Update FileSegment.cpp
 2023-05-03 11:50:13 +02:00
Nikolay Degterinsky
4c038ef538
Merge branch 'master' into bcrypt 2023-05-02 21:52:21 +02:00
kssenii
ecfbf1e304 Remove dependency from DB::Context in readers 2023-05-02 21:45:27 +02:00
kssenii
ac9ef077ad Fix 2023-05-02 21:41:17 +02:00
Alexander Gololobov
2e20f2a14d Do not skip building set even when reading from remote 2023-05-02 21:31:56 +02:00
Azat Khuzhin
2b21b83979 Reduce inter-header dependencies 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-02 21:15:18 +02:00
Dmitry Novik
bab112a1bc
Merge branch 'master' into revert-47476-revert_46622 2023-05-02 20:54:25 +02:00
Dmitry Novik
0c521eb10b Another atempt to fix test 2023-05-02 18:51:36 +00:00
Nikolay Degterinsky
46c86dce93 Fix bcrypt enum value 2023-05-02 14:36:48 +00:00
Smita Kulkarni
8205398f31 Fixed comment 2023-05-02 16:31:39 +02:00
Nikita Taranov
67bd861bc3 Merge branch 'master' into optimize_reading2 2023-05-02 14:23:56 +00:00
kssenii
22487de34a Fix 2023-05-02 16:09:33 +02:00
Kseniia Sumarokova
bccc4d3b7e
Update FileSegment.cpp 2023-05-02 15:43:03 +02:00
Robert Schulze
6bdbee33ee
Incorporate review feedback 2023-05-02 12:29:56 +00:00
Nikita Mikhaylov
aa4c5fe958
Enhancements for background merges (#49313) 2023-05-02 13:43:59 +02:00
Antonio Andelic
f695040a19
Merge pull request #48897 from ClickHouse/keeper-check-not-exists 
Add `CheckNotExists` request to Keeper
 2023-05-02 13:14:22 +02:00
Kseniia Sumarokova
3ec53152e5
Update Metadata.cpp 2023-05-02 12:02:54 +02:00
Vladimir C
96d109c3db
Merge pull request #49137 from ClickHouse/vdimir/mutliple_storage_join 2023-05-02 11:40:16 +02:00
Alexey Milovidov
1267fbca1c
Merge pull request #48998 from ongkong/fix-join-clause-create-column 
Fix incorrect createColumn call on join clause
 2023-05-02 02:23:07 +03:00
Alexey Milovidov
51847fbf10
Merge pull request #49321 from nickitat/rand_exp 
Presumably better buffer size for reading from fs
 2023-05-02 02:06:25 +03:00
alekseygolub
64d232f1aa Fix memory leak 2023-05-01 18:00:26 +00:00
Han Fei
4d18a4f8d3
Merge branch 'master' into hanfei/fiu 2023-05-01 17:51:41 +02:00
Han Fei
b701293d46 support named fault injection and add a simple test 2023-05-01 17:44:37 +02:00
Anton Popov
36d53e0d6c
Merge branch 'master' into add-async-insert-mt-setting 2023-05-01 16:13:23 +02:00
Nikita Taranov
266c9d403a review fixes 2023-05-01 13:09:55 +00:00
ongkong
1bffb28adc add comment, change check location 2023-05-01 12:14:44 +02:00
ongkong
35a736f811 use sanitizeBlock 2023-05-01 12:14:44 +02:00
Nikolay Degterinsky
f64ef88197 Merge remote-tracking branch 'upstream/master' into bcrypt 2023-04-30 21:39:16 +00:00
Alexey Milovidov
15d0379e2c
Merge pull request #48792 from ClickHouse/fix_some_tests3 
Fix some tests
 2023-04-29 22:04:11 +03:00
Nikita Taranov
0be9c0751e
Merge branch 'master' into rand_exp 2023-04-29 15:22:08 +02:00
Smita Kulkarni
0726db7d9a Renamed parameter to max_parsing_threads 2023-04-29 10:34:59 +02:00
Nikita Taranov
af558e282c impl 2023-04-28 20:48:37 +00:00
Kseniia Sumarokova
d4aa96e262
Merge branch 'master' into better-local-object-storage 2023-04-28 21:05:05 +02:00
Mike Kot
2d20658885
Merge branch 'master' into feature/show-secrets-option 2023-04-28 20:19:35 +03:00
Kseniia Sumarokova
a04025623c
Update FileCache.cpp 2023-04-28 16:47:15 +02:00
Kseniia Sumarokova
b2aa2aa244
Fix 2023-04-28 16:45:00 +02:00
Alexey Milovidov
911d4e9977
Merge branch 'master' into fix_some_tests3 2023-04-28 17:23:46 +03:00
Nikita Taranov
c862eca0f7 fix build 2023-04-28 14:08:46 +00:00
Mike Kot
f1d0ed367d separate name for format setting 2023-04-28 13:39:38 +00:00
Mike Kot
7bffa89e96 add all 3 settings 2023-04-28 13:22:36 +00:00
Alexey Milovidov
6dcfdbfaf2
Merge branch 'master' into clusters_is_active_column 2023-04-28 16:16:25 +03:00
Nikita Taranov
71a092d201 Merge branch 'master' into optimize_reading2 2023-04-28 12:29:27 +00:00
kssenii
298ed454ce Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-28 13:26:55 +02:00
Mike Kot
e428af0b63 replace format settings with server setting 2023-04-27 21:27:18 +00:00
Mike Kot
b24de29c70 reverting mr changes 2023-04-27 20:37:12 +00:00
Mike Kot
357cde95e9 Merge branch 'master' into feature/show-secrets-option 2023-04-27 20:19:40 +00:00
kssenii
f7404d7e72 Fix 2023-04-27 21:40:55 +02:00
Nikita Taranov
de03b90570 impl 2023-04-27 17:07:00 +00:00
Nikolay Degterinsky
df1e56b985 Merge remote-tracking branch 'upstream/master' into bcrypt 2023-04-27 17:01:58 +00:00
Kseniia Sumarokova
45e2d296f9
Merge branch 'master' into better-local-object-storage 2023-04-27 14:54:04 +02:00
Alexey Milovidov
3fa19c5ee2
Merge branch 'master' into fix_some_tests3 2023-04-27 15:04:15 +03:00
kssenii
641b64c4e8 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-27 13:31:41 +02:00
Kseniia Sumarokova
c72bd3eeef
Merge pull request #49260 from ClickHouse/kssenii-patch-7 
Add a comment in FileCache.cpp
 2023-04-27 13:29:19 +02:00
Nikolay Degterinsky
ea55222131
Merge pull request #44674 from evillique/add_default_password_type 
Add default password type
 2023-04-27 13:25:58 +02:00
kssenii
fab84345fd Corrections after merge with master 2023-04-27 13:18:32 +02:00
kssenii
3d4413183f Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-27 13:16:36 +02:00
Kseniia Sumarokova
9325787884
Update FileCache.cpp 2023-04-27 13:03:41 +02:00
Kseniia Sumarokova
28dee371e0
Merge pull request #49203 from kssenii/fix-system-error-from-cache 
Catch exceptions from create_directories in fs cache
 2023-04-27 12:29:38 +02:00
Nikolay Degterinsky
527b136ddf
Merge pull request #48806 from ucasfl/in-operator 
IN operator support compare Date and Date32
 2023-04-27 01:17:05 +02:00
Kseniia Sumarokova
5a01673118
Merge branch 'master' into fix-after-49110 2023-04-26 22:19:13 +02:00
kssenii
fd432a3ee4 Fix 2023-04-26 17:54:10 +02:00
kssenii
8ba9ab67c2 Fix 2023-04-26 17:13:15 +02:00
kssenii
3939498ef7 Review fixes 2023-04-26 15:43:09 +02:00
Alexey Milovidov
d151e03b04
Merge branch 'master' into rs/show-indexes 2023-04-26 13:55:35 +03:00
Nikolay Degterinsky
a8d124ac0f
Merge branch 'master' into add_default_password_type 2023-04-26 12:53:23 +02:00
Alexey Milovidov
815767a064
Merge branch 'master' into vdimir/mutliple_storage_join 2023-04-26 13:44:22 +03:00
Alexey Milovidov
2a42bcd4ae
Merge pull request #49120 from CurtizJ/add-field-async-insert-log 
Add field with number of rows to `system.asynchronous_insert_log`
 2023-04-26 13:43:09 +03:00
Alexey Milovidov
4dceee0dc3
Merge branch 'master' into rs/qc-totals-extremes2 2023-04-26 13:40:12 +03:00
Robert Schulze
c58e0e347c
Various cleanups 2023-04-26 09:46:35 +00:00
Aleksei Golub
26812f36fb Added read-only database setting; Fixed error messages for filesystem database; added tests 2023-04-26 10:42:20 +03:00
Robert Schulze
403396d0bd
Fix style 2023-04-25 21:04:55 +00:00
Robert Schulze
c4ab1e12f1
Initial version of SHOW INDEXES 2023-04-25 20:48:11 +00:00
Nikolai Kochetov
80a2f30a0c Refactor PreparedSets [2] 2023-04-25 18:14:08 +00:00
kssenii
cbf0b98102 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-25 15:15:22 +02:00
vdimir
7731ea7905
Fix key not found error for queries with multiple StorageJoin 2023-04-25 11:28:54 +00:00
Kseniia Sumarokova
81a53ab5f8
Merge branch 'master' into fix-assertions-from-cache 2023-04-25 11:42:15 +02:00
Anton Popov
cc5acfbe62 add profile event 2023-04-25 01:37:37 +00:00
Anton Popov
c7db58be66 add MergeTree setting async_insert 2023-04-25 01:11:58 +00:00
Anton Popov
aa81937eee add field with number of rows to async insert log 2023-04-25 00:31:55 +00:00
Robert Schulze
b565257a39
Support totals and extremes in query cache 2023-04-24 22:23:27 +00:00
Alexey Milovidov
dee0b78303
Merge pull request #48745 from Algunenano/arenita 
Don't count unused arena bytes as read_bytes in AggregateFunction types
 2023-04-25 00:53:11 +03:00
Nikolai Kochetov
65d23af611 Refactor PreparedSets [1] 2023-04-24 19:35:49 +00:00
Nikolay Degterinsky
2fe3a69ba1 Merge remote-tracking branch 'upstream/master' into add_default_password_type 2023-04-24 18:51:32 +00:00
Smita Kulkarni
b70878aa0e Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-04-24 19:39:36 +02:00
Nikolay Degterinsky
c90d5ffed8 Fix style 2023-04-24 15:27:24 +00:00
Nikolay Degterinsky
12ad5b710a Separate AuthenticationData and AuthenticationType, small fixes 2023-04-24 15:12:45 +00:00
pufit
1985fb785e
Merge pull request #48262 from ClickHouse/42497-inherit-grant 
GRANT CURRENT GRANTS implementation
 2023-04-24 11:11:10 -04:00
kssenii
4048756bdb Fix 2023-04-24 16:50:07 +02:00
Alexander Tokmakov
19f43ee7c7 Merge branch 'master' into fix_some_tests3 2023-04-24 16:13:27 +02:00
Alexander Tokmakov
332a736a59 Merge branch 'master' into clusters_is_active_column 2023-04-24 15:54:03 +02:00
Robert Schulze
ded8eca041
Merge pull request #48284 from ClickHouse/rs/qc-quota 
Query Cache: Allow per-user quotas
 2023-04-24 11:32:32 +02:00
Raúl Marín
f0e045bb3d Merge remote-tracking branch 'blessed/master' into arenita 2023-04-24 10:42:56 +02:00
Alexey Milovidov
76230be7c8
Merge pull request #49061 from ClickHouse/parameter-exception-context 
Add the context to the exception message about query paramters
 2023-04-23 21:25:01 +03:00
Robert Schulze
38d5f12a32
Query Cache: Implement per-user quotas 2023-04-23 16:13:51 +00:00
Robert Schulze
7fe0562de5
Make the cfg parameter naming more consistent 2023-04-23 14:00:22 +00:00
Alexey Milovidov
e2bb8467d5
Merge pull request #48961 from evillique/default_shard_replica 
Use default `{replica}`, `{shard}` arguments in Replicated engine
 2023-04-23 16:09:57 +03:00
Alexey Milovidov
4f6e8b0b3c
Merge pull request #48999 from liuneng1994/fix_aggregate_empty_string_key 
Fix aggregate empty string error
 2023-04-23 16:08:44 +03:00
kssenii
d8023806a9 Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-23 12:39:34 +02:00
kssenii
ef4c4e0192 Fix incorrectly resolved conflicts after merge with master 2023-04-23 12:37:04 +02:00
Kseniia Sumarokova
be0cb808de
Merge pull request #48973 from kssenii/randomize-fs-read-settings 
Randomize prefetch settings in ci
 2023-04-23 12:04:30 +02:00
Alexey Milovidov
2ae9dad9b8 Add the context to the exception message about query paramters 2023-04-23 09:34:12 +02:00
kssenii
1fbaaf0445 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-22 17:14:50 +02:00
Alexey Milovidov
48c2884ef0 Update Aggregator.h 2023-04-22 11:55:32 +08:00
liuneng
eea363a8c3 fix aggregate empty string error 2023-04-22 11:55:32 +08:00
Nikolay Degterinsky
61f217be1d Better solution, add test 2023-04-22 01:55:18 +00:00
Nikolay Degterinsky
b599d0bd01 Better 2023-04-21 22:03:56 +00:00
Dmitry Novik
e23105b322 Merge remote-tracking branch 'origin/master' into revert-47476-revert_46622 2023-04-21 16:40:04 +00:00
Alexey Milovidov
bd64a4a6fc
Merge pull request #48904 from ClickHouse/fix_set_withzk_and_analyzer 
Fix segfault when set is not built yet
 2023-04-21 15:14:23 +03:00
Kruglov Pavel
032b39e50b
Merge pull request #48976 from Avogar/fix-unknown-identifier-row-policy 
Fix UNKNOWN_IDENTIFIER error while select from table with row policy and column with dots
 2023-04-21 13:28:46 +02:00
kssenii
e58c16318f Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-21 12:49:40 +02:00
pufit
544972bcda Solve PR issues 2023-04-20 18:31:35 -04:00
kssenii
0a34f3bcea Merge remote-tracking branch 'upstream/master' into randomize-fs-read-settings 2023-04-20 19:04:01 +02:00
kssenii
6552ec6afc Small improvement 2023-04-20 18:15:09 +02:00
avogar
3b72a6821c Fix UNKNOWN_IDENTIFIER error while select from table with row policy and column with dots 2023-04-20 12:28:39 +00:00
Vladimir C
99ac321f5d
Merge pull request #48664 from ClickHouse/vdimir/http_tmp_buffer_fs_cache 2023-04-20 14:15:30 +02:00
Antonio Andelic
09156ab2b7 Merge branch 'master' into keeper-check-not-exists 2023-04-20 12:07:16 +00:00
Han Fei
6805edf9e6
Merge pull request #48850 from sichenzhao/add_alias_for_parts_name 
add aliases for parts name and names
 2023-04-20 13:04:11 +02:00
Nikolay Degterinsky
75e283323f Use default parameters in Replicated engine 2023-04-19 20:03:02 +00:00
kssenii
fef179a9ea Remove redundant code 2023-04-19 19:23:50 +02:00
kssenii
69de3a3c2d Fix tsan in test test_temporary_data_in_cache 2023-04-19 17:48:56 +02:00
Vladimir C
b31f696aae Fix iteration in Context::getClusters 2023-04-19 16:44:54 +02:00
vdimir
b1f0e0d62f Lock clusters_mutex for cluster_discovery 2023-04-19 16:44:54 +02:00
vdimir
d6416a1321 Store clusters from ClusterDiscovery in separate map 2023-04-19 16:44:54 +02:00
Vladimir C
61635eb2d3 Update TemporaryDataOnDisk::createRawStream 2023-04-19 16:44:21 +02:00
vdimir
92d0d9d4ff Http temporary buffer integration with fs cache 2023-04-19 16:44:21 +02:00
Antonio Andelic
3f00d46785
Update enum for ZooKeeperLog 2023-04-19 14:07:38 +02:00
Alexander Gololobov
2a829d9838 Check getData() result in other places 2023-04-18 17:11:19 +02:00
kssenii
67d695580a Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-18 12:42:25 +02:00
kssenii
c34f277fe7 Fix conflicts after merge 2023-04-18 00:00:38 +02:00
Alexander Gololobov
ba5ca15c40
Merge pull request #46835 from ClickHouse/reduce_mem_in_mutation_with_subquery 
Reduce memory consumption by mutations with big subqueries used with IN
 2023-04-17 23:55:08 +02:00
kssenii
cd25d61795 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-17 19:47:43 +02:00
kssenii
1322ad53b8 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-17 18:22:21 +02:00
kssenii
53181c938a Fixes 2023-04-17 16:32:08 +02:00
Sichen Zhao
eb4c26d81f add aliases for parts name and names 2023-04-17 13:18:40 +00:00
Raúl Marín
db71bbb08d Fix examples build 2023-04-17 10:59:40 +02:00
Raúl Marín
39f8c43a60 Merge remote-tracking branch 'blessed/master' into arenita 2023-04-17 10:33:38 +02:00
Michael Kolupaev
473f212c82 Hopefully fix assertion failure in CachedOnDiskReadBufferFromFile 2023-04-17 04:58:32 +00:00
kssenii
0517de58d1 Simplify FileCacheFactory 2023-04-15 16:15:20 +02:00
flynn
203df96d5c IN operator support compare Date and Date32 2023-04-15 12:41:42 +00:00
Alexander Gololobov
018f768e7e Cleanups 2023-04-14 20:07:37 +02:00
Alexander Tokmakov
70c1000c22 fix some tests 2023-04-14 19:42:17 +02:00
Vitaly Baranov
567111f146
Merge pull request #48683 from azat/table-deps-fix 
Fix table dependencies in case of failed RENAME TABLE
 2023-04-14 17:22:14 +02:00
Dmitry Novik
cf5d9a175a Revert "Merge pull request #48760 from ClickHouse/revert-46089-background-memory-tracker" 
This reverts commit a61ed33223, reversing
changes made to 5f01b8a2b5.
 2023-04-14 16:34:19 +02:00
Alexey Milovidov
84cb2ff879 Added setting "use_index_for_in_with_subqueries_max_values" #2571 2023-04-14 16:12:18 +02:00
Alexander Gololobov
28d0cc974a Cleanups 2023-04-14 16:12:18 +02:00
Alexander Gololobov
0ae74ddf42 Use different types for referencing local set and set that is stored in cache shared between tasks 2023-04-14 16:12:18 +02:00
Alexander Gololobov
05a9d1cbfc Cleanups 2023-04-14 16:12:18 +02:00
Alexander Gololobov
0aa7979dd5 Unblock tests with ENGINE=Set 2023-04-14 16:12:18 +02:00
Alexander Gololobov
ae2da38549 Check that the set has been built before it is used 2023-04-14 16:12:18 +02:00
Alexander Gololobov
fbf09a1115 Fixes for new analyzer 2023-04-14 16:12:18 +02:00
Alexander Gololobov
adbe87e9d3 Fixes 2023-04-14 16:12:18 +02:00
Alexander Gololobov
0b9579bfb8 Reworked, testing 2023-04-14 16:12:18 +02:00
Alexander Gololobov
02876ed87d Share PreparedSet-s between task of the same mutation to reduce mem usage 2023-04-14 16:12:18 +02:00
Dmitry Novik
352a240a20
Merge pull request #47984 from ClickHouse/optimize-compilation 
Optimize compilation
 2023-04-14 15:34:46 +02:00
kssenii
ed2a5d25e5 Fix 2023-04-14 15:26:25 +02:00
kssenii
f7232ef537 Fix 2023-04-14 12:46:36 +02:00
Robert Schulze
2378df0a47
Merge pull request #48763 from ClickHouse/rs/fix-squashing 
Fix squashing in query cache
 2023-04-14 11:44:58 +02:00
Robert Schulze
a09db7f4f5
Fix squashing in query cache 2023-04-13 20:09:29 +00:00
Robert Schulze
43b8cbc50c
Merge pull request #48737 from ClickHouse/rs/fix-wshadow 
Remove `-Wshadow` suppression which leaked into global namespace
 2023-04-13 21:18:50 +02:00
Alexander Tokmakov
af1bf08663
Revert "Add MemoryTracker for the background tasks" 2023-04-13 21:05:02 +03:00
Anton Popov
5f01b8a2b5
Merge pull request #48517 from CurtizJ/fix-memory-compressed-tables 
Fix storage `Memory` with enabled compression
 2023-04-13 19:52:49 +02:00
pufit
abe3c9b9db
Merge pull request #48758 from ClickHouse/revert-48679-scheme-infer 
Revert "Make Schema inference works for CREATE AS SELECT"
 2023-04-13 13:46:30 -04:00
Anton Popov
4a1c868ca9
Merge pull request #48663 from CurtizJ/fix-async-insert-no-data 
Fix async inserts with empty data
 2023-04-13 19:44:47 +02:00
pufit
d717372c6e
Revert "Make Schema inference works for CREATE AS SELECT" 2023-04-13 13:35:32 -04:00
pufit
a8c892f925
Merge pull request #48679 from ucasfl/scheme-infer 
Make Schema inference works for CREATE AS SELECT
 2023-04-13 13:02:04 -04:00
kssenii
599a02ca0d Fix 2023-04-13 18:53:16 +02:00
kssenii
461477dabf Fix style check 2023-04-13 16:38:38 +02:00
Dmitry Novik
5cc9b46f78 Merge remote-tracking branch 'origin/master' into optimize-compilation 2023-04-13 16:04:09 +02:00
kssenii
ce723ec32d Fix style check, better priority->iterate 2023-04-13 14:45:28 +02:00
kssenii
58a30213c9 Fix after merge 2023-04-13 13:34:19 +02:00
Dmitry Novik
467ecf45e1
Merge pull request #46089 from ClickHouse/background-memory-tracker 
Add MemoryTracker for the background tasks
 2023-04-13 13:29:27 +02:00
kssenii
ce45105448 Cleanup a bit 2023-04-13 13:27:01 +02:00
kssenii
09c23287aa Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-13 13:25:45 +02:00
Raúl Marín
2b70e08f23 Don't count unreserved bytes in Arenas as read_bytes 2023-04-13 12:43:24 +02:00
Robert Schulze
7a21d5888c
Remove -Wshadow suppression which leaked into global namespace 2023-04-13 08:46:40 +00:00
Raúl Marín
da9a539cf7 Reduce the usage of Arena.h 2023-04-13 10:31:32 +02:00
pufit
aa0f3bc994 Solve pr issues 2023-04-12 20:28:17 -04:00
kssenii
b773b506b2 Better 2023-04-12 23:27:06 +02:00
Anton Popov
05a93ef123 add flush id and time for async insert with empty data 2023-04-12 20:57:55 +00:00
Alexey Milovidov
9a3030803e
Merge pull request #48596 from azat/ci/second_deadlock_stack 
Add second_deadlock_stack=1 for TSan on CI and fix some lock-order-inversion problems
 2023-04-12 22:11:06 +03:00
Anton Popov
08aa629e56
Merge branch 'master' into fix-memory-compressed-tables 2023-04-12 15:38:54 +02:00
Nikolai Kochetov
f66e9c5a26
Merge pull request #48427 from ClickHouse/update_sort_desc_follow_up 
Update sort desc: more efficient original node search in ActionsDAG
 2023-04-12 13:59:37 +02:00
Azat Khuzhin
5277c802f7 Fix table dependencies in case of failed RENAME TABLE 
CI found this [1]

    [ 382620 ] {} <Error> Application: Caught exception while loading metadata: Code: 60. DB::Exception: Table test_25.join doesn\'t exist: While processing _CAST(joinGet(test_25.join, \'m\', CAST(\'42\', \'int\')) AS m_tmp_alter1351896088559986425, \'Int32\') AS m: default expression and column type are incomp
    [ 382620 ] {} <Error> Application: Code: 60. DB::Exception: Table test_25.join doesn\'t exist: While processing _CAST(joinGet(test_25.join, \'m\', CAST(\'42\', \'int\')) AS m_tmp_alter1351896088559986425, \'Int32\') AS m: default expression and column type are incompatible.: Cannot attach table `test_25`.`t

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/48670/d7f865037266ed87538cf4df7ec7e8165681871b/stress_test__asan_.html

The problem here is that after failed RENAME dependencies got lost:

    [4e937f39d866] 2023.04.12 00:51:06.833624 [ 13419 ] {61429225-64cb-4fce-b60d-01e0dac6e52c} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 34.65 GiB (attempt to allocate chunk of 2097419 bytes), maximum: 34.29 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory. (MEMORY_LIMIT_EXCEEDED) (version 23.4.1.1) (from [::1]:45710) (comment: 01160_table_dependencies.sh) (in query: create database test_25_1), Stack trace (when copying this message, always include the lines below):
    [4e937f39d866] 2023.04.12 00:51:07.351914 [ 5151 ] {66d8bdd4-668e-4239-a8af-6b8f17bb5222} <Error> executeQuery: Code: 81. DB::Exception: Database test_25_1 doesn't exist. (UNKNOWN_DATABASE) (version 23.4.1.1) (from [::1]:45762) (comment: 01160_table_dependencies.sh) (in query: rename table t to test_25_1.t), Stack trace (when copying this message, always include the lines below):

And from the test output:

    OK
    OK
    OK
    OK
    a       []      []      []
    data_02344      []      []      []
    date_table      []      []      []
    dict1   []      ['dict_src']    ['join']
    dict2   []      ['join']        []
    dict_src        []      []      ['dict1']
    dist_02346      []      []      []
    join    []      ['dict1']       ['dict2','s']
                                    ^^^^^^^^^^^^^ no "t"

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-12 11:11:05 +02:00
flynn
ea9187511e Make Schema inference works for CREATE AS SELECT 2023-04-12 04:21:03 +00:00
pufit
078b8f5399 Small refactor, additional tests 2023-04-11 19:55:59 -04:00
Dmitry Novik
06e6794fc0 Merge remote-tracking branch 'origin/master' into background-memory-tracker 2023-04-11 15:29:35 +00:00
Anton Popov
09c66c3879 fix async inserts with empty data 2023-04-11 15:13:46 +00:00
Mikhail f. Shiryaev
89e4509e68
Merge branch 'master' into fix-memory-compressed-tables 2023-04-11 16:38:51 +02:00
Alexander Tokmakov
a1fda047f6 add docs 2023-04-11 16:32:40 +02:00
Alexander Tokmakov
dfff30e1a9 Merge branch 'master' into clusters_is_active_column 2023-04-11 16:32:15 +02:00
Anton Popov
3896375247 fix getting subcolumn from in-memory data 2023-04-11 14:26:31 +00:00
Nikolay Degterinsky
0621222737 Fix crashes with incorrect query parameters 2023-04-11 14:22:55 +00:00
Nikolay Degterinsky
f34b304707 Fixes, add test 2023-04-11 12:54:54 +00:00
SmitaRKulkarni
e2010c0257
Merge branch 'master' into zk_retry_timeout 2023-04-11 14:05:52 +02:00
Nikolay Degterinsky
7606336fd8 Merge remote-tracking branch 'upstream/master' into add_default_password_type 2023-04-11 11:59:45 +00:00
vdimir
4605d71947
Fix build src/Interpreters/InterpreterInsertQuery.h 2023-04-11 11:17:02 +00:00
Sema Checherinda
4dd86a406a
Merge pull request #48543 from azat/mv-uniq-thread-group 
Use one ThreadGroup while pushing to materialized views (and some refactoring for ThreadGroup)
 2023-04-11 11:47:46 +02:00
Alexey Milovidov
d259217cf3
Merge pull request #48570 from azat/build/logger_useful 
Remove superfluous includes of logger_userful.h from headers
 2023-04-11 03:56:39 +03:00
Nikolay Degterinsky
908be1e324 Fix tests 2023-04-10 22:04:10 +00:00
Alexey Milovidov
f4d6596eaa
Merge pull request #48577 from ClickHouse/evillique-patch-2 
Don't use type conversion with String query parameters
 2023-04-10 22:18:39 +03:00
Azat Khuzhin
013a3ed269 Fix possible lock order inversion in Context::setTemporaryStorageInCache() 
TSan report [1]:

    WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=1)
      Cycle in lock order graph: M0 (0x7b9000000008) => M1 (0x7b90000000a8) => M0

      Mutex M1 acquired here while holding mutex M0 in main thread:
        0 pthread_mutex_lock <null> (clickhouse+0xc66a8da) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        3 std::__1::lock_guard<std::__1::mutex>::lock_guard[abi:v15000](std::__1::mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:94:27 (clickhouse+0x1d0ccb3a) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        4 DB::Context::getDisk() const build_docker/./src/Interpreters/Context.cpp:3281:21 (clickhouse+0x1d0ccb3a)
        5 DB::Context::setTemporaryStorageInCache() build_docker/./src/Interpreters/Context.cpp:927:21 (clickhouse+0x1d0cc188) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        6 DB::Server::main() build_docker/./programs/server/Server.cpp:1055:25 (clickhouse+0x148504c2) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)

      Mutex M0 previously acquired by the same thread here:
        0 pthread_mutex_lock <null> (clickhouse+0xc66a8da) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        3 std::__1::unique_lock<std::__1::recursive_mutex>::unique_lock[abi:v15000](std::__1::recursive_mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:122:61 (clickhouse+0x1d0cc148) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        4 DB::Context::getLock() const build_docker/./src/Interpreters/Context.cpp:690:12 (clickhouse+0x1d0cc148)
        5 DB::Context::setTemporaryStorageInCache() build_docker/./src/Interpreters/Context.cpp:922:17 (clickhouse+0x1d0cc148)
        6 DB::Server::main() build_docker/./programs/server/Server.cpp:1055:25 (clickhouse+0x148504c2) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)

      Muex M0 acquired here while holding mutex M1 in main thread:
        0 pthread_mutex_lock <null> (clickhouse+0xc66a8da) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        3 std::__1::unique_lock<std::__1::recursive_mutex>::unique_lock[abi:v15000](std::__1::recursive_mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:122:61 (clickhouse+0x1d0c7ce1) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        4 DB::Context::getLock() const build_docker/./src/Interpreters/Context.cpp:690:12 (clickhouse+0x1d0c7ce1)
        5 DB::Context::getPath() const build_docker/./src/Interpreters/Context.cpp:713:17 (clickhouse+0x1d0c7ce1)
        6 DB::loadDiskLocalConfig() build_docker/./src/Disks/DiskLocal.cpp:75:30 (clickhouse+0x1ce7ed8a) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        7 DB::DiskLocal::applyNewSettings() build_docker/./src/Disks/DiskLocal.cpp:491:5 (clickhouse+0x1ce7e7ef) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        8 DB::DiskSelector::updateFromConfig() const build_docker/./src/Disks/DiskSelector.cpp:92:19 (clickhouse+0x1d14088e) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        9 DB::Context::updateStorageConfiguration() build_docker/./src/Interpreters/Context.cpp:3388:49 (clickhouse+0x1d0ec002) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        ..
        17 DB::ConfigReloader::reloadIfNewer() build_docker/./src/Common/Config/ConfigReloader.cpp:149:13 (clickhouse+0x1fff4250)

      Mutex M1 previously acquired by the same thread here:
        0 pthread_mutex_lock <null> (clickhouse+0xc66a8da) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        3 std::__1::lock_guard<std::__1::mutex>::lock_guard[abi:v15000](std::__1::mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:94:27 (clickhouse+0x1d0ebf25) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280)
        4 DB::Context::updateStorageConfiguration() build_docker/./src/Interpreters/Context.cpp:3384:21 (clickhouse+0x1d0ebf25)
        ..
        12 DB::ConfigReloader::reloadIfNewer() build_docker/./src/Common/Config/ConfigReloader.cpp:149:13 (clickhouse+0x1fff4250)

    SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/usr/bin/clickhouse+0xc66a8da) (BuildId: 174c9c4bf3606a5fc390211bdc2feff052246280) in pthread_mutex_lock
    ==================

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/48596/a1272e8536265929255fdf5020836f057859e425/integration_tests__tsan__[6/6].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-10 19:56:52 +02:00
Nikolay Degterinsky
def8611d2c Fix style 2023-04-10 17:05:31 +00:00
Nikolay Degterinsky
6f57a952d9 Improvements 2023-04-10 16:40:49 +00:00
Azat Khuzhin
79b83c4fd2 Remove superfluous includes of logger_userful.h from headers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-10 17:59:30 +02:00
Nikolai Kochetov
28484cd762
Merge pull request #48522 from ClickHouse/many-mutations-memory-limit 
Reduce memory usage for multiple alter delete mutations.
 2023-04-10 12:50:42 +02:00
Raúl Marín
45ad555c39
Merge branch 'master' into zk_retry_timeout 2023-04-10 10:04:16 +02:00
Nikolay Degterinsky
8524f6958a
Update src/Interpreters/ReplaceQueryParameterVisitor.cpp 
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2023-04-10 09:09:57 +02:00
Nikolay Degterinsky
7957752128 Merge remote-tracking branch 'upstream/master' into add_default_password_type 2023-04-10 02:25:46 +00:00
Nikolay Degterinsky
64dc60a8d1 Better version, introduce ASTAuthenticationData 2023-04-10 02:20:57 +00:00
Nikolay Degterinsky
e71c7fe744
Fix 2023-04-10 03:48:58 +02:00
Nikolay Degterinsky
6b1c86c6ec
Don't use type conversion with String query parameters 2023-04-10 03:39:13 +02:00
Vitaly Baranov
62e9c419de
Merge pull request #48519 from azat/build/fwd-decl-threadpool 
Use forward declaration of ThreadPool
 2023-04-08 00:44:01 +02:00
Alexey Milovidov
e09e80b493
Merge pull request #48500 from CurtizJ/fix-sparse-column-query-cache 
Fix query cache with sparse columns
 2023-04-07 19:34:10 +03:00
Alexander Tokmakov
9ec91acef3 add some columns to system.clusters 2023-04-07 18:26:23 +02:00
Azat Khuzhin
44ae8485f1 Use one ThreadGroup while pushing to materialized views 
Before this patch only the case when ThreadStatus (current_thread)
already has ThreadGroup works that way, after they will be identical in
this aspect.

But this should not affect anything, but it just make sense.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 15:37:35 +02:00
Azat Khuzhin
aacf2a0838 Move ThreadGroupSwitcher to ThreadStatus.h (out from MergeTree code) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 15:32:08 +02:00
Azat Khuzhin
5b2b20a0b0 Rename ThreadGroupStatus to ThreadGroup 
There are methods like getThreadGroup() and ThreadGroupSwitcher class,
so seems that this is logical.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 15:31:48 +02:00
Sema Checherinda
0503ed6fb8
Merge pull request #46668 from azat/fix-query_views_log-from-bg 
Fix system.query_views_log for MVs that are pushed from background threads
 2023-04-07 14:47:05 +02:00
Azat Khuzhin
011480924a Use forward declaration of ThreadPool 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 11:25:35 +02:00
Yakov Olkhovskiy
8d1924cc9a
Merge pull request #47962 from ClickHouse/imp-use-structure-from-insertion 
Improvement and fix for use_structure_from_insertion_table_in_table_functions
 2023-04-06 19:01:00 -04:00
Nikolai Kochetov
e8fb135180 Reduce memory usage for multiple alter delete mutations. 2023-04-06 19:07:53 +00:00
Anton Popov
8d1b535efb fix storage Memory with compression 2023-04-06 18:27:53 +00:00
Sema Checherinda
9d8d9e8ec9
Merge branch 'master' into fix-query_views_log-from-bg 2023-04-06 18:16:47 +02:00
Anton Popov
4757d76fbf fix query cache with sparse columns 2023-04-06 16:01:23 +00:00
Anton Popov
b5a3723c88
Merge pull request #48395 from ClickHouse/with_fill_clarification 
WITH FILL clarification and cleanup
 2023-04-06 17:02:57 +02:00
pufit
d1fa9596b8 Prevent execution on cluster 2023-04-06 10:56:35 -04:00
Sema Checherinda
b8729be97a
Merge pull request #47104 from CheSema/thead-group-for-merge 
merge and mutation make thread group for setting memory trackers right
 2023-04-06 15:46:56 +02:00
Vitaly Baranov
b45b662fe3
Merge pull request #48431 from Algunenano/system_referential_table 
Don't check dependencies when renaming system tables automatically
 2023-04-06 15:28:37 +02:00
Nikolai Kochetov
6c6335b686
Merge pull request #48357 from ClickHouse/fix-projections-analysis-with-analyzer 
Batch fix for projections analysis with analyzer.
 2023-04-06 15:13:56 +02:00
Yakov Olkhovskiy
d2e897d4b8
Merge branch 'master' into imp-use-structure-from-insertion 2023-04-06 08:52:38 -04:00
Antonio Andelic
28a33f0a47
Merge pull request #48435 from ClickHouse/kv-update-only-affected-rows 
Update only affected rows in KV storage
 2023-04-06 12:33:54 +02:00
Vladimir C
e7a5c96353
Merge pull request #48219 from ClickHouse/vdimir/lock_order_47072 2023-04-06 11:54:33 +02:00
Vladimir C
a2b71a0054
Merge pull request #48386 from ClickHouse/vdimir/reset-downloader-tmp-data-cache 2023-04-06 11:23:16 +02:00
Yakov Olkhovskiy
8a19ac4e73
Merge branch 'master' into imp-use-structure-from-insertion 2023-04-06 00:51:44 -04:00
pufit
144ee7aed4 Fix logic, additional tests 2023-04-05 22:52:31 -04:00
robot-ch-test-poll3
3ad0a6ac18
Merge pull request #48433 from ClickHouse/some-fixes-for-parallel-replicas 
Some fixes for parallel replicas
 2023-04-06 02:00:06 +02:00
Alexander Tokmakov
a154fa73f4
Merge pull request #48401 from ClickHouse/fix_00002_log_messages 
Find big allocations without memory limits checks
 2023-04-05 23:52:39 +03:00
Sema Checherinda
554c928534
Merge branch 'master' into thead-group-for-merge 2023-04-05 21:09:38 +02:00
Raúl Marín
1fa992f015
Merge branch 'master' into system_referential_table 2023-04-05 20:47:53 +02:00
robot-clickhouse
3d2c279859
Merge pull request #48242 from azat/better-throttling 
Per-query/per-server throttling for remote IO/local IO/BACKUPs
 2023-04-05 19:32:37 +02:00
kssenii
a405d06e8a Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-05 18:46:52 +02:00
Antonio Andelic
0260b84bc3 Update only affected rows in KV storage 2023-04-05 16:12:14 +00:00
Raúl Marín
5d9e006c46 PR comments 2023-04-05 18:04:03 +02:00
Igor Nikonov
46f7016d8c Add search alias by input name in ActionsDAG 2023-04-05 15:56:06 +00:00
Nikita Mikhaylov
3f44c6d936 Better 2023-04-05 15:19:45 +00:00
Raúl Marín
d65d00f496 Don't check dependencies when renaming system tables automatically 2023-04-05 16:15:24 +02:00
vdimir
d9abcf2f69
Reduce scope storage_policies_mutex in setTemporaryStoragePolicy 2023-04-05 13:42:49 +00:00
vdimir
01ce3baf0f
Rename Context::getTemporaryVolume to getGlobalTemporaryVolume 2023-04-05 13:42:13 +00:00
Igor Nikonov
41f6e0e7b5 More efficient original node search in ActionsDAG 2023-04-05 13:36:07 +00:00
vdimir
6f3fa33641
Add unbound flag to system.filesystem_cache 2023-04-05 13:12:20 +00:00
Nikita Mikhaylov
7b526d3df9
Always communicate with local replica through loopback interface for parallel replicas (#48246) 2023-04-05 13:32:46 +02:00
Sema Checherinda
f504cd5fed avoid race, do not set thread_group.counters.parent from query 2023-04-05 12:36:55 +02:00
Igor Nikonov
47e7e19ca9 Move suffix generation code into one function 2023-04-05 10:12:42 +00:00
vdimir
bae1286b63
Update temp_data_on_disk in shared context 2023-04-05 10:07:44 +00:00
vdimir
7fe06f8435
Do not take lock for shared context in setTempDataOnDisk 2023-04-05 09:22:09 +00:00
Azat Khuzhin
4008155a6e Add max_ prefix for backup_bandwidth settings 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 10:54:25 +02:00
Azat Khuzhin
61405b827d Use MAKE_DEPRECATED_BY_SERVER_CONFIG() for deprecated throttle settings 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
684ecc24e2 Ignore deprecated values max_remote_{read,write}_network_bandwidth_for_server 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
ad96d3c249 Do not require proper Poco Application for Context 
$ yes | head | clickhouse-obfuscator --structure 'id String' --seed foo --input-format TSV --output-format TSV
    Poco::Exception. Code: 1000, e.code() = 0, Null pointer: _pInstance in file "/src/ch/clickhouse/base/poco/Util/include/Poco/Util/Application.h", line 446, Stack trace (when copying this message, always include the lines below):

    0. /src/ch/clickhouse/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x15605622 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    1. /src/ch/clickhouse/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x156055ed in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    2. /src/ch/clickhouse/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x2ebd2d80 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    3. /src/ch/clickhouse/base/poco/Foundation/src/Exception.cpp:132: Poco::LogicException::LogicException(String const&, int) @ 0x2ebd3667 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    4. /src/ch/clickhouse/base/poco/Foundation/src/Exception.cpp:134: Poco::NullPointerException::NullPointerException(String const&, int) @ 0x2ebd3da7 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    5. /src/ch/clickhouse/base/poco/Foundation/src/Bugcheck.cpp:42: Poco::Bugcheck::nullPointer(char const*, char const*, int) @ 0x2ebc5851 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    6. /src/ch/clickhouse/base/poco/Util/include/Poco/Util/Application.h:446: Poco::Util::Application::instance() @ 0x1ee3e6cd in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    7. /src/ch/clickhouse/src/Interpreters/Context.cpp:358: DB::ContextSharedPart::ContextSharedPart() @ 0x263b0cca in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    8. /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::ContextSharedPart>::__unique_single std::make_unique[abi:v15000]<DB::ContextSharedPart>() @ 0x26387b6e in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    9. /src/ch/clickhouse/src/Interpreters/Context.cpp:652: DB::Context::createShared() @ 0x26361a3c in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    10. /src/ch/clickhouse/programs/obfuscator/Obfuscator.cpp:1293: mainEntryClickHouseObfuscator(int, char**) @ 0x1ef6eb26 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    11. /src/ch/clickhouse/programs/main.cpp:481: main @ 0x155e569a in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    12. ? @ 0x7ffff7dd2790 in ?
    13. __libc_start_main @ 0x7ffff7dd284a in ?
    14. _start @ 0x155e51ae in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
     (version 23.3.1.2537)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
a8c1407bc4 Use existing ServerSettings in Context instead of reading it again 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
c332d290d8 Keep only one throttler for BACKUPs IO (instead of separate read/write) 
There is no need in separate read/write throttling, because you cannot
write faster then read anyway, and plus this makes the code less cleaner

(and also it will allow avoid implementing throttling backups to S3,
since it does not use common S3 writer).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:17 +02:00
Azat Khuzhin
218b1f9c29 Add ability to throttle BACKUPs on per-server/backup basis 
Server settings:
- backup_read_bandwidth_for_server
- backup_write_bandwidth_for_server

Query settings:
- backup_read_bandwidth
- backup_write_bandwidth

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:48 +02:00
Azat Khuzhin
a25dd1d348 Add ability to throttle local IO on per-query/server basis 
Server settings:
- max_local_read_bandwidth_for_server
- max_local_write_bandwidth_for_server

Query settings:
- max_local_read_bandwidth
- max_local_write_bandwidth

This is the preparation for adding ability to throttle BACKUPs

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Azat Khuzhin
b3406beeb7 Add per-query network throttling 
Controlled with:
- max_remote_read_network_bandwidth
- max_remote_write_network_bandwidth

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Azat Khuzhin
abdb682048 Move some methods for Context into module part 
This will be required later, for per-query bandwidth limiting.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Azat Khuzhin
40cb87c2ac Make remote bandwidth settings server settings 
Right now those two settings are applied only at start:
- max_remote_read_network_bandwidth_for_server
- max_remote_write_network_bandwidth_for_server

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
pufit
8441b0a3e1 Fix comments, tests, docs 2023-04-04 20:43:15 -04:00
Alexander Tokmakov
ba65be22da fix bad error messages 2023-04-04 20:21:22 +02:00
vdimir
5966281f7c
Test WriteBufferToFileSegment with several threads 2023-04-04 18:15:27 +00:00