thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 02:12:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
137,459 Commits 9,330 Branches 1,783 Tags 2.1 GiB
1264b5337e
Commit Graph

16571 Commits

Author SHA1 Message Date
Anton Popov
d683977fe9 Revert "Revert "Add table function mergeTreeIndex"" 
This reverts commit 1e4199d704.
 2024-02-26 22:47:39 +00:00
Alexey Milovidov
04fe5dc55c Merge branch 'master' into fix-inconsistent-formatting 2024-02-26 23:46:24 +01:00
Alexey Milovidov
5b1a07053e Merge branch 'master' into dont-load-useless-index-parts-in-memory 2024-02-26 23:39:53 +01:00
Alexey Milovidov
0b90076a36
Revert "Revert "Check stack size in Parser"" 2024-02-27 01:34:50 +03:00
Robert Schulze
4794dc28f0
Merge remote-tracking branch 'ClickHouse/master' into dist/config-settings 2024-02-26 21:57:33 +00:00
Kirill Nikiforov
8b314a3ac8
Implement system.dns_cache table and dns_cache_max_size server setting (#60257) 
* system.dns_cache table WIP

* system.dns_cache table

* rollback unexpected contrib changes

* rollback unexpected changes

* add docs

* code style fixes

* optimize dns cache exporting

* stateful test for system.dns_cache table

* stateful test for system.dns_cache table

* cr fixes

* stateful test for system.dns_cache table

* stateful test for system.dns_cache table

* implement system.dns_cache table

* system.dns_cache test

* system.dns_cache fix docs

* system.dns_cache prettify

* dns_cache_max_size settings

* fix system.dns_cache test

* Update dns_cache.md

* fix doc

Co-authored-by: pufit <pufit@clickhouse.com>

---------

Co-authored-by: pufit <pufit@clickhouse.com>
 2024-02-26 15:48:02 -05:00
Raúl Marín
8a161bdc0a Fix unrestricted reads from keeper 2024-02-26 19:38:00 +01:00
Alexander Tokmakov
1e4199d704
Revert "Add table function mergeTreeIndex" 2024-02-26 19:37:06 +01:00
Alexey Milovidov
a3f1411679 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-inconsistent-formatting 2024-02-26 19:01:11 +01:00
HowePa
dbd8d35f01 use lower case in dict 2024-02-27 00:48:34 +08:00
vdimir
9f1a0734a1
review fix 2024-02-26 16:15:14 +00:00
vdimir
5a9026f39b
Analyzer: compute ALIAS columns right after reading 2024-02-26 16:15:05 +00:00
Anton Popov
814cd53918
Merge pull request #58140 from CurtizJ/storage-merge-tree-index 
Add table function `mergeTreeIndex`
 2024-02-26 16:31:56 +01:00
Dmitry Novik
42b91201a5 Merge remote-tracking branch 'origin/master' into analyzer-refactor-constant-name 2024-02-26 15:11:57 +00:00
HowePa
ea89fa0de9 replace to checkFormatName 2024-02-26 23:07:50 +08:00
Robert Schulze
1e40b63cb1
Merge remote-tracking branch 'ClickHouse/master' into dist/config-settings 2024-02-26 11:15:51 +00:00
Raúl Marín
f84209be2f Merge remote-tracking branch 'blessed/master' into max_threads_sanity 2024-02-26 11:56:29 +01:00
Alexey Milovidov
bf75d2878b Merge branch 'master' into fix-inconsistent-formatting 2024-02-26 00:09:02 +01:00
Amos Bird
38427fff35
Better name 2024-02-26 02:32:00 +08:00
Amos Bird
d577d1e4e2
Fix something 2024-02-26 02:32:00 +08:00
Amos Bird
534da3f40f
Fix more tests 2024-02-26 02:31:59 +08:00
Amos Bird
6137914604
Fix most tests 2024-02-26 02:31:58 +08:00
Amos Bird
8a1e23dcc4
Fix more tests 2024-02-26 02:31:57 +08:00
Amos Bird
7435568a0d
Fix tests 2024-02-26 02:31:57 +08:00
Amos Bird
c5e073320c
Unify prewhere optimization 2024-02-26 02:31:56 +08:00
Amos Bird
49352732cc
Fix more tests 2024-02-26 02:31:56 +08:00
Amos Bird
ffab16595e
Fix more tests 2024-02-26 02:31:55 +08:00
Amos Bird
07a0adceb9
Only use query_info.filter_actions_dag for key analysis 2024-02-26 02:31:55 +08:00
Azat Khuzhin
a70a100db0 Support io_uring for StorageFile 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-02-24 19:36:58 +01:00
Vitaly Baranov
f944faec6f Fix using table_id in StorageKeeperMap. 2024-02-23 20:27:19 +01:00
Alexey Milovidov
da2ae34c1d Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-inconsistent-formatting 2024-02-23 15:55:03 +01:00
Alexander Tokmakov
ef3b19167f
Merge pull request #58036 from MikhailBurdukov/backoff_for_failed_mutations 
Backoff policy for failed mutation.
 2024-02-23 15:27:41 +01:00
Alexey Milovidov
396402a9b6 Fix inconsistent formatting of operator function and nulls action 2024-02-23 15:22:21 +01:00
Nikita Mikhaylov
8fef0b71c4
Merge pull request #60350 from ClickHouse/revert-54901-pufit/views-sql-security 
Revert "Add definers for views"
 2024-02-23 15:15:36 +01:00
Dmitry Novik
d13b2a91c1 Fix virtual tables 2024-02-23 13:33:12 +00:00
Alexander Tokmakov
32607cc796
Merge pull request #60159 from ClickHouse/fix_create_replica 
Skip sanity checks on secondary CREATE query
 2024-02-23 13:53:47 +01:00
Sergei Trifonov
2c766cf75f
Merge pull request #59507 from ClickHouse/alter-table-forget-partition 
Add `FORGET PARTITION` query to remove old partition nodes from ZooKeeper
 2024-02-23 12:55:08 +01:00
Raúl Marín
464b5e0e15
Revert "Add definers for views (#54901)" 
This reverts commit 57306706b3.
 2024-02-23 12:44:31 +01:00
Raúl Marín
a431608dd4 Add sanity check for some settings 2024-02-23 12:32:17 +01:00
Kseniia Sumarokova
6a7fef13ba
Merge pull request #60282 from ClickHouse/s3queue-fix-bug-and-flaky-test 
s3queue: fix bug (also fixes flaky test_storage_s3_queue/test.py::test_shards_distributed)
 2024-02-23 11:57:34 +01:00
Dmitry Novik
81df9c5a27
Use possition to merge plans 2024-02-22 19:07:54 +01:00
Alexander Tokmakov
e56748a92c
Merge pull request #60044 from ClickHouse/fix_undead_sessions 
Ability to detect undead ZooKeeper sessions
 2024-02-22 18:13:15 +01:00
pufit
57306706b3
Add definers for views (#54901) 2024-02-22 17:47:50 +01:00
kssenii
f1e95fb78b Add a way to force read-through cache for merges 2024-02-22 15:40:00 +01:00
Azat Khuzhin
0cb6f205d7 Fix actions execution during preliminary filtering (PK, partition pruning) 
Previously preliminary filtering works incorrectly if input columns had
been used multiple times, because:
- position initialized only for the first user
- column had been moved, so the second user will get nullptr

And this was the case when it had been required multiple times.

Consider the following example:

    select * from data final prewhere indexHint(_partition_id = 'all') or indexHint(_partition_id = 'all')

Actions for this PREWHERE is the following:

    (lldb) p actions.__ptr_->dumpActions()
    (std::string) $1 = "input:
    _partition_id LowCardinality(String)
    _partition_id LowCardinality(String)

    actions:
    INPUT : 0 -> _partition_id LowCardinality(String) : 0
    COLUMN Const(String) -> 'all' String : 1
    COLUMN Const(String) -> 'UInt8'_String String : 2
    INPUT : 1 -> _partition_id LowCardinality(String) : 3
    COLUMN Const(String) -> 'all' String : 4
    COLUMN Const(String) -> 'UInt8'_String String : 5
    FUNCTION equals(_partition_id :: 0, 'all' :: 1) -> equals(_partition_id, 'all') LowCardinality(UInt8) : 6
    FUNCTION equals(_partition_id :: 3, 'all' :: 4) -> equals(_partition_id, 'all') LowCardinality(UInt8) : 1
    FUNCTION _CAST(equals(_partition_id, 'all') :: 6, 'UInt8'_String :: 2) -> _CAST(equals(_partition_id, 'all'), 'UInt8'_String) UInt8 : 4
    FUNCTION _CAST(equals(_partition_id, 'all') :: 1, 'UInt8'_String :: 5) -> _CAST(equals(_partition_id, 'all'), 'UInt8'_String) UInt8 : 2
    FUNCTION or(_CAST(equals(_partition_id, 'all'), 'UInt8'_String) :: 4, _CAST(equals(_partition_id, 'all'), 'UInt8'_String) :: 2) -> or(indexHint(), indexHint())"...

It has _partition_id column in input multiple times.

So fix this by adding a flag (`allow_duplicates_in_input`), if set,
columns will be fully initialized.

v2: fix "Position out of bound in Block::erase()"
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-02-22 15:01:57 +01:00
Alexander Tokmakov
97ed2c05b0 Merge branch 'master' into fix_undead_sessions 2024-02-22 14:36:36 +01:00
Nikolai Kochetov
0e6b55ad35
Merge pull request #60220 from kitaisreal/system-backups-system-backup-log-add-query-id-and-error-stacktrace 
Tables system.backups and system.backup_log add query_id and error stacktrace
 2024-02-22 12:43:52 +01:00
kssenii
222323f293 Fix 2024-02-22 12:38:13 +01:00
Azat Khuzhin
a4f765cae7
Improve performance of SELECTs with active mutations (#59531) 
* Configure keeper for perf tests

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Improve performance of SELECTs with active mutations

getAlterMutationCommandsForPart() can be a hot path for query execution
when there are pending mutations.

- LOG_TEST - it is not only check one bool, but actually a bunch of
  atomics as well.

- Return std::vector over std::map (map is not required there) - no
  changes in performance.

- Copy only RENAME_COLUMN (since only this mutation is required by
  AlterConversions).

And here are results:

run|result
-|-
SELECT w/o ALTER|queries: 1565, QPS: 355.259, RPS: 355.259
SELECT w/ ALTER unpatched|queries: 2099, QPS: 220.623, RPS: 220.623
SELECT w/ ALTER and w/o LOG_TEST|queries: 2730, QPS: 235.859, RPS: 235.859
SELECT w/ ALTER and w/o LOG_TEST and w/ RENAME_COLUMN only|queries: 2995, QPS: 290.982, RPS: 290.982

But there are still room for improvements, at least MergeTree engines
could implement getStorageSnapshotForQuery().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Add AlterConversions::supportsMutationCommandType(), flatten vector<vector<MutationCommand>>

* Work around what appears to be a clang static analysis bug

---------

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Co-authored-by: Michael Kolupaev <michael.kolupaev@clickhouse.com>
 2024-02-22 08:51:10 +00:00
Alexey Milovidov
e98d09c93e Do not load useless columns from the index in memory 2024-02-21 21:41:04 +01:00
Alexander Tokmakov
c5ab1893d7 Merge branch 'master' into modify-engine 2024-02-21 20:37:58 +01:00
Raúl Marín
453d4d30cf
Revert "Support resource request canceling" 2024-02-21 20:19:00 +01:00
Alexander Tokmakov
fcfea17657
Update gtest_storage_log.cpp 2024-02-21 20:16:14 +01:00
Anton Popov
c624e2fa8b fix style check 2024-02-21 18:56:28 +00:00
Anton Popov
6656cdb9fb refactoring of virtual columns 2024-02-21 17:24:12 +00:00
Anton Popov
3d57237fe8 refactoring of virtual columns 2024-02-21 16:40:27 +00:00
Sergei Trifonov
19e81c20ca
Merge pull request #59032 from ClickHouse/cpu-scheduler-and-cc 
Support resource request canceling
 2024-02-21 16:16:24 +01:00
serxa
69c5fae1bc review fixes 2024-02-21 15:11:50 +00:00
serxa
8b7ef7c895 assert is not needed 2024-02-21 14:53:19 +00:00
Igor Nikonov
a0105e94ba
Merge pull request #60077 from ClickHouse/cleanup-connection-pool-priority 
Cleanup: less confusion between config priority and balancing priority in connection pools
 2024-02-21 15:22:08 +01:00
serxa
f92ec2cd5d Merge branch 'master' into cpu-scheduler-and-cc 2024-02-21 14:05:37 +00:00
serxa
bf90da7057 Merge branch 'master' into cpu-scheduler-and-cc 2024-02-21 14:05:01 +00:00
Maksim Kita
ae91c655a0 Tables system.backups and system.backup_log add query_id and error stacktrace 2024-02-21 14:36:58 +03:00
Antonio Andelic
fee1565780
Revert "Check stack size in Parser" 2024-02-21 11:33:08 +01:00
Kseniia Sumarokova
d601534a4e
Merge pull request #60166 from ClickHouse/fix-s3-queue-system-table-uuid-column 
Fix system.s3queue_log
 2024-02-21 10:52:45 +01:00
Alexey Milovidov
c9686eb76c
Merge pull request #60123 from conicl/skip_log_empty_message 
skip printing meaningless log
 2024-02-21 09:02:26 +01:00
Anton Popov
74399253d9 refactoring of virtual columns 2024-02-21 02:28:30 +00:00
Anton Popov
23f300b581 refactoring of virtual columns 2024-02-21 00:57:35 +00:00
Alexander Tokmakov
45d407034f Merge branch 'master' into fix_undead_sessions 2024-02-20 22:21:37 +01:00
Alexander Tokmakov
ec0986af0b skip sanity checks on secondary create query 2024-02-20 21:59:28 +01:00
alesapin
392081256c
Merge pull request #60078 from kitaisreal/revert-60058-revert-59946-replicated-merge-tree-alter-update-metadata-version-in-single-transaction 
Revert "Revert "ReplicatedMergeTree invalid metadata_version fix""
 2024-02-20 19:24:35 +01:00
Anton Popov
533082495d fix build 2024-02-20 16:33:43 +00:00
Kseniia Sumarokova
f7de95cec3
Merge pull request #59775 from ClickHouse/rabbitmq-fix-possible-channel-ack-timeout 
rabbitmq: fix possible channel error state
 2024-02-20 16:08:33 +01:00
Sergei Trifonov
d2aebe2c9a
Merge branch 'master' into alter-table-forget-partition 2024-02-20 13:56:50 +01:00
Antonio Andelic
34e8e673e9
Merge pull request #60144 from ClickHouse/per-user-s3-settings 
Support specifying users for S3 settings
 2024-02-20 13:20:07 +01:00
Igor Nikonov
b6277d5e87 Merge remote-tracking branch 'origin/master' into cleanup-connection-pool-priority 2024-02-20 12:13:10 +00:00
Robert Schulze
7e11fc79d9
Merge pull request #58023 from canhld94/materialize_non_override_past_values 
Materializing an ordinary column with default expression should not override past values
 2024-02-20 11:12:31 +01:00
kssenii
3dac4c9743 Fix 2024-02-20 11:10:11 +01:00
kssenii
5ef4a30d6f Fix 2024-02-20 10:24:09 +01:00
MikhailBurdukov
c2d641c658
Merge branch 'master' into backoff_for_failed_mutations 2024-02-20 11:48:59 +03:00
Maksim Kita
9e0f607608 Revert "Revert "ReplicatedMergeTree invalid metadata_version fix"" 2024-02-20 11:39:57 +03:00
MikhailBurdukov
7cdfedaedd Enable setting by default 2024-02-20 08:39:11 +00:00
Alexey Milovidov
b216223aa0
Merge pull request #60139 from ClickHouse/revert-60093-lazy-primary-key-loading 
Fix data race in `IMergeTreeDataPart`
 2024-02-20 06:44:28 +01:00
Raúl Marín
9361946d15 Fix build in master 2024-02-19 17:48:53 +01:00
alesapin
31bc327005
Merge pull request #60111 from ClickHouse/buffer_bg_flush_parallel 
Flush StorageBuffer into multiple threads if num_layers > 1
 2024-02-19 16:56:24 +01:00
Antonio Andelic
175a1db787 Support specifying users for s3 settings 2024-02-19 16:43:27 +01:00
Anton Popov
0aa8dafe0a Merge remote-tracking branch 'upstream/master' into HEAD 2024-02-19 14:07:55 +00:00
Vitaly Baranov
f10a8868e1
Merge pull request #60097 from vitlibar/move-thread-pool-callback-runner-to-common 
Move threadPoolCallbackRunner to the "Common" folder
 2024-02-19 14:10:36 +01:00
Antonio Andelic
7c2654f62e Fix data race 2024-02-19 14:09:37 +01:00
Dmitry Novik
11970ddf82
Merge branch 'master' into analyzer-fix-test_select_table_name_from_merge_over_distributed 2024-02-19 14:07:58 +01:00
Dmitry Novik
2122454576 Temporary progress 2024-02-19 12:41:29 +00:00
Igor Nikonov
f06e15ce5f Merge remote-tracking branch 'origin/master' into cleanup-connection-pool-priority 2024-02-19 11:33:55 +00:00
Alexey Milovidov
a10eddecef
Merge pull request #59697 from ClickHouse/check-stack-size-in-parser 
Check stack size in Parser
 2024-02-19 11:33:04 +01:00
Alexey Milovidov
df48106cd5
Merge pull request #60015 from azat/values-quote-escape 
Fix INSERT into SQLite with single quote (by escaping single quotes with a quote instead of backslash)
 2024-02-19 10:37:45 +01:00
alesapin
6565423b1a Review fix 2024-02-19 10:32:36 +01:00
conicliu
b38ad8297c skip log empty message 2024-02-19 16:11:42 +08:00
Alexey Milovidov
fa1ca348a4 Miscellaneous 2024-02-19 03:19:21 +01:00
Alexey Milovidov
6fd563df52 Miscellaneous 2024-02-19 02:58:51 +01:00
MikhailBurdukov
c3ab9071fc
Merge branch 'master' into backoff_for_failed_mutations 2024-02-18 17:48:11 +03:00
MikhailBurdukov
b227e5dd3f Review changes 2024-02-18 13:37:12 +00:00
alesapin
29e3e7cb96 Better if only 1 layer 2024-02-18 13:19:35 +01:00
alesapin
8f29320a73 Flush StorageBuffer into multiple threads if num_layers > 1 2024-02-18 13:15:24 +01:00
Alexey Milovidov
8d6e6d67ed Merge branch 'fix-inconsistent-formatting-of-explain-in-subqueries' into lazy-primary-key-loading 2024-02-18 11:04:01 +01:00
Alexey Milovidov
232fe0dd1b Fix inconsistent formatting of SELECT ... FROM (EXPLAIN ...) 2024-02-18 05:38:34 +01:00
Alexey Milovidov
c083498cf8 Apply review comments 2024-02-18 05:13:17 +01:00
Vitaly Baranov
3323d5ce81 Move threadPoolCallbackRunner to the "Common" folder. 2024-02-17 21:59:51 +01:00
Alexey Milovidov
0e944d400c
Merge pull request #60092 from ClickHouse/index-preparation 
A small preparation for better handling of primary key in memory
 2024-02-17 10:15:06 +01:00
Alexey Milovidov
c5107afb94
Merge branch 'master' into lazy-primary-key-loading 2024-02-17 10:14:35 +01:00
Alexey Milovidov
6cae75751b
Merge pull request #60049 from ClickHouse/less-memory-usage-primary-key 
Maybe less memory usage for primary keys in memory
 2024-02-17 08:58:02 +01:00
Alexey Milovidov
021e11e4fc Add a setting for lazy loading 2024-02-17 06:42:53 +01:00
Alexey Milovidov
423ff759f1 Lazy loading of primary keys in memory 2024-02-17 06:35:03 +01:00
Alexey Milovidov
e9cf92245f Move methods to .cpp 2024-02-17 06:20:44 +01:00
Alexey Milovidov
fce3a8cafc A small preparation for better handling of primary key in memory 2024-02-17 06:16:56 +01:00
Alexey Milovidov
ac5bef7c74 A small preparation for better handling of primary key in memory 2024-02-17 06:16:01 +01:00
Alexey Milovidov
46c2f85314 Merge branch 'master' into less-memory-usage-primary-key 2024-02-17 04:55:59 +01:00
Alexey Milovidov
2f315e0eb5 Style check for abbreviations 2024-02-17 02:14:15 +01:00
Alexey Milovidov
731c484b3d
Revert "Implement system.dns_cache table (#59856)" 
This reverts commit b5ef034697.
 2024-02-17 01:46:41 +01:00
Kirill Nikiforov
b5ef034697
Implement system.dns_cache table (#59856) 
* system.dns_cache table WIP

* system.dns_cache table

* rollback unexpected contrib changes

* rollback unexpected changes

* add docs

* code style fixes

* optimize dns cache exporting

* stateful test for system.dns_cache table

* stateful test for system.dns_cache table

* cr fixes

* stateful test for system.dns_cache table

* stateful test for system.dns_cache table

* Update 02981_system_dns_cache_table.sql

* Update 02981_system_dns_cache_table.reference

---------

Co-authored-by: pufit <pufit@clickhouse.com>
 2024-02-16 14:31:22 -05:00
Alexander Tokmakov
b513f1d456 fix build 2024-02-16 18:27:22 +01:00
Alexander Tokmakov
31479649fe
Merge pull request #59908 from azat/rmt-uuid-macro 
[RFC] Allow uuid in replica_path if CREATE TABLE explicitly has it
 2024-02-16 17:31:59 +01:00
Alexander Tokmakov
3813a97775
Merge pull request #59895 from azat/rmt-mutations-polling 
Do not pull mutations if pulling replication log had been stopped
 2024-02-16 17:24:30 +01:00
Alexander Tokmakov
03ff171922 Merge branch 'master' into modify-engine 2024-02-16 17:22:24 +01:00
Dmitry Novik
a3c34a7cc2
Merge pull request #54211 from ClickHouse/Support_parameterized_view_with_analyzer 
Refactor parameterized view & add support for new analyzer
 2024-02-16 16:13:49 +01:00
Igor Nikonov
b85a68790a Cleanup: connection pool priority -> config priority 
- names were creating confusion between config priority and balancing priority for a reader
 2024-02-16 14:39:41 +00:00
Kruglov Pavel
4d6f167e0c
Merge pull request #59092 from Avogar/auto-format-detection 
Try to detect file format automatically during schema inference if it's unknown
 2024-02-16 14:32:18 +01:00
Alexander Tokmakov
228a29b93f fix 2024-02-16 14:05:22 +01:00
alesapin
0a189e830c
Merge pull request #59942 from kitaisreal/storage-system-tables-add-metadata-version 
StorageSystemTables add metadata_version
 2024-02-16 13:17:22 +01:00
Raúl Marín
e168329ade
Revert "ReplicatedMergeTree invalid metadata_version fix" 2024-02-16 11:44:16 +01:00
Alexey Milovidov
2f36c9e965 Maybe less memory usage for primary keys in memory 2024-02-16 00:56:24 +01:00
Anton Popov
ef6b8275b1 Merge branch 'virtual-column-prewhere' of https://github.com/amosbird/ClickHouse into refactor-virtual-columns 2024-02-15 21:45:58 +00:00
Alexander Tokmakov
22c97be2a4 ability to detect undead ZooKeeper sessions 2024-02-15 22:17:31 +01:00
kssenii
2a783321d7 Fix deletion of s3 queue 2024-02-15 17:56:38 +01:00
Azat Khuzhin
f10fc95933 Fix INSERT into SQLite with single quote 
Previously it leads to syntax error, due to incorrect escaping of single
quotes for SQLite, "\'" had been used instead of "''"

So set output_format_values_escape_quote_with_quote=true for SQLite to
fix this.

v2: prepare modified Context for writing on storage creation
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-02-15 12:47:29 +01:00
Robert Schulze
f8413481c4
Merge remote-tracking branch 'ClickHouse/master' into materialize_non_override_past_values 2024-02-15 10:29:00 +00:00
Alexey Milovidov
a98facee52
Merge pull request #59944 from ClickHouse/fewer_lambdas 
Replace lambdas with pointers to members to simplify stacks
 2024-02-15 06:02:35 +01:00
Alexey Milovidov
009d08561d
Merge pull request #59946 from kitaisreal/replicated-merge-tree-alter-update-metadata-version-in-single-transaction 
ReplicatedMergeTree invalid metadata_version fix
 2024-02-15 04:38:06 +01:00
Alexey Milovidov
681351fe1e
Merge pull request #59987 from nickitat/fix_data_race_storage_distributed 
Fix data race in `StorageDistributed`
 2024-02-15 02:55:08 +01:00
Alexey Milovidov
7a5e152d9e
Merge pull request #59967 from ClickHouse/dashboards-to-use-merge-tables 
Use merge table in the dashboard; add Cloud overview to the system table
 2024-02-14 21:56:37 +01:00
Nikita Taranov
5f87956fc8 impl 2024-02-14 16:22:14 +01:00
Dmitry Novik
8d2ad5383b Fix execution name for constants 2024-02-14 15:19:08 +00:00
Maksim Kita
d4778d858d Fixed tests 2024-02-14 18:05:29 +03:00
Maksim Kita
0cce386207 Added documentation 2024-02-14 18:05:29 +03:00
Maksim Kita
f6d211b44d StorageSystemTables add metadata_version 2024-02-14 18:05:29 +03:00
Maksim Kita
be490291af Updated implementation 2024-02-14 18:05:06 +03:00
Maksim Kita
dca627026c ReplicatedMergeTree ALTER update metadata_version in single transaction 2024-02-14 18:05:06 +03:00
kssenii
8fe9ede207 Add a test 2024-02-14 15:25:43 +01:00
Dmitry Novik
617e73ce28 Merge remote-tracking branch 'origin/master' into analyzer-fix-test_select_table_name_from_merge_over_distributed 2024-02-14 13:32:14 +00:00
Nikolai Kochetov
03720d558e
Merge branch 'master' into allow-parallel-replicas-for-join-with-analyzer-2 2024-02-14 12:54:29 +01:00
Alexander Gololobov
dda7c5d2b7 Add comment with motivation 2024-02-14 11:41:07 +01:00
Alexey Milovidov
7b3e57e2d2 Use merge table in the dashboard 2024-02-14 02:37:42 +01:00
Alexey Milovidov
102e64cf1d Merge branch 'master' of github.com:ClickHouse/ClickHouse into dashboards-to-use-merge-tables 2024-02-14 02:20:20 +01:00
Alexey Milovidov
27f115f581 Use merge table in the dashboard 2024-02-14 02:17:23 +01:00
Alexey Milovidov
7ee6b0bd87
Merge pull request #59941 from kitaisreal/register-storage-merge-tree-exception-message-fix 
Register StorageMergeTree exception message fix
 2024-02-14 01:31:00 +01:00
Alexey Milovidov
8b59977639
Merge pull request #59909 from azat/rmt-macros-cleanup 
Petite cleanup around macros and ReplicatedMergeTree
 2024-02-14 00:09:34 +01:00
Kruglov Pavel
1dbfeafb42
Merge branch 'master' into auto-format-detection 2024-02-13 19:08:33 +01:00
Alexander Gololobov
037bc043e6 Removed lambdas 2024-02-13 16:01:17 +01:00
Maksim Kita
4dc6ed3a39 Register StorageMergeTree exception message fix 2024-02-13 17:53:35 +03:00
Smita Kulkarni
d7056d642d Addressed review comments 2024-02-13 13:53:40 +01:00
Smita Kulkarni
3526d7982c Merge branch 'master' into Support_parameterized_view_with_analyzer 2024-02-13 13:08:35 +01:00
Kseniia Sumarokova
bb12fb45c4
Merge pull request #59873 from azat/file-log-shutdown-fix 
Fix long shutdown of FileLog storage
 2024-02-13 12:08:12 +01:00
Nikita Mikhaylov
4b85f6b39a
Revert "Less error prone interface of read buffers" (#59911) 2024-02-13 12:02:46 +01:00
Nikolai Kochetov
666b3d6664
Merge branch 'master' into allow-parallel-replicas-for-join-with-analyzer-2 2024-02-13 11:11:03 +01:00
Azat Khuzhin
58e5d7876b Allow uuid in replica_path if CREATE TABLE explicitly has it 
Right now the query with {uuid} macro without ON CLUSTER fails:

    CREATE TABLE x UUID 'aaaaaaaa-1111-2222-3333-aaaaaaaaaaaa' (key Int) ENGINE = ReplicatedMergeTree('/tables/{database}/{uuid}', 'r1') ORDER BY tuple();

There is a workaround right now to use ATTACH instead of CREATE, but
ATTACH is not CREATE.

CREATE still useful for proper RESTORE without ON CLUSTER.

So this patch allows this syntax, but only if UUID had been explicitly
passed in the query. This looks safe.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-02-13 08:01:09 +01:00
Alexey Milovidov
4dc59051a1
Merge pull request #59897 from ClickHouse/s3queue-fix-uninitialized-value 
s3queue: fix uninitialized value
 2024-02-13 02:44:45 +01:00
Azat Khuzhin
c61ac1d3bc Convert default_replica_path/default_replica_name into server settings 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-02-12 21:05:58 +01:00
kssenii
d008ee725f Add a test 2024-02-12 20:23:21 +01:00
kssenii
0628ae6268 S3 queue fix uninitialized value 2024-02-12 17:13:30 +01:00
Azat Khuzhin
c7c05c9881 Do not pull mutations if pulling replication log had been stopped 
Right now, mutations can be pulled even after:

    SYSTEM STOP PULLING REPLICATION LOG

Since they pulled from two places:
- StorageReplicatedMergeTree::mutationsUpdatingTask()
- ReplicatedMergeTreeQueue::pullLogsToQueue()

And only the last one checks action blocker.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-02-12 16:28:08 +01:00
Nikolai Kochetov
fd2ee19f65 Follow up for #58554. Cleanup. 2024-02-12 13:37:25 +00:00
Nikolai Kochetov
09d3b3c0d0
Merge pull request #58554 from ClickHouse/try-to-always-push-down-prewhere-from-query-plan 
Simplify optimize-push-to-prewhere from query plan
 2024-02-12 12:52:39 +01:00
Azat Khuzhin
67b73c3e49 Fix long shutdown of FileLog storage 
Previously it was possible to wait up to
poll_directory_watch_events_backoff_max (default is 32000) on shutdown,
because it was not possible to stop poll of inotify.

Before (takes 3 seconds):

    2024.02.12 11:27:55.058192 [ 10134 ] {} <Trace> StorageFileLog (file_log): Waiting for cleanup
    2024.02.12 11:27:58.178021 [ 10271 ] {} <Trace> directory_watch: Execution took 7519 ms.

After:

    2024.02.12 11:33:29.722403 [ 15866 ] {} <Trace> StorageFileLog (file_log): Waiting for cleanup
    2024.02.12 11:33:29.722473 [ 15956 ] {} <Trace> directory_watch: Execution took 6399 ms.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-02-12 11:52:09 +01:00
Robert Schulze
06dc9ed074
Merge remote-tracking branch 'ClickHouse/master' into materialize_non_override_past_values 2024-02-11 15:48:55 +00:00
Alexander Gololobov
309db427e7 Do not reinitialize ZooKeeperWithFaultInjection on each chunk 2024-02-11 13:13:49 +01:00
Michael Kolupaev
5424f235a9 Fix StorageURL doing some of the query execution in one thread instead of max_threads 2024-02-11 02:44:18 +00:00
Anton Popov
b4449535c2
Merge pull request #59733 from CurtizJ/check-minmax-index-types 
Validate types of arguments for `minmax` skipping index
 2024-02-09 19:39:59 +01:00
Dmitry Novik
dd8680018f Analyzer: WIP on test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 2024-02-09 16:43:20 +01:00
Nikolai Kochetov
862c6cd79c
Merge pull request #57430 from ClickHouse/non-ready-set-ttl 
Non ready set in TTL WHERE.
 2024-02-09 16:31:26 +01:00
Igor Nikonov
effaaceb26
Merge pull request #59501 from ClickHouse/pr-better-replicas-failover-2 
Parallel replicas: better initial replicas failover (2)
 2024-02-09 15:50:56 +01:00
Dmitry Novik
2e6ca6200e
Merge pull request #59774 from ClickHouse/analyzer-merge-follow-up 
Followup #50894
 2024-02-09 15:05:19 +01:00
Vitaly Baranov
09e727d48b
Merge pull request #56988 from ClickHouse/Azure_backup 
Backup & Restore support for AzureBlobStorage
 2024-02-09 14:45:35 +01:00
Dmitry Novik
8655c11280
Fix typo 2024-02-09 11:45:15 +01:00
Igor Nikonov
84d64a6ed8 Merge remote-tracking branch 'origin/master' into pr-better-replicas-failover-2 2024-02-08 18:03:16 +00:00
kssenii
e55b60e05c Fix 2024-02-08 17:43:16 +01:00
Dmitry Novik
5f75087178 Add comments 2024-02-08 15:48:20 +00:00
Nikolai Kochetov
73d2ff3933
Update MergeTask.cpp 2024-02-08 14:55:35 +01:00
Dmitry Novik
b8f74b5b05
Merge pull request #50894 from ClickHouse/storage-merge-aliases-analyzer 
Analyzer: support aliases and distributed JOINs in StorageMerge
 2024-02-08 11:22:00 +01:00
Igor Nikonov
3fb30a23d9 Merge remote-tracking branch 'origin/master' into pr-better-replicas-failover-2 2024-02-08 09:09:11 +00:00
Alexander Tokmakov
510f90edf8
Merge pull request #59688 from ClickHouse/revert_39507_again 
Temporarily remove a feature that doesn't work
 2024-02-08 02:27:55 +01:00
Anton Popov
9a4dbc843a validate type of arguments for minmax secondary index 2024-02-07 17:52:17 +00:00
Dmitry Novik
813020a1a6 remove redundant logging 2024-02-07 16:44:09 +01:00
Dmitry Novik
ff4729ccc4 Improve replaceTableExpressionAndRemoveJoin function 2024-02-07 16:40:23 +01:00
Smita Kulkarni
e594608c42 Merge branch 'master' into Azure_backup 2024-02-07 13:48:46 +01:00
Nikolai Kochetov
d1902cdba0 Fix some tests. 2024-02-07 12:36:24 +01:00
Nikolai Kochetov
06d112135e Simplify prewhere push down from query plan. Try to always use it. 2024-02-07 12:36:20 +01:00
Nikolai Kochetov
a3d1b3f502 Merge branch 'master' into non-ready-set-ttl 2024-02-07 11:29:04 +00:00
Nikolai Kochetov
c434748fc1
Merge branch 'master' into allow-parallel-replicas-for-join-with-analyzer-2 2024-02-07 12:25:11 +01:00
Raúl Marín
38c8e1b8cc
Merge pull request #59485 from Algunenano/strict_int8 
Strict aliasing for Int8
 2024-02-07 11:33:06 +01:00
Alexander Gololobov
4dcd806dfc
Merge pull request #59545 from ClickHouse/system_zookeeper_stream 
Produce stream of chunks from system.zookeeper instead of accumulating the whole result
 2024-02-07 10:17:38 +01:00
Alexander Tokmakov
77e4b0c8d9 Revert "Merge pull request #59122 from arthurpassos/bring_back_attach_partition_pr" 
This reverts commit ac90637170, reversing
changes made to fe2b5cde05.
 2024-02-07 02:52:43 +01:00
Igor Nikonov
0c00435cea Merge remote-tracking branch 'origin/master' into pr-better-replicas-failover-2 2024-02-07 00:52:50 +00:00
Alexey Milovidov
01c8542462
Merge pull request #59606 from vitlibar/fix-distributed-table-with-const-sharding-key 
Fix distributed table with a constant sharding key
 2024-02-06 22:10:34 +01:00
Raúl Marín
ff63a1cf9f Fix tidy 2024-02-06 20:34:12 +01:00
Dmitry Novik
d558728281
Merge branch 'master' into storage-merge-aliases-analyzer 2024-02-06 20:24:20 +01:00
Dmitry Novik
1abcf26df6 Fix filter expressions 2024-02-06 15:39:52 +00:00
Anton Popov
addbd78341
Merge pull request #59436 from nickitat/perf_introspection_for_inserts 
More perf introspection for `INSERT`-s
 2024-02-06 14:59:32 +01:00
Alexander Gololobov
ca4f46ab41 Check block size in bytes instead of rows 2024-02-06 14:27:15 +01:00
Alexander Gololobov
a26e8b3a9e Fix for filters w/o path column 2024-02-05 22:48:48 +01:00
Vitaly Baranov
86ac7b14d0 Fix Distributed table engine with a constant sharding key. 2024-02-05 21:56:29 +01:00
Igor Nikonov
dab078f7d2 Profile events 2024-02-05 20:21:48 +00:00
Nikolai Kochetov
6b06fcf7fc Fix more tests. 2024-02-05 17:54:15 +00:00
Nikolai Kochetov
15bf263153 Support non global in mode. 2024-02-05 17:05:21 +00:00
Nikolai Kochetov
8692e8f752 Add settings to squash external table blocks. 2024-02-05 17:05:21 +00:00
Nikolai Kochetov
1892318e46 Squash temporary tables. 2024-02-05 17:05:20 +00:00
Nikolai Kochetov
b60228af3f Allow to send a chain of subqueries for parallel replicas with analyzer. 2024-02-05 17:05:20 +00:00
Kseniia Sumarokova
83515e3656
Merge pull request #59543 from nickolaj-jepsen/master 
Add support for NATS credentials file
 2024-02-05 12:20:26 +01:00
Nikita Mikhaylov
6ba35b5459
Make comments for system tables also available in local mode and enforce comments not to be empty (#59493) 
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2024-02-05 11:17:58 +01:00
alesapin
db046c1bce
Merge pull request #59517 from ClickHouse/remove_scarry_message 
Remove a scary message if an error is retryable
 2024-02-04 17:43:06 +01:00
MikhailBurdukov
b90a5b94a8 Move setting to mergetree 2024-02-04 11:24:26 +00:00
Smita Kulkarni
b0994c5fa7 Addressed comments, added test for named collection 2024-02-04 11:28:20 +01:00
Alexander Gololobov
82caaa744a Produce stream of chunks instead of accumulating the whole result 2024-02-03 16:08:41 +01:00
Nickolaj Jepsen
e822ba3a47 Add support for NATS credentials file 2024-02-02 23:34:01 +01:00
Dmitry Novik
05a7d22c24 WIP on calculate aliases only once 2024-02-02 13:45:04 +00:00
Alexander Sapin
d5e3b71590 Remove scarry message if error is retryable 2024-02-02 14:07:37 +01:00
Raúl Marín
87d493533c
Revert "Poco Logger small refactoring" 2024-02-02 12:10:05 +01:00
Alexander Gololobov
095648d7bc
Merge pull request #59388 from ClickHouse/system_zookeeper_retries 
Retry disconnects and expired sessions when reading system.zookeeper
 2024-02-02 11:52:15 +01:00
serxa
1d31ac2a0c add FORGET PARTITION query to remove old parition nodes from ZooKeeper 2024-02-02 10:03:31 +00:00
Alexey Milovidov
15467f2efb
Merge pull request #59451 from ClickHouse/remove-unused-headers 
CLion says these headers are unused
 2024-02-01 21:47:17 +01:00
Nikita Taranov
d73abc17f5 better 2024-02-01 21:42:22 +01:00
Nikita Taranov
698d00dbfe add to MutateTask 2024-02-01 20:43:43 +01:00
Nikita Taranov
76f2ae08b5 collect stats build time 2024-02-01 18:16:09 +01:00
Nikita Taranov
4795149863 print stats once 2024-02-01 17:16:42 +01:00
Kseniia Sumarokova
e029021696
Merge pull request #59446 from ClickHouse/s3-queue-allow-to-add-start-point 
Allow to define a starting point for s3queue ordered mode at creation
 2024-02-01 16:46:24 +01:00
Alexey Milovidov
f52b08070f Merge branch 'master' into remove-unused-headers 2024-02-01 13:42:37 +01:00
kssenii
3a16427e00 Fix test 2024-02-01 12:35:01 +01:00
Maksim Kita
ecbea6f625 Updated implementation 2024-02-01 12:51:50 +03:00
Kseniia Sumarokova
b3418c506e
Update src/Storages/S3Queue/S3QueueSettings.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2024-02-01 10:28:33 +01:00
Dmitry Novik
5b630ed745
Merge pull request #59399 from azat/build/format-fwd-decl 
Forward declaration for PeekableReadBuffer
 2024-02-01 09:15:07 +01:00
Alexey Milovidov
33f6940214 CLion says these headers are unused 2024-02-01 03:50:09 +01:00
kssenii
361b2f107b Allow to define a starting point for s3queue ordered mode 2024-01-31 21:28:18 +01:00
Alexander Gololobov
5947e5b5a8 Use retry settings and fault injection 2024-01-31 16:48:46 +01:00
Dmitry Novik
572d47acc7 Remove logging 2024-01-31 14:43:06 +00:00
Dmitry Novik
0d21004218 WIP on StorageMerge 2024-01-31 13:50:15 +00:00
Kseniia Sumarokova
b20567a055
Merge pull request #59167 from ClickHouse/s3-queue-parallelize-ordered-mode 
S3Queue: allow parallel & disrtibuted processing for ordered mode
 2024-01-31 11:20:41 +01:00
Kseniia Sumarokova
caaafbcedb
Merge pull request #56864 from ClickHouse/broken-projections-better-handling 
Do not consider data part as broken if projection is broken
 2024-01-31 11:17:15 +01:00
Dmitry Novik
ac17ad1e66
Merge pull request #59073 from ClickHouse/fix-analyzer-resolve-distr-id 
Analyzer: Do not resolve remote table id on initiator
 2024-01-31 09:08:30 +01:00
Nikita Taranov
1be8b61c5b add test 2024-01-30 21:52:32 +01:00
Dmitry Novik
722f3db738 Merge remote-tracking branch 'origin/master' into storage-merge-aliases-analyzer 2024-01-30 19:50:01 +00:00
Nikita Taranov
17ab2674f4 impl 2024-01-30 20:35:10 +01:00
Azat Khuzhin
010f1c6cb7 Forward declaration for PeekableReadBuffer 
ReadHelpers.h is very common header and is the root cause of "recompile
everything".

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-30 19:24:19 +01:00
Alexander Gololobov
c348c4e828 Move ZooKeeperRetries.h to Common 2024-01-30 19:02:17 +01:00