thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 10:22:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
140,865 Commits 9,328 Branches 1,783 Tags 2.1 GiB
aea3090074
Commit Graph

17020 Commits

Author SHA1 Message Date
Alexander Gololobov
be3c498833 Support more that 255 replicas in system table 2024-03-28 18:22:26 +01:00
Yarik Briukhovetskyi
e2b0fde0f0
Remove unused setting 2024-03-28 17:22:14 +01:00
kssenii
19b90d8348 Resolve conflicts after merge, add system.filesystem_cache_settings 2024-03-28 16:28:59 +01:00
Nikita Taranov
7e5b5420cb impl 2024-03-28 15:08:23 +00:00
vdimir
4b990e732c
Fix style 2024-03-28 15:22:26 +01:00
Kseniia Sumarokova
0244e0851a
Merge pull request #61954 from ClickHouse/follow-up-broken-projections 
Follow up to #60452
 2024-03-28 14:23:28 +01:00
Azat Khuzhin
f4c5be3bb4 Fix CREATE TABLE w/o columns definition for ReplicatedMergeTree 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-28 14:06:52 +01:00
zhongyuankai
f1ae99b113 Support ALTER MODIFY SETTING for Memory tables 2024-03-28 19:55:22 +08:00
Anton Popov
56b3c95d54 Merge remote-tracking branch 'upstream/master' into HEAD 2024-03-27 22:38:00 +00:00
Han Fei
b824e94e8c
Merge pull request #61936 from hanfei1991/hanfei/refreshable_view_race 
fix a race in refreshable view
 2024-03-27 22:55:22 +01:00
Alexey Milovidov
795026af72
Merge pull request #61911 from ClickHouse/jamesweb 
Make userspace page cache work with 'web' disks
 2024-03-28 00:18:27 +03:00
Arthur Passos
5561f0567b Merge branch 'master' into forbid_attach_table_duplicated_replica_path 2024-03-27 17:51:02 -03:00
Han Fei
f3616f67aa Merge branch 'master' into hanfei/refreshable_view_race 2024-03-27 21:33:59 +01:00
Han Fei
e849c48e21 add a test 2024-03-27 18:56:13 +01:00
Alexey Milovidov
4250c5fb19 Update autogenerated version to 24.4.1.1 and contributors 2024-03-27 18:48:14 +01:00
Aleksandr Musorin
f9d81bc774 refactore: improve reading several configurations for kafka 
Simplify and do some refactoring for kafka client settings.

Allows to set up separate
settings for consumer and producer like:

```
<consumer>
    ...
</consumer>

<producer>
    <kafka_topic>
        <name>topic_name</name>
        ...
    </kafka_topic>
</producer>
```

Moreover, this fixes warnings from kafka client like:
`Configuration property session.timeout.ms is a consumer property and
will be ignored by this producer instance`
 2024-03-27 14:53:33 +01:00
János Benjamin Antal
2c5c589a88
Merge pull request #61779 from ClickHouse/revert-61774-revert-61479-61220_fix_read_system_parts 
Revert "Revert "Fix bug when reading system.parts using UUID (issue 61220).""
 2024-03-27 14:27:32 +01:00
Igor Nikonov
15a0f5381f
Merge pull request #59176 from azat/dist/readonly-replicas 
Add ability to skip read-only replicas for INSERT into Distributed engine
 2024-03-27 13:19:12 +01:00
Azat Khuzhin
c1a44f887e Check for "SYSTEM STOP MERGES" primarily for MERGE_PARTS/MUTATE_PART 
Since after it will obtain part information (getPartIfExists()), that
will try to acquire DataPartsLock.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-27 12:52:29 +01:00
János Benjamin Antal
cf5517a099 Cancel merges during move/replace partition 2024-03-27 11:31:41 +00:00
vdimir
128c51c6fc
Fix storage join loading order 2024-03-27 11:27:30 +00:00
Han Fei
af2e86d72d make TSAN happy 2024-03-27 10:21:16 +01:00
Michael Kolupaev
d5da446d21 Tidy 2024-03-27 02:32:15 +00:00
kssenii
0a5ce69880 Review fix 2024-03-26 22:02:50 +01:00
Han Fei
2110bfd278 address comment 2024-03-26 18:53:54 +01:00
Han Fei
97fca6682e fix logical-error when undoing quorum insert transaction 2024-03-26 18:48:25 +01:00
kssenii
a777045dd3 Fix and more tests for broken projections handling 2024-03-26 18:47:48 +01:00
Nikita Mikhaylov
1b828ab6f6 Fix for ASync 2024-03-26 16:05:43 +00:00
Vitaly Baranov
83e006e372
Merge pull request #61734 from vitlibar/fix-columns-after-alter-table-modify-query 
Fix columns after executing MODIFY QUERY for a materialized view with internal table
 2024-03-26 15:35:22 +01:00
Han Fei
73deb9a24c fix a race in refreshable view 2024-03-26 14:20:54 +01:00
Alexander Gololobov
4e94510e32 Finish chunk if the size exceeds max_block_size 2024-03-26 12:49:19 +01:00
Alexander Gololobov
2bf1d118c7 Stream rows when reading from system.replicas 2024-03-26 12:49:19 +01:00
Arthur Passos
2ed53e309b
Reset part level upon attach from disk on MergeTree (#61536) 
* hackish change to check how CI reacts

* remove unnecessary renameTo

* introduce separate metho

* remove unused arg

* rename meth od

* add simple test

* add current database

* set mutations sync

* Update tests/queries/0_stateless/03013_test_part_level_is_reset_attach_from_disk_mt.sql

---------

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <benjamin.antal@clickhouse.com>
 2024-03-26 11:44:55 +01:00
Azat Khuzhin
c7b919b19e Replace distributed_insert_replicas_preferences with distributed_insert_skip_read_only_replicas 
v2: fix test
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-26 11:24:57 +01:00
Azat Khuzhin
fe613ddd06 Convert distributed_insert_replicas_preferences into enum 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-26 11:24:14 +01:00
Azat Khuzhin
eb334d717a Remove superior check for empty pools 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-26 11:23:21 +01:00
Azat Khuzhin
dbdf080108 Remove insert argument for ConnectionPoolWithFailover 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-26 11:22:38 +01:00
Azat Khuzhin
b9469e2729 Do not try to INSERT into readonly replicas for Distributed engine 
Sometimes replica may be readonly for a long time, and this will make
some INSERT queries fail, but it does not make sense to INSERT into
readonly replica, so let's ignore them.

But note, that this will require to extend TableStatus (not extend, but
introduce new version), that will have is_readonly field.

Also before background INSERT into Distributed does not uses
getManyChecked() which means that they do not request TableStatus
packet, while now they would, though this is minor (just a note).

v2: Add a note about max_replica_delay_for_distributed_queries for INSERT
v3: Skip read-only replicas for async INSERT into Distributed
v4: Remove extra @insert parameter for ConnectionPool*::get*
    It make sense only when the table name had passed --
    ConnectionPoolWithFailover::getManyChecked()
v5: rebase on top LoggerPtr
v6: rebase
v7: rebase
v8: move TryResult::is_readonly into the end

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-26 11:21:38 +01:00
Azat Khuzhin
b2945a5ad4 Mark Replicated table as readonly in case of static disk (read-only/write-once) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-26 11:19:10 +01:00
Nikolai Kochetov
2f382ac8a8
Merge pull request #61787 from ClickHouse/fix-virtual-columns-storage-merge-analyzer 
Fix more tests with virtual columns in StorageMerge.
 2024-03-26 11:16:25 +01:00
Kseniia Sumarokova
fd8e34a0ba
Merge pull request #60452 from ClickHouse/revert-refert-of-broken-projections-handling 
Revert "Revert "Merge pull request #56864 from ClickHouse/broken-projections-better-handling""
 2024-03-26 10:34:48 +01:00
pufit
cad66f3e68
Fix ALTER QUERY MODIFY SQL SECURITY (#61480) 2024-03-25 20:03:02 -04:00
Yakov Olkhovskiy
6c4cd355e6
Merge pull request #61511 from jewelzqiu/existing-count 
Add docs for lwd-aware merge settings
 2024-03-25 16:59:58 -04:00
Alexander Tokmakov
87dea7885f
Merge pull request #61675 from ClickHouse/replace_range_sucks 
Fix actual_part_name for REPLACE_RANGE (`Entry actual part isn't empty yet`)
 2024-03-25 18:59:49 +01:00
Alexander Tokmakov
071a8ff95f less unformatted exceptions 2024-03-25 18:55:46 +01:00
Arthur Passos
6c56c0c89e Do not allow table to be attached if there already is an active replica path 2024-03-25 11:19:12 -03:00
kssenii
38b15fb2a3 Fix 02494_zero_copy_and_projection_and_mutation_work_together.sql with shared merge tree 2024-03-25 14:59:37 +01:00
Vitaly Baranov
12a7f6afff Fix columns of a materialized views after MODIFY QUERY. 2024-03-25 14:50:36 +01:00
alesapin
d4063e378e
Merge pull request #61554 from ClickHouse/fix_wait_for_mutation_done 
Wait for done mutation with more logs and asserts
 2024-03-25 14:27:34 +01:00
kssenii
e3d2fb28cf Merge remote-tracking branch 'origin/master' into revert-refert-of-broken-projections-handling 2024-03-25 13:54:39 +01:00
Nikolai Kochetov
75150033c8 Cleanup 2024-03-25 12:45:20 +00:00
Nikolai Kochetov
6d5f2cd7b5 Cleanup 2024-03-25 12:44:28 +00:00
Yarik Briukhovetskyi
20a45b4073
Merge pull request #61564 from liuneng1994/optimize_in_single_value 
New analyzer pass to optimize in single value
 2024-03-25 12:53:01 +01:00
Nikolai Kochetov
abe78e1a7b Merge branch 'master' into fix-virtual-columns-storage-merge-analyzer 2024-03-25 10:43:23 +00:00
Azat Khuzhin
911fab7824 Fix possible connections data-race for distributed_foreground_insert/distributed_background_insert_batch 
Previously connection will marked as not in use earlier than
RemoteInserter, while it still may use it and this will lead to crashes,
like in [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/61832/d0cd8d0d7f74ff251d52e1871cba57d26a51873b/stateless_tests_flaky_check__asan_.html

I was able to reproduce the crash locally by running the test
03030_system_flush_distributed_settings in parallel.

Fixes: #45491
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-25 11:25:11 +01:00
János Benjamin Antal
80a7cda59a Fix style 2024-03-25 09:21:45 +00:00
Alexey Milovidov
8c6a6f2b71
Merge pull request #61832 from azat/dist/flush-settings 
Add ability to override initial INSERT SETTINGS via SYSTEM FLUSH DISTRIBUTED
 2024-03-25 03:09:01 +03:00
Azat Khuzhin
35722be8ec Add ability to override initial INSERT SETTINGS via SYSTEM FLUSH DISTRIBUTED 
This is useful to overwrite settings in the initial query during force
flush with SYSTEM FLUSH DISTRIBUTED.

For example imagine that your server is out of queries
(max_max_concurrent_queries_for_all_users had been reached), but you
want to flush the distributed table anyway, after this patch you can
use something like this:

    SYSTEM FLUSH DISTRIBUTED dist SETTINGS max_concurrent_queries_for_all_users=1000000

And also fix flush_on_detach for SYSTEM FLUSH DISTRIBUTED, it should
ignore flush_on_detach.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-24 21:32:34 +01:00
Alexey Milovidov
9210712f4f
Merge pull request #60112 from Unalian/feat-59377 
[feature]: allow to attach parts from a different disk
 2024-03-24 23:12:43 +03:00
Alexey Milovidov
717603802d
Merge pull request #61785 from danipozo/azure-use-managed-identity-for-backups 
Use managed identity for backups IO in Azure Blob Storage
 2024-03-24 20:31:12 +03:00
Alexey Milovidov
8ddfe9a48b
Merge pull request #61665 from ClickHouse/build-improvements-magic-enum-template-instantiation 
Build improvements by removing magic enum from header and apply some explicit template instantiation
 2024-03-24 19:31:46 +03:00
Alexey Milovidov
93beef8bb5
Merge pull request #61399 from azat/order-by-SimpleAggregateFunction 
Forbid SimpleAggregateFunction in ORDER BY of MergeTree tables
 2024-03-24 19:31:18 +03:00
alesapin
49f007af60 Merge remote-tracking branch 'origin/master' into fix_wait_for_mutation_done 2024-03-24 13:54:39 +01:00
Alexey Milovidov
8936528372 Merge branch 'master' into order-by-SimpleAggregateFunction 2024-03-24 02:16:57 +01:00
Alexey Milovidov
aeefba57b5
Update StorageAzureBlob.cpp 2024-03-23 22:49:23 +03:00
Alexey Milovidov
13b4a2838c
Update StorageReplicatedMergeTree.cpp 2024-03-23 21:26:23 +03:00
Alexey Milovidov
25201919a5 Fix errors 2024-03-23 19:23:01 +01:00
Alexey Milovidov
a5dc6241bb
Update StorageMaterializedView.cpp 2024-03-23 21:10:42 +03:00
Alexey Milovidov
7c189ee94a
Merge branch 'master' into add-materialized-view-query-check 2024-03-23 21:07:58 +03:00
Alexey Milovidov
056c8ce021
Merge pull request #59965 from nickitat/express_support 
S3Express support
 2024-03-23 20:52:44 +03:00
Alexey Milovidov
a56c113c83 Fix for clang-19 2024-03-22 22:12:59 +01:00
Nikolai Kochetov
b2ae11d6b6 Fixing tests. 2024-03-22 17:49:01 +00:00
Dani Pozo
2736b4ef64 Use managed identity for backups IO in Azure Blob Storage 
Also adds option to prevent ClickHouse from trying to create a non-existing container, which requires
a role assignment at the storage account level.
 2024-03-22 17:58:07 +01:00
Nikolai Kochetov
cf91d5dcbc FIx 2024-03-22 16:57:42 +00:00
Nikolai Kochetov
5207526910 Fix more tests with wirtual columns in StorageMerge. 2024-03-22 16:55:01 +00:00
Alexander Tokmakov
9082a018a5
Merge pull request #59082 from ClickHouse/fix_race_async_inserts_queue 
Fix race on `Context::async_insert_queue`
 2024-03-22 16:06:13 +01:00
János Benjamin Antal
cff56a5c73 Unify column names 2024-03-22 15:05:10 +00:00
János Benjamin Antal
6028d007c3
Revert "Revert "Fix bug when reading system.parts using UUID (issue 61220)." …" 
This reverts commit 8c49c9bf9e.
 2024-03-22 15:52:54 +01:00
Anton Popov
18bc53264f fix reading without marks 2024-03-22 14:34:07 +00:00
János Benjamin Antal
8c49c9bf9e
Revert "Fix bug when reading system.parts using UUID (issue 61220)." (#61774) 2024-03-22 15:20:01 +01:00
Anton Popov
887a9ef184 Merge remote-tracking branch 'upstream/master' into HEAD 2024-03-22 13:37:46 +00:00
Alexey Milovidov
037c910c35
Merge pull request #61535 from CurtizJ/refactor-compact-parts-reading 
Refactoring of reading from compact parts
 2024-03-22 16:33:59 +03:00
Nikolai Kochetov
3ed30959fa
Merge pull request #61666 from ClickHouse/fix-not-ready-set-system-parts 
Fix Non-ready set for system.parts.
 2024-03-22 11:49:26 +01:00
robot-ch-test-poll
03a4b5fb58
Merge pull request #61053 from ClickHouse/async-loader-waiters-limit 
Separate limits on number of waiting and executing queries
 2024-03-22 03:15:25 +01:00
Zhuo Qiu
6a95163a24
Merge branch 'ClickHouse:master' into existing-count 2024-03-22 09:51:27 +08:00
Alexey Milovidov
96a45fc9ec
Merge pull request #61119 from ilejn/update_cppkafka 
update cppkafka to v0.4.1
 2024-03-22 01:19:42 +03:00
Alexey Milovidov
12e35ad1ff
Merge pull request #61616 from nickitat/02908_many_requests_again 
Try to fix `02908_many_requests_to_system_replicas` again
 2024-03-22 01:15:50 +03:00
Kseniia Sumarokova
0c824dab17
Merge pull request #60585 from aalexfvk/traverse_shadow_remote_data_paths 
Traverse shadow directory for system.remote_data_paths
 2024-03-21 19:02:36 +01:00
Vitaly Baranov
1acc0ebe79
Merge pull request #61356 from vitlibar/fix-replace-dictionary 
Fix CREATE OR REPLACE DICTIONARY
 2024-03-21 16:45:06 +01:00
divanik
9dbe7beef8
Merge pull request #59390 from divanik/divanik/generate_series_function 
Add step to the generating numbers table functions
 2024-03-21 16:27:25 +01:00
Dmitry Novik
e4209218d4
Merge pull request #60497 from ClickHouse/analyzer-distr-json 
Analyzer: Support different ObjectJSON on shards
 2024-03-21 16:25:30 +01:00
János Benjamin Antal
1944162088
Merge pull request #61479 from wudanzy/61220_fix_read_system_parts 
Fix bug when reading system.parts using UUID (issue 61220).
 2024-03-21 14:43:29 +01:00
Nikolai Kochetov
e828acd0e5 Review fixes. 2024-03-21 13:17:15 +00:00
Alexander Tokmakov
81fd904b54 I hate REPLACE_RANGE and actual_part_name 2024-03-21 13:02:50 +01:00
Sergei Trifonov
816a6b81c4
Merge pull request #61545 from wiledusc/fix-typo-legacy-max-level 
Fix typo in variable name use_leagcy_max_level
 2024-03-21 12:31:00 +01:00
divanik
cc7a4182e3 Merge branch 'master' of github.com:ClickHouse/ClickHouse into divanik/generate_series_function 2024-03-21 11:27:48 +00:00
Antonio Andelic
c4856e326b
Merge pull request #60225 from azat/dist/parallel-flush 
Parallel flush of pending INSERT blocks of Distributed engine
 2024-03-21 12:25:59 +01:00
serxa
c31b958f88 Merge branch 'master' into async-loader-waiters-limit 2024-03-21 10:58:22 +00:00
unashi
adc964568c [update] Merge the on same disk and on another disk scenarios into cloneAndLoadDataPart; remove the try catch; instead of obtaining the copied destination hard disk through traversal, the destination hard disk is obtained through reservation. 2024-03-21 15:44:27 +08:00
liuneng
ada4384deb fix ut failed 2024-03-21 10:38:51 +08:00
Alexey Milovidov
97b7daef84
Merge pull request #61604 from Algunenano/nested_includes 
Reduce header dependencies
 2024-03-21 02:02:17 +03:00
János Benjamin Antal
0b54b34b5b Avoid instantiation of CacheBase's ctor in header file 2024-03-20 17:00:36 +00:00
János Benjamin Antal
9e8c731e1e Remove some inclusion of magic enum 2024-03-20 17:00:36 +00:00
Nikolai Kochetov
a7ff6d63c3 Fix Non-ready set for system.parts. 2024-03-20 16:48:24 +00:00
Raúl Marín
e6256fa380 Fix build 2024-03-20 15:43:09 +01:00
alesapin
a6051f8038
Merge pull request #61620 from ClickHouse/fix-part-log-with-async-with-dedup 
Fix `system.part_log` for async insert with deduplication
 2024-03-20 15:36:21 +01:00
divanik
52b2220dc2 Resolved issues 2024-03-20 13:20:39 +00:00
Nikita Taranov
22f6bb3645
Merge branch 'master' into debug_info 2024-03-20 14:11:34 +01:00
Raúl Marín
c98634ad8c Merge remote-tracking branch 'blessed/master' into nested_includes 2024-03-20 14:08:10 +01:00
Raúl Marín
c380de864a
Merge pull request #61606 from Algunenano/magic_trash 
Remove some magic_enum from headers
 2024-03-20 14:03:11 +01:00
William Schoeffel
34e3eb223e
Merge branch 'ClickHouse:master' into fix-typo-legacy-max-level 2024-03-20 09:45:37 -03:00
divanik
8ec2dd31c7 Merge branch 'master' of github.com:ClickHouse/ClickHouse into divanik/generate_series_function 2024-03-20 10:54:51 +00:00
vdimir
7ba425a76d
Merge pull request #60182 from ClickHouse/vdimir/analyzer/join_using_parent_scope 
Analyzer: support join using column from select list
 2024-03-20 10:59:31 +01:00
vdimir
9a43f7941b
Merge pull request #61456 from kitaisreal/temporary-data-use-temporary-files-codec-setting 
Temporary data use temporary_files_codec setting
 2024-03-20 10:54:17 +01:00
Kseniia Sumarokova
05516112ca
Merge pull request #61537 from ClickHouse/s3queue-disallow-sharded-processing 
Disallow s3queue sharded mode
 2024-03-20 10:53:51 +01:00
alesapin
af17145c82 Merge remote-tracking branch 'origin/master' into fix_wait_for_mutation_done 2024-03-20 10:47:19 +01:00
Dmitry Novik
10bf164aa7 Cleanup #ci_set_analyzer 2024-03-20 10:47:15 +01:00
Nikolai Kochetov
134215e3c3
Merge pull request #61550 from ClickHouse/fix-01083_expressions_in_engine_arguments-analyzer 
Fixing 01584_distributed_buffer_cannot_find_column with analyzer.
 2024-03-20 09:01:29 +01:00
Antonio Andelic
2312eb886f Fix part log for async insert with deduplication 2024-03-20 08:53:26 +01:00
Nikita Taranov
13ff65474a impl 2024-03-19 21:09:03 +00:00
Azat Khuzhin
11d0b0a9fa Forbid SimpleAggregateFunction in ORDER BY of MergeTree tables 
Like AggregateFunction is forbidden, but they are forbidden because they
are not comparable.

New setting (allow_suspicious_primary_key) had been added for backard
compatiblity (turned OFF by default).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-19 21:09:30 +01:00
Alexey Milovidov
99a81c848e
Merge pull request #60659 from azat/rmt-part_log-fix 
Fix missed entries in system.part_log in case of fetch preferred over merges/mutations
 2024-03-19 22:52:05 +03:00
Dmitry Novik
221b74c77f Fix #ci_set_analyzer 2024-03-19 19:44:15 +01:00
Alexander Tokmakov
5d7e5e5a72 Merge branch 'master' into fix_race_async_inserts_queue 2024-03-19 19:39:25 +01:00
Dmitry Novik
302179c81d Merge remote-tracking branch 'origin/master' into analyzer-distr-json 2024-03-19 18:56:04 +01:00
alesapin
f89d73d683
Merge pull request #60079 from ClickHouse/fix_max_insert_delayed_streams_for_parallel_write 
Make max_insert_delayed_streams_for_parallel_write actually work
 2024-03-19 17:52:00 +01:00
Raúl Marín
c6f0a434c0 Remove some magic_enum from headers 2024-03-19 17:40:27 +01:00
Kseniia Sumarokova
79bbcbdf1e
Update StorageS3Queue.cpp 2024-03-19 17:20:31 +01:00
Maksim Kita
ec6c276a92 Temporary data use temporary_files_codec setting 2024-03-19 19:07:45 +03:00
Raúl Marín
de855ca917 Reduce header dependencies 2024-03-19 17:04:29 +01:00
Ilya Golshtein
939554ec0a contrib: update cppkafka to v0.4.1@9c5ea0e3 2024-03-19 14:54:00 +00:00
Dmitry Novik
a8885e361e
Merge branch 'master' into vdimir/analyzer/join_using_parent_scope 2024-03-19 15:51:53 +01:00
Azat Khuzhin
8b1df937a5 Use scheduleFromThreadPool for parallel processing in distributed storage 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-19 13:20:51 +01:00
alesapin
9b14d17d7b Fxi 2024-03-19 12:58:36 +01:00
János Benjamin Antal
9cd7e6155c
Merge pull request #60031 from azat/merge_tree_mutations_cleanup-fix 
Fix finished_mutations_to_keep=0 for MergeTree (as docs says 0 is to keep everything)
 2024-03-19 12:55:46 +01:00
János Benjamin Antal
922df7d60e
Merge pull request #61003 from amosbird/less-header-1 
Replace some header includes with forward declarations
 2024-03-19 12:14:06 +01:00
alesapin
667aade4e3 Fix build 2024-03-19 12:12:35 +01:00
alesapin
c90ec72fde Merge branch 'master' into fix_wait_for_mutation_done 2024-03-19 12:10:07 +01:00
Azat Khuzhin
929dc6fa12 Attach to query for threads in Distributed engine background ops 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-19 06:58:38 -04:00
Azat Khuzhin
eb75926e50 Parallel flush of pending INSERT blocks of Distributed engine 
Parallelism will work only if you have multi disk policy for table (like
everything in Distributed engine right now).

This will work for DETACH/server shutdown and SYSTEM FLUSH DISTRIBUTED

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-19 06:58:38 -04:00
Han Fei
75e94438e0
Update src/Storages/StorageReplicatedMergeTree.cpp 2024-03-19 10:21:19 +01:00
Dan Wu
411fb050d3
Merge branch 'ClickHouse:master' into 61220_fix_read_system_parts 2024-03-19 11:16:17 +08:00
Vitaly Baranov
306f642d97 Update dictionary's comment in the same safe manner too. 2024-03-19 02:56:33 +01:00
robot-ch-test-poll3
e82cf5d5ba
Merge pull request #61528 from ClickHouse/revert-61518-revert-61298-analyzer-merge-virtual-columns 
Revert "Revert "Analyzer: Fix virtual columns in StorageMerge""
 2024-03-18 23:51:55 +01:00
Robert Schulze
0daaeeab78
Merge pull request #61406 from rschu1ze/enable-clang-tidy-in-headers 
Enable clang-tidy in headers
 2024-03-18 21:43:26 +01:00
Anton Popov
5ac15574b7 do not load marks on merges 2024-03-18 20:28:10 +00:00
Alexander Sapin
a3a549bc24 Wait for update of value in memory and in zookeeper 2024-03-18 21:03:00 +01:00
Alexander Sapin
82da2d56e9 Add chassert 2024-03-18 20:58:15 +01:00
Alexander Sapin
1264b5337e More diagnostics 2024-03-18 20:57:14 +01:00
Alexander Sapin
083d0776ed Wait for done mutation properly 2024-03-18 20:47:31 +01:00
Nikolai Kochetov
e97443ddd1
Update StorageBuffer.cpp 2024-03-18 20:15:34 +01:00
Nikolai Kochetov
c99a874305
Update StorageValues.cpp 2024-03-18 20:13:52 +01:00
Kirill Nikiforov
ebaee8e5f6
Add support for ATTACH PARTITION ALL (#61107) 2024-03-18 15:00:27 -04:00
Nikolai Kochetov
3def01f1f6 Fixing 01083_expressions_in_engine_arguments with analyzer. 2024-03-18 18:53:01 +00:00
William Schoeffel
4f30ac72a1 fix typo in variable name legacy 2024-03-18 15:18:47 -03:00
Nikita Taranov
c1fb34da0d
Merge branch 'master' into debug_info 2024-03-18 17:54:31 +01:00
Nikita Taranov
b0131a278b impl 2024-03-18 16:48:53 +00:00
Arthur Passos
c67c04b89a fix typo 2024-03-18 13:44:36 -03:00
kssenii
b814fabcd3 Disallow s3queue sharded mode 2024-03-18 17:22:38 +01:00
Anton Popov
13b27a8d36 refactoring of reading from compact parts 2024-03-18 16:13:02 +00:00
divanik
265d91747f Remove log debug 2024-03-18 16:03:12 +00:00
Nikita Mikhaylov
15896ecaf6
Add table system.keywords [Part 1] (#51808) 2024-03-18 16:58:57 +01:00
Anton Popov
efe5373f2b refactoring of reading from compact parts 2024-03-18 15:57:58 +00:00
Kseniia Sumarokova
d4895c2e52
Merge pull request #61324 from ClickHouse/add-zk-retries 
Add zookeeper retries for exists check in forcefullyRemoveBrokenOutdatedPartFromZooKeeper
 2024-03-18 16:17:37 +01:00
Vitaly Baranov
08d64f9e11 Get rid of modifying a shared AbstractConfiguration (between StorageDictionary and ExternalDictionariesLoader). 
Always create a new configuration when we need to modify it.
 2024-03-18 14:43:38 +01:00
Dmitry Novik
15e19206b4 Merge remote-tracking branch 'origin/master' into analyzer-distr-json-copy 2024-03-18 14:24:51 +01:00
Dmitry Novik
a65f027cf7
Revert "Revert "Analyzer: Fix virtual columns in StorageMerge"" 2024-03-18 14:00:20 +01:00
Yarik Briukhovetskyi
5e5a39d039
Merge pull request #61467 from wiledusc/remove-duplicated-check-containing-part-empty 
Remove duplicated check containing_part.empty()
 2024-03-18 13:44:25 +01:00
divanik
c5d35bb187 Merge branch 'master' of github.com:ClickHouse/ClickHouse into divanik/generate_series_function 2024-03-18 12:26:43 +00:00
Yarik Briukhovetskyi
c85d410cda
Merge pull request #60612 from JakeBamrah/master 
[59558] Add size cap to Memory tables
 2024-03-18 13:21:56 +01:00
Robert Schulze
0dff126942
Merge remote-tracking branch 'rschu1ze/master' into enable-clang-tidy-in-headers 2024-03-18 11:34:13 +00:00
alesapin
b01526aeb2
Merge pull request #61267 from CurtizJ/better-memory-usage-inserts 
Free memory earlier in inserts
 2024-03-18 12:30:42 +01:00
vdimir
e27e6fb2f5
Analyzer: support join using column from select list 2024-03-18 11:11:00 +00:00
Kseniia Sumarokova
6d2e48d085
Merge branch 'master' into add-zk-retries 2024-03-18 11:24:36 +01:00
Antonio Andelic
6549a96ced
Revert "Analyzer: Fix virtual columns in StorageMerge" 2024-03-18 10:38:28 +01:00
János Benjamin Antal
4b4a03c0f1
Merge branch 'master' into less-header-1 2024-03-18 10:25:42 +01:00
Robert Schulze
c17536c3b3
Enable clang-tidy in headers 2024-03-18 08:00:09 +00:00
Zhuo Qiu
bf85f7f368 add docs & minor fix for lwd-aware merge settings 2024-03-18 11:02:02 +08:00
Alexey Milovidov
01136bbc3b Limit backtracking in parser 2024-03-17 19:54:45 +01:00
Kseniia Sumarokova
d4205afe82
Merge branch 'master' into add-zk-retries 2024-03-17 17:56:32 +01:00
Alexey Milovidov
994bf8765c Merge branch 'master' of github.com:ClickHouse/ClickHouse into split-cast-overload-resolver 2024-03-17 14:20:02 +01:00
Alexey Milovidov
cb8d859453
Merge branch 'master' into analyzer-distr-json 2024-03-17 15:15:06 +03:00
Alexey Milovidov
bf98386a33
Merge pull request #61298 from ClickHouse/analyzer-merge-virtual-columns 
Analyzer: Fix virtual columns in StorageMerge
 2024-03-17 15:05:23 +03:00
Vitaly Baranov
4fe1c9a78f Get rid of checking the "last modification time" when the configuration is generated 
to allow ExternalLoader::reloadConfig() work for generated configurations too.
 2024-03-17 12:10:57 +01:00
Vitaly Baranov
41d8621d97 Fix replacing dictionaries: StorageID is now updated immediately for every dictionary during CREATE OR REPLACE 
even if the dictionary is not loaded yet or loading right now.
 2024-03-17 12:10:52 +01:00
Alexey Milovidov
9be38c064a Merge branch 'master' into split-cast-overload-resolver 2024-03-16 14:07:59 +01:00
Dan Wu
adaf262dc0 Fix bug when reading system.parts using UUID (issue 61220). 
During the predicates pushing down of system.parts table, there is a
logic to use that predicates to filter on the storages. This makes sense
if the predicate is on the `database`, `table`, `engine`. But it can be
problem if the predicate is about `uuid`, because storage UUID is
apparently different from parts UUID.

Rename the column name from `uuid` to `storage_uuid` fixed this.
 2024-03-16 02:02:21 +00:00
William Schoeffel
4dbf3ee1bb fix: remove duplicated check containing part 2024-03-15 14:05:24 -03:00
Yakov Olkhovskiy
a31f551bc7
Merge pull request #58223 from jewelzqiu/existing-count 
Consider lightweight deleted rows when selecting parts to merge
 2024-03-15 12:42:35 -04:00
Kruglov Pavel
d045ab150e
Merge pull request #61362 from ClickHouse/revert-61359-revert-61234-Avogar-patch-2 
Revert "Revert "Fix usage of session_token in S3 engine""
 2024-03-15 16:00:05 +01:00
Dmitry Novik
061cd5a53d Fixup #ci_set_analyzer 2024-03-15 14:22:46 +00:00
Anton Popov
6ad2f95837 Merge remote-tracking branch 'upstream/master' into HEAD 2024-03-15 13:52:20 +00:00
Dmitry Novik
1cdd280c9f
Merge pull request #61392 from ClickHouse/analyzer-fix-context-in-merge 
Analyzer: Fix planner context for subquery in StorageMerge
 2024-03-15 14:05:28 +01:00
Sergei Trifonov
273d221781
Merge pull request #60558 from ClickHouse/revert-60253-revert-59032-cpu-scheduler-and-cc 
Revert "Revert "Support resource request canceling""
 2024-03-15 13:24:50 +01:00
divanik
e57446b6f5 Resolved conflicts 2024-03-15 11:57:59 +00:00
Antonio Andelic
0ba400d065
Merge pull request #61380 from ClickHouse/fix-heap-use-after-free-alias-merge 
Fix `heap-use-after-free` for Merge table with alias
 2024-03-15 11:15:02 +01:00
Zhuo Qiu
7f97b11ce6 only read _row_exists column when necessary 2024-03-15 15:48:20 +08:00
Alexey Milovidov
c9fb8ef54b Merge branch 'master' into split-cast-overload-resolver 2024-03-15 07:09:51 +01:00
Nikolai Kochetov
1cae77997a Try revert stage back 2024-03-14 21:34:25 +00:00
avogar
e08eaebc99 Add sanity check for poll_max_batch_size FileLog setting to avoid big untracked allocations 2024-03-14 20:36:18 +00:00
Anton Popov
edaf2c8023 Merge remote-tracking branch 'upstream/master' into HEAD 2024-03-14 17:06:22 +00:00
Dmitry Novik
90669ef487 Merge remote-tracking branch 'origin/master' into analyzer-distr-json 2024-03-14 17:43:26 +01:00
Dmitry Novik
0b3b734c9b Analyzer: Fix planner context for subquery in StorageMerge 2024-03-14 17:37:49 +01:00
Kseniia Sumarokova
ceeb926fa8
Merge branch 'master' into add-zk-retries 2024-03-14 17:27:40 +01:00
kssenii
b778b01408 Add settings 2024-03-14 16:39:32 +01:00
Dmitry Novik
d558c4dcb7 Cleanup 2024-03-14 16:17:58 +01:00
Dmitry Novik
e145115ef1 Fixup 2024-03-14 16:15:06 +01:00
Antonio Andelic
026ac4deb1 Fix heap-use-after-free for Merge table with alias 2024-03-14 14:20:22 +00:00
Robert Schulze
06d917fcf0
Merge pull request #61150 from rschu1ze/clang-tidy-list 
Some clang-tidy fixes
 2024-03-14 14:54:41 +01:00
vdimir
0b97f3ac16
Merge pull request #61320 from ClickHouse/vdimir/fix_rabbitmq_logical_error 
Fix logical error in RabbitMQ storage with MATERIALIZED columns
 2024-03-14 13:28:05 +01:00
Alexey Milovidov
8094dd88d9 Merge branch 'master' into split-cast-overload-resolver 2024-03-14 13:08:37 +01:00
Alexey Milovidov
1a41aa4ac7 Merge branch 'master' of github.com:ClickHouse/ClickHouse into split-cast-overload-resolver 2024-03-14 13:06:23 +01:00
divanik
9c77aa9510 Merge branch 'master' of github.com:ClickHouse/ClickHouse into divanik/generate_series_function 2024-03-14 11:41:14 +00:00
Kruglov Pavel
56220b5105
Revert "Revert "Fix usage of session_token in S3 engine"" 2024-03-14 12:23:32 +01:00
Antonio Andelic
c6f0b2a5a4
Revert "Fix usage of session_token in S3 engine" 2024-03-14 11:57:03 +01:00
Robert Schulze
3486b4a6e3
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-list 2024-03-14 10:44:20 +00:00
jsc0218
b3b46ea708 Merge remote-tracking branch 'origin/master' into TableEngineGrant_version2 2024-03-14 00:36:29 +00:00
kssenii
208a91a2f7 Add retries exists check in loadOutdatedDataParts 2024-03-13 19:03:29 +01:00
Jake Bamrah
5e66cd7154 fix settings seg-fault and expand test case coverage 2024-03-13 18:47:13 +01:00
Yarik Briukhovetskyi
4aab1170ef fix style 2024-03-13 18:47:13 +01:00
Yarik Briukhovetskyi
77223d682d reload tests 2024-03-13 18:47:13 +01:00
Jake Bamrah
c73e299820 add size cap to memory tables 2024-03-13 18:47:13 +01:00
Kruglov Pavel
843e2dc968
Merge pull request #60586 from Avogar/fix-filter-virtual-columns 
Fix reading from MergeTree with non-deterministic functions in filter
 2024-03-13 18:22:12 +01:00
vdimir
b2e067b3da
Fix logical error in RabbitMQ storage with MATERIALIZED columns 2024-03-13 16:41:35 +00:00
Kruglov Pavel
dd0caaf9b4
Merge pull request #61234 from ClickHouse/Avogar-patch-2 
Fix usage of session_token in S3 engine
 2024-03-13 17:05:39 +01:00
divanik
cf639bab44 Fix attach bug 2024-03-13 13:37:26 +00:00
Dmitry Novik
80723134d6 Fix fast test #ci_set_analyzer 2024-03-13 14:00:57 +01:00
Dmitry Novik
ffdb84b8df Fix style 2024-03-13 13:38:23 +01:00
Dmitry Novik
1a47682c12 Analyzer: Fix virtual columns in StorageMerge #ci_set_analyzer 2024-03-13 13:08:12 +01:00
Dmitry Novik
2e49140de7 Small progress 2024-03-13 11:37:42 +01:00
Dmitry Novik
5ef241cc85 WIP on virtual columns in StorageMerge 2024-03-13 11:30:11 +01:00
unashi
da06ba432e [fix] merge and resolve problems 2024-03-13 12:09:35 +08:00
Alexey Milovidov
146e401851 Merge branch 'master' into split-cast-overload-resolver 2024-03-13 04:28:24 +01:00
unashi
c78224c2ff Merge branch 'master' into feat-59377 2024-03-13 11:24:00 +08:00
divanik
3c2514bc77 Correct small issues 2024-03-12 18:33:32 +00:00
Anton Popov
f35987928b free memory earlier in inserts 2024-03-12 18:05:14 +00:00
Anton Popov
fc648439f3
Merge branch 'master' into add-virtual-columns 2024-03-12 18:00:12 +01:00
divanik
8f44201f92 Preliminary commit 2024-03-12 16:22:33 +00:00
Azat Khuzhin
fb0ad3eb15 Disable retries for replicated fetches (they are handled by the engine itself) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-12 17:08:42 +01:00
Azat Khuzhin
df44ed173c Fix missed entries in system.part_log in case of fetch preferred over merges/mutations 
The NEED_PREPARE do not call part_log_writer() before.

v2:
- Add a test for missed entries in system.part_log for merge
- Fix part_log_writer
- Add a test for missed entries in system.part_log for mutate

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-12 17:08:42 +01:00
Kseniia Sumarokova
3257bfa918
Merge branch 'master' into traverse_shadow_remote_data_paths 2024-03-12 15:18:52 +01:00
Robert Schulze
1b1685e3f1
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-list 2024-03-12 13:02:59 +00:00
Kruglov Pavel
d431276045
Fix usage of session_token in S3 2024-03-12 12:43:48 +01:00
serxa
defed92331 do nothing in waitForOutdatedPartsToBeLoaded() if loading is not required 2024-03-12 11:38:27 +00:00
Kseniia Sumarokova
61543edc8b
Merge branch 'master' into fix-usage-of-plain-metadata 2024-03-12 11:53:00 +01:00
vdimir
439fe42f88
Merge pull request #61013 from ClickHouse/vdimir/fix_astrename_clone 
Fix ASTRenameQuery::clone
 2024-03-12 11:13:28 +01:00
Azat Khuzhin
6c74ab18c0 Do not clean mutations for MergeTree on DROP PART (to match ReplicatedMergeTree) 
Even if this should be done, only related mutations should be cleaned,
not all.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-12 10:39:41 +01:00
Azat Khuzhin
f559767d22 Fix finished_mutations_to_keep=0 for MergeTree (as docs says 0 is to keep everything) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-03-12 10:39:41 +01:00
unashi
c1b94b2170 [debug] fast test again again 2024-03-12 15:33:43 +08:00
Zhuo Qiu
05969a39f3 resolve conflicts 2024-03-12 14:45:25 +08:00