Commit Graph

14500 Commits

Author SHA1 Message Date
Azat Khuzhin
ffa82e9297 Fix filtering parts with indexHint for non analyzer
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-09-08 18:17:06 +02:00
Andrey Zvonov
ff4586e663
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-09-08 18:17:06 +02:00
Anton Popov
1aa34c16e3 Merge remote-tracking branch 'upstream/master' into HEAD 2023-09-08 14:00:45 +00:00
Han Fei
ddcb64f39f update docs and refine statements 2023-09-08 02:27:17 +02:00
Anton Popov
67b27643ed fix creash in case of exception in prefetched pool 2023-09-07 16:56:13 +00:00
Petr Vasilev
d0884d3267 Merge branch 'master' of https://github.com/vahpetr/ClickHouse into fix-nats-high-cpu-usage 2023-09-07 18:48:01 +03:00
Petr Vasilev
cb969aa54a Remove empty line 2023-09-07 18:47:31 +03:00
Petr Vasilev
4ef241445f Remove received_mutex and received_cv 2023-09-07 18:44:18 +03:00
Antonio Andelic
fae0e13baa Properly remove added interserver endpoint 2023-09-07 14:34:36 +00:00
robot-ch-test-poll2
f28ad1e136
Merge pull request #54368 from ClickHouse/fix_replacing_final_error
Fix logical error in vertical merge + replacing merge tree + optimize cleanup
2023-09-07 15:45:43 +02:00
Petr Vasilev
c0333263a0 Rollback do_reschedule flag 2023-09-07 15:07:21 +03:00
Petr Vasilev
9c61ceb051 Fix nats high cpu usage 2023-09-07 13:59:32 +03:00
zvonand
c36a3fbd39 renamed setting 2023-09-07 12:00:25 +03:00
Alexey Milovidov
1b30f91e7a
Merge pull request #51304 from ClickHouse/alter-table-add-comment
Add support for ALTER TABLE MODIFY COMMENT
2023-09-07 07:30:57 +03:00
Alexey Gerasimchuck
c2f4dc0f14 Implemented peak_threads_usage 2023-09-07 01:32:39 +00:00
serxa
77697211cd add docs for system.scheduler + small fix 2023-09-06 19:30:39 +00:00
SmitaRKulkarni
f0146c0ff0
Increase retries for test_merge_tree_azure_blob_storage (#54069)
* Increase max_single_download_retries & max_single_read_retries for test_merge_tree_azure_blob_storage

* Updated logs to log exception leading to detaching part

* Updated to catch RequestFailedException azure exception as its the base exception

* Updated isRetryableExcepion to take std::exception_ptr as input

* Fixed style check fails

* Fix build issue & moved NetException & TimeoutException to retryable exceptions

* Fixed style check with catch
2023-09-06 19:57:59 +02:00
Alexander Sapin
4368c00052 Fix logical error in vertical merge + replacing merge tree + optimize cleanup 2023-09-06 18:43:56 +02:00
Kruglov Pavel
5b29640b22
Merge branch 'master' into group-by-constant-keys 2023-09-06 18:06:51 +02:00
robot-ch-test-poll4
8c71582316
Merge pull request #54185 from ClickHouse/analyzer-partition-id-pruning
Fix partition id pruning for analyzer.
2023-09-06 16:53:35 +02:00
robot-ch-test-poll1
9e50e8ff20
Merge pull request #54193 from ClickHouse/fix_replace_partition_with_zero_copy
Fix REPLACE/MOVE PARTITION with zero-copy replication
2023-09-06 16:04:25 +02:00
Alexander Sapin
a176c38876 Fix tests with database replicated 2023-09-06 14:17:46 +02:00
Alexander Sapin
518d4b4357 Merge branch 'master' into alter-table-add-comment 2023-09-06 12:34:04 +02:00
Anton Popov
401496aa1f Apply suggestions from code review
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
2023-09-06 10:27:46 +00:00
robot-ch-test-poll1
d9746b5ea1
Merge pull request #54336 from arenadata/ADQM-1157
Used assert_cast instead of dynamic_cast in ExternalDataSourceCache
2023-09-06 08:28:21 +02:00
robot-clickhouse-ci-1
02339a1f22
Merge pull request #54326 from ClickHouse/fix_segfault_system_zookeeper
Fix segfault in system.zookeeper
2023-09-06 02:20:50 +02:00
Alexey Gerasimchuck
f9edf01280 used assert_cast instead of dynamic_cast 2023-09-05 22:57:40 +00:00
robot-clickhouse-ci-1
7403ee689f
Merge pull request #53877 from vitlibar/add-restore-setting-allow-broken-parts
Add restore setting "restore_broken_parts_as_detached"
2023-09-05 21:28:08 +02:00
Kruglov Pavel
de801ce563
Merge pull request #54293 from ClickHouse/pasha-returned-from-vacation
Code improvement for reading from archives
2023-09-05 21:09:09 +02:00
Alexander Tokmakov
a8489c21da fix segfault in system.zookeeper 2023-09-05 20:23:16 +02:00
robot-ch-test-poll
2e7edf4413
Merge pull request #53638 from arenadata/ADQM-987
Add `backup_log` system table
2023-09-05 17:03:41 +02:00
Antonio Andelic
88930a335c Apply comments 2023-09-05 12:32:07 +00:00
Anton Popov
5da7133c94
Merge pull request #53735 from priera/fix_zk_error_when_doing_replace
37737 fixed internal error when replacing with byte-equal parts
2023-09-05 12:51:14 +02:00
alesapin
095bb37bc4
Merge pull request #54244 from bigo-sg/hdfs-config
Minor improvement, load from config
2023-09-05 12:36:38 +02:00
Kruglov Pavel
fd42f0bac0
Merge pull request #54271 from Avogar/better-count-and-progress-archives
Fix using count from cache and improve progress bar for reading from archives
2023-09-05 12:12:39 +02:00
Antonio Andelic
a51c333c5f Improve code for reading from archives 2023-09-05 08:37:02 +00:00
Victor Krasnov
132ab6067b Merge branch 'master' into ADQM-987 2023-09-05 10:01:44 +03:00
Alexey Milovidov
a59c694e4f
Merge pull request #54225 from Hexta/recreate-secondary-index-system-data-size
Fix system.data_skipping_indices for recreated indices
2023-09-05 01:13:09 +03:00
Alexey Milovidov
1a458d524e
Merge pull request #54260 from ClickHouse/vdimir/enable_allow_remove_stale_moving_parts
Enable allow_remove_stale_moving_parts by default
2023-09-05 01:11:22 +03:00
robot-clickhouse
926c5636dd
Merge pull request #42599 from ClickHouse/build-fuzzer-protocol
libFuzzer: add CI fuzzers build, add tcp protocol fuzzer, fix other fuzzers.
2023-09-04 22:41:54 +02:00
avogar
e9456082c1 Fix using count from cache and improve progress bar for reading from archives 2023-09-04 17:33:59 +00:00
Vitaly Baranov
8060407c70 Add backup setting "check_parts" and restore setting "restore_broken_parts_as_detached". 2023-09-04 18:54:03 +02:00
vdimir
f2ff693a39
Enable allow_remove_stale_moving_parts by default 2023-09-04 15:41:22 +00:00
Alexander Tokmakov
dfcc6f78b3 fix 2023-09-04 16:51:25 +02:00
serxa
aa8e9c3cbc fix style 2023-09-04 14:18:54 +00:00
Han Fei
f60dad0598 Merge branch 'master' into hanfei/statistic 2023-09-04 16:17:07 +02:00
Alexander Tokmakov
ea996b0a00 Merge branch 'master' into fix_replace_partition_with_zero_copy 2023-09-04 15:55:51 +02:00
Han Fei
2330a28f7b support estimate by stat 2023-09-04 15:51:00 +02:00
Sergei Trifonov
258da91b5f
Merge branch 'master' into disks-and-io-sheduler 2023-09-04 15:14:28 +02:00
serxa
b5233efc20 fix system table and scheduler config reloading 2023-09-04 12:35:17 +00:00
Victor Krasnov
78760639d2 Merge branch 'master' into ADQM-987 2023-09-04 13:59:20 +03:00
zhanglistar
a2fd3a4de8 remove unnecessary check for HDFS load config 2023-09-04 17:52:43 +08:00
Alexey Milovidov
864e66cbf7 Fix TOCTOU 2023-09-04 04:06:24 +02:00
Alexey Milovidov
e618dc993e Fix bad code in the system.filesystem_cache 2023-09-04 03:53:18 +02:00
serxa
8fdd118e0c fix build 2023-09-03 22:40:00 +00:00
serxa
adcab639a4 add system.scheduler table for introspection 2023-09-03 21:48:33 +00:00
Alexey Milovidov
86223699be
Merge pull request #54200 from ClickHouse/fix-ipv6-bloom
Fix: allow IPv6 for bloom filter
2023-09-03 23:08:51 +03:00
Artur Malchanau
04d0b0c09a
Fix system.data_skipping_indices for recreated indices
Fix secondary index size in system.data_skipping_indices table in the
case of dropped and added again index.
2023-09-03 20:36:55 +03:00
Kseniia Sumarokova
b3319f7908
Minor changes (#54171) 2023-09-03 15:47:52 +02:00
robot-ch-test-poll2
776f232ec0
Merge pull request #54215 from ClickHouse/remove-metadata-cache
Deprecate metadata cache, part 2
2023-09-03 07:17:26 +02:00
Alexey Milovidov
7cb4d52d92 Deprecate metadata cache, part 2 2023-09-03 03:56:11 +02:00
Alexey Milovidov
282636fd74 S3Queue is experimental 2023-09-03 00:00:37 +02:00
Michael Kolupaev
8505b3c940
Merge pull request #54134 from kssenii/refactor-reading-from-remote-fs
Small refactoring for read from object storage
2023-09-02 11:13:03 -07:00
Yakov Olkhovskiy
a3216c78ef allow IPv6 for bloom filter 2023-09-01 22:44:05 +00:00
Alexander Tokmakov
59d28f3787 fix 2023-09-01 22:40:08 +02:00
Alexander Tokmakov
bba5a1d480 update zero copy locks correctly 2023-09-01 20:10:23 +02:00
Sergei Trifonov
cf5ea46713
Merge pull request #53338 from azat/throttling-fixes
Fix IO throttling during copying whole directories
2023-09-01 18:29:15 +02:00
Alexey Milovidov
f8c2ea986d Update autogenerated version to 23.9.1.1 and contributors 2023-09-01 17:46:49 +02:00
Nikolai Kochetov
46087fd3e5 Refactor. 2023-09-01 15:45:08 +00:00
Victor Krasnov
52cce27e81 Merge branch 'master' into ADQM-987 2023-09-01 18:00:20 +03:00
Nikolai Kochetov
4cbfca78e4 Fixing 01748_partition_id_pruning 2023-09-01 14:29:31 +00:00
Yakov Olkhovskiy
361b21b416 fix fuzzers, cmake refactor, add target fuzzers 2023-09-01 14:20:50 +00:00
robot-clickhouse-ci-2
ae4f6c9683
Merge pull request #54137 from kssenii/s3-queue-better-default-path
Better path for storage `S3Queue` metadata
2023-09-01 16:01:38 +02:00
Anton Popov
033dba4772
Merge pull request #54126 from ClickHouse/CurtizJ-patch-8
Small improvements in `getAlterMutationCommandsForPart`
2023-09-01 14:10:45 +02:00
kssenii
9eb1dfcd12 Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
Nikolai Kochetov
291bd76f1d
Merge pull request #54133 from ClickHouse/fixing-01086_odbc_roundtrip-with-analyzer
Fixing 01086_odbc_roundtrip with analyzer.
2023-09-01 12:54:50 +02:00
Kseniia Sumarokova
c51b6ca60c
Update StorageS3Queue.cpp 2023-08-31 18:29:06 +02:00
kssenii
da185fc8b1 Better paths for s3 queue metadata 2023-08-31 18:16:07 +02:00
Nikolai Kochetov
dc062856b0 Fixing 01086_odbc_roundtrip with analyzer. 2023-08-31 15:23:27 +00:00
Anton Popov
af70fdfcfa mute noisy log message 2023-08-31 15:05:35 +00:00
Igor Nikonov
c3caead859
Merge pull request #52688 from Algunenano/better_zk_retries_on_insert
Improve insert retries on keeper session expiration
2023-08-31 17:00:02 +02:00
pufit
1bc6b40a12
Merge pull request #35102 from tonickkozlov/bloom-filter-index/deduplicate
Prune bloom filter indices based on data cardinality
2023-08-31 10:04:19 -04:00
Anton Kozlov
af21620ab5
Update src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h
Co-authored-by: pufit <pufit@yandex.ru>
2023-08-31 14:06:21 +01:00
Anton Kozlov
d3123da824
Update src/Storages/MergeTree/MergeTreeIndexAggregatorBloomFilter.cpp
Co-authored-by: pufit <pufit@yandex.ru>
2023-08-31 14:05:57 +01:00
Anton Kozlov
3f6ef6b5d9
Update src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h
Co-authored-by: pufit <pufit@yandex.ru>
2023-08-31 14:05:41 +01:00
Anton Kozlov
e6895688d7
Update src/Storages/MergeTree/MergeTreeIndexAggregatorBloomFilter.cpp
Co-authored-by: pufit <pufit@yandex.ru>
2023-08-31 14:05:28 +01:00
Anton Popov
579ec699bf Merge remote-tracking branch 'upstream/master' into HEAD 2023-08-31 12:59:59 +00:00
Anton Popov
77111a229d fix clang-tidy 2023-08-31 12:59:21 +00:00
Raúl Marín
4703c463aa Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-31 14:37:02 +02:00
Andrey Zvonov
83405a8808
Merge branch 'master' into zvonand-globs-small-fix 2023-08-31 08:42:48 +02:00
Antonio Andelic
0148e15aee
Merge pull request #53880 from ClickHouse/archive-improvements-2
Improve schema inference for archives
2023-08-30 21:11:34 +02:00
Alexander Tokmakov
83c5e2fba6
Revert "Fix bug on mutations with subcolumns of type JSON in predicates of UPDATE and DELETE queries." 2023-08-30 12:56:17 +02:00
Antonio Andelic
f406019413 Apply PR comments 2023-08-30 09:26:01 +00:00
Antonio Andelic
ddb58217d4 Merge branch 'master' into archive-improvements-2 2023-08-30 07:43:25 +00:00
Antonio Andelic
36fb7cfbd1
Merge pull request #54012 from ClickHouse/refactor-async-insert-with-dedup
Refactor logic around async insert with deduplication
2023-08-30 09:37:55 +02:00
Azat Khuzhin
51e8b16916 Fix IO throttling during copying whole directories
v2: pass local context during ALTER TABLE MOVE PARTITION TO VOLUME/DISK as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-29 23:13:46 +02:00
Nikita Mikhaylov
97efd14e90
Merge pull request #54009 from ClickHouse/change-istorage-interface
Change IStorage interface by random walk, no goal in particular
2023-08-29 22:22:53 +02:00
Sergei Trifonov
802579f3f1
Merge pull request #49618 from ClickHouse/concurrency-control-controllable
Make concurrency control controllable
2023-08-29 19:44:51 +02:00
Anton Popov
8197580746 fix tests and better interface 2023-08-29 17:23:52 +00:00
Antonio Andelic
e96c665be7 Refactor logic around async insert with dedup 2023-08-29 14:48:43 +00:00
Alexander Tokmakov
17f3754193
Merge pull request #53851 from ClickHouse/distributed_no_flush_on_shutdown
Follow-up to #53501
2023-08-29 16:34:22 +02:00
Nikita Mikhaylov
96cabcae3a Change interface 2023-08-29 16:26:48 +02:00
Anton Popov
7bc5627912 fix typos 2023-08-29 14:21:58 +00:00
Anton Popov
02391eafe1 refactoring of reading from MergeTree tables 2023-08-29 13:40:12 +00:00
Anton Popov
6ce4a82a04
Merge pull request #53677 from VanDarkholme7/mutations_subcolumns
Fix bug on mutations with subcolumns of type JSON in predicates of UPDATE and DELETE queries.
2023-08-29 15:08:00 +02:00
Antonio Andelic
a0dd350e5e Fix assert 2023-08-29 09:59:16 +00:00
Andrey Zvonov
6195390c77
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-08-29 13:46:09 +05:00
János Benjamin Antal
a1a45ee905
Merge pull request #53766 from ilejn/exceptions_kafka_consumers
system.kafka_consumers exception info improvements
2023-08-29 10:22:26 +02:00
Antonio Andelic
8708ebd7fb Correctly check files 2023-08-29 07:40:08 +00:00
Anton Popov
03f5d3feb4 fix checkDataPart 2023-08-28 14:59:03 +00:00
Raúl Marín
174f7a4e63 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-28 16:27:45 +02:00
Alexander Tokmakov
d41eca1dcc rename new method 2023-08-28 16:01:00 +02:00
Antonio Andelic
9b99f25d75 Improve schema inference 2023-08-28 13:11:52 +00:00
Robert Schulze
4c78c08847
Add test 2023-08-28 13:02:24 +00:00
Davit Vardanyan
9190fb2ba3 Fix: Deserialize 2023-08-28 16:18:12 +04:00
Andrey Zvonov
c9b0db9c61
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-08-28 17:15:33 +05:00
Andrey Zvonov
2447eb027c
Update StorageHDFS.cpp 2023-08-28 15:12:44 +03:00
Igor Nikonov
a2686af496
Merge pull request #53862 from ClickHouse/cleanup-parallel-replicas-0
Remove unused parallel replicas coordinator in query info
2023-08-28 13:30:23 +02:00
Antonio Andelic
5a0c2ca108 Merge branch 'master' into archive-improvements-2 2023-08-28 08:34:42 +00:00
VanDarkholme7
bac257743b
Merge branch 'ClickHouse:master' into mutations_subcolumns 2023-08-28 14:10:31 +08:00
Igor Nikonov
7368bf65b3 Remove unused parallel replicas coordinator in query info 2023-08-27 22:12:39 +00:00
Victor Krasnov
87c90767d9 Merge branch 'master' into ADQM-987 2023-08-27 09:14:42 +03:00
alesapin
0e601b856f
Merge pull request #53751 from Algunenano/clang18
Support clang-18 (Wmissing-field-initializers)
2023-08-25 23:27:06 +02:00
Andrey Zvonov
911bf94fbd
Merge branch 'master' into zvonand-globs-small-fix 2023-08-25 23:50:07 +05:00
Victor Krasnov
921f0c8b5d Merge branch 'master' into ADQM-987 2023-08-25 21:20:23 +03:00
Alexander Tokmakov
32efbe77d1
Merge pull request #53811 from ClickHouse/fix_02443_2
Fix a bug in attach partition
2023-08-25 20:16:53 +03:00
Alexander Tokmakov
9ab545e28c do not wait for flush on shutdown 2023-08-25 19:09:10 +02:00
Alexander Gololobov
684d5aab88
Merge pull request #53797 from ClickHouse/add_profile_event_for_lock
Add event for tracking data parts lock
2023-08-25 18:51:49 +02:00
Kseniia Sumarokova
34ac113af6
Merge pull request #53121 from kssenii/fix-cache-logical-error
Fix possible error from cache "Read unexpected size"
2023-08-25 12:16:24 +02:00
Kruglov Pavel
d80be4673b
Merge pull request #53692 from Avogar/cache-count
Cache number of rows in files for count in file/s3/url/hdfs/azure functions
2023-08-25 09:08:52 +02:00
Antonio Andelic
f0e800092f
Merge pull request #53796 from ClickHouse/fix-filtered-url-paths
Fix reading from `url` with all filtered paths
2023-08-25 08:49:33 +02:00
Yakov Olkhovskiy
415a993c91 fix fuzzers build 2023-08-24 23:22:39 +00:00
alesapin
3a289534cb Fxi 2023-08-24 20:48:10 +02:00
Alexander Tokmakov
cbbb81f5cc
Merge pull request #53405 from canhld94/limit_concurent_background_optimize
Limit number of concurrent background partition optimize
2023-08-24 20:49:38 +03:00
Alexander Tokmakov
75d32bfe77
Merge pull request #53750 from ClickHouse/Add_retries_for_azure_retryable_errors
Add retries for azure retryable errors
2023-08-24 20:26:42 +03:00
Alexander Tokmakov
e1d135dc01 fix another bug in attach partition 2023-08-24 19:10:25 +02:00
Raúl Marín
6490213207 Merge remote-tracking branch 'blessed/master' into clang18 2023-08-24 17:35:26 +02:00
Kruglov Pavel
44db5fa992
Merge branch 'master' into cache-count 2023-08-24 17:21:18 +02:00
avogar
c4f8113a78 Fix tests 2023-08-24 15:05:45 +00:00
Robert Schulze
45d924c62c
Merge pull request #53616 from wzb5212/skip_idx_bug_fix
Fix number of dropped granules in EXPLAIN PLAN index=1
2023-08-24 16:46:47 +02:00
Han Fei
10c622fa08 Merge branch 'master' into hanfei/statistic 2023-08-24 15:56:06 +02:00
Antonio Andelic
6eb400c6ae Fix reading from url with all filtered paths 2023-08-24 13:37:11 +00:00
alesapin
415b3826e0 Add event for tracking data parts lock 2023-08-24 15:28:38 +02:00
Ilya Golshtein
f58b4a812d exceptions_kafka_consumers: code review suggestions are addressed 2023-08-24 13:10:21 +00:00
Arthur Passos
2bade7db08
Add global proxy setting (#51749)
* initial impl

* fix env ut

* move ut directory

* make sure no null proxy resolver is returned by ProxyConfigurationResolverProvider

* minor adjustment

* add a few tests, still incomplete

* add proxy support for url table function

* use proxy for select from url as well

* remove optional from return type, just returns empty config

* fix style

* style

* black

* ohg boy

* rm in progress file

* god pls don't let me kill anyone

* ...

* add use_aws guards

* remove hard coded s3 proxy resolver

* add concurrency-mt-unsafe

* aa

* black

* add logging back

* revert change

* imrpove code a bit

* helper functions and separate tests

* for some reason, this env test is not working..

* formatting

* :)

* clangtidy

* lint

* revert some stupid things

* small test adjusmtments

* simplify tests

* rename test

* remove extra line

* freaking style change

* simplify a bit

* fix segfault & remove an extra call

* tightly couple proxy provider with context..

* remove useless include

* rename config prefix parameter

* simplify provider a bit

* organize provider a bit

* add a few comments

* comment out proxy env tests

* fix nullptr in unit tests

* make sure old storage proxy config is properly covered without global context instance

* move a few functions from class to anonymous namespace

* fix no fallback for specific storage conf

* change API to accept http method instead of bool

* implement http/https distinction in listresolver, any still not implemented

* implement http/https distinction in remote resolver

* progress on code, improve tests and add url function working test

* use protcol instead of method for http and https

* small fix

* few more adjustments

* fix style

* black

* move enum to proxyconfiguration

* wip

* fix build

* fix ut

* delete atomicroundrobin class

* remove stale include

* add some tests.. need to spend some more time on the design..

* change design a bit

* progress

* use existing context for tests

* rename aux function and fix ut

* ..

* rename test

* try to simplify tests a bit

* simplify tests a bit more

* attempt to fix tests, accept more than one remote resolver

* use proper log id

* try waiting for resolver

* proper wait logic

* black

* empty

* address a few comments

* refactor tests

* remove old tests

* baclk

* use RAII to set/unset env

* black

* clang tidy

* fix env proxy not respecting any

* use log trace

* fix wrong logic in getRemoteREsolver

* fix wrong logic in getRemoteREsolver

* fix test

* remove unwanted code

* remove ClientConfigurationperRequest and auxilary classes

* remove unwanted code

* remove adapter test

* few adjustments and add test for s3 storage conf  with new proxy settings

* black

* use chassert for context

* Add getenv comment
2023-08-24 16:07:26 +03:00
Antonio Andelic
8e1d38d377 Merge branch 'master' into archive-improvements-2 2023-08-24 13:03:36 +00:00
Han Fei
d80fd7c74c small improvement 2023-08-24 14:20:34 +02:00
Han Fei
c29261e5b4 support estimate by t-digest 2023-08-24 14:10:22 +02:00
Raúl Marín
1c98568df9 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-24 12:22:42 +02:00
Robert Schulze
8c9fade58c
Merge remote-tracking branch 'rschu1ze/master' into skip_idx_bug_fix 2023-08-24 10:17:34 +00:00
Kseniia Sumarokova
a4f39fe134
Merge branch 'master' into fix-cache-logical-error 2023-08-24 12:00:18 +02:00
Alexander Tokmakov
011c7f7064
Revert "Planner prepare filters for analysis" 2023-08-24 12:52:03 +03:00
Kseniia Sumarokova
187706558a
Merge pull request #51622 from kssenii/system-sync-cache
Add `system sync filesystem cache` command
2023-08-24 11:42:32 +02:00
Ilya Golshtein
901240eede exceptions_kafka_consumers: clang-tidy improvement 2023-08-24 08:44:08 +00:00
Ilya Golshtein
ec0c12960a exceptions_kafka_consumers: stylecheck 2023-08-23 23:10:25 +00:00
Ilya Golshtein
e7213ef5b9 exceptions_kafka_consumers: new test, MV exceptions propagated 2023-08-23 22:46:04 +00:00
Kruglov Pavel
685ffc5856
Fix style 2023-08-23 22:51:14 +02:00
Kruglov Pavel
f7e1abd774
Merge branch 'master' into cache-count 2023-08-23 22:31:49 +02:00
Kruglov Pavel
063c986680
Merge pull request #53637 from Avogar/fast-count-from-files
Optimize count from files in most input formats
2023-08-23 22:17:29 +02:00
avogar
93ba73715a Merge branch 'master' of github.com:ClickHouse/ClickHouse into group-by-constant-keys 2023-08-23 18:47:42 +00:00
avogar
894513f6cd Fix tests 2023-08-23 18:43:08 +00:00
Nikolai Kochetov
3a4a7d70a2
Merge pull request #52762 from kitaisreal/planner-prepare-filters-for-analysis
Planner prepare filters for analysis
2023-08-23 18:59:43 +02:00
Anton Kozlov
a5b6a91b54 [bloom filter] Write hashes for bloom filter into a hash set for optimal sizing 2023-08-23 14:17:55 +00:00
Victor Krasnov
8b2342c205 Minor code cleanup: remove some redundant includes of InterpreterAlterQuery.h 2023-08-23 14:16:36 +00:00
Raúl Marín
93dac0c880 Support clang-18 (Wmissing-field-initializers) 2023-08-23 15:53:45 +02:00
Smita Kulkarni
262c39329b Updated to retry loading parts for azure TransportException 2023-08-23 15:50:11 +02:00
Smita Kulkarni
1a074ecf93 Updated to not mark part as broken for azure TransportException in loadDataPart 2023-08-23 15:33:01 +02:00
Smita Kulkarni
7fdf90ccf1 Added azure TransportException to retryable errors 2023-08-23 15:21:37 +02:00
avogar
68e3af56d4 Address comments 2023-08-23 13:19:15 +00:00
Kruglov Pavel
592fa77987
Merge branch 'master' into cache-count 2023-08-23 15:18:02 +02:00
Kruglov Pavel
7e362a2110
Merge branch 'master' into fast-count-from-files 2023-08-23 15:13:20 +02:00
Robert Schulze
bea651e828
Merge pull request #53657 from rschu1ze/cache-improvements
Expose more settings from caches
2023-08-23 14:58:45 +02:00
Robert Schulze
a093f519fc
Merge remote-tracking branch 'rschu1ze/master' into skip_idx_bug_fix 2023-08-23 12:37:46 +00:00
Robert Schulze
5dec6f19f3
Merge pull request #51325 from xinhuitian/fix-annoy-index-update
Do not reset Annoy index during build-up with > 1 mark
2023-08-23 14:31:06 +02:00
robot-ch-test-poll1
c22ffa6195
Merge pull request #53529 from Avogar/filter-files-all-table-functions
Use filter by file/path before reading in url/file/hdfs table functins
2023-08-23 14:21:23 +02:00
Robert Schulze
761268f86a
Merge remote-tracking branch 'rschu1ze/master' into skip_idx_bug_fix 2023-08-23 10:55:04 +00:00
Kruglov Pavel
e193aec583
Merge branch 'master' into fast-count-from-files 2023-08-23 12:15:34 +02:00
kssenii
325e2d3172 Merge remote-tracking branch 'upstream/master' into system-sync-cache 2023-08-23 11:44:19 +02:00
priera
fd51839b97 basic implementation 2023-08-23 11:19:01 +02:00
Antonio Andelic
83d4b819f3 Better support for reading from archives 2023-08-23 08:10:30 +00:00
xinhuitian
920762e66d fix USearch 2023-08-23 10:54:56 +08:00
Tian Xinhui
7cf057e716
Merge branch 'ClickHouse:master' into fix-annoy-index-update 2023-08-23 10:33:14 +08:00
pufit
926533306c
Merge pull request #53668 from ClickHouse/pufit/fix_s3_threads
Limiting the number of parsing threads for the S3 source
2023-08-22 19:09:33 -04:00
Kruglov Pavel
a83a868acd
Update src/Storages/HDFS/StorageHDFS.cpp
Co-authored-by: Dmitry Novik <mrnovikd@gmail.com>
2023-08-22 19:08:04 +02:00
Robert Schulze
d5a7277a59
Better parameter name 2023-08-22 15:43:13 +00:00
pufit
e42da9411b Fix variables names 2023-08-22 11:23:10 -04:00
Raúl Marín
abfd48bcd2 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-22 17:01:22 +02:00
Nikolai Kochetov
5ee9b56040 Do not change the snapshot after at reading from MT 2023-08-22 13:50:51 +00:00
Victor Krasnov
258f9927b4 Merge branch 'master' into ADQM-987 2023-08-22 16:49:32 +03:00
Kruglov Pavel
de960f3c35
Fix style 2023-08-22 15:22:17 +02:00
Kruglov Pavel
a34c9c3ebc
Fix build 2023-08-22 15:21:54 +02:00
Kruglov Pavel
67c5c0203b
Merge branch 'master' into fast-count-from-files 2023-08-22 15:03:48 +02:00
avogar
89b402f4e2 Fix 2023-08-22 13:01:07 +00:00
avogar
7f9e81d504 Clean up 2023-08-22 12:55:00 +00:00
alesapin
b4aaeb1a82 Fix bad log line format 2023-08-22 14:53:35 +02:00
Kruglov Pavel
c0bdd0e00b
Merge branch 'master' into cache-count 2023-08-22 14:42:22 +02:00
avogar
b4145aeddc Cache number of rows in files for count in file/s3/url/hdfs/azure functions 2023-08-22 11:59:59 +00:00
Robert Schulze
f2fb7365e6
Add settings for 1. size_ratio (all caches) and 2. policy (index caches)
1. is an expert-level setting, default is 0.5, applies only to SLRU.

Also, I noticed that we expose cache policy settings for the mark and
the uncompresed cache but not for the index mark and the index
uncompressed cache. Changed that as well, it simplifies the code a bit.
2023-08-22 11:04:29 +00:00
Ilya Golshtein
7cd8237a42 exceptions_kafka_consumers: initial 2023-08-22 10:26:52 +00:00
Robert Schulze
acaa2eee58
Merge pull request #53556 from ClibMouse/feature/big-endian-UUID-correction
Change Big Endian-UUID to work the same as Little Endian-UUID
2023-08-22 11:57:44 +02:00
alesapin
1a5a9fca87
Merge pull request #53605 from irenjj/feat_53538
Add column 'name' to system.clusters as an alias to cluster.
2023-08-22 11:34:21 +02:00
VanDarkholme7
2638cc69d7 fix bug on DELETE and UPDATE with JSON subcolumns 2023-08-22 09:26:17 +00:00
Tian Xinhui
76016d9593
Merge branch 'master' into fix-annoy-index-update 2023-08-22 16:58:29 +08:00
pufit
9d454d9afc Merge branch 'master' into pufit/fix_s3_threads
# Conflicts:
#	src/Storages/StorageS3.cpp
#	src/Storages/StorageS3.h
#	src/Storages/StorageURL.cpp
#	src/Storages/StorageURL.h
2023-08-21 21:32:15 -04:00
pufit
98a701e2c1 Limiting number of parsing threads for S3 source 2023-08-21 21:21:03 -04:00
Robert Schulze
a20ceef398
Merge pull request #53447 from davvard/master
USearch HNSW index for ANN search
2023-08-21 23:34:26 +02:00
Michael Kolupaev
2f4d433e69 Parquet filter pushdown 2023-08-21 14:15:52 -07:00
Nikolai Kochetov
4222ac528d Merge branch 'master' into planner-prepare-filters-for-analysis 2023-08-21 16:48:33 +00:00
Kruglov Pavel
aebbf00a5a
Fix 2023-08-21 17:47:30 +02:00
Anton Popov
3e9a182555 better interfaces for getting of stream name in part 2023-08-21 13:52:25 +00:00
Kruglov Pavel
a2f81bf4ea
Fix typo 2023-08-21 15:20:44 +02:00
Kruglov Pavel
dfdd26ee71
Fix build 2023-08-21 15:04:03 +02:00
Kruglov Pavel
88aee95122
Merge branch 'master' into fast-count-from-files 2023-08-21 14:46:33 +02:00
avogar
584bd57f5c Clean up 2023-08-21 12:43:11 +00:00
avogar
47304bf7aa Optimize count from files in most input formats 2023-08-21 12:30:52 +00:00
Kruglov Pavel
9d9bdde0e3
Merge branch 'master' into filter-files-all-table-functions 2023-08-21 14:26:59 +02:00
Kruglov Pavel
9869196021
Update VirtualColumnUtils.cpp 2023-08-21 14:22:48 +02:00
Antonio Andelic
ed15babddd
Merge pull request #53198 from ClickHouse/improve-archive-reading
Improve reading from archives
2023-08-21 14:11:25 +02:00
Victor Krasnov
a7813260c0 Merge branch 'master' into ADQM-987 2023-08-21 14:45:13 +03:00
Kruglov Pavel
21413636ac
Remove debug logging 2023-08-21 13:30:08 +02:00
Robert Schulze
2c330516b8
Another compile fix 2023-08-21 11:12:39 +00:00
Duc Canh Le
2c07065f81 increase the default value
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
2023-08-21 11:04:24 +00:00
Raúl Marín
f2d3c9c281 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-21 12:14:31 +02:00
Robert Schulze
066ec559ad
Do not reset Annoy index during build-up with > 1 mark 2023-08-21 10:05:43 +00:00
王智博
8bb0f18019 bad size of mark skip idx 2023-08-20 22:57:53 +08:00
Kseniia Sumarokova
404fb0588a
Merge branch 'master' into fix-cache-logical-error 2023-08-20 16:28:13 +02:00
irenjj
a9b65ee929 Add Alias for system.clusters. 2023-08-20 14:47:07 +08:00
irenjj
71459c6d65 Revert 2023-08-20 14:35:59 +08:00
irenjj
e64a1fc26d Add column 'name' to system.clusters as an alias to cluster. 2023-08-20 11:52:14 +08:00
Igor Nikonov
a69ea02418 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-19 21:43:25 +00:00
Kseniia Sumarokova
aeebedfa0d
Merge branch 'master' into system-sync-cache 2023-08-19 13:32:48 +04:00
Kseniia Sumarokova
c0b8d7eddf
Merge pull request #50999 from ilejn/system_kafka_consumers
system.kafka_consumers table to monitor kafka consumers
2023-08-19 13:31:18 +04:00
kssenii
247abe0a2a Merge remote-tracking branch 'upstream/master' into system-sync-cache 2023-08-19 08:43:53 +04:00
Alexey Milovidov
8520a7baca
Merge pull request #53559 from azat/skip-index-set-fix
Fix possible UB in Set skipping index for functions with incorrect args
2023-08-19 02:06:18 +03:00
Alexey Milovidov
6fb24fe74d
Merge pull request #53560 from azat/skip-index-inverted-fix-ub
Fix possible UB in inverted indexes (experimental feature)
2023-08-19 02:06:04 +03:00
Igor Nikonov
aebda10f7c Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-18 23:00:58 +00:00
Alexey Milovidov
7ad8ea78bb
Merge pull request #48990 from incfly/partition
Add partition to the system part_log and merge table.
2023-08-18 21:38:53 +03:00
avogar
8e445b5461 Fixes 2023-08-18 17:49:40 +00:00
Azat Khuzhin
c78b94bed6 Fix possible UB in inverted indexes (experimental feature)
It is possible to have incorrect types there in case of index contains
functions, add a proper check after value had been changed.

<details>

<summary>MSan report</summary>

```
==182==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55658547c59e in DB::MergeTreeConditionInverted::traverseASTEquals() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:551:26
    1 0x556585475566 in DB::MergeTreeConditionInverted::traverseAtomAST() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:438:21
    2 0x55658547e73e in DB::MergeTreeConditionInverted::MergeTreeConditionInverted()::$_1::operator()(DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:228:73
    9 0x556585489e35 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:252:14
    10 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    11 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    12 0x5565854804e6 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::RPNBuilder(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::Block, std::__1::shared_ptr<DB::PreparedSets>, std::__1::function<bool (DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&)> const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:218:9
    13 0x55658546fb87 in DB::MergeTreeConditionInverted::MergeTreeConditionInverted(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>, DB::Block const&, DB::GinFilterParameters const&, DB::ITokenExtractor const*) build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:223:28
    19 0x55658547ceda in DB::MergeTreeIndexInverted::createIndexCondition(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:716:12
    20 0x556587125734 in DB::buildIndexes(std::__1::optional<DB::ReadFromMergeTree::Indexes>&, std::__1::shared_ptr<DB::ActionsDAG>, DB::MergeTreeData const&, std::__1::shared_ptr<DB::Context const> const&, DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&) build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1292:48
    21 0x556587121aaa in DB::ReadFromMergeTree::applyFilters() build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1305:5
    22 0x55658726cf4f in DB::QueryPlanOptimizations::optimizeTreeThirdPass(DB::QueryPlan::Node&, std::__1::list<DB::QueryPlan::Node, std::__1::allocator<DB::QueryPlan::Node>>&) build_docker/./src/Processors/QueryPlan/Optimizations/optimizeTree.cpp:201:38
    23 0x5565870bc489 in DB::QueryPlan::optimize(DB::QueryPlanOptimizationSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:485:5
    24 0x5565870b919a in DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:161:5
    25 0x55658206385a in DB::InterpreterSelectWithUnionQuery::execute() build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379:31
    26 0x556582c15c50 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:751:40
    27 0x556582c09681 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1173:30
    28 0x5565860e7bd7 in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:421:24
    29 0x556586126e9e in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2057:9
```

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-18 18:15:17 +02:00
Azat Khuzhin
1e2b0c8dc4 Fix possible UB in Set skipping index for functions with incorrect args
Usually functions are validated before index analysis, however it is
not the case for parallel replicas, and it requires additional checks
before interpreting function arguments.

<details>

<summary>stack trace</summary>

```
==172==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55cf82aedd6c in DB::ASTFunction* typeid_cast<DB::ASTFunction*, DB::IAST>(DB::IAST*) build_docker/./src/Common/typeid_cast.h:42:73
    1 0x55cf82aedd6c in DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, false>::value(DB::IAST*) build_docker/./src/Common/TypePromotion.h:38:43
    2 0x55cf82aedd6c in std::__1::invoke_result<decltype(&CastHelper<DB::ASTFunction, false>::value), DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, std::is_reference_v<DB::ASTFunction>>, DB::IAST*>::type DB::TypePromotion<DB::IAST>::as<DB::ASTFunction>() build_docker/./src/>
    3 0x55cf82aedd6c in DB::MergeTreeIndexConditionSet::operatorFromAST(std::__1::shared_ptr<DB::IAST>&) build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:602:25
    4 0x55cf82ae5bc1 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:547:9
    5 0x55cf82ae5de6 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:552:13
    6 0x55cf82ae06db in DB::MergeTreeIndexConditionSet::MergeTreeIndexConditionSet() build_docker/.>
    ...
    12 0x55cf82aef09c in DB::MergeTreeIndexSet::createIndexCondition() const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:703:12
    13 0x55cf84951cd4 in DB::buildIndexes()
    14 0x55cf84955ed3 in DB::ReadFromMergeTree::selectRangesToReadImpl()
    15 0x55cf8494caef in DB::ReadFromMergeTree::selectRangesToRead()
    16 0x55cf82a409a9 in DB::MergeTreeDataSelectExecutor::estimateNumMarksToRead()
    17 0x55cf827f728d in DB::MergeTreeData::canUseParallelReplicasBasedOnPKAnalysis()
    18 0x55cf827f627e in DB::MergeTreeData::getQueryProcessingStage()
    19 0x55cf7f2f4969 in DB::InterpreterSelectQuery::getSampleBlockImpl() build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:937:31
    20 0x55cf7f2daa00 in DB::InterpreterSelectQuery::InterpreterSelectQuery()
    24 0x55cf7f520b98 in DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter()
    25 0x55cf7f51b6cd in DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery()
    27 0x55cf7f1d4ea9 in DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) build_docker/./src/Interpreters/InterpreterFactory.cpp:162:16
    28 0x55cf8012e485 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1032:31
    29 0x55cf80121bc1 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1229:30
    30 0x55cf8389295f in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:424:24
    31 0x55cf838d7dfb in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2050:9
```

</details>

CI: https://s3.amazonaws.com/clickhouse-test-reports/53214/d99b10c340909ed4ee2e6edf0921e8a2f8561b0d/fuzzer_astfuzzermsan/report.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-18 18:02:46 +02:00
Austin Kothig
6b42975d33
Change BE-UUID to work the same as LE-UUID. Included high and low getters to provide cleaner code when accessing undertype. 2023-08-18 08:19:46 -07:00
Yakov Olkhovskiy
84cb358c14
Merge pull request #43210 from amosbird/storage_executable_stderr
Add different reactions when external command stderr has data
2023-08-18 11:17:46 -04:00
Alexander Tokmakov
a6684f7a37
Merge pull request #53484 from azat/rmt-queue-cleanup
Remove outdated code in ReplicatedMergeTreeQueue::initialize()
2023-08-18 18:11:33 +03:00
avogar
2d4167c612 FRemove unneded changes 2023-08-18 14:50:07 +00:00
avogar
aac5131859 Fix for no_more_keys 2023-08-18 13:21:16 +00:00
avogar
7fa72f5cdf Optimize group by constant keys 2023-08-18 12:47:26 +00:00
Alexander Gololobov
50061b9cb4
Merge pull request #53452 from ClickHouse/such_much_watch
Deduplicate same watch callbacks registered multiple times
2023-08-18 14:43:03 +02:00
Alexander Tokmakov
0bd378afd9
Merge pull request #53381 from bakwc/bakwc-patch-2
Fixed system.data_skipping_indices for MaterializedMySQL
2023-08-18 14:22:40 +03:00
Duc Canh Le
2d353f8c43 Merge branch 'master' into limit_concurent_background_optimize 2023-08-18 09:25:27 +00:00
Amos Bird
b6de70b042
StorageExecutable check_exit_code default to false 2023-08-18 15:38:47 +08:00