thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
127,340 Commits 9,326 Branches 1,783 Tags 2.1 GiB
be0f0b6879
Commit Graph

14771 Commits

Author SHA1 Message Date
Robert Schulze
f3bf8388b6
Incorporate feedback 2023-09-20 10:00:00 +00:00
SmitaRKulkarni
d8adf05de2
Added a new column _block_number (#47532) 
Added a new virtual column _block_number which is persisted on merges when allow_experimental_block_number_column is enabled
 2023-09-20 11:31:12 +02:00
Azat Khuzhin
d9a634eb0f Fix filtering parts with indexHint for non analyzer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit ffa82e9297)
 2023-09-20 11:29:35 +02:00
pufit
34aecc0bf3 Adjusting num_streams by expected work in StorageS3 2023-09-19 23:05:48 -04:00
slvrtrn
636fc506aa Replace information_schema view columns aliases with uppercase 2023-09-19 23:42:40 +02:00
Igor Nikonov
e28f4e819c
Merge branch 'master' into parallel-replicas-not-enough-replicas 2023-09-19 23:05:24 +02:00
Azat Khuzhin
c439c4bca2
Revert "Fix filtering parts with indexHint for non analyzer" 2023-09-19 21:39:21 +02:00
Han Fei
eb86c113e5
Merge pull request #54184 from priera/fix_mv_insertion_when_duplicated_parts 
Inserting only non-duplicate chunks in MV
 2023-09-19 20:41:45 +02:00
Kruglov Pavel
d555fdffa5
Merge pull request #54449 from azat/parts-prune-indexHint 
Fix filtering parts with indexHint for non analyzer
 2023-09-19 19:15:41 +02:00
slvrtrn
e4e9727572 Update information_schema docs 2023-09-19 14:14:17 +02:00
alesapin
9dcc41cf83 Fxi 2023-09-19 13:47:32 +02:00
alesapin
28b71163b4 Merge branch 'master' into better_exception_in_check_data_part 2023-09-19 13:45:56 +02:00
alesapin
7b1cf9d912 Revert "Better check for exceptions in checkData of non replicated merge tree" 
This reverts commit 363ee7cae5.
 2023-09-19 13:45:34 +02:00
alesapin
363ee7cae5 Better check for exceptions in checkData of non replicated merge tree 2023-09-19 13:41:55 +02:00
priera
b92e37d2df Fixing PR comments 2023-09-19 12:15:01 +02:00
priera
94d737264c inserting non-duplicate chunks 2023-09-19 12:15:00 +02:00
serxa
311db94640 default max_burst equals to max_speed as in Throttler 2023-09-19 10:10:21 +00:00
Maksim Kita
c7ddbab9bc Updated implementation 2023-09-19 13:06:19 +03:00
flynn
c3d12ce0e1 update comment 2023-09-19 08:18:37 +00:00
Anton Popov
72f2bfd166 fix build 2023-09-19 05:35:57 +00:00
VanDarkholme7
783b48b730
Merge branch 'ClickHouse:master' into missing_subcolumns 2023-09-19 10:33:06 +08:00
Anton Popov
624782bd8f Merge remote-tracking branch 'upstream/master' into HEAD 2023-09-18 23:13:20 +00:00
Anton Popov
d94c1b1ec9 add check for collisions in filenames 2023-09-18 22:27:04 +00:00
slvrtrn
d91a656dd4 Revert schemata.sql format 2023-09-18 22:33:16 +02:00
slvrtrn
ad7b707658 Update information_schema.tables sql and docs 2023-09-18 22:30:27 +02:00
Robert Schulze
097217628e
Merge remote-tracking branch 'rschu1ze/master' into is_deterministic 2023-09-18 20:17:17 +00:00
slvrtrn
9a0d07c682 Update information_schema docs 2023-09-18 22:14:53 +02:00
slvrtrn
54f902f0b7 Adjust information_schema for Tableau Online 2023-09-18 19:50:05 +02:00
Sema Checherinda
e7550523c8
Merge pull request #54651 from CheSema/limit_backoff_timeout 
limit the delay before next try in S3
 2023-09-18 19:41:26 +02:00
Robert Schulze
4373ed0f98
Added field "is_deterministic" to system.functions 2023-09-18 17:27:53 +00:00
Alexander Sapin
326eaa9bc5 Better exception message in checkDataPart 2023-09-18 18:26:53 +02:00
Igor Nikonov
047d214436 Merge remote-tracking branch 'origin/master' into parallel-replicas-not-enough-replicas 2023-09-18 15:29:56 +00:00
Alexey Milovidov
e4d47539ab
Merge pull request #54477 from azat/view-sampling 
Support SAMPLE BY for VIEW
 2023-09-18 14:00:52 +03:00
Kruglov Pavel
9c888ea42b
Merge pull request #53549 from Avogar/group-by-constant-keys 
Optimize group by constant keys
 2023-09-18 12:12:40 +02:00
Yakov Olkhovskiy
711876dfa8
Merge pull request #54696 from ClickHouse/fix-match-keycondition 
Fix: avoid using regex match, possibly containing alternation, as a key condition.
 2023-09-17 22:45:13 -04:00
Robert Schulze
de4f22e20a
Typo 2023-09-17 15:27:30 +00:00
Robert Schulze
9f009cccd5
Incorporate review feedback 2023-09-17 15:22:51 +00:00
Robert Schulze
f5e8028bb1
Merge pull request #54642 from rschu1ze/broken-re2st 
Remove broken lockless variant of re2
 2023-09-17 15:30:57 +02:00
robot-clickhouse-ci-1
a878d104d7
Merge pull request #52839 from zvonand/zvonand-globs-small-fix 
Fix directory permissions for multi-directory globs. Follow-up #50559
 2023-09-16 20:52:30 +02:00
robot-ch-test-poll1
a2ed5c2cae
Merge pull request #54174 from ClickHouse/vdimir/fix_hostname_const 
Fix hostname and co result constness in new analyzer
 2023-09-16 19:37:08 +02:00
VanDarkholme7
cf8c614fed Fix bug on DELETE and UPDATE with json subcolumns 2023-09-16 13:04:19 +00:00
robot-ch-test-poll4
5fb8e46967
Merge pull request #54635 from Avogar/cluster-func-names-with-spaces 
Fix schema inference for *Cluster functions for column names with spaces
 2023-09-15 20:36:33 +02:00
Yakov Olkhovskiy
f52e87f474 fix 2023-09-15 18:30:26 +00:00
Yakov Olkhovskiy
c2d46286e7 we can't process regex with alternation here - bypass 2023-09-15 18:21:38 +00:00
Bhavna Jindal
dc99de3c44 Log engine Mark file to read and write in little Endian for s390x compatibility 2023-09-15 08:59:40 -07:00
Maksim Kita
29e4352c17 Updated implementation 2023-09-15 17:04:24 +03:00
Maksim Kita
7a7697059c Fixed tests 2023-09-15 17:04:24 +03:00
Maksim Kita
32cad222bf KeyCondition IN function support different types 2023-09-15 17:04:24 +03:00
Kruglov Pavel
2075f9c667
Merge branch 'master' into group-by-constant-keys 2023-09-15 15:10:08 +02:00
kssenii
6846fe3c58 Fxi 2023-09-15 14:21:08 +02:00
Andrey Zvonov
ffbed02f14
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-09-15 14:18:07 +03:00
avogar
e647346b36 Better 2023-09-15 11:11:04 +00:00
Kruglov Pavel
1bc29696be
Merge pull request #54576 from HarryLeeIBM/hlee-s390x-rand-gen 
Fix UInt256 and IPv4 random data generation on s390x
 2023-09-15 11:42:47 +02:00
Robert Schulze
945179be46
Annoy: Fix LOGICAL_ERROR with default values #52258 2023-09-14 19:23:32 +00:00
Robert Schulze
c422a8f0dc
Cosmetics 2023-09-14 18:05:20 +00:00
Robert Schulze
f0eadd4b22
Merge pull request #54103 from ClickHouse/ustweaks 
Small usearch index improvements: metrics and configurable internal data type
 2023-09-14 19:58:53 +02:00
Sema Checherinda
d9e15c00c9 limit the delay before next try in S3 2023-09-14 19:45:07 +02:00
vdimir
1aa18e0eb6
Analyzer: Remove constants from header in StorageDistributed 2023-09-14 16:44:18 +00:00
kssenii
342755d35e Better 2023-09-14 18:41:31 +02:00
Robert Schulze
7b378dbad3
Remove broken lockless variant of re2 2023-09-14 16:40:42 +00:00
Alexander Tokmakov
52de64c2bd remove unneeded code 2023-09-14 18:30:12 +02:00
kssenii
afcb0b2b9a Merge remote-tracking branch 'upstream/master' into s3-queue-fixes 2023-09-14 17:16:29 +02:00
Mikhail f. Shiryaev
bf962a8674
Merge pull request #54517 from HarryLeeIBM/hlee-s390x-jemalloc-bins 
Fix endian issue in jemalloc_bins for s390x
 2023-09-14 15:43:50 +02:00
avogar
f240243c86 Fix schema inference for *Cluster functions for column names with spaces 2023-09-14 13:34:11 +00:00
Michael Kolupaev
8997464867
Small usearch index improvements: metrics and f16 2023-09-14 11:24:47 +00:00
serxa
09d4bf6930 add IO scheduling node for bandwidth throttling 2023-09-14 08:51:47 +00:00
Andrey Zvonov
ff16ddbbb3
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-09-14 11:31:49 +03:00
robot-clickhouse-ci-2
e111d4abd3
Merge pull request #54610 from CurtizJ/fix-after-refactoring 
Fix reading of virtual columns in reverse order
 2023-09-14 03:38:27 +02:00
Anton Popov
df2d1ed638 fix reading of virtual columns in reverse order 2023-09-13 21:09:49 +00:00
Tim Windelschmidt
7979090333 use env to find bash 2023-09-13 18:57:11 +02:00
Robert Schulze
59cded9d84
Merge pull request #54563 from rschu1ze/issue-54541 
Fix AST fuzzer crash in MergeTreeIndex{FullText|Inverted}
 2023-09-13 16:29:06 +02:00
Alexander Tokmakov
a365557d9e
Merge pull request #54550 from ClickHouse/fix_zero_copy_garbage 
Fix zero copy garbage
 2023-09-13 15:02:05 +02:00
HarryLeeIBM
735039673a Fix UInt256 and IPv4 random data generation on s390x 2023-09-13 04:55:02 -07:00
alesapin
e8055ba509
Merge pull request #54549 from ClickHouse/fix_redundant_checksums_doesnt_match_error 
Fix rare case of CHECKSUM_DOESNT_MATCH error
 2023-09-13 13:02:50 +02:00
Robert Schulze
8c7a256632
Fix full text index fix 2023-09-13 07:57:48 +00:00
Robert Schulze
e69c1c2f9a
Fix inverted index fix 2023-09-13 07:56:35 +00:00
Robert Schulze
28068f11e1
Fix #54541 2023-09-12 21:23:23 +00:00
alesapin
18ab92eb4f Fix style 2023-09-12 21:57:46 +02:00
alesapin
e17a824fec Fxi 2023-09-12 21:42:32 +02:00
alesapin
6ec65d6be7 Make function slightly more readable 2023-09-12 21:41:05 +02:00
Alexander Tokmakov
46fae52c29 fix zero copy garbage 2023-09-12 20:01:46 +02:00
alesapin
f3729da64b Fix rare case of CHECKSUM_DOESNT_MATCH error 2023-09-12 19:55:56 +02:00
Igor Nikonov
7b3f32b95a
Merge pull request #54520 from ClickHouse/pr-cleanup 
Parallel replicas: cleanup unused params
 2023-09-12 19:48:18 +02:00
robot-clickhouse
1c8ee76ba2
Merge pull request #54513 from Avogar/formats-with-names-no-header 
Fix possible parsing error in WithNames formats with disabled input_format_with_names_use_header
 2023-09-12 17:58:03 +02:00
Anton Popov
38e0689b3b
Merge pull request #54512 from CurtizJ/fix-use-after-free 
Fix use-after-free in `MergeTreePrefetchedReadPool`
 2023-09-12 16:51:12 +02:00
Dmitry Novik
7edc0a30a3
Merge pull request #54335 from arenadata/ADQM-1109 
Added peak_threads_usage to query_log table
 2023-09-12 16:03:27 +02:00
Dmitry Novik
f730a003ea
Merge pull request #54467 from ClickHouse/drop-lost-part-log 
Add a log message on replicated table drop
 2023-09-12 15:59:18 +02:00
Igor Nikonov
1287f68745 Handle clusterAllReplicas/remote cases to avoid unnecessary logging 2023-09-12 12:52:29 +00:00
Igor Nikonov
3acc26001a Merge remote-tracking branch 'origin/master' into pr-cleanup 2023-09-12 10:38:06 +00:00
Alexey Milovidov
e9daff6c2f
Merge pull request #54506 from azat/system.detached_parts-modification_time 
Add modification_time into system.detached_parts
 2023-09-12 04:53:59 +03:00
Alexey Milovidov
44a9dbd2b7
Update StorageSystemDetachedParts.cpp 2023-09-12 04:53:51 +03:00
Sema Checherinda
0ea0799118
Merge pull request #54430 from CheSema/remove-recursive-with-projection 
reproduce and fix the bug in removeSharedRecursive
 2023-09-12 00:18:55 +02:00
Igor Nikonov
d5ea047ab8 Parallel replicas: cleanup unused params 2023-09-11 21:52:40 +00:00
HarryLeeIBM
26b17f88d5 Add comment for the fix 2023-09-11 13:52:58 -07:00
HarryLeeIBM
0a81389359 Fix endian issue in jemalloc_bins for s390x 2023-09-11 12:14:01 -07:00
Azat Khuzhin
8cd6153c83 Add modification_time into system.detached_parts 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-11 20:29:55 +02:00
Dmitry Novik
b12d1219f2 Merge remote-tracking branch 'origin/master' into drop-lost-part-log 2023-09-11 17:05:05 +00:00
Dmitry Novik
b69a90d61f Change log message level 2023-09-11 14:57:25 +00:00
avogar
2d8f33bfa2 Fix parsing error in WithNames formats while reading subset of columns with disabled input_format_with_names_use_header 2023-09-11 14:55:37 +00:00
Anton Popov
ec37e93461 fix use-after-free in MergeTreePrefetchedReadPool 2023-09-11 14:55:21 +00:00
Nikolai Kochetov
903c966cc8
Merge branch 'master' into planner-prepare-filters-for-analysis-2 2023-09-11 16:14:03 +02:00
Sergei Trifonov
08bad4d440
Merge pull request #47009 from ClickHouse/disks-and-io-sheduler 2023-09-11 07:22:14 +02:00
robot-ch-test-poll
6f3374d684
Merge pull request #54481 from ucasfl/hints 
Refactor IHints
 2023-09-10 15:46:36 +02:00
Azat Khuzhin
5535c7462d Support SAMPLE BY for VIEW 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-10 08:24:46 +02:00
flynn
6ec783a746 Refactor IHints 2023-09-10 03:48:12 +00:00
Igor Nikonov
d5ffff93a7
Merge pull request #54468 from ClickHouse/distributed-cleanup-select-info 
Cleanup: unnecessary SelectQueryInfo usage around distributed
 2023-09-09 21:47:36 +02:00
Sema Checherinda
1298dba0c6 fix the bug, add useful logs 2023-09-09 15:54:28 +02:00
Igor Nikonov
2fdc700da2 Style 2023-09-09 13:33:18 +00:00
Igor Nikonov
a12a375a1d Style check 2023-09-09 12:51:34 +00:00
Igor Nikonov
fff2a2d81c Fix style 2023-09-09 08:29:33 +00:00
Igor Nikonov
5470a5b60f Fix style 2023-09-09 06:58:18 +00:00
Mingliang Pan
4d05b57c21
Merge branch 'ClickHouse:master' into master 2023-09-09 11:12:45 +08:00
Igor Nikonov
7a396139df Cleanup: unnecessary SelectQueryInfo usage around distributed 2023-09-08 21:53:38 +00:00
Dmitry Novik
da69a9fda0 Add a log message on replicated table drop 2023-09-08 21:41:42 +00:00
Azat Khuzhin
ffa82e9297 Fix filtering parts with indexHint for non analyzer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-08 18:17:06 +02:00
Andrey Zvonov
ff4586e663
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-09-08 18:17:06 +02:00
kssenii
220a67eca7 Better 2023-09-08 16:38:05 +02:00
Anton Popov
1aa34c16e3 Merge remote-tracking branch 'upstream/master' into HEAD 2023-09-08 14:00:45 +00:00
kssenii
7338b560a8 S3Queue fixes 2023-09-07 19:17:21 +02:00
Anton Popov
67b27643ed fix creash in case of exception in prefetched pool 2023-09-07 16:56:13 +00:00
Petr Vasilev
d0884d3267 Merge branch 'master' of https://github.com/vahpetr/ClickHouse into fix-nats-high-cpu-usage 2023-09-07 18:48:01 +03:00
Petr Vasilev
cb969aa54a Remove empty line 2023-09-07 18:47:31 +03:00
Petr Vasilev
4ef241445f Remove received_mutex and received_cv 2023-09-07 18:44:18 +03:00
Antonio Andelic
fae0e13baa Properly remove added interserver endpoint 2023-09-07 14:34:36 +00:00
robot-ch-test-poll2
f28ad1e136
Merge pull request #54368 from ClickHouse/fix_replacing_final_error 
Fix logical error in vertical merge + replacing merge tree + optimize cleanup
 2023-09-07 15:45:43 +02:00
Petr Vasilev
c0333263a0 Rollback do_reschedule flag 2023-09-07 15:07:21 +03:00
Petr Vasilev
9c61ceb051 Fix nats high cpu usage 2023-09-07 13:59:32 +03:00
zvonand
c36a3fbd39 renamed setting 2023-09-07 12:00:25 +03:00
Alexey Milovidov
1b30f91e7a
Merge pull request #51304 from ClickHouse/alter-table-add-comment 
Add support for ALTER TABLE MODIFY COMMENT
 2023-09-07 07:30:57 +03:00
Alexey Gerasimchuck
c2f4dc0f14 Implemented peak_threads_usage 2023-09-07 01:32:39 +00:00
serxa
77697211cd add docs for system.scheduler + small fix 2023-09-06 19:30:39 +00:00
SmitaRKulkarni
f0146c0ff0
Increase retries for test_merge_tree_azure_blob_storage (#54069) 
* Increase max_single_download_retries & max_single_read_retries for test_merge_tree_azure_blob_storage

* Updated logs to log exception leading to detaching part

* Updated to catch RequestFailedException azure exception as its the base exception

* Updated isRetryableExcepion to take std::exception_ptr as input

* Fixed style check fails

* Fix build issue & moved NetException & TimeoutException to retryable exceptions

* Fixed style check with catch
 2023-09-06 19:57:59 +02:00
Alexander Sapin
4368c00052 Fix logical error in vertical merge + replacing merge tree + optimize cleanup 2023-09-06 18:43:56 +02:00
Kruglov Pavel
5b29640b22
Merge branch 'master' into group-by-constant-keys 2023-09-06 18:06:51 +02:00
robot-ch-test-poll4
8c71582316
Merge pull request #54185 from ClickHouse/analyzer-partition-id-pruning 
Fix partition id pruning for analyzer.
 2023-09-06 16:53:35 +02:00
robot-ch-test-poll1
9e50e8ff20
Merge pull request #54193 from ClickHouse/fix_replace_partition_with_zero_copy 
Fix REPLACE/MOVE PARTITION with zero-copy replication
 2023-09-06 16:04:25 +02:00
Alexander Sapin
a176c38876 Fix tests with database replicated 2023-09-06 14:17:46 +02:00
Alexander Sapin
518d4b4357 Merge branch 'master' into alter-table-add-comment 2023-09-06 12:34:04 +02:00
Anton Popov
401496aa1f Apply suggestions from code review 
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-09-06 10:27:46 +00:00
robot-ch-test-poll1
d9746b5ea1
Merge pull request #54336 from arenadata/ADQM-1157 
Used assert_cast instead of dynamic_cast in ExternalDataSourceCache
 2023-09-06 08:28:21 +02:00
robot-clickhouse-ci-1
02339a1f22
Merge pull request #54326 from ClickHouse/fix_segfault_system_zookeeper 
Fix segfault in system.zookeeper
 2023-09-06 02:20:50 +02:00
Alexey Gerasimchuck
f9edf01280 used assert_cast instead of dynamic_cast 2023-09-05 22:57:40 +00:00
robot-clickhouse-ci-1
7403ee689f
Merge pull request #53877 from vitlibar/add-restore-setting-allow-broken-parts 
Add restore setting "restore_broken_parts_as_detached"
 2023-09-05 21:28:08 +02:00
Kruglov Pavel
de801ce563
Merge pull request #54293 from ClickHouse/pasha-returned-from-vacation 
Code improvement for reading from archives
 2023-09-05 21:09:09 +02:00
Alexander Tokmakov
a8489c21da fix segfault in system.zookeeper 2023-09-05 20:23:16 +02:00
robot-ch-test-poll
2e7edf4413
Merge pull request #53638 from arenadata/ADQM-987 
Add `backup_log` system table
 2023-09-05 17:03:41 +02:00
Antonio Andelic
88930a335c Apply comments 2023-09-05 12:32:07 +00:00
Anton Popov
5da7133c94
Merge pull request #53735 from priera/fix_zk_error_when_doing_replace 
37737 fixed internal error when replacing with byte-equal parts
 2023-09-05 12:51:14 +02:00
alesapin
095bb37bc4
Merge pull request #54244 from bigo-sg/hdfs-config 
Minor improvement, load from config
 2023-09-05 12:36:38 +02:00
Kruglov Pavel
fd42f0bac0
Merge pull request #54271 from Avogar/better-count-and-progress-archives 
Fix using count from cache and improve progress bar for reading from archives
 2023-09-05 12:12:39 +02:00
Antonio Andelic
a51c333c5f Improve code for reading from archives 2023-09-05 08:37:02 +00:00
Victor Krasnov
132ab6067b Merge branch 'master' into ADQM-987 2023-09-05 10:01:44 +03:00
Alexey Milovidov
a59c694e4f
Merge pull request #54225 from Hexta/recreate-secondary-index-system-data-size 
Fix system.data_skipping_indices for recreated indices
 2023-09-05 01:13:09 +03:00
Alexey Milovidov
1a458d524e
Merge pull request #54260 from ClickHouse/vdimir/enable_allow_remove_stale_moving_parts 
Enable allow_remove_stale_moving_parts by default
 2023-09-05 01:11:22 +03:00
robot-clickhouse
926c5636dd
Merge pull request #42599 from ClickHouse/build-fuzzer-protocol 
libFuzzer: add CI fuzzers build, add tcp protocol fuzzer, fix other fuzzers.
 2023-09-04 22:41:54 +02:00
avogar
e9456082c1 Fix using count from cache and improve progress bar for reading from archives 2023-09-04 17:33:59 +00:00
Vitaly Baranov
8060407c70 Add backup setting "check_parts" and restore setting "restore_broken_parts_as_detached". 2023-09-04 18:54:03 +02:00
vdimir
f2ff693a39
Enable allow_remove_stale_moving_parts by default 2023-09-04 15:41:22 +00:00
Alexander Tokmakov
dfcc6f78b3 fix 2023-09-04 16:51:25 +02:00
serxa
aa8e9c3cbc fix style 2023-09-04 14:18:54 +00:00
Alexander Tokmakov
ea996b0a00 Merge branch 'master' into fix_replace_partition_with_zero_copy 2023-09-04 15:55:51 +02:00
Sergei Trifonov
258da91b5f
Merge branch 'master' into disks-and-io-sheduler 2023-09-04 15:14:28 +02:00
serxa
b5233efc20 fix system table and scheduler config reloading 2023-09-04 12:35:17 +00:00
Victor Krasnov
78760639d2 Merge branch 'master' into ADQM-987 2023-09-04 13:59:20 +03:00
zhanglistar
a2fd3a4de8 remove unnecessary check for HDFS load config 2023-09-04 17:52:43 +08:00
Alexey Milovidov
864e66cbf7 Fix TOCTOU 2023-09-04 04:06:24 +02:00
Alexey Milovidov
e618dc993e Fix bad code in the system.filesystem_cache 2023-09-04 03:53:18 +02:00
serxa
8fdd118e0c fix build 2023-09-03 22:40:00 +00:00
serxa
adcab639a4 add system.scheduler table for introspection 2023-09-03 21:48:33 +00:00
Alexey Milovidov
86223699be
Merge pull request #54200 from ClickHouse/fix-ipv6-bloom 
Fix: allow IPv6 for bloom filter
 2023-09-03 23:08:51 +03:00
Artur Malchanau
04d0b0c09a
Fix system.data_skipping_indices for recreated indices 
Fix secondary index size in system.data_skipping_indices table in the
case of dropped and added again index.
 2023-09-03 20:36:55 +03:00
Kseniia Sumarokova
b3319f7908
Minor changes (#54171) 2023-09-03 15:47:52 +02:00
robot-ch-test-poll2
776f232ec0
Merge pull request #54215 from ClickHouse/remove-metadata-cache 
Deprecate metadata cache, part 2
 2023-09-03 07:17:26 +02:00
Alexey Milovidov
7cb4d52d92 Deprecate metadata cache, part 2 2023-09-03 03:56:11 +02:00
Alexey Milovidov
282636fd74 S3Queue is experimental 2023-09-03 00:00:37 +02:00
Michael Kolupaev
8505b3c940
Merge pull request #54134 from kssenii/refactor-reading-from-remote-fs 
Small refactoring for read from object storage
 2023-09-02 11:13:03 -07:00
Yakov Olkhovskiy
a3216c78ef allow IPv6 for bloom filter 2023-09-01 22:44:05 +00:00
Alexander Tokmakov
59d28f3787 fix 2023-09-01 22:40:08 +02:00
Alexander Tokmakov
bba5a1d480 update zero copy locks correctly 2023-09-01 20:10:23 +02:00
Sergei Trifonov
cf5ea46713
Merge pull request #53338 from azat/throttling-fixes 
Fix IO throttling during copying whole directories
 2023-09-01 18:29:15 +02:00
Alexey Milovidov
f8c2ea986d Update autogenerated version to 23.9.1.1 and contributors 2023-09-01 17:46:49 +02:00
Nikolai Kochetov
46087fd3e5 Refactor. 2023-09-01 15:45:08 +00:00
Victor Krasnov
52cce27e81 Merge branch 'master' into ADQM-987 2023-09-01 18:00:20 +03:00
Nikolai Kochetov
4cbfca78e4 Fixing 01748_partition_id_pruning 2023-09-01 14:29:31 +00:00
Yakov Olkhovskiy
361b21b416 fix fuzzers, cmake refactor, add target fuzzers 2023-09-01 14:20:50 +00:00
robot-clickhouse-ci-2
ae4f6c9683
Merge pull request #54137 from kssenii/s3-queue-better-default-path 
Better path for storage `S3Queue` metadata
 2023-09-01 16:01:38 +02:00
Anton Popov
033dba4772
Merge pull request #54126 from ClickHouse/CurtizJ-patch-8 
Small improvements in `getAlterMutationCommandsForPart`
 2023-09-01 14:10:45 +02:00
kssenii
9eb1dfcd12 Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
Nikolai Kochetov
291bd76f1d
Merge pull request #54133 from ClickHouse/fixing-01086_odbc_roundtrip-with-analyzer 
Fixing 01086_odbc_roundtrip with analyzer.
 2023-09-01 12:54:50 +02:00
Kseniia Sumarokova
c51b6ca60c
Update StorageS3Queue.cpp 2023-08-31 18:29:06 +02:00
kssenii
da185fc8b1 Better paths for s3 queue metadata 2023-08-31 18:16:07 +02:00
Nikolai Kochetov
dc062856b0 Fixing 01086_odbc_roundtrip with analyzer. 2023-08-31 15:23:27 +00:00
Anton Popov
af70fdfcfa mute noisy log message 2023-08-31 15:05:35 +00:00
Igor Nikonov
c3caead859
Merge pull request #52688 from Algunenano/better_zk_retries_on_insert 
Improve insert retries on keeper session expiration
 2023-08-31 17:00:02 +02:00
Dmitry Novik
5266511281 Merge remote-tracking branch 'origin/master' into revert-53782-revert-52762-planner-prepare-filters-for-analysis 2023-08-31 14:40:53 +00:00
pufit
1bc6b40a12
Merge pull request #35102 from tonickkozlov/bloom-filter-index/deduplicate 
Prune bloom filter indices based on data cardinality
 2023-08-31 10:04:19 -04:00
Anton Kozlov
af21620ab5
Update src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h 
Co-authored-by: pufit <pufit@yandex.ru>
 2023-08-31 14:06:21 +01:00
Anton Kozlov
d3123da824
Update src/Storages/MergeTree/MergeTreeIndexAggregatorBloomFilter.cpp 
Co-authored-by: pufit <pufit@yandex.ru>
 2023-08-31 14:05:57 +01:00
Anton Kozlov
3f6ef6b5d9
Update src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h 
Co-authored-by: pufit <pufit@yandex.ru>
 2023-08-31 14:05:41 +01:00
Anton Kozlov
e6895688d7
Update src/Storages/MergeTree/MergeTreeIndexAggregatorBloomFilter.cpp 
Co-authored-by: pufit <pufit@yandex.ru>
 2023-08-31 14:05:28 +01:00
Anton Popov
579ec699bf Merge remote-tracking branch 'upstream/master' into HEAD 2023-08-31 12:59:59 +00:00
Anton Popov
77111a229d fix clang-tidy 2023-08-31 12:59:21 +00:00
Raúl Marín
4703c463aa Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-31 14:37:02 +02:00
Andrey Zvonov
83405a8808
Merge branch 'master' into zvonand-globs-small-fix 2023-08-31 08:42:48 +02:00
Antonio Andelic
0148e15aee
Merge pull request #53880 from ClickHouse/archive-improvements-2 
Improve schema inference for archives
 2023-08-30 21:11:34 +02:00
Alexander Tokmakov
83c5e2fba6
Revert "Fix bug on mutations with subcolumns of type JSON in predicates of UPDATE and DELETE queries." 2023-08-30 12:56:17 +02:00
Antonio Andelic
f406019413 Apply PR comments 2023-08-30 09:26:01 +00:00
Antonio Andelic
ddb58217d4 Merge branch 'master' into archive-improvements-2 2023-08-30 07:43:25 +00:00
Antonio Andelic
36fb7cfbd1
Merge pull request #54012 from ClickHouse/refactor-async-insert-with-dedup 
Refactor logic around async insert with deduplication
 2023-08-30 09:37:55 +02:00
Azat Khuzhin
51e8b16916 Fix IO throttling during copying whole directories 
v2: pass local context during ALTER TABLE MOVE PARTITION TO VOLUME/DISK as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-29 23:13:46 +02:00
Nikita Mikhaylov
97efd14e90
Merge pull request #54009 from ClickHouse/change-istorage-interface 
Change IStorage interface by random walk, no goal in particular
 2023-08-29 22:22:53 +02:00
Sergei Trifonov
802579f3f1
Merge pull request #49618 from ClickHouse/concurrency-control-controllable 
Make concurrency control controllable
 2023-08-29 19:44:51 +02:00
Anton Popov
8197580746 fix tests and better interface 2023-08-29 17:23:52 +00:00
Antonio Andelic
e96c665be7 Refactor logic around async insert with dedup 2023-08-29 14:48:43 +00:00
Alexander Tokmakov
17f3754193
Merge pull request #53851 from ClickHouse/distributed_no_flush_on_shutdown 
Follow-up to #53501
 2023-08-29 16:34:22 +02:00
Nikita Mikhaylov
96cabcae3a Change interface 2023-08-29 16:26:48 +02:00
Anton Popov
7bc5627912 fix typos 2023-08-29 14:21:58 +00:00
Anton Popov
02391eafe1 refactoring of reading from MergeTree tables 2023-08-29 13:40:12 +00:00
Anton Popov
6ce4a82a04
Merge pull request #53677 from VanDarkholme7/mutations_subcolumns 
Fix bug on mutations with subcolumns of type JSON in predicates of UPDATE and DELETE queries.
 2023-08-29 15:08:00 +02:00
Antonio Andelic
a0dd350e5e Fix assert 2023-08-29 09:59:16 +00:00
Raúl Marín
45e8185f9b Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-08-29 11:17:34 +02:00
Andrey Zvonov
6195390c77
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-08-29 13:46:09 +05:00
János Benjamin Antal
a1a45ee905
Merge pull request #53766 from ilejn/exceptions_kafka_consumers 
system.kafka_consumers exception info improvements
 2023-08-29 10:22:26 +02:00
Antonio Andelic
8708ebd7fb Correctly check files 2023-08-29 07:40:08 +00:00
Anton Popov
03f5d3feb4 fix checkDataPart 2023-08-28 14:59:03 +00:00
Raúl Marín
174f7a4e63 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-28 16:27:45 +02:00
Alexander Tokmakov
d41eca1dcc rename new method 2023-08-28 16:01:00 +02:00
Antonio Andelic
9b99f25d75 Improve schema inference 2023-08-28 13:11:52 +00:00
Robert Schulze
4c78c08847
Add test 2023-08-28 13:02:24 +00:00
Davit Vardanyan
9190fb2ba3 Fix: Deserialize 2023-08-28 16:18:12 +04:00
Andrey Zvonov
c9b0db9c61
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-08-28 17:15:33 +05:00
Andrey Zvonov
2447eb027c
Update StorageHDFS.cpp 2023-08-28 15:12:44 +03:00
Igor Nikonov
a2686af496
Merge pull request #53862 from ClickHouse/cleanup-parallel-replicas-0 
Remove unused parallel replicas coordinator in query info
 2023-08-28 13:30:23 +02:00
Antonio Andelic
5a0c2ca108 Merge branch 'master' into archive-improvements-2 2023-08-28 08:34:42 +00:00
VanDarkholme7
bac257743b
Merge branch 'ClickHouse:master' into mutations_subcolumns 2023-08-28 14:10:31 +08:00
Igor Nikonov
7368bf65b3 Remove unused parallel replicas coordinator in query info 2023-08-27 22:12:39 +00:00
Victor Krasnov
87c90767d9 Merge branch 'master' into ADQM-987 2023-08-27 09:14:42 +03:00
alesapin
0e601b856f
Merge pull request #53751 from Algunenano/clang18 
Support clang-18 (Wmissing-field-initializers)
 2023-08-25 23:27:06 +02:00
Andrey Zvonov
911bf94fbd
Merge branch 'master' into zvonand-globs-small-fix 2023-08-25 23:50:07 +05:00
Victor Krasnov
921f0c8b5d Merge branch 'master' into ADQM-987 2023-08-25 21:20:23 +03:00
Alexander Tokmakov
32efbe77d1
Merge pull request #53811 from ClickHouse/fix_02443_2 
Fix a bug in attach partition
 2023-08-25 20:16:53 +03:00
Alexander Tokmakov
9ab545e28c do not wait for flush on shutdown 2023-08-25 19:09:10 +02:00
Alexander Gololobov
684d5aab88
Merge pull request #53797 from ClickHouse/add_profile_event_for_lock 
Add event for tracking data parts lock
 2023-08-25 18:51:49 +02:00
Kseniia Sumarokova
34ac113af6
Merge pull request #53121 from kssenii/fix-cache-logical-error 
Fix possible error from cache "Read unexpected size"
 2023-08-25 12:16:24 +02:00
Nikolai Kochetov
4e4620250d Fixing clang tidy. 2023-08-25 09:51:52 +00:00
Kruglov Pavel
d80be4673b
Merge pull request #53692 from Avogar/cache-count 
Cache number of rows in files for count in file/s3/url/hdfs/azure functions
 2023-08-25 09:08:52 +02:00
Antonio Andelic
f0e800092f
Merge pull request #53796 from ClickHouse/fix-filtered-url-paths 
Fix reading from `url` with all filtered paths
 2023-08-25 08:49:33 +02:00
Yakov Olkhovskiy
415a993c91 fix fuzzers build 2023-08-24 23:22:39 +00:00
alesapin
3a289534cb Fxi 2023-08-24 20:48:10 +02:00
Nikolai Kochetov
506c5667a7 Remove comment. 2023-08-24 18:21:41 +00:00
Alexander Tokmakov
cbbb81f5cc
Merge pull request #53405 from canhld94/limit_concurent_background_optimize 
Limit number of concurrent background partition optimize
 2023-08-24 20:49:38 +03:00
Alexander Tokmakov
75d32bfe77
Merge pull request #53750 from ClickHouse/Add_retries_for_azure_retryable_errors 
Add retries for azure retryable errors
 2023-08-24 20:26:42 +03:00
Alexander Tokmakov
e1d135dc01 fix another bug in attach partition 2023-08-24 19:10:25 +02:00
Nikolai Kochetov
9737dbca75 Fix window functions. 2023-08-24 16:37:27 +00:00
Raúl Marín
6490213207 Merge remote-tracking branch 'blessed/master' into clang18 2023-08-24 17:35:26 +02:00
Kruglov Pavel
44db5fa992
Merge branch 'master' into cache-count 2023-08-24 17:21:18 +02:00
avogar
c4f8113a78 Fix tests 2023-08-24 15:05:45 +00:00
Robert Schulze
45d924c62c
Merge pull request #53616 from wzb5212/skip_idx_bug_fix 
Fix number of dropped granules in EXPLAIN PLAN index=1
 2023-08-24 16:46:47 +02:00
Antonio Andelic
6eb400c6ae Fix reading from url with all filtered paths 2023-08-24 13:37:11 +00:00
alesapin
415b3826e0 Add event for tracking data parts lock 2023-08-24 15:28:38 +02:00
Ilya Golshtein
f58b4a812d exceptions_kafka_consumers: code review suggestions are addressed 2023-08-24 13:10:21 +00:00
Arthur Passos
2bade7db08
Add global proxy setting (#51749) 
* initial impl

* fix env ut

* move ut directory

* make sure no null proxy resolver is returned by ProxyConfigurationResolverProvider

* minor adjustment

* add a few tests, still incomplete

* add proxy support for url table function

* use proxy for select from url as well

* remove optional from return type, just returns empty config

* fix style

* style

* black

* ohg boy

* rm in progress file

* god pls don't let me kill anyone

* ...

* add use_aws guards

* remove hard coded s3 proxy resolver

* add concurrency-mt-unsafe

* aa

* black

* add logging back

* revert change

* imrpove code a bit

* helper functions and separate tests

* for some reason, this env test is not working..

* formatting

* :)

* clangtidy

* lint

* revert some stupid things

* small test adjusmtments

* simplify tests

* rename test

* remove extra line

* freaking style change

* simplify a bit

* fix segfault & remove an extra call

* tightly couple proxy provider with context..

* remove useless include

* rename config prefix parameter

* simplify provider a bit

* organize provider a bit

* add a few comments

* comment out proxy env tests

* fix nullptr in unit tests

* make sure old storage proxy config is properly covered without global context instance

* move a few functions from class to anonymous namespace

* fix no fallback for specific storage conf

* change API to accept http method instead of bool

* implement http/https distinction in listresolver, any still not implemented

* implement http/https distinction in remote resolver

* progress on code, improve tests and add url function working test

* use protcol instead of method for http and https

* small fix

* few more adjustments

* fix style

* black

* move enum to proxyconfiguration

* wip

* fix build

* fix ut

* delete atomicroundrobin class

* remove stale include

* add some tests.. need to spend some more time on the design..

* change design a bit

* progress

* use existing context for tests

* rename aux function and fix ut

* ..

* rename test

* try to simplify tests a bit

* simplify tests a bit more

* attempt to fix tests, accept more than one remote resolver

* use proper log id

* try waiting for resolver

* proper wait logic

* black

* empty

* address a few comments

* refactor tests

* remove old tests

* baclk

* use RAII to set/unset env

* black

* clang tidy

* fix env proxy not respecting any

* use log trace

* fix wrong logic in getRemoteREsolver

* fix wrong logic in getRemoteREsolver

* fix test

* remove unwanted code

* remove ClientConfigurationperRequest and auxilary classes

* remove unwanted code

* remove adapter test

* few adjustments and add test for s3 storage conf  with new proxy settings

* black

* use chassert for context

* Add getenv comment
 2023-08-24 16:07:26 +03:00
Antonio Andelic
8e1d38d377 Merge branch 'master' into archive-improvements-2 2023-08-24 13:03:36 +00:00
Nikolai Kochetov
cb851fcee0 Cleanup. 2023-08-24 11:07:17 +00:00
Nikolai Kochetov
33b8b93d1b Re-implement getOptimizedQueryProcessingStage with analyzer. 2023-08-24 11:07:17 +00:00
Nikolai Kochetov
fc90a1a0bd Fix some skip_unused_shards tests. 2023-08-24 11:07:17 +00:00
Nikolai Kochetov
26e0ad8d72 Re-impl evaluateExpressionOverConstantCondition [part 1] 2023-08-24 11:07:17 +00:00
Nikolai Kochetov
93ab53163a
Revert "Revert "Planner prepare filters for analysis"" 2023-08-24 12:32:56 +02:00
Raúl Marín
1c98568df9 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-24 12:22:42 +02:00
Robert Schulze
8c9fade58c
Merge remote-tracking branch 'rschu1ze/master' into skip_idx_bug_fix 2023-08-24 10:17:34 +00:00
Kseniia Sumarokova
a4f39fe134
Merge branch 'master' into fix-cache-logical-error 2023-08-24 12:00:18 +02:00
Alexander Tokmakov
011c7f7064
Revert "Planner prepare filters for analysis" 2023-08-24 12:52:03 +03:00
Kseniia Sumarokova
187706558a
Merge pull request #51622 from kssenii/system-sync-cache 
Add `system sync filesystem cache` command
 2023-08-24 11:42:32 +02:00
Ilya Golshtein
901240eede exceptions_kafka_consumers: clang-tidy improvement 2023-08-24 08:44:08 +00:00
helifu
adfdba4b83 Display reloaded values of server settings 2023-08-24 12:32:30 +08:00
Ilya Golshtein
ec0c12960a exceptions_kafka_consumers: stylecheck 2023-08-23 23:10:25 +00:00
Ilya Golshtein
e7213ef5b9 exceptions_kafka_consumers: new test, MV exceptions propagated 2023-08-23 22:46:04 +00:00
Kruglov Pavel
685ffc5856
Fix style 2023-08-23 22:51:14 +02:00
Kruglov Pavel
f7e1abd774
Merge branch 'master' into cache-count 2023-08-23 22:31:49 +02:00
Kruglov Pavel
063c986680
Merge pull request #53637 from Avogar/fast-count-from-files 
Optimize count from files in most input formats
 2023-08-23 22:17:29 +02:00
avogar
93ba73715a Merge branch 'master' of github.com:ClickHouse/ClickHouse into group-by-constant-keys 2023-08-23 18:47:42 +00:00
avogar
894513f6cd Fix tests 2023-08-23 18:43:08 +00:00
Nikolai Kochetov
3a4a7d70a2
Merge pull request #52762 from kitaisreal/planner-prepare-filters-for-analysis 
Planner prepare filters for analysis
 2023-08-23 18:59:43 +02:00
Anton Kozlov
a5b6a91b54 [bloom filter] Write hashes for bloom filter into a hash set for optimal sizing 2023-08-23 14:17:55 +00:00
Victor Krasnov
8b2342c205 Minor code cleanup: remove some redundant includes of InterpreterAlterQuery.h 2023-08-23 14:16:36 +00:00
Raúl Marín
93dac0c880 Support clang-18 (Wmissing-field-initializers) 2023-08-23 15:53:45 +02:00
Smita Kulkarni
262c39329b Updated to retry loading parts for azure TransportException 2023-08-23 15:50:11 +02:00
Smita Kulkarni
1a074ecf93 Updated to not mark part as broken for azure TransportException in loadDataPart 2023-08-23 15:33:01 +02:00
Smita Kulkarni
7fdf90ccf1 Added azure TransportException to retryable errors 2023-08-23 15:21:37 +02:00
avogar
68e3af56d4 Address comments 2023-08-23 13:19:15 +00:00
Kruglov Pavel
592fa77987
Merge branch 'master' into cache-count 2023-08-23 15:18:02 +02:00
Kruglov Pavel
7e362a2110
Merge branch 'master' into fast-count-from-files 2023-08-23 15:13:20 +02:00
Robert Schulze
bea651e828
Merge pull request #53657 from rschu1ze/cache-improvements 
Expose more settings from caches
 2023-08-23 14:58:45 +02:00
Robert Schulze
a093f519fc
Merge remote-tracking branch 'rschu1ze/master' into skip_idx_bug_fix 2023-08-23 12:37:46 +00:00
Robert Schulze
5dec6f19f3
Merge pull request #51325 from xinhuitian/fix-annoy-index-update 
Do not reset Annoy index during build-up with > 1 mark
 2023-08-23 14:31:06 +02:00
robot-ch-test-poll1
c22ffa6195
Merge pull request #53529 from Avogar/filter-files-all-table-functions 
Use filter by file/path before reading in url/file/hdfs table functins
 2023-08-23 14:21:23 +02:00
Robert Schulze
761268f86a
Merge remote-tracking branch 'rschu1ze/master' into skip_idx_bug_fix 2023-08-23 10:55:04 +00:00
Kruglov Pavel
e193aec583
Merge branch 'master' into fast-count-from-files 2023-08-23 12:15:34 +02:00
kssenii
325e2d3172 Merge remote-tracking branch 'upstream/master' into system-sync-cache 2023-08-23 11:44:19 +02:00
priera
fd51839b97 basic implementation 2023-08-23 11:19:01 +02:00
Antonio Andelic
83d4b819f3 Better support for reading from archives 2023-08-23 08:10:30 +00:00
xinhuitian
920762e66d fix USearch 2023-08-23 10:54:56 +08:00
Tian Xinhui
7cf057e716
Merge branch 'ClickHouse:master' into fix-annoy-index-update 2023-08-23 10:33:14 +08:00
pufit
926533306c
Merge pull request #53668 from ClickHouse/pufit/fix_s3_threads 
Limiting the number of parsing threads for the S3 source
 2023-08-22 19:09:33 -04:00
Kruglov Pavel
a83a868acd
Update src/Storages/HDFS/StorageHDFS.cpp 
Co-authored-by: Dmitry Novik <mrnovikd@gmail.com>
 2023-08-22 19:08:04 +02:00
Robert Schulze
d5a7277a59
Better parameter name 2023-08-22 15:43:13 +00:00
pufit
e42da9411b Fix variables names 2023-08-22 11:23:10 -04:00
Raúl Marín
abfd48bcd2 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-22 17:01:22 +02:00
Nikolai Kochetov
5ee9b56040 Do not change the snapshot after at reading from MT 2023-08-22 13:50:51 +00:00
Victor Krasnov
258f9927b4 Merge branch 'master' into ADQM-987 2023-08-22 16:49:32 +03:00
Kruglov Pavel
de960f3c35
Fix style 2023-08-22 15:22:17 +02:00
Kruglov Pavel
a34c9c3ebc
Fix build 2023-08-22 15:21:54 +02:00
Kruglov Pavel
67c5c0203b
Merge branch 'master' into fast-count-from-files 2023-08-22 15:03:48 +02:00
avogar
89b402f4e2 Fix 2023-08-22 13:01:07 +00:00
avogar
7f9e81d504 Clean up 2023-08-22 12:55:00 +00:00
alesapin
b4aaeb1a82 Fix bad log line format 2023-08-22 14:53:35 +02:00
Kruglov Pavel
c0bdd0e00b
Merge branch 'master' into cache-count 2023-08-22 14:42:22 +02:00
avogar
b4145aeddc Cache number of rows in files for count in file/s3/url/hdfs/azure functions 2023-08-22 11:59:59 +00:00
Robert Schulze
f2fb7365e6
Add settings for 1. size_ratio (all caches) and 2. policy (index caches) 
1. is an expert-level setting, default is 0.5, applies only to SLRU.

Also, I noticed that we expose cache policy settings for the mark and
the uncompresed cache but not for the index mark and the index
uncompressed cache. Changed that as well, it simplifies the code a bit.
 2023-08-22 11:04:29 +00:00
Ilya Golshtein
7cd8237a42 exceptions_kafka_consumers: initial 2023-08-22 10:26:52 +00:00
Robert Schulze
acaa2eee58
Merge pull request #53556 from ClibMouse/feature/big-endian-UUID-correction 
Change Big Endian-UUID to work the same as Little Endian-UUID
 2023-08-22 11:57:44 +02:00
alesapin
1a5a9fca87
Merge pull request #53605 from irenjj/feat_53538 
Add column 'name' to system.clusters as an alias to cluster.
 2023-08-22 11:34:21 +02:00
VanDarkholme7
2638cc69d7 fix bug on DELETE and UPDATE with JSON subcolumns 2023-08-22 09:26:17 +00:00
Tian Xinhui
76016d9593
Merge branch 'master' into fix-annoy-index-update 2023-08-22 16:58:29 +08:00
pufit
9d454d9afc Merge branch 'master' into pufit/fix_s3_threads 
# Conflicts:
#	src/Storages/StorageS3.cpp
#	src/Storages/StorageS3.h
#	src/Storages/StorageURL.cpp
#	src/Storages/StorageURL.h
 2023-08-21 21:32:15 -04:00
pufit
98a701e2c1 Limiting number of parsing threads for S3 source 2023-08-21 21:21:03 -04:00
Robert Schulze
a20ceef398
Merge pull request #53447 from davvard/master 
USearch HNSW index for ANN search
 2023-08-21 23:34:26 +02:00
Michael Kolupaev
2f4d433e69 Parquet filter pushdown 2023-08-21 14:15:52 -07:00
Nikolai Kochetov
4222ac528d Merge branch 'master' into planner-prepare-filters-for-analysis 2023-08-21 16:48:33 +00:00
Kruglov Pavel
aebbf00a5a
Fix 2023-08-21 17:47:30 +02:00
Anton Popov
3e9a182555 better interfaces for getting of stream name in part 2023-08-21 13:52:25 +00:00
Kruglov Pavel
a2f81bf4ea
Fix typo 2023-08-21 15:20:44 +02:00
Kruglov Pavel
dfdd26ee71
Fix build 2023-08-21 15:04:03 +02:00
Kruglov Pavel
88aee95122
Merge branch 'master' into fast-count-from-files 2023-08-21 14:46:33 +02:00
avogar
584bd57f5c Clean up 2023-08-21 12:43:11 +00:00
avogar
47304bf7aa Optimize count from files in most input formats 2023-08-21 12:30:52 +00:00
Kruglov Pavel
9d9bdde0e3
Merge branch 'master' into filter-files-all-table-functions 2023-08-21 14:26:59 +02:00
Kruglov Pavel
9869196021
Update VirtualColumnUtils.cpp 2023-08-21 14:22:48 +02:00
Antonio Andelic
ed15babddd
Merge pull request #53198 from ClickHouse/improve-archive-reading 
Improve reading from archives
 2023-08-21 14:11:25 +02:00
Victor Krasnov
a7813260c0 Merge branch 'master' into ADQM-987 2023-08-21 14:45:13 +03:00
Kruglov Pavel
21413636ac
Remove debug logging 2023-08-21 13:30:08 +02:00
Robert Schulze
2c330516b8
Another compile fix 2023-08-21 11:12:39 +00:00
Duc Canh Le
2c07065f81 increase the default value 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-08-21 11:04:24 +00:00
Raúl Marín
f2d3c9c281 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-21 12:14:31 +02:00
Robert Schulze
066ec559ad
Do not reset Annoy index during build-up with > 1 mark 2023-08-21 10:05:43 +00:00
王智博
8bb0f18019 bad size of mark skip idx 2023-08-20 22:57:53 +08:00
Kseniia Sumarokova
404fb0588a
Merge branch 'master' into fix-cache-logical-error 2023-08-20 16:28:13 +02:00
irenjj
a9b65ee929 Add Alias for system.clusters. 2023-08-20 14:47:07 +08:00
irenjj
71459c6d65 Revert 2023-08-20 14:35:59 +08:00
irenjj
e64a1fc26d Add column 'name' to system.clusters as an alias to cluster. 2023-08-20 11:52:14 +08:00
Igor Nikonov
a69ea02418 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-19 21:43:25 +00:00
Kseniia Sumarokova
aeebedfa0d
Merge branch 'master' into system-sync-cache 2023-08-19 13:32:48 +04:00
Kseniia Sumarokova
c0b8d7eddf
Merge pull request #50999 from ilejn/system_kafka_consumers 
system.kafka_consumers table to monitor kafka consumers
 2023-08-19 13:31:18 +04:00
kssenii
247abe0a2a Merge remote-tracking branch 'upstream/master' into system-sync-cache 2023-08-19 08:43:53 +04:00
Alexey Milovidov
8520a7baca
Merge pull request #53559 from azat/skip-index-set-fix 
Fix possible UB in Set skipping index for functions with incorrect args
 2023-08-19 02:06:18 +03:00
Alexey Milovidov
6fb24fe74d
Merge pull request #53560 from azat/skip-index-inverted-fix-ub 
Fix possible UB in inverted indexes (experimental feature)
 2023-08-19 02:06:04 +03:00
Igor Nikonov
aebda10f7c Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-18 23:00:58 +00:00
Alexey Milovidov
7ad8ea78bb
Merge pull request #48990 from incfly/partition 
Add partition to the system part_log and merge table.
 2023-08-18 21:38:53 +03:00
avogar
8e445b5461 Fixes 2023-08-18 17:49:40 +00:00
Azat Khuzhin
c78b94bed6 Fix possible UB in inverted indexes (experimental feature) 
It is possible to have incorrect types there in case of index contains
functions, add a proper check after value had been changed.

<details>

<summary>MSan report</summary>

```
==182==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55658547c59e in DB::MergeTreeConditionInverted::traverseASTEquals() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:551:26
    1 0x556585475566 in DB::MergeTreeConditionInverted::traverseAtomAST() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:438:21
    2 0x55658547e73e in DB::MergeTreeConditionInverted::MergeTreeConditionInverted()::$_1::operator()(DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:228:73
    9 0x556585489e35 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:252:14
    10 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    11 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    12 0x5565854804e6 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::RPNBuilder(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::Block, std::__1::shared_ptr<DB::PreparedSets>, std::__1::function<bool (DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&)> const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:218:9
    13 0x55658546fb87 in DB::MergeTreeConditionInverted::MergeTreeConditionInverted(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>, DB::Block const&, DB::GinFilterParameters const&, DB::ITokenExtractor const*) build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:223:28
    19 0x55658547ceda in DB::MergeTreeIndexInverted::createIndexCondition(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:716:12
    20 0x556587125734 in DB::buildIndexes(std::__1::optional<DB::ReadFromMergeTree::Indexes>&, std::__1::shared_ptr<DB::ActionsDAG>, DB::MergeTreeData const&, std::__1::shared_ptr<DB::Context const> const&, DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&) build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1292:48
    21 0x556587121aaa in DB::ReadFromMergeTree::applyFilters() build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1305:5
    22 0x55658726cf4f in DB::QueryPlanOptimizations::optimizeTreeThirdPass(DB::QueryPlan::Node&, std::__1::list<DB::QueryPlan::Node, std::__1::allocator<DB::QueryPlan::Node>>&) build_docker/./src/Processors/QueryPlan/Optimizations/optimizeTree.cpp:201:38
    23 0x5565870bc489 in DB::QueryPlan::optimize(DB::QueryPlanOptimizationSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:485:5
    24 0x5565870b919a in DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:161:5
    25 0x55658206385a in DB::InterpreterSelectWithUnionQuery::execute() build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379:31
    26 0x556582c15c50 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:751:40
    27 0x556582c09681 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1173:30
    28 0x5565860e7bd7 in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:421:24
    29 0x556586126e9e in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2057:9
```

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-18 18:15:17 +02:00
Azat Khuzhin
1e2b0c8dc4 Fix possible UB in Set skipping index for functions with incorrect args 
Usually functions are validated before index analysis, however it is
not the case for parallel replicas, and it requires additional checks
before interpreting function arguments.

<details>

<summary>stack trace</summary>

```
==172==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55cf82aedd6c in DB::ASTFunction* typeid_cast<DB::ASTFunction*, DB::IAST>(DB::IAST*) build_docker/./src/Common/typeid_cast.h:42:73
    1 0x55cf82aedd6c in DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, false>::value(DB::IAST*) build_docker/./src/Common/TypePromotion.h:38:43
    2 0x55cf82aedd6c in std::__1::invoke_result<decltype(&CastHelper<DB::ASTFunction, false>::value), DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, std::is_reference_v<DB::ASTFunction>>, DB::IAST*>::type DB::TypePromotion<DB::IAST>::as<DB::ASTFunction>() build_docker/./src/>
    3 0x55cf82aedd6c in DB::MergeTreeIndexConditionSet::operatorFromAST(std::__1::shared_ptr<DB::IAST>&) build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:602:25
    4 0x55cf82ae5bc1 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:547:9
    5 0x55cf82ae5de6 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:552:13
    6 0x55cf82ae06db in DB::MergeTreeIndexConditionSet::MergeTreeIndexConditionSet() build_docker/.>
    ...
    12 0x55cf82aef09c in DB::MergeTreeIndexSet::createIndexCondition() const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:703:12
    13 0x55cf84951cd4 in DB::buildIndexes()
    14 0x55cf84955ed3 in DB::ReadFromMergeTree::selectRangesToReadImpl()
    15 0x55cf8494caef in DB::ReadFromMergeTree::selectRangesToRead()
    16 0x55cf82a409a9 in DB::MergeTreeDataSelectExecutor::estimateNumMarksToRead()
    17 0x55cf827f728d in DB::MergeTreeData::canUseParallelReplicasBasedOnPKAnalysis()
    18 0x55cf827f627e in DB::MergeTreeData::getQueryProcessingStage()
    19 0x55cf7f2f4969 in DB::InterpreterSelectQuery::getSampleBlockImpl() build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:937:31
    20 0x55cf7f2daa00 in DB::InterpreterSelectQuery::InterpreterSelectQuery()
    24 0x55cf7f520b98 in DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter()
    25 0x55cf7f51b6cd in DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery()
    27 0x55cf7f1d4ea9 in DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) build_docker/./src/Interpreters/InterpreterFactory.cpp:162:16
    28 0x55cf8012e485 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1032:31
    29 0x55cf80121bc1 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1229:30
    30 0x55cf8389295f in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:424:24
    31 0x55cf838d7dfb in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2050:9
```

</details>

CI: https://s3.amazonaws.com/clickhouse-test-reports/53214/d99b10c340909ed4ee2e6edf0921e8a2f8561b0d/fuzzer_astfuzzermsan/report.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-18 18:02:46 +02:00
Austin Kothig
6b42975d33
Change BE-UUID to work the same as LE-UUID. Included high and low getters to provide cleaner code when accessing undertype. 2023-08-18 08:19:46 -07:00
Yakov Olkhovskiy
84cb358c14
Merge pull request #43210 from amosbird/storage_executable_stderr 
Add different reactions when external command stderr has data
 2023-08-18 11:17:46 -04:00
Alexander Tokmakov
a6684f7a37
Merge pull request #53484 from azat/rmt-queue-cleanup 
Remove outdated code in ReplicatedMergeTreeQueue::initialize()
 2023-08-18 18:11:33 +03:00
avogar
2d4167c612 FRemove unneded changes 2023-08-18 14:50:07 +00:00
avogar
aac5131859 Fix for no_more_keys 2023-08-18 13:21:16 +00:00
avogar
7fa72f5cdf Optimize group by constant keys 2023-08-18 12:47:26 +00:00
Alexander Gololobov
50061b9cb4
Merge pull request #53452 from ClickHouse/such_much_watch 
Deduplicate same watch callbacks registered multiple times
 2023-08-18 14:43:03 +02:00
Alexander Tokmakov
0bd378afd9
Merge pull request #53381 from bakwc/bakwc-patch-2 
Fixed system.data_skipping_indices for MaterializedMySQL
 2023-08-18 14:22:40 +03:00
Duc Canh Le
2d353f8c43 Merge branch 'master' into limit_concurent_background_optimize 2023-08-18 09:25:27 +00:00
Amos Bird
b6de70b042
StorageExecutable check_exit_code default to false 2023-08-18 15:38:47 +08:00
Amos Bird
c43bf153f5
Refactor 2023-08-18 15:38:46 +08:00
Amos Bird
dd0c71b32a
Add error_exit_reaction 2023-08-18 15:38:46 +08:00
Amos Bird
476f3cedc1
Various reactions when executable stderr has data 2023-08-18 15:38:45 +08:00
Alexey Milovidov
a882bf1bb8
Merge pull request #53495 from ClickHouse/i_hate_materialized_views 
Fix a race condition between RESTART REPLICAS and DROP DATABASE
 2023-08-18 06:40:04 +03:00
Igor Nikonov
06c98956b3 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-17 20:42:50 +00:00
Jianfei Hu
0c81546960 remove the comments. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-17 20:13:33 +00:00
pufit
a52249872e
Merge pull request #53313 from jorisgio/joris/max_threads_for_indexes 
add max_threads_for_indexes settings
 2023-08-17 13:59:53 -04:00
avogar
60b0b88d50 Clean up 2023-08-17 16:59:57 +00:00
avogar
4c32097df3 Use filter by file/path before reading in url/file/hdfs table functions, reduce code duplication 2023-08-17 16:54:43 +00:00
Dmitry Novik
1bd478dba2 Merge remote-tracking branch 'origin/master' into fix-prewhere-lost-column 2023-08-17 16:00:35 +00:00
Alexander Gololobov
54e2be4e32 Use WatchCallbackPtr 2023-08-17 17:32:04 +02:00
Dmitry Novik
01cb96d39a Leave only necessary conditions in output 2023-08-17 15:27:24 +00:00
Alexander Tokmakov
89c365803f
Merge pull request #53424 from ClickHouse/exception_message_patterns6 
Less exceptions with runtime format string
 2023-08-17 17:58:04 +03:00
Igor Nikonov
3e9ae1751e Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-17 14:45:37 +00:00
Filipp Ozinov
ce87451b66
Merge branch 'ClickHouse:master' into bakwc-patch-2 2023-08-17 18:37:00 +04:00
Anton Popov
0865c0794e Merge remote-tracking branch 'upstream/master' into HEAD 2023-08-17 14:08:59 +00:00
Alexander Tokmakov
72b52250ea
Merge pull request #53501 from azat/dist-flush_on_detach 
Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown
 2023-08-17 16:56:04 +03:00
Azat Khuzhin
fcb66379d1 Remove outdated code in ReplicatedMergeTreeQueue::initialize() 
Initially added in #28817
But got cleaned up in #41981

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-17 13:45:26 +02:00
Robert Schulze
ec7daed52f
Merge pull request #50276 from ClibMouse/feature/mergetree-checksum-big-endian-support 
MergeTree & SipHash checksum big-endian support
 2023-08-17 13:27:18 +02:00
Robert Schulze
42644bfbf5
Rename file to MergeTreeIndexUSearch.h/cpp 2023-08-17 09:54:12 +00:00
Robert Schulze
f420006e09
Make the code consistent with Annoy index and fix build 2023-08-17 09:42:13 +00:00
Robert Schulze
456a02da1b
Merge remote-tracking branch 'rschu1ze/master' 2023-08-17 09:30:27 +00:00
Robert Schulze
1215a23de6
Merge pull request #53422 from rschu1ze/consistency-fixes-for-caches 
Refactorings for configuration of in-memory caches
 2023-08-17 11:08:32 +02:00
Mingliang Pan
2db0b4f52a
Update StorageEmbeddedRocksDB.cpp 
add drop function to close file before drop table
 2023-08-17 15:14:14 +08:00
Mingliang Pan
349f3b9c02
Update StorageEmbeddedRocksDB.h 
add drop method to close file before drop table
 2023-08-17 15:12:42 +08:00
Azat Khuzhin
17ca2661a1 Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown 
Sometimes you can have tons of data there, i.e. few TiBs, and sending
them on server shutdown does not looks sane (maybe there is a bug and
you need to update/restart to fix flushing).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-17 08:58:06 +02:00
Igor Nikonov
49f7b4acd6 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-16 23:31:49 +00:00
Igor Nikonov
21ef1f1d1c Remove unnecessary Scalars usage for cluster_for_parallel_replicas 2023-08-16 23:24:01 +00:00
Alexander Tokmakov
d7272410bd the worst fix for a deadlock ever 2023-08-17 00:42:51 +02:00