thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
117,803 Commits 9,328 Branches 1,783 Tags 2.1 GiB
71678f64b1
Commit Graph

13198 Commits

Author SHA1 Message Date
alesapin
cc3897a84a
Merge pull request #50033 from kssenii/disk-object-storage-minor-changes 
Get rid of indirect write buffer in object storages
 2023-05-22 14:08:17 +02:00
Alexander Tokmakov
487e510103 Merge branch 'master' into less_zookeeper_requests 2023-05-22 13:59:26 +02:00
Alexander Tokmakov
c89f92e1f6
Merge pull request #50052 from amosbird/fix_49913 
Fix reporting broken projection parts
 2023-05-22 14:20:06 +03:00
Nikolay Degterinsky
183f90e45a Update MongoDB protocol 2023-05-22 09:05:23 +00:00
Antonio Andelic
93cd47fd5f
Merge pull request #50026 from ClickHouse/fix-deadlock-attach-thread 
Avoid deadlock when starting table in attach thread of `ReplicatedMergeTree`
 2023-05-22 09:22:33 +02:00
Amos Bird
0a3d986e42
Fix reporting projection broken part 2023-05-21 20:58:58 +08:00
Nikita Taranov
c93836b962 fix 2023-05-19 22:27:53 +00:00
zvonand
7dccb51bc0 fix conflict and update 2023-05-19 23:40:51 +02:00
alekseygolub
2b68a6a22a Fix style 2023-05-19 16:03:22 +00:00
alekseygolub
c85c3afa1f Added option to rename files, loaded via TableFunctionFile, after success processing 2023-05-19 16:03:22 +00:00
kssenii
3e42ee7f2b Get rid of finalize callback in object storages 2023-05-19 17:29:37 +02:00
Antonio Andelic
4af8187464 Activate restarting thread in both cases 2023-05-19 15:06:02 +00:00
mateng915
5237dd0245
New system table zookeeper connection (#45245) 
* Feature: Support new system table to show which zookeeper node be connected

Description:
============
Currently we have no place to check which zk node be connected otherwise using
lsof command. It not convenient

Solution:
=========
Implemented a new system table, system.zookeeper_host when CK Server has zk
this table will show the zk node dir which connected by current CK server

Noted: This table can support multi-zookeeper cluster scenario.

* fixed review comments

* added test case

* update test cases

* remove unused code

* fixed review comments and removed unused code

* updated test cases for print host, port and is_expired

* modify the code comments

* fixed CI Failed

* fixed code style check failure

* updated test cases by added Tags

* update test reference

* update test cases

* added system.zookeeper_connection doc

* Update docs/en/operations/system-tables/zookeeper_connection.md

* Update docs/en/operations/system-tables/zookeeper_connection.md

* Update docs/en/operations/system-tables/zookeeper_connection.md

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-05-19 17:06:43 +03:00
Antonio Andelic
acf71c5b9a
Fix typo 2023-05-19 15:48:31 +02:00
Antonio Andelic
3107070e76 Avoid deadlock when starting table in attach thread 2023-05-19 12:48:19 +00:00
Nikolay Degterinsky
a09a8d60c4
Merge branch 'master' into postgresql-uuid 2023-05-19 13:44:17 +02:00
alesapin
e741450b88
Merge branch 'master' into fix_another_zero_copy_bug 2023-05-19 12:40:48 +02:00
alesapin
e5b001abda
Merge branch 'master' into fix_some_tests4 2023-05-19 12:34:03 +02:00
Michael Kolupaev
e84f0895e7 Support hardlinking parts transactionally 2023-05-18 21:05:56 -07:00
Nikolay Degterinsky
ef45956713 Fix style 2023-05-19 01:31:45 +00:00
Nikolay Degterinsky
b8be714830 Add schema inference to more table engines 2023-05-19 00:44:27 +00:00
Nikita Taranov
971cc092d4
Update src/Storages/MergeTree/MergeTreePrefetchedReadPool.cpp 2023-05-18 15:16:47 +02:00
SmitaRKulkarni
a91c793684
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-05-18 09:24:25 +02:00
Kseniia Sumarokova
1c04085e8f
Update MergeTreeWriteAheadLog.h 2023-05-17 18:15:51 +02:00
Han Fei
ed1d036151
Merge pull request #49884 from azat/dist-fix-async-block-processing 
Fix processing pending batch for Distributed async INSERT after restart
 2023-05-17 15:19:42 +02:00
Nikolay Degterinsky
194ce2d881 Better 2023-05-17 13:13:57 +00:00
Nikita Taranov
0dd67bacf2
Merge branch 'master' into optimize_reading2 2023-05-17 15:06:41 +02:00
Alexander Tokmakov
36c31e1d79
Improve concurrent parts removal with zero copy replication (#49630) 
* improve concurrent parts removal

* fix

* fix
 2023-05-17 14:07:34 +03:00
Vitaly Baranov
6c8a923c9d
Merge branch 'master' into write-encrypted-to-backup 2023-05-17 12:37:05 +02:00
Vitaly Baranov
f4ac4c3f9d Corrections after review. 2023-05-17 03:23:16 +02:00
Alexander Tokmakov
242a3fc520 Merge branch 'master' into less_zookeeper_requests 2023-05-16 18:24:11 +02:00
Alexander Tokmakov
0da82945ac fix 2023-05-16 18:18:48 +02:00
Kruglov Pavel
4530f38fdf
Merge branch 'master' into urlCluster 2023-05-16 16:21:23 +02:00
alesapin
50a536bba8 Remove unused code 2023-05-16 15:26:24 +02:00
Alexander Tokmakov
b6716a8f0f Merge branch 'master' into fix_some_tests4 2023-05-16 14:46:27 +02:00
Vitaly Baranov
943707963f Add backup setting "decrypt_files_from_encrypted_disks" 2023-05-16 14:27:27 +02:00
Vitaly Baranov
5198997fd8 Remove ReadSettings from backup entries. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
517e119e03 Move checksum calculation to IBackupEntry. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
c92219f01b BACKUP now writes encrypted data for tables on encrypted disks. 2023-05-16 14:26:33 +02:00
Vitaly Baranov
cc50fcc60a Remove the 'temporary_file_' argument from BackupEntryFromImmutableFile's constructor. 2023-05-16 14:25:37 +02:00
alesapin
93bd09ddd6
Merge branch 'master' into fix_another_zero_copy_bug 2023-05-16 12:24:52 +02:00
alesapin
0b4ab70dd9
Merge pull request #49891 from hanfei1991/hanfei/chassert-1 
use chassert in MergeTreeDeduplicationLog to have better log info
 2023-05-16 11:50:11 +02:00
Sema Checherinda
03c51208d1
Merge pull request #44869 from CheSema/multi_part_upload 
rework WriteBufferFromS3, add tests, add abortion
 2023-05-16 10:52:01 +02:00
Nikolay Degterinsky
d9be88a36a Add UUID data type to PostgreSQL 2023-05-15 21:40:10 +00:00
Robert Schulze
59bc3e25be
Merge pull request #49824 from AVMusorin/allow-alias-column-kafka 
KafkaEngine: Allow usage of Alias column type
 2023-05-15 23:40:03 +02:00
Vitaly Baranov
801cacc13f
Merge pull request #49831 from vitlibar/fix-setting-null-in-profile-def 
Fix setting NULL in profile definition
 2023-05-15 22:24:49 +02:00
Alexander Tokmakov
65bc702b0b fix 2023-05-15 20:02:30 +02:00
Han Fei
4137a5e058 use chassert in MergeTreeDeduplicationLog to have better log info 2023-05-15 18:51:16 +02:00
avogar
334f062fa0 fix style 2023-05-15 16:39:26 +00:00
avogar
f9e0eb47d7 Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster 2023-05-15 16:35:24 +00:00
Alexander Tokmakov
05ae7b2c2d fix some tests 2023-05-15 18:28:12 +02:00
Azat Khuzhin
f2a023140e Fix processing pending batch for Distributed async INSERT after restart 
After abnormal server restart current_batch.txt (that contains list of
files to send to the remote shard), may not have all files, if it was
terminated between unlink .bin files and truncation of current_batch.txt

But it should be fixed in a more reliable way, though to backport the
patch I kept it simple.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-15 15:57:30 +02:00
AVMusorin
418a61a68c
Allow using Alias column type for KafkaEngine 
```
create table kafka
(
 a UInt32,
 a_str String Alias toString(a)
) engine = Kafka;

create table data
(
  a UInt32;
  a_str String
) engine = MergeTree
order by tuple();

create materialized view data_mv to data
(
  a UInt32,
  a_str String
) as
select a, a_str from kafka;
```
Alias type works as expected in comparison with MATERIALIZED/EPHEMERAL
or column with default expression.

Ref: https://github.com/ClickHouse/ClickHouse/pull/47138

Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2023-05-15 15:39:58 +02:00
Alexander Tokmakov
25912a2673
Merge pull request #49876 from JackyWoo/fix_typo 
fix typo
 2023-05-15 13:32:58 +03:00
Kruglov Pavel
558eda4146
Merge pull request #49412 from azat/block-use-dense-hash-map 
Switch Block::NameMap to google::dense_hash_map over HashMap
 2023-05-15 12:22:55 +02:00
JackyWoo
8d1bcb5c2f fix typo 2023-05-15 16:51:20 +08:00
Alexey Milovidov
b4b5b4dcc5 Amend 2023-05-14 02:52:23 +02:00
Alexey Milovidov
0ca36d4f89 Merge branch 'master' into clang-17 2023-05-14 01:57:40 +02:00
Smita Kulkarni
fd58eac75a Fixed max_threads datatype issue for builds 2023-05-13 10:22:37 +02:00
Alexey Milovidov
a2d1cc1333
Merge pull request #49844 from MikhailBurdukov/in_with_tuple 
TYPE_MISMATCH exception for in operator with  single column tuples.
 2023-05-13 04:21:12 +03:00
Alexey Milovidov
5a44dc26e7 Fixes for clang-17 2023-05-13 02:57:31 +02:00
Anton Popov
401030e2be
Merge pull request #49757 from CurtizJ/add-forgotten-lock 
Add forgotten lock (addition to #49117)
 2023-05-12 23:55:09 +02:00
MikhailBurdukov
8c7b63f1e7 Fix 2023-05-12 16:42:01 +00:00
Smita Kulkarni
792565d858 Updated to ULL 2023-05-12 17:23:37 +02:00
avogar
3d00f4cdf6 Remove unneded files 2023-05-12 14:03:04 +00:00
avogar
70a8fd2c50 Fix schema inference with named collection, refactor Cluster table functions 2023-05-12 13:58:45 +00:00
Robert Schulze
b9c185af44
Merge pull request #49678 from azat/build/llvm-16 
Switch to LLVM/clang 16 (16.0.3)
 2023-05-12 13:47:36 +02:00
Sema Checherinda
a1ee7d52e1
Merge pull request #49432 from CheSema/lost-blobs 
all s3-blobs removed when merge aborted, remove part from failed fetch without unlock keper
 2023-05-12 13:19:27 +02:00
Sema Checherinda
5706e05757 adding more comments 2023-05-12 13:00:56 +02:00
Nikita Taranov
536c5ac261
Merge branch 'master' into optimize_reading2 2023-05-12 12:59:32 +02:00
Alexander Gololobov
7c7565f094
Merge pull request #49737 from ClickHouse/fix_lwd_and_object 
Ignore LWD column in checkPartDynamicColumns
 2023-05-12 12:20:07 +02:00
Robert Schulze
922420420c
Merge pull request #49300 from ClickHouse/rs/functdocs 
Introduce more fields for in-source function documentation
 2023-05-12 11:36:04 +02:00
Vitaly Baranov
eb62030fa4 Fix assigning a setting to NULL in settings profile's definition. 2023-05-12 11:04:30 +02:00
Azat Khuzhin
2c40dd6a4c Switch Block::NameMap to google::dense_hash_map over HashMap 
Since HashMap creates 2^8 elements by default, while dense_hash_map
should be good here.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-12 05:52:57 +02:00
Alexey Milovidov
ea979b40a9
Merge pull request #49717 from ClickHouse/disable-mmap-for-server 
Disable mmap for StorageFile in clickhouse-server
 2023-05-11 23:53:01 +03:00
Anton Popov
d21dc829d9
Merge branch 'master' into add-forgotten-lock 2023-05-11 16:56:13 +02:00
Azat Khuzhin
e5c4eb35e8 Merge remote-tracking branch 'u/master' into build/llvm-16 
* u/master:
  enable used flags's reinit only when the hash talbe rehash
  Fix build of libfiu on clang-16
  fix flaky test 02504_regexp_dictionary_ua_parser
  fix convertation
  Fix test that expected CH to apply a wrong optimization
  ActionsDAG: remove wrong optimization
 2023-05-11 16:32:25 +02:00
avogar
3ee8de792c Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster 2023-05-11 12:46:20 +00:00
Smita Kulkarni
ef1100bb90 Added include to fix build issue 2023-05-11 14:44:36 +02:00
Sema Checherinda
8470aa8fbb
Merge branch 'master' into convertations_to_timespan 2023-05-11 08:24:40 +02:00
Sema Checherinda
66e3712dca
Merge branch 'master' into lost-blobs 2023-05-11 08:23:25 +02:00
Sema Checherinda
9724688608 rename field, do not fetch PreActive 2023-05-11 08:22:37 +02:00
Alexey Milovidov
cfd3f76217
Merge branch 'master' into disable-mmap-for-server 2023-05-11 06:17:32 +03:00
Anton Popov
ff287c7bba add lock 2023-05-10 23:00:49 +00:00
zvonand
ecd0b4cc06 update settings from master 2023-05-11 00:37:58 +02:00
Azat Khuzhin
d56e1423ef
Merge branch 'master' into build/llvm-16 2023-05-10 21:23:26 +02:00
avogar
9096f62efc Merge branch 'master' of github.com:ClickHouse/ClickHouse into random-structure 2023-05-10 18:46:19 +00:00
Sema Checherinda
7fbf87be17 rework WriteBufferFromS3, squashed 2023-05-10 18:31:47 +00:00
Alexander Gololobov
049ef9dda5
Merge branch 'master' into fix_lwd_and_object 2023-05-10 19:54:26 +02:00
Sema Checherinda
f2ad1122a1 fix convertation 2023-05-10 17:50:42 +00:00
Nikita Taranov
ec63a6ee7c
Merge branch 'master' into optimize_reading2 2023-05-10 17:24:37 +02:00
Alexey Milovidov
4a4eb5b171
Merge pull request #49660 from CurtizJ/fix-sparse-columns-reload 
Fix reading from sparse columns after restart
 2023-05-10 18:08:22 +03:00
Alexey Milovidov
33a58aacea
Merge pull request #49719 from ClickHouse/make-projections-non-experimental 
Make projections production-ready
 2023-05-10 18:07:02 +03:00
Azat Khuzhin
a0cc5b7e47 Merge remote-tracking branch 'u/master' into build/llvm-16 2023-05-10 16:28:36 +02:00
Alexander Gololobov
e6f4e5cd10 Ignore LWD column in checkPartDynamicColumns 2023-05-10 16:10:04 +02:00
Han Fei
d17aa828b3
Merge pull request #49361 from hanfei1991/hanfei/fiu 
support named fault injection and add a simple test
 2023-05-10 15:51:51 +02:00
zvonand
7762dde325 Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-implicit-tz 2023-05-10 14:49:17 +02:00
Roman Vasin
632c0eeefc
Add support of temporary tables in FROM part of ATTACH|REPLACE PARTITION FROM (#49436) 
* Add temp tables after FROM

* Resolve works fine with temp and non-temp tables

* Add usage of temporary table in  00626_replace_partition_from_table test

* Update documentation

* Add tests for ReplicatedMergeTree

* Move tests into 02731_replace_partition_from_temporary_table
 2023-05-10 15:25:12 +03:00
Azat Khuzhin
2f9a7cba8a Fix stack-use-after-scope in KafkaProducer::produce() 
ASan report:

    ==1==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f7540186b20 at pc 0x55656ef458ad bp 0x7f754114acf0 sp 0x7f754114a4b8
    READ of size 2 at 0x7f7540186b20 thread T2 (TCPHandler)
        0 0x55656ef458ac in __asan_memcpy (/usr/bin/clickhouse+0x11b028ac) (BuildId: 6ba63057df561f6dc16fb8fd7623fd858bf327d1)
        1 0x55659f62daf8 in rd_kafka_msg_new00 build_docker/./contrib/librdkafka/src/rdkafka_msg.c:192:3
        2 0x55659f62daf8 in rd_kafka_msg_new0 build_docker/./contrib/librdkafka/src/rdkafka_msg.c:257:8
        3 0x55659f630635 in rd_kafka_producev build_docker/./contrib/librdkafka/src/rdkafka_msg.c:642:23
        4 0x55659f4a865c in cppkafka::Producer::do_produce() build_docker/./contrib/cppkafka/src/producer.cpp:128:19
        5 0x55659f4a8306 in cppkafka::Producer::produce() build_docker/./contrib/cppkafka/src/producer.cpp:69:5
        6 0x5565961e748c in DB::KafkaProducer::produce() build_docker/./src/Storages/Kafka/KafkaProducer.cpp:70:23
        7 0x5565962155b2 in DB::MessageQueueSink::consume(DB::Chunk) build_docker/./src/Storages/MessageQueueSink.cpp:64:23
        8 0x5565980b9e8e in DB::SinkToStorage::onConsume(DB::Chunk) build_docker/./src/Processors/Sinks/SinkToStorage.cpp:18:5

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-10 12:15:17 +02:00
SmitaRKulkarni
5c030c428c
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-05-10 09:51:38 +02:00
Alexey Milovidov
8a6e07f0ea Make projections production-ready 2023-05-10 03:35:13 +02:00
Alexey Milovidov
a2c4b8e23d Disable mmap for server 2023-05-10 03:16:52 +02:00
Andrey Zvonov
9237b904be
Merge branch 'master' into zvonand-implicit-tz 2023-05-10 01:25:33 +02:00
Anton Popov
996fcfe120 fix column ttl with sparse columns 2023-05-09 21:37:43 +00:00
Alexander Tokmakov
6f53ee289d fix another issue, add a test 2023-05-09 23:26:56 +02:00
Azat Khuzhin
d03ae2abfa Fix modernize-loop-convert clang-tidy warning 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-09 21:19:37 +02:00
Alexey Milovidov
3556c26075
Merge pull request #49434 from ClickHouse/ins 
Make INSERT do more things in parallel to avoid getting bottlenecked on one thread
 2023-05-09 08:10:15 +03:00
Alexey Milovidov
a8bdb20fc4
Merge pull request #48787 from ClickHouse/background-memory-tracker 
Add MemoryTracker for the background tasks [Resubmit]
 2023-05-09 07:58:36 +03:00
Alexey Milovidov
9a73a04fe3
Merge pull request #42587 from ClickHouse/compress-marks-primary-key-by-default 
Compress marks and primary key by default
 2023-05-09 06:43:10 +03:00
Han Fei
64b67b9a37 Merge branch 'master' into hanfei/fiu 2023-05-09 01:14:30 +02:00
Alexander Tokmakov
aa829c4ddc better partitions hint in mutations finalization 2023-05-09 00:23:21 +02:00
Han Fei
82ce47ce56 address comments 2023-05-08 23:02:36 +02:00
Anton Popov
0612f5d782 fix sparse columns after reload 2023-05-08 11:50:21 +00:00
Kseniia Sumarokova
336bb41c5d
Merge branch 'master' into remove-dependency-from-context 2023-05-08 12:46:10 +02:00
Alexey Milovidov
3cf1da7bf3
Merge branch 'master' into deprecate-in-memory-parts 2023-05-08 01:38:00 +03:00
Alexander Tokmakov
91489243c9 provide better partitions hint for merge selecting task 2023-05-08 00:22:07 +02:00
Alexander Tokmakov
29bbde85a6 split selectPartsToMerge into smaller functions 2023-05-07 22:06:25 +02:00
robot-ch-test-poll
5078231c7e
Merge pull request #49619 from ClickHouse/remove_concurrent 
Better concurrent parts removal with zero copy
 2023-05-07 21:03:44 +02:00
alesapin
33e691ad51 Better name 2023-05-07 14:26:21 +02:00
alesapin
a2ed457219 Better concurrent parts removal with zero copy 2023-05-07 14:02:24 +02:00
Alexey Milovidov
6b742bb1d1
Merge branch 'master' into system-stack-trace-fix 2023-05-07 06:05:47 +03:00
Alexey Milovidov
3b6edbacab Update tests 2023-05-07 03:45:17 +02:00
Alexey Milovidov
de3e66385c Merge branch 'master' into compress-marks-primary-key-by-default 2023-05-07 02:40:25 +02:00
Sema Checherinda
08bfc8f58d inserted parts has exlusive blobs 2023-05-07 01:51:43 +02:00
Sema Checherinda
56f2917718
Update StorageReplicatedMergeTree.cpp 2023-05-06 17:59:07 +02:00
Alexey Milovidov
9efbe86110
Merge pull request #49563 from evillique/object-column-alter 
Forbid the change of data type of Object('json') column
 2023-05-06 18:17:16 +03:00
Robert Schulze
983e37048c
Merge pull request #48853 from ClickHouse/rs/qc-totals-extremes2 
QueryCache: Support queries with totals and extremes
 2023-05-06 15:41:10 +02:00
Igor Nikonov
c23600fb56
Merge pull request #49536 from ClickHouse/propa 
Make input_format_parquet_preserve_order imply !parallelize_output_from_storages
 2023-05-06 14:53:36 +02:00
Sema Checherinda
8f5afffaa0 Outdated parts has to be unlocked in general case 2023-05-06 14:23:57 +02:00
Sema Checherinda
f302c1986a use flag has_exclusive_blobs to track the part's originate 2023-05-06 03:07:29 +02:00
Alexey Milovidov
75a2589a42
Merge branch 'master' into deprecate-in-memory-parts 2023-05-06 00:49:59 +03:00
Robert Schulze
3dfc0bd265
Merge pull request #49413 from azat/build/headers 
Slightly reduce inter-header dependencies
 2023-05-05 23:37:58 +02:00
Robert Schulze
45c28e1221
Introduce more fields for in-source function documentation 2023-05-05 21:30:21 +00:00
Nikolay Degterinsky
d29193da51 Also prohibit complex types with Object('json') 2023-05-05 18:45:00 +00:00
Nikolay Degterinsky
f361d96f21 Forbid the change of data type of Object('json') column 2023-05-05 15:51:17 +00:00
Sema Checherinda
efcf7fbe4f
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-05-05 17:49:12 +02:00
Sema Checherinda
36b028bc6b
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-05-05 17:48:57 +02:00
Robert Schulze
aba5db40a1
Merge remote-tracking branch 'origin/master' into rs/qc-totals-extremes2 2023-05-05 14:11:10 +00:00
zvonand
8338d54c34 Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-implicit-tz 2023-05-05 15:50:26 +02:00
Nikolay Degterinsky
7a727bbbe3
Merge pull request #49520 from IvanTakarlikov-st/it-grammar-fixes 
Fix some grammar mistakes in documentation, code and tests
 2023-05-05 15:45:58 +02:00
wangxiaobo
f704c0dfec
Implement SYSTEM DROP REPLICA from auxillary ZooKeeper clusters (#48932) 
* multiple zookeeper drop replica bug fix.

* add an integration test.

* format code.

* set stay_alive=True for test.

* style check bug fix.

* add check for is_active

* format code

* remove table_settings and has_metadata_out param.
 2023-05-05 14:11:11 +03:00
Kseniia Sumarokova
7c203dbcd2
Merge pull request #49481 from snikch/master 
Fix postgres database setting
 2023-05-05 12:53:51 +02:00
Anton Popov
c87a6e8e7d
Merge pull request #49117 from CurtizJ/limit-mutations 
Add settings to delay or throw in case of too many mutations
 2023-05-05 12:18:57 +02:00
Alexey Milovidov
09f171822e Merge branch 'master' into deprecate-in-memory-parts 2023-05-05 06:43:23 +02:00
Michael Kolupaev
3bd1489f18 Propagate input_format_parquet_preserve_order to parallelizeOutputAfterReading() 2023-05-05 04:20:27 +00:00
Alexey Milovidov
ab1a1ea5b7
Merge pull request #49490 from ClickHouse/fix-s3cluster-with-headers 
Correctly handle s3Cluster arguments
 2023-05-05 05:31:23 +03:00
Michael Kolupaev
0fa311ee38 Make INSERT do more things in parallel to avoid getting bottlenecked on one thread 2023-05-05 00:49:27 +00:00
Michael Kolupaev
eb3b774ad0 Better control over Parquet row group size 2023-05-04 14:59:55 -07:00
Sema Checherinda
9a0bc891e4 fix can_remove_callback for Temporary parts 2023-05-04 19:39:16 +02:00
Ivan Takarlikov
8873856ce5 Fix some grammar mistakes in documentation, code and tests 2023-05-04 13:35:18 -03:00
Anton Popov
e61d7baa54 Merge remote-tracking branch 'upstream/master' into HEAD 2023-05-04 14:03:53 +00:00
Alexander Tokmakov
36f8fcdfaf
Merge pull request #49458 from ClickHouse/tavplubix-patch-6 
Follow-up to #48792
 2023-05-04 14:32:18 +03:00
kssenii
1433f5ffc9 Merge remote-tracking branch 'upstream/master' into remove-dependency-from-context 2023-05-04 13:24:02 +02:00
Antonio Andelic
a68a023ca7
Merge pull request #48724 from johanngan/sse-kms 
Support SSE-KMS configuration with S3 client
 2023-05-04 13:20:54 +02:00
Alexander Gololobov
f77cc89464
Merge pull request #49470 from ClickHouse/remove_no_delay_flag 
Replace `NO DELAY` with `SYNC` in tests
 2023-05-04 11:57:01 +02:00
alesapin
412b161104
Merge pull request #48791 from kssenii/better-local-object-storage 
Make local object storage work consistently with s3 object storage, fix problem with append, make it configurable as independent storage
 2023-05-04 11:47:43 +02:00
alesapin
89caf40840
Merge pull request #46528 from myrrc/feature/show-secrets-option 
Show secrets in `SHOW` and `SELECT` queries
 2023-05-04 11:21:30 +02:00
Antonio Andelic
8769ac2187 Correctly append arguments 2023-05-04 08:30:45 +00:00
SmitaRKulkarni
a28da97e87
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-05-04 10:14:11 +02:00
johanngan
731823b873 Add support for SSE-KMS configuration with S3 
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html

Similar to the server_side_encryption_customer_key_base64 option for
configuring SSE-C with S3, add the following settings to configure
SSE-KMS on a per-endpoint/disk basis:
  - server_side_encryption_kms_key_id
  - server_side_encryption_kms_encryption_context
  - server_side_encryption_kms_bucket_key_enabled
 2023-05-03 21:35:38 -05:00
Mal Curtis
5727585d95 s/use_tables_cache/use_table_cache 2023-05-03 23:40:12 +00:00
Alexey Milovidov
261907c143 Merge branch 'master' into deprecate-in-memory-parts 2023-05-04 01:07:38 +02:00
alesapin
610e63bfd2 Fix build 2023-05-03 23:54:24 +02:00
Alexander Gololobov
a90c2ec90d
Merge pull request #49425 from ClickHouse/fix_build_set_with_remote 
Fix fuzz bug when subquery set is not built when reading from remote()
 2023-05-03 21:47:29 +02:00
wangxiaobo
2924837c7e
Fix Attaching a table with non-existent ZK path does not increase the ReadonlyReplica metric (#48954) 
* Attaching a table with non-existent ZK path does not increase the ReadonlyReplica metric bug fix.

* add integration test.

* add integration test.

* Update test.py

* format code

* format code
 2023-05-03 22:41:20 +03:00
alesapin
913b63edc9 Fix another zero copy bug 2023-05-03 21:28:33 +02:00
Alexander Tokmakov
83b12ec5ca replace NO DELAY with SYNC in tests 2023-05-03 20:08:49 +02:00
Anton Popov
c9705d6bf1
Merging #49066 (Better error handling during loading of parts) (#49430) 
* catch filesystem error when loading parts

* better exception handling during part loading

---------

Co-authored-by: cangyin <excangyin@gmail.com>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-05-03 19:10:06 +02:00
Nikita Mikhaylov
954e3b724c
Speedup outdated parts loading (#49317) 2023-05-03 18:56:45 +02:00
Alexander Tokmakov
ff70d26c23
Update StorageMergeTree.cpp 2023-05-03 19:30:44 +03:00
Alexander Tokmakov
e399903030
Merge pull request #48548 from ClickHouse/clusters_is_active_column 
Add some columns to system.clusters
 2023-05-03 17:42:40 +03:00
Alexander Tokmakov
f3f6cfdcbb
Update StorageMergeTree.cpp 2023-05-03 17:39:31 +03:00
Anton Popov
7896d30737
Merge pull request #49122 from CurtizJ/add-async-insert-mt-setting 
Add `MergeTree` setting `async_insert`
 2023-05-03 16:01:13 +02:00
Nikita Taranov
30d216f863 fix 2023-05-03 13:49:46 +00:00
kssenii
35f437ac9c Address review comments 2023-05-03 14:37:18 +02:00
kssenii
189f276ff5 Fix 2023-05-03 13:16:08 +02:00
alesapin
6f3f202f7c
Merge branch 'master' into fix_zero_copy_not_atomic 2023-05-03 12:53:35 +02:00
Nikolai Kochetov
6388be2062
Merge pull request #49326 from k-morozov/refactoring/pipe-into-step 
move pipe compute into initializePipeline
 2023-05-03 12:43:21 +02:00
Alexey Milovidov
ee9fae6aa2
Merge pull request #49363 from tsolodov/master 
Disallow configuring compression CODECs for alias columns
 2023-05-03 03:37:48 +03:00
Alexey Milovidov
f449df85b6 Deprecate in-memory parts 2023-05-03 00:31:09 +02:00
kssenii
d8d0d78db8 Fix build 2023-05-03 00:09:52 +02:00
kssenii
ecfbf1e304 Remove dependency from DB::Context in readers 2023-05-02 21:45:27 +02:00
Alexander Gololobov
2e20f2a14d Do not skip building set even when reading from remote 2023-05-02 21:31:56 +02:00
Azat Khuzhin
2b21b83979 Reduce inter-header dependencies 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-02 21:15:18 +02:00
Smita Kulkarni
8205398f31 Fixed comment 2023-05-02 16:31:39 +02:00
Nikita Taranov
67bd861bc3 Merge branch 'master' into optimize_reading2 2023-05-02 14:23:56 +00:00
alesapin
a08225d9ea Fix typo 2023-05-02 15:34:57 +02:00
SmitaRKulkarni
997210c782
Merge pull request #48515 from ClickHouse/Clear_marks_cache_for_outdated_parts 
MergeTreeMarksLoader holds DataPart instead of DataPartStorage
 2023-05-02 15:11:14 +02:00
Konstantin Morozov
e4394debd4 forgot about query_plan 2023-05-02 14:59:41 +02:00
Konstantin Morozov
397a1700c7 apply comments 2023-05-02 14:59:41 +02:00
kst-morozov
f0619910d4 move pipe compute into initializePipeline 2023-05-02 14:59:41 +02:00
alesapin
24a744db4c Merge branch 'master' into fix_zero_copy_not_atomic 2023-05-02 14:57:55 +02:00
alesapin
619b6f2071 Fix typo 2023-05-02 14:57:52 +02:00
Nikita Mikhaylov
aa4c5fe958
Enhancements for background merges (#49313) 2023-05-02 13:43:59 +02:00
Anton Popov
bcdd2307fb add comments 2023-05-02 11:30:50 +00:00
Antonio Andelic
f695040a19
Merge pull request #48897 from ClickHouse/keeper-check-not-exists 
Add `CheckNotExists` request to Keeper
 2023-05-02 13:14:22 +02:00
Vladimir C
96d109c3db
Merge pull request #49137 from ClickHouse/vdimir/mutliple_storage_join 2023-05-02 11:40:16 +02:00
Ziy1-Tan
1bb0d1519e Fix style 
Signed-off-by: Ziy1-Tan <ajb459684460@gmail.com>
 2023-05-02 16:54:14 +08:00
vitac
5aa9dae09b
Merge branch 'master' into vcol 2023-05-02 15:36:26 +08:00
Ziy1-Tan
c93ceedbef Fix style 
Signed-off-by: Ziy1-Tan <ajb459684460@gmail.com>
 2023-05-02 10:38:37 +08:00
Timur Solodovnikov
e58a63b451 addressing PR comments 2023-05-01 16:13:45 -07:00
Han Fei
d608ba41e7 try to fix 2023-05-02 00:05:36 +02:00
Timur Solodovnikov
c7295db43e nit 2023-05-01 13:59:38 -07:00
Timur Solodovnikov
6566ed7863 removed loop & added tests 2023-05-01 13:51:46 -07:00
kssenii
fbdccbd25f Merge remote-tracking branch 'origin/better-local-object-storage' into better-local-object-storage 2023-05-01 22:25:28 +02:00
kssenii
f325ba1794 Better 2023-05-01 21:10:37 +02:00
alesapin
0f6a4b973e
Merge branch 'master' into ttl_on_insert 2023-05-01 21:02:04 +02:00
alesapin
a3f9a8b87b Fix bug in removal of existing part directory 2023-05-01 20:32:01 +02:00
Nikita Taranov
979b68a488 fix 2023-05-01 18:24:30 +00:00
Nikita Mikhaylov
503f099afe
Merge branch 'master' into master 2023-05-01 18:50:44 +02:00
Timur Solodovnikov
fc5b0783fd fix for linter 2023-05-01 09:40:56 -07:00
Timur Solodovnikov
e49842ff6c block setting codec for alias columns 2023-05-01 09:25:08 -07:00
Han Fei
4d18a4f8d3
Merge branch 'master' into hanfei/fiu 2023-05-01 17:51:41 +02:00
Han Fei
b701293d46 support named fault injection and add a simple test 2023-05-01 17:44:37 +02:00
alesapin
66847386df
Merge pull request #49146 from ClickHouse/don_t_write_version_file_on_start 
Don't add metadata_version file if it doesn't exist
 2023-05-01 17:04:47 +02:00
Anton Popov
36d53e0d6c
Merge branch 'master' into add-async-insert-mt-setting 2023-05-01 16:13:23 +02:00
Ziy1-Tan
2c159061ed Support _path and _file virtual columns for table function url. 2023-05-01 21:40:30 +08:00
Alexander Gololobov
c1c6955374 Deprecate delete-on-destroy.txt, do not create it any more 2023-05-01 14:47:20 +02:00
Duc Canh Le
abe0cfd10f
fix wrong query result when using nullable primary key (#49172) 2023-05-01 12:51:22 +02:00
alesapin
f03e75971a Remove on detach 2023-04-30 19:06:31 +02:00
alesapin
a098298c0f Remove file for empty parts 2023-04-30 19:06:23 +02:00
Smita Kulkarni
a5d47ea489 Fixed build issues 2023-04-30 19:01:06 +02:00
alesapin
a95a5f6fb6 Fix two bugs and add test 2023-04-30 16:33:23 +02:00
alesapin
ec2c860a39 Merge branch 'fix_zero_copy_not_atomic' of github.com:ClickHouse/ClickHouse into fix_zero_copy_not_atomic 2023-04-30 15:02:27 +02:00
Smita Kulkarni
408db4a25b Updated to store IMergeTreeDataPartInfoForReader instead of DataPart in MergeTreeMarksLoader 2023-04-30 10:56:43 +02:00
alesapin
daa0b095ed Merge branch 'master' into fix_zero_copy_not_atomic 2023-04-29 21:26:26 +02:00
Alexey Milovidov
15d0379e2c
Merge pull request #48792 from ClickHouse/fix_some_tests3 
Fix some tests
 2023-04-29 22:04:11 +03:00
Nikita Taranov
f1ff2c0135
Fix oss-fuzz build errors (#49236) 2023-04-29 18:44:50 +02:00
Alexey Milovidov
5bd4fc9e90
Merge pull request #48492 from azat/s3-multipart-strict-size 
Add ability to use strict parts size for S3 (compatibility with R2)
 2023-04-28 23:34:45 +03:00
Kseniia Sumarokova
d4aa96e262
Merge branch 'master' into better-local-object-storage 2023-04-28 21:05:05 +02:00
Mike Kot
2d20658885
Merge branch 'master' into feature/show-secrets-option 2023-04-28 20:19:35 +03:00
alesapin
70ee02a3eb Add node holder 2023-04-28 17:39:32 +02:00
Alexey Milovidov
911d4e9977
Merge branch 'master' into fix_some_tests3 2023-04-28 17:23:46 +03:00
Alexander Tokmakov
691fdbf6a6
Update StorageReplicatedMergeTree.cpp (#49302) 2023-04-28 17:21:03 +03:00
Alexey Milovidov
6dcfdbfaf2
Merge branch 'master' into clusters_is_active_column 2023-04-28 16:16:25 +03:00
alesapin
ef49603432 Merge branch 'master' into fix_zero_copy_not_atomic 2023-04-28 14:42:30 +02:00
Nikita Taranov
71a092d201 Merge branch 'master' into optimize_reading2 2023-04-28 12:29:27 +00:00
kssenii
e603adfd4c Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-28 13:26:57 +02:00
kssenii
298ed454ce Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-28 13:26:55 +02:00
Raúl Marín
bb4a7f65de
Slight improvements to coordinator logging (#49204) 2023-04-28 13:25:53 +02:00
Kseniia Sumarokova
107474a42f
Merge pull request #44985 from kssenii/cache-better-locks 
Rework locking in fs cache
 2023-04-28 12:24:55 +02:00
Azat Khuzhin
c8ec152f57 Add ability to use strict parts size for S3 (compatibility with R2) 
The are some implementations that does not allow to use parts of
different size for multi-part upload:

- CouldFlare R2 [1] and also got help on discord
- AWS S3 Glacier [2]
- and I also know of one bug in Google Cloud Storage (GCS), but it
  should be fixed in a couple of months.

  [1]: https://developers.cloudflare.com/r2/reference/changelog/#2022-10-28
       > Multipart upload part sizes are always expected to be of the
       > same size, but this enforcement is now done when you complete
       > an upload instead of being done very time you upload a part

  [2]: https://docs.aws.amazon.com/amazonglacier/latest/dev/api-multipart-initiate-upload.html

On top of this I'm going to add auto detection of this issue for disks,
but not for S3 function, since those checks are quite costly, and they
need to write at least 5MB*3, which may be questionable.

v2: mark test as long [1]
    [1]: https://s3.amazonaws.com/clickhouse-test-reports/48492/d4a90880e947ab3a17b3eddeea274afed8ebceb4/stateless_tests_flaky_check__asan_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-28 11:01:56 +02:00
Mike Kot
357cde95e9 Merge branch 'master' into feature/show-secrets-option 2023-04-27 20:19:40 +00:00
Nikita Taranov
de03b90570 impl 2023-04-27 17:07:00 +00:00
Robert Schulze
b6e65ddeff
Merge branch 'master' into add-examples-to-functions 2023-04-27 16:11:32 +02:00
Kseniia Sumarokova
45e2d296f9
Merge branch 'master' into better-local-object-storage 2023-04-27 14:54:04 +02:00
Alexey Milovidov
3fa19c5ee2
Merge branch 'master' into fix_some_tests3 2023-04-27 15:04:15 +03:00
Alexey Milovidov
aac037f51c
Merge pull request #46765 from zk-kiger/improve_cluster_table_funcion_skip_unavailable_shards 
add skip_unavailable_shards setting for table function cluster
 2023-04-27 14:54:18 +03:00
Alexander Tokmakov
fc9b2fbac8
Fix race on Outdated parts loading (#49223) 2023-04-27 13:52:09 +02:00
kssenii
3d4413183f Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-27 13:16:36 +02:00
Robert Schulze
096667be45
More fixes 2023-04-27 09:33:52 +00:00
Alexey Milovidov
0da0592f86
Merge pull request #49237 from ClickHouse/23.5-prepare 
Update version after release
 2023-04-27 02:47:22 +03:00
Alexey Milovidov
7333e964bc Update autogenerated version to 23.5.1.1 and contributors 2023-04-27 01:40:57 +02:00
Alexey Milovidov
354cf09892
Merge pull request #48549 from cangyin/add_zookeeper_name 
add zookeeper name in system.replicas
 2023-04-27 01:22:22 +03:00
Alexey Milovidov
3920eb987f
Merge pull request #49101 from ClickHouse/parallelize-output-setting 
Setting parallelize_output_from_storages
 2023-04-27 00:51:24 +03:00
Robert Schulze
d72030b876
Various fixes 2023-04-26 20:38:41 +00:00
DanRoscigno
fb39dc3f19 add some markdown 2023-04-26 14:36:34 -04:00
DanRoscigno
3e1e86cff2 examples added 2023-04-26 14:26:49 -04:00
alesapin
a34e465abf Fxi 2023-04-26 19:57:18 +02:00
alesapin
f3d1d3b58b Merge branch 'master' into fix_zero_copy_not_atomic 2023-04-26 18:00:31 +02:00
DanRoscigno
57164820ef fixed column to string 2023-04-26 11:53:50 -04:00
DanRoscigno
ddd285cda2 compiles, but table has map and table is empty 2023-04-26 11:37:10 -04:00
DanRoscigno
c480f81f94 add: map header, return 2023-04-26 09:50:23 -04:00
Robert Schulze
9bed1ef0aa
Merge branch 'master' into hlee-s390x-part-hash 2023-04-26 13:27:36 +02:00
Alexander Tokmakov
6434fd8a47
fix prepared statement insert already exists (#49154) 2023-04-26 14:06:10 +03:00
Alexey Milovidov
23132fbdbc
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards 2023-04-26 13:54:39 +03:00
Alexey Milovidov
ea34df25f4
Merge branch 'master' into add_zookeeper_name 2023-04-26 13:53:57 +03:00
Alexey Milovidov
c3ec6de0b2
Merge branch 'master' into parallelize-output-setting 2023-04-26 13:51:14 +03:00
Alexey Milovidov
815767a064
Merge branch 'master' into vdimir/mutliple_storage_join 2023-04-26 13:44:22 +03:00
Alexey Milovidov
57b8e74cea
Merge branch 'master' into Avogar-patch-3 2023-04-26 13:43:55 +03:00
Alexey Milovidov
cf097a9c0b
Merge pull request #49125 from azat/build/iceberg-avro 
Fix building iceberg without avro
 2023-04-26 13:41:50 +03:00
Alexey Milovidov
4dceee0dc3
Merge branch 'master' into rs/qc-totals-extremes2 2023-04-26 13:40:12 +03:00
Robert Schulze
90f74ccb8d
Merge branch 'master' into hlee-s390x-part-hash 2023-04-26 10:33:01 +02:00
alesapin
e5f71c3b74 Fix zero copy not atomic 2023-04-26 00:25:04 +02:00
Kseniia Sumarokova
b92234ed90
Merge branch 'master' into fix-postgres-setting 2023-04-25 23:55:13 +02:00
Igor Nikonov
928eb7433f Merge remote-tracking branch 'origin/master' into parallelize-output-setting 2023-04-25 21:52:15 +00:00
Robert Schulze
bc313184a5
Merge branch 'master' into hlee-s390x-part-hash 2023-04-25 18:56:40 +02:00
Alexander Tokmakov
fde55f3ea6 fix 2023-04-25 18:25:10 +02:00
alesapin
3a24f0b6c6 fxi 2023-04-25 15:33:41 +02:00
kssenii
cbf0b98102 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-25 15:15:22 +02:00
vdimir
7731ea7905
Fix key not found error for queries with multiple StorageJoin 2023-04-25 11:28:54 +00:00
Smita Kulkarni
307aa127d4 Updated to calculate and send max_parsing_threads 2023-04-25 13:27:20 +02:00
Robert Schulze
1c41967906
Merge remote-tracking branch 'origin/master' into hlee-s390x-part-hash 2023-04-25 09:11:00 +00:00
Azat Khuzhin
7ec2de0d3f Fix building iceberg without avro 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-25 08:42:42 +02:00
cangyin
623c8db1a1 add column zookeeper_name in system.replicas 2023-04-25 09:19:36 +08:00
Anton Popov
c7db58be66 add MergeTree setting async_insert 2023-04-25 01:11:58 +00:00
Robert Schulze
b565257a39
Support totals and extremes in query cache 2023-04-24 22:23:27 +00:00
Alexey Milovidov
1a42148262
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards 2023-04-25 00:44:58 +03:00
Robert Schulze
6ba6e24a5a
Merge pull request #48536 from lingtaolf/feature/support_settings_allow_suspicious_indices 
Reject identical expressions in indices (allow_suspicious_indices)
 2023-04-24 22:27:59 +02:00
Kruglov Pavel
75a3b6c322
Fix build 2023-04-24 21:08:53 +02:00
Anton Popov
38622d0770 add settings to delay or throw in case of too many mutations 2023-04-24 18:21:49 +00:00
Smita Kulkarni
b70878aa0e Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-04-24 19:39:36 +02:00
Kruglov Pavel
8ff864cd8b
Fix 2023-04-24 19:12:50 +02:00
avogar
c503f6532c Add more finalize() to avoid terminate 2023-04-24 15:11:36 +00:00
Alexander Tokmakov
19f43ee7c7 Merge branch 'master' into fix_some_tests3 2023-04-24 16:13:27 +02:00
Alexander Tokmakov
332a736a59 Merge branch 'master' into clusters_is_active_column 2023-04-24 15:54:03 +02:00
avogar
2949ceced1 Fix adding structure to cluster table functions, make it better 2023-04-24 13:20:04 +00:00
kssenii
277393fd8d Fix 2023-04-24 14:45:41 +02:00
Igor Nikonov
481d0a9dc5 Setting parallelize_output_from_storages 2023-04-24 12:22:45 +00:00
alesapin
516a0c9784
Merge pull request #48862 from ClickHouse/fix-zero-copy-replication-drop-detached-parts 
Fix unexpected part name error when trying to drop a ignored detached part with zero copy replication
 2023-04-24 12:57:25 +02:00
Robert Schulze
744e7c7601
Some fixups 2023-04-24 09:43:36 +00:00
Robert Schulze
ded8eca041
Merge pull request #48284 from ClickHouse/rs/qc-quota 
Query Cache: Allow per-user quotas
 2023-04-24 11:32:32 +02:00
Alexey Milovidov
ff5ec30b0b
Merge pull request #49005 from ClickHouse/fix-ibm 
Reimplement #48986
 2023-04-24 01:39:22 +03:00
Alexey Milovidov
3ae7bf754b Merge branch 'master' into compress-marks-primary-key-by-default 2023-04-23 22:46:13 +02:00
Alexey Milovidov
67de39c2d9
Merge pull request #48727 from ClickHouse/parallel-processing-from-storages 
Parallelize query processing right after reading FROM ...
 2023-04-23 23:10:32 +03:00
Robert Schulze
38d5f12a32
Query Cache: Implement per-user quotas 2023-04-23 16:13:51 +00:00
Nikita Taranov
54c1b73b5d
Revert "Adapt marks count for prefetch read pool" 2023-04-23 15:33:32 +02:00
Alexey Milovidov
31000978f4
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards 2023-04-23 16:18:48 +03:00
Alexey Milovidov
3621a4d7ac Merge branch 'master' into fix-ibm 2023-04-23 14:26:15 +02:00
kssenii
d8023806a9 Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-23 12:39:34 +02:00
Kseniia Sumarokova
7f6488cf37
Merge pull request #49051 from nickitat/adapt_for_prefetch_read_pool 
Adapt marks count for prefetch read pool
 2023-04-23 12:12:47 +02:00
Kseniia Sumarokova
be0cb808de
Merge pull request #48973 from kssenii/randomize-fs-read-settings 
Randomize prefetch settings in ci
 2023-04-23 12:04:30 +02:00
Alexey Milovidov
28e03f5781 Merge branch 'master' into fix-ibm 2023-04-23 06:01:41 +02:00
凌涛
08e81d0a52
Merge branch 'master' into feature/support_settings_allow_suspicious_indices 2023-04-23 10:08:00 +08:00
Nikita Taranov
d99c227b0d impl 2023-04-22 21:13:23 +00:00
kssenii
6054fc4c99 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-22 17:17:19 +02:00
kssenii
1fbaaf0445 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-22 17:14:50 +02:00
kssenii
e0eb88f9bc Fix clang-tidy 2023-04-22 17:02:17 +02:00
kssenii
475a893be3 Fix, add assertions 2023-04-21 21:52:17 +02:00
avogar
447189a6ca Better 2023-04-21 17:54:09 +00:00
avogar
0097230611 Better 2023-04-21 17:35:17 +00:00
avogar
0805b517ee Fix parsing failover options 2023-04-21 17:28:14 +00:00
avogar
6277e10f29 Remove some trash 2023-04-21 17:26:42 +00:00
avogar
944f54aadf Finish urlCluster, refactor code, reduce code duplication 2023-04-21 17:24:37 +00:00
Kseniia Sumarokova
bd748045ad
Fix typo 2023-04-21 18:54:23 +02:00
Yakov Olkhovskiy
334b052879
Merge pull request #48873 from ClickHouse/fix-window-view-schedule 
Fix unnecessary multiple rescheduling in StorageWindowView
 2023-04-21 08:34:25 -04:00
Alexey Milovidov
bd64a4a6fc
Merge pull request #48904 from ClickHouse/fix_set_withzk_and_analyzer 
Fix segfault when set is not built yet
 2023-04-21 15:14:23 +03:00
avogar
c949f0ebf5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster 2023-04-21 14:13:33 +02:00
avogar
86686fbbc3 Fix conflicts 2023-04-21 14:11:18 +02:00
Michael Lex
5f218353ed Add comment to explain why it's safe to assume format version 1 2023-04-21 14:01:38 +02:00
kssenii
e58c16318f Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-21 12:49:40 +02:00
robot-clickhouse-ci-1
32ac001448
Merge pull request #48838 from amosbird/issue_48735 
Fix key condition on duplicate primary keys
 2023-04-21 12:46:50 +02:00
Alexey Milovidov
252c755323 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-ibm 2023-04-21 12:39:13 +02:00
Alexey Milovidov
530b764953 Fix IBM 2023-04-21 12:38:45 +02:00
kssenii
0ccc3afe74 Fix 2023-04-21 12:26:28 +02:00
凌涛
797d5bbfe7 modify code 2023-04-21 18:08:14 +08:00
凌涛
de18bf72b4 modify code 2023-04-21 17:27:26 +08:00
Smita Kulkarni
aa9635e35b Removed is_part_outdated flag & usage, updated MergeTreeMarksLoader to hold data_part instead of data_part_storage 2023-04-21 11:24:02 +02:00
Alexey Milovidov
8a92eb0cf1
Update src/Storages/IStorage.h 2023-04-21 11:50:07 +03:00
Kruglov Pavel
9749448eb0
Merge pull request #47229 from Avogar/non-blocking-connect 
Improve async reading from socket
 2023-04-21 10:38:50 +02:00
kssenii
88784568dc Better 2023-04-20 23:24:04 +02:00
Andrey Zvonov
984e8fde41
Merge branch 'master' into zvonand-implicit-tz 2023-04-20 23:13:17 +02:00
kssenii
6edb61015d Fix style check 2023-04-20 19:35:49 +02:00
kssenii
0a34f3bcea Merge remote-tracking branch 'upstream/master' into randomize-fs-read-settings 2023-04-20 19:04:01 +02:00
kssenii
6552ec6afc Small improvement 2023-04-20 18:15:09 +02:00
Yakov Olkhovskiy
d34ee81be6 use Poco::Timestamp() instead of std::time 2023-04-20 14:36:54 +00:00
Smita Kulkarni
780eef674f Merge branch 'master' into Clear_marks_cache_for_outdated_parts 2023-04-20 16:17:26 +02:00
kssenii
59211c490a More correct accounting of limits 2023-04-20 15:43:14 +02:00
alesapin
d8e6c6ab88
Merge pull request #48731 from ClickHouse/limit_mutations_per_entry 
Limit the number of mutations commands per `MUTATE_PART` entry
 2023-04-20 15:05:46 +02:00
kssenii
ec1370b2ed Fix 2023-04-20 14:46:10 +02:00
Antonio Andelic
09156ab2b7 Merge branch 'master' into keeper-check-not-exists 2023-04-20 12:07:16 +00:00
Han Fei
6805edf9e6
Merge pull request #48850 from sichenzhao/add_alias_for_parts_name 
add aliases for parts name and names
 2023-04-20 13:04:11 +02:00
Vladimir C
ad516d2349
Merge pull request #48795 from ClickHouse/vdimir/fix-cluster-discovery-47176 2023-04-20 11:56:07 +02:00
Kseniia Sumarokova
9e481f5faa
Merge pull request #48845 from kssenii/fix-data-race-in-rabbitmq 
Fix data race in StorageRabbitMQ
 2023-04-20 11:43:18 +02:00
vdimir
d6416a1321 Store clusters from ClusterDiscovery in separate map 2023-04-19 16:44:54 +02:00
kssenii
323caae3c0 Fix 2023-04-19 16:29:38 +02:00
Kruglov Pavel
2ad161d2b7
Merge branch 'master' into non-blocking-connect 2023-04-19 13:39:40 +02:00
larryluogit
5fc7a489ae
Merge branch 'master' into noParquet 2023-04-18 21:17:00 -04:00
Igor Nikonov
d5eb65b5ea Remove redundant narrowPipe() 2023-04-18 22:41:28 +00:00
Igor Nikonov
2455334009 Merge remote-tracking branch 'origin/master' into parallel-processing-from-storages 2023-04-18 21:42:51 +00:00
Antonio Andelic
fb16623d48 Add CheckNotExists request to Keeper 2023-04-18 14:45:24 +00:00
Alexander Gololobov
f7fe747ddc Fix segfault 2023-04-18 16:30:58 +02:00
kssenii
16b027ed0c Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-18 16:25:08 +02:00
kssenii
59459cb287 Fix test 2023-04-18 15:54:36 +02:00
Anton Popov
c06b580890
Merge pull request #48844 from CheSema/wait_loading_parts 
fix the race wait loading parts
 2023-04-18 14:58:20 +02:00
kssenii
67d695580a Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-18 12:42:25 +02:00
Larry Luo
b929547ab2 fix build without parquet 2023-04-17 21:37:32 -07:00
凌涛
a975211185 add more tests 2023-04-18 10:26:35 +08:00
Yakov Olkhovskiy
55389def04
use Poco::Timestamp() instead of std::time 2023-04-17 19:29:02 -04:00
Nikita Mikhaylov
c04d632373 Fix style 2023-04-18 00:22:52 +02:00
Alexander Gololobov
ba5ca15c40
Merge pull request #46835 from ClickHouse/reduce_mem_in_mutation_with_subquery 
Reduce memory consumption by mutations with big subqueries used with IN
 2023-04-17 23:55:08 +02:00
Robert Schulze
d658305554
Merge pull request #47693 from azat/stringstream-INT_MAX 
Fix crash for uploading parts which size is greater then INT_MAX to S3
 2023-04-17 23:08:56 +02:00
alesapin
ca8f3a10b0
Update src/Storages/StorageReplicatedMergeTree.cpp 2023-04-17 20:20:39 +02:00
kssenii
cd25d61795 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-17 19:47:43 +02:00
robot-clickhouse-ci-2
45f4a5f74c
Merge pull request #47964 from ClickHouse/fast-parquet 
Read Parquet files faster
 2023-04-17 19:27:38 +02:00
kssenii
1322ad53b8 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-17 18:22:21 +02:00
Michael Lex
51c9ae05fb Fix unexpected part name error when trying to drop a ignored detache partition with zero copy replication 
Before this fix, the integration test failed with this error

E           Code: 233. DB::Exception: Received from 172.22.0.9:9000. DB::Exception: Unexpected part name: ignored_all_0_0_0 for format version: 1. Stack trace:
E
E           0. /home/ubuntu/clickhouse/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x19bc7d02 in /usr/bin/clickhouse
E           1. /home/ubuntu/clickhouse/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x19bc7ccd in /usr/bin/clickhouse
E           2. /home/ubuntu/clickhouse/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x3194fae0 in /usr/bin/clickhouse
E           3. /home/ubuntu/clickhouse/src/Common/Exception.cpp:89: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x2268ae6e in /usr/bin/clickhouse
E           4. /home/ubuntu/clickhouse/src/Common/Exception.h:54: DB::Exception::Exception(String&&, int, bool) @ 0x19bbb00a in /usr/bin/clickhouse
E           5. /home/ubuntu/clickhouse/src/Common/Exception.h:81: DB::Exception::Exception<String const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>&>::type>, String const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>&) @ 0x2c4f318c in /usr/bin/clickhouse
E           6. /home/ubuntu/clickhouse/src/Storages/MergeTree/MergeTreePartInfo.cpp:23: DB::MergeTreePartInfo::fromPartName(String const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>) @ 0x2c4f0867 in /usr/bin/clickhouse
E           7. /home/ubuntu/clickhouse/src/Storages/StorageReplicatedMergeTree.cpp:8305: DB::(anonymous namespace)::getParentLockedBlobs(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, String const&, String const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>, Poco::Logger*) @ 0x2bca90f7 in /usr/bin/clickhouse
 2023-04-17 15:57:41 +00:00
kssenii
b77e9c1ef0 Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-17 16:44:10 +02:00
kssenii
d2c73a5522 Better 2023-04-17 16:41:21 +02:00
Sichen Zhao
eb4c26d81f add aliases for parts name and names 2023-04-17 13:18:40 +00:00
凌涛
d1a14900ea remove unused h 2023-04-17 20:26:31 +08:00
凌涛
7ed4c413e3 add settings allow_suspicious_indices 2023-04-17 20:23:19 +08:00
kssenii
0b62354261 Fix data race, refactor 2023-04-17 13:40:53 +02:00
Amos Bird
97d27c87a6
Fix key condition on duplicate primary keys 
Fix key condition usage when encountering duplicate primary keys. This can happen in projections. This bug was introduced in https://github.com/ClickHouse/ClickHouse/pull/45686.

This PR is also an optimization because we don't need to create unused FieldRefs anymore.
 2023-04-17 19:13:42 +08:00
Sema Checherinda
daa55ae0d2 fix the race wait loading parts 2023-04-17 13:01:28 +02:00
Kseniia Sumarokova
8bc0a3a899
Merge pull request #48759 from kssenii/fix-mysql-named-collection 
Fix some settings of mysql dictionary with named collection
 2023-04-17 11:18:04 +02:00
Azat Khuzhin
f5c3f3a4ab Remove quirk for s3_max_single_part_upload_size (when it exceed INT_MAX) 
Reverts: #48816
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-17 10:49:55 +02:00
Michael Kolupaev
473f212c82 Hopefully fix assertion failure in CachedOnDiskReadBufferFromFile 2023-04-17 04:58:32 +00:00
Michael Kolupaev
87be78e6de Better 2023-04-17 04:58:32 +00:00
Michael Kolupaev
e133633359 Parallel decoding with one row group per thread 2023-04-17 04:58:32 +00:00
Michael Kolupaev
683077890f Highly questionable refactoring (getInputMultistream() nonsense) 2023-04-17 04:58:32 +00:00
Michael Kolupaev
2d4fe85513 Something 2023-04-17 04:58:32 +00:00
Kseniia Sumarokova
54f660bd4f
Merge pull request #48805 from kssenii/simplify-cache-factory 
Simplify FileCacheFactory
 2023-04-16 19:47:01 +02:00
robot-ch-test-poll1
69cac0a788
Merge pull request #48816 from ClickHouse/fix-s3 
Prevent a bug in AWS S3
 2023-04-16 14:46:02 +02:00
Alexey Milovidov
8e0b1eae69
Update StorageS3Settings.cpp 2023-04-16 08:53:52 +03:00
Igor Nikonov
908ad29106 Do not parallelize output for zeroes() 2023-04-15 22:14:18 +00:00
Alexey Milovidov
1174f4dcbe Prevent a bug in AWS S3 2023-04-15 23:56:49 +02:00
Alexey Milovidov
7f620bf852 Fix double whitespace in exception message 2023-04-15 23:54:10 +02:00
kssenii
0517de58d1 Simplify FileCacheFactory 2023-04-15 16:15:20 +02:00
Igor Nikonov
8603807b57 Use generic way to parallelize output for file() 
+ disable parallelization for storage Null
 2023-04-15 12:35:24 +00:00
Igor Nikonov
60dbb7b406 Merge remote-tracking branch 'origin/master' into parallel-processing-from-storages 2023-04-14 20:11:51 +00:00
Igor Nikonov
9e92c26d53 Fix test_storage_mysql/test.py::test_settings_connection_wait_timeout 2023-04-14 20:10:57 +00:00
Alexander Tokmakov
70c1000c22 fix some tests 2023-04-14 19:42:17 +02:00
Dmitry Novik
cf5d9a175a Revert "Merge pull request #48760 from ClickHouse/revert-46089-background-memory-tracker" 
This reverts commit a61ed33223, reversing
changes made to 5f01b8a2b5.
 2023-04-14 16:34:19 +02:00
Alexander Gololobov
a7b05589b6 Caches for multiple mutations 2023-04-14 16:12:19 +02:00
Alexander Gololobov
7a381e9232 Added enable_sharing_sets_for_mutations setting 2023-04-14 16:12:18 +02:00
Alexander Gololobov
633aaa73aa Use use_index_for_in_with_subqueries_max_values setting to limit sizes of sets used in KeyCondition 2023-04-14 16:12:18 +02:00
Alexander Gololobov
28d0cc974a Cleanups 2023-04-14 16:12:18 +02:00
Alexander Gololobov
adbe87e9d3 Fixes 2023-04-14 16:12:18 +02:00
Alexander Gololobov
0b9579bfb8 Reworked, testing 2023-04-14 16:12:18 +02:00
Alexander Gololobov
02876ed87d Share PreparedSet-s between task of the same mutation to reduce mem usage 2023-04-14 16:12:18 +02:00
Dmitry Novik
352a240a20
Merge pull request #47984 from ClickHouse/optimize-compilation 
Optimize compilation
 2023-04-14 15:34:46 +02:00
Alexander Tokmakov
de9dc2b0b3
Merge branch 'master' into limit_mutations_per_entry 2023-04-14 15:19:08 +03:00
Igor Nikonov
7c84dc48c6 Better way to define for which storage output is parallelized 2023-04-14 12:00:02 +00:00
zvonand
089306820f Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-implicit-tz 2023-04-14 13:21:49 +02:00
Kseniia Sumarokova
9eee7c72fe
Merge pull request #47307 from kssenii/better-tests-for-data-lakes 
deltalake/iceberg/hudi improvements
 2023-04-14 12:13:17 +02:00
Kseniia Sumarokova
6a0d9a37ce
Merge branch 'master' into fix-mysql-named-collection 2023-04-14 12:03:51 +02:00
Igor Nikonov
267cd097bf
Merge pull request #48729 from ClickHouse/fix-memory-order-in-number_mt 
Fix memory order in number_mt
 2023-04-13 23:57:36 +02:00
Kseniia Sumarokova
54518bf928
Merge branch 'master' into better-tests-for-data-lakes 2023-04-13 23:45:12 +02:00
Igor Nikonov
fa6346008a Merge remote-tracking branch 'origin/master' into parallel-processing-from-storages 2023-04-13 20:55:18 +00:00
Igor Nikonov
fcd2eaea72 Disable 'Dictionary' storage due to count() can return incorrect result 
see test_dictionaries_redis/test_long.py::test_redis_dict_long
 2023-04-13 20:23:59 +00:00
Robert Schulze
43b8cbc50c
Merge pull request #48737 from ClickHouse/rs/fix-wshadow 
Remove `-Wshadow` suppression which leaked into global namespace
 2023-04-13 21:18:50 +02:00
Alexander Tokmakov
af1bf08663
Revert "Add MemoryTracker for the background tasks" 2023-04-13 21:05:02 +03:00
kssenii
ad48e1d010 Fox 2023-04-13 19:36:25 +02:00
Alexander Tokmakov
8d207e8752 Merge branch 'master' into limit_mutations_per_entry 2023-04-13 18:13:44 +02:00
Dmitry Novik
5cc9b46f78 Merge remote-tracking branch 'origin/master' into optimize-compilation 2023-04-13 16:04:09 +02:00
kssenii
6f53784f22 Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes 2023-04-13 15:56:40 +02:00
Alexander Tokmakov
93368b4d55
Merge pull request #48730 from ClickHouse/remove_some_code 
Do not remove broken detached parts on startup
 2023-04-13 16:18:59 +03:00
kssenii
3ac7f999bc Review fixes 2023-04-13 13:38:16 +02:00
kssenii
188b41cdb2 Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-13 13:33:30 +02:00
Dmitry Novik
467ecf45e1
Merge pull request #46089 from ClickHouse/background-memory-tracker 
Add MemoryTracker for the background tasks
 2023-04-13 13:29:27 +02:00
kssenii
09c23287aa Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-13 13:25:45 +02:00
Robert Schulze
7a21d5888c
Remove -Wshadow suppression which leaked into global namespace 2023-04-13 08:46:40 +00:00
Alexander Tokmakov
d6bff4c5a8 limit the number of mutations commands per MUTATE_PART 2023-04-13 00:20:43 +02:00
Igor Nikonov
ba1adeba2b Fix fast tests 
+ avoid parallelization after SourceFromSingleChunk, SystemOne
 2023-04-12 21:46:09 +00:00
Alexander Tokmakov
5d77bb149e do not remove broken detached parts on startup 2023-04-12 23:26:34 +02:00
Igor Nikonov
ac540335e7 Let's see blast radius w/o parallelization after numbers() 2023-04-12 21:05:24 +00:00
Robert Schulze
3f7ce60e03
Merge branch 'master' into rs/gcc-removal 2023-04-12 22:17:04 +02:00
robot-ch-test-poll2
d1793a9bc5
Merge pull request #48692 from ClickHouse/rs/mergetree-syntax 
Update MergeTree syntax for optional index granularity argument
 2023-04-12 21:35:40 +02:00
Igor Nikonov
e3b61a4ea7 Fix memory order in number_mt 2023-04-12 19:35:12 +00:00
Alexey Milovidov
4d74e3da4d
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards 2023-04-12 22:13:52 +03:00
Igor Nikonov
174885368f Parallelize query processing right after reading FROM ... 
- mostly simple or external storages
 2023-04-12 18:46:07 +00:00
alesapin
8067cfcb51
Merge pull request #46946 from ClickHouse/revert-46909-revert-45911-mutations_rename_hang 
Fix several `RENAME COLUMN` bugs
 2023-04-12 19:07:48 +02:00
Dmitry Novik
6c11acccc9 Remove unnecessary code 2023-04-12 14:08:09 +00:00
Robert Schulze
6b441b40dd
Update MergeTree syntax for optional index granularity argument 
This is follow up to #45451.
 2023-04-12 11:16:55 +00:00
zvonand
2a9f28b73b resolve 2023-04-12 12:54:39 +02:00
zvonand
a9499eed79 moved getting server TZ DateLUT to separate place, upd tests and fix 2023-04-12 12:47:05 +02:00
alesapin
55c3f1ee29
Merge pull request #48666 from ClickHouse/remove_strange_code_from_mutation_task 
Remove strange code from MutateTask
 2023-04-12 12:38:43 +02:00
SmitaRKulkarni
7ad05c1bc3
Merge branch 'master' into Clear_marks_cache_for_outdated_parts 2023-04-12 09:21:59 +02:00
SmitaRKulkarni
dd64eaed66
Merge pull request #47985 from Algunenano/zk_retry_timeout 
Do not continue retrying to connect to ZK if the query is killed or over limits
 2023-04-12 08:49:47 +02:00