thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-24 00:22:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,500 Commits 9,167 Branches 1,781 Tags 2.1 GiB
dcbd3b9c26
Commit Graph

131500 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexey Milovidov
dcbd3b9c26
Merge pull request #58267 from ClickHouse/fix_is_deleted_compatibility 
Re-introduce `is_deleted` column for ReplacingMergeTree
 2023-12-28 00:13:01 +01:00
Alexey Milovidov
d005193815
Merge pull request #58269 from ClickHouse/tavplubix-patch-9 
Allow a few retries when committing a part during shutdown
 2023-12-28 00:12:02 +01:00
Alexey Milovidov
5406c79c89
Merge pull request #58264 from ClickHouse/fs-cache-better-multi-cache-configuration 
Make fs caches with the same path use the same cache objects
 2023-12-28 00:10:15 +01:00
Sema Checherinda
cc8a35293b
Merge pull request #58163 from danipozo/remove-overwritten-assignment 
Remove retryStrategy assignments overwritten in ClientFactory::create()
 2023-12-27 20:29:55 +01:00
Alexander Tokmakov
a3cba8e06f
Update StorageReplicatedMergeTree.cpp 2023-12-27 20:27:15 +01:00
Sema Checherinda
617ee5abaf
Merge pull request #57551 from bharatnc/ncb/async-metrics-pri-key-size 
add asynchronous metrics for total primary key size in memory and allocated
 2023-12-27 20:24:20 +01:00
Alexander Tokmakov
f5bcfaffa5 disable vertical merges with cleanup 2023-12-27 19:28:50 +01:00
kssenii
041c5d4219 Make caches with the same path use the same cache objects 2023-12-27 18:37:43 +01:00
Alexander Tokmakov
f924848347 partially revert #54368 (f28ad1e136) 2023-12-27 18:17:59 +01:00
Alexander Tokmakov
d7a6148828 partially revert #54706 (135df6870c) 2023-12-27 18:13:51 +01:00
Kruglov Pavel
745d9bb47f
Merge pull request #58214 from kevinmingtarja/optimize-long-and-notequals-chain 
Add optimization for AND notEquals chain in logical expression optimizer
 2023-12-27 15:51:59 +01:00
Alexey Milovidov
0e678fb6c1
Merge pull request #56996 from ClickHouse/vdimir/hash_join_max_block_size 
HashJoin respects max_joined_block_size_rows
 2023-12-27 15:46:46 +01:00
Alexey Milovidov
f00337e2ba
Merge pull request #57872 from CurtizJ/optimize-aggregation-consecutive-keys 
Better optimization of consecutive keys in aggregation
 2023-12-27 15:44:22 +01:00
Alexey Milovidov
27bcdbe14a
Update src/Core/Settings.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-12-27 17:41:30 +03:00
Alexey Milovidov
1a93fd7f7d
Merge pull request #57829 from azat/kafka-fix-stat-leak 
Create consumers for Kafka tables on fly (but keep them for some period since last used)
 2023-12-27 15:35:05 +01:00
Alexey Milovidov
ad42115994
Merge pull request #58175 from Avogar/fix-array-subcolumns-read 
Fix possible PARAMETER_OUT_OF_BOUND error during subcolumns reading from wide part in MergeTree
 2023-12-27 15:30:02 +01:00
Alexander Tokmakov
01d042c490 Revert "Merge pull request #57932 from ClickHouse/remove-shit-cleanup" 
This reverts commit 2d58dc512c, reversing
changes made to 41873dc4a3.
 2023-12-27 13:46:06 +01:00
robot-clickhouse-ci-2
bd43660255
Merge pull request #58245 from Algunenano/fix_perf_readme 
Fix perf test README
 2023-12-27 13:33:50 +01:00
Alexey Milovidov
5f6318e51b
Merge pull request #58151 from ClickHouse/fix-legend 
Fix dashboard legend sorting and rows number
 2023-12-27 13:33:00 +01:00
Alexey Milovidov
d0bc4fafdb
Merge pull request #58220 from wangtZJU/fix_AddDefaultDatabaseVisitor_bad_performance 
fix CREATE VIEW hang
 2023-12-27 13:30:37 +01:00
Alexey Milovidov
77ec4a0422
Merge pull request #58218 from kitaisreal/merge-tree-automatically-derive-do-not-merge-across-partitions-select-final-setting 
MergeTree derive do_not_merge_across_partitions_select_final setting
 2023-12-27 13:26:17 +01:00
Raúl Marín
f8d9a850c7 Fix perf test README 2023-12-27 12:16:17 +00:00
Alexander Tokmakov
4c4623e3ab Revert "Merge pull request #58100 from ClickHouse/optimize-cleanup-as-noop" 
This reverts commit 3b1aa53460, reversing
changes made to b9466023ce.
 2023-12-27 13:03:53 +01:00
Alexander Tokmakov
eeadeaa89d Revert "Merge pull request #58104 from ClickHouse/cleanup-replication-compatibility" 
This reverts commit 34fd555ee6, reversing
changes made to cb53ee63be.
 2023-12-27 13:03:38 +01:00
Kevin Mingtarja
3d7596da4b Rename test, fix style 2023-12-27 17:28:01 +07:00
Azat Khuzhin
ebad1bf4f3 Move StorageKafka::createConsumer() into KafkaConsumer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
03218202d3 Fix data-race between StorageKafka::startup() and cleanConsumers() 
Actually now we can create consumer object in the ctor, no need to do
this in startup(), since consumer now do not connects to kafka.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
1f03a21033 Update comment for statistics.interval.ms librdkafka option 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
06a9e9a9ca Use separate thread for kafka consumers cleanup 
Since pool may exceed threads, while we need to run this thread always
to avoid memory leaking.

And this should not be a problem since librdkafka has multiple threads
for each consumer (5!) anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
a7453f7f14 Allow setThreadName() to truncate thread name instead of throw an error 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
b19b70b8fc Add ability to configure TTL for kafka consumers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
2ff0bfb0a1 Preserve KafkaConsumer objects 
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
db74549940 Enable stats for system.kafka_consumers back by default 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
e7592c140e Create consumers for Kafka tables on fly (but keep them for 1min since last used) 
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
51d4f583e6 Properly set shutdown_called in StorageKafka::shutdown() 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
123d63e824 Remove StorageKafka::num_created_consumers (in favor of all_consumers.size()) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Alexey Milovidov
a0fccb0498
Merge pull request #58224 from amosbird/part_offset_pk 
Primary key analysis for _part_offset
 2023-12-26 14:51:57 +01:00
Alexey Milovidov
4fbe41b47e
Update 02950_part_offset_as_primary_key.sql 2023-12-26 16:51:39 +03:00
Alexey Milovidov
31a081bd83
Merge pull request #58226 from Algunenano/cleanup_known_short 
Cleanup some known short messages
 2023-12-26 14:40:58 +01:00
Alexey Milovidov
ee1d7f25de
Merge pull request #58221 from ClickHouse/fix_syntax_and_doc 
Fix syntax and doc
 2023-12-26 14:38:54 +01:00
Yarik Briukhovetskyi
41c275274b
Merge pull request #58148 from yariks5s/fix_s3_regions 
S3-links region independency
 2023-12-26 12:59:00 +01:00
Raúl Marín
e87b9751bd Cleanup some known short messages 2023-12-26 12:58:50 +01:00
Maksim Kita
71921086ae Fixed tests 2023-12-26 12:41:21 +03:00
Maksim Kita
cbf9304d1f MergeTree automatically derive do_not_merge_across_partitions_select_final setting 2023-12-26 12:41:21 +03:00
Amos Bird
66660ee4e2
Add comment 2023-12-26 17:04:00 +08:00
Amos Bird
bfcccf9fa3
Primary key analysis for _part_offset 2023-12-26 17:03:59 +08:00
santrancisco
91835256aa
fix doc 2023-12-26 16:57:18 +11:00
santrancisco
a59d874bf9
fix syntax 2023-12-26 16:56:58 +11:00
wangtao.2077
7ad278eefb fix AddDefaultDatabaseVisitor bad performance 2023-12-26 13:31:45 +08:00
Nikolay Degterinsky
85b149395a
Merge pull request #57796 from evillique/replicated-database-forbid-create-as-select 
Forbid CREATE AS SELECT for database Replicated
 2023-12-25 19:43:28 +01:00