thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,502 Commits 9,328 Branches 1,783 Tags 2.1 GiB
418423a304
Commit Graph

131502 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael Kolupaev
418423a304 Slightly more things 2023-12-27 20:24:55 +00:00
Michael Kolupaev
ef4cc5ec7f Things 2023-12-27 20:24:55 +00:00
Michael Kolupaev
a7c369e14f Overhaul timestamp arithmetic 2023-12-27 20:24:55 +00:00
Michael Kolupaev
01369a0a8a Overhaul dependencies 2023-12-27 20:24:54 +00:00
Michael Kolupaev
01345981e2 Overhaul RefreshTask 2023-12-27 20:24:54 +00:00
Michael Kolupaev
5dc04a13a7 Simple review comments 2023-12-27 20:24:54 +00:00
koloshmet
808cb0fa05 fix fix fix 2023-12-27 20:24:54 +00:00
koloshmet
f1161566b4 proper tmp table cleanup 2023-12-27 20:24:54 +00:00
koloshmet
f14114dafc proper tmp table cleanup 2023-12-27 20:24:54 +00:00
koloshmet
d1932763f3 fixed style 2023-12-27 20:24:54 +00:00
koloshmet
3556c15c51 fixed tests 2023-12-27 20:24:54 +00:00
koloshmet
ba766edb53 refreshable view query test 2023-12-27 20:24:54 +00:00
koloshmet
c762898adb refreshable materialized views 2023-12-27 20:24:54 +00:00
Sema Checherinda
cc8a35293b
Merge pull request #58163 from danipozo/remove-overwritten-assignment 
Remove retryStrategy assignments overwritten in ClientFactory::create()
 2023-12-27 20:29:55 +01:00
Sema Checherinda
617ee5abaf
Merge pull request #57551 from bharatnc/ncb/async-metrics-pri-key-size 
add asynchronous metrics for total primary key size in memory and allocated
 2023-12-27 20:24:20 +01:00
Kruglov Pavel
745d9bb47f
Merge pull request #58214 from kevinmingtarja/optimize-long-and-notequals-chain 
Add optimization for AND notEquals chain in logical expression optimizer
 2023-12-27 15:51:59 +01:00
Alexey Milovidov
0e678fb6c1
Merge pull request #56996 from ClickHouse/vdimir/hash_join_max_block_size 
HashJoin respects max_joined_block_size_rows
 2023-12-27 15:46:46 +01:00
Alexey Milovidov
f00337e2ba
Merge pull request #57872 from CurtizJ/optimize-aggregation-consecutive-keys 
Better optimization of consecutive keys in aggregation
 2023-12-27 15:44:22 +01:00
Alexey Milovidov
27bcdbe14a
Update src/Core/Settings.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-12-27 17:41:30 +03:00
Alexey Milovidov
1a93fd7f7d
Merge pull request #57829 from azat/kafka-fix-stat-leak 
Create consumers for Kafka tables on fly (but keep them for some period since last used)
 2023-12-27 15:35:05 +01:00
Alexey Milovidov
ad42115994
Merge pull request #58175 from Avogar/fix-array-subcolumns-read 
Fix possible PARAMETER_OUT_OF_BOUND error during subcolumns reading from wide part in MergeTree
 2023-12-27 15:30:02 +01:00
robot-clickhouse-ci-2
bd43660255
Merge pull request #58245 from Algunenano/fix_perf_readme 
Fix perf test README
 2023-12-27 13:33:50 +01:00
Alexey Milovidov
5f6318e51b
Merge pull request #58151 from ClickHouse/fix-legend 
Fix dashboard legend sorting and rows number
 2023-12-27 13:33:00 +01:00
Alexey Milovidov
d0bc4fafdb
Merge pull request #58220 from wangtZJU/fix_AddDefaultDatabaseVisitor_bad_performance 
fix CREATE VIEW hang
 2023-12-27 13:30:37 +01:00
Alexey Milovidov
77ec4a0422
Merge pull request #58218 from kitaisreal/merge-tree-automatically-derive-do-not-merge-across-partitions-select-final-setting 
MergeTree derive do_not_merge_across_partitions_select_final setting
 2023-12-27 13:26:17 +01:00
Raúl Marín
f8d9a850c7 Fix perf test README 2023-12-27 12:16:17 +00:00
Kevin Mingtarja
3d7596da4b Rename test, fix style 2023-12-27 17:28:01 +07:00
Azat Khuzhin
ebad1bf4f3 Move StorageKafka::createConsumer() into KafkaConsumer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
03218202d3 Fix data-race between StorageKafka::startup() and cleanConsumers() 
Actually now we can create consumer object in the ctor, no need to do
this in startup(), since consumer now do not connects to kafka.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
1f03a21033 Update comment for statistics.interval.ms librdkafka option 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
06a9e9a9ca Use separate thread for kafka consumers cleanup 
Since pool may exceed threads, while we need to run this thread always
to avoid memory leaking.

And this should not be a problem since librdkafka has multiple threads
for each consumer (5!) anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
a7453f7f14 Allow setThreadName() to truncate thread name instead of throw an error 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
b19b70b8fc Add ability to configure TTL for kafka consumers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
2ff0bfb0a1 Preserve KafkaConsumer objects 
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
db74549940 Enable stats for system.kafka_consumers back by default 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
e7592c140e Create consumers for Kafka tables on fly (but keep them for 1min since last used) 
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
51d4f583e6 Properly set shutdown_called in StorageKafka::shutdown() 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
123d63e824 Remove StorageKafka::num_created_consumers (in favor of all_consumers.size()) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Alexey Milovidov
a0fccb0498
Merge pull request #58224 from amosbird/part_offset_pk 
Primary key analysis for _part_offset
 2023-12-26 14:51:57 +01:00
Alexey Milovidov
4fbe41b47e
Update 02950_part_offset_as_primary_key.sql 2023-12-26 16:51:39 +03:00
Alexey Milovidov
31a081bd83
Merge pull request #58226 from Algunenano/cleanup_known_short 
Cleanup some known short messages
 2023-12-26 14:40:58 +01:00
Alexey Milovidov
ee1d7f25de
Merge pull request #58221 from ClickHouse/fix_syntax_and_doc 
Fix syntax and doc
 2023-12-26 14:38:54 +01:00
Yarik Briukhovetskyi
41c275274b
Merge pull request #58148 from yariks5s/fix_s3_regions 
S3-links region independency
 2023-12-26 12:59:00 +01:00
Raúl Marín
e87b9751bd Cleanup some known short messages 2023-12-26 12:58:50 +01:00
Maksim Kita
71921086ae Fixed tests 2023-12-26 12:41:21 +03:00
Maksim Kita
cbf9304d1f MergeTree automatically derive do_not_merge_across_partitions_select_final setting 2023-12-26 12:41:21 +03:00
Amos Bird
66660ee4e2
Add comment 2023-12-26 17:04:00 +08:00
Amos Bird
bfcccf9fa3
Primary key analysis for _part_offset 2023-12-26 17:03:59 +08:00
santrancisco
91835256aa
fix doc 2023-12-26 16:57:18 +11:00
santrancisco
a59d874bf9
fix syntax 2023-12-26 16:56:58 +11:00