thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-28 18:42:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,655 Commits 9,187 Branches 1,782 Tags 2.2 GiB
1013f6b23f
Commit Graph

131655 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexey Milovidov
ad42115994
Merge pull request #58175 from Avogar/fix-array-subcolumns-read 
Fix possible PARAMETER_OUT_OF_BOUND error during subcolumns reading from wide part in MergeTree
 2023-12-27 15:30:02 +01:00
Alexander Tokmakov
01d042c490 Revert "Merge pull request #57932 from ClickHouse/remove-shit-cleanup" 
This reverts commit 2d58dc512c, reversing
changes made to 41873dc4a3.
 2023-12-27 13:46:06 +01:00
robot-clickhouse-ci-2
bd43660255
Merge pull request #58245 from Algunenano/fix_perf_readme 
Fix perf test README
 2023-12-27 13:33:50 +01:00
Alexey Milovidov
5f6318e51b
Merge pull request #58151 from ClickHouse/fix-legend 
Fix dashboard legend sorting and rows number
 2023-12-27 13:33:00 +01:00
Alexey Milovidov
d0bc4fafdb
Merge pull request #58220 from wangtZJU/fix_AddDefaultDatabaseVisitor_bad_performance 
fix CREATE VIEW hang
 2023-12-27 13:30:37 +01:00
Alexey Milovidov
77ec4a0422
Merge pull request #58218 from kitaisreal/merge-tree-automatically-derive-do-not-merge-across-partitions-select-final-setting 
MergeTree derive do_not_merge_across_partitions_select_final setting
 2023-12-27 13:26:17 +01:00
Raúl Marín
f8d9a850c7 Fix perf test README 2023-12-27 12:16:17 +00:00
Alexander Tokmakov
4c4623e3ab Revert "Merge pull request #58100 from ClickHouse/optimize-cleanup-as-noop" 
This reverts commit 3b1aa53460, reversing
changes made to b9466023ce.
 2023-12-27 13:03:53 +01:00
Alexander Tokmakov
eeadeaa89d Revert "Merge pull request #58104 from ClickHouse/cleanup-replication-compatibility" 
This reverts commit 34fd555ee6, reversing
changes made to cb53ee63be.
 2023-12-27 13:03:38 +01:00
Kevin Mingtarja
3d7596da4b Rename test, fix style 2023-12-27 17:28:01 +07:00
Raúl Marín
68787cedd6 Linter warnings 2023-12-27 08:51:17 +00:00
Azat Khuzhin
ebad1bf4f3 Move StorageKafka::createConsumer() into KafkaConsumer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
03218202d3 Fix data-race between StorageKafka::startup() and cleanConsumers() 
Actually now we can create consumer object in the ctor, no need to do
this in startup(), since consumer now do not connects to kafka.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
1f03a21033 Update comment for statistics.interval.ms librdkafka option 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
06a9e9a9ca Use separate thread for kafka consumers cleanup 
Since pool may exceed threads, while we need to run this thread always
to avoid memory leaking.

And this should not be a problem since librdkafka has multiple threads
for each consumer (5!) anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
a7453f7f14 Allow setThreadName() to truncate thread name instead of throw an error 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
b19b70b8fc Add ability to configure TTL for kafka consumers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
2ff0bfb0a1 Preserve KafkaConsumer objects 
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
db74549940 Enable stats for system.kafka_consumers back by default 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
e7592c140e Create consumers for Kafka tables on fly (but keep them for 1min since last used) 
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
51d4f583e6 Properly set shutdown_called in StorageKafka::shutdown() 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
123d63e824 Remove StorageKafka::num_created_consumers (in favor of all_consumers.size()) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Pengyuan Bian
3dcc1f5448 update. 2023-12-27 07:54:18 +00:00
Raúl Marín
76abd32196 Speedup MIN and MAX for native types 2023-12-26 20:23:55 +01:00
Alexey Milovidov
a0fccb0498
Merge pull request #58224 from amosbird/part_offset_pk 
Primary key analysis for _part_offset
 2023-12-26 14:51:57 +01:00
Alexey Milovidov
4fbe41b47e
Update 02950_part_offset_as_primary_key.sql 2023-12-26 16:51:39 +03:00
Alexey Milovidov
31a081bd83
Merge pull request #58226 from Algunenano/cleanup_known_short 
Cleanup some known short messages
 2023-12-26 14:40:58 +01:00
Alexey Milovidov
ee1d7f25de
Merge pull request #58221 from ClickHouse/fix_syntax_and_doc 
Fix syntax and doc
 2023-12-26 14:38:54 +01:00
Yarik Briukhovetskyi
41c275274b
Merge pull request #58148 from yariks5s/fix_s3_regions 
S3-links region independency
 2023-12-26 12:59:00 +01:00
Raúl Marín
e87b9751bd Cleanup some known short messages 2023-12-26 12:58:50 +01:00
Maksim Kita
71921086ae Fixed tests 2023-12-26 12:41:21 +03:00
Maksim Kita
cbf9304d1f MergeTree automatically derive do_not_merge_across_partitions_select_final setting 2023-12-26 12:41:21 +03:00
Amos Bird
66660ee4e2
Add comment 2023-12-26 17:04:00 +08:00
Amos Bird
bfcccf9fa3
Primary key analysis for _part_offset 2023-12-26 17:03:59 +08:00
Pengyuan Bian
2765acfe45 wip test. 2023-12-26 07:50:46 +00:00
santrancisco
91835256aa
fix doc 2023-12-26 16:57:18 +11:00
santrancisco
a59d874bf9
fix syntax 2023-12-26 16:56:58 +11:00
wangtao.2077
7ad278eefb fix AddDefaultDatabaseVisitor bad performance 2023-12-26 13:31:45 +08:00
Nikolay Degterinsky
85b149395a
Merge pull request #57796 from evillique/replicated-database-forbid-create-as-select 
Forbid CREATE AS SELECT for database Replicated
 2023-12-25 19:43:28 +01:00
Alexey Milovidov
005023a16d
Merge pull request #58210 from chhetripradeep/pchhetri/param-query-clickhouse-local 
Add support for specifying query parameters in the command line in clickhouse-local
 2023-12-25 13:34:04 +01:00
Maksim Kita
c5959068cb Fixed tests 2023-12-25 12:51:17 +03:00
Maksim Kita
46446ed0c4 Fixed tests 2023-12-25 12:51:17 +03:00
Maksim Kita
51d6c444f5 Fixed tests 2023-12-25 12:51:17 +03:00
Maksim Kita
269e9706fb MergeTree FINAL extract non intersecting parts ranges 2023-12-25 12:51:17 +03:00
Alexey Milovidov
35e27ab1a3
Merge pull request #58213 from azat/tests/processes-cleanup-v2 
Fix leftover processes/hangs in tests (resubmit)
 2023-12-25 05:07:43 +01:00
Alexey Milovidov
9196c2b994 Follow-up 2023-12-25 04:54:54 +01:00
Alexey Milovidov
c2f93ecf4d
Merge pull request #58211 from ClickHouse/binary-viewer 
Binary (symbols) viewer
 2023-12-25 04:52:57 +01:00
Nikolay Degterinsky
98a6d67ae3 Disable tests with CREATE AS SELECT for database Replicated 2023-12-24 23:49:26 +00:00
Nikolay Degterinsky
d524951416
Merge pull request #58198 from azat/exception-fmt 
Fix all Exception with missing arguments
 2023-12-25 00:40:18 +01:00
Azat Khuzhin
3be3b0a280 Fix incorrect Exceptions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 21:26:32 +01:00