thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-02 12:32:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
132,161 Commits 9,194 Branches 1,782 Tags 2.2 GiB
ddbdd89904
Commit Graph

132161 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Azat Khuzhin
87f3f6619a Fix data-race between StorageKafka::startup() and cleanConsumers() 
Actually now we can create consumer object in the ctor, no need to do
this in startup(), since consumer now do not connects to kafka.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 03218202d3)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
3c139d7135 Update comment for statistics.interval.ms librdkafka option 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 1f03a21033)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
6f85306510 Use separate thread for kafka consumers cleanup 
Since pool may exceed threads, while we need to run this thread always
to avoid memory leaking.

And this should not be a problem since librdkafka has multiple threads
for each consumer (5!) anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 06a9e9a9ca)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
8ac68b64d7 Allow setThreadName() to truncate thread name instead of throw an error 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit a7453f7f14)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
7d2b82c37c Add ability to configure TTL for kafka consumers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit b19b70b8fc)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
bea1610219 Preserve KafkaConsumer objects 
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 2ff0bfb0a1)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
71fdde76c2 Enable stats for system.kafka_consumers back by default 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit db74549940)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
d66be02dc3 Create consumers for Kafka tables on fly (but keep them for 1min since last used) 
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit e7592c140e)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
a6841c8915 Properly set shutdown_called in StorageKafka::shutdown() 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 51d4f583e6)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
3541d9a05f Remove StorageKafka::num_created_consumers (in favor of all_consumers.size()) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 123d63e824)
 2023-12-28 15:32:39 +01:00
avogar
146de5b220 Ignore MVs with dropped target table during pushing to views 2023-12-28 13:56:57 +00:00
avogar
e1a9baa5b0 Fix 2023-12-28 13:51:37 +00:00
Nikolai Kochetov
737563296b
Merge branch 'master' into filter-virtual-columns-storage-merge 2023-12-28 14:47:41 +01:00
vdimir
e20cb2d052
Disable max_bytes_before_external* in 00172_hits_joins 2023-12-28 13:35:34 +00:00
Antonio Andelic
8ace9978b3 Handle another case for preprocessing 2023-12-28 13:20:47 +00:00
Alexander Tokmakov
bdada351c8 Revert "Merge pull request #58274 from ClickHouse/revert-58267" 
This reverts commit 583b9637c2, reversing
changes made to 224e937620.
 2023-12-28 14:07:59 +01:00
Alexander Tokmakov
5fcbf9cfb0 Revert "Merge pull request #58251 from ClickHouse/reintroduce-compatibility-with-a-misfeature" 
This reverts commit a811d5b761, reversing
changes made to 583b9637c2.
 2023-12-28 14:06:56 +01:00
vdimir
017e4998a1
Fix timeouts in 01732_race_condition_storage_join_long 2023-12-28 12:55:51 +00:00
Alexander Tokmakov
d58b76ce06
Merge pull request #58296 from ClickHouse/revert-56946-mv 
Revert "Refreshable materialized views (takeover)"
 2023-12-28 13:33:22 +01:00
Alexander Tokmakov
38fe70c68a
Revert "Refreshable materialized views (takeover)" 2023-12-28 13:12:20 +01:00
kssenii
529a1d67cb Fix 2023-12-28 12:48:56 +01:00
Alexey Milovidov
c6731df0cf Update changelog 2023-12-28 12:13:43 +01:00
Alexey Milovidov
09352205d8
Merge pull request #58288 from ClickHouse/fix-release 
Fix an error in the release script - it didn't allow to make 23.12.
 2023-12-28 12:09:23 +01:00
Alexey Milovidov
0c57260ad3
Merge pull request #58290 from ClickHouse/auto/v23.12.1.1368-stable 
Update version_date.tsv and changelogs after v23.12.1.1368-stable
 2023-12-28 12:09:12 +01:00
Duc Canh Le
238c5e66d5 use ChunkInfo to carry part level 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-28 11:01:18 +00:00
robot-clickhouse
27657a3eb6 Update version_date.tsv and changelogs after v23.12.1.1368-stable 2023-12-28 10:32:09 +00:00
Alexey Milovidov
4dbd07fbb9
Merge pull request #58286 from ClickHouse/23.13-prepare 
Update version after release
 2023-12-28 11:27:58 +01:00
Raúl Marín
2fdb56bb4b Merge remote-tracking branch 'blessed/master' into speedup_numbers 2023-12-28 11:26:16 +01:00
Alexey Milovidov
4bb8592434 Update autogenerated version to 23.13.1.1 and contributors 2023-12-28 11:22:16 +01:00
Alexey Milovidov
9d1e124102 Let me do the release 2023-12-28 11:21:58 +01:00
Alexey Milovidov
d764d8d72b Update Changelog 2023-12-28 11:19:16 +01:00
vdimir
f53281a823
fix 02862_index_inverted_incorrect_args 2023-12-28 10:00:05 +00:00
Alexey Milovidov
9f895fd038 Add a new item into the changelog 2023-12-28 09:33:01 +01:00
Alexey Milovidov
a2faa65b08
Merge pull request #56946 from ClickHouse/mv 
Refreshable materialized views (takeover)
 2023-12-28 09:29:44 +01:00
Alexey Milovidov
3f10ad7573 Remove reverted item from the changelog 2023-12-28 07:16:57 +01:00
Alexey Milovidov
643dddb619
Merge pull request #58277 from ClickHouse/revert-57969-punycode-is-cool 
Revert "Implement punycode encoding/decoding"
 2023-12-28 07:12:36 +01:00
Alexey Milovidov
345d29a3c1
Revert "Implement punycode encoding/decoding" 2023-12-28 09:12:22 +03:00
Alexey Milovidov
a719c67bd0 Remove reverted item from the changelog 2023-12-28 07:03:45 +01:00
Duc Canh Le
e622cf9ee9 address review comments 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-28 05:01:55 +00:00
Alexey Milovidov
524d53199d
Merge branch 'master' into mv 2023-12-28 04:11:48 +01:00
Alexey Milovidov
a811d5b761
Merge pull request #58251 from ClickHouse/reintroduce-compatibility-with-a-misfeature 
Reintroduce compatibility with `is_deleted` on a syntax level
 2023-12-28 04:11:04 +01:00
Alexey Milovidov
583b9637c2
Merge pull request #58274 from ClickHouse/revert-58267 
Revert #58267
 2023-12-28 04:10:18 +01:00
Alexey Milovidov
c7efd2afea Revert #58267 2023-12-28 04:09:33 +01:00
Alexey Milovidov
224e937620
Merge pull request #58273 from ClickHouse/changelog-23.12 
Add changelog for 23.12
 2023-12-28 04:04:08 +01:00
Alexey Milovidov
a1062dec92 Add changelog for 23.12 2023-12-28 03:52:09 +01:00
Alexey Milovidov
40a5dbdeba
Merge branch 'master' into mv 2023-12-28 03:16:27 +01:00
Alexey Milovidov
a62f4ee278
Merge pull request #58272 from ClickHouse/revert-57829-kafka-fix-stat-leak 
Revert "Create consumers for Kafka tables on fly (but keep them for some period since last used)"
 2023-12-28 01:36:09 +01:00
Alexey Milovidov
c52886eb81
Revert "Create consumers for Kafka tables on fly (but keep them for some period since last used)" 2023-12-28 03:35:57 +03:00
Alexey Milovidov
1d9dbfd18b
Merge pull request #49103 from ClickHouse/check-about-global-sorting 
Fixed a sorting order breakage in TTL GROUP BY
 2023-12-28 01:35:14 +01:00
Alexey Milovidov
30f6d03b37
Merge pull request #57977 from bianpengyuan/add-s3-read-only-setting 
Add a setting to specify s3 disk is read only.
 2023-12-28 01:24:50 +01:00