Azat Khuzhin
7d2b82c37c
Add ability to configure TTL for kafka consumers
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit b19b70b8fc
2023-12-28 15:32:39 +01:00
Azat Khuzhin
bea1610219
Preserve KafkaConsumer objects
...
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 2ff0bfb0a1
2023-12-28 15:32:39 +01:00
Azat Khuzhin
71fdde76c2
Enable stats for system.kafka_consumers back by default
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit db74549940
2023-12-28 15:32:39 +01:00
Azat Khuzhin
d66be02dc3
Create consumers for Kafka tables on fly (but keep them for 1min since last used)
...
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).
Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
moves entries from this queue into another linked list, but in a
with sorting, which is incredibly slow for linked lists)
So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.
This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit e7592c140e
2023-12-28 15:32:39 +01:00
Azat Khuzhin
a6841c8915
Properly set shutdown_called in StorageKafka::shutdown()
...
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 51d4f583e6
2023-12-28 15:32:39 +01:00
Azat Khuzhin
3541d9a05f
Remove StorageKafka::num_created_consumers (in favor of all_consumers.size())
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 123d63e824
2023-12-28 15:32:39 +01:00
vdimir
e20cb2d052
Disable max_bytes_before_external* in 00172_hits_joins
2023-12-28 13:35:34 +00:00
Antonio Andelic
8ace9978b3
Handle another case for preprocessing
2023-12-28 13:20:47 +00:00
vdimir
017e4998a1
Fix timeouts in 01732_race_condition_storage_join_long
2023-12-28 12:55:51 +00:00
Alexander Tokmakov
d58b76ce06
Merge pull request #58296 from ClickHouse/revert-56946-mv
...
Revert "Refreshable materialized views (takeover)"
2023-12-28 13:33:22 +01:00
Alexander Tokmakov
38fe70c68a
Revert "Refreshable materialized views (takeover)"
2023-12-28 13:12:20 +01:00
kssenii
529a1d67cb
Fix
2023-12-28 12:48:56 +01:00
Alexey Milovidov
c6731df0cf
Update changelog
2023-12-28 12:13:43 +01:00
Alexey Milovidov
09352205d8
Merge pull request #58288 from ClickHouse/fix-release
...
Fix an error in the release script - it didn't allow to make 23.12.
2023-12-28 12:09:23 +01:00
Alexey Milovidov
0c57260ad3
Merge pull request #58290 from ClickHouse/auto/v23.12.1.1368-stable
...
Update version_date.tsv and changelogs after v23.12.1.1368-stable
2023-12-28 12:09:12 +01:00
Duc Canh Le
238c5e66d5
use ChunkInfo to carry part level
...
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
2023-12-28 11:01:18 +00:00
robot-clickhouse
27657a3eb6
Update version_date.tsv and changelogs after v23.12.1.1368-stable
2023-12-28 10:32:09 +00:00
Alexey Milovidov
4dbd07fbb9
Merge pull request #58286 from ClickHouse/23.13-prepare
...
Update version after release
2023-12-28 11:27:58 +01:00
Alexey Milovidov
4bb8592434
Update autogenerated version to 23.13.1.1 and contributors
2023-12-28 11:22:16 +01:00
Alexey Milovidov
9d1e124102
Let me do the release
2023-12-28 11:21:58 +01:00
Alexey Milovidov
d764d8d72b
Update Changelog
2023-12-28 11:19:16 +01:00
vdimir
f53281a823
fix 02862_index_inverted_incorrect_args
2023-12-28 10:00:05 +00:00
Alexey Milovidov
9f895fd038
Add a new item into the changelog
2023-12-28 09:33:01 +01:00
Alexey Milovidov
a2faa65b08
Merge pull request #56946 from ClickHouse/mv
...
Refreshable materialized views (takeover)
2023-12-28 09:29:44 +01:00
Alexey Milovidov
3f10ad7573
Remove reverted item from the changelog
2023-12-28 07:16:57 +01:00
Alexey Milovidov
643dddb619
Merge pull request #58277 from ClickHouse/revert-57969-punycode-is-cool
...
Revert "Implement punycode encoding/decoding"
2023-12-28 07:12:36 +01:00
Alexey Milovidov
345d29a3c1
Revert "Implement punycode encoding/decoding"
2023-12-28 09:12:22 +03:00
Alexey Milovidov
a719c67bd0
Remove reverted item from the changelog
2023-12-28 07:03:45 +01:00
Duc Canh Le
e622cf9ee9
address review comments
...
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
2023-12-28 05:01:55 +00:00
Alexey Milovidov
524d53199d
Merge branch 'master' into mv
2023-12-28 04:11:48 +01:00
Alexey Milovidov
a811d5b761
Merge pull request #58251 from ClickHouse/reintroduce-compatibility-with-a-misfeature
...
Reintroduce compatibility with `is_deleted` on a syntax level
2023-12-28 04:11:04 +01:00
Alexey Milovidov
583b9637c2
Merge pull request #58274 from ClickHouse/revert-58267
...
Revert #58267
2023-12-28 04:10:18 +01:00
Alexey Milovidov
c7efd2afea
Revert #58267
2023-12-28 04:09:33 +01:00
Alexey Milovidov
224e937620
Merge pull request #58273 from ClickHouse/changelog-23.12
...
Add changelog for 23.12
2023-12-28 04:04:08 +01:00
Alexey Milovidov
a1062dec92
Add changelog for 23.12
2023-12-28 03:52:09 +01:00
Alexey Milovidov
40a5dbdeba
Merge branch 'master' into mv
2023-12-28 03:16:27 +01:00
Alexey Milovidov
a62f4ee278
Merge pull request #58272 from ClickHouse/revert-57829-kafka-fix-stat-leak
...
Revert "Create consumers for Kafka tables on fly (but keep them for some period since last used)"
2023-12-28 01:36:09 +01:00
Alexey Milovidov
c52886eb81
Revert "Create consumers for Kafka tables on fly (but keep them for some period since last used)"
2023-12-28 03:35:57 +03:00
Alexey Milovidov
1d9dbfd18b
Merge pull request #49103 from ClickHouse/check-about-global-sorting
...
Fixed a sorting order breakage in TTL GROUP BY
2023-12-28 01:35:14 +01:00
Alexey Milovidov
30f6d03b37
Merge pull request #57977 from bianpengyuan/add-s3-read-only-setting
...
Add a setting to specify s3 disk is read only.
2023-12-28 01:24:50 +01:00
Alexey Milovidov
e7c987058c
Merge pull request #58250 from Avogar/json-compact-each-row-parallel-parsing
...
Fix parallel parsing for JSONCompactEachRow
2023-12-28 01:23:30 +01:00
robot-ch-test-poll2
04178a9028
Merge pull request #58231 from Algunenano/new_minmax
...
Speedup MIN and MAX for native types
2023-12-28 01:22:38 +01:00
Alexey Milovidov
1b6c5f6a98
Merge pull request #58120 from kitaisreal/merge-tree-final-extract-non-intersecting-parts-ranges
...
MergeTree FINAL extract non intersecting parts ranges
2023-12-28 01:18:26 +01:00
Alexey Milovidov
38d050060e
Merge branch 'master' into merge-tree-final-extract-non-intersecting-parts-ranges
2023-12-28 01:18:18 +01:00
Alexey Milovidov
d7a35773c1
Merge pull request #58252 from Algunenano/i51543
...
Avoid throwing ABORTED on normal situations
2023-12-28 00:28:51 +01:00
Alexey Milovidov
8d984df135
Merge pull request #58237 from azat/build/fwd-decl-exception
...
Some code refactoring (was an attempt to improve build time, but failed)
2023-12-28 00:21:09 +01:00
Alexey Milovidov
79fa5b7bd9
Merge pull request #58249 from Algunenano/i57086
...
[Analyzer] Add test for #57086
2023-12-28 00:20:22 +01:00
Alexey Milovidov
c024dc9c3d
Merge pull request #58265 from ClickHouse/remove-mayBenefitFromIndexForIn
...
Remove mayBenefitFromIndexForIn
2023-12-28 00:15:04 +01:00
Alexey Milovidov
dcbd3b9c26
Merge pull request #58267 from ClickHouse/fix_is_deleted_compatibility
...
Re-introduce `is_deleted` column for ReplacingMergeTree
2023-12-28 00:13:01 +01:00
Alexey Milovidov
d005193815
Merge pull request #58269 from ClickHouse/tavplubix-patch-9
...
Allow a few retries when committing a part during shutdown
2023-12-28 00:12:02 +01:00
