thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-27 18:12:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
77396d00d9
ClickHouse/tests/integration/test_storage_kafka/configs/kafka.xml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

59 lines
2.3 KiB
XML
Raw Normal View History

Change <yandex> to <clickhouse> in configs
2021-09-19 22:38:53 +00:00
<clickhouse>
Add row_delimiter argument to StorageKafka. There are common cases where a message doesn't end with a row delimiter. This patch allows specifying a row_delimiter char to compensate that. https://github.com/yandex/ClickHouse/issues/2298
2018-07-18 05:22:01 +00:00
<kafka>
Enable librdkafka debugging in test_storage_kafka
2020-05-04 11:27:15 +00:00
<!-- Debugging of possible issues, like:
- https://github.com/edenhill/librdkafka/issues/2077
- https://github.com/edenhill/librdkafka/issues/1778
Add integration test for kafka consumer hang This test triggers heartbeat failrues which leads to REBALANCE_IN_PROGRESS which causes hang.
2020-04-27 18:56:35 +00:00
- #5615
Enable librdkafka debugging in test_storage_kafka
2020-05-04 11:27:15 +00:00
XXX: for now this messages will appears in stderr.
-->
<debug>cgrp,consumer,topic,protocol</debug>
Allow configuration of Kafka topics with periods The Kafka table engine allows global configuration and per-Kafka-topic configuration. The latter uses syntax <kafka_TOPIC>, e.g. for topic "football": <kafka_football> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </kafka_football> Some users had to find out the hard way that such configuration doesn't take effect if the topic name contains a period, e.g. "sports.football". The reason is that ClickHouse configuration framework already uses periods as level separators to descend the configuration hierarchy. (Besides that, per-topic configuration at the same level as global configuration could be considered ugly.) Note that Kafka topics may contain characters "a-zA-Z0-9._-" (*) and a tree-like topic organization using periods is quite common in practice. This PR deprecates the existing per-topic configuration syntax (but continues to support it for backward compat) and introduces a new per-topic configuration syntax below the global Kafka configuration of the form: <kafka> <topic name="football"> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </topic> </kafka> The period restriction doesn't apply to XML attributes, so <topic name="sports.football"> will work. Also, everything Kafka-related is below <kafka>. Considered but rejected alternatives: - Extending Poco ConfigurationView with custom separators (e.g."/" instead of "."). Won't work easily because ConfigurationView only builds a path but defers descending the configuration tree to the normal configuration classes. - Reloading the configuration file in StorageKafka (instead of reading the loaded file) but with a custom separator. This mode is supported by XML configuration. Too ugly and error-prone since the true configuration is composed from multiple configuration files. (*) https://stackoverflow.com/a/37067544
2023-02-22 19:58:48 +00:00
system_kafka_consumers: cleanup, test should be more stable
2023-07-10 14:03:49 +00:00
<!-- librdkafka stat in system.kafka_consumers -->
Support sql created named collections in Kafka Storage for librdkafka settings
2024-02-27 18:36:36 +00:00
<!-- default 3000 (every three second) seems too long for test -->
<statistics_interval_ms>600</statistics_interval_ms>
system_kafka_consumers: cleanup, test should be more stable
2023-07-10 14:03:49 +00:00
Change to S3 cfg syntax
2023-02-23 20:04:41 +00:00
<kafka_topic>
<name>consumer_hang</name>
Allow configuration of Kafka topics with periods The Kafka table engine allows global configuration and per-Kafka-topic configuration. The latter uses syntax <kafka_TOPIC>, e.g. for topic "football": <kafka_football> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </kafka_football> Some users had to find out the hard way that such configuration doesn't take effect if the topic name contains a period, e.g. "sports.football". The reason is that ClickHouse configuration framework already uses periods as level separators to descend the configuration hierarchy. (Besides that, per-topic configuration at the same level as global configuration could be considered ugly.) Note that Kafka topics may contain characters "a-zA-Z0-9._-" (*) and a tree-like topic organization using periods is quite common in practice. This PR deprecates the existing per-topic configuration syntax (but continues to support it for backward compat) and introduces a new per-topic configuration syntax below the global Kafka configuration of the form: <kafka> <topic name="football"> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </topic> </kafka> The period restriction doesn't apply to XML attributes, so <topic name="sports.football"> will work. Also, everything Kafka-related is below <kafka>. Considered but rejected alternatives: - Extending Poco ConfigurationView with custom separators (e.g."/" instead of "."). Won't work easily because ConfigurationView only builds a path but defers descending the configuration tree to the normal configuration classes. - Reloading the configuration file in StorageKafka (instead of reading the loaded file) but with a custom separator. This mode is supported by XML configuration. Too ugly and error-prone since the true configuration is composed from multiple configuration files. (*) https://stackoverflow.com/a/37067544
2023-02-22 19:58:48 +00:00
<!-- default: 3000 -->
refactore: improve reading several configurations for kafka Simplify and do some refactoring for kafka client settings. Allows to set up separate settings for consumer and producer like: ``` <consumer> ... </consumer> <producer> <kafka_topic> <name>topic_name</name> ... </kafka_topic> </producer> ``` Moreover, this fixes warnings from kafka client like: `Configuration property session.timeout.ms is a consumer property and will be ignored by this producer instance`
2024-03-27 11:19:26 +00:00
<heartbeat_interval_ms>301</heartbeat_interval_ms>
Change to S3 cfg syntax
2023-02-23 20:04:41 +00:00
</kafka_topic>
refactore: improve reading several configurations for kafka Simplify and do some refactoring for kafka client settings. Allows to set up separate settings for consumer and producer like: ``` <consumer> ... </consumer> <producer> <kafka_topic> <name>topic_name</name> ... </kafka_topic> </producer> ``` Moreover, this fixes warnings from kafka client like: `Configuration property session.timeout.ms is a consumer property and will be ignored by this producer instance`
2024-03-27 11:19:26 +00:00
<kafka_separate_settings>
<!-- This is old syntax. This setting will be set for consumer and producer-->
<!-- default 3000 -->
<heartbeat_interval_ms>302</heartbeat_interval_ms>
</kafka_separate_settings>
<consumer>
<auto_offset_reset>earliest</auto_offset_reset>
<kafka_topic>
<!-- Setting for topic will be applied only for consumer -->
<name>consumer_hang</name>
<!-- default: 10000 -->
<session_timeout_ms>6000</session_timeout_ms>
</kafka_topic>
<kafka_topic>
<name>separate_settings</name>
<!-- default: 10000 -->
<session_timeout_ms>6001</session_timeout_ms>
</kafka_topic>
</consumer>
<producer>
<kafka_separate_settings>
<!-- This is old syntax. And doesn't work inside producer and consumer tags-->
<!-- default 300000 -->
<message_timeout_ms>300001</message_timeout_ms>
</kafka_separate_settings>
<kafka_topic>
<name>separate_settings</name>
<!-- This is only producer setting, if it was applied to consumer,
it would create warning message in logs. -->
<!-- default 30000 -->
<request_timeout_ms>30001</request_timeout_ms>
</kafka_topic>
<!-- default 60000 -->
<transaction_timeout_ms>60001</transaction_timeout_ms>
</producer>
Add row_delimiter argument to StorageKafka. There are common cases where a message doesn't end with a row delimiter. This patch allows specifying a row_delimiter char to compensate that. https://github.com/yandex/ClickHouse/issues/2298
2018-07-18 05:22:01 +00:00
</kafka>
Add integration test for kafka consumer hang This test triggers heartbeat failrues which leads to REBALANCE_IN_PROGRESS which causes hang.
2020-04-27 18:56:35 +00:00
Change <yandex> to <clickhouse> in configs
2021-09-19 22:38:53 +00:00
</clickhouse>
Reference in New Issue Copy Permalink