thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-27 18:12:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
803a80b098
ClickHouse/tests/integration/test_storage_kafka/configs/kafka.xml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

23 lines
717 B
XML
Raw Normal View History

Change <yandex> to <clickhouse> in configs
2021-09-19 22:38:53 +00:00
<clickhouse>
Add row_delimiter argument to StorageKafka. There are common cases where a message doesn't end with a row delimiter. This patch allows specifying a row_delimiter char to compensate that. https://github.com/yandex/ClickHouse/issues/2298
2018-07-18 05:22:01 +00:00
<kafka>
<auto_offset_reset>earliest</auto_offset_reset>
Enable librdkafka debugging in test_storage_kafka
2020-05-04 11:27:15 +00:00
<!-- Debugging of possible issues, like:
- https://github.com/edenhill/librdkafka/issues/2077
- https://github.com/edenhill/librdkafka/issues/1778
Add integration test for kafka consumer hang This test triggers heartbeat failrues which leads to REBALANCE_IN_PROGRESS which causes hang.
2020-04-27 18:56:35 +00:00
- #5615
Enable librdkafka debugging in test_storage_kafka
2020-05-04 11:27:15 +00:00
XXX: for now this messages will appears in stderr.
-->
<debug>cgrp,consumer,topic,protocol</debug>
Allow configuration of Kafka topics with periods The Kafka table engine allows global configuration and per-Kafka-topic configuration. The latter uses syntax <kafka_TOPIC>, e.g. for topic "football": <kafka_football> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </kafka_football> Some users had to find out the hard way that such configuration doesn't take effect if the topic name contains a period, e.g. "sports.football". The reason is that ClickHouse configuration framework already uses periods as level separators to descend the configuration hierarchy. (Besides that, per-topic configuration at the same level as global configuration could be considered ugly.) Note that Kafka topics may contain characters "a-zA-Z0-9._-" (*) and a tree-like topic organization using periods is quite common in practice. This PR deprecates the existing per-topic configuration syntax (but continues to support it for backward compat) and introduces a new per-topic configuration syntax below the global Kafka configuration of the form: <kafka> <topic name="football"> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </topic> </kafka> The period restriction doesn't apply to XML attributes, so <topic name="sports.football"> will work. Also, everything Kafka-related is below <kafka>. Considered but rejected alternatives: - Extending Poco ConfigurationView with custom separators (e.g."/" instead of "."). Won't work easily because ConfigurationView only builds a path but defers descending the configuration tree to the normal configuration classes. - Reloading the configuration file in StorageKafka (instead of reading the loaded file) but with a custom separator. This mode is supported by XML configuration. Too ugly and error-prone since the true configuration is composed from multiple configuration files. (*) https://stackoverflow.com/a/37067544
2023-02-22 19:58:48 +00:00
Change to S3 cfg syntax
2023-02-23 20:04:41 +00:00
<kafka_topic>
<name>consumer_hang</name>
Allow configuration of Kafka topics with periods The Kafka table engine allows global configuration and per-Kafka-topic configuration. The latter uses syntax <kafka_TOPIC>, e.g. for topic "football": <kafka_football> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </kafka_football> Some users had to find out the hard way that such configuration doesn't take effect if the topic name contains a period, e.g. "sports.football". The reason is that ClickHouse configuration framework already uses periods as level separators to descend the configuration hierarchy. (Besides that, per-topic configuration at the same level as global configuration could be considered ugly.) Note that Kafka topics may contain characters "a-zA-Z0-9._-" (*) and a tree-like topic organization using periods is quite common in practice. This PR deprecates the existing per-topic configuration syntax (but continues to support it for backward compat) and introduces a new per-topic configuration syntax below the global Kafka configuration of the form: <kafka> <topic name="football"> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </topic> </kafka> The period restriction doesn't apply to XML attributes, so <topic name="sports.football"> will work. Also, everything Kafka-related is below <kafka>. Considered but rejected alternatives: - Extending Poco ConfigurationView with custom separators (e.g."/" instead of "."). Won't work easily because ConfigurationView only builds a path but defers descending the configuration tree to the normal configuration classes. - Reloading the configuration file in StorageKafka (instead of reading the loaded file) but with a custom separator. This mode is supported by XML configuration. Too ugly and error-prone since the true configuration is composed from multiple configuration files. (*) https://stackoverflow.com/a/37067544
2023-02-22 19:58:48 +00:00
<!-- default: 3000 -->
<heartbeat_interval_ms>300</heartbeat_interval_ms>
<!-- default: 10000 -->
<session_timeout_ms>6000</session_timeout_ms>
Change to S3 cfg syntax
2023-02-23 20:04:41 +00:00
</kafka_topic>
Add row_delimiter argument to StorageKafka. There are common cases where a message doesn't end with a row delimiter. This patch allows specifying a row_delimiter char to compensate that. https://github.com/yandex/ClickHouse/issues/2298
2018-07-18 05:22:01 +00:00
</kafka>
Add integration test for kafka consumer hang This test triggers heartbeat failrues which leads to REBALANCE_IN_PROGRESS which causes hang.
2020-04-27 18:56:35 +00:00
Change <yandex> to <clickhouse> in configs
2021-09-19 22:38:53 +00:00
</clickhouse>
Reference in New Issue Copy Permalink