ClickHouse/tests/integration/test_storage_kafka/configs/kafka.xml

<clickhouse>
    <kafka>
        <auto_offset_reset>earliest</auto_offset_reset>
        <!-- Debugging of possible issues, like:
             - https://github.com/edenhill/librdkafka/issues/2077
             - https://github.com/edenhill/librdkafka/issues/1778
             - #5615

             XXX: for now this messages will appears in stderr.
        -->
        <debug>cgrp,consumer,topic,protocol</debug>

        <!-- librdkafka stat in system.kafka_consumers -->
				<!-- default 3000 (every three second) seems too long for test -->
				<statistics_interval_ms>600</statistics_interval_ms>

        <kafka_topic>
            <name>consumer_hang</name>
            <!-- default: 3000 -->
            <heartbeat_interval_ms>300</heartbeat_interval_ms>
            <!-- default: 10000 -->
            <session_timeout_ms>6000</session_timeout_ms>
        </kafka_topic>
    </kafka>

</clickhouse>
Change <yandex> to <clickhouse> in configs 2021-09-19 22:38:53 +00:00			`<clickhouse>`
Add row_delimiter argument to StorageKafka. There are common cases where a message doesn't end with a row delimiter. This patch allows specifying a row_delimiter char to compensate that. https://github.com/yandex/ClickHouse/issues/2298 2018-07-18 05:22:01 +00:00			`<kafka>`
			`<auto_offset_reset>earliest</auto_offset_reset>`
Enable librdkafka debugging in test_storage_kafka 2020-05-04 11:27:15 +00:00			`<!-- Debugging of possible issues, like:`
			`- https://github.com/edenhill/librdkafka/issues/2077`
			`- https://github.com/edenhill/librdkafka/issues/1778`
Add integration test for kafka consumer hang This test triggers heartbeat failrues which leads to REBALANCE_IN_PROGRESS which causes hang. 2020-04-27 18:56:35 +00:00			`- #5615`
Enable librdkafka debugging in test_storage_kafka 2020-05-04 11:27:15 +00:00
			`XXX: for now this messages will appears in stderr.`
			`-->`
			`<debug>cgrp,consumer,topic,protocol</debug>`
Allow configuration of Kafka topics with periods The Kafka table engine allows global configuration and per-Kafka-topic configuration. The latter uses syntax <kafka_TOPIC>, e.g. for topic "football": <kafka_football> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </kafka_football> Some users had to find out the hard way that such configuration doesn't take effect if the topic name contains a period, e.g. "sports.football". The reason is that ClickHouse configuration framework already uses periods as level separators to descend the configuration hierarchy. (Besides that, per-topic configuration at the same level as global configuration could be considered ugly.) Note that Kafka topics may contain characters "a-zA-Z0-9._-" () and a tree-like topic organization using periods is quite common in practice. This PR deprecates the existing per-topic configuration syntax (but continues to support it for backward compat) and introduces a new per-topic configuration syntax below the global Kafka configuration of the form: <kafka> <topic name="football"> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </topic> </kafka> The period restriction doesn't apply to XML attributes, so <topic name="sports.football"> will work. Also, everything Kafka-related is below <kafka>. Considered but rejected alternatives: - Extending Poco ConfigurationView with custom separators (e.g."/" instead of "."). Won't work easily because ConfigurationView only builds a path but defers descending the configuration tree to the normal configuration classes. - Reloading the configuration file in StorageKafka (instead of reading the loaded file) but with a custom separator. This mode is supported by XML configuration. Too ugly and error-prone since the true configuration is composed from multiple configuration files. () https://stackoverflow.com/a/37067544 2023-02-22 19:58:48 +00:00
system_kafka_consumers: cleanup, test should be more stable 2023-07-10 14:03:49 +00:00			`<!-- librdkafka stat in system.kafka_consumers -->`
			`<!-- default 3000 (every three second) seems too long for test -->`
			`<statistics_interval_ms>600</statistics_interval_ms>`

Change to S3 cfg syntax 2023-02-23 20:04:41 +00:00			`<kafka_topic>`
			`<name>consumer_hang</name>`
Allow configuration of Kafka topics with periods The Kafka table engine allows global configuration and per-Kafka-topic configuration. The latter uses syntax <kafka_TOPIC>, e.g. for topic "football": <kafka_football> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </kafka_football> Some users had to find out the hard way that such configuration doesn't take effect if the topic name contains a period, e.g. "sports.football". The reason is that ClickHouse configuration framework already uses periods as level separators to descend the configuration hierarchy. (Besides that, per-topic configuration at the same level as global configuration could be considered ugly.) Note that Kafka topics may contain characters "a-zA-Z0-9._-" () and a tree-like topic organization using periods is quite common in practice. This PR deprecates the existing per-topic configuration syntax (but continues to support it for backward compat) and introduces a new per-topic configuration syntax below the global Kafka configuration of the form: <kafka> <topic name="football"> <retry_backoff_ms>250</retry_backoff_ms> <fetch_min_bytes>100000</fetch_min_bytes> </topic> </kafka> The period restriction doesn't apply to XML attributes, so <topic name="sports.football"> will work. Also, everything Kafka-related is below <kafka>. Considered but rejected alternatives: - Extending Poco ConfigurationView with custom separators (e.g."/" instead of "."). Won't work easily because ConfigurationView only builds a path but defers descending the configuration tree to the normal configuration classes. - Reloading the configuration file in StorageKafka (instead of reading the loaded file) but with a custom separator. This mode is supported by XML configuration. Too ugly and error-prone since the true configuration is composed from multiple configuration files. () https://stackoverflow.com/a/37067544 2023-02-22 19:58:48 +00:00			`<!-- default: 3000 -->`
			`<heartbeat_interval_ms>300</heartbeat_interval_ms>`
			`<!-- default: 10000 -->`
			`<session_timeout_ms>6000</session_timeout_ms>`
Change to S3 cfg syntax 2023-02-23 20:04:41 +00:00			`</kafka_topic>`
Add row_delimiter argument to StorageKafka. There are common cases where a message doesn't end with a row delimiter. This patch allows specifying a row_delimiter char to compensate that. https://github.com/yandex/ClickHouse/issues/2298 2018-07-18 05:22:01 +00:00			`</kafka>`
Add integration test for kafka consumer hang This test triggers heartbeat failrues which leads to REBALANCE_IN_PROGRESS which causes hang. 2020-04-27 18:56:35 +00:00
Change <yandex> to <clickhouse> in configs 2021-09-19 22:38:53 +00:00			`</clickhouse>`