Commit Graph

19 Commits

Author SHA1 Message Date
Aleksandr Musorin
f9d81bc774 refactore: improve reading several configurations for kafka
Simplify and do some refactoring for kafka client settings.

Allows to set up separate
settings for consumer and producer like:

```
<consumer>
    ...
</consumer>

<producer>
    <kafka_topic>
        <name>topic_name</name>
        ...
    </kafka_topic>
</producer>
```

Moreover, this fixes warnings from kafka client like:
`Configuration property session.timeout.ms is a consumer property and
will be ignored by this producer instance`
2024-03-27 14:53:33 +01:00
Pervakov Grigorii
d3b82ab7c1 Support sql created named collections in Kafka Storage for librdkafka settings 2024-02-27 18:36:36 +00:00
Ilya Golshtein
46be5e5782 system_kafka_consumers: cleanup, test should be more stable 2023-08-10 22:35:05 +00:00
Alexey Milovidov
b2164ab97e Fix integration tests 2023-07-18 22:13:28 +02:00
kssenii
8073e0bad1 Fix tests 2023-06-14 13:45:53 +02:00
Robert Schulze
6906cafa99
Change to S3 cfg syntax 2023-02-23 20:04:41 +00:00
Robert Schulze
81bf43157f
Allow configuration of Kafka topics with periods
The Kafka table engine allows global configuration and per-Kafka-topic
configuration. The latter uses syntax <kafka_TOPIC>, e.g. for topic
"football":

  <kafka_football>
      <retry_backoff_ms>250</retry_backoff_ms>
      <fetch_min_bytes>100000</fetch_min_bytes>
  </kafka_football>

Some users had to find out the hard way that such configuration doesn't
take effect if the topic name contains a period, e.g. "sports.football".
The reason is that ClickHouse configuration framework already uses
periods as level separators to descend the configuration hierarchy.
(Besides that, per-topic configuration at the same level as global
configuration could be considered ugly.)

Note that Kafka topics may contain characters "a-zA-Z0-9._-" (*) and
a tree-like topic organization using periods is quite common in
practice.

This PR deprecates the existing per-topic configuration syntax (but
continues to support it for backward compat) and introduces a new
per-topic configuration syntax below the global Kafka configuration of
the form:

<kafka>
   <topic name="football">
       <retry_backoff_ms>250</retry_backoff_ms>
       <fetch_min_bytes>100000</fetch_min_bytes>
   </topic>
</kafka>

The period restriction doesn't apply to XML attributes, so <topic
name="sports.football"> will work. Also, everything Kafka-related is
below <kafka>.

Considered but rejected alternatives:
- Extending Poco ConfigurationView with custom separators (e.g."/"
  instead of "."). Won't work easily because ConfigurationView only
  builds a path but defers descending the configuration tree to the
  normal configuration classes.
- Reloading the configuration file in StorageKafka (instead of reading
  the loaded file) but with a custom separator. This mode is supported
  by XML configuration. Too ugly and error-prone since the true
  configuration is composed from multiple configuration files.

(*) https://stackoverflow.com/a/37067544
2023-02-22 20:35:09 +00:00
Alexey Milovidov
4a2bd6e0aa Adapt a test 2023-02-21 00:43:10 +01:00
Alexey Milovidov
dc4b52e54e Adapt a test 2023-02-21 00:41:40 +01:00
Alexey Milovidov
b42d26acfe Remove one line from XML, because we do not care 2022-08-28 02:44:02 +02:00
kssenii
2ea4e7c4e7 Better 2021-11-29 15:05:02 +00:00
kssenii
980c87c466 Add named collection for kafka / rabbit 2021-11-24 14:53:01 +03:00
Alexey Milovidov
e513a5db32 Change <yandex> to <clickhouse> in configs 2021-09-20 01:38:53 +03:00
Vitaly Baranov
3a3cbeb7aa Collect stderr.log and stdout.log in all integration tests by default. 2021-06-29 17:30:47 +03:00
Yatsishin Ilya
70963e4396 Fixed test_kafka_flush_by_time test. 2021-01-28 18:03:23 +03:00
Mikhail Filimonov
0d55d364da Add test for macros usage in certain kafka setting.
https://github.com/ClickHouse/ClickHouse/issues/2249
918dbc2902
2020-07-30 11:28:52 +02:00
Azat Khuzhin
7ab0c3493c Add integration test for kafka consumer hang
This test triggers heartbeat failrues which leads to
REBALANCE_IN_PROGRESS which causes hang.
2020-05-04 18:46:30 +03:00
Azat Khuzhin
fc5f0bfe52 Enable librdkafka debugging in test_storage_kafka 2020-05-04 17:23:09 +03:00
Ivan
97f2a2213e
Move all folders inside /dbms one level up (#9974)
* Move some code outside dbms/src folder
* Fix paths
2020-04-02 02:51:21 +03:00