Aleksandr Musorin
f9d81bc774
refactore: improve reading several configurations for kafka
...
Simplify and do some refactoring for kafka client settings.
Allows to set up separate
settings for consumer and producer like:
```
<consumer>
...
</consumer>
<producer>
<kafka_topic>
<name>topic_name</name>
...
</kafka_topic>
</producer>
```
Moreover, this fixes warnings from kafka client like:
`Configuration property session.timeout.ms is a consumer property and
will be ignored by this producer instance`
2024-03-27 14:53:33 +01:00
Alexey Milovidov
06a9ac7126
Fix tests
2024-03-24 23:23:13 +01:00
Max Kainov
55737de0e6
disable more hdfs and kafka inttests
...
#ci_set_arm
2024-03-20 17:24:55 +00:00
Kseniia Sumarokova
aec76d6653
Merge pull request #58877 from ClickHouse/multiple-read-file-log-storage-in-mv
...
Multiple read file log storage in mv
2024-01-22 12:02:41 +01:00
Kruglov Pavel
6d064512e1
Merge pull request #58614 from Blargian/58363_disable_ansi_pretty_automatically
...
58363 Automatically disable ANSI escape sequences in Pretty formats if the output is not a terminal
2024-01-17 13:45:41 +01:00
Blargian
1b81efc337
fix failing 02421 stateless and (hopefully) fix failing integration tests
2024-01-16 19:48:23 +02:00
robot-clickhouse
7262e93a27
Automatic style fix
2024-01-16 16:26:28 +00:00
János Benjamin Antal
5a178ce1ef
Add proper cleanup to kafka test
2024-01-16 16:03:44 +00:00
robot-clickhouse
224a58ddc5
Automatic style fix
2024-01-04 10:30:44 +00:00
János Benjamin Antal
1a1fa07128
Add missing asserts
2024-01-04 10:18:04 +00:00
János Benjamin Antal
2517d23e27
Add tests
2024-01-04 10:14:41 +00:00
avogar
f20917048e
Fix kafka tests
2023-12-29 17:02:46 +00:00
Kruglov Pavel
9389ea8745
Fix style
2023-12-28 18:47:57 +00:00
Kruglov Pavel
69f6147499
Fix style
2023-12-28 18:47:57 +00:00
avogar
c6085b3eb9
Fix working with read buffers in StreamingFormatExecutor
2023-12-28 18:47:57 +00:00
Raúl Marín
6c76ad1ea4
Cleanup
2023-08-22 11:57:56 +02:00
Raúl Marín
f2d3c9c281
Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert
2023-08-21 12:14:31 +02:00
Ilya Golshtein
978d8bf9a6
system_kafka_consumers: cleanup with exceptions
2023-08-10 22:35:05 +00:00
Ilya Golshtein
7dab2dc041
system_kafka_consumers: exceptions seem to work
2023-08-10 22:35:05 +00:00
Ilya Golshtein
e8b10f0126
system_kafka_consumers: black formatter thing
2023-08-10 22:35:05 +00:00
Ilya Golshtein
80829a6bf7
system_kafka_consumers: no offset_committed, new test with MV
2023-08-10 22:35:05 +00:00
Ilya Golshtein
ae3e6f0fac
system_kafka_consumers: getMemberId introduced to avoid leak
2023-08-10 22:35:05 +00:00
Ilya Golshtein
2cc4bb355c
system_kafka_consumers: make black formatter happy
2023-08-10 22:35:05 +00:00
Ilya Golshtein
46be5e5782
system_kafka_consumers: cleanup, test should be more stable
2023-08-10 22:35:05 +00:00
Ilya Golshtein
1f03e320ee
system_kafka_consumers: test fix + black formatter
2023-08-10 22:35:05 +00:00
Ilya Golshtein
b71c2bedfc
system_kafka_consumers: rebalance test is ok, statistics_interval_ms
2023-08-10 22:35:05 +00:00
Ilya Golshtein
ae6e0469ad
system_kafka_consumers: rdfakfa stat with attempt to use json
2023-08-10 22:35:05 +00:00
Ilya Golshtein
40fd53c3e3
system_kafka_consumers: style check
2023-08-10 22:35:05 +00:00
Ilya Golshtein
88f5dd5db8
system_kafka_consumers: integration tests passed
2023-08-10 22:35:05 +00:00
Ilya Golshtein
5e1c72a0d0
system_kafka_consumers: test with rebalance
2023-08-10 22:35:05 +00:00
Ilya Golshtein
957787d96a
system_kafka_consumers: kind of works, test added
2023-08-10 22:35:05 +00:00
Yury Bogomolov
ba081ed402
Merge branch 'master' into support-predefined-macro-in-kafka-settings
2023-07-29 15:00:21 +04:00
Yury Bogomolov
b2ee4505fb
review fixes
2023-07-28 16:06:03 +00:00
Raúl Marín
e2a3a0e37a
Adapt test
2023-07-28 13:33:14 +02:00
Alexey Milovidov
a1e9a76ead
Merge branch 'master' into 51844-fix-test_storage_kafkatestpytest_kafka_formats_with_broken_message-test
2023-07-28 13:18:10 +03:00
Yury Bogomolov
c8b128aad4
add predefined macros support in kafka engine settings
2023-07-21 01:06:49 +04:00
János Benjamin Antal
16cc007842
Fix table ad variable name
2023-07-20 09:21:18 +00:00
Ilya Golshtein
aa2b7e248d
test_for_basic_auth_registry - fix port in test_kafka_formats
2023-07-19 08:32:45 +00:00
János Benjamin Antal
6a104cc3f7
Add checked waits to improve kafka tests
...
It might take a few minutes to receive all the messages for the last
materialized view. By waiting to the number of expected results the
happy path of execution takes minimal time while becoming more stable,
while the erroneous path might take a bit longer.
2023-07-18 16:04:41 +00:00
AVMusorin
418a61a68c
Allow using Alias column type for KafkaEngine
...
```
create table kafka
(
a UInt32,
a_str String Alias toString(a)
) engine = Kafka;
create table data
(
a UInt32;
a_str String
) engine = MergeTree
order by tuple();
create materialized view data_mv to data
(
a UInt32,
a_str String
) as
select a, a_str from kafka;
```
Alias type works as expected in comparison with MATERIALIZED/EPHEMERAL
or column with default expression.
Ref: https://github.com/ClickHouse/ClickHouse/pull/47138
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
2023-05-15 15:39:58 +02:00
Mikhail f. Shiryaev
4f2c336cfc
Update black version, run it for the repository
2023-03-24 11:54:38 +01:00
AVMusorin
0f2ae72141
prohibit DEFAULT/EPHEMERAL/ALIAS in KafkaEngine
2023-03-03 16:25:42 +01:00
Antonio Andelic
890f094829
Set retention policy for Kafka topic
2023-01-04 12:04:00 +00:00
Kruglov Pavel
c3405be272
Increase the number of attempts in tests
2022-12-27 16:49:32 +01:00
avogar
8e13d1f1ec
Improve and refactor Kafka/StorageMQ/NATS and data formats
2022-10-28 16:41:10 +00:00
Kruglov Pavel
d4d0ec1c95
Fix style
2022-08-26 16:44:54 +02:00
avogar
b8061532ac
Add setting to disable limit on kafka_num_consumers
2022-08-26 12:49:39 +00:00
Anton Popov
e911900054
remove last mentions of data streams
2022-05-09 19:15:24 +00:00
Nikolai Kochetov
ebfdadd6f8
Fix firmatting.
2022-04-07 12:05:21 +00:00
Mikhail Filimonov
f9a4cd12d4
Make parallel reading of kafka consumers work again (issue #35153 )
2022-04-05 21:46:56 +02:00
