Mikhail f. Shiryaev
e6f5a3f98b
Apply black formatter to all *.py files in the repo
2022-03-22 17:39:58 +01:00
alesapin
1eefdc8df6
Fix more test
2022-03-15 17:31:33 +01:00
alesapin
ace8169732
Fix kafka test
2022-01-31 19:18:34 +03:00
Azat Khuzhin
63eac4b0cf
tests/integration/test_storage_kafka: cover CustomSeparated
2021-12-31 12:02:21 +03:00
Azat Khuzhin
f1c726a675
tests/integration/test_storage_kafka: cover Template format
2021-12-31 11:25:08 +03:00
Azat Khuzhin
52ad7a4538
Fix test_storage_kafka failures by adjusting retention.ms
...
This test uses predefined timestamps, and default retention.ms is too
small:
kafka1_1 | [2021-12-28 21:40:21,842] INFO Created log for partition virt2_0-0 in /var/lib/kafka/data with properties {compression.type -> producer, message.format.version -> 2.2-IV1, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, message.downconversion.enable -> true, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
kafka1_1 | [2021-12-28 21:40:24,540] INFO [Log partition=virt2_0-0, dir=/var/lib/kafka/data] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log)
v2: fix tests with 0 timestamp_ms
and this should be all tests that was left:
$ fgrep 'Found deletable segments with base offsets' test_storage_kafka/_instances_0/docker.log
kafka1_1 | [2021-12-29 09:46:15,610] INFO [Log partition=avro1-0, dir=/var/lib/kafka/data] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log)
kafka1_1 | [2021-12-29 09:51:15,609] INFO [Log partition=virt1-0, dir=/var/lib/kafka/data] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log)
2021-12-30 10:25:10 +03:00
Vitaly Baranov
b5b195f4e2
Merge branch 'master' into kafka_protobuf_issue26643
2021-12-10 23:22:35 +03:00
kssenii
2ea4e7c4e7
Better
2021-11-29 15:05:02 +00:00
kssenii
2f0fe4a21f
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into rabbit-kafka-select-commit
2021-11-29 10:08:14 +00:00
Kruglov Pavel
b39c19399f
Merge pull request #31691 from kssenii/testing-1
...
Allow named_collections for kafka / rabbit
2021-11-25 14:06:31 +03:00
kssenii
980c87c466
Add named collection for kafka / rabbit
2021-11-24 14:53:01 +03:00
kssenii
35a4be601a
Select query commit only with settings
2021-11-03 15:43:58 +03:00
Yatsishin Ilya
d5138e1427
Merge remote-tracking branch 'origin' into integration--7
2021-11-02 17:41:40 +03:00
Kruglov Pavel
901ebcede6
Merge pull request #30351 from arenadata/ADQM-335
...
output_format_avro_rows_in_file
2021-11-02 12:25:27 +03:00
Ilya Golshtein
8efa1743cf
output_format_avro_rows_in_file fix and test for _timestamp
2021-11-01 13:09:18 +03:00
avogar
0ce893cb10
Fix kafka test
2021-10-28 14:20:29 +03:00
Kruglov Pavel
11c0d08d1c
Fix test again
2021-10-21 16:46:27 +03:00
Kruglov Pavel
bd016f384e
Try fix tests
2021-10-21 12:54:10 +03:00
avogar
cd17643eb8
Fix tests
2021-10-21 09:22:42 +03:00
Ilya Golshtein
d90302aa3b
output_format_avro_rows_in_file
2021-10-18 19:01:06 +03:00
Yatsishin Ilya
0efeaf5dc4
Merge remote-tracking branch 'origin' into integration--7
2021-10-08 11:22:55 +03:00
Nikolai Kochetov
78e1db209f
Remove more data streams ( #29491 )
...
* Remove more streams.
* Fixing build.
* Fixing build.
* Rename files.
* Fix fast test.
* Fix StorageKafka.
* Try fix kafka test.
* Move createBuffer to KafkaSource ctor.
* Revert "Move createBuffer to KafkaSource ctor."
This reverts commit 81fa94d27e2107e54969
2021-10-07 11:26:08 +03:00
Yatsishin Ilya
c32e833f25
better
2021-10-06 16:23:29 +03:00
Yatsishin Ilya
b0a5a8532f
improve test_storage_kafka test
2021-10-06 16:08:25 +03:00
Mikhail Filimonov
ea320c96d5
Test for issue #26643
2021-10-04 14:38:10 +00:00
alexey-milovidov
995913e89c
Merge branch 'master' into yandex-to-clickhouse-in-configs
2021-09-28 02:27:36 +03:00
Nikolai Kochetov
193f566568
Fix exception message.
2021-09-24 14:54:39 +03:00
Nikolai Kochetov
24dac111b7
Try fix kafka test.
2021-09-24 13:30:57 +03:00
Alexey Milovidov
e513a5db32
Change <yandex> to <clickhouse> in configs
2021-09-20 01:38:53 +03:00
alesapin
36abf9f2f4
Fix broken kafka test
2021-09-03 10:51:37 +03:00
Yatsishin Ilya
da4e688293
Improve kafka topic creation and deletion. Improve mongodb tests
2021-08-30 15:58:21 +03:00
Yatsishin Ilya
d9e782d379
Merge remote-tracking branch 'origin' into mute-integration-debug-fails
2021-08-20 18:59:16 +03:00
Yatsishin Ilya
812a6ffb80
kafka assertion was fixed in master
2021-08-20 18:56:15 +03:00
Yatsishin Ilya
9c22d07dea
Mute current failures
2021-08-19 14:32:32 +03:00
Mikhail Filimonov
05d77d2873
Safer processing of NULL messages in Kafka for some formats
2021-08-18 11:26:57 +02:00
Kseniia Sumarokova
0917488cec
Merge pull request #27382 from kssenii/cast-internal
...
add Cast internal function
2021-08-13 13:13:51 +03:00
Raúl Marín
d1d47658f5
Merge remote-tracking branch 'blessed/master' into materialization_log
2021-08-10 10:49:05 +02:00
kssenii
484c3a5d2e
Update kafka test
2021-08-09 11:55:38 +00:00
Raúl Marín
b4e417e507
test_kafka_virtual_columns_with_materialized_view: Limit iterations and show better error
2021-08-06 13:39:33 +02:00
Raúl Marín
4f1b02426a
test_storage_kafka: Remove duplicated code
2021-08-06 13:38:56 +02:00
Raúl Marín
145699d0c2
test_kafka_no_holes_when_write_suffix_failed: Fix expected error message
...
The old error message was incorrect (it said a failure happened when
pushing the prefix when it was happening while pushing the suffix.
2021-07-27 16:29:34 +02:00
Raúl Marín
ff249a14af
Fix test_kafka_no_holes_when_write_suffix_failed
2021-07-27 16:29:34 +02:00
vdimir
dd3e1a8a4d
Adjust exception message in test_replicated_fetches_timeouts
2021-07-15 11:58:21 +03:00
Vitaly Baranov
3a3cbeb7aa
Collect stderr.log and stdout.log in all integration tests by default.
2021-06-29 17:30:47 +03:00
alexey-milovidov
77721734ed
Merge pull request #21267 from filimonov/kafka-failover
...
Kafka failover issue fix
2021-06-14 01:42:42 +03:00
alexey-milovidov
77ab26e217
Merge branch 'master' into less_timeouts
2021-06-10 02:03:16 +03:00
filimonov
3f1d3f2e1d
Update test.py
2021-06-09 23:20:44 +02:00
Mikhail Filimonov
3627ab7cbe
Fix kafka failover issue ( #21118 ).
...
Distringuish lack of assignment vs lack of partitions. Bit better / clearer logging.
2021-06-09 16:02:56 +02:00
filimonov
7469f054f0
Update test.py
...
Missed one of the refactorings which were merged in the meanwhile 872e36c207 (diff-8129a3b04ff2293c2d89b35395be9ef49918cdafd4420a745d05733b00979646L86-R100)
2021-06-09 12:27:42 +02:00
alesapin
7179fafef0
Less timeouts in integration tests
2021-06-09 10:49:29 +03:00
mergify[bot]
1172807442
Merge branch 'master' into recreate_kafka_table_case
2021-06-06 12:15:55 +00:00
Mikhail Filimonov
c8fe168a8d
Fix the test after #20393
2021-06-04 15:55:07 +02:00
filimonov
74ae079b19
Update test.py
2021-06-04 15:30:25 +02:00
Mikhail Filimonov
e11f7db28b
Add test #23430
2021-06-03 20:41:01 +02:00
Yatsishin Ilya
28af06f3d0
Merge remote-tracking branch 'origin' into integration-2
2021-05-27 07:01:22 +03:00
Yatsishin Ilya
893cd47bd2
better
2021-05-25 15:40:59 +03:00
filimonov
88919080d5
Try to improve kafka flaky test
2021-05-24 22:37:16 +02:00
Yatsishin Ilya
e1176acc04
Merge remote-tracking branch 'origin' into integration-2
2021-05-17 09:53:00 +03:00
Mikhail Filimonov
dd619e1e37
Try to stabilize flapping test
2021-05-13 16:03:32 +02:00
Yatsishin Ilya
9d159072ef
mERGe remote-tracking branch 'origin' into integration-2
2021-05-11 13:32:44 +03:00
Chao Ma
ec8ca5bf5e
Fix kafka storage does not work with parquet format messages
2021-05-06 11:32:58 +08:00
Mikhail Filimonov
80065b08f8
Revert "avoid race in librdkafka"
...
This reverts commit bd8c3eb117
2021-05-04 10:06:33 +02:00
Yatsishin Ilya
c59a11f798
Merge remote-tracking branch 'origin' into integration-2
2021-04-27 18:34:33 +03:00
Chao Ma
713eb9486c
Fix kafka storage does not work with arrow and arrowstream format messages
2021-04-21 19:44:09 +08:00
Yatsishin Ilya
f6d455732e
better
2021-04-14 19:04:17 +03:00
Yatsishin Ilya
db657c03c8
fix
2021-04-12 18:08:09 +03:00
Yatsishin Ilya
ec6adb692d
Merge remote-tracking branch 'origin' into integration-2
2021-04-12 10:04:11 +03:00
Peng Jian
f73faf1578
fix test case
2021-04-08 14:26:32 +08:00
Peng Jian
16e971ee52
fix test case
2021-04-08 07:19:57 +08:00
Peng Jian
604ba9db60
add test case
2021-04-07 20:15:15 +08:00
Peng Jian
e30c07db20
add test cases
2021-03-31 20:22:36 +08:00
Peng Jian
cc56302e8b
fix test case
2021-03-31 17:16:51 +08:00
Peng Jian
17a5fc6e0e
add test
2021-03-31 17:16:51 +08:00
Yatsishin Ilya
7b7580204c
fix
2021-03-28 00:26:53 +03:00
Yatsishin Ilya
fbf0fe6436
Merge remote-tracking branch 'origin' into integration-2
2021-03-20 22:47:33 +03:00
Vladimir
c8b5be636f
Merge pull request #21438 from arenadata/ADQM-170
2021-03-19 17:06:55 +03:00
Ilya Golshtein
518103f851
old Avro test data removed
2021-03-16 19:39:58 +03:00
Ilya Golshtein
abcf581005
avro bug with format_setting reference and a typo in kafka test
2021-03-15 12:19:33 +03:00
Ilya Golshtein
a579fcb424
kafka/avro works against table
2021-03-14 15:55:44 +03:00
Yatsishin Ilya
7bfa4bb64b
Merge remote-tracking branch 'origin' into integration-2
2021-03-09 10:33:15 +03:00
filimonov
bd8c3eb117
avoid race in librdkafka
...
see https://github.com/edenhill/librdkafka/issues/3279
2021-03-04 15:01:10 +01:00
Yatsishin Ilya
298824c926
fix
2021-03-04 14:52:56 +03:00
Yatsishin Ilya
f03cf47a82
fix
2021-03-04 14:22:19 +03:00
Yatsishin Ilya
8122003c98
more fixes
2021-03-04 13:53:50 +03:00
Yatsishin Ilya
34609445f7
fix
2021-03-04 12:06:39 +03:00
Ilya Golshtein
f11a8cd13c
multimessage avro format for kafka test
2021-03-04 11:07:29 +03:00
Yatsishin Ilya
babea212c7
Merge remote-tracking branch 'origin' into integration-2
2021-03-03 11:56:15 +03:00
Mikhail Filimonov
9e3547681b
try to address python kafka client disconnects better
2021-02-26 07:17:31 +01:00
Mikhail Filimonov
60ede36068
simple fix for CI failure
2021-02-26 06:41:11 +01:00
Mikhail Filimonov
c4470ff283
proper placement for test
2021-02-26 06:38:14 +01:00
Mikhail Filimonov
2bcfff257c
Remove most of sleeps in tests
2021-02-25 17:02:59 +01:00
Mikhail Filimonov
6c6eaf2a60
Fixes
2021-02-25 17:02:57 +01:00
Mikhail Filimonov
a4c9e62d6d
Integration tests: introduce wait_for_log_line.
...
It uses tail -f executed in container, this way i don't need to pull
the file many times, rememeber positions etc.
2021-02-25 17:02:56 +01:00
Yatsishin Ilya
905cc7a2bd
fixes
2021-02-24 14:46:58 +03:00
Yatsishin Ilya
2df4317aba
wip
2021-02-19 00:21:50 +03:00
Yatsishin Ilya
609c9b272f
fix
2021-02-18 16:42:31 +03:00
Yatsishin Ilya
872e36c207
wip
2021-02-18 15:57:45 +03:00
Yatsishin Ilya
70963e4396
Fixed test_kafka_flush_by_time test.
2021-01-28 18:03:23 +03:00
Nikita Mikhaylov
736cc6959a
remove kafka flaky fix
2020-12-30 17:24:39 +03:00
filimonov
e81a99156f
Attempt to prevent test flap
...
That extra SELECT is executed synchronously, after that all further actions will have a more stable timing.
2020-12-29 18:51:00 +01:00
Nikita Mikhailov
fbf2ac35e8
fix tsan
2020-12-17 18:14:09 +03:00
Azat Khuzhin
2e569899d1
Add a test for kafka with snappy compression method (regression for UB in snappy)
...
I've checked it manually and it works with clang build and does not with
gcc (there are NULL bytes in the middle of value).
2020-12-14 00:38:07 +03:00
filimonov
8e2ca9518b
Update test.py
...
adjusting timeouts a bit, in the good hope that it will prevent flakiness of the test.
2020-12-11 10:13:03 +01:00
Mikhail Filimonov
b072bc7d2b
Test for the issue #12615
2020-11-06 17:54:44 +01:00
Mikhail Filimonov
e17d4e1eeb
Tests and some docs
2020-10-12 07:51:46 +02:00
Mikhail Filimonov
b602d18dd8
ProtobufSingle input format
...
Allows to read Single protobuf message at once (w/o length-delemeters).
2020-10-12 07:51:36 +02:00
filimonov
1ce0d4dbc2
Update test.py
...
similar change for test_kafka_duplicates_when_commit_failed
2020-10-10 23:36:59 +02:00
Mikhail Filimonov
60f4e561fb
fix test_kafka_produce_key_timestamp
2020-10-10 17:01:22 +02:00
Mikhail Filimonov
5fb549ca24
fix flap in no_holes_when_write_suffix_failed
2020-10-09 17:06:36 +02:00
Azat Khuzhin
9cb3c743bd
Convert to python3 ( #15007 )
2020-10-02 19:54:07 +03:00
bharatnc
b75963d370
Format and optimize imports in integration test files
...
This PR formats all the `*.py` files found under the `tests/integration`
folder. It also reorders the imports and cleans up a bunch of unused
imports.
The formatting also takes care of other things like wrapping lines and
fixing spaces and indents such that the tests look more readable.
2020-09-15 23:10:54 -07:00
Peng Jian
77420c82ea
Add test case for Kafka engine with kafka_thread_per_consumer enabled.
2020-09-06 11:17:19 +08:00
Yatsishin Ilya
6902bbce50
Merge remote-tracking branch 'origin' into integration-explicit-configs
2020-09-02 12:48:07 +03:00
Dongdong Yang
9651c36ceb
testcase for issue 14202
2020-08-31 15:42:01 +08:00
Yatsishin Ilya
adc2c117c8
fixes
2020-08-18 17:34:04 +03:00
alesapin
d387b97a69
Merge pull request #13285 from filimonov/kafka_fix_test_rdkafka_15
...
Fix for test_kafka_flush_by_block_size after rdkafka 1.5
2020-08-06 11:48:21 +03:00
Andrew Onyshchuk
a46049f86d
test_storage_kafka: test empty message when supported by format
2020-08-04 02:43:42 -05:00
Andrew Onyshchuk
e472c83601
Add AvroConfluent kafka test
2020-08-04 02:43:42 -05:00
Mikhail Filimonov
991c673870
Fix for test_kafka_flush_by_block_size after rdkafka 1.5
...
Since rdkafka 1.5 (https://github.com/edenhill/librdkafka/releases/tag/v1.5.0 )
don't auto-create topics when subscribing, so test start failing
because it was intentionally time-limited to check if flushing by block size worked.
Since 20.5 it's possible to increase kafka_flush_interval_ms per table, so i've
adjusted the test using that feature to test if data is flushed by block size,
not by time.
2020-08-03 18:43:37 +02:00
Alexander Kuzmenkov
4049f23188
Merge pull request #13103 from filimonov/test_for_macros_in_kafka
...
Add test for macros usage in some kafka settings.
2020-08-03 17:16:12 +03:00
alexey-milovidov
5e0437eeac
Merge pull request #12442 from ClickHouse/deprecate-special-zero-datetime
...
Deprecate special printing of zero Date/DateTime
2020-08-02 00:36:34 +03:00
alexey-milovidov
dc733e7c45
Fix Python
2020-07-31 01:23:16 +03:00
Alexey Milovidov
cfd4e9da3d
Update tests
2020-07-31 00:24:14 +03:00
Alexey Milovidov
83c9d9ef45
Merge branch 'master' into deprecate-special-zero-datetime
2020-07-31 00:17:04 +03:00
Alexey Milovidov
362655fa43
Fix some bad tests
2020-07-30 22:59:49 +03:00
Mikhail Filimonov
0d55d364da
Add test for macros usage in certain kafka setting.
...
https://github.com/ClickHouse/ClickHouse/issues/2249
918dbc2902
2020-07-30 11:28:52 +02:00
Azat Khuzhin
cdb959ec8e
Fix test_storage_kafka::test_kafka_unavailable
2020-07-22 11:42:30 +03:00
Alexey Milovidov
b0b0690e1c
Fix Python
2020-07-16 06:56:46 +03:00
Alexey Milovidov
126a94ab69
Update integration tests
2020-07-16 04:48:50 +03:00
Anton Popov
890bb2f582
add failing test with kafka
2020-07-14 12:24:12 +03:00
Mikhail Filimonov
9609bd9dee
Kafka better states, formats based on PeekableReadBuffer, and other minor fixes.
...
Add formats tests, fixes for JSONCompactEachRowWithNamesAndTypes, TSVWithNamesAndTypes. Some CR fixes
Add sanitizing for kafka_max_block_size and kafka_poll_max_batch_size
2020-06-19 10:49:05 +02:00
Ivan
c8808e89fd
Update test.py
2020-06-06 19:48:03 +03:00
Mikhail Filimonov
26d93fdfba
Fix the test, adjust default for queued.min.messages
2020-06-04 14:14:24 +02:00
Mikhail Filimonov
bf1e5db0ee
Tests from issue #11308 , from issue #4116
2020-06-03 19:01:45 +02:00
alesapin
c22e8f6b9b
Merge pull request #11283 from filimonov/kafka_virtual_column_headers
...
Virtual columns for Kafka headers
2020-06-01 12:06:00 +03:00
alesapin
364ebe9246
Merge pull request #11260 from filimonov/kafka_virtual_column_timestamp_ms
...
Add _timestamp_ms virtual columns to Kafka engine
2020-06-01 12:04:17 +03:00
alexey-milovidov
9a362eb02e
Merge pull request #11252 from filimonov/kafka-clientid
...
Kafka clientid (finishing #11073 )
2020-05-31 18:37:30 +03:00
Mikhail Filimonov
b885337454
Virtual columns for accessing kafka message headers
2020-05-29 10:44:10 +02:00
Mikhail Filimonov
03430b1763
Change the test, accept new librdkafka behaviour
2020-05-28 23:11:29 +02:00
Mikhail Filimonov
e39c4cdf71
Add _timestamp_ms virtual columns to Kafka engine
2020-05-28 19:41:57 +02:00
Mikhail Filimonov
6b694d54b4
Better kafka client.id
2020-05-28 15:39:55 +02:00
filimonov
c73b8372e7
Update test.py
2020-05-27 07:44:13 +02:00
Mikhail Filimonov
efc0b97707
Fix Kafka performance issue related to reschedules based on limits, which were always applied.
2020-05-27 07:39:06 +02:00
filimonov
73a0ef6c0e
Fix for the hang during deletion of engine=Kafka (one more time) ( #11145 )
...
* Added flag for safer rdkafka destruction, but more testing detected another hang case, which has no straigt solutions and can be workarounded currenly only by draining the consumer queue, so destructor is back
* After review fixes
* After review fixes2
* After review fixes3
2020-05-26 00:13:10 +03:00
Mikhail Filimonov
b50480cbde
Fixes the potential missed data during termination of Kafka engine table
2020-05-19 18:55:37 +02:00
Mikhail Filimonov
791f3a6c90
Merge branch 'kafka_drop_hang_fix' of github.com:filimonov/ClickHouse into kafka_drop_hang_fix
2020-05-15 22:20:45 +02:00
Mikhail Filimonov
405cce2c53
Another variant of fix. It looks like we do not need destructor at all
2020-05-15 22:19:21 +02:00
filimonov
f97c14f7be
bit better comment
2020-05-14 12:18:17 +02:00
Mikhail Filimonov
6499d702f9
After review fixes
2020-05-14 12:15:38 +02:00
Mikhail Filimonov
a1013f1e71
Fix for the hang during deletion of engine=Kafka
2020-05-14 10:38:02 +02:00
