thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-24 00:22:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
126,732 Commits 9,167 Branches 1,781 Tags 2.1 GiB
85e3c31f14
Commit Graph

200 Commits

Author SHA1 Message Date
Azat Khuzhin
f1c726a675 tests/integration/test_storage_kafka: cover Template format 2021-12-31 11:25:08 +03:00
Azat Khuzhin
52ad7a4538 Fix test_storage_kafka failures by adjusting retention.ms 
This test uses predefined timestamps, and default retention.ms is too
small:

    kafka1_1           | [2021-12-28 21:40:21,842] INFO Created log for partition virt2_0-0 in /var/lib/kafka/data with properties {compression.type -> producer, message.format.version -> 2.2-IV1, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, message.downconversion.enable -> true, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)

    kafka1_1           | [2021-12-28 21:40:24,540] INFO [Log partition=virt2_0-0, dir=/var/lib/kafka/data] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log)

v2: fix tests with 0 timestamp_ms
    and this should be all tests that was left:

    $ fgrep 'Found deletable segments with base offsets' test_storage_kafka/_instances_0/docker.log
    kafka1_1           | [2021-12-29 09:46:15,610] INFO [Log partition=avro1-0, dir=/var/lib/kafka/data] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log)
    kafka1_1           | [2021-12-29 09:51:15,609] INFO [Log partition=virt1-0, dir=/var/lib/kafka/data] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log)
 2021-12-30 10:25:10 +03:00
Vitaly Baranov
b5b195f4e2
Merge branch 'master' into kafka_protobuf_issue26643 2021-12-10 23:22:35 +03:00
kssenii
2ea4e7c4e7 Better 2021-11-29 15:05:02 +00:00
kssenii
2f0fe4a21f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into rabbit-kafka-select-commit 2021-11-29 10:08:14 +00:00
Kruglov Pavel
b39c19399f
Merge pull request #31691 from kssenii/testing-1 
Allow named_collections for kafka / rabbit
 2021-11-25 14:06:31 +03:00
kssenii
980c87c466 Add named collection for kafka / rabbit 2021-11-24 14:53:01 +03:00
kssenii
35a4be601a Select query commit only with settings 2021-11-03 15:43:58 +03:00
Yatsishin Ilya
d5138e1427 Merge remote-tracking branch 'origin' into integration--7 2021-11-02 17:41:40 +03:00
Kruglov Pavel
901ebcede6
Merge pull request #30351 from arenadata/ADQM-335 
output_format_avro_rows_in_file
 2021-11-02 12:25:27 +03:00
Ilya Golshtein
8efa1743cf output_format_avro_rows_in_file fix and test for _timestamp 2021-11-01 13:09:18 +03:00
avogar
0ce893cb10 Fix kafka test 2021-10-28 14:20:29 +03:00
Kruglov Pavel
11c0d08d1c
Fix test again 2021-10-21 16:46:27 +03:00
Kruglov Pavel
bd016f384e
Try fix tests 2021-10-21 12:54:10 +03:00
avogar
cd17643eb8 Fix tests 2021-10-21 09:22:42 +03:00
Ilya Golshtein
d90302aa3b output_format_avro_rows_in_file 2021-10-18 19:01:06 +03:00
Yatsishin Ilya
0efeaf5dc4 Merge remote-tracking branch 'origin' into integration--7 2021-10-08 11:22:55 +03:00
Nikolai Kochetov
78e1db209f
Remove more data streams (#29491) 
* Remove more streams.

* Fixing build.

* Fixing build.

* Rename files.

* Fix fast test.

* Fix StorageKafka.

* Try fix kafka test.

* Move createBuffer to KafkaSource ctor.

* Revert "Move createBuffer to KafkaSource ctor."

This reverts commit 81fa94d27e.

* Revert "Try fix kafka test."

This reverts commit 2107e54969.

* Comment some rows in test.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-10-07 11:26:08 +03:00
Yatsishin Ilya
c32e833f25 better 2021-10-06 16:23:29 +03:00
Yatsishin Ilya
b0a5a8532f improve test_storage_kafka test 2021-10-06 16:08:25 +03:00
Mikhail Filimonov
ea320c96d5 Test for issue #26643 2021-10-04 14:38:10 +00:00
alexey-milovidov
995913e89c
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-28 02:27:36 +03:00
Nikolai Kochetov
193f566568 Fix exception message. 2021-09-24 14:54:39 +03:00
Nikolai Kochetov
24dac111b7 Try fix kafka test. 2021-09-24 13:30:57 +03:00
Alexey Milovidov
e513a5db32 Change <yandex> to <clickhouse> in configs 2021-09-20 01:38:53 +03:00
alesapin
36abf9f2f4 Fix broken kafka test 2021-09-03 10:51:37 +03:00
Yatsishin Ilya
da4e688293 Improve kafka topic creation and deletion. Improve mongodb tests 2021-08-30 15:58:21 +03:00
Yatsishin Ilya
d9e782d379 Merge remote-tracking branch 'origin' into mute-integration-debug-fails 2021-08-20 18:59:16 +03:00
Yatsishin Ilya
812a6ffb80 kafka assertion was fixed in master 2021-08-20 18:56:15 +03:00
Yatsishin Ilya
9c22d07dea Mute current failures 2021-08-19 14:32:32 +03:00
Mikhail Filimonov
05d77d2873
Safer processing of NULL messages in Kafka for some formats 2021-08-18 11:26:57 +02:00
Kseniia Sumarokova
0917488cec
Merge pull request #27382 from kssenii/cast-internal 
add Cast internal function
 2021-08-13 13:13:51 +03:00
Raúl Marín
d1d47658f5 Merge remote-tracking branch 'blessed/master' into materialization_log 2021-08-10 10:49:05 +02:00
kssenii
484c3a5d2e Update kafka test 2021-08-09 11:55:38 +00:00
Raúl Marín
b4e417e507 test_kafka_virtual_columns_with_materialized_view: Limit iterations and show better error 2021-08-06 13:39:33 +02:00
Raúl Marín
4f1b02426a test_storage_kafka: Remove duplicated code 2021-08-06 13:38:56 +02:00
Raúl Marín
145699d0c2 test_kafka_no_holes_when_write_suffix_failed: Fix expected error message 
The old error message was incorrect (it said a failure happened when
pushing the prefix when it was happening while pushing the suffix.
 2021-07-27 16:29:34 +02:00
Raúl Marín
ff249a14af Fix test_kafka_no_holes_when_write_suffix_failed 2021-07-27 16:29:34 +02:00
vdimir
dd3e1a8a4d
Adjust exception message in test_replicated_fetches_timeouts 2021-07-15 11:58:21 +03:00
Vitaly Baranov
3a3cbeb7aa Collect stderr.log and stdout.log in all integration tests by default. 2021-06-29 17:30:47 +03:00
alexey-milovidov
77721734ed
Merge pull request #21267 from filimonov/kafka-failover 
Kafka failover issue fix
 2021-06-14 01:42:42 +03:00
alexey-milovidov
77ab26e217
Merge branch 'master' into less_timeouts 2021-06-10 02:03:16 +03:00
filimonov
3f1d3f2e1d
Update test.py 2021-06-09 23:20:44 +02:00
Mikhail Filimonov
3627ab7cbe
Fix kafka failover issue (#21118). 
Distringuish lack of assignment vs lack of partitions. Bit better / clearer logging.
 2021-06-09 16:02:56 +02:00
filimonov
7469f054f0
Update test.py 
Missed one of the refactorings which were merged in the meanwhile 872e36c207 (diff-8129a3b04ff2293c2d89b35395be9ef49918cdafd4420a745d05733b00979646L86-R100)
 2021-06-09 12:27:42 +02:00
alesapin
7179fafef0 Less timeouts in integration tests 2021-06-09 10:49:29 +03:00
mergify[bot]
1172807442
Merge branch 'master' into recreate_kafka_table_case 2021-06-06 12:15:55 +00:00
Mikhail Filimonov
c8fe168a8d
Fix the test after #20393 2021-06-04 15:55:07 +02:00
filimonov
74ae079b19
Update test.py 2021-06-04 15:30:25 +02:00
Mikhail Filimonov
e11f7db28b
Add test #23430 2021-06-03 20:41:01 +02:00
Yatsishin Ilya
28af06f3d0 Merge remote-tracking branch 'origin' into integration-2 2021-05-27 07:01:22 +03:00
Yatsishin Ilya
893cd47bd2 better 2021-05-25 15:40:59 +03:00
filimonov
88919080d5
Try to improve kafka flaky test 2021-05-24 22:37:16 +02:00
Yatsishin Ilya
e1176acc04 Merge remote-tracking branch 'origin' into integration-2 2021-05-17 09:53:00 +03:00
Mikhail Filimonov
dd619e1e37
Try to stabilize flapping test 2021-05-13 16:03:32 +02:00
Yatsishin Ilya
9d159072ef mERGe remote-tracking branch 'origin' into integration-2 2021-05-11 13:32:44 +03:00
Chao Ma
ec8ca5bf5e Fix kafka storage does not work with parquet format messages 2021-05-06 11:32:58 +08:00
Mikhail Filimonov
80065b08f8
Revert "avoid race in librdkafka" 
This reverts commit bd8c3eb117.
 2021-05-04 10:06:33 +02:00
Yatsishin Ilya
c59a11f798 Merge remote-tracking branch 'origin' into integration-2 2021-04-27 18:34:33 +03:00
Chao Ma
713eb9486c Fix kafka storage does not work with arrow and arrowstream format messages 2021-04-21 19:44:09 +08:00
Yatsishin Ilya
f6d455732e better 2021-04-14 19:04:17 +03:00
Yatsishin Ilya
db657c03c8 fix 2021-04-12 18:08:09 +03:00
Yatsishin Ilya
ec6adb692d Merge remote-tracking branch 'origin' into integration-2 2021-04-12 10:04:11 +03:00
Peng Jian
f73faf1578 fix test case 2021-04-08 14:26:32 +08:00
Peng Jian
16e971ee52 fix test case 2021-04-08 07:19:57 +08:00
Peng Jian
604ba9db60 add test case 2021-04-07 20:15:15 +08:00
Peng Jian
e30c07db20 add test cases 2021-03-31 20:22:36 +08:00
Peng Jian
cc56302e8b fix test case 2021-03-31 17:16:51 +08:00
Peng Jian
17a5fc6e0e add test 2021-03-31 17:16:51 +08:00
Yatsishin Ilya
7b7580204c fix 2021-03-28 00:26:53 +03:00
Yatsishin Ilya
fbf0fe6436 Merge remote-tracking branch 'origin' into integration-2 2021-03-20 22:47:33 +03:00
Vladimir
c8b5be636f
Merge pull request #21438 from arenadata/ADQM-170 2021-03-19 17:06:55 +03:00
Ilya Golshtein
518103f851 old Avro test data removed 2021-03-16 19:39:58 +03:00
Ilya Golshtein
abcf581005 avro bug with format_setting reference and a typo in kafka test 2021-03-15 12:19:33 +03:00
Ilya Golshtein
a579fcb424 kafka/avro works against table 2021-03-14 15:55:44 +03:00
Yatsishin Ilya
7bfa4bb64b Merge remote-tracking branch 'origin' into integration-2 2021-03-09 10:33:15 +03:00
filimonov
bd8c3eb117
avoid race in librdkafka 
see https://github.com/edenhill/librdkafka/issues/3279
 2021-03-04 15:01:10 +01:00
Yatsishin Ilya
298824c926 fix 2021-03-04 14:52:56 +03:00
Yatsishin Ilya
f03cf47a82 fix 2021-03-04 14:22:19 +03:00
Yatsishin Ilya
8122003c98 more fixes 2021-03-04 13:53:50 +03:00
Yatsishin Ilya
34609445f7 fix 2021-03-04 12:06:39 +03:00
Ilya Golshtein
f11a8cd13c multimessage avro format for kafka test 2021-03-04 11:07:29 +03:00
Yatsishin Ilya
babea212c7 Merge remote-tracking branch 'origin' into integration-2 2021-03-03 11:56:15 +03:00
Mikhail Filimonov
9e3547681b
try to address python kafka client disconnects better 2021-02-26 07:17:31 +01:00
Mikhail Filimonov
60ede36068
simple fix for CI failure 2021-02-26 06:41:11 +01:00
Mikhail Filimonov
c4470ff283
proper placement for test 2021-02-26 06:38:14 +01:00
Mikhail Filimonov
2bcfff257c
Remove most of sleeps in tests 2021-02-25 17:02:59 +01:00
Mikhail Filimonov
6c6eaf2a60
Fixes 2021-02-25 17:02:57 +01:00
Mikhail Filimonov
a4c9e62d6d
Integration tests: introduce wait_for_log_line. 
It uses tail -f executed in container, this way i don't need to pull
the file many times, rememeber positions etc.
 2021-02-25 17:02:56 +01:00
Yatsishin Ilya
905cc7a2bd fixes 2021-02-24 14:46:58 +03:00
Yatsishin Ilya
2df4317aba wip 2021-02-19 00:21:50 +03:00
Yatsishin Ilya
609c9b272f fix 2021-02-18 16:42:31 +03:00
Yatsishin Ilya
872e36c207 wip 2021-02-18 15:57:45 +03:00
Yatsishin Ilya
70963e4396 Fixed test_kafka_flush_by_time test. 2021-01-28 18:03:23 +03:00
Nikita Mikhaylov
736cc6959a
remove kafka flaky fix 2020-12-30 17:24:39 +03:00
filimonov
e81a99156f
Attempt to prevent test flap 
That extra SELECT is executed synchronously, after that all further actions will have a more stable timing.
 2020-12-29 18:51:00 +01:00
Nikita Mikhailov
fbf2ac35e8 fix tsan 2020-12-17 18:14:09 +03:00
Azat Khuzhin
2e569899d1 Add a test for kafka with snappy compression method (regression for UB in snappy) 
I've checked it manually and it works with clang build and does not with
gcc (there are NULL bytes in the middle of value).
 2020-12-14 00:38:07 +03:00
filimonov
8e2ca9518b
Update test.py 
adjusting timeouts a bit, in the good hope that it will prevent flakiness of the test.
 2020-12-11 10:13:03 +01:00
Mikhail Filimonov
b072bc7d2b
Test for the issue #12615 2020-11-06 17:54:44 +01:00