Mikhail Filimonov
dc43728dea
Test showing the problem with partial commit of polled batch + fix
2020-01-28 15:21:36 +01:00
Ivan
7ba4e36e73
Don't push to MVs when inserting into Kafka table ( #7265 )
...
* Do not insert values into MV when inserting directly to Kafka
* Add method noPushingToViews() to IStorage interface
To separate InterpreterInsertQuery and StorageKafka
2019-10-22 13:31:28 +03:00
Ivan
cffc254922
Always read and insert Kafka messages as a whole ( #6950 )
...
* Always read and insert Kafka messages as a whole.
2019-09-20 15:12:32 +03:00
Ivan Lezhankin
c5f1352394
Decide to put delimiter only after polled message
2019-09-11 16:16:43 +03:00
Ivan
6c32fc3fc1
Store offsets manually for each message ( #6872 )
2019-09-09 19:59:51 +03:00
Ivan
ab7df6b6dd
Fix kafka tests ( #6805 )
...
* Commit offsets more precisely
* Get rid of DelimitedReadBuffer since read buffers don't line up well
* Increase timeouts
2019-09-05 00:25:33 +03:00
Ivan
5851316742
Don't interrupt copyData() used inside Kafka materialized views ( #6597 )
...
* Print better offsets on commit
* Stop consumption on the read buffer level
* Don't use cancellation in the middle of copyData()
* Add test
* Disable squashing stream for Kafka materialized views
* Commit last read message, since rdkafka auto-modifies offset
2019-08-29 18:36:07 +03:00
Ivan
1bf6e034e8
Fix infinite loop when reading Kafka messages ( #6354 )
...
* Do not pause/resume consumer at all
* Fix kafka tests
* Try to ensure the subscription
* Set timeout for kafka tests and return 'while True'
* Update cluster.py
* When doing a raw select from kafka, ignore client errors.
They may rise due to 'Local: Timed out' while subscribing.
2019-08-07 19:10:14 +03:00
alexey-milovidov
7219e167d2
Merge pull request #6026 from bopohaa/fix-kafka-unclean-stream
...
Clear Kafka's buffer if an invalid message is found.
2019-07-25 04:30:37 +03:00
Ivan Lezhankin
f9945494d9
Always resume consumer before subscription.
...
Also add more logs to see the difference between rd_kafka_assignment() vs rd_kafka_subscription()
2019-07-19 18:01:34 +03:00
Nik
922c3eb22e
Clear Kafka's buffer if an invalid message is found.
2019-07-16 19:27:42 +04:00
Ivan
fdae12a51c
Keep on reading of new Kafka messages after stalling ( #5852 )
2019-07-03 19:51:11 +03:00
Ivan Lezhankin
0917573493
Merge remote-tracking branch 'upstream/master' into issue-5286
2019-06-24 14:42:58 +03:00
Ivan Lezhankin
b75db2ef04
Fix dropping message to early when stalling
2019-06-21 17:29:10 +03:00
Ivan Lezhankin
61750d09d2
Commit offsets for SELECTing from Kafka table too
2019-06-20 15:34:25 +03:00
Ivan Lezhankin
fdd3f07860
Don't do premature subscription to topics
2019-06-19 19:15:30 +03:00
Ivan Lezhankin
d907d8e8ca
Commit offsets for SELECTing from Kafka table too
2019-06-17 19:27:18 +03:00
Ivan Lezhankin
8326021d74
Fix style
2019-06-03 17:36:59 +03:00
Ivan Lezhankin
13212c9b01
Fix the hang on dropping Kafka table when there is no mat. views
2019-05-27 20:25:34 +03:00
Ivan Lezhankin
a14013dff7
Add setting to regulate Kafka committing policy.
2019-05-16 18:20:30 +03:00
Ivan
1ea9e3019d
Freeze the Kafka buffer after first empty response ( #5283 )
...
* Check inside inferior streams for cancellation while reading.
* Stop reading from Kafka buffer after first empty read.
2019-05-15 19:11:50 +03:00
Ivan
4a5832b18a
Check the time limit every (flush_interval / poll_timeout) number of rows from Kafka ( #5249 )
2019-05-14 18:52:03 +03:00
Ivan Lezhankin
94bba1918f
Do not drop Kafka consumer buffers after closing stream
2019-04-22 16:23:05 +03:00
Ivan Lezhankin
681266abd0
Poll messages in batches of max_block_size size.
2019-01-25 15:48:59 +03:00
Ivan Lezhankin
862d419f31
Fix losing the next message after a broken one.
2019-01-24 14:58:25 +03:00
Ivan Lezhankin
c8e605327d
Split StorageKafka.cpp on smaller files per class
2019-01-24 14:58:24 +03:00
