Ivan
7ba4e36e73
Don't push to MVs when inserting into Kafka table ( #7265 )
...
* Do not insert values into MV when inserting directly to Kafka
* Add method noPushingToViews() to IStorage interface
To separate InterpreterInsertQuery and StorageKafka
2019-10-22 13:31:28 +03:00
Ivan
cffc254922
Always read and insert Kafka messages as a whole ( #6950 )
...
* Always read and insert Kafka messages as a whole.
2019-09-20 15:12:32 +03:00
Ivan
ab7df6b6dd
Fix kafka tests ( #6805 )
...
* Commit offsets more precisely
* Get rid of DelimitedReadBuffer since read buffers don't line up well
* Increase timeouts
2019-09-05 00:25:33 +03:00
Ivan
5851316742
Don't interrupt copyData() used inside Kafka materialized views ( #6597 )
...
* Print better offsets on commit
* Stop consumption on the read buffer level
* Don't use cancellation in the middle of copyData()
* Add test
* Disable squashing stream for Kafka materialized views
* Commit last read message, since rdkafka auto-modifies offset
2019-08-29 18:36:07 +03:00
Ivan
a502424c33
Implement support for insertion into Kafka tables ( #6012 )
...
* Add write-callback on each row for RowOutputStream
* Fix build of new rdkafka library
* Poll messages if Kafka outgoing queue is full
* Add test
* Add test producer-consumer
* Truncate delimiter from last row in message
2019-08-20 14:17:57 +03:00
Ivan Lezhankin
b55da1888a
Add "_timestamp" virtual column
2019-08-06 17:18:37 +03:00
Ivan Lezhankin
4bc2b32c3e
Merge branch 'master' into issue-5286
2019-08-06 16:25:51 +03:00
Ivan Lezhankin
969cb2f826
Append _partition virtual column
2019-07-22 14:32:11 +03:00
Ivan Lezhankin
f9945494d9
Always resume consumer before subscription.
...
Also add more logs to see the difference between rd_kafka_assignment() vs rd_kafka_subscription()
2019-07-19 18:01:34 +03:00
Ivan Lezhankin
0917573493
Merge remote-tracking branch 'upstream/master' into issue-5286
2019-06-24 14:42:58 +03:00
Ivan Lezhankin
fdd3f07860
Don't do premature subscription to topics
2019-06-19 19:15:30 +03:00
Ivan Lezhankin
13212c9b01
Fix the hang on dropping Kafka table when there is no mat. views
2019-05-27 20:25:34 +03:00
Ivan Lezhankin
800854119e
Add buffer callback to fill in virtual columns
2019-05-23 14:15:18 +03:00
Ivan Lezhankin
dd906eabdc
[WIP] refactoring
2019-05-22 22:38:43 +03:00
Ivan Lezhankin
a14013dff7
Add setting to regulate Kafka committing policy.
2019-05-16 18:20:30 +03:00
Ivan
1ea9e3019d
Freeze the Kafka buffer after first empty response ( #5283 )
...
* Check inside inferior streams for cancellation while reading.
* Stop reading from Kafka buffer after first empty read.
2019-05-15 19:11:50 +03:00
Ivan
4a5832b18a
Check the time limit every (flush_interval / poll_timeout) number of rows from Kafka ( #5249 )
2019-05-14 18:52:03 +03:00
Ivan Lezhankin
94bba1918f
Do not drop Kafka consumer buffers after closing stream
2019-04-22 16:23:05 +03:00
Ivan Lezhankin
681266abd0
Poll messages in batches of max_block_size size.
2019-01-25 15:48:59 +03:00
Ivan Lezhankin
862d419f31
Fix losing the next message after a broken one.
2019-01-24 14:58:25 +03:00
Ivan Lezhankin
c8e605327d
Split StorageKafka.cpp on smaller files per class
2019-01-24 14:58:24 +03:00
