Commit Graph

19 Commits

Author SHA1 Message Date
alesapin
70ca7f4156 Merge with master 2019-08-21 11:57:35 +03:00
Ivan
a502424c33
Implement support for insertion into Kafka tables (#6012)
* Add write-callback on each row for RowOutputStream
* Fix build of new rdkafka library
* Poll messages if Kafka outgoing queue is full
* Add test
* Add test producer-consumer
* Truncate delimiter from last row in message
2019-08-20 14:17:57 +03:00
alesapin
13e4581317 More general 2019-08-07 18:21:45 +03:00
Ivan Lezhankin
dd906eabdc [WIP] refactoring 2019-05-22 22:38:43 +03:00
Ivan Lezhankin
cf39c4cc47 Embed virtual columns into IStorage 2019-05-21 14:24:32 +03:00
Ivan Lezhankin
a14013dff7 Add setting to regulate Kafka committing policy. 2019-05-16 18:20:30 +03:00
Ivan Lezhankin
94bba1918f Do not drop Kafka consumer buffers after closing stream 2019-04-22 16:23:05 +03:00
Ivan
957d23267b
Minimal implementation of row-level security CLICKHOUSE-4315 (#4792)
For detailed description see the related PR
2019-03-29 23:31:06 +03:00
Vitaly Baranov
309df832ec IStorage::read() function, "max_block_size" parameter: change type UInt64 -> size_t. 2019-02-19 14:56:08 +03:00
proller
4078a62573 Fix build without contrib/cppkafka submodule 2019-02-11 19:34:32 +03:00
Alexey Milovidov
58a6accee5 Updated test #4246 2019-02-10 19:55:12 +03:00
Ivan Lezhankin
89ffb577b1 Implement additional Kafka engine setting for allowed parsing errors 2019-01-18 15:48:38 +03:00
Ivan Lezhankin
337c092c7e Use cppkafka instead of raw C interface 2019-01-14 14:15:57 +03:00
Ivan Lezhankin
9c35598373 Add cppkafka to contrib 2019-01-14 14:15:57 +03:00
Alexey Milovidov
1d4701b0bc Added some warnings from clang's -Weverything 2019-01-04 15:10:00 +03:00
Marek Vavruša
37e03bd938 StorageKafka: check dependencies recursively, add max_block setting
Changes:

* Check table dependencies recursively before starting streaming
* Use actual number of created consumers when streaming
* Add size limits hint to Kafka consumer stream
* Configurable `kafka_max_block_size` per table
2018-10-18 09:20:34 -07:00
Marek Vavruša
ea32434888 Storages/Kafka: fixed deadlocks, moved to background pool
This fixes two deadlocks in Kafka engine found previously:
* When exception is thrown before starting reading, consumer
  was never returned to the storage. Now it is claimed only
  when actually starting reading.
* Fixed lockup on deinitialization when consumer only unsubscribed,
  but didn't close, and the endine then timeouted when waiting for
  consumer destruction.

This also moves the stream thread to background worker pool.
The reason for that is that it will compete with other tasks for
time, so it will form a backpressure on insertion when the system
is busy.
2018-09-25 22:51:07 -07:00
Alexey Milovidov
2016374d20 Merging #2226 2018-08-05 10:05:43 +03:00
Alexander Marshalov
154d7e3228 Added SETTINGS clause for Kafka storage engine 2018-08-03 21:51:28 +03:00