Commit Graph

13 Commits

Author SHA1 Message Date
Ivan Lezhankin
94bba1918f Do not drop Kafka consumer buffers after closing stream 2019-04-22 16:23:05 +03:00
Ivan
957d23267b
Minimal implementation of row-level security CLICKHOUSE-4315 (#4792)
For detailed description see the related PR
2019-03-29 23:31:06 +03:00
Vitaly Baranov
309df832ec IStorage::read() function, "max_block_size" parameter: change type UInt64 -> size_t. 2019-02-19 14:56:08 +03:00
proller
4078a62573 Fix build without contrib/cppkafka submodule 2019-02-11 19:34:32 +03:00
Alexey Milovidov
58a6accee5 Updated test #4246 2019-02-10 19:55:12 +03:00
Ivan Lezhankin
89ffb577b1 Implement additional Kafka engine setting for allowed parsing errors 2019-01-18 15:48:38 +03:00
Ivan Lezhankin
337c092c7e Use cppkafka instead of raw C interface 2019-01-14 14:15:57 +03:00
Ivan Lezhankin
9c35598373 Add cppkafka to contrib 2019-01-14 14:15:57 +03:00
Alexey Milovidov
1d4701b0bc Added some warnings from clang's -Weverything 2019-01-04 15:10:00 +03:00
Marek Vavruša
37e03bd938 StorageKafka: check dependencies recursively, add max_block setting
Changes:

* Check table dependencies recursively before starting streaming
* Use actual number of created consumers when streaming
* Add size limits hint to Kafka consumer stream
* Configurable `kafka_max_block_size` per table
2018-10-18 09:20:34 -07:00
Marek Vavruša
ea32434888 Storages/Kafka: fixed deadlocks, moved to background pool
This fixes two deadlocks in Kafka engine found previously:
* When exception is thrown before starting reading, consumer
  was never returned to the storage. Now it is claimed only
  when actually starting reading.
* Fixed lockup on deinitialization when consumer only unsubscribed,
  but didn't close, and the endine then timeouted when waiting for
  consumer destruction.

This also moves the stream thread to background worker pool.
The reason for that is that it will compete with other tasks for
time, so it will form a backpressure on insertion when the system
is busy.
2018-09-25 22:51:07 -07:00
Alexey Milovidov
2016374d20 Merging #2226 2018-08-05 10:05:43 +03:00
Alexander Marshalov
154d7e3228 Added SETTINGS clause for Kafka storage engine 2018-08-03 21:51:28 +03:00