Commit Graph

33 Commits

Author SHA1 Message Date
Alexander Tokmakov
cf5c998b30 refactor storage name 2019-12-26 19:23:38 +03:00
Mikhail Filimonov
f3fa71c707 Init query context for Kafka to make subqueries work 2019-12-13 02:21:40 +01:00
Ivan
7ba4e36e73
Don't push to MVs when inserting into Kafka table (#7265)
* Do not insert values into MV when inserting directly to Kafka
* Add method noPushingToViews() to IStorage interface
  To separate InterpreterInsertQuery and StorageKafka
2019-10-22 13:31:28 +03:00
alesapin
7382a9f3c3 Rename method for settings check 2019-09-01 00:15:40 +03:00
alesapin
6f30338c08 Fixes 2019-08-30 21:31:35 +03:00
alesapin
808f4d0b8a Intermediate refactoring 2019-08-30 19:50:59 +03:00
Alexey Milovidov
a1101e5278 Fixed build 2019-08-28 16:21:19 +03:00
Alexey Milovidov
c8798b2ffa Slightly improve table locks 2019-08-28 02:47:30 +03:00
alexey-milovidov
8db93d392b
Merge pull request #6685 from yandex/merge_tree_settings_alter
Alter merge tree settings with other types of alter
2019-08-27 16:07:28 +03:00
alesapin
36472b14f9 Defend kafka storage from alters 2019-08-27 12:48:20 +03:00
alesapin
7851d8fe0a Add ability to alter settings with other types of alter 2019-08-27 12:34:53 +03:00
alexey-milovidov
561b173ddc
Merge pull request #6675 from yandex/simplify-shared-ptr-helper
Simplify shared_ptr_helper
2019-08-27 00:36:52 +03:00
Alexey Milovidov
0eec226c9a Simplify shared_ptr_helper 2019-08-26 22:07:29 +03:00
alesapin
0d0ebf885f Remove two redundant arguments from alter 2019-08-26 17:50:34 +03:00
alesapin
70ca7f4156 Merge with master 2019-08-21 11:57:35 +03:00
Ivan
a502424c33
Implement support for insertion into Kafka tables (#6012)
* Add write-callback on each row for RowOutputStream
* Fix build of new rdkafka library
* Poll messages if Kafka outgoing queue is full
* Add test
* Add test producer-consumer
* Truncate delimiter from last row in message
2019-08-20 14:17:57 +03:00
alesapin
13e4581317 More general 2019-08-07 18:21:45 +03:00
Ivan Lezhankin
dd906eabdc [WIP] refactoring 2019-05-22 22:38:43 +03:00
Ivan Lezhankin
cf39c4cc47 Embed virtual columns into IStorage 2019-05-21 14:24:32 +03:00
Ivan Lezhankin
a14013dff7 Add setting to regulate Kafka committing policy. 2019-05-16 18:20:30 +03:00
Ivan Lezhankin
94bba1918f Do not drop Kafka consumer buffers after closing stream 2019-04-22 16:23:05 +03:00
Ivan
957d23267b
Minimal implementation of row-level security CLICKHOUSE-4315 (#4792)
For detailed description see the related PR
2019-03-29 23:31:06 +03:00
Vitaly Baranov
309df832ec IStorage::read() function, "max_block_size" parameter: change type UInt64 -> size_t. 2019-02-19 14:56:08 +03:00
proller
4078a62573 Fix build without contrib/cppkafka submodule 2019-02-11 19:34:32 +03:00
Alexey Milovidov
58a6accee5 Updated test #4246 2019-02-10 19:55:12 +03:00
Ivan Lezhankin
89ffb577b1 Implement additional Kafka engine setting for allowed parsing errors 2019-01-18 15:48:38 +03:00
Ivan Lezhankin
337c092c7e Use cppkafka instead of raw C interface 2019-01-14 14:15:57 +03:00
Ivan Lezhankin
9c35598373 Add cppkafka to contrib 2019-01-14 14:15:57 +03:00
Alexey Milovidov
1d4701b0bc Added some warnings from clang's -Weverything 2019-01-04 15:10:00 +03:00
Marek Vavruša
37e03bd938 StorageKafka: check dependencies recursively, add max_block setting
Changes:

* Check table dependencies recursively before starting streaming
* Use actual number of created consumers when streaming
* Add size limits hint to Kafka consumer stream
* Configurable `kafka_max_block_size` per table
2018-10-18 09:20:34 -07:00
Marek Vavruša
ea32434888 Storages/Kafka: fixed deadlocks, moved to background pool
This fixes two deadlocks in Kafka engine found previously:
* When exception is thrown before starting reading, consumer
  was never returned to the storage. Now it is claimed only
  when actually starting reading.
* Fixed lockup on deinitialization when consumer only unsubscribed,
  but didn't close, and the endine then timeouted when waiting for
  consumer destruction.

This also moves the stream thread to background worker pool.
The reason for that is that it will compete with other tasks for
time, so it will form a backpressure on insertion when the system
is busy.
2018-09-25 22:51:07 -07:00
Alexey Milovidov
2016374d20 Merging #2226 2018-08-05 10:05:43 +03:00
Alexander Marshalov
154d7e3228 Added SETTINGS clause for Kafka storage engine 2018-08-03 21:51:28 +03:00