Mikhail Filimonov
6e426592a7
Make the formats like RowBinaryWithNamesAndTypes work in Kafka & do not try to parse buffer when it at eof
2019-12-03 22:03:22 +01:00
alexey-milovidov
b6edda194f
Update KafkaBlockInputStream.cpp
2019-12-03 03:52:02 +03:00
Mikhail Filimonov
ad136ec62d
get rid of cloneWithColumns in internal loop and some other trash
2019-11-29 11:46:25 +01:00
Mikhail Filimonov
4cc0d0ca5d
make virtual_columns local
2019-11-29 07:21:31 +01:00
Mikhail Filimonov
07722dfa93
more progress - cleanup internal cycles
2019-11-29 00:06:03 +01:00
Mikhail Filimonov
6217cb51f6
Get rid of InputStream layer for Kafka, use InputStream directly. Reopening port during reset.
2019-11-28 17:43:06 +01:00
Mikhail Filimonov
deec48a3e0
attempt to fix kafka parsing performance regression https://github.com/ClickHouse/ClickHouse/issues/7261
2019-11-27 00:46:19 +01:00
Ivan
21db9157f7
Don't initialize some constant blocks in a loop ( #7475 )
2019-11-01 14:34:29 +03:00
Ivan
05b17976d1
Commit to Kafka explicitly after the writing is finalized ( #7175 )
2019-10-03 00:17:19 +03:00
Ivan
cffc254922
Always read and insert Kafka messages as a whole ( #6950 )
...
* Always read and insert Kafka messages as a whole.
2019-09-20 15:12:32 +03:00
Ivan
ab7df6b6dd
Fix kafka tests ( #6805 )
...
* Commit offsets more precisely
* Get rid of DelimitedReadBuffer since read buffers don't line up well
* Increase timeouts
2019-09-05 00:25:33 +03:00
Alexey Milovidov
d1c268c2ed
Added setting "kafka_max_wait_ms"
2019-08-28 18:24:23 +03:00
Ivan
a502424c33
Implement support for insertion into Kafka tables ( #6012 )
...
* Add write-callback on each row for RowOutputStream
* Fix build of new rdkafka library
* Poll messages if Kafka outgoing queue is full
* Add test
* Add test producer-consumer
* Truncate delimiter from last row in message
2019-08-20 14:17:57 +03:00
Ivan Lezhankin
fd27ac9e9e
Merge remote-tracking branch 'upstream/master' into issue-5286
2019-08-11 00:11:44 +03:00
kreuzerkrieg
112fc71276
adding -Wshadow for GCC
2019-08-09 23:58:16 +03:00
Ivan Lezhankin
b55da1888a
Add "_timestamp" virtual column
2019-08-06 17:18:37 +03:00
Ivan Lezhankin
4bc2b32c3e
Merge branch 'master' into issue-5286
2019-08-06 16:25:51 +03:00
Ivan Lezhankin
969cb2f826
Append _partition virtual column
2019-07-22 14:32:11 +03:00
Nik
4a30eba32f
Fix style check
2019-07-16 22:34:47 +04:00
Nik
922c3eb22e
Clear Kafka's buffer if an invalid message is found.
2019-07-16 19:27:42 +04:00
Ivan Lezhankin
436052c67b
Merge branch 'master' into issue-5286
2019-06-18 16:03:29 +03:00
proller
fc7ae399c4
Use system rapidjson; Fix types for macos;
2019-06-14 13:28:30 +03:00
Ivan Lezhankin
1eccbc39c5
Don't add virtual column to empty block
2019-05-28 00:01:24 +03:00
Ivan Lezhankin
19a850ad75
Use read callback to populate virtual columns in Kafka Engine
2019-05-23 16:20:25 +03:00
Ivan Lezhankin
800854119e
Add buffer callback to fill in virtual columns
2019-05-23 14:15:18 +03:00
Ivan Lezhankin
dd906eabdc
[WIP] refactoring
2019-05-22 22:38:43 +03:00
Ivan
4a5832b18a
Check the time limit every (flush_interval / poll_timeout) number of rows from Kafka ( #5249 )
2019-05-14 18:52:03 +03:00
Ivan Lezhankin
94bba1918f
Do not drop Kafka consumer buffers after closing stream
2019-04-22 16:23:05 +03:00
proller
8227f5fe63
Arcadia fix
2019-03-12 15:06:17 +03:00
Ivan Lezhankin
15ad409875
Tests are now passing locally
2019-02-11 14:54:30 +03:00
Ivan Lezhankin
39e2563f93
Create new consumers on-demand
...
and give them some time for graceful assignment.
2019-01-30 20:58:49 +03:00
Ivan Lezhankin
681266abd0
Poll messages in batches of max_block_size size.
2019-01-25 15:48:59 +03:00
Ivan Lezhankin
9636bdcd67
Fix build
2019-01-24 15:44:58 +03:00
Ivan Lezhankin
862d419f31
Fix losing the next message after a broken one.
2019-01-24 14:58:25 +03:00
Ivan Lezhankin
c8e605327d
Split StorageKafka.cpp on smaller files per class
2019-01-24 14:58:24 +03:00
