thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 03:42:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
36,034 Commits 9,315 Branches 1,783 Tags 2.1 GiB
8f90c6ec12
Commit Graph

176 Commits

Author SHA1 Message Date
Ivan Lezhankin
c5f1352394 Decide to put delimiter only after polled message 2019-09-11 16:16:43 +03:00
Ivan
6c32fc3fc1
Store offsets manually for each message (#6872) 2019-09-09 19:59:51 +03:00
Ivan
ab7df6b6dd
Fix kafka tests (#6805) 
* Commit offsets more precisely
* Get rid of DelimitedReadBuffer since read buffers don't line up well
* Increase timeouts
 2019-09-05 00:25:33 +03:00
alesapin
7382a9f3c3 Rename method for settings check 2019-09-01 00:15:40 +03:00
alesapin
2647d4ca15 Rename immutable to readonly 2019-08-30 23:12:26 +03:00
alesapin
6f30338c08 Fixes 2019-08-30 21:31:35 +03:00
alesapin
808f4d0b8a Intermediate refactoring 2019-08-30 19:50:59 +03:00
Ivan
5851316742
Don't interrupt copyData() used inside Kafka materialized views (#6597) 
* Print better offsets on commit
* Stop consumption on the read buffer level
* Don't use cancellation in the middle of copyData()
* Add test
* Disable squashing stream for Kafka materialized views
* Commit last read message, since rdkafka auto-modifies offset
 2019-08-29 18:36:07 +03:00
alexey-milovidov
44dd3bd792
Merge pull request #6692 from yandex/remove-queue_max_wait_ms 
Set default value of "queue_max_wait_ms" to zero, because current value makes no sense
 2019-08-28 21:06:09 +03:00
Alexey Milovidov
d1c268c2ed Added setting "kafka_max_wait_ms" 2019-08-28 18:24:23 +03:00
Alexey Milovidov
a1101e5278 Fixed build 2019-08-28 16:21:19 +03:00
Alexey Milovidov
c8798b2ffa Slightly improve table locks 2019-08-28 02:47:30 +03:00
alesapin
106b9717cd Refactoring of immutable settings 2019-08-27 16:14:19 +03:00
alexey-milovidov
8db93d392b
Merge pull request #6685 from yandex/merge_tree_settings_alter 
Alter merge tree settings with other types of alter
 2019-08-27 16:07:28 +03:00
alesapin
36472b14f9 Defend kafka storage from alters 2019-08-27 12:48:20 +03:00
alesapin
7851d8fe0a Add ability to alter settings with other types of alter 2019-08-27 12:34:53 +03:00
alexey-milovidov
561b173ddc
Merge pull request #6675 from yandex/simplify-shared-ptr-helper 
Simplify shared_ptr_helper
 2019-08-27 00:36:52 +03:00
Alexey Milovidov
0eec226c9a Simplify shared_ptr_helper 2019-08-26 22:07:29 +03:00
alesapin
cf31187254 Resolve conflicts 2019-08-26 21:08:58 +03:00
alesapin
0d0ebf885f Remove two redundant arguments from alter 2019-08-26 17:50:34 +03:00
Alexey Milovidov
60d660e263 Merge branch 'master' into merge-constraints 2019-08-26 16:13:44 +03:00
Alexey Milovidov
3fef8e92ad Merging CONSTRAINTS support 2019-08-25 00:20:20 +03:00
alesapin
70ca7f4156 Merge with master 2019-08-21 11:57:35 +03:00
Ivan
a502424c33
Implement support for insertion into Kafka tables (#6012) 
* Add write-callback on each row for RowOutputStream
* Fix build of new rdkafka library
* Poll messages if Kafka outgoing queue is full
* Add test
* Add test producer-consumer
* Truncate delimiter from last row in message
 2019-08-20 14:17:57 +03:00
alesapin
7b1039568c Revert old changed 2019-08-13 14:24:18 +03:00
alesapin
428c753ed7 Merge with master 2019-08-13 11:37:47 +03:00
Ivan Lezhankin
fd27ac9e9e Merge remote-tracking branch 'upstream/master' into issue-5286 2019-08-11 00:11:44 +03:00
kreuzerkrieg
112fc71276 adding -Wshadow for GCC 2019-08-09 23:58:16 +03:00
alesapin
2803fcc2ba Make atomic settings 2019-08-09 16:07:19 +03:00
alesapin
e0d18c0fe8 Merge branch 'master' into merge_tree_settings_alter 2019-08-08 18:16:22 +03:00
Ivan Lezhankin
e90bf74ce8 Merge branch 'master' into issue-5286 2019-08-08 12:57:00 +03:00
Ivan
1bf6e034e8
Fix infinite loop when reading Kafka messages (#6354) 
* Do not pause/resume consumer at all
* Fix kafka tests
* Try to ensure the subscription
* Set timeout for kafka tests and return 'while True'
* Update cluster.py
* When doing a raw select from kafka, ignore client errors.
  They may rise due to 'Local: Timed out' while subscribing.
 2019-08-07 19:10:14 +03:00
alesapin
7409f1a3de More comments 2019-08-07 18:33:10 +03:00
alesapin
13e4581317 More general 2019-08-07 18:21:45 +03:00
Ivan Lezhankin
b55da1888a Add "_timestamp" virtual column 2019-08-06 17:18:37 +03:00
alesapin
ca29343f54 Uniq settings 2019-08-06 17:09:36 +03:00
Ivan Lezhankin
4bc2b32c3e Merge branch 'master' into issue-5286 2019-08-06 16:25:51 +03:00
alesapin
86831fe500 Add ability to create immutable settings 2019-07-29 14:03:50 +03:00
alexey-milovidov
7219e167d2
Merge pull request #6026 from bopohaa/fix-kafka-unclean-stream 
Clear Kafka's buffer if an invalid message is found.
 2019-07-25 04:30:37 +03:00
Ivan Lezhankin
969cb2f826 Append _partition virtual column 2019-07-22 14:32:11 +03:00
Ivan Lezhankin
f9945494d9 Always resume consumer before subscription. 
Also add more logs to see the difference between rd_kafka_assignment() vs rd_kafka_subscription()
 2019-07-19 18:01:34 +03:00
Nik
4a30eba32f Fix style check 2019-07-16 22:34:47 +04:00
Nik
922c3eb22e Clear Kafka's buffer if an invalid message is found. 2019-07-16 19:27:42 +04:00
alexey-milovidov
a1cfdf2a60
Merge pull request #5855 from abyss7/no-undefined 
Use --no-undefined linker option for split build
 2019-07-07 01:47:11 +03:00
Ivan Lezhankin
3b995bdb02 Add missing dependecies on sight to enable --no-undefined option 2019-07-04 17:05:55 +03:00
Ivan
fdae12a51c
Keep on reading of new Kafka messages after stalling (#5852) 2019-07-03 19:51:11 +03:00
Ivan Lezhankin
0917573493 Merge remote-tracking branch 'upstream/master' into issue-5286 2019-06-24 14:42:58 +03:00
Ivan Lezhankin
b75db2ef04 Fix dropping message to early when stalling 2019-06-21 17:29:10 +03:00
Ivan Lezhankin
61750d09d2 Commit offsets for SELECTing from Kafka table too 2019-06-20 15:34:25 +03:00
Ivan Lezhankin
fdd3f07860 Don't do premature subscription to topics 2019-06-19 19:15:30 +03:00
Ivan Lezhankin
9fd048cdbd Allow to select virtual columns in materialized view 2019-06-18 19:32:37 +03:00
Ivan Lezhankin
436052c67b Merge branch 'master' into issue-5286 2019-06-18 16:03:29 +03:00
Ivan Lezhankin
d907d8e8ca Commit offsets for SELECTing from Kafka table too 2019-06-17 19:27:18 +03:00
proller
fc7ae399c4 Use system rapidjson; Fix types for macos; 2019-06-14 13:28:30 +03:00
Alexey Milovidov
c846d0f6af Namespace comments are unneeded according to the code style. 
find dbms -name '*.h' -or -name '*.cpp' | xargs grep -l -P '}\s*//+\s*namespace\s*' | xargs sed -i -r -e 's/}\s*\/\/+\s*namespace\s*.*$/}/'
 2019-06-13 13:43:37 +03:00
Alexey Milovidov
8b87511e25 Namespace comments are unneeded according to the code style. 
find dbms -name '*.h' -or -name '*.cpp' | xargs grep -l -P '}\s*//+\s*namespace\s*' | xargs sed -i -r -e 's/}\s*\/\/+\s*namespace\s*\w+/}/'
 2019-06-13 13:37:13 +03:00
Ivan Lezhankin
97558c1b41 Merge remote-tracking branch 'upstream/master' into issue-5286 2019-06-10 14:15:14 +03:00
Ivan
f70eed78d8
Don't do intermediate commits in Kafka tables by default (#5445) 
* Don't do intermediate commits in Kafka tables by default
* Use default settings' values as defaults in code
 2019-06-03 19:40:22 +03:00
Ivan Lezhankin
8326021d74 Fix style 2019-06-03 17:36:59 +03:00
Ivan Lezhankin
b311984879 Merge remote-tracking branch 'upstream/master' into issue-5286 2019-06-03 17:19:09 +03:00
Ivan Lezhankin
1eccbc39c5 Don't add virtual column to empty block 2019-05-28 00:01:24 +03:00
Ivan Lezhankin
13212c9b01 Fix the hang on dropping Kafka table when there is no mat. views 2019-05-27 20:25:34 +03:00
alexey-milovidov
81ed764d6a
Merge pull request #5285 from abyss7/issue-5085 
Don't crash the server when Kafka consumers have failed to start
 2019-05-25 16:09:26 +03:00
Ivan Lezhankin
19a850ad75 Use read callback to populate virtual columns in Kafka Engine 2019-05-23 16:20:25 +03:00
Ivan Lezhankin
800854119e Add buffer callback to fill in virtual columns 2019-05-23 14:15:18 +03:00
Ivan Lezhankin
dd906eabdc [WIP] refactoring 2019-05-22 22:38:43 +03:00
Ivan Lezhankin
cf39c4cc47 Embed virtual columns into IStorage 2019-05-21 14:24:32 +03:00
Ivan Lezhankin
a14013dff7 Add setting to regulate Kafka committing policy. 2019-05-16 18:20:30 +03:00
alexey-milovidov
8e2bdde93b
Update StorageKafka.cpp 2019-05-15 23:57:24 +03:00
Ivan
1ea9e3019d
Freeze the Kafka buffer after first empty response (#5283) 
* Check inside inferior streams for cancellation while reading.
* Stop reading from Kafka buffer after first empty read.
 2019-05-15 19:11:50 +03:00
Ivan Lezhankin
337ddf5047 Catch cppkafka exceptions about consumer creation failures 2019-05-15 17:23:52 +03:00
Ivan
4a5832b18a
Check the time limit every (flush_interval / poll_timeout) number of rows from Kafka (#5249) 2019-05-14 18:52:03 +03:00
Vitaly Baranov
28d9a80072 Rename macro APPLY_FOR_SETTINGS => LIST_OF_SETTINGS. 2019-04-29 03:22:59 +03:00
Vitaly Baranov
8277e9d8f1 Refactoring of Settings based on new template class SettingsCollection. 
Add new class SettingsConstraints.
 2019-04-26 19:11:05 +03:00
proller
ba4687e110
Cmake: dont use LINK_MODE (but use BUILD_SHARED_LIBS) (#5091) 2019-04-24 12:06:28 +03:00
Ivan Lezhankin
94bba1918f Do not drop Kafka consumer buffers after closing stream 2019-04-22 16:23:05 +03:00
Ivan
957d23267b
Minimal implementation of row-level security CLICKHOUSE-4315 (#4792) 
For detailed description see the related PR
 2019-03-29 23:31:06 +03:00
Alexey Milovidov
25ab402d21 ParserCreateQuery: moved code to cpp 2019-03-24 01:45:28 +03:00
proller
fce9db12b9 Move Interpreters/Settings* -> Core/Settings* 2019-03-22 15:08:30 +03:00
Ivan Lezhankin
4f6a494af8 Revert references where they were before refactoring. 2019-03-15 19:14:13 +03:00
Ivan Lezhankin
4c1a0ca255 Merge remote-tracking branch 'upstream/master' into iast-as 2019-03-12 15:51:18 +03:00
proller
8227f5fe63 Arcadia fix 2019-03-12 15:06:17 +03:00
Ivan Lezhankin
d1ead22073 Rename As<…> → as<…> 2019-03-11 16:22:51 +03:00
Ivan Lezhankin
a7e838e328 Replace all other casts to method IAST::As<Derived> 
Also remove some redundant methods like IsIdentifier() et. al.
 2019-03-11 15:49:39 +03:00
Ivan Lezhankin
0e325c7a4a Replace const typeid_casts with As<Derived>() method invocations 2019-03-08 12:48:20 +03:00
Vitaly Baranov
309df832ec IStorage::read() function, "max_block_size" parameter: change type UInt64 -> size_t. 2019-02-19 14:56:08 +03:00
proller
bdefba6c32 fix link order 2019-02-12 15:59:40 +03:00
proller
558ebbcc31 New library clickhouse_storage_kafka 2019-02-12 14:17:46 +03:00
proller
4078a62573 Fix build without contrib/cppkafka submodule 2019-02-11 19:34:32 +03:00
Ivan Lezhankin
15ad409875 Tests are now passing locally 2019-02-11 14:54:30 +03:00
Ivan Lezhankin
2bd14117f4 Merge remote-tracking branch 'upstream/master' into issue-2675 2019-02-11 13:57:40 +03:00
Alexey Milovidov
58a6accee5 Updated test #4246 2019-02-10 19:55:12 +03:00
Ivan Lezhankin
39e2563f93 Create new consumers on-demand 
and give them some time for graceful assignment.
 2019-01-30 20:58:49 +03:00
Ivan Lezhankin
681266abd0 Poll messages in batches of max_block_size size. 2019-01-25 15:48:59 +03:00
Ivan Lezhankin
9636bdcd67 Fix build 2019-01-24 15:44:58 +03:00
Ivan Lezhankin
862d419f31 Fix losing the next message after a broken one. 2019-01-24 14:58:25 +03:00
Ivan Lezhankin
425d483cfb Add integration tests 2019-01-24 14:58:25 +03:00
Ivan Lezhankin
c8e605327d Split StorageKafka.cpp on smaller files per class 2019-01-24 14:58:24 +03:00
Ivan Lezhankin
05ab15cc84 Merge IBlockInputStream and IProfilingBlockInputStream into one 2019-01-23 17:48:50 +03:00
maiha
6c83beafdd dbms: Fixed misspells in comments 2019-01-23 04:56:53 +09:00