thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
72,889 Commits 9,328 Branches 1,783 Tags 2.1 GiB
0009cab962
Commit Graph

146 Commits

Author SHA1 Message Date
Anton Popov
abebce2f18 fix kafka 2021-09-09 17:48:01 +03:00
Anton Popov
e8ac8e3454 execute asynchronous inserts separatly for each client 2021-08-27 06:00:12 +03:00
Anton Popov
6d3274c22c Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-26 03:31:46 +03:00
Azat Khuzhin
5947e54c1b Fix incorrect assertion during writing to StorageKafka 
The problem it does not triggered in CI always because buffers was not
destroyed by that time.

Fixes: #26547
 2021-08-19 22:07:20 +03:00
alexey-milovidov
bcfab277e7
Update ReadBufferFromKafkaConsumer.cpp 2021-08-18 15:25:54 +03:00
Mikhail Filimonov
05d77d2873
Safer processing of NULL messages in Kafka for some formats 2021-08-18 11:26:57 +02:00
Anton Popov
3a0d4807a5 Merge branch 'async-insert' of git://github.com/abyss7/ClickHouse into merging-20557 2021-08-06 16:20:02 +03:00
Nikolai Kochetov
0eb563dc1b Fix more tests. 2021-07-26 17:47:29 +03:00
Nikolai Kochetov
2dc5c89b66 Update Storage::write 2021-07-23 17:25:35 +03:00
feihengye
6230ad0160
#26640 change kafka engine max consumers from 16 to physical cpu cores (#26642) 
Co-authored-by: jhonye <jhonye@tencent.com>
 2021-07-22 11:54:08 +03:00
Alexey Milovidov
261a220227 Remove some code 2021-07-17 21:06:46 +03:00
Ivan Lezhankin
37365589ed Merge branch 'master' into async-insert 2021-06-17 16:57:45 +03:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Mikhail Filimonov
3627ab7cbe
Fix kafka failover issue (#21118). 
Distringuish lack of assignment vs lack of partitions. Bit better / clearer logging.
 2021-06-09 16:02:56 +02:00
Ivan Lezhankin
774107738c Fix build 2021-06-02 19:52:33 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
Ivan Lezhankin
ee194928d2 CC 2021-04-21 16:19:28 +03:00
Azat Khuzhin
4b69684cd2 Add logging about pushing to underlying in StorageKafka 2021-04-14 23:15:31 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
Peng Jian
26b5482b4d remove the flag in the parser 2021-03-31 22:25:51 +08:00
Peng Jian
ec54f2a361 fix compilation error 2021-03-31 17:16:51 +08:00
Peng Jian
278d2d7df0 fix typo 2021-03-31 17:16:51 +08:00
Peng Jian
cad54d4e2c reset the parser if exception catched. 2021-03-31 17:16:51 +08:00
Peng Jian
85cfb927e6 move HandleKafkaErrorMode to KafkaSettings 2021-03-31 17:16:51 +08:00
Peng Jian
909d5ad2b5 Handle errors for Kafka engine 2021-03-31 17:15:57 +08:00
Vitaly Baranov
18e036d19b Improved serialization for data types combined of Arrays and Tuples. 
Improved matching enum data types to protobuf enum type.
Fixed serialization of the Map data type.
Omitted values are now set by default.
 2021-02-17 20:50:09 +03:00
Azat Khuzhin
fb4596f433 Fix clang-analyzer-optin.cplusplus.VirtualCall 2021-02-04 02:08:41 +03:00
Azat Khuzhin
46f5268135 Fix incorrect WriteBuffer interface usage in message brokers producers 
WriteBufferToKafkaProducer and WriteBufferToRabbitMQProducer uses
WriteBuffer::set(nullptr, 0) which will leave the WriteBuffer in the
invalid state, since after this available() is 0 and next() will not
call nextImpl().

Stacktrace:

    2021.02.02 05:27:16.248056 [ 97 ] {} <Fatal> BaseDaemon: ########################################
    2021.02.02 05:27:16.248697 [ 97 ] {} <Fatal> BaseDaemon: (version 21.3.1.5880, build id: 03F265087EF651DC4D8E569775FE9E91FFA5DE76) (from thread 93) (query_id: 8c34c220-f66a-45d2-8f4f-01193cede98b) Received signal Segmentation fault (11)
    2021.02.02 05:27:16.249030 [ 97 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: write. Address not mapped to object.
    2021.02.02 05:27:16.249623 [ 97 ] {} <Fatal> BaseDaemon: Stack trace: 0x12f3dbeb 0x12efd075 0x12efc90e 0x12ebb0e9 0x12ebe0cb 0x12e1da18 0x12a8c857 0x1219aa62 0x1218fc82 0x1218fdaa 0x11e3f120 0x12dfa3df 0x12df42a4 0x12e00fa8 0x158cdc63 0x158ce38f 0x15a36b72 0x15a35110 0x15a33918 0x8adf12d 0x7fb5e0fe3609 0x7fb5e0f0a293
    2021.02.02 05:27:16.408985 [ 97 ] {} <Fatal> BaseDaemon: 5. ./obj-x86_64-linux-gnu/../src/IO/WriteHelpers.h:64: DB::JSONEachRowRowOutputFormat::writeRowStartDelimiter() @ 0x12f3dbeb in /usr/bin/clickhouse
    2021.02.02 05:27:16.534797 [ 97 ] {} <Fatal> BaseDaemon: 6. ./obj-x86_64-linux-gnu/../src/Processors/Formats/IRowOutputFormat.cpp:80: DB::IRowOutputFormat::write(std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn> > > const&, unsigned long) @ 0x12efd075 in /usr/bin/clickhouse
    2021.02.02 05:27:16.599210 [ 97 ] {} <Fatal> BaseDaemon: 7. ./obj-x86_64-linux-gnu/../src/Processors/Formats/IRowOutputFormat.cpp:0: DB::IRowOutputFormat::consume(DB::Chunk) @ 0x12efc90e in /usr/bin/clickhouse
    2021.02.02 05:27:16.680733 [ 97 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3211: DB::IOutputFormat::write(DB::Block const&) @ 0x12ebb0e9 in /usr/bin/clickhouse
    2021.02.02 05:27:16.798768 [ 97 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Processors/Formats/OutputStreamToOutputFormat.cpp:15: DB::OutputStreamToOutputFormat::write(DB::Block const&) @ 0x12ebe0cb in /usr/bin/clickhouse
    2021.02.02 05:27:16.981593 [ 97 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/DataStreams/MaterializingBlockOutputStream.h:0: DB::MaterializingBlockOutputStream::write(DB::Block const&) @ 0x12e1da18 in /usr/bin/clickhouse
    2021.02.02 05:27:17.090383 [ 97 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Storages/RabbitMQ/RabbitMQBlockOutputStream.cpp:61: DB::RabbitMQBlockOutputStream::write(DB::Block const&) @ 0x12a8c857 in /usr/bin/clickhouse
    2021.02.02 05:27:17.140425 [ 97 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../src/DataStreams/AddingDefaultBlockOutputStream.cpp:0: DB::AddingDefaultBlockOutputStream::write(DB::Block const&) @ 0x1219aa62 in /usr/bin/clickhouse
    2021.02.02 05:27:17.190282 [ 97 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../src/DataStreams/SquashingBlockOutputStream.cpp:0: DB::SquashingBlockOutputStream::finalize() @ 0x1218fc82 in /usr/bin/clickhouse
    2021.02.02 05:27:17.240052 [ 97 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2844: DB::SquashingBlockOutputStream::writeSuffix() @ 0x1218fdaa in /usr/bin/clickhouse
    2021.02.02 05:27:17.294527 [ 97 ] {} <Fatal> BaseDaemon: 15. ./obj-x86_64-linux-gnu/../src/DataStreams/CountingBlockOutputStream.h:37: DB::CountingBlockOutputStream::writeSuffix() @ 0x11e3f120 in /usr/bin/clickhouse
    2021.02.02 05:27:17.405442 [ 97 ] {} <Fatal> BaseDaemon: 16. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:521: DB::TCPHandler::processInsertQuery(DB::Settings const&) @ 0x12dfa3df in /usr/bin/clickhouse
    2021.02.02 05:27:17.475822 [ 97 ] {} <Fatal> BaseDaemon: 17. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x12df42a4 in /usr/bin/clickhouse
    2021.02.02 05:27:17.609845 [ 97 ] {} <Fatal> BaseDaemon: 18. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1419: DB::TCPHandler::run() @ 0x12e00fa8 in /usr/bin/clickhouse
    2021.02.02 05:27:17.695292 [ 97 ] {} <Fatal> BaseDaemon: 19. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x158cdc63 in /usr/bin/clickhouse
    2021.02.02 05:27:17.833612 [ 97 ] {} <Fatal> BaseDaemon: 20. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:0: Poco::Net::TCPServerDispatcher::run() @ 0x158ce38f in /usr/bin/clickhouse
    2021.02.02 05:27:17.907391 [ 97 ] {} <Fatal> BaseDaemon: 21. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/ScopedLock.h:36: Poco::PooledThread::run() @ 0x15a36b72 in /usr/bin/clickhouse
    2021.02.02 05:27:18.033949 [ 97 ] {} <Fatal> BaseDaemon: 22. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x15a35110 in /usr/bin/clickhouse
    2021.02.02 05:27:18.122761 [ 97 ] {} <Fatal> BaseDaemon: 23. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/SharedPtr.h:277: Poco::ThreadImpl::runnableEntry(void*) @ 0x15a33918 in /usr/bin/clickhouse
    2021.02.02 05:27:18.283663 [ 97 ] {} <Fatal> BaseDaemon: 24. __tsan_thread_start_func @ 0x8adf12d in /usr/bin/clickhouse
    2021.02.02 05:27:18.284153 [ 97 ] {} <Fatal> BaseDaemon: 25. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    2021.02.02 05:27:18.284544 [ 97 ] {} <Fatal> BaseDaemon: 26. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.02.02 05:27:21.675458 [ 97 ] {} <Fatal> BaseDaemon: Calculated checksum of the binary: B53B58DB6CF8186EEC9EA9273F135E44. There is no information about the reference checksum.
    2021.02.02 05:27:28.078805 [ 1 ] {} <Fatal> Application: Child process was terminated by signal 11.
 2021-02-03 23:46:28 +03:00
Azat Khuzhin
fabace394d Cleanup message brokers output streams 2021-02-03 00:32:36 +03:00
Azat Khuzhin
acde56ff8a Add separate pool for message brokers (RabbitMQ and Kafka) 
background_message_broker_schedule_pool_size was introduced in #13939,
but never used actually, use it for RabbitMQ and Kafka.
 2021-01-27 21:08:41 +03:00
Azat Khuzhin
64cb6405ac Drop IStorage::supportsSettings() (replaced with StorageFeatures::supports_settings) 2021-01-08 14:36:26 +03:00
Nikita Mikhailov
c5f92e5096 better formatfactory 2020-12-30 06:07:30 +03:00
Nikolai Kochetov
8de5cd5bc7 Merge branch 'master' into async-read-from-socket 2020-12-14 17:45:38 +03:00
Azat Khuzhin
2bf226decc Fix typo in KafkaSettings comments 2020-12-14 00:38:10 +03:00
Nikolai Kochetov
088c128832 Fixing build and tests. 2020-12-09 17:11:20 +03:00
tavplubix
085359c110
Merge pull request #17274 from ClickHouse/fix_ast_formatting_in_logs 
Fix AST formatting in log messages
 2020-11-24 19:00:56 +03:00
Alexander Tokmakov
60a5782c75 fix AST formatting in log messages 2020-11-22 20:23:12 +03:00
Amos Bird
1d9d586e20
Make global_context consistent. 2020-11-20 18:23:14 +08:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
tavplubix
67099f28ac
Merge pull request #16591 from ClickHouse/aku/create-file 
Support `SETTINGS` clause for File engine
 2020-11-09 14:15:42 +03:00
alexey-milovidov
f4ba5f1f9a
Merge pull request #16772 from ClickHouse/fix-stringstream 
Fix "server failed to start" error
 2020-11-08 14:27:08 +03:00
Alexey Milovidov
5314185e25 Merge branch 'master' into azat-optimize_skip_unused_shards-optimization 2020-11-08 00:17:59 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Alexander Kuzmenkov
99ee127620 Support SETTINGS clause for File engine 
Accept the usual user settings related to file formats.

Most of the diff are the mechanistic code changes required to allow
providing the required FormatSettings to the format factory. The File
engine then extracts these settings from the `CREATE` query, and specifies
them when creating the format parser.
 2020-11-02 10:50:38 +03:00
Alexey Milovidov
d5e277f080 Better exceptions rethrow 2020-10-26 06:35:57 +03:00
Mikhail Filimonov
87a8ba63f7
Few more CR fixes 2020-10-12 15:05:40 +02:00
Mikhail Filimonov
f6b00f2cb6
After CR fixes 2020-10-12 07:52:09 +02:00
Mikhail Filimonov
6df42c580b
ProtobufSingle output format 2020-10-12 07:51:45 +02:00
Alexey Milovidov
5b482f4191 Cleanups 2020-10-10 19:31:10 +03:00
Azat Khuzhin
1f51de362f Take memory allocations from librdkafka threads into account 2020-10-07 23:18:20 +03:00