thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
99,301 Commits 9,328 Branches 1,783 Tags 2.1 GiB
e7ca30cd11
Commit Graph

260 Commits

Author SHA1 Message Date
Nikolai Kochetov
3d3e143c29
Merge pull request #30001 from ClickHouse/remove-streams-from-formats 
Remove streams from formats.
 2021-10-14 12:39:07 +03:00
Maksim Kita
659a46a972 WriteBufferToRabbitMQProduced fixed queue usage 2021-10-14 00:33:18 +03:00
Maksim Kita
04047f76c7 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
d55561422f Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
c9b6c2661b Refactor ConcurrentBoundedQueue 2021-10-14 00:33:18 +03:00
Nikolai Kochetov
2957971ee3 Remove some last streams. 2021-10-13 21:22:02 +03:00
Nikolai Kochetov
93b3fcc195 Fix rabbitmq 2021-10-12 13:05:20 +03:00
Nikolai Kochetov
ec18340351 Remove streams from formats. 2021-10-11 19:11:50 +03:00
Nikolai Kochetov
78e1db209f
Remove more data streams (#29491) 
* Remove more streams.

* Fixing build.

* Fixing build.

* Rename files.

* Fix fast test.

* Fix StorageKafka.

* Try fix kafka test.

* Move createBuffer to KafkaSource ctor.

* Revert "Move createBuffer to KafkaSource ctor."

This reverts commit 81fa94d27e.

* Revert "Try fix kafka test."

This reverts commit 2107e54969.

* Comment some rows in test.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-10-07 11:26:08 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Nikolai Kochetov
9d3e8fb9c3 Tiny fix. 2021-09-24 13:41:52 +03:00
Nikolai Kochetov
cc53ba283d Try fix other tests. 2021-09-24 13:23:28 +03:00
Nikolai Kochetov
a8c3b02598 Merge branch 'master' into rewrite-pushing-to-views 2021-09-17 15:38:11 +03:00
Kseniia Sumarokova
e9d44d0848
Merge pull request #28797 from kssenii/rabbit-fix 
Rewrite StorageRabbitMQ a little
 2021-09-16 23:58:51 +03:00
kssenii
7b068317c1 Fix build 2021-09-16 13:55:18 +00:00
kssenii
432ca1ed93 Fix tests 2021-09-16 10:46:43 +00:00
Nikolai Kochetov
e616732743 Small refactoring. 2021-09-15 22:35:48 +03:00
Kseniia Sumarokova
fbc048b2bf
Merge branch 'master' into rabbit-fix 2021-09-15 12:11:57 +03:00
kssenii
32c9f76ddc Some fixes 2021-09-15 09:09:24 +00:00
Anton Popov
fc17936c12 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-14 23:02:30 +03:00
Nikolai Kochetov
0e267c50b4 Merge branch 'master' into rewrite-pushing-to-views 2021-09-14 16:13:54 +03:00
kssenii
0978c8baab Fix tests 2021-09-12 16:55:37 +03:00
Vitaly Baranov
51dbedafd9 Expand macros in RabbitMQ settings. 2021-09-12 16:42:03 +03:00
Kseniia Sumarokova
6cc4ad97c3
Update WriteBufferToRabbitMQProducer.h 2021-09-12 00:35:18 +03:00
kssenii
453bd27c0a Review fixes 2021-09-11 20:40:18 +03:00
kssenii
f26f64993a Rewrite StorageRabbitMQ a bit 2021-09-10 13:35:24 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
kssenii
24010689e6 Fix possible throw from shutdown 2021-09-09 15:08:18 +03:00
Anton Popov
4c388e3d84 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-09-09 14:10:16 +03:00
Anton Popov
59128412ea Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-03 22:17:11 +03:00
Nikolai Kochetov
66a76ab70f Rewrite PushingToViewsBlockOutputStream part 6 2021-09-03 20:29:36 +03:00
Anton Popov
c9a0cc5ce9 fix race on table schema in insert-select 2021-09-03 17:23:21 +03:00
Kseniia Sumarokova
09c9dd489b
Update StorageRabbitMQ.cpp 2021-09-01 13:28:34 +03:00
kssenii
335b6f12fd Fix 2021-08-30 20:59:03 +03:00
Anton Popov
e8ac8e3454 execute asynchronous inserts separatly for each client 2021-08-27 06:00:12 +03:00
Anton Popov
6d3274c22c Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-26 03:31:46 +03:00
Vitaly Baranov
64f69462c6
Merge pull request #27921 from vitlibar/add-tests-for-field-init 
Allow implicit cast bool to Field
 2021-08-24 17:15:44 +03:00
Vitaly Baranov
ff1fad8521 Use setSetting() with implicit conversion true/false -> Field. 2021-08-23 08:57:00 +03:00
Azat Khuzhin
b60e5ac801 Fix incorrect assertion during writing to StorageRabbitMQ 2021-08-20 07:14:20 +03:00
Anton Popov
3a0d4807a5 Merge branch 'async-insert' of git://github.com/abyss7/ClickHouse into merging-20557 2021-08-06 16:20:02 +03:00
Anton Popov
e36736b50c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-02 22:52:02 +03:00
kssenii
c6ee94db2e Fix 2021-07-28 08:11:10 +00:00
Nikolai Kochetov
61d8f880cd Rename some files. 2021-07-26 19:48:25 +03:00
Nikolai Kochetov
0eb563dc1b Fix more tests. 2021-07-26 17:47:29 +03:00
Nikolai Kochetov
2dc5c89b66 Update Storage::write 2021-07-23 17:25:35 +03:00
Anton Popov
f99374cca6 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-07-20 18:20:21 +03:00
alexey-milovidov
3bcef76a9d
Merge pull request #26529 from kssenii/fixing-rabbitmq 
Fix rabbitmq shutdown in case rabbitmq setup was not started
 2021-07-20 16:41:25 +03:00
kssenii
63c05e6e14 Fix rabbitmq shutdown in case rabbitmq setup was not started 2021-07-19 21:06:10 +00:00
Alexey Milovidov
261a220227 Remove some code 2021-07-17 21:06:46 +03:00
Anton Popov
3ed7f5a6cc dynamic subcolumns: add snapshot for storage 2021-07-09 06:15:41 +03:00
Anton Popov
072e65b728 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-07-07 17:20:38 +03:00
Ivan Lezhankin
37365589ed Merge branch 'master' into async-insert 2021-06-17 16:57:45 +03:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Ivan Lezhankin
774107738c Fix build 2021-06-02 19:52:33 +03:00
Ivan Lezhankin
791cbcf0b1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-02 18:50:37 +03:00
Ivan Lezhankin
84c23dc060 Merge branch 'master' into async-insert 2021-06-02 18:06:21 +03:00
Anton Popov
9f52362b81 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-05-31 23:08:45 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
Anton Popov
0bdf9d207c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-05-26 00:25:47 +03:00
kssenii
53ce22ae65 Exponencial backoff if queues are empty 2021-05-22 09:44:57 +00:00
kssenii
e524d11222 Remove setting 2021-05-16 20:16:10 +00:00
kssenii
55c573f920 Fix 2021-05-14 07:11:32 +00:00
kssenii
7115045317 Tiny changes 2021-05-13 09:39:57 +00:00
kssenii
0cb0bd48f7 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into rabbitmq-fixes 2021-05-13 06:36:51 +00:00
Alexey Milovidov
e23a0d8f7e Fix warnings by PVS-Studio 2021-05-08 23:57:08 +03:00
Alexey Milovidov
367f7fe6c9 Fix warnings by PVS-Studio 2021-05-08 23:57:08 +03:00
Alexey Milovidov
ad88819ee4 Fix a bunch of warnings from PVS-Studio 2021-05-08 19:13:10 +03:00
Alexey Milovidov
3025f9e141 More fixes for PVS-Studio 2021-05-08 19:12:49 +03:00
Azat Khuzhin
c18ed16699 Make StorageRabbitMQ.cpp self compilable (missing AbstractConfiguration.h) 2021-05-06 09:07:58 +03:00
kssenii
0b6fb7660b Fix checks 2021-05-05 09:15:56 +00:00
Anton Popov
e44706911e dynamic columns: better getting of sample block 2021-05-05 02:02:54 +03:00
kssenii
973ee4e9c4 Remove strange code, some fixes 2021-05-04 20:24:57 +00:00
kssenii
2112521304 Allow all RabbitMQ setup to be done only by user 2021-05-04 19:54:16 +00:00
kssenii
a3feaa48f4 Allow user to define specific queue settings 2021-05-04 18:57:49 +00:00
kssenii
13ba5287e3 Add cleanup in case of drop table 2021-05-04 16:26:47 +00:00
kssenii
4aad69dc90 Better commnets 2021-05-04 13:03:53 +00:00
kssenii
2e6c753add Allow separate vhost for each table 2021-04-21 15:51:05 +00:00
Ivan Lezhankin
ee194928d2 CC 2021-04-21 16:19:28 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
Maksim Kita
749dd8d555
Merge pull request #21193 from kitaisreal/storage-rabbit-mq-added-uvloop 
StorageRabbitMQ added UVLoop
 2021-02-26 11:04:02 +03:00
Maksim Kita
29e783999c Fixed typo 2021-02-25 17:43:32 +03:00
Maksim Kita
f077a25a7a StorageRabbitMQ added UVLoop 2021-02-25 17:02:34 +03:00
kssenii
6eaf9281b5 Review fix 2021-02-24 16:52:14 +00:00
kssenii
96f22b4290 Fix rabbit bug 2021-02-24 10:41:08 +00:00
alexey-milovidov
2299930c31
Merge pull request #20545 from kssenii/rabbit-format-settings 
rabbitmq: add missing format factory settings
 2021-02-19 20:24:17 +03:00
Vitaly Baranov
3cbb325913
Merge pull request #20506 from vitlibar/refactor-protobuf-format-io 
Improved serialization in Protobuf format.
 2021-02-18 11:31:37 +03:00
Vitaly Baranov
18e036d19b Improved serialization for data types combined of Arrays and Tuples. 
Improved matching enum data types to protobuf enum type.
Fixed serialization of the Map data type.
Omitted values are now set by default.
 2021-02-17 20:50:09 +03:00
Kseniia Sumarokova
20ff183549
Merge branch 'master' into rabbit-format-settings 2021-02-17 19:01:53 +03:00
kssenii
29362bb483 Support vhost 2021-02-16 22:05:14 +00:00
kssenii
2a887b9772 Add missing format factory settings 2021-02-15 22:06:10 +00:00
Azat Khuzhin
fb4596f433 Fix clang-analyzer-optin.cplusplus.VirtualCall 2021-02-04 02:08:41 +03:00
Azat Khuzhin
46f5268135 Fix incorrect WriteBuffer interface usage in message brokers producers 
WriteBufferToKafkaProducer and WriteBufferToRabbitMQProducer uses
WriteBuffer::set(nullptr, 0) which will leave the WriteBuffer in the
invalid state, since after this available() is 0 and next() will not
call nextImpl().

Stacktrace:

    2021.02.02 05:27:16.248056 [ 97 ] {} <Fatal> BaseDaemon: ########################################
    2021.02.02 05:27:16.248697 [ 97 ] {} <Fatal> BaseDaemon: (version 21.3.1.5880, build id: 03F265087EF651DC4D8E569775FE9E91FFA5DE76) (from thread 93) (query_id: 8c34c220-f66a-45d2-8f4f-01193cede98b) Received signal Segmentation fault (11)
    2021.02.02 05:27:16.249030 [ 97 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: write. Address not mapped to object.
    2021.02.02 05:27:16.249623 [ 97 ] {} <Fatal> BaseDaemon: Stack trace: 0x12f3dbeb 0x12efd075 0x12efc90e 0x12ebb0e9 0x12ebe0cb 0x12e1da18 0x12a8c857 0x1219aa62 0x1218fc82 0x1218fdaa 0x11e3f120 0x12dfa3df 0x12df42a4 0x12e00fa8 0x158cdc63 0x158ce38f 0x15a36b72 0x15a35110 0x15a33918 0x8adf12d 0x7fb5e0fe3609 0x7fb5e0f0a293
    2021.02.02 05:27:16.408985 [ 97 ] {} <Fatal> BaseDaemon: 5. ./obj-x86_64-linux-gnu/../src/IO/WriteHelpers.h:64: DB::JSONEachRowRowOutputFormat::writeRowStartDelimiter() @ 0x12f3dbeb in /usr/bin/clickhouse
    2021.02.02 05:27:16.534797 [ 97 ] {} <Fatal> BaseDaemon: 6. ./obj-x86_64-linux-gnu/../src/Processors/Formats/IRowOutputFormat.cpp:80: DB::IRowOutputFormat::write(std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn> > > const&, unsigned long) @ 0x12efd075 in /usr/bin/clickhouse
    2021.02.02 05:27:16.599210 [ 97 ] {} <Fatal> BaseDaemon: 7. ./obj-x86_64-linux-gnu/../src/Processors/Formats/IRowOutputFormat.cpp:0: DB::IRowOutputFormat::consume(DB::Chunk) @ 0x12efc90e in /usr/bin/clickhouse
    2021.02.02 05:27:16.680733 [ 97 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3211: DB::IOutputFormat::write(DB::Block const&) @ 0x12ebb0e9 in /usr/bin/clickhouse
    2021.02.02 05:27:16.798768 [ 97 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Processors/Formats/OutputStreamToOutputFormat.cpp:15: DB::OutputStreamToOutputFormat::write(DB::Block const&) @ 0x12ebe0cb in /usr/bin/clickhouse
    2021.02.02 05:27:16.981593 [ 97 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/DataStreams/MaterializingBlockOutputStream.h:0: DB::MaterializingBlockOutputStream::write(DB::Block const&) @ 0x12e1da18 in /usr/bin/clickhouse
    2021.02.02 05:27:17.090383 [ 97 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Storages/RabbitMQ/RabbitMQBlockOutputStream.cpp:61: DB::RabbitMQBlockOutputStream::write(DB::Block const&) @ 0x12a8c857 in /usr/bin/clickhouse
    2021.02.02 05:27:17.140425 [ 97 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../src/DataStreams/AddingDefaultBlockOutputStream.cpp:0: DB::AddingDefaultBlockOutputStream::write(DB::Block const&) @ 0x1219aa62 in /usr/bin/clickhouse
    2021.02.02 05:27:17.190282 [ 97 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../src/DataStreams/SquashingBlockOutputStream.cpp:0: DB::SquashingBlockOutputStream::finalize() @ 0x1218fc82 in /usr/bin/clickhouse
    2021.02.02 05:27:17.240052 [ 97 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2844: DB::SquashingBlockOutputStream::writeSuffix() @ 0x1218fdaa in /usr/bin/clickhouse
    2021.02.02 05:27:17.294527 [ 97 ] {} <Fatal> BaseDaemon: 15. ./obj-x86_64-linux-gnu/../src/DataStreams/CountingBlockOutputStream.h:37: DB::CountingBlockOutputStream::writeSuffix() @ 0x11e3f120 in /usr/bin/clickhouse
    2021.02.02 05:27:17.405442 [ 97 ] {} <Fatal> BaseDaemon: 16. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:521: DB::TCPHandler::processInsertQuery(DB::Settings const&) @ 0x12dfa3df in /usr/bin/clickhouse
    2021.02.02 05:27:17.475822 [ 97 ] {} <Fatal> BaseDaemon: 17. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x12df42a4 in /usr/bin/clickhouse
    2021.02.02 05:27:17.609845 [ 97 ] {} <Fatal> BaseDaemon: 18. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1419: DB::TCPHandler::run() @ 0x12e00fa8 in /usr/bin/clickhouse
    2021.02.02 05:27:17.695292 [ 97 ] {} <Fatal> BaseDaemon: 19. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x158cdc63 in /usr/bin/clickhouse
    2021.02.02 05:27:17.833612 [ 97 ] {} <Fatal> BaseDaemon: 20. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:0: Poco::Net::TCPServerDispatcher::run() @ 0x158ce38f in /usr/bin/clickhouse
    2021.02.02 05:27:17.907391 [ 97 ] {} <Fatal> BaseDaemon: 21. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/ScopedLock.h:36: Poco::PooledThread::run() @ 0x15a36b72 in /usr/bin/clickhouse
    2021.02.02 05:27:18.033949 [ 97 ] {} <Fatal> BaseDaemon: 22. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x15a35110 in /usr/bin/clickhouse
    2021.02.02 05:27:18.122761 [ 97 ] {} <Fatal> BaseDaemon: 23. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/SharedPtr.h:277: Poco::ThreadImpl::runnableEntry(void*) @ 0x15a33918 in /usr/bin/clickhouse
    2021.02.02 05:27:18.283663 [ 97 ] {} <Fatal> BaseDaemon: 24. __tsan_thread_start_func @ 0x8adf12d in /usr/bin/clickhouse
    2021.02.02 05:27:18.284153 [ 97 ] {} <Fatal> BaseDaemon: 25. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    2021.02.02 05:27:18.284544 [ 97 ] {} <Fatal> BaseDaemon: 26. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021.02.02 05:27:21.675458 [ 97 ] {} <Fatal> BaseDaemon: Calculated checksum of the binary: B53B58DB6CF8186EEC9EA9273F135E44. There is no information about the reference checksum.
    2021.02.02 05:27:28.078805 [ 1 ] {} <Fatal> Application: Child process was terminated by signal 11.
 2021-02-03 23:46:28 +03:00
Azat Khuzhin
fabace394d Cleanup message brokers output streams 2021-02-03 00:32:36 +03:00
Azat Khuzhin
acde56ff8a Add separate pool for message brokers (RabbitMQ and Kafka) 
background_message_broker_schedule_pool_size was introduced in #13939,
but never used actually, use it for RabbitMQ and Kafka.
 2021-01-27 21:08:41 +03:00
Azat Khuzhin
64cb6405ac Drop IStorage::supportsSettings() (replaced with StorageFeatures::supports_settings) 2021-01-08 14:36:26 +03:00
Nikita Mikhailov
c5f92e5096 better formatfactory 2020-12-30 06:07:30 +03:00
Nikolai Kochetov
8de5cd5bc7 Merge branch 'master' into async-read-from-socket 2020-12-14 17:45:38 +03:00
Azat Khuzhin
5b3ab48861 More forward declaration for generic headers 
The following headers are pretty generic, so use forward declaration as
much as possible:
- Context.h
- Settings.h
- ConnectionTimeouts.h
(Also this shows that some missing some includes -- this has been fixed)

And split ConnectionTimeouts.h into ConnectionTimeoutsContext.h (since
module part cannot be added for it, due to recursive build dependencies
that will be introduced)

Also remove Settings from the RemoteBlockInputStream/RemoteQueryExecutor
and just pass the context, since settings was passed only in speicifc
places, that can allow making a copy of Context (i.e. Copier).

Approx results (How much units will be recompiled after changing file X?):

- ConnectionTimeouts.h
  - mainline: 100

- Context.h:
  - mainline: ~800
  - patched:  415

- Settings.h:
  - mainline: 900-1K
  - patched:  440 (most of them because of the Context.h)
 2020-12-12 17:43:10 +03:00
Nikolai Kochetov
32b38f389e Merge branch 'master' into async-read-from-socket 2020-12-09 17:15:36 +03:00
kssenii
8f1ed36897 Fix races 2020-12-06 00:24:49 +00:00