thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-10-01 06:00:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
47,241 Commits 8,961 Branches 1,767 Tags 1.8 GiB
d7be761484
Commit Graph

679 Commits

Author SHA1 Message Date
Nikolai Kochetov
d7be761484 Try fix test. 2020-09-04 20:14:36 +03:00
Nikolai Kochetov
c761970bc5 Merge branch 'master' into pipes-and-creating-sets 2020-09-04 11:38:35 +03:00
Nikolai Kochetov
b7e53208d1 Fix tests. 2020-09-04 11:36:47 +03:00
Matwey V. Kornilov
9b4da1e82f Add missed #include <atomic> 
std::atomic<Data*> at line 199 requires including <atomic>
 2020-09-02 23:40:27 +03:00
Nikolai Kochetov
acfd35a908 Update QueryPipeline::addCreatingSetsTransform 2020-09-02 16:04:54 +03:00
Nikolai Kochetov
811e44a937
Merge pull request #14315 from azat/EXPLAIN-SIGSEGV-fix 
[RFC] Fix QueryPlan lifetime (for EXPLAIN PIPELINE graph=1) for queries with nested interpreter
 2020-09-02 12:46:59 +03:00
Nikolai Kochetov
f82b799f12 Update CreatingSetsTransform. 2020-09-01 16:53:11 +03:00
Nikolai Kochetov
8fac595428 Stop query execution if exception happened in PipelineExecutor itself. 2020-09-01 13:29:10 +03:00
Azat Khuzhin
d04cda0367 Fix QueryPlan lifetime (for EXPLAIN PIPELINE graph=1) for queries with nested interpreter 
Example of such queries are distributed queries, which creates local
InterpreterSelectQuery, which will have it's own QueryPlan but returns
Pipes that has that IQueryPlanStep attached.

After EXPLAIN PIPELINE graph=1 tries to use them, and will get SIGSEGV.

- TSAN:

<details>

```
==2782113==ERROR: AddressSanitizer: heap-use-after-free on address 0x6120000223c0 at pc 0x00002b8f3f3e bp 0x7fff18cfbff0 sp 0x7fff18cfbfe8
READ of size 8 at 0x6120000223c0 thread T22 (TCPHandler)
    #0 0x2b8f3f3d in DB::printPipelineCompact(std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > > const&, DB::WriteBuffer&, bool) /build/obj-x86_64-linux-gnu/../src/Processors/printPipeline.cpp:116:53
    #1 0x29ee698c in DB::InterpreterExplainQuery::executeImpl() /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterExplainQuery.cpp:275:17
    #2 0x29ee2e40 in DB::InterpreterExplainQuery::execute() /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterExplainQuery.cpp:73:14
    #3 0x2a7b44a2 in DB::executeQueryImpl(char const*, char const*, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*) /build/obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:389:28
    #4 0x2a7b1cb3 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool) /build/obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:675:30
    #5 0x2b7993b2 in DB::TCPHandler::runImpl() /build/obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:253:24
    #6 0x2b7b649a in DB::TCPHandler::run() /build/obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1217:9
    #7 0x31d9c57e in Poco::Net::TCPServerConnection::start() /build/obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3
    #8 0x31d9d281 in Poco::Net::TCPServerDispatcher::run() /build/obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:114:20
    #9 0x3206b5d5 in Poco::PooledThread::run() /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14
    #10 0x320657ad in Poco::ThreadImpl::runnableEntry(void*) /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27
    #11 0x7ffff7f853e8 in start_thread (/usr/lib/libpthread.so.0+0x93e8)
    #12 0x7ffff7ea2292 in clone (/usr/lib/libc.so.6+0x100292)

0x6120000223c0 is located 0 bytes inside of 272-byte region [0x6120000223c0,0x6120000224d0)
freed by thread T22 (TCPHandler) here:
    #0 0x122f3b62 in operator delete(void*, unsigned long) (/src/ch/tmp/master-20200831/clickhouse+0x122f3b62)
    #1 0x2bd9e9fa in std::__1::default_delete<DB::IQueryPlanStep>::operator()(DB::IQueryPlanStep*) const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2363:5
    #2 0x2bd9e9fa in std::__1::unique_ptr<DB::IQueryPlanStep, std::__1::default_delete<DB::IQueryPlanStep> >::reset(DB::IQueryPlanStep*) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2618:7
    #3 0x2bd9e9fa in std::__1::unique_ptr<DB::IQueryPlanStep, std::__1::default_delete<DB::IQueryPlanStep> >::~unique_ptr() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2572:19
    #4 0x2bd9e9fa in DB::QueryPlan::Node::~Node() /build/obj-x86_64-linux-gnu/../src/Processors/QueryPlan/QueryPlan.h:66:12
    #5 0x2bd9e9fa in void std::__1::allocator_traits<std::__1::allocator<std::__1::__list_node<DB::QueryPlan::Node, void*> > >::__destroy<DB::QueryPlan::Node>(std::__1::integral_constant<bool, false>, std::__1::allocator<std::__1::__list_node<DB::QueryPlan::Node, void*> >&,
 DB::QueryPlan::Node*) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:1798:23
    #6 0x2bd9e9fa in void std::__1::allocator_traits<std::__1::allocator<std::__1::__list_node<DB::QueryPlan::Node, void*> > >::destroy<DB::QueryPlan::Node>(std::__1::allocator<std::__1::__list_node<DB::QueryPlan::Node, void*> >&, DB::QueryPlan::Node*) /build/obj-x86_64-lin
ux-gnu/../contrib/libcxx/include/memory:1630:14
    #7 0x2bd9e9fa in std::__1::__list_imp<DB::QueryPlan::Node, std::__1::allocator<DB::QueryPlan::Node> >::clear() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/list:762:13
    #8 0x29fece08 in DB::InterpreterSelectQuery::execute() /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:492:1
    #9 0x2abf7484 in DB::ClusterProxy::(anonymous namespace)::createLocalStream(std::__1::shared_ptr<DB::IAST> const&, DB::Block const&, DB::Context const&, DB::QueryProcessingStage::Enum) /build/obj-x86_64-linux-gnu/../src/Interpreters/ClusterProxy/SelectStreamFactory.cpp:
78:33
    #10 0x2abea85d in DB::ClusterProxy::SelectStreamFactory::createForShard(DB::Cluster::ShardInfo const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, std::__1::shar
ed_ptr<DB::Throttler> const&, DB::SelectQueryInfo const&, std::__1::vector<DB::Pipe, std::__1::allocator<DB::Pipe> >&)::$_0::operator()() const /build/obj-x86_64-linux-gnu/../src/Interpreters/ClusterProxy/SelectStreamFactory.cpp:133:51
    #11 0x2abea85d in DB::ClusterProxy::SelectStreamFactory::createForShard(DB::Cluster::ShardInfo const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, std::__1::shar
ed_ptr<DB::Throttler> const&, DB::SelectQueryInfo const&, std::__1::vector<DB::Pipe, std::__1::allocator<DB::Pipe> >&) /build/obj-x86_64-linux-gnu/../src/Interpreters/ClusterProxy/SelectStreamFactory.cpp:189:13
    #12 0x2abe6d99 in DB::ClusterProxy::executeQuery(DB::ClusterProxy::IStreamFactory&, std::__1::shared_ptr<DB::Cluster> const&, Poco::Logger*, std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::Settings const&, DB::SelectQueryInfo const&) /build/obj-x86_64-lin
ux-gnu/../src/Interpreters/ClusterProxy/executeQuery.cpp:107:24
    #13 0x2abc4b74 in DB::StorageDistributed::read(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, s
td::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, DB::Context const&, DB::QueryProcessingStage::Enum, unsigned long, unsigned int) /build/obj-x86_64-linux-gnu/../src/Storages/StorageDistributed.cpp:514:12
    #14 0x2bda1c5a in DB::ReadFromStorageStep::ReadFromStorageStep(std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>, std::__1::shared_ptr<DB::StorageInMemoryMetadata const>&, DB::SelectQueryOptions, std::__1::shared_ptr<DB::IStorage>, std::__1::vector<std::__1::basic_st
ring<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context>, DB::QueryProcessingStage
::Enum, unsigned long, unsigned long) /build/obj-x86_64-linux-gnu/../src/Processors/QueryPlan/ReadFromStorageStep.cpp:39:26
    #15 0x2a01ca70 in std::__1::__unique_if<DB::ReadFromStorageStep>::__unique_single std::__1::make_unique<DB::ReadFromStorageStep, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const>&, DB::SelectQueryOptions&, std
::__1::shared_ptr<DB::IStorage>&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&, DB::SelectQueryInfo&, st
d::__1::shared_ptr<DB::Context>&, DB::QueryProcessingStage::Enum&, unsigned long&, unsigned long&>(std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const>&, DB::SelectQueryOptions&, std::__1::shared_ptr<DB::IStorage>&,
std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&, DB::SelectQueryInfo&, std::__1::shared_ptr<DB::Context>&,
DB::QueryProcessingStage::Enum&, unsigned long&, unsigned long&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3028:32
    #16 0x29ff556a in DB::InterpreterSelectQuery::executeFetchColumns(DB::QueryProcessingStage::Enum, DB::QueryPlan&, std::__1::shared_ptr<DB::PrewhereInfo> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::
__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:1383:26
    #17 0x29fe6b83 in DB::InterpreterSelectQuery::executeImpl(DB::QueryPlan&, std::__1::shared_ptr<DB::IBlockInputStream> const&, std::__1::optional<DB::Pipe>) /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:795:9
    #18 0x29fe5771 in DB::InterpreterSelectQuery::buildQueryPlan(DB::QueryPlan&) /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:473:5
    #19 0x2a47d370 in DB::InterpreterSelectWithUnionQuery::buildQueryPlan(DB::QueryPlan&) /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectWithUnionQuery.cpp:182:38
    #20 0x29ee5bff in DB::InterpreterExplainQuery::executeImpl() /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterExplainQuery.cpp:265:21
    #21 0x29ee2e40 in DB::InterpreterExplainQuery::execute() /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterExplainQuery.cpp:73:14
    #22 0x2a7b44a2 in DB::executeQueryImpl(char const*, char const*, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*) /build/obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:389:28
    #23 0x2a7b1cb3 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool) /build/obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:675:30
    #24 0x2b7993b2 in DB::TCPHandler::runImpl() /build/obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:253:24
    #25 0x2b7b649a in DB::TCPHandler::run() /build/obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1217:9
    #26 0x31d9c57e in Poco::Net::TCPServerConnection::start() /build/obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3
    #27 0x31d9d281 in Poco::Net::TCPServerDispatcher::run() /build/obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:114:20
    #28 0x3206b5d5 in Poco::PooledThread::run() /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14
    #29 0x320657ad in Poco::ThreadImpl::runnableEntry(void*) /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27
    #30 0x7ffff7f853e8 in start_thread (/usr/lib/libpthread.so.0+0x93e8)
```

</details>
 2020-09-01 10:04:08 +03:00
Mark Papadakis
ff76790a59
Update AggregatingTransform.cpp 
Replaced redundant call to chunk.getNumRows() with local var.
 2020-08-28 16:08:06 +03:00
alexey-milovidov
65ee7dcb18
Merge pull request #13941 from gervarela/avro_confluent_skip_missing_magic_or_schema_id 
Skip abnormaly small messages in AvroConfluent format
 2020-08-26 19:34:42 +03:00
alexey-milovidov
3c6675f7f4
Update AvroRowInputFormat.cpp 2020-08-26 02:26:27 +03:00
alexey-milovidov
5d4507907d
Update AvroRowInputFormat.cpp 2020-08-26 00:19:51 +03:00
alexey-milovidov
efa5b76605
Update AvroRowInputFormat.cpp 2020-08-26 00:17:39 +03:00
Alexander Kuzmenkov
e3c919ec19
Merge pull request #13847 from hexiaoting/dev_rmcode 
Deprecate ODBCDriver format
 2020-08-24 22:34:52 +03:00
alexey-milovidov
b71ef6db22
Update AvroRowInputFormat.cpp 2020-08-21 18:34:15 +03:00
alexey-milovidov
84db9a3a66
Merge pull request #13906 from tomjiang1987/optimize_error_msg_for_null 
optimize error msg for null value of TabSeparatedRow format
 2020-08-21 16:06:16 +03:00
Gervasio Varela
a6a18b62f9 AvroConfluent + Kafla: Skip malformed messages that do not contain at least the AvroConfluent magic number and the schema id definition. 2020-08-21 11:11:41 +02:00
tao jiang
294aa985e0 fix typo exception name 2020-08-21 08:53:00 +08:00
alexey-milovidov
c5a7b1c456
Update RowInputFormatWithDiagnosticInfo.cpp 2020-08-20 20:34:55 +03:00
alexey-milovidov
911946b517
Update TabSeparatedRowInputFormat.cpp 2020-08-20 20:32:49 +03:00
Nikolai Kochetov
7c0fcb2039
Merge pull request #13611 from ClickHouse/array-join-processor 
Refactor ARRAY JOIN
 2020-08-20 17:56:37 +03:00
tao jiang
189c284597 fix if statement code style check 2020-08-20 18:42:14 +08:00
tao jiang
70ef84ee4b fix code style check 2020-08-20 18:06:41 +08:00
Jiang Tao
613e3f7247 optimize error msg for null value 2020-08-20 11:25:28 +08:00
Mark Papadakis
734f57462b
Update PipelineExecutor.cpp 
Should have used back_edges instead of direct_edges
 2020-08-18 15:09:23 +03:00
hexiaoting
ddbfe0ce21 Deprecate ODBCDriver format 2020-08-18 11:29:40 +08:00
Alexey Milovidov
09846a92cd Fix typos 2020-08-16 15:09:41 +03:00
tavplubix
18ef911fe8
Merge pull request #13672 from zhang2014/refactor/materialize_mysql_database 
ISSUES-4006 split mysql protocol to multiple packets
 2020-08-16 13:55:11 +03:00
alexey-milovidov
c5c79e9408
Merge pull request #13590 from azat/TSV-fix-CR-assertion 
Fix assertion for carriage return in TSV format header
 2020-08-15 11:15:23 +03:00
alexey-milovidov
d95b9f73a4
Update TabSeparatedRowInputFormat.cpp 2020-08-15 11:11:10 +03:00
Nikolai Kochetov
8e631a98ea Refactor ActionsChain. 2020-08-14 21:24:58 +03:00
Nikolai Kochetov
6934974fc5 Lift up ARRAY JOIN step. 2020-08-14 21:24:31 +03:00
Nikolai Kochetov
85d0f1a480 Add ARRAY JOIN to ActionsChain 2020-08-14 21:24:19 +03:00
Nikolai Kochetov
95bbda3667 Specify columns after ARRAY JOIN and JOIN. 
Add ArrayJoinTransform and ArrayJoinStep.
 2020-08-14 21:24:03 +03:00
zhang2014
4199baf69d Merge branch 'master' of github.com:yandex/ClickHouse into refactor/materialize_mysql_database 2020-08-14 13:30:12 +08:00
alexey-milovidov
eeefcd6493
Merge pull request #13337 from ClickHouse/refactor-pipes-3 
Refactor Pipe
 2020-08-14 04:12:22 +03:00
alexey-milovidov
3b05c315e4
Update Pipe.h 2020-08-14 04:10:10 +03:00
zhang2014
96bd3ac34b ISSUES-4006 split mysql protocol 2020-08-13 20:41:36 +08:00
zhang2014
34f4c8972e ISSUES-4006 split replication packets 2020-08-13 20:07:02 +08:00
zhang2014
c76d7e3121 ISSUES-4006 split generic packets 2020-08-13 18:17:13 +08:00
zhang2014
c0d42b764e ISSUES-4006 split protocol text packet 2020-08-13 17:20:45 +08:00
zhang2014
4fcc178deb ISSUES-4006 split mysql protocol 2020-08-13 16:17:33 +08:00
zhang2014
688836cdc4 ISSUES-4006 split msql protocol & fix build 2020-08-13 14:30:29 +08:00
Azat Khuzhin
6097f769c7 Fix assertion for carriage return in the header in TSVWithNames* format. 
TabSeparatedRowInputFormat::readPrefix() tries to check for carriage
return (via checkForCarriageReturn()), however it does the check after
checking the column is exists, reorder the codepath a little bit to
run checkForCarriageReturn() before.
 2020-08-12 11:13:19 +03:00
Azat Khuzhin
72205cd326 Call writePrefix() from ctor in SinkToOutputStream 
This will fix lack of headers for the following formats if there are
zero rows:
- TSVWithNames
- CSVWithNames
- TSVWithNamesAndTypes
 2020-08-11 00:52:35 +03:00
Azat Khuzhin
d5a2e1e61c Call doWritePrefix() if IOutputFormat works as Processor (i.e. clickhouse-local) 
This fixes the *WithNames output format (lack of header before this
patch).
 2020-08-11 00:52:35 +03:00
Azat Khuzhin
ba92213d21 Write header only once in TSVWithNames* 2020-08-11 00:52:35 +03:00
Nikolai Kochetov
9b67cd9faf Merge branch 'master' into refactor-pipes-3 2020-08-10 10:50:17 +03:00
Alexey Milovidov
edd89a8610 Fix half of typos 2020-08-08 03:47:03 +03:00