thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-21 15:12:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
58,304 Commits 9,143 Branches 1,778 Tags 2.1 GiB
d305b23338
Commit Graph

58304 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
tavplubix
d305b23338
Merge pull request #21079 from azat/distributed_ddl_pool_size-zk-fix 
Fix various issues in DDLWorker (SIGSEGV and others)
 2021-03-02 11:59:30 +03:00
olgarev
15b3f379a5
DOCSUP-4915: documented geo data types (#21294) 
* Initial

* Fix links

* Fix master

* Mistakes corrected

* Minor fix

* Fixes and intro.

* Update geo.md

* Apply suggestions from code review

Co-authored-by: Anton Popov <pad11rus@gmail.com>

Co-authored-by: Olga Revyakina <revolg@yandex-team.ru>
Co-authored-by: Anton Popov <pad11rus@gmail.com>
 2021-03-01 19:41:16 +03:00
Maksim Kita
ca98bb68d4
Merge pull request #21353 from kitaisreal/aggregate-function-sum-map-better-comment 
AggregateFunctionSumMap better comment message
 2021-03-01 19:19:07 +03:00
Maksim Kita
7ed5900251
Updated style 2021-03-01 19:18:56 +03:00
Maksim Kita
019a2090c7
Fix unused variable 2021-03-01 19:18:14 +03:00
Maksim Kita
3764a2a2be AggregateFunctionSumMap better comment message 2021-03-01 19:15:59 +03:00
Alexander Kuzmenkov
977b98ad69
Merge pull request #21094 from den-crane/tests/window_functions 
test for window functions
 2021-03-01 19:00:45 +03:00
tavplubix
10aa26576c
Update DatabaseReplicatedWorker.cpp 2021-03-01 17:11:45 +03:00
Roman Bug
a4f2ee0752
DOCSUP-6144: Edit and translate PR to Russian (#19780) 
Co-authored-by: Anna <42538400+adevyatova@users.noreply.github.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>
Co-authored-by: Vladimir <vdimir@yandex-team.ru>
 2021-03-01 16:51:38 +03:00
Nikita Mikhaylov
cb2a26308f
Merge pull request #20347 from gyuton/gyuton-DOCSUP-6549-Edit_and_translate 
DOCSUP-6549: Edited and translated to Russian
 2021-03-01 16:49:06 +03:00
Alexander Kuzmenkov
366fba4b04
Update 01592_long_window_functions1.sql 2021-03-01 16:06:44 +03:00
Anton Popov
c98e6bb1ad
Merge pull request #20550 from ClickHouse/fix-number-of-threads-in-subqueries 
Fix number of threads in subqueries
 2021-03-01 15:49:38 +03:00
Nikolai Kochetov
976dbe8077
Merge pull request #20341 from ClickHouse/filter-push-down 
Filter push down
 2021-03-01 12:35:06 +03:00
alesapin
11f2a271a2 Remove useless unit test 2021-03-01 10:40:00 +03:00
alexey-milovidov
174f49a9f1
Merge pull request #21308 from fuwhu/refine-wal-restore-code 
remove unused code in MergeTreeWriteAheadLog::restore
 2021-03-01 01:44:23 +03:00
alexey-milovidov
9b633ac564
Update ontime.md 2021-02-28 23:27:01 +03:00
alexey-milovidov
81be2d1ed9
Merge pull request #21305 from azat/terminate-nested-writers 
Fix abnormal server termination for nested writers
 2021-02-28 21:54:13 +03:00
alexey-milovidov
3bd180c416
Merge pull request #20149 from keen-wolf/timezone_about 
fix toMinute function to handle special timezone
 2021-02-28 21:53:51 +03:00
alexey-milovidov
b924b28c57
Merge pull request #21306 from azat/unused-writers 
[RFC] Remove unused writers
 2021-02-28 21:47:01 +03:00
Vladimir
6aa72bcd43
Merge pull request #19591 from ka1bi4/romanzhukov-DOCSUP-5266-PR-translate 2021-02-28 20:40:04 +03:00
fuwhu
2df33be7c2 Remove unused code in MergeTreeWriteAheadLog::restore 2021-02-28 23:17:31 +08:00
alexey-milovidov
467fcbec06
Merge pull request #21292 from ClickHouse/extract-text-from-html 
Rewrite extractTextFromHTML function
 2021-02-28 16:36:18 +03:00
Azat Khuzhin
eb0387c5a9 Fix abnormal server termination for nested writers 
Writers with nested writer can call next() from the dtor for nested
writer and this will cause exception again, so the buffer position
should be updated on exceptions.

Found by stress test (thread) here [1] and here [2]:

    2021.02.27 19:27:53.498977 [ 302 ] {} <Fatal> BaseDaemon: (version 21.3.1.6130, build id: 2DAEC5DEBF03C5A1C3BF66B7779C886F16239345) (from thread 1614) Terminate called for uncaught exception:
    Code: 24, e.displayText() = DB::Exception: Cannot write to ostream at offset 2097498, Stack trace (when copying this message, always include the lines below):

    0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:0: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x15bef2ab in /usr/bin/clickhouse
    1. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8aea92e in /usr/bin/clickhouse
    2. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:0: DB::WriteBufferFromOStream::nextImpl() @ 0x8bbbc45 in /usr/bin/clickhouse
    3. ./obj-x86_64-linux-gnu/../src/IO/BufferBase.h:39: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8bbc077 in /usr/bin/clickhouse
    4. ./obj-x86_64-linux-gnu/../src/IO/WriteBufferFromOStream.cpp:44: DB::WriteBufferFromOStream::~WriteBufferFromOStream() @ 0x8bbc18a in /usr/bin/clickhouse
    5. ./obj-x86_64-linux-gnu/../src/IO/BufferWithOwnMemory.h:137: DB::ZstdDeflatingWriteBuffer::~ZstdDeflatingWriteBuffer() @ 0x118bdc29 in /usr/bin/clickhouse
    6. ./obj-x86_64-linux-gnu/../src/IO/ZstdDeflatingWriteBuffer.cpp:32: DB::ZstdDeflatingWriteBuffer::~ZstdDeflatingWriteBuffer() @ 0x118be3ea in /usr/bin/clickhouse
    7. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:0: DB::WriteBufferFromHTTPServerResponse::finalize() @ 0x12f1dceb in /usr/bin/clickhouse
    8. ./obj-x86_64-linux-gnu/../src/Server/HTTPHandler.cpp:703: DB::HTTPHandler::trySendExceptionToClient(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x12e9fecc in /usr/bin/clickhouse
    9. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/string:1444: DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x12ea0d60 in /usr/bin/clickhouse
    10. ./obj-x86_64-linux-gnu/../src/Server/HTTP/HTTPServerConnection.cpp:0: DB::HTTPServerConnection::run() @ 0x12f16db1 in /usr/bin/clickhouse
    11. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x15b184f3 in /usr/bin/clickhouse
    12. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:0: Poco::Net::TCPServerDispatcher::run() @ 0x15b18c1f in /usr/bin/clickhouse
    13. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/ScopedLock.h:36: Poco::PooledThread::run() @ 0x15c7fdb2 in /usr/bin/clickhouse
    14. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x15c7e350 in /usr/bin/clickhouse
    15. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/SharedPtr.h:277: Poco::ThreadImpl::runnableEntry(void*) @ 0x15c7cb58 in /usr/bin/clickhouse
    16. __tsan_thread_start_func @ 0x8a04ced in /usr/bin/clickhouse
    17. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    18. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
     (version 21.3.1.6130)

  [1]: https://clickhouse-test-reports.s3.yandex.net/21279/4f61ef3099f42f17b496a0b0424773978d9a32dc/stress_test_(thread).html#fail1
  [2]: https://clickhouse-test-reports.s3.yandex.net/21292/ae9fea1d0af118a8f87b224d194d61da1567188b/stress_test_(thread).html#fail1

v2: https://clickhouse-test-reports.s3.yandex.net/21305/e969daa6e86c5e09cfef08cfde19712982b64e59/stress_test_(thread).html#fail1
 2021-02-28 15:13:42 +03:00
keenwolf
303c389b36 change the time_t to unsigned to handle properly the start date of 1970-01-01 2021-02-28 18:11:54 +08:00
Azat Khuzhin
568a49dad5 Remove unused AsynchronousWriteBuffer 2021-02-28 11:19:56 +03:00
Azat Khuzhin
8b58dba094 Drop unused HexWriteBuffer 2021-02-28 10:52:09 +03:00
Azat Khuzhin
d42d4cfd6b Improve test_distributed_ddl_parallel to cover more cases 
Refs: #21264
 2021-02-28 05:45:40 +03:00
Azat Khuzhin
7d51ae3212 DDLWorker: Fix processing of tasks in parallel (correct queue begin) 
Otherwise it will:
- start from incorrect task and bail because finished node already
  exists
- and also process alreayd processed items
 2021-02-28 05:45:12 +03:00
Azat Khuzhin
0e68fc67aa Add other distributed_ddl settings into config.xml as an example 2021-02-28 05:45:12 +03:00
Azat Khuzhin
955974a8b5 DDLWorker: avoid NULL dereference on termination and failed zookeeper initialization 
Log snipped shows the problem:

    2021.02.24 04:40:29.349181 [ 39 ] {} <Warning> DDLWorker: DDLWorker is configured to use multiple threads. It's not recommended because queries can be reordered. Also it may cause some unknown issues to appear.
    2021.02.24 04:40:29.349516 [ 39 ] {} <Information> Application: Ready for connections.
    2021.02.24 04:40:29.349602 [ 74 ] {} <Debug> DDLWorker: Started DDLWorker cleanup thread
    2021.02.24 04:40:29.349639 [ 73 ] {} <Debug> DDLWorker: Starting DDLWorker thread
    2021.02.24 04:40:29.349698 [ 73 ] {} <Debug> DDLWorker: Started DDLWorker thread
    2021.02.24 04:40:29.352548 [ 73 ] {} <Error> virtual void DB::DDLWorker::initializeMainThread(): Code: 999, e.displayText() = Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: 192.168.112.3:2181
    Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 21.3.1.1), 192.168.112.3:2181
    Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 21.3.1.1), 192.168.112.3:2181
    Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 21.3.1.1), 192.168.112.3:2181
     (Connection loss), Stack trace (when copying this message, always include the lines below):

    0. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Coordination::Error, int) @ 0xfe93923 in /usr/bin/clickhouse
    1. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Coordination::Error) @ 0xfe93ba2 in /usr/bin/clickhouse
    2. Coordination::ZooKeeper::connect(std::__1::vector<Coordination::ZooKeeper::Node, std::__1::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0xfed3a01 in /usr/bin/clickhouse
    3. Coordination::ZooKeeper::ZooKeeper(std::__1::vector<Coordination::ZooKeeper::Node, std::__1::allocator<Coordination::ZooKeeper::Node> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Timespan, Poco::Timespan, Poco::Timespan) @ 0xfed2222 in /usr/bin/clickhouse
    4. zkutil::ZooKeeper::init(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xfe961cd in /usr/bin/clickhouse
    5. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xfe97a96 in /usr/bin/clickhouse
    6. void std::__1::allocator_traits<std::__1::allocator<zkutil::ZooKeeper> >::__construct<zkutil::ZooKeeper, Poco::Util::AbstractConfiguration const&, char const (&) [10]>(std::__1::integral_constant<bool, true>, std::__1::allocator<zkutil::ZooKeeper>&, zkutil::ZooKeeper*, Poco::Util::AbstractConfiguration const&, char const (&) [10]) @ 0xed98387 in /usr/bin/clickhouse
    7. DB::Context::getZooKeeper() const @ 0xed75190 in /usr/bin/clickhouse
    8. DB::DDLWorker::getAndSetZooKeeper() @ 0xedb81c9 in /usr/bin/clickhouse
    9. DB::DDLWorker::initializeMainThread() @ 0xedc9eb0 in /usr/bin/clickhouse
    10. DB::DDLWorker::runMainThread() @ 0xedb5d01 in /usr/bin/clickhouse
    11. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'()::operator()() @ 0xedcafa1 in /usr/bin/clickhouse
    12. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x892651f in /usr/bin/clickhouse
    13. ? @ 0x8929fb3 in /usr/bin/clickhouse
    14. start_thread @ 0x8ea7 in /lib/x86_64-linux-gnu/libpthread-2.31.so
    15. __clone @ 0xfddef in /lib/x86_64-linux-gnu/libc-2.31.so
     (version 21.3.1.1)
    ...
    2021.02.24 04:40:30.025278 [ 41 ] {} <Trace> BaseDaemon: Received signal 15
    2021.02.24 04:40:30.025336 [ 41 ] {} <Information> Application: Received termination signal (Terminated)
    ...
    2021.02.24 04:40:30.582078 [ 39 ] {} <Information> Application: Closed all listening sockets.
    2021.02.24 04:40:30.582124 [ 39 ] {} <Information> Application: Closed connections.
    2021.02.24 04:40:30.583770 [ 39 ] {} <Information> Application: Shutting down storages.
    2021.02.24 04:40:30.583932 [ 39 ] {} <Information> Context: Shutdown disk data
    2021.02.24 04:40:30.583951 [ 39 ] {} <Information> Context: Shutdown disk default
    2021.02.24 04:40:30.584163 [ 46 ] {} <Trace> SystemLog (system.query_log): Terminating
    2021.02.24 04:40:30.586025 [ 39 ] {} <Trace> BackgroundSchedulePool/BgSchPool: Waiting for threads to finish.
    2021.02.24 04:40:34.352701 [ 73 ] {} <Debug> DDLWorker: Initialized DDLWorker thread
    2021.02.24 04:40:34.352758 [ 73 ] {} <Debug> DDLWorker: Scheduling tasks
 2021-02-28 05:45:12 +03:00
Azat Khuzhin
dfd1c73b31 Load dictionaries before starting accepting connections and DDLWorker 2021-02-28 05:45:12 +03:00
Azat Khuzhin
3c7e765b27 DDLWorker: process unfinished tasks only after reinitialization 2021-02-28 05:45:12 +03:00
Azat Khuzhin
54b1496408 DDLWorker: optimize processing of unfinished tasks 
Do not look at zookeeper if was_executed is false
 2021-02-28 05:45:12 +03:00
Azat Khuzhin
eeda1fe756 DDLWorker: wait for pending async tasks 2021-02-28 05:45:12 +03:00
Azat Khuzhin
0cd67ed051 DDLWorker: remove only completed tasks (significant for distributed_ddl.pool_size > 1) 
Otherwise it will SIGSEGV due to invalid-read (UAF).
 2021-02-28 05:45:12 +03:00
Azat Khuzhin
7481b05d27 DDLWorker: avoid comparing pointers for getting min element 
The current_tasks always updated with emplace_back, so front() is ok.
 2021-02-28 05:45:11 +03:00
Azat Khuzhin
a795db57f1 tests: decrease distributed_ddl_task_timeout to 60 in test_distributed_ddl_parallel 2021-02-28 05:45:11 +03:00
Azat Khuzhin
06e8bb955d tests: catch exceptions in threads in test_distributed_ddl_parallel 
Otherwise the test will not report an error.
 2021-02-28 05:45:11 +03:00
Azat Khuzhin
b899d8d919 tests: regression for test for invalid reference to zookeeper for distributed_ddl.pool_size > 1 2021-02-28 05:45:11 +03:00
Azat Khuzhin
78c6e0527d DDLWorker: Fix reference to zookeeper with distributed_ddl.pool_size > 1 (thread pool) 2021-02-28 05:45:11 +03:00
Azat Khuzhin
f7dbcfc12a DDLTask: Use std::uncaught_exceptions() over deprecated std::uncaught_exception() 2021-02-28 05:45:11 +03:00
Alexey Milovidov
220a494fa4 Fix idiotic syntax highlight in docs #18432 2021-02-28 04:42:45 +03:00
Alexey Milovidov
ae9fea1d0a Fix gcc and clang-tidy 2021-02-28 04:05:04 +03:00
Alexey Milovidov
107d8ec811 Merge branch 'master' into extract-text-from-html 2021-02-28 04:03:32 +03:00
Alexey Milovidov
e5ae9cbb63 Fix Arcadia 2021-02-28 04:03:22 +03:00
alexey-milovidov
9f0400f015
Merge pull request #21171 from azat/tests-bump-zookeeper 
Bump zookeeper version to 3.6.2 in tests
 2021-02-28 02:35:58 +03:00
Alexey Milovidov
01ef06a42c Fix broken links 2021-02-28 02:31:47 +03:00
alexey-milovidov
528f1d9a39
Merge pull request #21266 from Sin4wd/patch-1 
Fix broken links in "See Also"
 2021-02-28 02:29:18 +03:00
alexey-milovidov
b8fba768e5
Merge pull request #21264 from ClickHouse/fix_zookeeper_update 
Fix several bugs with ZooKeeper client
 2021-02-28 01:57:04 +03:00
alexey-milovidov
470bc2b01b
Merge pull request #21279 from kssenii/add-tests 
Add tests for zstd and zlib http compression
 2021-02-28 01:33:21 +03:00