Commit Graph

131 Commits

Author SHA1 Message Date
Vitaly Baranov
f3d72b9ec1 Skip unknown settings with warnings. 2019-11-17 03:51:58 +03:00
Alexey Milovidov
9ac4e4c7fb Little better #7600 2019-11-05 22:31:07 +03:00
Azat Khuzhin
4dfffdd24a Write current batch for distributed send atomically (using .tmp + rename)
Otherwise the following can happen after reboot:

    2019.11.01 11:46:12.217143 [ 187 ] {} <Error> dist.Distributed.DirectoryMonitor: Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected \n before: S\'^A\0^]\0\0<BE>4^A\0r<87>\0\0<A2><D7>^D^Y\0<F2>{^E<CD>\0\0Hy\0\0<F2>^_^C\0^_&\0\0<FF><D3>\0\0
    <8D><91>\0\0<C0>9\0\0<C0><B0>^A\0^G<AA>\0\0<B5><FE>^A\0<BF><A7>^A\0<9B><CB>^A\0I^R^A\0<B7><AB>^A\0<BC><8F>\0\0˲^B\0Zy\0\0<94><AA>\0\0<98>
    <8F>\0\0\f<A5>\0\0^QN\0\0<E3><C6>\0\0<B1>6^B\0ɳ\0\0W<99>\0\0<B9><A2>\0\0:<BB>\0\0)<B1>\0\0#<8B>\0\0aW\0\0<ED>#\0\0<F1>@\0\0ˀ^B\0<D7><FC>\0\0<DF>, Stack trace:

    0. 0x559e27222e60 StackTrace::StackTrace() /usr/bin/clickhouse
    1. 0x559e27222c45 DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) /usr/bin/clickhouse
    2. 0x559e26de4473 ? /usr/bin/clickhouse
    3. 0x559e272494b5 DB::assertString(char const*, DB::ReadBuffer&) /usr/bin/clickhouse
    4. 0x559e2a5dab45 DB::StorageDistributedDirectoryMonitor::processFilesWithBatching(std::map<unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) /usr/bin/clickhouse
    5. 0x559e2a5db5fa DB::StorageDistributedDirectoryMonitor::processFiles() /usr/bin/clickhouse
    6. 0x559e2a5dba78 DB::StorageDistributedDirectoryMonitor::run() /usr/bin/clickhouse
    7. 0x559e2a5ddbbc ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::StorageDistributedDirectoryMonitor::*)(), DB::StorageDistributedDirectoryMonitor*>(void (DB::StorageDistributedDirectoryMonitor::*&&)(), DB::StorageDistributedDirectoryMonitor*&&)::{lambda()#1}::operator()() const /usr/bin/clickhouse
    8. 0x559e2726b07c ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) /usr/bin/clickhouse
    9. 0x559e2bbc3640 ? /usr/bin/clickhouse
    10. 0x7fbd62b3cfb7 start_thread /lib/x86_64-linux-gnu/libpthread-2.29.so
    11. 0x7fbd62a692ef __clone /lib/x86_64-linux-gnu/libc-2.29.so
     (version 19.17.1.1)

v2: remove fsync, to avoid possible stalls (https://github.com/ClickHouse/ClickHouse/pull/7600#discussion_r342010874)
2019-11-05 00:24:38 +03:00
Alexander Kuzmenkov
050de71ef4
Update DistributedBlockOutputStream.cpp 2019-10-24 12:33:45 +03:00
Azat Khuzhin
ab9d9f8997 Fix INSERT into Distributed non local node with MATERIALIZED columns
Previous patch e527def18a ("Fix INSERT
into Distributed() table with MATERIALIZED column") fixes it only for
cases when the node is local, i.e. direct insert.

This patch address the problem when the node is not local
(`is_local == false`), by erasing materialized columns on INSERT into
Distributed.

And this patch fixes two cases, depends on `insert_distributed_sync`
setting:

- `insert_distributed_sync=0`

    ```
    Not found column value in block. There are only columns: date. Stack trace:

    2. 0x7ffff7be92e0 DB::Exception::Exception() dbms/src/Common/Exception.h:27
    3. 0x7fffec5d6cf6 DB::Block::getByName(...) dbms/src/Core/Block.cpp:187
    4. 0x7fffec2fe067 DB::NativeBlockInputStream::readImpl() dbms/src/DataStreams/NativeBlockInputStream.cpp:159
    5. 0x7fffec2d223f DB::IBlockInputStream::read() dbms/src/DataStreams/IBlockInputStream.cpp:61
    6. 0x7ffff7c6d40d DB::TCPHandler::receiveData() dbms/programs/server/TCPHandler.cpp:971
    7. 0x7ffff7c6cc1d DB::TCPHandler::receivePacket() dbms/programs/server/TCPHandler.cpp:855
    8. 0x7ffff7c6a1ef DB::TCPHandler::readDataNext(unsigned long const&, int const&) dbms/programs/server/TCPHandler.cpp:406
    9. 0x7ffff7c6a41b DB::TCPHandler::readData(DB::Settings const&) dbms/programs/server/TCPHandler.cpp:437
    10. 0x7ffff7c6a5d9 DB::TCPHandler::processInsertQuery(DB::Settings const&) dbms/programs/server/TCPHandler.cpp:464
    11. 0x7ffff7c687b5 DB::TCPHandler::runImpl() dbms/programs/server/TCPHandler.cpp:257
    ```

- `insert_distributed_sync=1`

    ```
    2019.10.18 13:23:22.114578 [ 44 ] {a78f669f-0b08-4337-abf8-d31e958f6d12} <Error> executeQuery: Code: 171, e.displayText() = DB::Exception: Block structure mismatch in RemoteBlockOutputStream stream: different number of columns:
    date Date UInt16(size = 1), value Date UInt16(size = 1)
    date Date UInt16(size = 0): Insertion status:
    Wrote 1 blocks and 0 rows on shard 0 replica 0, 127.0.0.1:59000 (average 0 ms per block)
    Wrote 0 blocks and 0 rows on shard 1 replica 0, 127.0.0.2:59000 (average 2 ms per block)
     (version 19.16.1.1) (from [::1]:3624) (in query: INSERT INTO distributed_00952 VALUES ), Stack trace:

    2. 0x7ffff7be92e0 DB::Exception::Exception() dbms/src/Common/Exception.h:27
    3. 0x7fffec5da4e9 DB::checkBlockStructure<void>(...)::{...}::operator()(...) const dbms/src/Core/Block.cpp:460
    4. 0x7fffec5da671 void DB::checkBlockStructure<void>(...) dbms/src/Core/Block.cpp:467
    5. 0x7fffec5d8d58 DB::assertBlocksHaveEqualStructure(...) dbms/src/Core/Block.cpp:515
    6. 0x7fffec326630 DB::RemoteBlockOutputStream::write(DB::Block const&) dbms/src/DataStreams/RemoteBlockOutputStream.cpp:68
    7. 0x7fffe98bd154 DB::DistributedBlockOutputStream::runWritingJob(DB::DistributedBlockOutputStream::JobReplica&, DB::Block const&)::{lambda()#1}::operator()() const dbms/src/Storages/Distributed/DistributedBlockOutputStream.cpp:280
    <snip>
    ````

Fixes: #7365
Fixes: #5429
Refs: #6891
2019-10-23 21:54:27 +03:00
tavplubix
0046b9f137 Wait for jobs to finish on exception (fixes rare segfaults) (#7350) 2019-10-17 17:41:27 +03:00
Alexander Kuzmenkov
8a1d57bddb Include PODArray.h into fewer files. 2019-10-11 14:28:16 +03:00
Azat Khuzhin
b22b65dd29
Avoid SIGSEGV on batch send failure (file with index XX is absent)
In case of the following error:
  Failed to send batch: file with index 23742 is absent

NULL dereference will occur for the "remote".
2019-09-21 02:22:25 +03:00
Azat Khuzhin
02befdd8c2
Introduce distributed_directory_monitor_max_sleep_time_ms setting
This will replace static 30 seconds max_sleep_time for the
StorageDistributedDirectoryMonitor, since in some cases 30 seconds can
be too large.
2019-09-12 03:16:06 +03:00
Vasily Nemkov
c2fc71b5e5 Post-PR fixes
Renamed settings, updated docs.
2019-09-05 13:35:36 +03:00
Vasily Nemkov
f98c488834 Post-review fixes
* Actually using the replica recovery settings for cluster
* A bit of doc on DBMS_CONNECTION_POOL_WITH_FAILOVER_MAX_ERROR_COUNT
* StorageDistributedDirectoryMonitor using settings for ConnectionPoolWithFailover
* Using SettingSeconds instead of SettingUInt64 for replica_error_decrease_period
2019-09-02 17:26:25 +03:00
Alexey Milovidov
6bbf3bd634 Added a metric for the number of files to send in Distributed tables 2019-08-22 04:30:49 +03:00
kreuzerkrieg
112fc71276 adding -Wshadow for GCC 2019-08-09 23:58:16 +03:00
Alexander Tokmakov
5cf183cbe3 fixes after review 2019-08-07 15:55:27 +03:00
Alexander Tokmakov
aefc664819 refactor throwFromErrno 2019-08-06 21:54:06 +03:00
alesapin
fd3abbe93d
Merge pull request #4935 from zhang2014/feature/support_system_replicas
Support system replicas queries for distributed
2019-06-17 15:30:30 +03:00
Konstantin Podshumok
a61a8386f3 update timeouts usage in DistributedBlockOutputStream 2019-06-02 12:43:20 +03:00
Konstantin Podshumok
b801fcb79a fix timeouts usage in directory monitor 2019-06-02 12:43:20 +03:00
zhang2014
80788cd7a8 Rename system sync distributed to system flush distributed 2019-05-29 10:43:52 +08:00
zhang2014
1a33840964 fix review 2019-05-29 10:43:52 +08:00
zhang2014
c44f608868 support replicas system queries for distributed 2019-05-29 10:43:52 +08:00
tai
b394a79af3 try fix batch async insert settings for Distributed 2019-04-29 17:02:23 +08:00
tai
0d39968233 Suport settings for async(#4852)
There is no hash operation on Settings. Because it is less valuable for hash operations.

And I dont know how to write a testcase.
2019-04-29 17:02:23 +08:00
Alexey Milovidov
571bc7065a Removed unused code related to MemoryTracker 2019-04-06 01:35:56 +03:00
Alexey Milovidov
65f8e4d4a8 Merge branch 'master' into global-thread-pool 2019-01-25 16:33:50 +03:00
Alexey Milovidov
c70e8cc5f0 Miscellaneous #3726 2019-01-21 22:45:26 +03:00
zhang2014
998591b30d fix build failure 2019-01-18 10:31:28 +08:00
alexey-milovidov
3d4bd7cbf8
Merge branch 'master' into feature/use_cluster_pool 2019-01-18 01:02:39 +03:00
Alexey Milovidov
f6b9b06307 Attempt to implemnt global thread pool #4018 2019-01-14 22:22:09 +03:00
Alexey Milovidov
abcd5a2a49 Attempt to implemnt global thread pool #4018 2019-01-11 22:12:36 +03:00
Alexey Milovidov
afda68d25c Added some warnings from clang's -Weverything 2019-01-04 17:18:49 +03:00
Alexey Milovidov
1d4701b0bc Added some warnings from clang's -Weverything 2019-01-04 15:10:00 +03:00
Alexey Milovidov
e33e5150b7 Miscellaneous [#CLICKHOUSE-2] 2019-01-02 09:44:36 +03:00
proller
300bbdfb43 Fix cyclic lib depend ppart 2: clean 2018-12-28 21:15:26 +03:00
alexey-milovidov
4305509236
Update DistributedBlockOutputStream.cpp 2018-12-07 07:15:26 +03:00
alexey-milovidov
e7f4d5a12b
Update DistributedBlockOutputStream.cpp 2018-12-07 07:14:52 +03:00
root
8c9e9649fb Add test and code comment. 2018-12-03 21:11:26 +08:00
zhang2014
994d1c0fe0 fix excessive copy 2018-12-03 13:26:27 +08:00
zhang2014
4798ab1924 Use cluster connections pool in DirectoryMonitor 2018-12-02 01:45:29 +08:00
root
94164ef250 Compatible with LowCardinality type data write 2018-11-28 12:51:04 +08:00
Alexey Milovidov
ab5d76fc42 Fixed error introduced in #3553 and prevent it happening 2018-11-21 23:56:37 +03:00
Marek Vavruša
8fad726d95 Added null guards for all CurrentThread::attachToIfDetached()
refs #3622
2018-11-20 09:10:34 -08:00
Vitaliy Lyudvichenko
e13ba09004 Rewritten ThreadStatus via adding shared thread state. [#CLICKHOUSE-2910]
Fixed race condition in SystemLog.
2018-06-19 23:30:35 +03:00
Vitaliy Lyudvichenko
4cd230f85c Merge remote-tracking branch 'origin/master' into CLICKHOUSE-2910 2018-06-14 16:04:00 +03:00
Alexey Milovidov
d864f39ec7 Merge branch 'master' into zhang2014-feature/support_truncate 2018-06-09 19:10:06 +03:00
Alexey Milovidov
b9b89a5590 Less dependencies [#CLICKHOUSE-2] 2018-06-05 22:46:49 +03:00
Vitaliy Lyudvichenko
e265e5c2ab Refactoring. [#CLICKHOUSE-2910] 2018-05-29 21:14:31 +03:00
Vitaliy Lyudvichenko
aa40931824 Merge remote-tracking branch 'origin/master' into CLICKHOUSE-2910 2018-05-28 20:22:27 +03:00
zhang2014
6613e56784 ISSUES-2259 support truncate syntax 2018-05-21 11:38:46 +08:00
Alexey Milovidov
577268d8f5 Style #2387 2018-05-18 21:31:18 +03:00