Commit Graph

2007 Commits

Author SHA1 Message Date
Nikolai Kochetov
1c106691b5
Merge pull request #16423 from amosbird/jbodread
Balanced reading from JBOD
2020-10-29 19:22:45 +03:00
Maxim Akhmedov
3627fabfb9 Remove -g0 form Arcadia build settings. 2020-10-29 17:37:23 +03:00
alesapin
40fc512e79 Merge branch 'master' into no_background_pool_no_more 2020-10-29 12:53:34 +03:00
Azat Khuzhin
b42f77a791 Covert error codes list to lookup table and drop generation of them. 2020-10-29 10:55:41 +03:00
Azat Khuzhin
b2e2322895 Add system.errors table
Contains error codes with number of times they have been triggered.

Columns:

-   `name` ([String](../../sql-reference/data-types/string.md)) — name of the error (`errorCodeToName`).
-   `code` ([Int32](../../sql-reference/data-types/int-uint.md)) — code number of the error.
-   `value` ([UInt64](../../sql-reference/data-types/int-uint.md)) - number of times this error has been happened.

**Example**

``` sql
SELECT *
FROM system.errors
WHERE value > 0
ORDER BY code ASC
LIMIT 1

┌─name─────────────┬─code─┬─value─┐
│ CANNOT_OPEN_FILE │   76 │     1 │
└──────────────────┴──────┴───────┘
2020-10-29 10:55:38 +03:00
Nikolai Kochetov
671d2b7f1b
Update MergeTreeReadPool.cpp
Fix comment
2020-10-29 10:04:23 +03:00
alexey-milovidov
0faf2bc7e3
Merge pull request #16332 from ClickHouse/fix-queue-processing-of-very-large-entries
Fix processing of very large entries in queue
2020-10-29 09:09:29 +03:00
alexey-milovidov
bf3acecf6f
Merge pull request #16032 from Avogar/mysql_bug
Don't touch MySQL database if it's unnecessary
2020-10-29 04:14:26 +03:00
Nikolai Kochetov
10bad32fb4 Refactor code a little bit. Add comment. 2020-10-29 00:52:26 +03:00
Amos Bird
f995ef9797
Balanced reading from JBOD 2020-10-29 04:05:07 +08:00
alesapin
57c3935a26
Merge pull request #16426 from kssenii/rabbit-optimize
Optimize rabbitmq engine
2020-10-28 22:24:22 +03:00
alesapin
9830ff1003
Merge pull request #16094 from nvartolomei/nv/wal-metadata-compatibility
RFC: Add metadata section to MergeTreeWriteAheadLog
2020-10-28 15:54:46 +03:00
alesapin
8e8bdeb5d7
Merge pull request #16434 from ClickHouse/fix_fake_race_on_merges_list
Fix fake race condition on system.merges merge_algorithm
2020-10-28 15:51:53 +03:00
tavplubix
cd70dc82cd
Merge pull request #16409 from azat/drop-distributed-fix
Fix DROP TABLE for Distributed (racy with INSERT)
2020-10-28 14:10:32 +03:00
alesapin
3c31a5134e Fix bug 2020-10-28 14:03:26 +03:00
tavplubix
4a63f1271b
Merge pull request #16244 from ClickHouse/fix_race_data_parts_exchange_endpoint
Fix race on StorageReplicatedMergeTree::data_parts_exchange_endpoint
2020-10-28 13:25:55 +03:00
alesapin
c10370f98d Merge branch 'master' into fix-queue-processing-of-very-large-entries 2020-10-28 11:19:34 +03:00
alesapin
617e42ddb4
Merge branch 'master' into rabbit-optimize 2020-10-28 10:24:05 +03:00
alesapin
1c9e24b5e2 Merge branch 'master' into system_fetches_table 2020-10-28 10:20:04 +03:00
kssenii
4b59882c30 Remove redundant, add virt col 2020-10-27 20:34:31 +00:00
Alexander Kuzmenkov
6509f1eb39
Merge branch 'master' into minumum 2020-10-27 21:38:23 +03:00
Azat Khuzhin
fc14fde24a Fix DROP TABLE for Distributed (racy with INSERT)
<details>

```

drop() on T1275:

    0  DB::StorageDistributed::drop (this=0x7f9ed34f0000) at ../contrib/libcxx/include/__hash_table:966
    1  0x000000000d557242 in DB::DatabaseOnDisk::dropTable (this=0x7f9fc22706d8, context=..., table_name=...)
       at ../contrib/libcxx/include/new:340
    2  0x000000000d6fcf7c in DB::InterpreterDropQuery::executeToTable (this=this@entry=0x7f9e42560dc0, query=...)
       at ../contrib/libcxx/include/memory:3826
    3  0x000000000d6ff5ee in DB::InterpreterDropQuery::execute (this=0x7f9e42560dc0) at ../src/Interpreters/InterpreterDropQuery.cpp:50
    4  0x000000000daa40c0 in DB::executeQueryImpl (begin=<optimized out>, end=<optimized out>, context=..., internal=<optimized out>,
       stage=DB::QueryProcessingStage::Complete, has_query_tail=false, istr=0x0) at ../src/Interpreters/executeQuery.cpp:420
    5  0x000000000daa59df in DB::executeQuery (query=..., context=..., internal=internal@entry=false, stage=<optimized out>,
       may_have_embedded_data=<optimized out>) at ../contrib/libcxx/include/string:1487
    6  0x000000000e1369e6 in DB::TCPHandler::runImpl (this=this@entry=0x7f9ddf3a9000) at ../src/Server/TCPHandler.cpp:254
    7  0x000000000e1379c9 in DB::TCPHandler::run (this=0x7f9ddf3a9000) at ../src/Server/TCPHandler.cpp:1326
    8  0x000000001086fac7 in Poco::Net::TCPServerConnection::start (this=this@entry=0x7f9ddf3a9000)
       at ../contrib/poco/Net/src/TCPServerConnection.cpp:43
    9  0x000000001086ff2b in Poco::Net::TCPServerDispatcher::run (this=0x7f9e4eba5c00)
       at ../contrib/poco/Net/src/TCPServerDispatcher.cpp:114
    10 0x00000000109dbe8e in Poco::PooledThread::run (this=0x7f9e4a2d2f80) at ../contrib/poco/Foundation/src/ThreadPool.cpp:199
    11 0x00000000109d78f9 in Poco::ThreadImpl::runnableEntry (pThread=<optimized out>)
       at ../contrib/poco/Foundation/include/Poco/SharedPtr.h:401
    12 0x00007f9fc3cccea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
    13 0x00007f9fc3bebeaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

StorageDistributedDirectoryMonitor on T166:

    0  DB::StorageDistributedDirectoryMonitor::StorageDistributedDirectoryMonitor (this=0x7f9ea7ab1400, storage_=..., path_=...,
       pool_=..., monitor_blocker_=..., bg_pool_=...) at ../src/Storages/Distributed/DirectoryMonitor.cpp:81
    1  0x000000000dbf684e in std::__1::make_unique<> () at ../contrib/libcxx/include/memory:3474
    2  DB::StorageDistributed::requireDirectoryMonitor (this=0x7f9ed34f0000, disk=..., name=...)
       at ../src/Storages/StorageDistributed.cpp:682
    3  0x000000000de3d5fa in DB::DistributedBlockOutputStream::writeToShard (this=this@entry=0x7f9ed39c7418, block=..., dir_names=...)
       at ../src/Storages/Distributed/DistributedBlockOutputStream.cpp:634
    4  0x000000000de3e214 in DB::DistributedBlockOutputStream::writeAsyncImpl (this=this@entry=0x7f9ed39c7418, block=...,
       shard_id=shard_id@entry=79) at ../src/Storages/Distributed/DistributedBlockOutputStream.cpp:539
    5  0x000000000de3e47b in DB::DistributedBlockOutputStream::writeSplitAsync (this=this@entry=0x7f9ed39c7418, block=...)
       at ../contrib/libcxx/include/vector:1546
    6  0x000000000de3eab0 in DB::DistributedBlockOutputStream::writeAsync (block=..., this=0x7f9ed39c7418)
       at ../src/Storages/Distributed/DistributedBlockOutputStream.cpp:141
    7  DB::DistributedBlockOutputStream::write (this=0x7f9ed39c7418, block=...)
       at ../src/Storages/Distributed/DistributedBlockOutputStream.cpp:135
    8  0x000000000d73b376 in DB::PushingToViewsBlockOutputStream::write (this=this@entry=0x7f9ea7a8cf58, block=...)
       at ../src/DataStreams/PushingToViewsBlockOutputStream.cpp:157
    9  0x000000000d7853eb in DB::AddingDefaultBlockOutputStream::write (this=0x7f9ed383d118, block=...)
       at ../contrib/libcxx/include/memory:3826
    10 0x000000000d740790 in DB::SquashingBlockOutputStream::write (this=0x7f9ed383de18, block=...)
       at ../contrib/libcxx/include/memory:3826
    11 0x000000000d68c308 in DB::CountingBlockOutputStream::write (this=0x7f9ea7ac6d60, block=...)
       at ../contrib/libcxx/include/memory:3826
    12 0x000000000ddab449 in DB::StorageBuffer::writeBlockToDestination (this=this@entry=0x7f9fbd56a000, block=..., table=...)
       at ../src/Storages/StorageBuffer.cpp:747
    13 0x000000000ddabfa6 in DB::StorageBuffer::flushBuffer (this=this@entry=0x7f9fbd56a000, buffer=...,
       check_thresholds=check_thresholds@entry=true, locked=locked@entry=false, reset_block_structure=reset_block_structure@entry=false)
       at ../src/Storages/StorageBuffer.cpp:661
    14 0x000000000ddac415 in DB::StorageBuffer::flushAllBuffers (reset_blocks_structure=false, check_thresholds=true, this=0x7f9fbd56a000)
        at ../src/Storages/StorageBuffer.cpp:605

shutdown() on T1275:

    0  DB::StorageDistributed::shutdown (this=0x7f9ed34f0000) at ../contrib/libcxx/include/atomic:1612
    1  0x000000000d6fd938 in DB::InterpreterDropQuery::executeToTable (this=this@entry=0x7f98530c79a0, query=...)
       at ../src/Storages/TableLockHolder.h:12
    2  0x000000000d6ff5ee in DB::InterpreterDropQuery::execute (this=0x7f98530c79a0) at ../src/Interpreters/InterpreterDropQuery.cpp:50
    3  0x000000000daa40c0 in DB::executeQueryImpl (begin=<optimized out>, end=<optimized out>, context=..., internal=<optimized out>,
       stage=DB::QueryProcessingStage::Complete, has_query_tail=false, istr=0x0) at ../src/Interpreters/executeQuery.cpp:420
    4  0x000000000daa59df in DB::executeQuery (query=..., context=..., internal=internal@entry=false, stage=<optimized out>,
       may_have_embedded_data=<optimized out>) at ../contrib/libcxx/include/string:1487
    5  0x000000000e1369e6 in DB::TCPHandler::runImpl (this=this@entry=0x7f9ddf3a9000) at ../src/Server/TCPHandler.cpp:254
    6  0x000000000e1379c9 in DB::TCPHandler::run (this=0x7f9ddf3a9000) at ../src/Server/TCPHandler.cpp:1326
    7  0x000000001086fac7 in Poco::Net::TCPServerConnection::start (this=this@entry=0x7f9ddf3a9000)
       at ../contrib/poco/Net/src/TCPServerConnection.cpp:43
    8  0x000000001086ff2b in Poco::Net::TCPServerDispatcher::run (this=0x7f9e4eba5c00)
       at ../contrib/poco/Net/src/TCPServerDispatcher.cpp:114
    9  0x00000000109dbe8e in Poco::PooledThread::run (this=0x7f9e4a2d2f80) at ../contrib/poco/Foundation/src/ThreadPool.cpp:199
    10 0x00000000109d78f9 in Poco::ThreadImpl::runnableEntry (pThread=<optimized out>)
       at ../contrib/poco/Foundation/include/Poco/SharedPtr.h:401
    11 0x00007f9fc3cccea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
    12 0x00007f9fc3bebeaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

```

</details>
2020-10-27 21:19:36 +03:00
alesapin
3ec5fc7a4c Add explicit 2020-10-27 18:29:06 +03:00
alesapin
60f2d822d7 Fix fake race condition on system.merges merge_algorithm 2020-10-27 18:27:12 +03:00
Alexander Kuzmenkov
60544ad46e
Merge pull request #16373 from ClickHouse/better-rethrow
Better exceptions rethrow
2020-10-27 16:38:32 +03:00
alesapin
6253d1466a Fix comments 2020-10-27 16:09:14 +03:00
alesapin
3d3d246bdc Rename table 2020-10-27 16:07:50 +03:00
alesapin
e40c509392 More clear callback name 2020-10-27 16:00:40 +03:00
alesapin
f35edfb2a5 Remove some unused fields 2020-10-27 15:57:55 +03:00
alesapin
ee4e6caf70 Remove redundant metric increment 2020-10-27 15:52:49 +03:00
alesapin
3824fcb076 Rename file and some tweaks 2020-10-27 15:47:42 +03:00
alesapin
31d63efdac Fix style 2020-10-27 15:26:10 +03:00
alesapin
dca35b1913 Add awesome test for fetch 2020-10-27 15:24:10 +03:00
kssenii
a48bc83763 Fix style 2020-10-27 12:04:07 +00:00
Pavel Kruglov
331545fe0d Fix typo 2020-10-27 14:29:36 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell 2020-10-27 12:04:03 +01:00
kssenii
db03cd7dd7 Much more optimal queues setup 2020-10-27 11:00:01 +00:00
kssenii
7beddceadd Fix bug that caused awful CPU usage 2020-10-27 10:02:27 +00:00
kssenii
cd0dbcb9d8 Minimize event loop lifetime 2020-10-27 10:01:39 +00:00
alesapin
d6130f13ef Add table to system tables 2020-10-27 10:21:49 +03:00
alesapin
880f4bbd05 System fetches 2020-10-26 19:38:35 +03:00
Alexander Kuzmenkov
5a70e26c70 Merge remote-tracking branch 'origin/master' into HEAD 2020-10-26 19:21:13 +03:00
alesapin
9ed4668dbb Refactor common part of background list 2020-10-26 15:40:55 +03:00
alesapin
d5d0819698 Merge branch 'master' into no_background_pool_no_more 2020-10-26 12:58:51 +03:00
alexey-milovidov
3aa49204d6
Update MergeTreeSettings.cpp 2020-10-26 07:39:20 +03:00
Alexey Milovidov
d5e277f080 Better exceptions rethrow 2020-10-26 06:35:57 +03:00
Alexey Milovidov
22a89fca01 Fix processing very large entries in queue 2020-10-25 05:57:03 +03:00
Alexey Milovidov
a2d888f19f Fix processing very large entries in queue 2020-10-25 05:57:03 +03:00
Ivan
1d170f5745
ASTTableIdentifier Part #1: improve internal representation of ASTIdentifier name (#16149)
* Use only |name_parts| as primary name source

* Restore legacy logic for table restoration

* Fix build

* Fix tests

* Add pytest server config

* Fix tests

* Fixes due to review
2020-10-24 21:46:10 +03:00
alexey-milovidov
c579611420
Merge pull request #16306 from azat/Set-Join-total-rows-bytes
Add total_rows/total_bytes support for Set/Join
2020-10-24 07:26:32 +03:00
alexey-milovidov
6037982343
Merge pull request #16284 from ClickHouse/primary_key_without_order_by
Add ability to specify PRIMARY KEY without ORDER BY
2020-10-24 06:05:04 +03:00
Alexey Milovidov
13b8dce792 Update contributors 2020-10-24 00:26:22 +03:00
Azat Khuzhin
4d1385ef19 Add total_rows/total_bytes support for Set/Join 2020-10-23 21:11:55 +03:00
alesapin
bc331a8029 Better set 2020-10-23 19:34:41 +03:00
alesapin
b28894c43b More compatible way 2020-10-23 19:31:51 +03:00
alesapin
59ca0d071c Fix comment 2020-10-23 14:17:37 +03:00
alesapin
69f2f9838e Add ability to specify PRIMARY KEY without ORDER BY 2020-10-23 14:16:27 +03:00
alesapin
39e47c5338 Less strange lambdas 2020-10-23 11:54:00 +03:00
Alexander Kuzmenkov
5cbf645de4 Merge remote-tracking branch 'origin/master' into tmp 2020-10-22 19:48:02 +03:00
alesapin
3c559c699f Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 17:16:46 +03:00
alesapin
1d07ece5d1 Fix clang tidy warning 2020-10-22 17:15:56 +03:00
alesapin
7a63b373f9 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 15:47:11 +03:00
alesapin
4b65d5469f Fix some unrelated performance issues in select parts for merge 2020-10-22 15:41:01 +03:00
alesapin
945dd3aa59 Merge branch 'master' into blind_fix_of_read_in_order 2020-10-22 10:37:37 +03:00
alexey-milovidov
adeba6bdd8
Merge pull request #15074 from amosbird/btc
Extend trivial count optimization.
2020-10-22 02:50:57 +03:00
Alexander Tokmakov
ac42199ae5 fix race on StorageReplicatedMergeTree::data_parts_exchange_endpoint 2020-10-21 22:25:22 +03:00
Kruglov Pavel
0b75885c8a
Merge branch 'master' into select_final 2020-10-21 20:42:39 +03:00
Pavel Kruglov
89fdeb4e15 Fix style, move setting and add checking level>0 2020-10-21 20:35:31 +03:00
Nikolai Kochetov
a0da588f06 Merge branch 'master' into new-block-for-functions-4 2020-10-21 16:16:58 +03:00
alexey-milovidov
0b7430dda1
Merge pull request #16206 from ClickHouse/fix-memory-tracking
Fix multiple issues with memory tracking
2020-10-21 14:34:22 +03:00
alesapin
c53f59dece
Merge pull request #15984 from ClickHouse/fix_insert_mv_dangling_ref
Fix possibly dangling reference to target table of MV
2020-10-21 12:13:51 +03:00
alesapin
11ea6d7911 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-21 11:39:48 +03:00
alesapin
b45d42bb9c Maybe devirtualization can help 2020-10-21 11:35:36 +03:00
alesapin
9f47c82bbd Sleep a little when pool is full 2020-10-21 11:31:57 +03:00
alesapin
b1a13d4884 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-21 10:30:09 +03:00
Nikolai Kochetov
68c9a57788 Merge branch 'master' into new-block-for-functions-4 2020-10-21 09:17:25 +03:00
Alexey Milovidov
2619efadc8 Fix multiple issues with memory tracking 2020-10-21 03:31:34 +03:00
alexey-milovidov
8084ce75cb
Merge pull request #16105 from azat/allow_nondeterministic_optimize_skip_unused_shards
Add allow_nondeterministic_optimize_skip_unused_shards
2020-10-21 00:16:26 +03:00
alesapin
c77b154398 Better locks in merge tree 2020-10-21 00:10:55 +03:00
alesapin
8097c696de Trying another fix 2020-10-20 23:05:20 +03:00
alexey-milovidov
75a02112bc
Merge pull request #15592 from sundy-li/readonly-metrics
sub ReadonlyReplica metric when detach readonly tables
2020-10-20 19:17:52 +03:00
alesapin
439bbd9861 Blind performance fix 2020-10-20 18:30:46 +03:00
Vladimir Chebotarev
aa5f207fd4
Added disable_merges option for volumes in multi-disk configuration (#13956)
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-10-20 18:10:24 +03:00
Nikolai Kochetov
a1d5adf8d5 Merge branch 'master' into new-block-for-functions-4 2020-10-20 16:16:08 +03:00
alesapin
30ca17b09b Comment to settings 2020-10-20 16:12:25 +03:00
alesapin
7256d01579 Normal backoff value instead of 10 seconds 2020-10-20 16:11:01 +03:00
Alexander Kuzmenkov
84908df6d8 fix setting parent from HTTP traceparent 2020-10-20 14:35:13 +03:00
alesapin
ce8ee01d75 Fix background moves start 2020-10-20 14:27:50 +03:00
alesapin
d126aa8a09 Merge branch 'master' into no_background_pool_no_more 2020-10-20 12:54:42 +03:00
alexey-milovidov
5e46e7c670
Merge pull request #15723 from Jokser/idempotent-part-move
Possibility to move part to another disk/volume if first attempt was failed
2020-10-20 04:26:50 +03:00
alexey-milovidov
124379cccc
Update IMergeTreeDataPart.h 2020-10-20 04:24:30 +03:00
alexey-milovidov
26517ff08d
Update IMergeTreeDataPart.h 2020-10-20 04:23:23 +03:00
Nikolai Kochetov
bc58637ec2 Fixing build. 2020-10-19 21:37:44 +03:00
Alexander Kuzmenkov
6dcfd26b29 Merge remote-tracking branch 'origin/master' into tmp 2020-10-19 21:36:30 +03:00
Alexander Kuzmenkov
defcd29b31
Merge pull request #16036 from ClickHouse/stress_test_improvemets
Minor improvements of stress test
2020-10-19 16:12:30 +03:00
Nicolae Vartolomei
af76ff1dbf RFC: Add metadata section to MergeTreeWriteAheadLog
Current WAL format doesn't seem to be extendable without breaking
reverse compatibility.

This PR proposes to introduces a metadata section
for each action in the WAL to which optional fields can be added.

Adding new fields is trivial and doesn't have to break backwards
compatibility if these are not critical for correctness.

Metadata contains the minimum version of the reader allowed
to interpret it.

Removing fields is not supported.
2020-10-19 12:16:31 +01:00
alesapin
514a95c292 Fixup 2020-10-19 12:06:51 +03:00
alesapin
cb139abe2a Better schedule after job already running 2020-10-19 11:47:15 +03:00
alesapin
e09713ece1 Merge branch 'master' into no_background_pool_no_more 2020-10-19 10:58:31 +03:00
alesapin
963bc57835
Merge pull request #16011 from ClickHouse/fix_alter_add_and_modify_order_by
Fix alter sorting key for ReplicatedVersionedCollapsingMergeTree
2020-10-19 10:31:13 +03:00