thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 19:32:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
76,490 Commits 9,319 Branches 1,783 Tags 2.1 GiB
4cfb105d5a
Commit Graph

2628 Commits

Author SHA1 Message Date
Amos Bird
f2374a6916
Better nullable primary key implementation. 2021-08-28 17:48:28 +08:00
Amos Bird
b68857d086
Simplify projection, add minmax_count projection. 2021-08-28 11:25:37 +08:00
Alexey Milovidov
79e0433ba7 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-28 01:19:16 +03:00
Amos Bird
ce96b2c35a
bump 2021-08-27 20:44:26 +08:00
tavplubix
703101fe4d
Merge pull request #27931 from ClickHouse/wait_for_all_replicas_timeouts 
Avoid too long waiting for inactive replicas
 2021-08-27 14:31:36 +03:00
Azat Khuzhin
19458a0c61 Fix removing of parts in a Temporary state 
Parts in a temporary state is not exists in data_parts_by_info, so do
not try to search there and throw LOGICAL_ERROR in case of failure:

<details>

```
    02:45:49.037546 [ 5890 ] {} <Error> test_iy9rta.concurrent_kill_4 (526aa7c8-db2a-4f0e-926a-a7c8db2a9f0e): Code: 40. DB::Exception: Part all_0_0_0_1 from r11 has different columns hash. (CHECKSUM_DOESNT_MATCH) (version 21.10.1.7910 (official build)). Data after mutation is not byte-identical to data on another replicas. We will download merged part from replica to force byte-identical result.
    02:45:49.049422 [ 5890 ] {} <Trace> test_iy9rta.concurrent_kill_4 (526aa7c8-db2a-4f0e-926a-a7c8db2a9f0e): Trying to immediately remove part all_0_0_0_1 (state Temporary)
    02:45:49.060210 [ 5890 ] {} <Fatal> : Logical error: 'Part all_0_0_0_1 doesn't exist'.
    02:47:01.572508 [ 29208 ] {} <Fatal> BaseDaemon: (version 21.10.1.7910 (official build), build id: 9309CECED9A0D32CBB00BB8EC77B91456679868E) (from thread 5890) (no query) Received signal Aborted (6)
    ...
    02:47:06.815000 [ 29208 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeData.cpp:2593: DB::MergeTreeData::tryRemovePartImmediately(std::__1::shared_ptr<DB::IMergeTreeDataPart const>&&) @ 0x1f959e0d in /usr/bin/clickhouse
    02:47:06.816309 [ 29209 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeData.cpp:2593: DB::MergeTreeData::tryRemovePartImmediately(std::__1::shared_ptr<DB::IMergeTreeDataPart const>&&) @ 0x1f959e0d in /usr/bin/clickhouse
    02:47:09.455665 [ 29208 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:1939: DB::StorageReplicatedMergeTree::tryExecutePartMutation(DB::ReplicatedMergeTreeLogEntry const&) @ 0x1f5f2bf6 in /usr/bin/clickhouse
    02:47:09.468738 [ 29209 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:1939: DB::StorageReplicatedMergeTree::tryExecutePartMutation(DB::ReplicatedMergeTreeLogEntry const&) @ 0x1f5f2bf6 in /usr/bin/clickhouse
    02:47:11.776857 [ 29208 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:1581: DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&) @ 0x1f5e484c in /usr/bin/clickhouse
    02:47:11.904232 [ 29209 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:1581: DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&) @ 0x1f5e484c in /usr/bin/clickhouse
    02:47:13.941811 [ 29208 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:3176: DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18::operator() shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) const @ 0x1f65faa5 in /usr/bin/clickhouse
    02:47:14.477158 [ 29209 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:3176: DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18::operator() shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) const @ 0x1f65faa5 in /usr/bin/clickhouse
    02:47:16.475373 [ 29208 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676: decltype(std::__1::forward<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>))(std::__1::forward<std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&>(fp0))) std::__1::__invoke<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMEntry>&>(DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) @ 0x1f65fa32 in /usr/bin/clickhouse
    02:47:16.970325 [ 29209 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676: decltype(std::__1::forward<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>))(std::__1::forward<std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&>(fp0))) std::__1::__invoke<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMEntry>&>(DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) @ 0x1f65fa32 in /usr/bin/clickhouse
    02:47:18.979481 [ 29208 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:317: bool std::__1::__invoke_void_return_wrapper<bool>::__call<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplireeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&>(DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogE x1f65f9f2 in /usr/bin/clickhouse
    02:47:19.450807 [ 29209 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:317: bool std::__1::__invoke_void_return_wrapper<bool>::__call<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplireeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&>(DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogE x1f65f9f2 in /usr/bin/clickhouse
    02:47:21.055007 [ 29208 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608: std::__1::__function::__default_alloc_func<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::ry>)::$_18, bool (std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&)>::operator()(std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) @ 0x1f65f9b0 in /usr/bin/clickhouse
    02:47:23.546946 [ 413 ] {} <Fatal> Application: Child process was terminated by signal 6.
```

</details>

CI: https://clickhouse-test-reports.s3.yandex.net/0/4a8b82232c11512232df3ecdf4ffaec287116ad5/stress_test_(debug).html#fail1
 2021-08-27 10:08:44 +03:00
Alexey Milovidov
46732e781f Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-27 01:57:48 +03:00
Mike Kot
1f3701d485 Disk empty name fix 2021-08-26 19:28:33 +02:00
Amos Bird
103e0abe1c
Reserve protocol number for ALTER PRIMARY KEY. 2021-08-26 22:08:58 +08:00
Amos Bird
0169fce78e
Projection bug fixes and refactoring. 2021-08-26 19:09:31 +08:00
Ilya Yatsishin
5553a064be
Merge pull request #28139 from lthaooo/fix_prewhere_total_size_of_queried_columns_bug 
Fix bug in MergeTreeWhereOptimizer
 2021-08-26 10:42:23 +03:00
Anton Popov
6d3274c22c Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-26 03:31:46 +03:00
Alexey Milovidov
7dd0146223 Fix error 2021-08-26 01:56:08 +03:00
Alexey Milovidov
3cb9cf7242 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-26 01:55:46 +03:00
Maksim Kita
2403686bb9
Merge pull request #28135 from amosbird/projection-improve3 
Get rid of useless projection columns during merge
 2021-08-25 22:14:38 +03:00
Mike Kot
407af64280 Merge remote-tracking branch 'upstream/master' into improvement/merge-tree-part-opt-parse 2021-08-25 18:24:43 +02:00
lthaooo
eecd5b21c9 fix total_size_of_queried_columns bug 2021-08-25 23:02:13 +08:00
Amos Bird
a6b2a6ee36
Get rid of useless projection columns during merge 2021-08-25 20:54:27 +08:00
lthaooo
314ba1e682
Merge branch 'master' into materialize_ttl_recalculate_only 2021-08-25 10:56:24 +08:00
Alexey Milovidov
848c2ec23e Tuneable prefetch 2021-08-25 02:38:08 +03:00
Alexey Milovidov
fd27a2faef Remove imperfect code 2021-08-25 02:05:55 +03:00
Alexey Milovidov
7c1d0a3baf Progress on development 2021-08-25 01:24:47 +03:00
Alexey Milovidov
f7ca674b0c Progress on development 2021-08-25 01:07:06 +03:00
Alexey Milovidov
8f57216180 Progress on development 2021-08-25 00:45:58 +03:00
Alexey Milovidov
4d9ad3725d Merge branch 'master' into async-reads 2021-08-25 00:45:40 +03:00
Mike Kot
d621605954 Merge remote-tracking branch 'upstream/master' into improvement/merge-tree-part-opt-parse 2021-08-24 19:43:33 +02:00
Mike Kot
f85e2e027c Optional semantics for [Detached]MergeTreePartInfo 2021-08-24 19:43:09 +02:00
Vitaly Baranov
64f69462c6
Merge pull request #27921 from vitlibar/add-tests-for-field-init 
Allow implicit cast bool to Field
 2021-08-24 17:15:44 +03:00
mergify[bot]
ae22bab2c4
Merge branch 'master' into materialize_ttl_recalculate_only 2021-08-24 08:49:19 +00:00
Vitaly Baranov
63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00
Anton Popov
0db8b524f0
Merge pull request #27984 from BraulioVM/optimize/has-all-bloom-filter 
Make `hasAll` filter condition leverage bloom filters
 2021-08-23 22:19:50 +03:00
Alexander Tokmakov
4a4a0b4823 fix 2021-08-23 16:32:07 +03:00
Alexander Tokmakov
cc9c2fd63b make code better 2021-08-23 15:57:50 +03:00
Yatsishin Ilya
17bb938541 fix style 2021-08-23 13:59:01 +03:00
Amos Bird
a2256b1307
Dedicated Mark/Uncompressed cache for skip indices 2021-08-23 15:27:31 +08:00
Vitaly Baranov
89166a655a Init SettingFieldBool with true or false, not with 1 or 0. 2021-08-23 09:47:36 +03:00
Vitaly Baranov
ff1fad8521 Use setSetting() with implicit conversion true/false -> Field. 2021-08-23 08:57:00 +03:00
Braulio Valdivielso
d48f6122e3 clean up implementation a bit 2021-08-22 10:06:30 +01:00
Braulio Valdivielso
e95563fb82 leverage bloom filter index for 'hasAll' 
Fixes #26151. For now, this is a draft implementation. There is some
cleanup that can be done, but the new test is passing
 2021-08-21 20:14:56 +01:00
Alexander Tokmakov
59eb3aa9a9 avoid too long waiting for inactive replicas 2021-08-20 15:59:57 +03:00
nvartolomei
c09c90125f
Merge branch 'master' into nv/last-queue-update-exception 2021-08-19 10:29:16 +01:00
Nikolai Kochetov
7dcff5c90e
Merge pull request #27742 from amosbird/projection-improvement3 
Improve projection analysis.
 2021-08-19 12:11:20 +03:00
alesapin
0aec151719
Merge pull request #27808 from ClickHouse/fix_replicas_may_diverge 
Fix a couple of bugs that may cause replicas to diverge
 2021-08-19 09:58:07 +03:00
Alexander Tokmakov
09ff66da0e fix a couple of bugs that may cause replicas to diverge 2021-08-18 12:50:46 +03:00
alesapin
5cc98c67a4 Fix 2021-08-18 11:56:18 +03:00
mergify[bot]
f11e396151
Merge branch 'master' into nv/last-queue-update-exception 2021-08-18 07:00:50 +00:00
Nikita Mikhaylov
cc8ad0a4e7
Merge pull request #27733 from Algunenano/i27620_take2 
Refactor arrayJoin check on partition expressions
 2021-08-17 15:57:16 +03:00
Nikolai Kochetov
98eb619b4b
Merge pull request #23367 from Avogar/short-circuit 
Implement short circuit function evaluation
 2021-08-17 12:36:07 +03:00
Amos Bird
ea7dc495cf
Better code. 2021-08-17 12:32:49 +08:00
Raúl Marín
90f224807a Style 2021-08-16 17:31:37 +02:00
Amos Bird
90881aab09
Better code style 2021-08-16 22:17:07 +08:00
Raúl Marín
3ec904a953 Only analyze if tuple has arguments 2021-08-16 14:47:41 +02:00
Amos Bird
b162a2b699
Improve projection analysis. 
Remove duplicate index analysis and avoid possible invalid limit checks
during projection analysis.
 2021-08-16 20:11:57 +08:00
Raúl Marín
34099bc196 Refactor arrayJoin check on partition expressions 2021-08-16 12:46:18 +02:00
Alexey Milovidov
d184b79bba Progress on async reads. 2021-08-16 03:00:32 +03:00
alexey-milovidov
4a36366925
Merge pull request #27648 from Algunenano/i27620 
Disable arrayJoin on partition expressions
 2021-08-16 02:53:36 +03:00
Alexey Milovidov
0cd3f25ba4 Merge remote-tracking branch 'origin/master' into async-reads 2021-08-16 02:41:01 +03:00
alexey-milovidov
76b0502482
Merge pull request #27690 from kitaisreal/remove-dense-hash-map-set 
Removed DenseHashMap, DenseHashSet
 2021-08-16 02:39:41 +03:00
Maksim Kita
5f5470c2cd Removed DenseHashMap, DenseHashSet 2021-08-15 13:32:56 +03:00
Alexey Milovidov
020beb885d Merge branch 'master' into async-reads 2021-08-15 12:08:19 +03:00
mergify[bot]
bf426cfb96
Merge branch 'master' into fix_intersecting_parts 2021-08-15 09:07:20 +00:00
mergify[bot]
353a324887
Merge branch 'master' into i27620 2021-08-15 02:32:26 +00:00
alexey-milovidov
ab4415a22b
Update MergeTreeData.cpp 2021-08-15 05:28:38 +03:00
Alexander Tokmakov
c534363abe fix intersecting parts 2021-08-13 20:33:10 +03:00
Raúl Marín
73cb7d55ec Disable arrayJoin on partition expressions 2021-08-13 16:18:46 +02:00
mergify[bot]
319e5af6eb
Merge branch 'master' into short-circuit 2021-08-13 13:01:46 +00:00
Nikolai Kochetov
66fdb859f3
Merge pull request #27528 from amosbird/projection-fix12 
Fix projection materialization with missing columns
 2021-08-13 15:20:51 +03:00
Kseniia Sumarokova
0917488cec
Merge pull request #27382 from kssenii/cast-internal 
add Cast internal function
 2021-08-13 13:13:51 +03:00
Nikolai Kochetov
5e5a6ff654 Merge branch 'master' into qoega-fix-access-gtest-in-arcadia 2021-08-12 12:02:15 +03:00
mergify[bot]
80eaea1c51
Merge branch 'master' into short-circuit 2021-08-12 08:38:45 +00:00
kssenii
dcc6f597dc Move templates from FunctionsConvertion 2021-08-11 19:09:51 +00:00
kssenii
7555fc19fd Revert "Move some templates" 
This reverts commit 7f247becca.
 2021-08-11 18:49:25 +00:00
mergify[bot]
68a52fc065
Merge branch 'master' into projection-fix12 2021-08-11 17:08:04 +00:00
Alexander Kuzmenkov
8f9845722a
Merge pull request #27250 from azat/minmax-nullable-new-format 
Add new index data skipping minmax index format for proper Nullable support
 2021-08-11 13:56:16 +03:00
kssenii
7d8b694a9f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into cast-internal 2021-08-11 07:37:21 +00:00
kssenii
7f247becca Move some templates 2021-08-11 07:32:46 +00:00
Alexey Milovidov
08f8511854 Fix Style 2021-08-11 06:52:28 +03:00
alexey-milovidov
54d8282209
Merge pull request #27248 from azat/mt-mutation-stuck-fix 
Fix mutation stuck on invalid partitions in non-replicated MergeTree
 2021-08-11 06:10:15 +03:00
alexey-milovidov
a0a5c0da32
Update BackgroundJobsExecutor.cpp 2021-08-11 06:08:30 +03:00
Amos Bird
e63c26edb7
Fix projection materialization with missing columns 2021-08-10 21:47:27 +08:00
kssenii
eb4746ac69 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into cast-internal 2021-08-10 09:38:41 +00:00
Nikolai Kochetov
a1ec7f75c5 Merge branch 'master' into qoega-fix-access-gtest-in-arcadia 2021-08-10 11:31:47 +03:00
Nikolai Kochetov
bb4c11cd27
Merge pull request #27298 from ClickHouse/fix-23515 
Fix bug from #23515.
 2021-08-09 23:25:30 +03:00
Nikolai Kochetov
8cc493a3cd Try fix build. 2021-08-09 18:09:29 +03:00
Pavel Kruglov
70b51133c1 Try to simplify code 2021-08-09 18:01:08 +03:00
Pavel Kruglov
0662df8b76 Fix performance with JIT, add arguments to function isSuitableForShortCircuitArgumentsExecution 2021-08-09 17:54:14 +03:00
Pavel Kruglov
e792fa588f Mark all Functions as sutable or not for executing as short circuit arguments 2021-08-09 17:50:09 +03:00
Nicolae Vartolomei
3f291b024a
Use plain mutex instead of MultiVersion 2021-08-09 13:58:23 +01:00
Nikolai Kochetov
4a00e402ae Fix spelling. 2021-08-09 12:09:09 +03:00
Nikolai Kochetov
372293b85d
Merge branch 'master' into fix-27179 2021-08-09 11:59:22 +03:00
Alexey Milovidov
8b9da6fe26 Merge branch 'master' into async-reads 2021-08-09 03:33:32 +03:00
kssenii
7991bb6e83 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into cast-internal 2021-08-08 21:30:17 +00:00
Azat Khuzhin
038241b6ed Add new index data skipping minmax index format for proper Nullable support 
Note, that it cannot be done w/o new extension, since index does not
have any header.

v2: use IDisk interface for existence check
v3: remove extra file existence check
v4: fix MATERIALIZE INDEX
 2021-08-08 19:30:12 +03:00
Azat Khuzhin
5139067631 Guard BackgroundJobsExecutor from thread termination in case of uncaught exception 2021-08-08 19:27:58 +03:00
alexey-milovidov
ba3fbbaab0
Update ReplicatedMergeTreeMergeStrategyPicker.cpp 2021-08-08 04:43:59 +03:00
mergify[bot]
9869c8bb10
Merge branch 'master' into zero_copy_by_default 2021-08-07 16:53:44 +00:00
kssenii
9ca422f0c5 Introduce CAST for internal usage 2021-08-07 09:03:10 +00:00
Alexander Tokmakov
5e2bfd5ba1 fix partition id validation 2021-08-06 23:56:51 +03:00
Nikolai Kochetov
3cbeab8959 Fix incorrect row-level filtering 2021-08-06 21:16:06 +03:00
Nikolai Kochetov
7320411a7f Fix bug from #23515. 2021-08-06 16:39:11 +03:00
Anton Popov
3a0d4807a5 Merge branch 'async-insert' of git://github.com/abyss7/ClickHouse into merging-20557 2021-08-06 16:20:02 +03:00
Alexey Milovidov
cea9690327 Merge branch 'master' into async-reads 2021-08-05 23:58:57 +03:00
alexey-milovidov
1a3d8cebd2
Merge pull request #27197 from azat/nullable-index-fix 
Fix on-disk format breakage for secondary indices over Nullable column
 2021-08-05 21:52:18 +03:00
Kruglov Pavel
59a94bd322
Merge pull request #27183 from ClickHouse/fix_detached_parts_name_partsing 
Fix part name parsing in system.detached_parts
 2021-08-05 18:21:03 +03:00
Alexander Kuzmenkov
d852207b0e
Merge pull request #25721 from CurtizJ/read-in-order-perf-2 
Reduce memory usage in queries with `ORDER BY primary_key
 2021-08-05 16:13:48 +03:00
terrylin
dd539f0a0d improvement of materilize ttl 2021-08-05 15:33:02 +08:00
terrylin
bd3d9a4518 materialize ttl recalculate only (optional) 2021-08-05 14:17:48 +08:00
Azat Khuzhin
dee27fcbb9 Fix on-disk format breakage for secondary indices over Nullable column 
[1] breaks on disk format (and the relevant change in the:

  [1]: https://github.com/ClickHouse/ClickHouse/pull/12455#discussion_r682830812

Too bad that I checked this patchset only for compatibility after
reverting this patch [2] (use case: I've applied it manually, then
revert it, and data skipping indexes over Nullable column had been
broken)

  [2]: https://github.com/ClickHouse/ClickHouse/pull/12455#issuecomment-823423772

But this patchset actually breaks compatibility with older versions of
clickhouse for Nullable data skipping indexes after simple upgrade:

Here is a simple reproducer:

    --
    -- run this with 21.6 or similar (i.e. w/o this patch)
    --

    CREATE TABLE data
    (
        `key` Int,
        `value` Nullable(Int),
        INDEX value_index value TYPE minmax GRANULARITY 1
    )
    ENGINE = MergeTree
    ORDER BY key;

    INSERT INTO data SELECT
        number,
        number
    FROM numbers(10000);

    SELECT * FROM data WHERE value = 20000 SETTINGS force_data_skipping_indices = 'value_index' SETTINGS force_data_skipping_indices = 'value_index', max_rows_to_read=1;

Now upgrade and run the query again:

    SELECT * FROM data WHERE value = 20000 SETTINGS force_data_skipping_indices = 'value_index' SETTINGS force_data_skipping_indices = 'value_index', max_rows_to_read=1;

And it will fail because of on disk format changes:

    $ ll --time-style=+ data/*/data/all_1_1_0/skp*.idx
    -rw-r----- 1 azat azat 36  data/with_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
    -rw-r----- 1 azat azat 37  data/without_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx

    $ md5sum data/*/data/all_1_1_0/skp*.idx
    a19c95c4a14506c65665a1e30ab404bf  data/with_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
    e50e2fcfa873b232196623d56ab26105  data/without_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx

Note, that there is no stable release with this patch included yet, so
no need to backport.

Also note that you may create data skipping indexes over Nullable
column even before [3].

  [3]: https://github.com/ClickHouse/ClickHouse/pull/12455

v2: break cases when granulas has Null in values due to backward
compatibility
 2021-08-05 00:19:38 +03:00
Anton Popov
9586bb7e90 fix pvs 2021-08-04 20:42:22 +03:00
terrylin
6f42ec6b9b fix compile error 2021-08-05 00:16:21 +08:00
Anton Popov
915ef50c2d optimize reading in order of key more 2021-08-04 18:18:03 +03:00
Alexander Tokmakov
23f8b3d07d fix part name parsing in system.detached_parts 2021-08-04 17:42:48 +03:00
terrylin
88b9e7f24f materialize ttl recalculate only optional 2021-08-04 22:16:13 +08:00
Anton Ivashkin
61016da2b1 Set allow_remote_fs_zero_copy_replication to true by default 2021-08-04 17:08:09 +03:00
Yatsishin Ilya
55e06ea16b fix 2021-08-04 13:06:01 +03:00
Alexey Milovidov
fd3b5680e2 Merge branch 'master' into async-reads 2021-08-03 23:13:31 +03:00
Anton Popov
fc9d72e75d fix performance of short queries with large number of columns 2021-08-02 15:03:55 +03:00
Nikolai Kochetov
6951e8147d
Merge pull request #27020 from amosbird/shardlevelconstness 
Shard-level const column
 2021-08-02 13:45:20 +03:00
Anton Popov
cc3ed12ed6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-02 02:56:00 +03:00
Alexey Milovidov
2a0cba2b9f Merge branch 'master' into async-reads 2021-08-01 01:25:13 +03:00
alexey-milovidov
701e2ffd79
Merge pull request #27002 from azat/mutation-stuck 
Fix possible mutation stuck due to race with DROP_RANGE
 2021-07-31 21:17:46 +03:00
Amos Bird
cd302eacc1
Fix projection match 2021-07-31 15:44:16 +08:00
tavplubix
00ac8194dc
Merge pull request #26896 from ClickHouse/small_improvement_in_queue 
Maybe fix extremely rare `intersecting parts`.
 2021-07-30 13:04:20 +03:00
terrylin
c49c2e5e18 fix compile error 2021-07-30 17:15:04 +08:00
mergify[bot]
41273ef5f3
Merge branch 'master' into mutation-stuck 2021-07-30 08:42:45 +00:00
Azat Khuzhin
bc9a4366d8 Fix possible mutation stack due to race with DROP_RANGE 
After #25884 parts that was dropped (DROP_RANGE) will not be removed
from old mutations and this will stuck the mutation.

Interesting, that this mutation may continue after some merge.
 2021-07-30 08:22:10 +03:00
terrylin
55f3e7e65a improvement of materilize ttl 2021-07-30 12:27:15 +08:00
alesapin
21599ffefc Simple validation for partition id before drop partition 2021-07-29 16:11:05 +03:00
alesapin
5799487caa Fix stupid bug 2021-07-29 14:54:36 +03:00
alesapin
bdc6858801 More correct comparsion 2021-07-28 13:19:34 +03:00
alesapin
26056cfb25 Fix extremely rare race condition on remove of unsuccessful entry from queue 2021-07-28 13:06:22 +03:00
Alexey Milovidov
9baf84224a Fix build 2021-07-28 02:54:20 +03:00
Alexey Milovidov
fd5c1561e2 Merge branch 'master' into async-reads 2021-07-27 23:58:32 +03:00
Nicolae Vartolomei
8b07a7f180 Store exception generated when we tried to update the queue last time 
The use case is to alert when queue contains broken entries. Especially
important when ClickHouse breaks backwards compatibility between
versions and log entries written by newer versions aren't parseable by
old versions.

```
Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected 'quorum: ' before: 'merge_type: 2\n'
```
 2021-07-27 15:42:40 +01:00
tavplubix
168edaed73
Merge pull request #26808 from amosbird/safe2 
Make sure table is readonly when restarting fails.
 2021-07-27 17:21:08 +03:00
Nikolai Kochetov
97bc754dd8
Merge pull request #26758 from ClickHouse/output-streams-to-processors 
Remove some output streams
 2021-07-27 17:19:26 +03:00
alexey-milovidov
d7b2404119
Merge pull request #15408 from ClickHouse/random_timezone_for_stateless_tests 
Use random timezone in stateless and stateful tests
 2021-07-27 16:18:14 +03:00
Alexey Milovidov
5e2ea4e042 Whitespace 2021-07-27 05:20:03 +03:00
Nikolai Kochetov
61d8f880cd Rename some files. 2021-07-26 19:48:25 +03:00
Nikolai Kochetov
9b5a816b43 Merge branch 'master' into output-streams-to-processors 2021-07-26 18:03:11 +03:00
Nikolai Kochetov
0eb563dc1b Fix more tests. 2021-07-26 17:47:29 +03:00
Amos Bird
623faf47e4
Make sure table is readonly when restarting fails. 2021-07-26 19:21:14 +08:00
Nikolai Kochetov
fa1c223269 Fix some tests. 2021-07-26 13:08:40 +03:00
Alexey Milovidov
5c3f5cb35a Experiment with asynchronous readers 2021-07-26 03:34:36 +03:00
Nikolai Kochetov
9c92f43359 Update storages. 2021-07-23 22:33:59 +03:00
Azat Khuzhin
00e2083421 Fix event_time_microseconds for REMOVE_PART in system.part_log 2021-07-23 00:59:08 +03:00
alexey-milovidov
fcca3e3e56
Merge pull request #26423 from ClickHouse/fix-header-for-scalar-query-with-empty-result 
Fix incompatible result type for scalar queries with empty result.
 2021-07-21 02:07:04 +03:00
Anton Popov
fa6857920d fix arcadia 2021-07-19 16:45:35 +03:00
Nikolai Kochetov
7a24e72e76 Merge branch 'master' into fix-header-for-scalar-query-with-empty-result 2021-07-19 15:48:44 +03:00
Nikolai Kochetov
96e20e2641 Fix some tests. 2021-07-19 15:35:55 +03:00
Kseniia Sumarokova
b0ddc4fb30
Merge pull request #26364 from kssenii/fixing-date32 
Continuation of #25774
 2021-07-18 21:44:45 +03:00
alexey-milovidov
b52411a715
Merge pull request #12455 from amosbird/npc 
Nullable primary key with correct KeyCondition
 2021-07-18 17:52:20 +03:00
Alexey Milovidov
261a220227 Remove some code 2021-07-17 21:06:46 +03:00
Azat Khuzhin
071dc7c3f5 Improve "Not executing log entry queue-X of type MUTATE_PART" message 2021-07-17 11:50:04 +03:00
Alexey Milovidov
bd924a5b8a Whitespace 2021-07-17 01:59:39 +03:00
alexey-milovidov
c373b86743
Merge pull request #26371 from CurtizJ/improve-perf-get-columns 
Improve performance of getting list of columns
 2021-07-17 01:34:15 +03:00
Anton Popov
048e089a58 remove useless code 2021-07-16 17:47:03 +03:00
Anton Popov
03c785931a fix pushdown of limit to reading stage 2021-07-16 17:27:38 +03:00
Anton Popov
c45a9e3bd6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-16 17:11:34 +03:00
Anton Popov
9e3af27f40 better performance of getSampleBlockForColumns 2021-07-15 20:36:48 +03:00
Anton Popov
5f71a6f5bb improve performance of getting columns list 2021-07-15 16:29:01 +03:00
kssenii
3ba2d114d5 Revert "Merge pull request #26352 from ClickHouse/revert-25774-datatype-date32" 
This reverts commit 94a210650f, reversing
changes made to 0893b9ff8e.
 2021-07-15 11:41:52 +00:00
alexey-milovidov
9ba4064c86
Revert "Datatype Date32, support range 1925 to 2283" 2021-07-15 10:53:35 +03:00
alexey-milovidov
0893b9ff8e
Merge pull request #26218 from amosbird/projection-improve1 
Aggressive IN index analysis for projections.
 2021-07-15 08:13:45 +03:00
Filatenkov Artur
f1702d356e
Merge pull request #26120 from lthaooo/settings-merge_selecting_sleep_ms 
add merge_selecting_sleep_ms setting
 2021-07-14 18:28:47 +03:00
Amos Bird
28b567a01b
Update 2021-07-14 23:09:17 +08:00
alexey-milovidov
55fc7323fd
Update MergeTreeDataSelectExecutor.h 2021-07-14 22:56:52 +08:00
alexey-milovidov
1ef068596c
Update MergeTreeDataSelectExecutor.cpp 2021-07-14 22:56:52 +08:00
Amos Bird
0558ecdc3f
Aggressive IN index analysis for projections. 2021-07-14 22:56:52 +08:00
Anton Popov
e2249bf1e5
Merge pull request #26256 from zxc111/issue-822 
Check hash function at table creation
 2021-07-14 14:16:02 +03:00
Zhichang Yu
fe5d17680e Merge remote-tracking branch 'origin/master' into zero_copy_hdfs 2021-07-14 02:12:28 +00:00
Maksim Kita
3d05f07ece
Merge pull request #25917 from excitoon-favorites/defaultblocksizeprediction 
Using required columns for block size prediction
 2021-07-13 20:05:41 +03:00
Vitaly Baranov
3c395389b0
Merge pull request #24206 from alexelex/alexelex-encryption-1 
Data encryption on VFS level
 2021-07-13 19:50:25 +03:00
Anton Popov
cb042afa9d pushdown limit while reading in order of primary key 2021-07-13 17:24:45 +03:00
zxc111
7b2fed2951 use flag to check sample_condition 2021-07-13 21:17:22 +08:00
Anton Popov
19d0f1bbfc Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-13 16:12:08 +03:00
Ivan Lezhankin
ab26aed6f9 Merge remote-tracking branch 'upstream/master' into async-insert 2021-07-13 13:54:09 +03:00
xiedeyantu
df438f4288 Modify code comments 2021-07-13 17:52:24 +08:00
Vitaly Baranov
e086e5e384
Merge branch 'master' into alexelex-encryption-1 2021-07-13 11:28:41 +03:00
Zhichang Yu
b4e6689bf9 fix test_hdfs_zero_copy_replication_move[tiered_copy-2] 2021-07-13 07:20:23 +00:00
Zhichang Yu
5047c758f4 fix per review 2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1 hdfs zero copy 2021-07-13 07:19:12 +00:00
alexey-milovidov
4b1e38eed9
Merge branch 'master' into projection-fix11 2021-07-13 04:52:15 +03:00
alexey-milovidov
5295f7ebb2
Merge pull request #26235 from ClickHouse/check-non-deterministic-functions-in-keys 
Check for non-deterministic functions in keys, including constant expressions
 2021-07-13 02:46:44 +03:00
Kseniia Sumarokova
4a6b4d6c26
Merge pull request #25774 from liuneng1994/datatype-date32 
Datatype Date32, support range 1925 to 2283
 2021-07-12 20:33:39 +03:00
zxc111
e0261866a4 add settings && add test case 2021-07-13 01:15:35 +08:00
mergify[bot]
b7ae4e4ced
Merge branch 'master' into async-insert 2021-07-12 09:21:37 +00:00
Alexey Milovidov
2a2aa69459 Handle non-existent "detached" directories 2021-07-12 10:01:26 +03:00
Alexey Milovidov
ff5c433f10 Check for non-deterministic functions in keys, including constant expressions 2021-07-12 08:58:19 +03:00
alexey-milovidov
8f1d7e67ca
Merge pull request #26003 from ClickHouse/ditch-aio 
Remove AIO for sequential reads
 2021-07-12 03:25:23 +03:00
zxc111
cd97478c06 check sampling condition when create table 2021-07-12 00:38:20 +08:00
Alexey Milovidov
1ce40d6629 Fix bad code (default function argument) 2021-07-11 04:01:23 +03:00
Alexey Milovidov
81646f8389 Merge branch 'master' into ditch-aio 2021-07-11 03:06:45 +03:00
terrylin
faed6263bb add merge_selecting_sleep_ms setting 2021-07-09 17:29:17 +08:00
Anton Popov
9f1ffa777f remove unused code 2021-07-08 15:06:33 +03:00
alesapin
4c85dae572
Merge pull request #25743 from ClickHouse/fix_aggregation_ttl 
Fix bug in execution of TTL GROUP BY
 2021-07-07 10:49:16 +03:00
alesapin
0d8844c510
Merge pull request #25884 from ClickHouse/fix_drop_part_in_queue 
Relax `DROP PART` guarantees and turn on checks in ReplicationQueue.
 2021-07-07 10:48:48 +03:00
mergify[bot]
446474932b
Merge branch 'master' into projection-fix11 2021-07-07 05:02:44 +00:00
Amos Bird
55981cb0ae
Fix normal projection 2021-07-07 13:01:30 +08:00
alesapin
3dee74df54 Comment 2021-07-06 19:53:54 +03:00
alesapin
a8fdc41193 Fix bug and add more trash to test 2021-07-06 19:51:23 +03:00
mergify[bot]
7ff8ac51a9
Merge branch 'master' into fix-merge-tree-settings-change 2021-07-06 12:14:33 +00:00
alesapin
75e26b93d0 Review bug fixes 2021-07-06 15:05:58 +03:00
alesapin
e9540f0621 Remove debug logs 2021-07-06 14:13:35 +03:00
mergify[bot]
8dd378f7c2
Merge branch 'master' into fix_drop_part_in_queue 2021-07-06 11:00:04 +00:00
alesapin
53b23775a9 Fix drop part 2021-07-06 13:58:53 +03:00
alesapin
2600922786 Followup fix 2021-07-06 11:36:39 +03:00
Alexey Milovidov
21fb6ddea5 Remove AIO for sequential reads 2021-07-05 23:02:24 +03:00
alesapin
f7e1cfdb24 Some partially working code 2021-07-05 22:58:55 +03:00
Anton Popov
9071ecd428 fix alter of settings in MergeTree 2021-07-05 15:44:58 +03:00
NengLiu
8b88988618 Merge branch 'master' into datatype-date32 
# Conflicts:
#	src/DataStreams/PostgreSQLBlockInputStream.cpp
 2021-07-05 10:18:38 +08:00
NengLiu
dc0c65ef4f Merge remote-tracking branch 'origin/master' into datatype-date32 2021-07-05 10:15:08 +08:00
alexey-milovidov
fef7c557f6
Merge pull request #24789 from kitaisreal/compile-aggregation-functions 
Compile aggregate functions
 2021-07-05 02:57:49 +03:00
mergify[bot]
1559c09977
Merge branch 'master' into fix_drop_part_in_queue 2021-07-04 16:00:39 +00:00
alesapin
cf2fc94d9d
Merge pull request #25820 from ClickHouse/better_remove_empty_parts 
Replace lost parts with empty parts instead of hacking replication queue
 2021-07-04 18:55:49 +03:00
Vladimir Chebotarev
e5cbd82879 Took PREWHERE into account. 2021-07-04 18:06:15 +03:00
alexey-milovidov
51bf60a938
Merge pull request #17769 from xjewer/alex/remove_engine_setting 
CLICKHOUSE-784: reset merge tree setting
 2021-07-03 18:27:15 +03:00
alexey-milovidov
1dda771864
Merge pull request #25554 from CurtizJ/fix-alter-ttl-column 
Fix `ALTER MODIFY COLUMN` of columns, which participates in TTL expressions
 2021-07-03 18:26:37 +03:00
mergify[bot]
9da1c98998
Merge branch 'master' into fix_aggregation_ttl 2021-07-03 15:07:44 +00:00
alesapin
7b4a56977d
Merge pull request #25893 from ClickHouse/no_backoff_when_pool_is_full 
Reset counter if background pool is full
 2021-07-03 18:03:29 +03:00
mergify[bot]
fbe1ff0fb4
Merge branch 'master' into fix_drop_part_in_queue 2021-07-03 15:02:15 +00:00
mergify[bot]
a2183dff39
Merge branch 'master' into better_remove_empty_parts 2021-07-03 15:01:11 +00:00
mergify[bot]
02cb287dc9
Merge branch 'master' into fix-alter-ttl-column 2021-07-02 23:49:46 +00:00
alesapin
b4b54a0549 Fix bug 2021-07-02 18:25:05 +03:00
Vladimir Chebotarev
6f5a72c963 Using required columns for block size prediction. 2021-07-02 16:55:00 +03:00
alesapin
72b84141e1 Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-07-02 12:41:28 +03:00
alesapin
f8f734d3dc Better comment 2021-07-02 12:40:13 +03:00
alesapin
abc3aca19b Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-07-02 12:30:54 +03:00
alesapin
2e29dc2975 More safe empty parts creation 2021-07-02 12:30:17 +03:00
alesapin
5e79c68e8e Debug 2021-07-02 11:54:22 +03:00
Alexey Milovidov
0e621788c7 Allow constexpr parameters for aggregate functions 2021-07-02 03:53:08 +03:00
Maksim Kita
da8c957167 Aggregator added CompiledExpressionCache 2021-07-01 22:56:36 +03:00
Maksim Kita
d24d3ae992 Added second variant of compilation 2021-07-01 22:56:36 +03:00
Maksim Kita
3fe559b31f Compile aggregate functions 2021-07-01 22:56:35 +03:00
Aleksei Semiglazov
9a5365fc41
CLICKHOUSE-784: reset merge tree setting 
Add an ability to reset custom setting to default and remove it from table's metadata.
This will allow to rollback the change without knowing the system/config's default.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>
 2021-07-01 20:09:54 +01:00
alesapin
7792b09911 Reset counter if background pool is full 2021-07-01 16:10:07 +03:00
alesapin
1a4ccab8e6 Fix style 2021-07-01 15:12:27 +03:00
mergify[bot]
98fa9f7951
Merge branch 'master' into better_remove_empty_parts 2021-07-01 11:32:43 +00:00
tavplubix
afbc6bf17d
Merge pull request #25684 from ClickHouse/cancel_merges_on_drop_partition 
Cancel merges on DROP PARTITION
 2021-06-30 23:41:04 +03:00
alesapin
c845a205f9 Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 22:42:02 +03:00
alesapin
45edc8c861 Fix tidy 2021-06-30 22:11:00 +03:00
alesapin
89a401a84f Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 18:53:17 +03:00
alesapin
6a73c8b49e Review fixes 2021-06-30 18:24:51 +03:00
tavplubix
c958f876b8
Update MergeList.h 2021-06-30 15:55:26 +03:00
alesapin
21f15be990 Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 15:30:32 +03:00
alesapin
2eb27540b2 Some test version 2021-06-30 15:29:09 +03:00
alesapin
94d4d32e4c
Merge pull request #25783 from ClickHouse/followup_drop_part_fix 
Relax check during DROP PART
 2021-06-30 12:25:42 +03:00
mergify[bot]
1799804243
Merge branch 'master' into fix_aggregation_ttl 2021-06-30 09:15:46 +00:00
mergify[bot]
8182b6e890
Merge branch 'master' into better_remove_empty_parts 2021-06-30 08:42:26 +00:00
alesapin
0af53ace55 Fix comment 2021-06-29 22:49:41 +03:00
alesapin
ca1d21b663
Update src/Storages/MergeTree/MergeTreeData.cpp 
Co-authored-by: tavplubix <avtokmakov@yandex-team.ru>
 2021-06-29 22:48:46 +03:00
alesapin
a6834213a1 Add tests 2021-06-29 22:47:54 +03:00
Nikolai Kochetov
d5bd2b1fa9
Merge pull request #25719 from ClickHouse/remove-PrewhereDAGInfo 
Remove PrewhereDAGInfo.
 2021-06-29 18:58:03 +03:00
alesapin
5822f0ba29 Replace lost parts with empty parts instead of hacking replication queue 2021-06-29 18:14:44 +03:00
Nikolai Kochetov
bf827936b7 Rename PrewhereActions 2021-06-29 14:53:34 +03:00
alesapin
7680eab0e4 one more fix 2021-06-29 13:03:00 +03:00
alesapin
941a6e539f Fix 2021-06-29 10:36:20 +03:00
Ivan Lezhankin
1bb41593d1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-28 22:55:56 +03:00
alesapin
72b281987e Add more debug 2021-06-28 17:15:51 +03:00
alesapin
d20feeb42c Relax check during DROP PART 2021-06-28 15:05:22 +03:00
alesapin
b11254d191
Merge branch 'master' into fix_aggregation_ttl 2021-06-28 13:31:29 +03:00
alesapin
4c213b639f
Merge pull request #25548 from nikitamikhaylov/background-processing 
A little improvement in BackgroundJobsExecutor
 2021-06-28 13:24:21 +03:00
alexey-milovidov
a11351a473
Merge pull request #25484 from vdimir/issue-19589 
Fix assert with non uint8 in prewhere
 2021-06-28 00:03:12 +03:00
alesapin
c977c33d6d Fix bug in execution of TTL GROUP BY 2021-06-27 19:18:15 +03:00
Anton Popov
2165ff679a
Merge branch 'master' into read-in-order-perf-2 2021-06-27 17:45:00 +03:00
alexey-milovidov
6662e7ab0f
Merge pull request #25732 from matwey/missed_include_optional_2 
Fix some more missed includes
 2021-06-26 14:04:06 +03:00
Matwey V. Kornilov
a29bd2c4b3 Add missed #include <limits> 
Signed-off-by: Matwey V. Kornilov <matwey.kornilov@gmail.com>
 2021-06-26 11:25:54 +03:00
Anton Popov
fafae3ab8d fix tests 2021-06-25 20:00:23 +03:00
Anton Popov
fa8ad1a7c6 reduce memory usage in queries with 'ORDER BY primary_key' 2021-06-25 19:23:41 +03:00
Ivan
14e43cfafa
Add CI check for darwin-aarch64 (#25560) 
* Add support for darwin-aarch64

* Fix PVS warnings

* Add build configuration

* Fix packager args
 2021-06-25 18:21:56 +03:00
Nikolai Kochetov
3544b059ce Merge branch 'master' into remove-PrewhereDAGInfo 2021-06-25 17:52:03 +03:00
Nikolai Kochetov
6bc0a628cd Remove PrewhereDAGInfo. 2021-06-25 17:49:28 +03:00
alesapin
f6afc381be
Merge pull request #25665 from ClickHouse/fix_24869 
Fix REPLACE_RANGE entry removal
 2021-06-25 11:11:25 +03:00
alesapin
d138ba79c7
Merge pull request #25549 from ClickHouse/add_test_for_concurrent_merges 
Fix bug in TTL merges assignment
 2021-06-25 11:02:29 +03:00
Alexander Tokmakov
76156af5cc cancel merges on drop partition 2021-06-24 17:07:43 +03:00
Ivan
e9e49a9085
Fix ANTLR parser and enable it back in CI (#25638) 
* Fix

* Skip more tests
 2021-06-24 14:45:29 +03:00
Alexander Tokmakov
7e6963bb39 fix REPLACE_RANGE entry removal 2021-06-24 11:16:54 +03:00
alesapin
e6da53dc46 better 2021-06-24 00:09:06 +03:00
alesapin
24876291d1 Followup fix 2021-06-24 00:01:07 +03:00
alesapin
4be4bc21e2 Fix for fix 2021-06-23 23:57:49 +03:00
alesapin
81c74435a3 Fix drop part bug 2021-06-23 22:25:30 +03:00
Nikita Mikhaylov
f4d6a6e128 support dynamic reloading of server config 2021-06-22 23:24:47 +00:00
Nikita Mikhaylov
a39bca0acf better 2021-06-22 23:24:47 +00:00
Nikita Mikhaylov
c66a3b22b5 done 2021-06-22 23:24:47 +00:00
mergify[bot]
c15493f0df
Merge branch 'master' into add_test_for_concurrent_merges 2021-06-22 15:49:35 +00:00
alesapin
47b29092b4 Fix bug in MergerMutator parts selector 2021-06-22 16:47:42 +03:00
Neng Liu
7ed1728a37 Merge branch 'master' into datatype-date32 
# Conflicts:
#	src/DataTypes/IDataType.h
#	src/Functions/CustomWeekTransforms.h
 2021-06-22 10:00:18 +08:00
Anton Popov
662d5b8495 fix ALTER MODIFY COLUMN of columns, that participates in TTL expressions 2021-06-21 18:20:18 +03:00
vdimir
46a5dd6701
Fix MergeTreeBaseSelectProcessor::executePrewhereActions 2021-06-21 16:01:02 +03:00
Anton Popov
ffa56bde24 fix usage of index with array columns and ARRAY JOIN 2021-06-21 15:34:05 +03:00
vdimir
eb85c5a8e7
Fix assert with non uint8 in prewhere 2021-06-21 15:32:29 +03:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652) 
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
 2021-06-20 11:24:43 +03:00
Amos Bird
f2ed5ef42b
Nullable primary key with correct KeyCondition 2021-06-18 23:04:24 +08:00
Latysheva Alexandra
48d3419160
Merge branch 'master' into alexelex-encryption-1 2021-06-18 11:28:08 +06:00
Ivan Lezhankin
37365589ed Merge branch 'master' into async-insert 2021-06-17 16:57:45 +03:00
Neng Liu
b6eab5fd2f add datetype32 function support 2021-06-17 16:05:30 +08:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Kseniia Sumarokova
d6240f2bd4
Merge pull request #25309 from azat/fs-fix 
Catch ErrnoException during parts cleaning
 2021-06-16 22:03:54 +03:00
tavplubix
07ce932282
Merge pull request #25306 from ClickHouse/tavplubix-patch-2 
Forcefully stop queue tasks on restarting
 2021-06-16 14:22:36 +03:00
Nikolai Kochetov
8bf55ae2b5
Merge pull request #25166 from ClickHouse/always-detach-parts-with-wrong-partition-id 
Always detach parts with wrong partition ID
 2021-06-16 11:06:07 +03:00
Kseniia Sumarokova
e08c05cdf5
Merge pull request #25295 from ClickHouse/remove-some-code-from-key-condition 
Remove some code from KeyCondition.
 2021-06-16 10:12:12 +03:00
Azat Khuzhin
de9689379f Catch ErrnoException during parts cleaning 
Avoids noisy messages like:

    2021.06.15 09:27:35.546982 [ 58176 ] {} <Error> auto DB::IBackgroundJobExecutor::jobExecutingTask()::(anonymous class)::operator()() const: Code: 481, e.displayText() = DB::ErrnoException: Cannot check modification time for file: /var/lib/clickhouse/store/c52/c52619f8-99bd-435e-8526-19f899bdb35e/tmp_insert_202106_7010_7010_0/, errno: 2, strerror: No such file or directory, Stack trace (when copying this message, always include the lines below):

    0. DB::Exception::Exception() @ 0x903209a in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    1. DB::throwFromErrnoWithPath() @ 0x9033089 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    2. FS::getModificationTime() @ 0x907b166 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    3. DB::DiskLocal::getLastModified() @ 0xf9a6669 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    4. DB::MergeTreeData::clearOldTemporaryDirectories(long) @ 0x1069a6de in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
 2021-06-16 06:56:55 +03:00
tavplubix
60c8dc1c0b
Update ReplicatedMergeTreeRestartingThread.cpp 2021-06-16 00:48:38 +03:00
Nikolai Kochetov
96d98ff020 Add comment 2021-06-15 21:42:26 +03:00
mergify[bot]
7959d92029
Merge branch 'master' into minor-changes-3 2021-06-15 18:07:24 +00:00
Nikolai Kochetov
80a13c489b Revert back moduloLegacy check for canConstantBeWrappedByFunctions. 2021-06-15 18:21:31 +03:00
Nikolai Kochetov
481b87b37a Remove some code from keyCondition. 2021-06-15 16:47:37 +03:00
Nikita Mikhaylov
a52bba91b7
Merge pull request #16401 from abyss7/ast-table-identifier-2 
ASTTableIdentifier part #2: Introduce ASTTableIdentifier
 2021-06-15 13:51:30 +03:00
tavplubix
e2ecc51a1f
Merge pull request #25087 from ClickHouse/srmt_remove_copypaste 
Remove copypaste from StorageReplicatedMergeTree
 2021-06-15 12:43:55 +03:00
Alexey Milovidov
4e982a3ae6 Merge branch 'master' into minor-changes-3 2021-06-15 00:45:45 +03:00
Alexey Milovidov
f49463d1e5 Fix clang-tidy 2021-06-14 07:28:17 +03:00
Alexey Milovidov
447d7bb8cd Minor changes 2021-06-14 07:13:35 +03:00
Alexey Milovidov
2211ba53d4 Copy paste some code 2021-06-14 05:48:10 +03:00
mergify[bot]
ab2a1a410f
Merge branch 'master' into test-key-condition-mon 2021-06-12 00:32:38 +00:00
alexey-milovidov
30f0124e77
Merge branch 'master' into always-detach-parts-with-wrong-partition-id 2021-06-12 02:55:55 +03:00
alexey-milovidov
eb7b491da9
Merge pull request #24900 from Enmk/bloom_filter_hasAny 
Added support of hasAny function to bloom_filter index.
 2021-06-12 00:58:25 +03:00
Ivan Lezhankin
2c7cb41a69 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-11 20:04:26 +03:00
Vasily Nemkov
2ac288c303 index is not used if types in hasAny are unsuported 
updated tests
 2021-06-11 18:29:52 +03:00
Alexander Tokmakov
73ff1728ae rename flag to more generic name 2021-06-11 15:41:48 +03:00
Nikolai Kochetov
79c69ea316 Fix build. 2021-06-11 11:29:39 +03:00
Vasily Nemkov
55064e49a7 Fixed the test 
Previous test version wasn't causing bug on unfixed code, this new one does.
 2021-06-10 19:09:43 +03:00
Nikolai Kochetov
fea19c2037
Merge pull request #25163 from amosbird/projection-fix10 
Disable projection when final is used in select query
 2021-06-10 18:44:29 +03:00
Vasily Nemkov
0ab28360c9 Added support of hasAny function to bloom_filter index. 
Also fixed bug discovered by AST fuzzer.
 2021-06-10 18:22:04 +03:00
Ivan
a7fa4e641c
Merge branch 'master' into ast-table-identifier-2 2021-06-10 16:43:22 +03:00
Nikolai Kochetov
6222fcffe0 Never remove parts on start at all. Always move. 2021-06-10 15:47:55 +03:00
Nikolai Kochetov
aa90309abf Never remove parts on start at all. Always move. 2021-06-10 15:46:03 +03:00
Nikolai Kochetov
2936fdd16c Always detach parts with wrong partition id. 2021-06-10 15:04:36 +03:00
Nikolai Kochetov
847dfb9914 Merge remote-tracking branch 'origin' into always-detach-parts-with-wrong-partition-id 2021-06-10 12:37:46 +03:00
Nikolai Kochetov
d52b87d47d Always detach proken parts with wrong partition id. 2021-06-10 12:37:33 +03:00
Amos Bird
4e358b28f1
Disable projection with final is used 2021-06-10 17:11:17 +08:00
Alexey Milovidov
6c5b2b827c Move code to more appropriate place 2021-06-10 02:58:51 +03:00
alexey-milovidov
0636f40349
Merge pull request #25127 from ClickHouse/fix-partition-hash-for-uuid 
Revert old UUID hash used for partition ID.
 2021-06-10 02:31:01 +03:00
Nikolai Kochetov
12791431da Revert old UUID hash used for partition ID. 2021-06-09 16:51:40 +03:00
Alexander Tokmakov
cef22688ff make code less ugly 2021-06-09 15:36:47 +03:00
Alexander Tokmakov
3ade38df82 remove copypaste 2021-06-08 22:17:45 +03:00
alexey-milovidov
e25d9843d1
Merge pull request #25025 from azat/parallel-kill-mutation 
Distinguish KILL MUTATION for different tables.
 2021-06-08 21:40:53 +03:00
Nikolai Kochetov
6197d20c18
Update KeyCondition.cpp 2021-06-08 15:48:14 +03:00
Ivan Lezhankin
973cea73b7 Merge branch 'master' into async-insert 2021-06-08 15:10:30 +03:00
Azat Khuzhin
1062d0ec91 Distinguish KILL MUTATION for different tables. 
Before this patch KILL MUTATION marks mutation as canceled just by name
(and part numbers) so if you have multiple tables with the same part
name, then killing mutation for one table, will mark it as killed for
another too.

Fix this by comparing StorageID too (it is better to use StorageID over
database/table to avoid ambiguity by using UUIDs for comparing).

Here is a failure of the 01414_freeze_does_not_prevent_alters on CI [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/24069/9fb69dcf98c71a939d200cad3c8491bf43a44622/functional_stateless_tests_(ubsan).html#fail1
 2021-06-08 10:51:22 +03:00
Nikolai Kochetov
c4832fd3c0 Added test. 2021-06-07 21:24:32 +03:00
Ivan
579250168f
Merge branch 'master' into ast-table-identifier-2 2021-06-07 20:30:39 +03:00
Latysheva Alexandra
1a2cbe9275
Merge branch 'master' into alexelex-encryption-1 2021-06-07 21:18:54 +06:00
Amos Bird
840895509e
Fix using aggregate projection when group by without key 2021-06-07 22:44:42 +08:00
Nikolai Kochetov
0d2a839ca4 Fix tests. 2021-06-07 16:41:40 +03:00
alesapin
170c49db69
Merge pull request #24961 from ClickHouse/more_general_check_in_queue 
Trying to fix 'Tagging already tagged part'
 2021-06-07 11:52:05 +03:00
Alexey Milovidov
17962459f5 Merge branch 'master' into issue-16775 2021-06-06 02:18:28 +03:00
Nikolai Kochetov
397f6133e0 Refactor canConstantBeWrappedByMonotonicFunctions function. 2021-06-04 20:56:56 +03:00
alesapin
1e69128443 Trying to fix 'Tagging already tagged part' 2021-06-04 14:49:00 +03:00
Kruglov Pavel
8acf031c1b
Merge pull request #24398 from Avogar/fix-deltasum 
Fix logical error AggregateFunctionFactory returned nullptr
 2021-06-03 19:40:06 +03:00
Nikolai Kochetov
3e5a1cda60 Revert 2021-06-03 17:44:59 +03:00
Nikolai Kochetov
726e22ea1d Always return false for canConstantBeWrappedByMonotonicFunctions. 2021-06-03 16:26:04 +03:00
tavplubix
1aff716f18
Merge pull request #24777 from ClickHouse/fix_intersection_with_lost_part 
Do not crash on intersecting virtual parts
 2021-06-03 15:42:46 +03:00
alesapin
282e8b52c4
Merge pull request #24872 from ClickHouse/fix_background_pool 
Paranoid fix for background pool task decrement
 2021-06-03 11:43:58 +03:00
alesapin
4a1bf51ad1
Merge branch 'master' into limited_fetches 2021-06-02 23:01:47 +03:00
Ivan Lezhankin
791cbcf0b1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-02 18:50:37 +03:00
Ivan Lezhankin
84c23dc060 Merge branch 'master' into async-insert 2021-06-02 18:06:21 +03:00
Ivan Lezhankin
ae38b4a3f1 Merge remote-tracking branch 'upstream/master' into ast-table-identifier-2 2021-06-02 16:07:59 +03:00
Nikolai Kochetov
134ad18249
Merge pull request #24574 from ClickHouse/refactor-MergeTreeDataSelectExecutor 
Refactor MergeTreeDataSelectExecutor
 2021-06-02 13:17:54 +03:00
alesapin
68d251ec11 Paranoid fix for background pool task decrement 2021-06-02 12:40:03 +03:00
alesapin
1c0973162d
Merge pull request #24803 from ClickHouse/followupfix_zookeeper_timeouts 
Don't capture temporary references in ZooKeeper client callbacks.
 2021-06-02 01:08:17 +03:00
tavplubix
a4ea5783d5
Update ActiveDataPartSet.cpp 2021-06-01 20:03:52 +03:00
Alexander Tokmakov
7d7c5638a5 Merge branch 'master' into fix_intersection_with_lost_part 2021-06-01 16:32:59 +03:00
tavplubix
275a616bb6
Update ActiveDataPartSet.cpp 2021-06-01 16:27:09 +03:00
Alexander Tokmakov
5969891611 do not crash on intersecting parts 2021-06-01 16:25:23 +03:00
Nikolai Kochetov
e615858b10 Merge branch 'master' into refactor-MergeTreeDataSelectExecutor 2021-06-01 16:25:04 +03:00
Alexander Tokmakov
cdd46aa117 Revert "try fix intersecting virtual parts" 
This reverts commit 2571ad7d43.
 2021-06-01 14:52:25 +03:00
Ivan Lezhankin
365e52817b More fixes due to "in" function arguments being incorrectly checked as ASTIdentifier 2021-06-01 14:20:03 +03:00
alexey-milovidov
c7bd3b8674
Merge pull request #23315 from CurtizJ/empty-parts-fix 
Fix waiting while dropping empty parts
 2021-06-01 13:52:49 +03:00
Nikolai Kochetov
b54ff696e5 Merge branch 'master' into refactor-MergeTreeDataSelectExecutor 2021-06-01 12:08:37 +03:00
Nikolai Kochetov
d4cbce3761
Merge pull request #24782 from amosbird/projection-fix7 
Fix empty part set with force_use_projection = 1
 2021-06-01 11:36:50 +03:00
alesapin
4e038b2a61 Remove copy-paste and rename (review fixes) 2021-06-01 10:30:06 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs 
Poco::File to std::filesystem
 2021-05-31 23:17:02 +03:00
Nikolai Kochetov
b9aa404508 Review fixes. 2021-05-31 21:25:43 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
alesapin
2fe75e33ee Replace Poco::Event with std future in synchronous calls in ZooKeeper client 2021-05-31 16:03:30 +03:00
Amos Bird
a206b76e5b
Fix empty part set with force_use_projection = 1 2021-05-31 10:31:45 +08:00
Alexander Tokmakov
2571ad7d43 try fix intersecting virtual parts 2021-05-31 00:30:50 +03:00
Alexander Tokmakov
16647fe8ce some unrelated fixes 2021-05-31 00:29:37 +03:00
kssenii
2a631aaf08 Final fixes 2021-05-29 00:34:44 +03:00
Nikolai Kochetov
58fbc544cc Add more comments. 2021-05-28 20:16:09 +03:00
Nikolai Kochetov
295a302bc8 Remove settings from ReadFromMergeTree. 2021-05-28 17:34:02 +03:00
Pavel Kruglov
42d57138f2 replace all isDateOrDateTime invocations to explicit calls to isDate and isDateTime and isDateTime64 2021-05-28 15:34:49 +03:00
Nikolai Kochetov
91e8c2c75d Remove MergingFinal step. 2021-05-28 12:52:33 +03:00
Nikolai Kochetov
94f1ac5a16 Remove some commented code. 2021-05-28 12:41:07 +03:00
Nikolai Kochetov
c80ab6166a Fix style. 2021-05-28 12:26:44 +03:00
Nikolai Kochetov
4e28b7cb02 Fix _sample_factor column. 2021-05-28 12:23:46 +03:00
Anton Popov
179f172c4c fix build 2021-05-28 02:10:44 +03:00
Anton Popov
336abfa475 Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-28 01:58:30 +03:00
Nikolai Kochetov
936bd8a011 Merge branch 'master' into refactor-MergeTreeDataSelectExecutor 2021-05-27 20:47:39 +03:00
Nikolai Kochetov
1aeb705b20 Fix some tests. 2021-05-27 19:53:58 +03:00
Nikolai Kochetov
cbdf3752ef Part 3. 2021-05-27 16:40:33 +03:00
kssenii
a0d8d689d3 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-27 15:44:12 +03:00
alesapin
6924dd51a2 Merge branch 'master' into limited_fetches 2021-05-27 11:17:36 +03:00
alesapin
ef17daaaa0
Merge pull request #24496 from ClickHouse/configurable_max_parts_to_merge 
Add `max_parts_to_merge_at_once` setting
 2021-05-27 10:56:58 +03:00
kssenii
b28157a0df Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-27 00:01:25 +03:00
alesapin
42b2383e01 Basic throttler implementation 2021-05-26 23:37:44 +03:00
kssenii
73f16ee9ee Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-26 23:08:08 +03:00
Nikolai Kochetov
a51a6ea0b7 Part 2. 2021-05-26 21:14:43 +03:00
alesapin
8ba6ed3fc2 Simplier test 2021-05-26 16:53:05 +03:00
tavplubix
a21e4b333e
Update ReplicatedMergeTreeQueue.cpp 2021-05-26 15:10:38 +03:00