Commit Graph

2628 Commits

Author SHA1 Message Date
Anton Popov
3a0d4807a5 Merge branch 'async-insert' of git://github.com/abyss7/ClickHouse into merging-20557 2021-08-06 16:20:02 +03:00
Alexey Milovidov
cea9690327 Merge branch 'master' into async-reads 2021-08-05 23:58:57 +03:00
alexey-milovidov
1a3d8cebd2
Merge pull request #27197 from azat/nullable-index-fix
Fix on-disk format breakage for secondary indices over Nullable column
2021-08-05 21:52:18 +03:00
Kruglov Pavel
59a94bd322
Merge pull request #27183 from ClickHouse/fix_detached_parts_name_partsing
Fix part name parsing in system.detached_parts
2021-08-05 18:21:03 +03:00
Alexander Kuzmenkov
d852207b0e
Merge pull request #25721 from CurtizJ/read-in-order-perf-2
Reduce memory usage in queries with `ORDER BY primary_key
2021-08-05 16:13:48 +03:00
terrylin
dd539f0a0d improvement of materilize ttl 2021-08-05 15:33:02 +08:00
terrylin
bd3d9a4518 materialize ttl recalculate only (optional) 2021-08-05 14:17:48 +08:00
Azat Khuzhin
dee27fcbb9 Fix on-disk format breakage for secondary indices over Nullable column
[1] breaks on disk format (and the relevant change in the:

  [1]: https://github.com/ClickHouse/ClickHouse/pull/12455#discussion_r682830812

Too bad that I checked this patchset only for compatibility after
reverting this patch [2] (use case: I've applied it manually, then
revert it, and data skipping indexes over Nullable column had been
broken)

  [2]: https://github.com/ClickHouse/ClickHouse/pull/12455#issuecomment-823423772

But this patchset actually breaks compatibility with older versions of
clickhouse for Nullable data skipping indexes after simple upgrade:

Here is a simple reproducer:

    --
    -- run this with 21.6 or similar (i.e. w/o this patch)
    --

    CREATE TABLE data
    (
        `key` Int,
        `value` Nullable(Int),
        INDEX value_index value TYPE minmax GRANULARITY 1
    )
    ENGINE = MergeTree
    ORDER BY key;

    INSERT INTO data SELECT
        number,
        number
    FROM numbers(10000);

    SELECT * FROM data WHERE value = 20000 SETTINGS force_data_skipping_indices = 'value_index' SETTINGS force_data_skipping_indices = 'value_index', max_rows_to_read=1;

Now upgrade and run the query again:

    SELECT * FROM data WHERE value = 20000 SETTINGS force_data_skipping_indices = 'value_index' SETTINGS force_data_skipping_indices = 'value_index', max_rows_to_read=1;

And it will fail because of on disk format changes:

    $ ll --time-style=+ data/*/data/all_1_1_0/skp*.idx
    -rw-r----- 1 azat azat 36  data/with_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
    -rw-r----- 1 azat azat 37  data/without_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx

    $ md5sum data/*/data/all_1_1_0/skp*.idx
    a19c95c4a14506c65665a1e30ab404bf  data/with_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
    e50e2fcfa873b232196623d56ab26105  data/without_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx

Note, that there is no stable release with this patch included yet, so
no need to backport.

Also note that you may create data skipping indexes over Nullable
column even before [3].

  [3]: https://github.com/ClickHouse/ClickHouse/pull/12455

v2: break cases when granulas has Null in values due to backward
compatibility
2021-08-05 00:19:38 +03:00
Anton Popov
9586bb7e90 fix pvs 2021-08-04 20:42:22 +03:00
terrylin
6f42ec6b9b fix compile error 2021-08-05 00:16:21 +08:00
Anton Popov
915ef50c2d optimize reading in order of key more 2021-08-04 18:18:03 +03:00
Alexander Tokmakov
23f8b3d07d fix part name parsing in system.detached_parts 2021-08-04 17:42:48 +03:00
terrylin
88b9e7f24f materialize ttl recalculate only optional 2021-08-04 22:16:13 +08:00
Anton Ivashkin
61016da2b1 Set allow_remote_fs_zero_copy_replication to true by default 2021-08-04 17:08:09 +03:00
Yatsishin Ilya
55e06ea16b fix 2021-08-04 13:06:01 +03:00
Alexey Milovidov
fd3b5680e2 Merge branch 'master' into async-reads 2021-08-03 23:13:31 +03:00
Anton Popov
fc9d72e75d fix performance of short queries with large number of columns 2021-08-02 15:03:55 +03:00
Nikolai Kochetov
6951e8147d
Merge pull request #27020 from amosbird/shardlevelconstness
Shard-level const column
2021-08-02 13:45:20 +03:00
Anton Popov
cc3ed12ed6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-02 02:56:00 +03:00
Alexey Milovidov
2a0cba2b9f Merge branch 'master' into async-reads 2021-08-01 01:25:13 +03:00
alexey-milovidov
701e2ffd79
Merge pull request #27002 from azat/mutation-stuck
Fix possible mutation stuck due to race with DROP_RANGE
2021-07-31 21:17:46 +03:00
Amos Bird
cd302eacc1
Fix projection match 2021-07-31 15:44:16 +08:00
tavplubix
00ac8194dc
Merge pull request #26896 from ClickHouse/small_improvement_in_queue
Maybe fix extremely rare `intersecting parts`.
2021-07-30 13:04:20 +03:00
terrylin
c49c2e5e18 fix compile error 2021-07-30 17:15:04 +08:00
mergify[bot]
41273ef5f3
Merge branch 'master' into mutation-stuck 2021-07-30 08:42:45 +00:00
Azat Khuzhin
bc9a4366d8 Fix possible mutation stack due to race with DROP_RANGE
After #25884 parts that was dropped (DROP_RANGE) will not be removed
from old mutations and this will stuck the mutation.

Interesting, that this mutation may continue after some merge.
2021-07-30 08:22:10 +03:00
terrylin
55f3e7e65a improvement of materilize ttl 2021-07-30 12:27:15 +08:00
alesapin
21599ffefc Simple validation for partition id before drop partition 2021-07-29 16:11:05 +03:00
alesapin
5799487caa Fix stupid bug 2021-07-29 14:54:36 +03:00
alesapin
bdc6858801 More correct comparsion 2021-07-28 13:19:34 +03:00
alesapin
26056cfb25 Fix extremely rare race condition on remove of unsuccessful entry from queue 2021-07-28 13:06:22 +03:00
Alexey Milovidov
9baf84224a Fix build 2021-07-28 02:54:20 +03:00
Alexey Milovidov
fd5c1561e2 Merge branch 'master' into async-reads 2021-07-27 23:58:32 +03:00
Nicolae Vartolomei
8b07a7f180 Store exception generated when we tried to update the queue last time
The use case is to alert when queue contains broken entries. Especially
important when ClickHouse breaks backwards compatibility between
versions and log entries written by newer versions aren't parseable by
old versions.

```
Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected 'quorum: ' before: 'merge_type: 2\n'
```
2021-07-27 15:42:40 +01:00
tavplubix
168edaed73
Merge pull request #26808 from amosbird/safe2
Make sure table is readonly when restarting fails.
2021-07-27 17:21:08 +03:00
Nikolai Kochetov
97bc754dd8
Merge pull request #26758 from ClickHouse/output-streams-to-processors
Remove some output streams
2021-07-27 17:19:26 +03:00
alexey-milovidov
d7b2404119
Merge pull request #15408 from ClickHouse/random_timezone_for_stateless_tests
Use random timezone in stateless and stateful tests
2021-07-27 16:18:14 +03:00
Alexey Milovidov
5e2ea4e042 Whitespace 2021-07-27 05:20:03 +03:00
Nikolai Kochetov
61d8f880cd Rename some files. 2021-07-26 19:48:25 +03:00
Nikolai Kochetov
9b5a816b43 Merge branch 'master' into output-streams-to-processors 2021-07-26 18:03:11 +03:00
Nikolai Kochetov
0eb563dc1b Fix more tests. 2021-07-26 17:47:29 +03:00
Amos Bird
623faf47e4
Make sure table is readonly when restarting fails. 2021-07-26 19:21:14 +08:00
Nikolai Kochetov
fa1c223269 Fix some tests. 2021-07-26 13:08:40 +03:00
Alexey Milovidov
5c3f5cb35a Experiment with asynchronous readers 2021-07-26 03:34:36 +03:00
Nikolai Kochetov
9c92f43359 Update storages. 2021-07-23 22:33:59 +03:00
Azat Khuzhin
00e2083421 Fix event_time_microseconds for REMOVE_PART in system.part_log 2021-07-23 00:59:08 +03:00
alexey-milovidov
fcca3e3e56
Merge pull request #26423 from ClickHouse/fix-header-for-scalar-query-with-empty-result
Fix incompatible result type for scalar queries with empty result.
2021-07-21 02:07:04 +03:00
Anton Popov
fa6857920d fix arcadia 2021-07-19 16:45:35 +03:00
Nikolai Kochetov
7a24e72e76 Merge branch 'master' into fix-header-for-scalar-query-with-empty-result 2021-07-19 15:48:44 +03:00
Nikolai Kochetov
96e20e2641 Fix some tests. 2021-07-19 15:35:55 +03:00
Kseniia Sumarokova
b0ddc4fb30
Merge pull request #26364 from kssenii/fixing-date32
Continuation of #25774
2021-07-18 21:44:45 +03:00
alexey-milovidov
b52411a715
Merge pull request #12455 from amosbird/npc
Nullable primary key with correct KeyCondition
2021-07-18 17:52:20 +03:00
Alexey Milovidov
261a220227 Remove some code 2021-07-17 21:06:46 +03:00
Azat Khuzhin
071dc7c3f5 Improve "Not executing log entry queue-X of type MUTATE_PART" message 2021-07-17 11:50:04 +03:00
Alexey Milovidov
bd924a5b8a Whitespace 2021-07-17 01:59:39 +03:00
alexey-milovidov
c373b86743
Merge pull request #26371 from CurtizJ/improve-perf-get-columns
Improve performance of getting list of columns
2021-07-17 01:34:15 +03:00
Anton Popov
048e089a58 remove useless code 2021-07-16 17:47:03 +03:00
Anton Popov
03c785931a fix pushdown of limit to reading stage 2021-07-16 17:27:38 +03:00
Anton Popov
c45a9e3bd6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-16 17:11:34 +03:00
Anton Popov
9e3af27f40 better performance of getSampleBlockForColumns 2021-07-15 20:36:48 +03:00
Anton Popov
5f71a6f5bb improve performance of getting columns list 2021-07-15 16:29:01 +03:00
kssenii
3ba2d114d5 Revert "Merge pull request #26352 from ClickHouse/revert-25774-datatype-date32"
This reverts commit 94a210650f, reversing
changes made to 0893b9ff8e.
2021-07-15 11:41:52 +00:00
alexey-milovidov
9ba4064c86
Revert "Datatype Date32, support range 1925 to 2283" 2021-07-15 10:53:35 +03:00
alexey-milovidov
0893b9ff8e
Merge pull request #26218 from amosbird/projection-improve1
Aggressive IN index analysis for projections.
2021-07-15 08:13:45 +03:00
Filatenkov Artur
f1702d356e
Merge pull request #26120 from lthaooo/settings-merge_selecting_sleep_ms
add merge_selecting_sleep_ms setting
2021-07-14 18:28:47 +03:00
Amos Bird
28b567a01b
Update 2021-07-14 23:09:17 +08:00
alexey-milovidov
55fc7323fd
Update MergeTreeDataSelectExecutor.h 2021-07-14 22:56:52 +08:00
alexey-milovidov
1ef068596c
Update MergeTreeDataSelectExecutor.cpp 2021-07-14 22:56:52 +08:00
Amos Bird
0558ecdc3f
Aggressive IN index analysis for projections. 2021-07-14 22:56:52 +08:00
Anton Popov
e2249bf1e5
Merge pull request #26256 from zxc111/issue-822
Check hash function at table creation
2021-07-14 14:16:02 +03:00
Zhichang Yu
fe5d17680e Merge remote-tracking branch 'origin/master' into zero_copy_hdfs 2021-07-14 02:12:28 +00:00
Maksim Kita
3d05f07ece
Merge pull request #25917 from excitoon-favorites/defaultblocksizeprediction
Using required columns for block size prediction
2021-07-13 20:05:41 +03:00
Vitaly Baranov
3c395389b0
Merge pull request #24206 from alexelex/alexelex-encryption-1
Data encryption on VFS level
2021-07-13 19:50:25 +03:00
Anton Popov
cb042afa9d pushdown limit while reading in order of primary key 2021-07-13 17:24:45 +03:00
zxc111
7b2fed2951 use flag to check sample_condition 2021-07-13 21:17:22 +08:00
Anton Popov
19d0f1bbfc Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-13 16:12:08 +03:00
Ivan Lezhankin
ab26aed6f9 Merge remote-tracking branch 'upstream/master' into async-insert 2021-07-13 13:54:09 +03:00
xiedeyantu
df438f4288 Modify code comments 2021-07-13 17:52:24 +08:00
Vitaly Baranov
e086e5e384
Merge branch 'master' into alexelex-encryption-1 2021-07-13 11:28:41 +03:00
Zhichang Yu
b4e6689bf9 fix test_hdfs_zero_copy_replication_move[tiered_copy-2] 2021-07-13 07:20:23 +00:00
Zhichang Yu
5047c758f4 fix per review 2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1 hdfs zero copy 2021-07-13 07:19:12 +00:00
alexey-milovidov
4b1e38eed9
Merge branch 'master' into projection-fix11 2021-07-13 04:52:15 +03:00
alexey-milovidov
5295f7ebb2
Merge pull request #26235 from ClickHouse/check-non-deterministic-functions-in-keys
Check for non-deterministic functions in keys, including constant expressions
2021-07-13 02:46:44 +03:00
Kseniia Sumarokova
4a6b4d6c26
Merge pull request #25774 from liuneng1994/datatype-date32
Datatype Date32, support range 1925 to 2283
2021-07-12 20:33:39 +03:00
zxc111
e0261866a4 add settings && add test case 2021-07-13 01:15:35 +08:00
mergify[bot]
b7ae4e4ced
Merge branch 'master' into async-insert 2021-07-12 09:21:37 +00:00
Alexey Milovidov
2a2aa69459 Handle non-existent "detached" directories 2021-07-12 10:01:26 +03:00
Alexey Milovidov
ff5c433f10 Check for non-deterministic functions in keys, including constant expressions 2021-07-12 08:58:19 +03:00
alexey-milovidov
8f1d7e67ca
Merge pull request #26003 from ClickHouse/ditch-aio
Remove AIO for sequential reads
2021-07-12 03:25:23 +03:00
zxc111
cd97478c06 check sampling condition when create table 2021-07-12 00:38:20 +08:00
Alexey Milovidov
1ce40d6629 Fix bad code (default function argument) 2021-07-11 04:01:23 +03:00
Alexey Milovidov
81646f8389 Merge branch 'master' into ditch-aio 2021-07-11 03:06:45 +03:00
terrylin
faed6263bb add merge_selecting_sleep_ms setting 2021-07-09 17:29:17 +08:00
Anton Popov
9f1ffa777f remove unused code 2021-07-08 15:06:33 +03:00
alesapin
4c85dae572
Merge pull request #25743 from ClickHouse/fix_aggregation_ttl
Fix bug in execution of TTL GROUP BY
2021-07-07 10:49:16 +03:00
alesapin
0d8844c510
Merge pull request #25884 from ClickHouse/fix_drop_part_in_queue
Relax `DROP PART` guarantees and turn on checks in ReplicationQueue.
2021-07-07 10:48:48 +03:00
mergify[bot]
446474932b
Merge branch 'master' into projection-fix11 2021-07-07 05:02:44 +00:00
Amos Bird
55981cb0ae
Fix normal projection 2021-07-07 13:01:30 +08:00
alesapin
3dee74df54 Comment 2021-07-06 19:53:54 +03:00
alesapin
a8fdc41193 Fix bug and add more trash to test 2021-07-06 19:51:23 +03:00
mergify[bot]
7ff8ac51a9
Merge branch 'master' into fix-merge-tree-settings-change 2021-07-06 12:14:33 +00:00
alesapin
75e26b93d0 Review bug fixes 2021-07-06 15:05:58 +03:00
alesapin
e9540f0621 Remove debug logs 2021-07-06 14:13:35 +03:00
mergify[bot]
8dd378f7c2
Merge branch 'master' into fix_drop_part_in_queue 2021-07-06 11:00:04 +00:00
alesapin
53b23775a9 Fix drop part 2021-07-06 13:58:53 +03:00
alesapin
2600922786 Followup fix 2021-07-06 11:36:39 +03:00
Alexey Milovidov
21fb6ddea5 Remove AIO for sequential reads 2021-07-05 23:02:24 +03:00
alesapin
f7e1cfdb24 Some partially working code 2021-07-05 22:58:55 +03:00
Anton Popov
9071ecd428 fix alter of settings in MergeTree 2021-07-05 15:44:58 +03:00
NengLiu
8b88988618 Merge branch 'master' into datatype-date32
# Conflicts:
#	src/DataStreams/PostgreSQLBlockInputStream.cpp
2021-07-05 10:18:38 +08:00
NengLiu
dc0c65ef4f Merge remote-tracking branch 'origin/master' into datatype-date32 2021-07-05 10:15:08 +08:00
alexey-milovidov
fef7c557f6
Merge pull request #24789 from kitaisreal/compile-aggregation-functions
Compile aggregate functions
2021-07-05 02:57:49 +03:00
mergify[bot]
1559c09977
Merge branch 'master' into fix_drop_part_in_queue 2021-07-04 16:00:39 +00:00
alesapin
cf2fc94d9d
Merge pull request #25820 from ClickHouse/better_remove_empty_parts
Replace lost parts with empty parts instead of hacking replication queue
2021-07-04 18:55:49 +03:00
Vladimir Chebotarev
e5cbd82879 Took PREWHERE into account. 2021-07-04 18:06:15 +03:00
alexey-milovidov
51bf60a938
Merge pull request #17769 from xjewer/alex/remove_engine_setting
CLICKHOUSE-784: reset merge tree setting
2021-07-03 18:27:15 +03:00
alexey-milovidov
1dda771864
Merge pull request #25554 from CurtizJ/fix-alter-ttl-column
Fix `ALTER MODIFY COLUMN` of columns, which participates in TTL expressions
2021-07-03 18:26:37 +03:00
mergify[bot]
9da1c98998
Merge branch 'master' into fix_aggregation_ttl 2021-07-03 15:07:44 +00:00
alesapin
7b4a56977d
Merge pull request #25893 from ClickHouse/no_backoff_when_pool_is_full
Reset counter if background pool is full
2021-07-03 18:03:29 +03:00
mergify[bot]
fbe1ff0fb4
Merge branch 'master' into fix_drop_part_in_queue 2021-07-03 15:02:15 +00:00
mergify[bot]
a2183dff39
Merge branch 'master' into better_remove_empty_parts 2021-07-03 15:01:11 +00:00
mergify[bot]
02cb287dc9
Merge branch 'master' into fix-alter-ttl-column 2021-07-02 23:49:46 +00:00
alesapin
b4b54a0549 Fix bug 2021-07-02 18:25:05 +03:00
Vladimir Chebotarev
6f5a72c963 Using required columns for block size prediction. 2021-07-02 16:55:00 +03:00
alesapin
72b84141e1 Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-07-02 12:41:28 +03:00
alesapin
f8f734d3dc Better comment 2021-07-02 12:40:13 +03:00
alesapin
abc3aca19b Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-07-02 12:30:54 +03:00
alesapin
2e29dc2975 More safe empty parts creation 2021-07-02 12:30:17 +03:00
alesapin
5e79c68e8e Debug 2021-07-02 11:54:22 +03:00
Alexey Milovidov
0e621788c7 Allow constexpr parameters for aggregate functions 2021-07-02 03:53:08 +03:00
Maksim Kita
da8c957167 Aggregator added CompiledExpressionCache 2021-07-01 22:56:36 +03:00
Maksim Kita
d24d3ae992 Added second variant of compilation 2021-07-01 22:56:36 +03:00
Maksim Kita
3fe559b31f Compile aggregate functions 2021-07-01 22:56:35 +03:00
Aleksei Semiglazov
9a5365fc41
CLICKHOUSE-784: reset merge tree setting
Add an ability to reset custom setting to default and remove it from table's metadata.
This will allow to rollback the change without knowing the system/config's default.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>
2021-07-01 20:09:54 +01:00
alesapin
7792b09911 Reset counter if background pool is full 2021-07-01 16:10:07 +03:00
alesapin
1a4ccab8e6 Fix style 2021-07-01 15:12:27 +03:00
mergify[bot]
98fa9f7951
Merge branch 'master' into better_remove_empty_parts 2021-07-01 11:32:43 +00:00
tavplubix
afbc6bf17d
Merge pull request #25684 from ClickHouse/cancel_merges_on_drop_partition
Cancel merges on DROP PARTITION
2021-06-30 23:41:04 +03:00
alesapin
c845a205f9 Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 22:42:02 +03:00
alesapin
45edc8c861 Fix tidy 2021-06-30 22:11:00 +03:00
alesapin
89a401a84f Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 18:53:17 +03:00
alesapin
6a73c8b49e Review fixes 2021-06-30 18:24:51 +03:00
tavplubix
c958f876b8
Update MergeList.h 2021-06-30 15:55:26 +03:00
alesapin
21f15be990 Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 15:30:32 +03:00
alesapin
2eb27540b2 Some test version 2021-06-30 15:29:09 +03:00
alesapin
94d4d32e4c
Merge pull request #25783 from ClickHouse/followup_drop_part_fix
Relax check during DROP PART
2021-06-30 12:25:42 +03:00
mergify[bot]
1799804243
Merge branch 'master' into fix_aggregation_ttl 2021-06-30 09:15:46 +00:00
mergify[bot]
8182b6e890
Merge branch 'master' into better_remove_empty_parts 2021-06-30 08:42:26 +00:00
alesapin
0af53ace55 Fix comment 2021-06-29 22:49:41 +03:00
alesapin
ca1d21b663
Update src/Storages/MergeTree/MergeTreeData.cpp
Co-authored-by: tavplubix <avtokmakov@yandex-team.ru>
2021-06-29 22:48:46 +03:00
alesapin
a6834213a1 Add tests 2021-06-29 22:47:54 +03:00
Nikolai Kochetov
d5bd2b1fa9
Merge pull request #25719 from ClickHouse/remove-PrewhereDAGInfo
Remove PrewhereDAGInfo.
2021-06-29 18:58:03 +03:00
alesapin
5822f0ba29 Replace lost parts with empty parts instead of hacking replication queue 2021-06-29 18:14:44 +03:00
Nikolai Kochetov
bf827936b7 Rename PrewhereActions 2021-06-29 14:53:34 +03:00
alesapin
7680eab0e4 one more fix 2021-06-29 13:03:00 +03:00
alesapin
941a6e539f Fix 2021-06-29 10:36:20 +03:00
Ivan Lezhankin
1bb41593d1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-28 22:55:56 +03:00
alesapin
72b281987e Add more debug 2021-06-28 17:15:51 +03:00
alesapin
d20feeb42c Relax check during DROP PART 2021-06-28 15:05:22 +03:00
alesapin
b11254d191
Merge branch 'master' into fix_aggregation_ttl 2021-06-28 13:31:29 +03:00
alesapin
4c213b639f
Merge pull request #25548 from nikitamikhaylov/background-processing
A little improvement in BackgroundJobsExecutor
2021-06-28 13:24:21 +03:00
alexey-milovidov
a11351a473
Merge pull request #25484 from vdimir/issue-19589
Fix assert with non uint8 in prewhere
2021-06-28 00:03:12 +03:00
alesapin
c977c33d6d Fix bug in execution of TTL GROUP BY 2021-06-27 19:18:15 +03:00
Anton Popov
2165ff679a
Merge branch 'master' into read-in-order-perf-2 2021-06-27 17:45:00 +03:00
alexey-milovidov
6662e7ab0f
Merge pull request #25732 from matwey/missed_include_optional_2
Fix some more missed includes
2021-06-26 14:04:06 +03:00
Matwey V. Kornilov
a29bd2c4b3 Add missed #include <limits>
Signed-off-by: Matwey V. Kornilov <matwey.kornilov@gmail.com>
2021-06-26 11:25:54 +03:00
Anton Popov
fafae3ab8d fix tests 2021-06-25 20:00:23 +03:00
Anton Popov
fa8ad1a7c6 reduce memory usage in queries with 'ORDER BY primary_key' 2021-06-25 19:23:41 +03:00
Ivan
14e43cfafa
Add CI check for darwin-aarch64 (#25560)
* Add support for darwin-aarch64

* Fix PVS warnings

* Add build configuration

* Fix packager args
2021-06-25 18:21:56 +03:00
Nikolai Kochetov
3544b059ce Merge branch 'master' into remove-PrewhereDAGInfo 2021-06-25 17:52:03 +03:00
Nikolai Kochetov
6bc0a628cd Remove PrewhereDAGInfo. 2021-06-25 17:49:28 +03:00
alesapin
f6afc381be
Merge pull request #25665 from ClickHouse/fix_24869
Fix REPLACE_RANGE entry removal
2021-06-25 11:11:25 +03:00
alesapin
d138ba79c7
Merge pull request #25549 from ClickHouse/add_test_for_concurrent_merges
Fix bug in TTL merges assignment
2021-06-25 11:02:29 +03:00
Alexander Tokmakov
76156af5cc cancel merges on drop partition 2021-06-24 17:07:43 +03:00
Ivan
e9e49a9085
Fix ANTLR parser and enable it back in CI (#25638)
* Fix

* Skip more tests
2021-06-24 14:45:29 +03:00
Alexander Tokmakov
7e6963bb39 fix REPLACE_RANGE entry removal 2021-06-24 11:16:54 +03:00
alesapin
e6da53dc46 better 2021-06-24 00:09:06 +03:00
alesapin
24876291d1 Followup fix 2021-06-24 00:01:07 +03:00
alesapin
4be4bc21e2 Fix for fix 2021-06-23 23:57:49 +03:00
alesapin
81c74435a3 Fix drop part bug 2021-06-23 22:25:30 +03:00
Nikita Mikhaylov
f4d6a6e128 support dynamic reloading of server config 2021-06-22 23:24:47 +00:00
Nikita Mikhaylov
a39bca0acf better 2021-06-22 23:24:47 +00:00
Nikita Mikhaylov
c66a3b22b5 done 2021-06-22 23:24:47 +00:00
mergify[bot]
c15493f0df
Merge branch 'master' into add_test_for_concurrent_merges 2021-06-22 15:49:35 +00:00
alesapin
47b29092b4 Fix bug in MergerMutator parts selector 2021-06-22 16:47:42 +03:00
Neng Liu
7ed1728a37 Merge branch 'master' into datatype-date32
# Conflicts:
#	src/DataTypes/IDataType.h
#	src/Functions/CustomWeekTransforms.h
2021-06-22 10:00:18 +08:00
Anton Popov
662d5b8495 fix ALTER MODIFY COLUMN of columns, that participates in TTL expressions 2021-06-21 18:20:18 +03:00
vdimir
46a5dd6701
Fix MergeTreeBaseSelectProcessor::executePrewhereActions 2021-06-21 16:01:02 +03:00
Anton Popov
ffa56bde24 fix usage of index with array columns and ARRAY JOIN 2021-06-21 15:34:05 +03:00
vdimir
eb85c5a8e7
Fix assert with non uint8 in prewhere 2021-06-21 15:32:29 +03:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652)
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
2021-06-20 11:24:43 +03:00
Amos Bird
f2ed5ef42b
Nullable primary key with correct KeyCondition 2021-06-18 23:04:24 +08:00
Latysheva Alexandra
48d3419160
Merge branch 'master' into alexelex-encryption-1 2021-06-18 11:28:08 +06:00
Ivan Lezhankin
37365589ed Merge branch 'master' into async-insert 2021-06-17 16:57:45 +03:00
Neng Liu
b6eab5fd2f add datetype32 function support 2021-06-17 16:05:30 +08:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Kseniia Sumarokova
d6240f2bd4
Merge pull request #25309 from azat/fs-fix
Catch ErrnoException during parts cleaning
2021-06-16 22:03:54 +03:00
tavplubix
07ce932282
Merge pull request #25306 from ClickHouse/tavplubix-patch-2
Forcefully stop queue tasks on restarting
2021-06-16 14:22:36 +03:00
Nikolai Kochetov
8bf55ae2b5
Merge pull request #25166 from ClickHouse/always-detach-parts-with-wrong-partition-id
Always detach parts with wrong partition ID
2021-06-16 11:06:07 +03:00
Kseniia Sumarokova
e08c05cdf5
Merge pull request #25295 from ClickHouse/remove-some-code-from-key-condition
Remove some code from KeyCondition.
2021-06-16 10:12:12 +03:00
Azat Khuzhin
de9689379f Catch ErrnoException during parts cleaning
Avoids noisy messages like:

    2021.06.15 09:27:35.546982 [ 58176 ] {} <Error> auto DB::IBackgroundJobExecutor::jobExecutingTask()::(anonymous class)::operator()() const: Code: 481, e.displayText() = DB::ErrnoException: Cannot check modification time for file: /var/lib/clickhouse/store/c52/c52619f8-99bd-435e-8526-19f899bdb35e/tmp_insert_202106_7010_7010_0/, errno: 2, strerror: No such file or directory, Stack trace (when copying this message, always include the lines below):

    0. DB::Exception::Exception() @ 0x903209a in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    1. DB::throwFromErrnoWithPath() @ 0x9033089 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    2. FS::getModificationTime() @ 0x907b166 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    3. DB::DiskLocal::getLastModified() @ 0xf9a6669 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    4. DB::MergeTreeData::clearOldTemporaryDirectories(long) @ 0x1069a6de in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
2021-06-16 06:56:55 +03:00
tavplubix
60c8dc1c0b
Update ReplicatedMergeTreeRestartingThread.cpp 2021-06-16 00:48:38 +03:00
Nikolai Kochetov
96d98ff020 Add comment 2021-06-15 21:42:26 +03:00
mergify[bot]
7959d92029
Merge branch 'master' into minor-changes-3 2021-06-15 18:07:24 +00:00
Nikolai Kochetov
80a13c489b Revert back moduloLegacy check for canConstantBeWrappedByFunctions. 2021-06-15 18:21:31 +03:00
Nikolai Kochetov
481b87b37a Remove some code from keyCondition. 2021-06-15 16:47:37 +03:00
Nikita Mikhaylov
a52bba91b7
Merge pull request #16401 from abyss7/ast-table-identifier-2
ASTTableIdentifier part #2: Introduce ASTTableIdentifier
2021-06-15 13:51:30 +03:00
tavplubix
e2ecc51a1f
Merge pull request #25087 from ClickHouse/srmt_remove_copypaste
Remove copypaste from StorageReplicatedMergeTree
2021-06-15 12:43:55 +03:00
Alexey Milovidov
4e982a3ae6 Merge branch 'master' into minor-changes-3 2021-06-15 00:45:45 +03:00
Alexey Milovidov
f49463d1e5 Fix clang-tidy 2021-06-14 07:28:17 +03:00
Alexey Milovidov
447d7bb8cd Minor changes 2021-06-14 07:13:35 +03:00
Alexey Milovidov
2211ba53d4 Copy paste some code 2021-06-14 05:48:10 +03:00
mergify[bot]
ab2a1a410f
Merge branch 'master' into test-key-condition-mon 2021-06-12 00:32:38 +00:00
alexey-milovidov
30f0124e77
Merge branch 'master' into always-detach-parts-with-wrong-partition-id 2021-06-12 02:55:55 +03:00
alexey-milovidov
eb7b491da9
Merge pull request #24900 from Enmk/bloom_filter_hasAny
Added support of hasAny function to bloom_filter index.
2021-06-12 00:58:25 +03:00
Ivan Lezhankin
2c7cb41a69 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-11 20:04:26 +03:00
Vasily Nemkov
2ac288c303 index is not used if types in hasAny are unsuported
updated tests
2021-06-11 18:29:52 +03:00
Alexander Tokmakov
73ff1728ae rename flag to more generic name 2021-06-11 15:41:48 +03:00
Nikolai Kochetov
79c69ea316 Fix build. 2021-06-11 11:29:39 +03:00
Vasily Nemkov
55064e49a7 Fixed the test
Previous test version wasn't causing bug on unfixed code, this new one does.
2021-06-10 19:09:43 +03:00
Nikolai Kochetov
fea19c2037
Merge pull request #25163 from amosbird/projection-fix10
Disable projection when final is used in select query
2021-06-10 18:44:29 +03:00
Vasily Nemkov
0ab28360c9 Added support of hasAny function to bloom_filter index.
Also fixed bug discovered by AST fuzzer.
2021-06-10 18:22:04 +03:00
Ivan
a7fa4e641c
Merge branch 'master' into ast-table-identifier-2 2021-06-10 16:43:22 +03:00
Nikolai Kochetov
6222fcffe0 Never remove parts on start at all. Always move. 2021-06-10 15:47:55 +03:00
Nikolai Kochetov
aa90309abf Never remove parts on start at all. Always move. 2021-06-10 15:46:03 +03:00
Nikolai Kochetov
2936fdd16c Always detach parts with wrong partition id. 2021-06-10 15:04:36 +03:00
Nikolai Kochetov
847dfb9914 Merge remote-tracking branch 'origin' into always-detach-parts-with-wrong-partition-id 2021-06-10 12:37:46 +03:00
Nikolai Kochetov
d52b87d47d Always detach proken parts with wrong partition id. 2021-06-10 12:37:33 +03:00
Amos Bird
4e358b28f1
Disable projection with final is used 2021-06-10 17:11:17 +08:00
Alexey Milovidov
6c5b2b827c Move code to more appropriate place 2021-06-10 02:58:51 +03:00
alexey-milovidov
0636f40349
Merge pull request #25127 from ClickHouse/fix-partition-hash-for-uuid
Revert old UUID hash used for partition ID.
2021-06-10 02:31:01 +03:00
Nikolai Kochetov
12791431da Revert old UUID hash used for partition ID. 2021-06-09 16:51:40 +03:00
Alexander Tokmakov
cef22688ff make code less ugly 2021-06-09 15:36:47 +03:00
Alexander Tokmakov
3ade38df82 remove copypaste 2021-06-08 22:17:45 +03:00
alexey-milovidov
e25d9843d1
Merge pull request #25025 from azat/parallel-kill-mutation
Distinguish KILL MUTATION for different tables.
2021-06-08 21:40:53 +03:00
Nikolai Kochetov
6197d20c18
Update KeyCondition.cpp 2021-06-08 15:48:14 +03:00
Ivan Lezhankin
973cea73b7 Merge branch 'master' into async-insert 2021-06-08 15:10:30 +03:00
Azat Khuzhin
1062d0ec91 Distinguish KILL MUTATION for different tables.
Before this patch KILL MUTATION marks mutation as canceled just by name
(and part numbers) so if you have multiple tables with the same part
name, then killing mutation for one table, will mark it as killed for
another too.

Fix this by comparing StorageID too (it is better to use StorageID over
database/table to avoid ambiguity by using UUIDs for comparing).

Here is a failure of the 01414_freeze_does_not_prevent_alters on CI [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/24069/9fb69dcf98c71a939d200cad3c8491bf43a44622/functional_stateless_tests_(ubsan).html#fail1
2021-06-08 10:51:22 +03:00
Nikolai Kochetov
c4832fd3c0 Added test. 2021-06-07 21:24:32 +03:00
Ivan
579250168f
Merge branch 'master' into ast-table-identifier-2 2021-06-07 20:30:39 +03:00
Latysheva Alexandra
1a2cbe9275
Merge branch 'master' into alexelex-encryption-1 2021-06-07 21:18:54 +06:00
Amos Bird
840895509e
Fix using aggregate projection when group by without key 2021-06-07 22:44:42 +08:00
Nikolai Kochetov
0d2a839ca4 Fix tests. 2021-06-07 16:41:40 +03:00
alesapin
170c49db69
Merge pull request #24961 from ClickHouse/more_general_check_in_queue
Trying to fix 'Tagging already tagged part'
2021-06-07 11:52:05 +03:00
Alexey Milovidov
17962459f5 Merge branch 'master' into issue-16775 2021-06-06 02:18:28 +03:00
Nikolai Kochetov
397f6133e0 Refactor canConstantBeWrappedByMonotonicFunctions function. 2021-06-04 20:56:56 +03:00
alesapin
1e69128443 Trying to fix 'Tagging already tagged part' 2021-06-04 14:49:00 +03:00
Kruglov Pavel
8acf031c1b
Merge pull request #24398 from Avogar/fix-deltasum
Fix logical error AggregateFunctionFactory returned nullptr
2021-06-03 19:40:06 +03:00
Nikolai Kochetov
3e5a1cda60 Revert 2021-06-03 17:44:59 +03:00
Nikolai Kochetov
726e22ea1d Always return false for canConstantBeWrappedByMonotonicFunctions. 2021-06-03 16:26:04 +03:00
tavplubix
1aff716f18
Merge pull request #24777 from ClickHouse/fix_intersection_with_lost_part
Do not crash on intersecting virtual parts
2021-06-03 15:42:46 +03:00
alesapin
282e8b52c4
Merge pull request #24872 from ClickHouse/fix_background_pool
Paranoid fix for background pool task decrement
2021-06-03 11:43:58 +03:00
alesapin
4a1bf51ad1
Merge branch 'master' into limited_fetches 2021-06-02 23:01:47 +03:00
Ivan Lezhankin
791cbcf0b1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-02 18:50:37 +03:00
Ivan Lezhankin
84c23dc060 Merge branch 'master' into async-insert 2021-06-02 18:06:21 +03:00
Ivan Lezhankin
ae38b4a3f1 Merge remote-tracking branch 'upstream/master' into ast-table-identifier-2 2021-06-02 16:07:59 +03:00
Nikolai Kochetov
134ad18249
Merge pull request #24574 from ClickHouse/refactor-MergeTreeDataSelectExecutor
Refactor MergeTreeDataSelectExecutor
2021-06-02 13:17:54 +03:00
alesapin
68d251ec11 Paranoid fix for background pool task decrement 2021-06-02 12:40:03 +03:00
alesapin
1c0973162d
Merge pull request #24803 from ClickHouse/followupfix_zookeeper_timeouts
Don't capture temporary references in ZooKeeper client callbacks.
2021-06-02 01:08:17 +03:00
tavplubix
a4ea5783d5
Update ActiveDataPartSet.cpp 2021-06-01 20:03:52 +03:00
Alexander Tokmakov
7d7c5638a5 Merge branch 'master' into fix_intersection_with_lost_part 2021-06-01 16:32:59 +03:00
tavplubix
275a616bb6
Update ActiveDataPartSet.cpp 2021-06-01 16:27:09 +03:00
Alexander Tokmakov
5969891611 do not crash on intersecting parts 2021-06-01 16:25:23 +03:00
Nikolai Kochetov
e615858b10 Merge branch 'master' into refactor-MergeTreeDataSelectExecutor 2021-06-01 16:25:04 +03:00
Alexander Tokmakov
cdd46aa117 Revert "try fix intersecting virtual parts"
This reverts commit 2571ad7d43.
2021-06-01 14:52:25 +03:00
Ivan Lezhankin
365e52817b More fixes due to "in" function arguments being incorrectly checked as ASTIdentifier 2021-06-01 14:20:03 +03:00
alexey-milovidov
c7bd3b8674
Merge pull request #23315 from CurtizJ/empty-parts-fix
Fix waiting while dropping empty parts
2021-06-01 13:52:49 +03:00
Nikolai Kochetov
b54ff696e5 Merge branch 'master' into refactor-MergeTreeDataSelectExecutor 2021-06-01 12:08:37 +03:00
Nikolai Kochetov
d4cbce3761
Merge pull request #24782 from amosbird/projection-fix7
Fix empty part set with force_use_projection = 1
2021-06-01 11:36:50 +03:00
alesapin
4e038b2a61 Remove copy-paste and rename (review fixes) 2021-06-01 10:30:06 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs
Poco::File to std::filesystem
2021-05-31 23:17:02 +03:00
Nikolai Kochetov
b9aa404508 Review fixes. 2021-05-31 21:25:43 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
alesapin
2fe75e33ee Replace Poco::Event with std future in synchronous calls in ZooKeeper client 2021-05-31 16:03:30 +03:00
Amos Bird
a206b76e5b
Fix empty part set with force_use_projection = 1 2021-05-31 10:31:45 +08:00
Alexander Tokmakov
2571ad7d43 try fix intersecting virtual parts 2021-05-31 00:30:50 +03:00
Alexander Tokmakov
16647fe8ce some unrelated fixes 2021-05-31 00:29:37 +03:00
kssenii
2a631aaf08 Final fixes 2021-05-29 00:34:44 +03:00
Nikolai Kochetov
58fbc544cc Add more comments. 2021-05-28 20:16:09 +03:00
Nikolai Kochetov
295a302bc8 Remove settings from ReadFromMergeTree. 2021-05-28 17:34:02 +03:00
Pavel Kruglov
42d57138f2 replace all isDateOrDateTime invocations to explicit calls to isDate and isDateTime and isDateTime64 2021-05-28 15:34:49 +03:00
Nikolai Kochetov
91e8c2c75d Remove MergingFinal step. 2021-05-28 12:52:33 +03:00
Nikolai Kochetov
94f1ac5a16 Remove some commented code. 2021-05-28 12:41:07 +03:00
Nikolai Kochetov
c80ab6166a Fix style. 2021-05-28 12:26:44 +03:00
Nikolai Kochetov
4e28b7cb02 Fix _sample_factor column. 2021-05-28 12:23:46 +03:00
Anton Popov
179f172c4c fix build 2021-05-28 02:10:44 +03:00
Anton Popov
336abfa475 Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-28 01:58:30 +03:00
Nikolai Kochetov
936bd8a011 Merge branch 'master' into refactor-MergeTreeDataSelectExecutor 2021-05-27 20:47:39 +03:00
Nikolai Kochetov
1aeb705b20 Fix some tests. 2021-05-27 19:53:58 +03:00
Nikolai Kochetov
cbdf3752ef Part 3. 2021-05-27 16:40:33 +03:00
kssenii
a0d8d689d3 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-27 15:44:12 +03:00
alesapin
6924dd51a2 Merge branch 'master' into limited_fetches 2021-05-27 11:17:36 +03:00
alesapin
ef17daaaa0
Merge pull request #24496 from ClickHouse/configurable_max_parts_to_merge
Add `max_parts_to_merge_at_once` setting
2021-05-27 10:56:58 +03:00
kssenii
b28157a0df Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-27 00:01:25 +03:00
alesapin
42b2383e01 Basic throttler implementation 2021-05-26 23:37:44 +03:00
kssenii
73f16ee9ee Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-26 23:08:08 +03:00
Nikolai Kochetov
a51a6ea0b7 Part 2. 2021-05-26 21:14:43 +03:00
alesapin
8ba6ed3fc2 Simplier test 2021-05-26 16:53:05 +03:00
tavplubix
a21e4b333e
Update ReplicatedMergeTreeQueue.cpp 2021-05-26 15:10:38 +03:00
alesapin
08c37a4c9e Merge branch 'master' into configurable_max_parts_to_merge 2021-05-26 11:30:39 +03:00
alesapin
069d45790f
Merge pull request #24157 from kssenii/modulo-partition-key
Adjust modulo function when used in partition key
2021-05-26 10:30:54 +03:00
alesapin
5638c18140
Merge pull request #22864 from ianton-ru/s3_zero_copy_replication
Fix Zero-Copy replication with several S3 volumes (issue 22679)
2021-05-26 10:28:23 +03:00
kssenii
940eab6924 Review fixes 2021-05-26 09:26:19 +03:00
Alexey Milovidov
20f59ffbb9 Merge branch 'master' into issue-16775 2021-05-26 03:46:43 +03:00
tavplubix
84f9d38620
Merge pull request #24321 from amosbird/fixdroppartition
Fix drop partition in rare case
2021-05-26 01:34:22 +03:00
Nikolai Kochetov
34eaa48294 Part 1. 2021-05-25 19:34:43 +03:00
alesapin
6e29404a75 Add max_parts_to_merge_at_once setting 2021-05-25 16:21:17 +03:00
Alexander Tokmakov
4849ea2e03 fix unexpected behavior in mutation finalization 2021-05-25 14:10:45 +03:00
kssenii
70469429c1 Fixes 2021-05-24 23:39:56 +00:00
kssenii
c3fc2ff04b Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into modulo-partition-key 2021-05-24 22:08:49 +00:00
kssenii
31107816bb fs::absolute to fs::canonical 2021-05-24 20:37:54 +03:00
Anton Ivashkin
c936dab6d5 Fix part type in Fetcher::downloadPartToS3 2021-05-24 19:37:37 +03:00
Anton Ivashkin
a98286b4c4 Merge master 2021-05-24 17:46:29 +03:00
Alexey Milovidov
d32819f068 Mark false positives for PVS-Studio 2021-05-24 06:59:12 +03:00
Alexey Milovidov
273226de32 Remove string parameter for Density 2021-05-24 06:43:25 +03:00
Alexey Milovidov
6f70feed2f Merge branch 'master' into issue-16775 2021-05-24 05:56:57 +03:00
Alexey Milovidov
ff7b5d34bb Mark false positives for PVS-Studio 2021-05-24 02:56:03 +03:00
kssenii
659e57e60f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into modulo-partition-key 2021-05-23 22:46:00 +00:00
kssenii
1ffaf1d793 Better check 2021-05-23 21:53:32 +00:00
kssenii
bbf56e35ac Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-23 10:54:48 +03:00
alesapin
6c0b683d3a
Merge pull request #17871 from nvartolomei/nv/parts-uuid-move-shard
Part movement between shards
2021-05-23 10:20:57 +03:00
mergify[bot]
3c6e6464a7
Merge branch 'master' into s3_zero_copy_replication 2021-05-23 07:09:17 +00:00
Alexey Milovidov
40d4f0678f Remove overload (harmful) 2021-05-23 04:25:06 +03:00
kssenii
1c43b333ea Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-22 21:26:48 +03:00
kssenii
5606e248a7 Fix 2021-05-22 16:19:56 +00:00
kssenii
b72f6ea78e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into modulo-partition-key 2021-05-22 16:15:15 +00:00
alexey-milovidov
a25f1910cc
Merge pull request #24320 from ClickHouse/remove-auto-array
Remove AutoArray
2021-05-22 02:19:59 +03:00
kssenii
fcfec83875 Fix comparisons with modulo key (version 2) 2021-05-21 16:40:47 +00:00
kssenii
94b5763dbd Revert "Different approach to fix modulo"
This reverts commit 665f1aa0f6.
2021-05-21 16:14:01 +00:00
kssenii
30845a383f Fix comparisons with modulo key 2021-05-21 15:01:41 +00:00
kssenii
665f1aa0f6 Different approach to fix modulo 2021-05-21 14:30:08 +00:00
Anton Ivashkin
29336a4a34 Use keep_s3_on_delet flag instead of DeleteOnDestroyKeepS3 state 2021-05-21 15:29:10 +03:00
kssenii
bd0bc8e3e7 Revert "Adjust minMax index"
This reverts commit c69f587f8a.
2021-05-21 12:01:08 +00:00
alesapin
20a34e951a Split fetch part into two stages, don't use replicated block output stream 2021-05-21 12:30:49 +03:00
Alexey Milovidov
ea30bb7abf Improve implementation 2021-05-21 04:17:18 +03:00
alesapin
deb784a321 Merge branch 'master' into nvartolomei-parts-move 2021-05-20 21:24:55 +03:00
kssenii
3dee003f9b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-20 19:20:09 +03:00
kssenii
c69f587f8a Adjust minMax index 2021-05-20 15:32:18 +00:00
Nikolai Kochetov
818d081292
Merge pull request #24296 from amosbird/projection-fix5
Skip data finalization when doing projection materialization.
2021-05-20 10:50:42 +03:00
Alexey Milovidov
1006a970f7 Remove AutoArray 2021-05-20 09:30:13 +03:00
kssenii
56b388addf Add check for nullptr 2021-05-19 19:32:04 +00:00
kssenii
477214a57f Fix partitionPrunner with modulo 2021-05-19 17:36:32 +00:00
Amos Bird
c68ef31065
Skip data finalization when doing projection materialization. 2021-05-20 00:28:50 +08:00
mergify[bot]
230bec70fb
Merge branch 'master' into modulo-partition-key 2021-05-19 10:08:41 +00:00
alesapin
4d6c71801c
Merge branch 'master' into nv/parts-uuid-move-shard 2021-05-19 13:03:14 +03:00
Alexandra
3f00f064af wide parts validation with file opening via disk 2021-05-18 20:06:09 +00:00
tavplubix
8b13e73ac2
Merge pull request #24208 from ClickHouse/fix_intersecting_parts_again
Addition to #23997
2021-05-18 21:13:25 +03:00
Kseniia Sumarokova
0ab15d0bc4
Merge branch 'master' into modulo-partition-key 2021-05-18 17:45:32 +03:00
Anton Ivashkin
d256cf4edf Fix PVS check (Two or more case-branches perform the same actions) 2021-05-18 17:41:09 +03:00
alesapin
c1c6a0d8e3
Merge pull request #24248 from ucasFL/remove
remove useless code
2021-05-18 16:09:00 +03:00
feng lv
403d0a305b remove useless code 2021-05-18 12:19:04 +00:00
kssenii
5f083b779d A little better, fix checks 2021-05-18 14:25:19 +03:00
alesapin
36aeb59705 Fix lazy test 2021-05-18 11:03:38 +03:00
alesapin
22cc1c953f Revert "PVS Studio fixes"
This reverts commit 87c6e39a29.
2021-05-18 10:34:16 +03:00
Alexander Tokmakov
20b9af7b26 do not allow concurrent alter partition 2021-05-18 00:20:58 +03:00
alesapin
87c6e39a29 PVS Studio fixes 2021-05-17 23:23:32 +03:00
kssenii
9b8df78fdd Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-17 17:42:05 +03:00
kssenii
8c955680ad Add comments and remove adjusted deserializarion 2021-05-17 14:33:04 +00:00
Anton Popov
e27acc26be better dropPart and dropPartition 2021-05-17 17:26:36 +03:00
Anton Popov
e043cc45bf Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-17 16:11:12 +03:00
Anton Ivashkin
8ed4a5de62 Fix Zero Copy after merge master 2021-05-17 16:01:08 +03:00
alesapin
5563a0d7f3 Fix style 2021-05-17 14:44:47 +03:00
alesapin
023b592f97 Fix style 2021-05-17 14:42:14 +03:00
alesapin
d8994350c1 Useful comments 2021-05-17 14:39:37 +03:00
Ilya Yatsishin
8ede5c736a
Merge pull request #23992 from MaxWk/enable-datatime64-version-column
enable DateTime64 to be version column
2021-05-17 14:34:15 +03:00
alesapin
77b1aeaac2 Use same zookeeper object to have an exception if something went wrong 2021-05-17 14:30:23 +03:00
alesapin
67d34c0136 merge with master 2021-05-17 14:13:18 +03:00
alesapin
17f229857c Merge branch 'master' into nvartolomei-parts-move 2021-05-17 13:52:48 +03:00
Anton Ivashkin
39a30b77fe Merge master 2021-05-17 11:47:48 +03:00
alesapin
eae2df6478
Merge pull request #23997 from ClickHouse/fix_intersecting_parts
Try fix intersecting parts
2021-05-17 10:46:58 +03:00
Nikolai Kochetov
98e6c27a95
Merge pull request #24168 from amosbird/projection-fix3
Fix distributed processing when using projection
2021-05-17 09:45:39 +03:00
Maksim Kita
20cced0a4c
Merge pull request #24155 from kitaisreal/function-refactoring
IFunction refactoring
2021-05-17 09:26:57 +03:00
kssenii
3b1bf2bae6 Poco::Path substitution 2021-05-16 23:38:23 +03:00
kssenii
807684812e Fix clang tidy 2021-05-16 18:18:11 +00:00
kssenii
d44f1680da Use unsigned type for (de)serialization 2021-05-16 18:09:55 +00:00
Amos Bird
07b1be5a76
Fix distributed processing when using projection 2021-05-16 22:40:06 +08:00
alesapin
29726b5510
Merge pull request #24122 from ClickHouse/fix_table_startup
Set readonly mode if restarting thread failed to activate replica
2021-05-16 16:45:51 +03:00
kssenii
22b6d0eb25 Adjust modulo recursively 2021-05-16 09:20:59 +00:00
Nikolai Kochetov
093b7e120a
Merge pull request #24062 from amosbird/projection-fix1
Fix empty key projection query analysis
2021-05-16 07:37:03 +03:00
kssenii
add5970b5f fs::absolute 2021-05-16 03:12:13 +03:00
kssenii
c7f51fe60e Modulo adjustment for partition key 2021-05-15 19:51:02 +00:00
Maksim Kita
947f28d430 IFunction refactoring 2021-05-15 20:33:15 +03:00
Alexander Tokmakov
df5f3fbc9d review suggestions 2021-05-14 19:11:40 +03:00
Amos Bird
a113acc40c
Fix empty key 2021-05-14 22:54:14 +08:00
Amos Bird
8de016d33a
Fix empty key projection query analysis 2021-05-14 22:26:26 +08:00
Alexander Tokmakov
41776ba5d9 fix 2021-05-14 15:55:30 +03:00
Alexander Tokmakov
970ad94851 Merge branch 'master' into fix_intersecting_parts 2021-05-14 13:17:02 +03:00
alesapin
d4c6a5a05e Better logging 2021-05-14 11:38:53 +03:00
alesapin
67e4393769 If table was not active set readonly mode 2021-05-14 11:32:41 +03:00
kssenii
f33a517d33 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-14 02:58:26 +03:00
Alexander Tokmakov
2da12f0b59 fix intersection of drop range and merge 2021-05-13 17:04:36 +03:00
Alexander Tokmakov
e114c7eb8b fix virtual parts in REPLACE_RANGE 2021-05-13 14:29:59 +03:00
feng lv
c6f8ab9826 fix 2021-05-13 02:05:53 +00:00
Nikolai Kochetov
690410009c Restor is_temporary flag for fetched part. 2021-05-12 19:17:01 +03:00
kssenii
0527f0ea33 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-12 16:54:18 +03:00
feng lv
9035e3a643 fix test and conflict 2021-05-11 16:22:24 +00:00
Nikolai Kochetov
817bc1377c Fix part check for projections. 2021-05-11 17:07:19 +03:00
kssenii
393b026f2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-11 16:49:23 +03:00
alesapin
46e136b5c4
Merge branch 'master' into nv/parts-uuid-move-shard 2021-05-11 15:36:40 +03:00
Nikolai Kochetov
a7d717d767 Remove unused code. 2021-05-11 14:59:56 +03:00
Nikolai Kochetov
89f16014b8 fix fetch for projection. 2021-05-11 14:44:59 +03:00
Amos Bird
074c2ead9c
Fix 2021-05-11 18:22:23 +08:00
Amos Bird
5e9fab6d10
Comment debug info out 2021-05-11 18:12:29 +08:00
Amos Bird
5836e2e277
Fix issues in normal projection 2021-05-11 18:12:29 +08:00
Amos Bird
51d685bb9c
Even more fixes 2021-05-11 18:12:29 +08:00
Amos Bird
ddd4256a15
More fixes 2021-05-11 18:12:28 +08:00
Amos Bird
718c284437
Fix more tests 2021-05-11 18:12:28 +08:00
Amos Bird
ba17acbd63
Fix tests 2021-05-11 18:12:28 +08:00
Amos Bird
a3fd0b6f2e
Fix tests 2021-05-11 18:12:28 +08:00
Amos Bird
42f161b1dc
Fix more 2021-05-11 18:12:28 +08:00
Amos Bird
ef918aeaff
Rebase and fix 2021-05-11 18:12:28 +08:00
Amos Bird
9c069ebdbf
support prewhere, row_filter, read_in_order and decent projection selection
TODO set index analysis in projection
2021-05-11 18:12:27 +08:00
Amos Bird
f7f949c1f9
Fix aggregation keys order 2021-05-11 18:12:27 +08:00
Amos Bird
e1e560765c
Fix missing columns 2021-05-11 18:12:27 +08:00
Amos Bird
35961c0c5d
Use ActionsDAG to rewrite projection queries 2021-05-11 18:12:27 +08:00
Nikolai Kochetov
29d7038e30
Add ProjectionDescription::Type enum. Remove some checks. 2021-05-11 18:12:27 +08:00
Amos Bird
ebaf42a448
Reformat and fix some tests 2021-05-11 18:12:27 +08:00
Nikolai Kochetov
672cfedd13
Disable normal projection by the number of granules. 2021-05-11 18:12:26 +08:00
Nikolai Kochetov
95431168c2
Try to enable normal projections. 2021-05-11 18:12:26 +08:00
Amos Bird
483bd165e7
Check if pipeline is simple and add more comments 2021-05-11 18:12:26 +08:00
Nikolai Kochetov
427f444ba4
Fix prewhere. 2021-05-11 18:12:26 +08:00
Nikolai Kochetov
3d36fb57eb
Cleanup code. 2021-05-11 18:12:26 +08:00
Nikolai Kochetov
3296c9292f
Try to merge projectons faster. 2021-05-11 18:12:26 +08:00
Amos Bird
264cff6415
Projections
TODO (suggested by Nikolai)

1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies

3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)

Also need to figure out how prewhere works for projections, and
row_filter_policies.

wip
2021-05-11 18:12:23 +08:00
Kruglov Pavel
49e7ed6e60
Merge pull request #23673 from amosbird/partitionvalue
Add _partition_value virtual column
2021-05-11 11:23:00 +03:00
Alexander Tokmakov
eef899ae63 better code, add test 2021-05-10 21:03:37 +03:00
万康
308b964ca4 enable DateTime64 to be version column 2021-05-10 20:30:52 +08:00
kssenii
ab1a05a1f4 Poco::Path to fs::path, less concatination 2021-05-09 14:59:49 +03:00
Alexey Milovidov
e23a0d8f7e Fix warnings by PVS-Studio 2021-05-08 23:57:08 +03:00
Alexey Milovidov
513c6891db Fix warnings by PVS-Studio 2021-05-08 23:54:49 +03:00
Alexey Milovidov
ad88819ee4 Fix a bunch of warnings from PVS-Studio 2021-05-08 19:13:10 +03:00
Alexey Milovidov
e905883c75 More fixes for PVS-Studio 2021-05-08 19:12:31 +03:00
Alexey Milovidov
8d62c42eb9 A bunch of changes for PVS-Studio 2021-05-08 19:11:50 +03:00
Alexey Milovidov
604daa9581 Fix a bunch of warnings from PVS-Studio 2021-05-08 19:09:17 +03:00
Alexey Milovidov
2fbecf36ca More fixes for PVS-Studio 2021-05-08 18:20:40 +03:00
kssenii
266567e04c Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-08 17:49:05 +03:00
Alexey Milovidov
9e1a5f41fc A bunch of changes for PVS-Studio 2021-05-08 17:43:03 +03:00
Alexey Milovidov
47a4c101b9 Merge branch 'master' into normalize-bigint 2021-05-08 16:41:15 +03:00
tavplubix
f8c7725ef5
Merge pull request #23742 from ClickHouse/update_metadata_on_replica_recovery
Update metadata on replica recovery
2021-05-08 16:34:03 +03:00
kssenii
02288359c5 Less manual concatenation of paths 2021-05-08 13:59:55 +03:00
kssenii
35f999bf04 Poco::createFile to fs::createFile 2021-05-08 02:41:47 +03:00
Alexander Tokmakov
24bcd9f7c6 better comments, fix pvs warnings 2021-05-07 16:39:21 +03:00
kssenii
140bf7e2ea Less Poco 2021-05-07 15:29:26 +03:00
kssenii
9ec92ec514 Fix tests, less manual concatination of paths 2021-05-05 18:39:30 +03:00
Alexey Milovidov
02ce9cc725 Untangle UUID 2021-05-04 01:46:51 +03:00
Alexey Milovidov
f4100cf70c Remove harmful std::regex 2021-05-02 22:04:26 +03:00
Alexey Milovidov
1fda866881 Merge branch 'master' into normalize-bigint 2021-05-02 21:24:45 +03:00
feng lv
39f68bf5ff fix conflict 2021-05-02 16:33:45 +00:00
kssenii
ee06936596 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-01 17:24:31 +03:00
Maksim Kita
14e879cef9
Merge pull request #23781 from ucasFL/fix-clear-column
Fix CLEAR COLUMN does not work after #21303
2021-04-30 17:28:59 +03:00
Nikolai Kochetov
4bb56849b3
Merge pull request #23763 from amosbird/rowfilterfix
Fix crash when prewhere and row policy filter are both in effect with empty result
2021-04-30 16:25:36 +03:00
feng lv
aed2f337e9 Fix CLEAR COLUMN does not work after #21303 2021-04-30 05:02:32 +00:00
Amos Bird
70ec13f3d3
Fix empty block with row_filter 2021-04-29 19:40:14 +08:00
kssenii
deb4903af8 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-04-28 20:57:13 +03:00
Alexander Tokmakov
bbf3bbc76d update metadata on replica recovery 2021-04-28 20:49:27 +03:00
alexey-milovidov
f541e4017c
Merge pull request #23710 from ClickHouse/fix-bad-error-message
Fix illiterate error message #23700
2021-04-28 14:32:33 +03:00
alexey-milovidov
fdc5a44202
Merge pull request #23644 from nikitamikhaylov/delete-unit-tests
Move non gtest unit tests to /examples folder
2021-04-28 07:28:18 +03:00
Alexey Milovidov
2cf9da72ff Fix illiterate error message #23700 2021-04-28 06:49:46 +03:00
kssenii
eeb71672a0 Change in Storages/* 2021-04-27 16:49:37 +03:00
Nicolae Vartolomei
1fa5871ff7 Fix bad rebase and introduce part_moves_between_shards_enable setting 2021-04-27 14:20:13 +01:00
Nicolae Vartolomei
94f3db3c78 Use ZK check requests as barriers to prevent issuing tasks based on stale state information 2021-04-27 14:20:13 +01:00
alesapin
d12deed53d bump ci 2021-04-27 14:20:12 +01:00
alesapin
b930ca5d59 Followup fix 2021-04-27 14:20:12 +01:00
Nicolae Vartolomei
53d57ffb52 Part movement between shards
Integrate query deduplication from #17348
2021-04-27 14:20:12 +01:00
Amos Bird
8a3b5c1fab
Add _partition_value virtual column 2021-04-27 16:15:59 +08:00
Alexey Milovidov
a5a4e64ba7 Fix a few PVS-Studio warnings 2021-04-27 07:22:32 +03:00
alexey-milovidov
186b1128d0
Merge pull request #23310 from amosbird/fixbugindex
Don't relax NOT conditions during partition pruning.
2021-04-27 07:13:18 +03:00
Nikita Mikhaylov
9f55424250 move to examples everywhere 2021-04-27 01:51:42 +03:00
tavplubix
81726de8fe
Merge pull request #23366 from amosbird/pid
Fix invalid virtual column expr
2021-04-26 11:10:24 +03:00
Alexey Milovidov
9eb60086b0 Attempt to normalize big integers 2021-04-25 13:02:19 +03:00
Alexey Milovidov
649550c5ab Attempt to normalize big integers 2021-04-25 12:30:43 +03:00
Alexey Milovidov
8f01af62d9 Merge branch 'master' into normalize-bigint 2021-04-25 06:57:44 +03:00
alexey-milovidov
cae5260817
Update KeyCondition.cpp 2021-04-24 05:34:35 +03:00
alexey-milovidov
20e897bc5a
Merge pull request #23301 from ClickHouse/fix_race_in_deduplication
Deduplication: check block version on removal
2021-04-24 04:59:00 +03:00
alexey-milovidov
2cfc2c8825
Merge pull request #23397 from CurtizJ/better-move-to-prewhere
Allow to move more conditions to PREWHERE
2021-04-24 04:39:33 +03:00
mergify[bot]
a5da512e39
Merge branch 'master' into pid 2021-04-24 00:37:39 +00:00
alexey-milovidov
c66e715f49
Merge pull request #23433 from ClickHouse/tavplubix-patch-1
Fix `Cannot unlink file` in `dropIfEmpty()` with multidisk confuguration
2021-04-24 03:34:06 +03:00
alexey-milovidov
cd1f14ba61
Update MergeTreeData.cpp 2021-04-24 03:33:56 +03:00
feng lv
0f5c05854c fix conflict 2021-04-23 12:22:51 +00:00
feng lv
4ffe199d39 Implement table comments 2021-04-23 12:18:23 +00:00
Ivan Lezhankin
d9626e73c3 Revert "CC"
This reverts commit ee194928d2.
2021-04-22 16:55:18 +03:00
hexiaoting
d24946f475 Fix bug when modify column without type 2021-04-22 14:48:57 +08:00
mergify[bot]
b9e229d19d
Merge branch 'master' into empty-parts-fix 2021-04-21 13:19:54 +00:00
Ivan Lezhankin
ee194928d2 CC 2021-04-21 16:19:28 +03:00
Nikita Mikhaylov
d4e8f9676b
Merge pull request #21304 from amosbird/indexhint
Resurrect indexHint function.
2021-04-21 16:14:43 +03:00
tavplubix
1a776b47a7
Update MergeTreeData.cpp 2021-04-21 15:34:42 +03:00
Amos Bird
50f2e488bd
Fix invalid virtual column expr 2021-04-21 10:29:03 +08:00
Anton Popov
a83a07246e allow to move more conditions to PREWHERE 2021-04-21 01:03:58 +03:00
Ivan
10c5ba3022
Merge branch 'master' into async-insert 2021-04-20 17:58:15 +03:00
Anton Popov
8f57141fad
Merge pull request #23195 from CurtizJ/nested-update
Merging #22503
2021-04-20 14:57:47 +03:00
Amos Bird
32c84f77c3
Resurrect indexHint function. 2021-04-20 19:27:23 +08:00
Amos Bird
d5f606c544
Another fix 2021-04-20 14:15:28 +08:00
Anton Popov
0a0c510aa2 minor fixes 2021-04-20 05:51:07 +03:00
Anton Popov
42a0416a2b better drop of empty parts 2021-04-20 05:31:08 +03:00
alexey-milovidov
4cef8c68f7
Merge pull request #23258 from azat/wundef
Add -Wundef for gcc builds
2021-04-19 22:45:22 +03:00
Anton Popov
66913c5865 fix waiting while dropping empty parts 2021-04-19 19:24:50 +03:00
Ivan Lezhankin
7a88ca31cc Merge branch 'master' into async-insert 2021-04-19 18:33:43 +03:00
Amos Bird
aeff06d67d
Don't relax NOT conditions during partition pruning. 2021-04-19 22:15:53 +08:00
tavplubix
c9ebc95faf
Update ReplicatedMergeTreeCleanupThread.cpp 2021-04-19 13:27:32 +03:00