Commit Graph

459 Commits

Author SHA1 Message Date
Vitaly Baranov
7a63feb3f7 Make restore tasks explicit. 2022-03-20 20:01:31 +01:00
Alexander Tokmakov
07d952b728 use snapshots for semistructured data, durability fixes 2022-03-17 18:26:18 +01:00
Alexander Tokmakov
d04dc03fa4 Merge branch 'master' into mvcc_prototype 2022-03-17 15:24:32 +01:00
Nikolai Kochetov
ee9c2ec735
Merge pull request #34780 from azat/mt-delayed-part-flush
Do not delay final part writing by default (fixes possible Memory limit exceeded during INSERT)
2022-03-17 12:30:51 +01:00
Anton Popov
0ba78c3c3a Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-16 15:28:09 +00:00
Alexander Tokmakov
9702b5177d Merge branch 'master' into mvcc_prototype 2022-03-14 21:45:38 +01:00
Alexander Tokmakov
278d779a01 log cleanup, more comments 2022-03-14 21:43:34 +01:00
Maksim Kita
2fdcf53a76 Fix clang-tidy warnings in Server, Storages folders 2022-03-14 18:17:35 +00:00
Anton Popov
063917786e minor fixes 2022-03-14 17:29:18 +00:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Alexander Tokmakov
061fa6a6f2 Merge branch 'master' into mvcc_prototype 2022-03-10 13:13:04 +01:00
Azat Khuzhin
3a5a39a9df Do not delay final part writing by default
For async s3 writes final part flushing was defered until all the INSERT
block was processed, however in case of too many partitions/columns you
may exceed max_memory_usage limit (since each stream has overhead).

Introduce max_insert_delayed_streams_for_parallel_writes (with default
to 1000 for S3, 0 otherwise), to avoid this.

This should "Memory limit exceeded" errors in performance tests.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-08 22:17:36 +03:00
Azat Khuzhin
caffc144b5 Fix possible "Part directory doesn't exist" during INSERT
In #33291 final part commit had been defered, and now it can take
significantly more time, that may lead to "Part directory doesn't exist"
error during INSERT:

    2022.02.21 18:18:06.979881 [ 11329 ] {insert} <Debug> executeQuery: (from 127.1:24572, user: default) INSERT INTO db.table (...) VALUES
    2022.02.21 20:58:03.933593 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18044_18044_0 to 20220214_270654_270654_0.
    2022.02.21 21:16:50.961917 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18197_18197_0 to 20220214_270689_270689_0.
    ...
    2022.02.22 21:16:57.632221 [ 64878 ] {} <Warning> db.table: Removing temporary directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/
    ...
    2022.02.23 12:23:56.277480 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18232_18232_0 to 20220214_273459_273459_0.
    2022.02.23 12:23:56.299218 [ 11329 ] {insert} <Error> executeQuery: Code: 107. DB::Exception: Part directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/ doesn't exist. Most likely it is a logical error. (FILE_DOESNT_EXIST) (version 22.2.1.1) (from 127.1:24572) (in query: INSERT INTO db.table (...) VALUES), Stack trace (when copying this message, always include the lines below):

Follow-up for: #28760
Refs: #33291

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-08 07:44:11 +03:00
taiyang-li
b4174b0bef merge master and fix conflicts 2022-03-08 11:39:25 +08:00
Anton Popov
2758db5341 add more comments 2022-03-01 19:32:55 +03:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
Anton Popov
a661eaf39f better performance of getting storage snapshot 2022-02-16 02:17:22 +03:00
Alexander Tokmakov
ae5aa8c12d write part version before other files 2022-02-15 02:24:51 +03:00
alesapin
b2886a429b Fix lock during fetch 2022-02-14 12:20:27 +03:00
Alexander Tokmakov
07e66e690d Merge branch 'master' into mvcc_prototype 2022-02-11 15:53:32 +03:00
Anton Popov
f012871a7c better caching of common types of object columns 2022-02-11 01:20:30 +03:00
alesapin
3af06b23f8 POC 2022-02-10 22:45:52 +03:00
Anton Popov
dcd7312d75 cache common type on objects in MergeTree 2022-02-09 23:47:53 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
taiyang-li
d04ccc0489 Merge branch 'master' into rocksdb_metacache 2022-02-09 11:54:10 +08:00
taiyang-li
b6132d490f merge master and solve conflict 2022-02-08 15:24:59 +08:00
Amos Bird
3fab7af541
Bug fix and improvement of minmax_count_projection 2022-02-06 16:46:11 +08:00
Alexander Tokmakov
fe30e0f162 fixes 2022-02-03 21:57:09 +03:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
Amos Bird
ec7d367814
DiskLocal checker
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
2022-02-01 05:55:27 +08:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
李扬
6d50d36405
Merge branch 'master' into rocksdb_metacache 2022-01-28 22:00:31 -06:00
Alexander Tokmakov
fb9b2d5326 Merge branch 'master' into mvcc_prototype 2022-01-28 21:18:36 +03:00
Alexander Tokmakov
e0304c2a58 review fixes, write tid into mutation entry 2022-01-28 20:47:37 +03:00
alexey-milovidov
f6684dbc62
Merge pull request #32304 from devcrafter/deduplication_token_7461
insert_deduplication_token setting for INSERT statement
2022-01-28 13:03:55 +03:00
zhongyuankai
a6254516e0 Fix Alter ttl modification unsupported table engine 2022-01-24 21:48:52 +08:00
taiyang-li
73def8b483 merge master and solve conflict 2022-01-24 11:01:43 +08:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
0ea0a8ccc7 Merge branch 'master' into mvcc_prototype 2022-01-20 17:05:55 +03:00
alexey-milovidov
994584df09
Merge pull request #33753 from azat/mt-fix-count-race
Fix tiny race between count() and INSERT/merges/... in MergeTree
2022-01-20 06:33:54 +03:00
Alexander Tokmakov
e9a5a64a71 Merge branch 'master' into mvcc_prototype 2022-01-19 21:41:23 +03:00
alexey-milovidov
d222cb9771
Merge pull request #33695 from amosbird/buildimprove
Some build improvement
2022-01-19 03:19:22 +03:00
Azat Khuzhin
c9e93b21c2 Fix tiny race between count() and INSERT/merges/... in MergeTree
Before it was possible to return stale counter from
StorageMergeTree::totalRows() (that is used for
optimize_trivial_count_query) since the accounting is done in two steps:
- subtract old number of rows
  <-- here the number can be zero, even though there are rows -->
- add new number of rows

This was found by CI [1] in 01615_random_one_shard_insertion test:

Here you can see that INSERT went to both tables:

<details>

    2022.01.16 09:07:34.288252 [ 154369 ] {a1905be0-93da-460c-8c6f-9b5adace72a0} <Debug> DistributedBlockOutputStream: It took 0.035197041 sec. to insert 100
    blocks, 2841.1479249065287 rows per second. Insertion status:
    Wrote 54 blocks and 54 rows on shard 0 replica 0, localhost:9000 (average 0 ms per block, the slowest block 1 ms)
    Wrote 46 blocks and 46 rows on shard 1 replica 0, localhost:9000 (average 0 ms per block, the slowest block 1 ms)

</details>

But the test fails, since select from shard1.tbl returns 0, and the
problem was concurrent merge:

<details>

    2022.01.16 09:07:34.289470 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> executeQuery: (from [::1]:42082) (comment: 01615_random_one_shard_insertion.sql) select count() != 0 from shard_0.tbl;
    2022.01.16 09:07:34.289564 [ 375 ] {c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab::all_1_54_9} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 54 rows, containing 1 columns (1 merged, 0 gathered) in 0.00171193 sec., 31543.345814373253 rows/sec., 246.43 KiB>
    2022.01.16 09:07:34.289810 [ 375 ] {c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab::all_1_54_9} <Trace> shard_0.tbl (c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab): Renaming temporary part tmp_merge_all_1_54_9 to all_1_54_9.
    2022.01.16 09:07:34.289858 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> ContextAccess (default): Access granted: SELECT(number) ON shard_0.tbl
    2022.01.16 09:07:34.289897 [ 375 ] {c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab::all_1_54_9} <Trace> shard_0.tbl (c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab) (MergerMutator): Merged 6 parts: from all_1_49_8 to all_54_54_0
    2022.01.16 09:07:34.289920 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> InterpreterSelectQuery: WithMergeableState -> Complete
    2022.01.16 09:07:34.289987 [ 375 ] {} <Debug> MemoryTracker: Peak memory usage Mutate/Merge: 3.12 MiB.
    2022.01.16 09:07:34.290305 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> MergingAggregatedTransform: Reading blocks of partially aggregated data.
    2022.01.16 09:07:34.290332 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> MergingAggregatedTransform: Read 1 blocks of partially aggregated data, total 1 rows.
    2022.01.16 09:07:34.290343 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> Aggregator: Merging partially aggregated single-level data.
    2022.01.16 09:07:34.290358 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> Aggregator: Merged partially aggregated single-level data.
    2022.01.16 09:07:34.290366 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> Aggregator: Converting aggregated data to blocks
    2022.01.16 09:07:34.290391 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> Aggregator: Converted aggregated data to blocks. 1 rows, 8.00 B in 1.0939e-05 sec. (91416.034 rows/sec., 714.19 KiB/sec.)
    2022.01.16 09:07:34.290709 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Information> executeQuery: Read 1 rows, 4.01 KiB in 0.001187722 sec., 841 rows/sec., 3.30 MiB/sec.
    2022.01.16 09:07:34.290774 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/33675/7848ea7d609e4c720e8e4494eb6207c0751f5aea/stateless_tests__ubsan__actions_.html

This also fixes a race between DROP TABLE check and INSERT/merges.

v0: use Active parts instead.
v2: fix total counters accounting instead.
2022-01-18 23:58:02 +03:00
Amos Bird
6d62060e16
Build improvement 2022-01-17 22:36:27 +08:00
alesapin
4caef03e6a Add ephemeral lock in zookeeper 2022-01-17 14:52:51 +03:00
Alexander Tokmakov
bf2b6c2c37 Merge branch 'master' into mvcc_prototype 2022-01-10 19:53:50 +03:00
taiyang-li
2cae8d552c remove useless code 2022-01-10 19:17:12 +08:00
taiyang-li
1b4b727d79 fix conflict 2022-01-10 10:43:13 +08:00
Igor Nikonov
102b9cdcfb Merge remote-tracking branch 'origin/master' into deduplication_token_7461 2022-01-09 19:15:33 +01:00
Igor Nikonov
0857a8d061 insert_deduplication_token support in non-replicated MergeTree 2022-01-09 19:06:09 +01:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474 2022-01-07 10:12:25 +00:00
taiyang-li
c0a9c2b916 refactor metadatacache to reduce using of USE_ROCKSDB 2022-01-05 19:51:50 +08:00
taiyang-li
3f6d830536 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-04 10:00:34 +08:00
Alexander Tokmakov
187c43eba8 rename Committed state to Active 2021-12-30 23:45:38 +03:00
taiyang-li
98f37afc80 fix building 2021-12-28 19:29:01 +08:00
Anton Ivashkin
0465aef15d Fixes by code review responces 2021-12-27 19:27:06 +03:00
Anton Ivashkin
e88b97dafb Fix typos 2021-12-21 19:56:29 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474 2021-12-21 17:38:36 +03:00
Anton Ivashkin
33cbfc89ef Move logic for replicated part to StorageReplicatedMergeTree class 2021-12-21 17:29:50 +03:00
Anton Ivashkin
0c0bf66334 Merge master 2021-12-21 17:27:54 +03:00
Anton Ivashkin
f0b9a4327a Use table UUID in zero-copy shared label in ZooKeeper 2021-12-20 16:19:00 +03:00
liyang
37ba8004ff Speep up mergetree starting up process 2021-12-18 16:39:59 +08:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
Alexander Tokmakov
32e62ed5c2 Merge branch 'master' into mvcc_prototype 2021-12-17 10:43:46 +03:00
Alexander Tokmakov
e185ad260b fix skipping of some mutations 2021-12-15 21:28:24 +03:00
Alexander Tokmakov
1b02136941 fix skipping of some mutations 2021-12-15 21:19:29 +03:00
Alexander Tokmakov
6e1c16c2e7 add support for mutations 2021-12-14 23:06:34 +03:00
nautaa
0da3881ff4 flush all InMemoryDataParts when wal is not enabled 2021-12-14 16:31:17 +08:00
Alexander Tokmakov
7fcb79ae72 Merge branch 'master' into mvcc_prototype 2021-12-07 14:39:29 +03:00
Anton Ivashkin
98bae1b313 Fix tests 2021-12-02 16:25:45 +03:00
Anton Ivashkin
e0a16a483e Merge master 2021-12-01 19:03:31 +03:00
Anton Ivashkin
80ab73c691 Fix Zero-Copy replication lost locks, fix remove used remote data in DROP DETACHED PART 2021-12-01 16:11:26 +03:00
Anton Popov
6f4d9a53b2 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-12-01 15:54:33 +03:00
Alexander Tokmakov
2cf4b51029 fix 2021-11-29 21:54:46 +03:00
Anton Ivashkin
0f9038ebed Zero-copy: move shared mark outside table node in ZooKeeper 2021-11-29 19:05:31 +03:00
Alexander Tokmakov
1ab8b5d4df fix segfault on attach parititon 2021-11-25 13:04:39 +03:00
Anton Popov
ccd78e3838 Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-22 17:19:35 +03:00
Nikolai Kochetov
b814f54f50
Merge pull request #31474 from amosbird/projection-improve6
Allow partition column to be used for minmax_count projection
2021-11-22 13:01:57 +03:00
Amos Bird
c47ec31617
fix 2021-11-19 11:51:05 +08:00
Alexander Tokmakov
0a4647f927 support alter partition 2021-11-17 21:14:14 +03:00
Alexander Tokmakov
672157b817 Merge branch 'master' into mvcc_prototype 2021-11-11 22:01:34 +03:00
Anton Popov
a20922b2d3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-11-09 15:36:25 +03:00
Alexander Tokmakov
51bd98294e better merge selecting backoff 2021-11-09 15:26:51 +03:00
Alexander Tokmakov
92eec74ad7 Merge branch 'master' into mvcc_prototype 2021-11-06 21:08:36 +03:00
Vitaly Baranov
a1f29d31ea Support BACKUP & RESTORE for log family. 2021-11-01 12:07:17 +03:00
Alexander Tokmakov
2e7e195e77 change alter_lock to std::timed_mutex 2021-10-26 13:37:00 +03:00
Nikita Mikhaylov
75487be899 White list of storages that supports final 2021-10-19 12:16:33 +00:00
Nikita Mikhaylov
4fbd332bf1 Done 2021-10-19 11:53:55 +00:00
Nikolai Kochetov
2a95d7fc87
Merge pull request #29918 from amosbird/projection-improve5
Add primary key to minmax_count_projection
2021-10-17 14:04:18 +03:00
alexey-milovidov
4f11cfa59d
Merge pull request #28760 from azat/mutator-forbid-cleaner
Forbid cleaning of tmp directories that can be used by an active mutation/merge.
2021-10-17 03:18:01 +03:00
Amos Bird
77c4a5fa18
Better 2021-10-16 18:13:41 +08:00
Amos Bird
77a2022cf5
Add primary key to minmax_count_projection 2021-10-16 18:13:41 +08:00
Azat Khuzhin
07e8b2b3c7 Do not try to remove temporary paths that is currently in written by merge/mutation
v2: rebase against MergeTask
v3: rebase due to conflicts in src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
v4:
- rebase due to conflicts in src/Storages/MergeTree/MergeTask.cpp
- drop common/scope_guard_safe.h (not used)
2021-10-16 00:43:52 +03:00
Amos Bird
23602f4607
Speed up part loading for JBOD 2021-10-13 12:33:54 +08:00
Nikolai Kochetov
c36bb9206e
Merge pull request #29879 from amosbird/queryidholderfix
Fix potential leak of query_id_holder
2021-10-12 15:42:02 +03:00
Maksim Kita
2d069acc22 System table data skipping indices added size 2021-10-11 11:39:50 +03:00
Amos Bird
ebfb013ea1
Fix potential leak of query_id_holder 2021-10-11 15:56:00 +08:00
Alexander Tokmakov
72b1b2e360 Merge branch 'master' into mvcc_prototype 2021-09-23 22:53:27 +03:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask (#25165)
Introduced MergeTask and MutateTask
2021-09-17 00:19:58 +03:00
Anton Popov
4c388e3d84 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-09-09 14:10:16 +03:00
Nikita Mikhaylov
6062dd0021 Better 2021-09-08 00:21:21 +00:00
Nikita Mikhaylov
ea0fbf81af Renaming 2021-09-06 12:01:16 +00:00
Nikita Mikhaylov
292a24abe8 Merge upstream/master into global-merge-executor (using imerge) 2021-09-03 00:34:24 +00:00
Nikita Mikhaylov
cc7c221fad Own PriorityQueue + prettifying the code 2021-09-02 21:31:32 +00:00
alesapin
b592400e54
Merge pull request #28404 from ClickHouse/remove_outdated_settings
Remove obsolete settings for replicated fetches.
2021-09-02 14:03:12 +03:00
Nikolai Kochetov
5f0b1bc4e3
Merge pull request #26286 from amosbird/projection-improve2
Virtual projection for min max indices.
2021-09-02 13:03:30 +03:00
mergify[bot]
5d299fbdee
Merge branch 'master' into remove_outdated_settings 2021-09-01 14:07:48 +00:00
alesapin
921e51e061 Remove some obsolete settings for replicated fetches 2021-08-31 15:22:56 +03:00
Nikita Mikhaylov
f8d4f04294 Merge upstream/master into global-merge-executor (using imerge) 2021-08-31 11:52:11 +00:00
Nikita Mikhaylov
c4416906c8 done 2021-08-30 19:37:03 +00:00
Amos Bird
b68857d086
Simplify projection, add minmax_count projection. 2021-08-28 11:25:37 +08:00
Alexey Milovidov
79e0433ba7 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-28 01:19:16 +03:00
Amos Bird
0169fce78e
Projection bug fixes and refactoring. 2021-08-26 19:09:31 +08:00
Alexey Milovidov
fd27a2faef Remove imperfect code 2021-08-25 02:05:55 +03:00
Alexey Milovidov
7c1d0a3baf Progress on development 2021-08-25 01:24:47 +03:00
Vitaly Baranov
63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00
Alexander Tokmakov
c74bfbf991 Merge branch 'master' into mvcc_prototype 2021-07-28 22:21:48 +03:00
Anton Popov
2b58f39c10 dynamic columns: support missed columns in distributed 2021-07-23 19:34:23 +03:00
Anton Popov
f99374cca6 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-07-20 18:20:21 +03:00
Zhichang Yu
5047c758f4 fix per review 2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1 hdfs zero copy 2021-07-13 07:19:12 +00:00
Alexey Milovidov
1ce40d6629 Fix bad code (default function argument) 2021-07-11 04:01:23 +03:00
Anton Popov
3ed7f5a6cc dynamic subcolumns: add snapshot for storage 2021-07-09 06:15:41 +03:00
Anton Popov
072e65b728 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-07-07 17:20:38 +03:00
Anton Popov
9071ecd428 fix alter of settings in MergeTree 2021-07-05 15:44:58 +03:00
Nikita Mikhaylov
c66a3b22b5 done 2021-06-22 23:24:47 +00:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652)
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
2021-06-20 11:24:43 +03:00
Alexander Tokmakov
9a88b9dacc set maxtid on drop part 2021-06-08 21:17:18 +03:00
Alexander Tokmakov
881889ef22 parts cleanup, fixes 2021-06-07 21:22:58 +03:00
Anton Popov
205a23282b Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-06-07 19:38:22 +03:00
Alexander Tokmakov
9915bd0a8b Merge branch 'master' into mvcc_prototype 2021-06-03 00:52:57 +03:00
alexey-milovidov
c7bd3b8674
Merge pull request #23315 from CurtizJ/empty-parts-fix
Fix waiting while dropping empty parts
2021-06-01 13:52:49 +03:00
Anton Popov
9f52362b81 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-05-31 23:08:45 +03:00
Alexander Tokmakov
5692db736c Merge branch 'master' into mvcc_prototype 2021-05-31 21:11:22 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
Anton Popov
24707e6604 dynamic subcolumns: better handling of incompatible types 2021-05-29 04:19:20 +03:00
Anton Popov
336abfa475 Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-28 01:58:30 +03:00
Anton Popov
0bdf9d207c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-05-26 00:25:47 +03:00
Alexander Tokmakov
f2ab5a05c6 Merge branch 'master' into mvcc_prototype 2021-05-18 13:36:38 +03:00
Alexander Tokmakov
1fd6142c1a try enable merges 2021-05-18 13:35:48 +03:00
Anton Popov
e27acc26be better dropPart and dropPartition 2021-05-17 17:26:36 +03:00
Anton Popov
e043cc45bf Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-17 16:11:12 +03:00
alesapin
17f229857c Merge branch 'master' into nvartolomei-parts-move 2021-05-17 13:52:48 +03:00
Alexander Tokmakov
970ad94851 Merge branch 'master' into fix_intersecting_parts 2021-05-14 13:17:02 +03:00
Alexander Tokmakov
df2849be90 Merge branch 'master' into mvcc_prototype 2021-05-13 21:48:36 +03:00
Alexander Tokmakov
2da12f0b59 fix intersection of drop range and merge 2021-05-13 17:04:36 +03:00
alesapin
46e136b5c4
Merge branch 'master' into nv/parts-uuid-move-shard 2021-05-11 15:36:40 +03:00
Amos Bird
9c069ebdbf
support prewhere, row_filter, read_in_order and decent projection selection
TODO set index analysis in projection
2021-05-11 18:12:27 +08:00