thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
100,258 Commits 9,338 Branches 1,783 Tags 2.1 GiB
11f93dec3f
Commit Graph

1196 Commits

Author SHA1 Message Date
Alexander Tokmakov
9702b5177d Merge branch 'master' into mvcc_prototype 2022-03-14 21:45:38 +01:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
zhanghuajie
53a8987b3b fix build fail with gcc --fix warnings without disabling some parameters 2022-03-11 21:59:19 +08:00
Alexander Tokmakov
061fa6a6f2 Merge branch 'master' into mvcc_prototype 2022-03-10 13:13:04 +01:00
Azat Khuzhin
caffc144b5 Fix possible "Part directory doesn't exist" during INSERT 
In #33291 final part commit had been defered, and now it can take
significantly more time, that may lead to "Part directory doesn't exist"
error during INSERT:

    2022.02.21 18:18:06.979881 [ 11329 ] {insert} <Debug> executeQuery: (from 127.1:24572, user: default) INSERT INTO db.table (...) VALUES
    2022.02.21 20:58:03.933593 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18044_18044_0 to 20220214_270654_270654_0.
    2022.02.21 21:16:50.961917 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18197_18197_0 to 20220214_270689_270689_0.
    ...
    2022.02.22 21:16:57.632221 [ 64878 ] {} <Warning> db.table: Removing temporary directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/
    ...
    2022.02.23 12:23:56.277480 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18232_18232_0 to 20220214_273459_273459_0.
    2022.02.23 12:23:56.299218 [ 11329 ] {insert} <Error> executeQuery: Code: 107. DB::Exception: Part directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/ doesn't exist. Most likely it is a logical error. (FILE_DOESNT_EXIST) (version 22.2.1.1) (from 127.1:24572) (in query: INSERT INTO db.table (...) VALUES), Stack trace (when copying this message, always include the lines below):

Follow-up for: #28760
Refs: #33291

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 07:44:11 +03:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
Alexander Tokmakov
aa6b9a2abc Merge branch 'master' into mvcc_prototype 2022-02-23 23:22:03 +03:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
Jianmei Zhang
ef0c3b99ff Merge remote-tracking branch 'upstream/master' 2022-02-17 14:02:27 +08:00
Jianmei Zhang
25c761b3b6 Early break for multiple leaders case when log updated by other leader 2022-02-16 16:06:41 +08:00
Alexander Tokmakov
e37ef4560c fix 2022-02-15 18:00:45 +03:00
alesapin
447cd56cb9 Fix comments 2022-02-15 12:11:50 +03:00
Alexander Tokmakov
ae5aa8c12d write part version before other files 2022-02-15 02:24:51 +03:00
alesapin
b75d551281 Fix clang tidy and add check for master 2022-02-14 14:37:41 +03:00
alesapin
b2886a429b Fix lock during fetch 2022-02-14 12:20:27 +03:00
alesapin
beb4400978 Fix 'same local part' check 2022-02-13 23:08:29 +03:00
Alexander Tokmakov
07e66e690d Merge branch 'master' into mvcc_prototype 2022-02-11 15:53:32 +03:00
alesapin
3af06b23f8 POC 2022-02-10 22:45:52 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
alesapin
02a93cb852 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-08 19:42:27 +03:00
alesapin
8e9ccbd077
Merge pull request #33933 from sunny19930321/fix-substr-zk-metadata 
Better local metadata comparison with ZooKeeper metadata
 2022-02-08 14:58:46 +03:00
alesapin
3af6012cb4 Revert "Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"""" 
This reverts commit 2bc2ea485e.
 2022-02-08 11:01:26 +03:00
alesapin
2bc2ea485e Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer""" 
This reverts commit fb77d7a7d5.
 2022-02-08 10:56:29 +03:00
alesapin
fb77d7a7d5 Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"" 
This reverts commit 875e5413ad.
 2022-02-07 12:36:54 +03:00
alesapin
bb361e3b01 Fix bug with existing part 2022-02-07 12:33:37 +03:00
Alexander Tokmakov
45be75b4db Merge branch 'master' into mvcc_prototype 2022-02-06 23:36:08 +03:00
alesapin
c028269e6f Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-04 18:02:25 +03:00
tavplubix
22de534fdc
Merge pull request #34297 from ClickHouse/fix_restarting_thread 
Try to fix race between pullLogsToQueue and RestartingThread
 2022-02-04 17:30:17 +03:00
alesapin
875e5413ad Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer" 
This reverts commit b92efed350, reversing
changes made to ecce006cb2.
 2022-02-04 14:30:33 +03:00
alesapin
2ed45b2a98 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-04 11:23:46 +03:00
Alexander Tokmakov
897e94c16c make restarting thread less bad 2022-02-03 23:29:24 +03:00
Alexander Tokmakov
ca5f951558 Merge branch 'master' into mvcc_prototype 2022-02-03 18:56:44 +03:00
alesapin
25375bc76b Remove unused param 2022-02-03 11:21:19 +03:00
alesapin
80800e051e Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-02 19:42:04 +03:00
Nikolai Kochetov
ea044fc6b2 Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer 2022-02-02 19:40:41 +03:00
alesapin
b9c118524f Fix race condition on hardlink/erase/read metadata 2022-02-02 19:40:21 +03:00
mergify[bot]
d8bea598b2
Merge branch 'master' into fix-substr-zk-metadata 2022-02-01 14:10:12 +00:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Alexander Tokmakov
d93c9f0be4 fix build after merge 2022-02-01 15:05:05 +03:00
alesapin
e87ed0ff02 Fix in-memory parts 2022-02-01 14:37:12 +03:00
Azat Khuzhin
5be76bc969 Use proper fmt-like logging 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3"" 2022-02-01 13:36:51 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
alexey-milovidov
99392b5ca7
Merge pull request #13544 from amosbird/mdha 
Multi-Disk auto-recovery.
 2022-02-01 06:13:26 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3" 2022-02-01 05:49:40 +03:00
Amos Bird
ec7d367814
DiskLocal checker 
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
 2022-02-01 05:55:27 +08:00
Nikolai Kochetov
321fa4a9e8
Merge pull request #33291 from ClickHouse/add-pool-to-s3-write-buffer 
Add pool to WriteBufferFromS3
 2022-01-31 19:37:40 +03:00
Maksim Kita
8513f20cfd
Merge pull request #34145 from kitaisreal/bitset-sort-performance-check 
pdqsort performance check
 2022-01-31 12:35:13 +01:00
tavplubix
d19e24f530
Merge pull request #34096 from ClickHouse/fix_race_merge_selecting_task 
Fix race between mergeSelectingTask and queue reinitialization
 2022-01-31 12:16:29 +03:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
Alexander Tokmakov
fb9b2d5326 Merge branch 'master' into mvcc_prototype 2022-01-28 21:18:36 +03:00
Alexander Tokmakov
e0304c2a58 review fixes, write tid into mutation entry 2022-01-28 20:47:37 +03:00
Nikolai Kochetov
1c9f026178 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-28 16:01:42 +03:00
Alexander Tokmakov
b3ddc601a5 fix race between mergeSelectingTask and queue reinitialization 2022-01-28 15:50:58 +03:00
liyang830
c2bdeb349c Merge branch 'fix-substr-zk-metadata' of github.com:sunny19930321/ClickHouse into fix-substr-zk-metadata 2022-01-27 16:35:02 +08:00
liyang830
eca0453564 fix local metadata differ zk metadata 2022-01-27 16:33:40 +08:00
Nikolai Kochetov
9b2998c639 Review fixes. 2022-01-26 18:08:01 +00:00
Nikolai Kochetov
506ee8c024 Refactor some code. 2022-01-24 15:55:29 +00:00
tavplubix
da9a38655b
Merge pull request #33847 from ClickHouse/fix_33806 
Fix logical error on RESTORE REPLICA
 2022-01-24 13:28:37 +03:00
tavplubix
34baca1298
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
 2022-01-24 12:12:08 +03:00
tavplubix
2f566d2af9
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
 2022-01-24 12:12:03 +03:00
Nikolai Kochetov
b3cbb63487 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:41:54 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001 Merge branch 'master' into mvcc_prototype 2022-01-21 18:59:01 +03:00
Alexander Tokmakov
8921725d71 fix logical error on RESTORE REPLICA 2022-01-21 14:32:19 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts 
Fix flushing of in-memory parts
 2022-01-21 02:18:03 +03:00
Alexander Tokmakov
e9a5a64a71 Merge branch 'master' into mvcc_prototype 2022-01-19 21:41:23 +03:00
alesapin
e5ec1aa446
Merge pull request #33642 from ClickHouse/add_integration_test_for_zero_duplication 
Fix bug with background moves in zero copy replication
 2022-01-18 18:32:16 +03:00
alesapin
eff85d3352 Review fixes 2022-01-18 11:27:01 +03:00
alesapin
4caef03e6a Add ephemeral lock in zookeeper 2022-01-17 14:52:51 +03:00
Nikolai Kochetov
6d49a62666 Some more async writes. 2022-01-14 19:53:55 +00:00
alesapin
bf803472b1 Add a tests for a TTL bug in zero copy replication 2022-01-14 18:44:10 +03:00
Nikolai Kochetov
28f2012b06 Add some more async writing. 2022-01-11 19:02:48 +00:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog 
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
 2022-01-10 22:01:41 +03:00
Alexander Tokmakov
bf2b6c2c37 Merge branch 'master' into mvcc_prototype 2022-01-10 19:53:50 +03:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474 2022-01-07 10:12:25 +00:00
Alexander Tokmakov
187c43eba8 rename Committed state to Active 2021-12-30 23:45:38 +03:00
alesapin
91e1ac437e Tiny improvements 2021-12-30 12:57:38 +03:00
alesapin
cbdba89d65 Merge branch 'master' into ianton-ru-MDB-15474 2021-12-30 11:21:19 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
Anton Popov
4ebf61b809 proper checking of atomic flags 2021-12-29 01:03:55 +03:00
Anton Ivashkin
2d87f0a0e3 Fix debug build 2021-12-28 11:45:38 +03:00
Anton Ivashkin
0465aef15d Fixes by code review responces 2021-12-27 19:27:06 +03:00
Anton Popov
a83832c03e fix flushing of in-memory parts 2021-12-27 18:54:28 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474 2021-12-21 17:38:36 +03:00
Anton Ivashkin
33cbfc89ef Move logic for replicated part to StorageReplicatedMergeTree class 2021-12-21 17:29:50 +03:00
Anton Ivashkin
0c0bf66334 Merge master 2021-12-21 17:27:54 +03:00
Alexander Tokmakov
9cd49bc0ec Merge branch 'master' into mvcc_prototype 2021-12-20 22:06:22 +03:00
Anton Ivashkin
c724b074ae Remove zero-copy version converter 2021-12-20 20:23:25 +03:00
Anton Ivashkin
f0b9a4327a Use table UUID in zero-copy shared label in ZooKeeper 2021-12-20 16:19:00 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
alesapin
6bd7e425c6
Merge pull request #22535 from CurtizJ/sparse-serialization 
Sparse serialization and ColumnSparse
 2021-12-17 15:26:17 +03:00
Alexander Tokmakov
e185ad260b fix skipping of some mutations 2021-12-15 21:28:24 +03:00
Alexander Tokmakov
1b02136941 fix skipping of some mutations 2021-12-15 21:19:29 +03:00
Alexander Tokmakov
d7ad72838c Merge branch 'master' into mvcc_prototype 2021-12-14 23:07:52 +03:00
Anton Popov
16312e7e4a Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-14 18:58:17 +03:00
tavplubix
25427719d4
Try fix 'Directory tmp_merge_<part_name>' already exists (#32201) 
* try fix 'directory tmp_merge_<part_name>' already exists

* fix

* fix

* fix

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-12-10 16:29:51 +03:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
Anton Popov
d8367334a3 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-08 18:26:19 +03:00
tavplubix
785d79531d
Update StorageReplicatedMergeTree.cpp 2021-12-08 15:27:32 +03:00
tavplubix
4f46ac6b30
Remove LeaderElection (#32140) 
* remove LeaderElection

* try fix tests

* Update test.py

* Update test.py
 2021-12-07 19:55:55 +03:00
Alexander Tokmakov
7fcb79ae72 Merge branch 'master' into mvcc_prototype 2021-12-07 14:39:29 +03:00
tavplubix
e9489ccd6f
Update StorageReplicatedMergeTree.cpp 2021-12-02 13:18:02 +03:00
Anton Ivashkin
e0a16a483e Merge master 2021-12-01 19:03:31 +03:00
Anton Ivashkin
80ab73c691 Fix Zero-Copy replication lost locks, fix remove used remote data in DROP DETACHED PART 2021-12-01 16:11:26 +03:00
Anton Popov
6f4d9a53b2 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-12-01 15:54:33 +03:00
Anton Popov
54f51444c0 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-01 15:49:02 +03:00
tavplubix
b623a387af
Merge pull request #31887 from ClickHouse/fix_cannot_create_empty_part 
Parse partition key value from `partition_id` when need to create part in empty partition
 2021-12-01 15:38:46 +03:00
tavplubix
78224ef273
Merge pull request #31738 from ClickHouse/fix_segfault_on_attach_partition 
Fix segfault on attach parititon
 2021-11-30 14:32:50 +03:00
Anton Ivashkin
0f9038ebed Zero-copy: move shared mark outside table node in ZooKeeper 2021-11-29 19:05:31 +03:00
Alexander Tokmakov
2fb00172a9 try parse partition key value from partition_id 2021-11-27 15:07:08 +03:00
Raúl Marín
b2cfa70541 Reduce dependencies on ASTFunction.h 
481 -> 230
 2021-11-26 18:21:54 +01:00
Alexander Tokmakov
1ab8b5d4df fix segfault on attach parititon 2021-11-25 13:04:39 +03:00
Anton Popov
ccd78e3838 Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-22 17:19:35 +03:00
Vitaly Baranov
6634fcbac7 Rename Quota::ResourceType -> QuotaType and move it to Access/Common. 2021-11-19 00:14:23 +03:00
Alexander Tokmakov
0a4647f927 support alter partition 2021-11-17 21:14:14 +03:00
Alexander Tokmakov
ce2f692bb4 Merge branch 'master' into mvcc_prototype 2021-11-15 14:56:50 +03:00
Kseniia Sumarokova
2313981fd7
Merge pull request #31260 from azat/external-cleanup 
Cleanup extern ProfileEvents/CurrentMetrics and add a style check
 2021-11-12 00:02:57 +03:00
Alexander Tokmakov
672157b817 Merge branch 'master' into mvcc_prototype 2021-11-11 22:01:34 +03:00
tavplubix
de6a6f3227
Merge pull request #31180 from ClickHouse/merge_selecting_backoff 
Slightly better backoff for merge selecting task
 2021-11-11 20:57:40 +03:00
Azat Khuzhin
baf14444e6 Cleanup ProfileEvents and CurrentMetrics 2021-11-10 21:15:27 +03:00
Anton Popov
a20922b2d3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-11-09 15:36:25 +03:00
Alexander Tokmakov
51bd98294e better merge selecting backoff 2021-11-09 15:26:51 +03:00
Amos Bird
33dfed584d
Fix exception message 2021-11-09 18:19:00 +08:00
Amos Bird
94fe925773
Better 2021-11-09 18:02:17 +08:00
Anton Popov
66973a2a28 Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-08 21:27:45 +03:00
Alexander Tokmakov
92eec74ad7 Merge branch 'master' into mvcc_prototype 2021-11-06 21:08:36 +03:00
Amos Bird
710f7d64e4
Add retry to check alter_partition_version 2021-11-03 15:48:16 +08:00
Anton Popov
d50137013c Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-01 16:55:53 +03:00
Azat Khuzhin
60a411581f Fix possible "The local set of parts of X doesn't look like the set of parts in ZooKeeper" error 
If during removing replica_path from zookeeper, some error occurred
(zookeeper goes away), then it may not remove everything from zookeeper.

And on DETACH/ATTACH (or server restart, like stress tests does in the
analysis from this comment [1]), it will trigger an error:

    The local set of parts of table test_1.alter_table_4 doesn't look like the set of parts in ZooKeeper:

  [1]: https://github.com/ClickHouse/ClickHouse/pull/28296#issuecomment-915829943

Fix this, by removing "metadata" at first, and only after this
everything else, this will avoid this error, since on ATTACH such table
will be marked as read-only.

v2: forget to remove remote_replica_path itself
v3: fix test_drop_replica by adding a check for remote_replica_path existence
 2021-11-01 08:59:28 +03:00
Azat Khuzhin
ca22e7acfb Use existing local remote_replica_path var in StorageReplicatedMergeTree::dropReplica() 2021-11-01 08:53:31 +03:00
Anton Popov
0099dfd523 refactoring of SerializationInfo 2021-10-29 20:21:02 +03:00
tavplubix
f50b2b651b
Merge pull request #30822 from ClickHouse/fix_29052 
Fix ambiguity when extracting auxiliary ZooKeeper name
 2021-10-29 12:15:29 +03:00
tavplubix
bb4a8e91f7
Create StorageReplicatedMergeTree.cpp 2021-10-28 20:25:31 +03:00
Alexander Tokmakov
c1310841d9 fix ambiguity when extracting auxiliary zk name 2021-10-28 19:19:41 +03:00
tavplubix
33ffe11d03
Merge pull request #30346 from ClickHouse/fix_some_fetches_may_stuck 
Try fix 'some fetches may stuck'
 2021-10-28 13:10:15 +03:00
Alexander Tokmakov
5e622d4c98 fix another issue 2021-10-27 15:59:26 +03:00
alesapin
2af950d4d5
Merge pull request #30658 from ClickHouse/std_mutex_alter_lock 
Change `alter_lock` from `RWLock` to `std::timed_mutex`
 2021-10-27 11:14:11 +03:00
Alexander Tokmakov
6be1e98342 fix 2021-10-26 17:06:48 +03:00
mergify[bot]
ce88a84e88
Merge branch 'master' into nv/parts-uuid-move-shard-kill 2021-10-26 11:09:19 +00:00
Alexander Tokmakov
2e7e195e77 change alter_lock to std::timed_mutex 2021-10-26 13:37:00 +03:00
Alexander Tokmakov
94088f8197 minor fixes 2021-10-25 17:01:36 +03:00
Alexander Tokmakov
075c3a37f4 Merge branch 'master' into fix_some_fetches_may_stuck 2021-10-25 15:28:54 +03:00
Anton Popov
b1a74af44d
Merge pull request #30180 from CurtizJ/remove-sample-by 
Allow to remove `SAMPLE BY` expression
 2021-10-21 17:09:53 +03:00
tavplubix
5e24f337a0
Update StorageReplicatedMergeTree.cpp 2021-10-20 09:29:41 +03:00
Alexander Tokmakov
caf972db48 fix 2021-10-19 14:56:04 +03:00
Alexander Tokmakov
89fe606d15 try fix 'some fetches may stuck' 2021-10-18 23:16:02 +03:00
Anton Popov
d71ffc355a Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-18 15:18:22 +03:00
alexey-milovidov
4f11cfa59d
Merge pull request #28760 from azat/mutator-forbid-cleaner 
Forbid cleaning of tmp directories that can be used by an active mutation/merge.
 2021-10-17 03:18:01 +03:00
Azat Khuzhin
07e8b2b3c7 Do not try to remove temporary paths that is currently in written by merge/mutation 
v2: rebase against MergeTask
v3: rebase due to conflicts in src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
v4:
- rebase due to conflicts in src/Storages/MergeTree/MergeTask.cpp
- drop common/scope_guard_safe.h (not used)
 2021-10-16 00:43:52 +03:00
Anton Popov
7aa6068fb2 Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-14 19:44:08 +03:00
Anton Popov
476d7a411f allow to remove SAMPLE BY expression 2021-10-14 17:05:20 +03:00
Nikolai Kochetov
ab28c6c855 Remove BlockInputStream interfaces. 2021-10-14 13:25:43 +03:00
Nikolai Kochetov
340b53ef85 Remove some more streams. 2021-10-08 17:03:54 +03:00
Nikolai Kochetov
9eddee5517
Merge pull request #29768 from ClickHouse/remove-merging-streams 
Remove some merging streams
 2021-10-08 13:15:10 +03:00
Alexander Tokmakov
4119848a41 better diagnostic for OPTIMIZE 2021-10-07 13:08:10 +03:00
mergify[bot]
ab55225c8f
Merge branch 'master' into remove-merging-streams 2021-10-05 14:16:34 +00:00
mergify[bot]
3b1f49aac7
Merge branch 'master' into nv/parts-uuid-move-shard-kill 2021-10-05 12:36:02 +00:00
Nikolai Kochetov
2001ebbf9d Fix build. 2021-10-04 21:52:31 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Nikita Mikhaylov
9756b8dc33
Added an ability to execute more merges and mutations than threads, added new settings (#29140) 2021-10-01 00:26:24 +03:00
Anton Popov
914781052e Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-29 17:37:07 +03:00
Alexander Tokmakov
72b1b2e360 Merge branch 'master' into mvcc_prototype 2021-09-23 22:53:27 +03:00
Nikolai Kochetov
998d29ebc7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-23 13:10:27 +03:00
Anton Popov
0fbfac056d better diagnostic in ReplicatedMergeTree 2021-09-21 22:42:03 +03:00
Nikolai Kochetov
a790d391c0 Merge branch 'master' into rewrite-pushing-to-views 2021-09-20 19:43:15 +03:00
Anton Popov
6f9e53197c Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-20 17:17:05 +03:00
Nicolae Vartolomei
9a02061d9c
Rewrite part movement between shards logic and add kill support 
See https://github.com/ClickHouse/ClickHouse/pull/24585#issuecomment-857735081
for extra context about the current implementation.
 2021-09-17 18:11:32 +01:00
Nicolae Vartolomei
0381c634d4
Add support for user defined identifier on log entries 
Sometimes we want to push a log entry once and only once. Because it is
not possible to create a sequential node in ZooKeeper and store its name
to a well known location in the same transaction we'll do it in the
other order. First somehow generate a unique identifier, then submit a
log entry with that identifier. Later, we can search through log entries
using the identifier we provided to find the node.

Required for part movement between shards.
 2021-09-17 15:32:35 +01:00
Nicolae Vartolomei
c428f433c3
Use do-while control loops for waiting on log processing 
This allows to use the same functions with very short timeouts while
ensuring that the actual state is checked at least once instead of
timing out before even looking at at ZK at least once.
 2021-09-17 15:32:35 +01:00
Nikolai Kochetov
a8c3b02598 Merge branch 'master' into rewrite-pushing-to-views 2021-09-17 15:38:11 +03:00
mergify[bot]
c23f1fc160
Merge branch 'master' into fix_part_checker 2021-09-17 07:23:12 +00:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask (#25165) 
Introduced MergeTask and MutateTask
 2021-09-17 00:19:58 +03:00
alesapin
2d9fa85c0a
Merge pull request #28889 from azat/fix-mutation-wait 
Fix waiting for mutation with mutations_sync=2
 2021-09-16 19:10:56 +03:00
Anton Popov
eef436fe22 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-16 18:07:42 +03:00
mergify[bot]
0cbd4eda60
Merge branch 'master' into fix_part_checker 2021-09-14 19:44:11 +00:00
alesapin
5a755d1e7c
Merge pull request #29009 from ClickHouse/improve_error_message 
Better error message on concurrent alters
 2021-09-14 17:21:26 +03:00
Nikolai Kochetov
0e267c50b4 Merge branch 'master' into rewrite-pushing-to-views 2021-09-14 16:13:54 +03:00
alesapin
797df22418 Fix build 2021-09-14 14:45:33 +03:00
alesapin
76b8d9d85f Better error message on concurrent alters 2021-09-14 13:57:05 +03:00
mergify[bot]
c54434084f
Merge branch 'master' into fix_part_checker 2021-09-13 13:19:34 +00:00
alesapin
1d05fab295
Merge pull request #28864 from ClickHouse/fix_replace_range_drop_part_reordering 
Fix reordering of REPLACE_RANGE and DROP PART
 2021-09-13 15:11:29 +03:00
alesapin
ab41384f63 Move queue initialization to restarting thread 2021-09-13 11:00:07 +03:00
alesapin
91e10adca5 Merge branch 'master' into fix_part_checker 2021-09-13 10:19:04 +03:00
alexey-milovidov
ea13a8b562
Merge pull request #28659 from myrrc/improvement/tostring_to_magic_enum 
Improving CH type system with concepts
 2021-09-12 15:26:29 +03:00
Azat Khuzhin
2f1a58f4b0 Fix waiting for mutation with mutations_sync=2 
CI: https://clickhouse-test-reports.s3.yandex.net/28827/c6b9816d45e3e8255eb1aee9b9059dfe7397fdab/functional_stateless_tests_(thread).html#fail1
 2021-09-11 15:23:00 +03:00
Alexander Tokmakov
518f542c3e fix reordering of REPLACE_RANGE and DROP PART 2021-09-10 17:32:45 +03:00
alexey-milovidov
2236a5df98
Merge pull request #28085 from myrrc/improvement/merge-tree-part-opt-parse 
optional<> semantics for parsing MergeTreePartInfo and DetachedPartInfo
 2021-09-10 00:04:03 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
alesapin
59edf6b5f6 Better logic on part checker 2021-09-09 18:19:12 +03:00
Anton Popov
4c388e3d84 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-09-09 14:10:16 +03:00
Anton Popov
8203bd1ac6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-09 14:04:37 +03:00
Nikita Mikhaylov
4db5062d6b
Merge pull request #28374 from nikitamikhaylov/global-merge-executor 
Introduced global executor for background MergeTree-related operations
 2021-09-09 11:30:21 +03:00
Nikolai Kochetov
999a4fe831 Fix other tests. 2021-09-08 21:29:38 +03:00
Nikita Mikhaylov
6062dd0021 Better 2021-09-08 00:21:21 +00:00
Azat Khuzhin
db0767a194 Implement detach_not_byte_identical_parts 
Maybe useful for further analysis of non byte-identical parts.
 2021-09-07 23:29:57 +03:00
Mike Kot
8e9aacadd1 Initial: replacing hardcoded toString for enums with magic_enum 2021-09-06 16:24:03 +02:00
Nikita Mikhaylov
ea0fbf81af Renaming 2021-09-06 12:01:16 +00:00
mergify[bot]
8eba5f29a1
Merge branch 'master' into improvement/merge-tree-part-opt-parse 2021-09-06 09:53:46 +00:00
Nikita Mikhaylov
292a24abe8 Merge upstream/master into global-merge-executor (using imerge) 2021-09-03 00:34:24 +00:00
Nikita Mikhaylov
cc7c221fad Own PriorityQueue + prettifying the code 2021-09-02 21:31:32 +00:00
mergify[bot]
5d299fbdee
Merge branch 'master' into remove_outdated_settings 2021-09-01 14:07:48 +00:00
alesapin
fd1581aee1 Fix style 2021-09-01 10:52:33 +03:00
Nikita Mikhaylov
dbc950caa4 added a test 2021-08-31 14:54:24 +00:00
Mike Kot
4194ed2ec7 Merge remote-tracking branch 'upstream/master' into improvement/merge-tree-part-opt-parse 2021-08-31 14:35:14 +02:00
alesapin
921e51e061 Remove some obsolete settings for replicated fetches 2021-08-31 15:22:56 +03:00
Nikita Mikhaylov
f8d4f04294 Merge upstream/master into global-merge-executor (using imerge) 2021-08-31 11:52:11 +00:00
Nikita Mikhaylov
1adb9bfe23 better 2021-08-31 11:02:39 +00:00
Nikita Mikhaylov
c4416906c8 done 2021-08-30 19:37:03 +00:00
Alexey Milovidov
79e0433ba7 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-28 01:19:16 +03:00
tavplubix
703101fe4d
Merge pull request #27931 from ClickHouse/wait_for_all_replicas_timeouts 
Avoid too long waiting for inactive replicas
 2021-08-27 14:31:36 +03:00
Amos Bird
0169fce78e
Projection bug fixes and refactoring. 2021-08-26 19:09:31 +08:00
Alexey Milovidov
7c1d0a3baf Progress on development 2021-08-25 01:24:47 +03:00
Mike Kot
d621605954 Merge remote-tracking branch 'upstream/master' into improvement/merge-tree-part-opt-parse 2021-08-24 19:43:33 +02:00
Mike Kot
f85e2e027c Optional semantics for [Detached]MergeTreePartInfo 2021-08-24 19:43:09 +02:00
tavplubix
0602d74a11
Merge pull request #28035 from ClickHouse/fix_replace_ranges_may_stuck 
Fix race between REPLACE PARTITION and MOVE PARTITION
 2021-08-24 17:49:57 +03:00
Alexander Tokmakov
d95131dd4a better check for alter partition version 2021-08-23 20:51:20 +03:00
Alexander Tokmakov
cc9c2fd63b make code better 2021-08-23 15:57:50 +03:00
Anton Popov
61239343e3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-20 16:33:30 +03:00
Alexander Tokmakov
59eb3aa9a9 avoid too long waiting for inactive replicas 2021-08-20 15:59:57 +03:00
Anton Popov
f3a77bb20d slightly better 2021-08-20 02:22:50 +03:00
Anton Popov
c3c3a06078 Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-20 01:45:38 +03:00
nvartolomei
c09c90125f
Merge branch 'master' into nv/last-queue-update-exception 2021-08-19 10:29:16 +01:00
Alexander Tokmakov
0ed046eb7b remove irrelevant comments 2021-08-18 15:33:11 +03:00
Alexander Tokmakov
09ff66da0e fix a couple of bugs that may cause replicas to diverge 2021-08-18 12:50:46 +03:00
mergify[bot]
f11e396151
Merge branch 'master' into nv/last-queue-update-exception 2021-08-18 07:00:50 +00:00
Nicolae Vartolomei
3f291b024a
Use plain mutex instead of MultiVersion 2021-08-09 13:58:23 +01:00
Maksim Kita
7fdf3cc263
Merge pull request #27180 from kitaisreal/storage-system-replicas-added-column-replica-is-active 
Storage system replicas added column replica is active
 2021-08-05 12:46:53 +03:00
Maksim Kita
3f48c85722 StorageSystemReplicas added column replica_is_active 2021-08-04 16:19:42 +03:00
Anton Popov
e36736b50c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-02 22:52:02 +03:00
Anton Popov
16ed0f6ed4 Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-02 17:55:17 +03:00
Alexander Tokmakov
42a8bb6872 fix assertions in Replicated database 2021-08-02 16:19:11 +03:00
Alexander Tokmakov
c74bfbf991 Merge branch 'master' into mvcc_prototype 2021-07-28 22:21:48 +03:00
alesapin
71169d7937
Merge pull request #26716 from nvartolomei/nv/part-cleanup-sequence 
Avoid deleting old parts from FS on shutdown for replicated engine
 2021-07-28 18:29:37 +03:00
Nicolae Vartolomei
8b07a7f180 Store exception generated when we tried to update the queue last time 
The use case is to alert when queue contains broken entries. Especially
important when ClickHouse breaks backwards compatibility between
versions and log entries written by newer versions aren't parseable by
old versions.

```
Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected 'quorum: ' before: 'merge_type: 2\n'
```
 2021-07-27 15:42:40 +01:00
Nikolai Kochetov
61d8f880cd Rename some files. 2021-07-26 19:48:25 +03:00
Nikolai Kochetov
9c92f43359 Update storages. 2021-07-23 22:33:59 +03:00
Nicolae Vartolomei
f35e6eee19 Avoid deleting old parts from FS on shutdown for replicated engine 
This was introduced in https://github.com/ClickHouse/ClickHouse/pull/8602.
The idea was to avoid data re-appearing in ClickHouse after DROP/DETACH
PARTITION. This problem was only present in MergeTree engine and I don't
understand why we need to do the same in ReplicatedMergeTree.

For ReplicatedMergeTree the state of truth is stored in ZK, deleting
things from filesystem just introduces inconsistencies and this is the
main source for errors like "No active replica has part X or covering
part".

The resulting problem is fixed by
https://github.com/ClickHouse/ClickHouse/pull/25820, but in my opinion
we would better avoid introducing the ZK/FS inconsistency in the first
place.

When does this inconsistency appear? Often the sequence is like this:

0. Write 2 parts to ZK [all_0_0_0, all_1_1_0]
1. A merge gets scheduled
2. New part replaces old parts [new: all_0_1_1, old: all_0_0_0, all_1_1_0]
3. Replica gets shutdown and old parts are removed from filesystem
4. Replica comes back online, metadata about all parts is still stored in ZK for this new replica.
5. Other replica after cleanup thread runs will have only [all_0_1_1] in
   ZK
5. User triggers a DROP_RANGE after a while (drop range is for all_0_1_9999*)
6. Each replica deletes from ZK only [all_0_1_1]. The replica that got
   restarted uses its in-memory state to choose nodes to delete from ZK.
7. Restart the replica again. It will now think that there are 2 parts
   that it lost and needs to fetch them [all_0_0_0, all_1_1_0].

`clearOldPartsAndRemoveFromZK` which is triggered from cleanup thread
runs cleanup sequence correctly, it first removes things from ZK and
then from filesystem. I don't see much benefit of triggering it on
shutdown and would rather have it called only from a single place.

---

This is a very, very edge case situation but it proves that the current
"fix" (https://github.com/ClickHouse/ClickHouse/pull/25820) isn't
complete.

```
create table test(
    v UInt64
)
engine=ReplicatedMergeTree('/clickhouse/test', 'one')
order by v
settings old_parts_lifetime = 30;

create table test2(
    v UInt64
)
engine=ReplicatedMergeTree('/clickhouse/test', 'two')
order by v
settings old_parts_lifetime = 30;

create table test3(
    v UInt64
)
engine=ReplicatedMergeTree('/clickhouse/test', 'three')
order by v
settings old_parts_lifetime = 30;

insert into table test values (1), (2), (3);
insert into table test values (4);

optimize table test final;

detach table test;
detach table test2;

alter table test3 drop partition tuple();

attach table test;
attach table test2;
```

```
(CONNECTED [localhost:9181]) /> ls /clickhouse/test/replicas/one/parts
all_0_0_0
all_1_1_0
(CONNECTED [localhost:9181]) /> ls /clickhouse/test/replicas/two/parts
all_0_0_0
all_1_1_0
(CONNECTED [localhost:9181]) /> ls /clickhouse/test/replicas/three/parts
```

```
detach table test;
attach table test;
```

`test` will now figure out that parts exist only in ZK and will issue `GET_PART`
after first removing parts from ZK.

`test2` will receive fetch for unknown parts and will trigger part checks itself.
Because `test` doesn't have the parts anymore in ZK `test2` will mark them as LostForever.
It will also not insert empty parts, because the partition is empty.

`test` is left with `GET_PART` in the queue and stuck.

```
SELECT
    table,
    type,
    replica_name,
    new_part_name,
    last_exception
FROM system.replication_queue

Query id: 74c5aa00-048d-4bc1-a2ea-6f69501c11a0

Row 1:
──────
table:          test
type:           GET_PART
replica_name:   one
new_part_name:  all_0_0_0
last_exception: Code: 234. DB::Exception: No active replica has part all_0_0_0 or covering part. (NO_REPLICA_HAS_PART) (version 21.9.1.1)

Row 2:
──────
table:          test
type:           GET_PART
replica_name:   one
new_part_name:  all_1_1_0
last_exception: Code: 234. DB::Exception: No active replica has part all_1_1_0 or covering part. (NO_REPLICA_HAS_PART) (version 21.9.1.1)
```
 2021-07-22 17:48:16 +01:00
Anton Popov
f99374cca6 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-07-20 18:20:21 +03:00
Anton Popov
c4b454494f Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-20 15:41:01 +03:00
tavplubix
41bb8acbb5
Update StorageReplicatedMergeTree.cpp 2021-07-14 20:05:50 +03:00
Filatenkov Artur
f1702d356e
Merge pull request #26120 from lthaooo/settings-merge_selecting_sleep_ms 
add merge_selecting_sleep_ms setting
 2021-07-14 18:28:47 +03:00
Zhichang Yu
b4e6689bf9 fix test_hdfs_zero_copy_replication_move[tiered_copy-2] 2021-07-13 07:20:23 +00:00
Zhichang Yu
5047c758f4 fix per review 2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1 hdfs zero copy 2021-07-13 07:19:12 +00:00
terrylin
faed6263bb add merge_selecting_sleep_ms setting 2021-07-09 17:29:17 +08:00
Anton Popov
3ed7f5a6cc dynamic subcolumns: add snapshot for storage 2021-07-09 06:15:41 +03:00
Anton Popov
14168b11f2 Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-07 17:05:11 +03:00
alesapin
4c85dae572
Merge pull request #25743 from ClickHouse/fix_aggregation_ttl 
Fix bug in execution of TTL GROUP BY
 2021-07-07 10:49:16 +03:00
alesapin
0d8844c510
Merge pull request #25884 from ClickHouse/fix_drop_part_in_queue 
Relax `DROP PART` guarantees and turn on checks in ReplicationQueue.
 2021-07-07 10:48:48 +03:00
alesapin
f7e1cfdb24 Some partially working code 2021-07-05 22:58:55 +03:00
Anton Popov
9071ecd428 fix alter of settings in MergeTree 2021-07-05 15:44:58 +03:00
mergify[bot]
9da1c98998
Merge branch 'master' into fix_aggregation_ttl 2021-07-03 15:07:44 +00:00
alesapin
13c008c7a8 Change exception type 2021-07-02 16:38:46 +03:00
alesapin
2e29dc2975 More safe empty parts creation 2021-07-02 12:30:17 +03:00
mergify[bot]
98fa9f7951
Merge branch 'master' into better_remove_empty_parts 2021-07-01 11:32:43 +00:00
tavplubix
afbc6bf17d
Merge pull request #25684 from ClickHouse/cancel_merges_on_drop_partition 
Cancel merges on DROP PARTITION
 2021-06-30 23:41:04 +03:00
alesapin
0193a9d087 Fix locking 2021-06-30 22:41:25 +03:00
alesapin
6a73c8b49e Review fixes 2021-06-30 18:24:51 +03:00
mergify[bot]
1799804243
Merge branch 'master' into fix_aggregation_ttl 2021-06-30 09:15:46 +00:00
alesapin
a6834213a1 Add tests 2021-06-29 22:47:54 +03:00
alesapin
5822f0ba29 Replace lost parts with empty parts instead of hacking replication queue 2021-06-29 18:14:44 +03:00
Raúl Marín
bfc122df64 Fix some typos in Storage classes 2021-06-28 19:03:56 +02:00
alesapin
b11254d191
Merge branch 'master' into fix_aggregation_ttl 2021-06-28 13:31:29 +03:00
alesapin
71603a7d13
Merge pull request #25772 from ClickHouse/fix_mutation_test 
Fix flaky test and wrong message
 2021-06-28 13:30:36 +03:00
alesapin
4c213b639f
Merge pull request #25548 from nikitamikhaylov/background-processing 
A little improvement in BackgroundJobsExecutor
 2021-06-28 13:24:21 +03:00
alesapin
7e73762b48 Fix flaky test and wrong message 2021-06-28 11:28:45 +03:00
alesapin
c977c33d6d Fix bug in execution of TTL GROUP BY 2021-06-27 19:18:15 +03:00
Alexander Tokmakov
76156af5cc cancel merges on drop partition 2021-06-24 17:07:43 +03:00
alesapin
4be4bc21e2 Fix for fix 2021-06-23 23:57:49 +03:00
alesapin
81c74435a3 Fix drop part bug 2021-06-23 22:25:30 +03:00
Nikita Mikhaylov
c66a3b22b5 done 2021-06-22 23:24:47 +00:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652) 
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
 2021-06-20 11:24:43 +03:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
tavplubix
e2ecc51a1f
Merge pull request #25087 from ClickHouse/srmt_remove_copypaste 
Remove copypaste from StorageReplicatedMergeTree
 2021-06-15 12:43:55 +03:00
alexey-milovidov
abe206f4fc
Merge pull request #25037 from nvartolomei/nv/queue-entry-wait-dead-code 
Update waitForTableReplicaToProcessLogEntry comments
 2021-06-12 03:11:13 +03:00
Alexander Tokmakov
6f5cf773ef Merge branch 'master' into mvcc_prototype 2021-06-09 12:50:54 +03:00
Alexander Tokmakov
3ade38df82 remove copypaste 2021-06-08 22:17:45 +03:00
Alexander Tokmakov
9a88b9dacc set maxtid on drop part 2021-06-08 21:17:18 +03:00
Azat Khuzhin
1062d0ec91 Distinguish KILL MUTATION for different tables. 
Before this patch KILL MUTATION marks mutation as canceled just by name
(and part numbers) so if you have multiple tables with the same part
name, then killing mutation for one table, will mark it as killed for
another too.

Fix this by comparing StorageID too (it is better to use StorageID over
database/table to avoid ambiguity by using UUIDs for comparing).

Here is a failure of the 01414_freeze_does_not_prevent_alters on CI [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/24069/9fb69dcf98c71a939d200cad3c8491bf43a44622/functional_stateless_tests_(ubsan).html#fail1
 2021-06-08 10:51:22 +03:00
Alexander Tokmakov
881889ef22 parts cleanup, fixes 2021-06-07 21:22:58 +03:00
Nicolae Vartolomei
19f64b3f25 Update waitForTableReplicaToProcessLogEntry comments 2021-06-07 11:01:57 +01:00
alesapin
2ea9d998e8
Merge pull request #24960 from nvartolomei/nv/queue-entry-wait-dead-code 
Delete support for waiting on queue- entries, is this dead code?
 2021-06-07 12:40:34 +03:00
Nicolae Vartolomei
a9d108fc5f Delete support for waiting on queue- entries, is this dead code? 2021-06-04 12:43:46 +01:00
alesapin
4127bbbb9c Remove compatibility code with 18.12 2021-06-04 13:20:15 +03:00
alesapin
4a3fd34a9d Fix typo 2021-06-04 11:33:07 +03:00
alesapin
f9c2f6925d Don't try if source replica was dropped 2021-06-04 11:32:33 +03:00
alesapin
1a421260b3 Fix endless wait in replica clone 2021-06-04 11:22:24 +03:00
tavplubix
1aff716f18
Merge pull request #24777 from ClickHouse/fix_intersection_with_lost_part 
Do not crash on intersecting virtual parts
 2021-06-03 15:42:46 +03:00
Alexander Tokmakov
9915bd0a8b Merge branch 'master' into mvcc_prototype 2021-06-03 00:52:57 +03:00
Alexander Tokmakov
9a9e95172f add test with mv 2021-06-02 23:03:44 +03:00
alesapin
4a1bf51ad1
Merge branch 'master' into limited_fetches 2021-06-02 23:01:47 +03:00
Nikolai Kochetov
134ad18249
Merge pull request #24574 from ClickHouse/refactor-MergeTreeDataSelectExecutor 
Refactor MergeTreeDataSelectExecutor
 2021-06-02 13:17:54 +03:00
alesapin
1c0973162d
Merge pull request #24803 from ClickHouse/followupfix_zookeeper_timeouts 
Don't capture temporary references in ZooKeeper client callbacks.
 2021-06-02 01:08:17 +03:00
Alexander Tokmakov
7d7c5638a5 Merge branch 'master' into fix_intersection_with_lost_part 2021-06-01 16:32:59 +03:00
Alexander Tokmakov
5969891611 do not crash on intersecting parts 2021-06-01 16:25:23 +03:00
Nikolai Kochetov
e615858b10 Merge branch 'master' into refactor-MergeTreeDataSelectExecutor 2021-06-01 16:25:04 +03:00
alexey-milovidov
c7bd3b8674
Merge pull request #23315 from CurtizJ/empty-parts-fix 
Fix waiting while dropping empty parts
 2021-06-01 13:52:49 +03:00
alesapin
c3321bef30 Merge branch 'master' into limited_fetches 2021-06-01 13:09:02 +03:00
Nikolai Kochetov
b54ff696e5 Merge branch 'master' into refactor-MergeTreeDataSelectExecutor 2021-06-01 12:08:37 +03:00
alesapin
b7c4f7dff8
Merge pull request #24809 from ClickHouse/fix_kill_mutation_wait 
Fix mutation wait on mutation kill
 2021-06-01 10:37:24 +03:00
alesapin
4e038b2a61 Remove copy-paste and rename (review fixes) 2021-06-01 10:30:06 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs 
Poco::File to std::filesystem
 2021-05-31 23:17:02 +03:00
Alexander Tokmakov
5692db736c Merge branch 'master' into mvcc_prototype 2021-05-31 21:11:22 +03:00
alesapin
18988a60ad Fix mutation wait 2021-05-31 18:03:45 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
alesapin
2fe75e33ee Replace Poco::Event with std future in synchronous calls in ZooKeeper client 2021-05-31 16:03:30 +03:00
Alexander Tokmakov
16647fe8ce some unrelated fixes 2021-05-31 00:29:37 +03:00
Nikolai Kochetov
a6ef5778cc
Merge pull request #24692 from amosbird/projection-fix6 
Fix alter table drop projection if exists
 2021-05-30 12:16:52 +03:00
Amos Bird
edd23d06ad
Fix alter table drop projection if exists 2021-05-28 21:27:36 +08:00
Anton Popov
179f172c4c fix build 2021-05-28 02:10:44 +03:00
Anton Popov
336abfa475 Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-28 01:58:30 +03:00
Nikolai Kochetov
936bd8a011 Merge branch 'master' into refactor-MergeTreeDataSelectExecutor 2021-05-27 20:47:39 +03:00
Nikolai Kochetov
1aeb705b20 Fix some tests. 2021-05-27 19:53:58 +03:00
alesapin
2985def91d Add tests 2021-05-27 15:54:47 +03:00
alesapin
42b2383e01 Basic throttler implementation 2021-05-26 23:37:44 +03:00
kssenii
73f16ee9ee Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-26 23:08:08 +03:00
alesapin
069d45790f
Merge pull request #24157 from kssenii/modulo-partition-key 
Adjust modulo function when used in partition key
 2021-05-26 10:30:54 +03:00
tavplubix
84f9d38620
Merge pull request #24321 from amosbird/fixdroppartition 
Fix drop partition in rare case
 2021-05-26 01:34:22 +03:00
Anton Popov
7d1431f6b6 better code near dropPartition 2021-05-25 20:25:00 +03:00
tavplubix
6b228f2630
Update StorageReplicatedMergeTree.cpp 2021-05-24 15:13:42 +03:00
tavplubix
e0f214d0e9
Update StorageReplicatedMergeTree.cpp 2021-05-24 14:23:23 +03:00
kssenii
659e57e60f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into modulo-partition-key 2021-05-23 22:46:00 +00:00
kssenii
bbf56e35ac Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-23 10:54:48 +03:00
kssenii
096d117f68 Review fixes 2021-05-22 00:12:46 +03:00
kssenii
94b5763dbd Revert "Different approach to fix modulo" 
This reverts commit 665f1aa0f6.
 2021-05-21 16:14:01 +00:00
kssenii
665f1aa0f6 Different approach to fix modulo 2021-05-21 14:30:08 +00:00
alesapin
20a34e951a Split fetch part into two stages, don't use replicated block output stream 2021-05-21 12:30:49 +03:00
alesapin
deb784a321 Merge branch 'master' into nvartolomei-parts-move 2021-05-20 21:24:55 +03:00
kssenii
3dee003f9b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-20 19:20:09 +03:00
Amos Bird
81dd60fdc2
Fix drop partition in rare case 2021-05-20 14:55:28 +08:00
mergify[bot]
29b3ed4551
Merge branch 'master' into modulo-partition-key 2021-05-20 05:45:11 +00:00
Nikolai Kochetov
e8f8b2ba18
Merge pull request #24162 from amosbird/projection-fix2 
Fix add projection to replicated mergetree
 2021-05-19 18:07:52 +03:00
mergify[bot]
230bec70fb
Merge branch 'master' into modulo-partition-key 2021-05-19 10:08:41 +00:00
alesapin
4d6c71801c
Merge branch 'master' into nv/parts-uuid-move-shard 2021-05-19 13:03:14 +03:00
Kseniia Sumarokova
0ab15d0bc4
Merge branch 'master' into modulo-partition-key 2021-05-18 17:45:32 +03:00
Alexander Tokmakov
f2ab5a05c6 Merge branch 'master' into mvcc_prototype 2021-05-18 13:36:38 +03:00
Alexander Tokmakov
1fd6142c1a try enable merges 2021-05-18 13:35:48 +03:00
Alexander Tokmakov
2dac182df3 fix 2021-05-18 13:00:34 +03:00
Alexander Tokmakov
20b9af7b26 do not allow concurrent alter partition 2021-05-18 00:20:58 +03:00
kssenii
9b8df78fdd Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-17 17:42:05 +03:00
Anton Popov
e27acc26be better dropPart and dropPartition 2021-05-17 17:26:36 +03:00
Anton Popov
e043cc45bf Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-17 16:11:12 +03:00
alesapin
67d34c0136 merge with master 2021-05-17 14:13:18 +03:00
alesapin
17f229857c Merge branch 'master' into nvartolomei-parts-move 2021-05-17 13:52:48 +03:00
Amos Bird
e5cd597111
Fix add projection to replicated mergetree 2021-05-16 16:49:38 +08:00
kssenii
c7f51fe60e Modulo adjustment for partition key 2021-05-15 19:51:02 +00:00
Alexander Tokmakov
df5f3fbc9d review suggestions 2021-05-14 19:11:40 +03:00