Alexander Tokmakov
b8d9066004
Revert "Resurrect parallel distributed insert select with s3Cluster ( #41535 )"
...
This reverts commit 860e34e760
2022-10-07 15:53:30 +02:00
alesapin
3373480ede
Merge pull request #42036 from ClickHouse/fix_one_more_no_such_key
...
Fix No such key during table drop
2022-10-06 14:50:02 +02:00
Nikita Mikhaylov
860e34e760
Resurrect parallel distributed insert select with s3Cluster ( #41535 )
2022-10-06 13:47:32 +02:00
alesapin
210882b9c4
Better fix
2022-10-03 23:30:50 +02:00
alesapin
3114ddcad2
Fix failure during table drop
2022-10-03 19:06:12 +02:00
Nikita Mikhaylov
dbcba1490c
Try to make MergeTreeRestartingThread more reactive in case of expired sessions ( #41092 )
2022-09-23 14:40:04 +02:00
alesapin
acde29e314
Nasty code
2022-09-20 17:55:06 +02:00
alesapin
cfe398cbbc
Fix lost table shared id during server restart
2022-09-20 14:51:16 +02:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze
...
Fix SYSTEM UNFREEZE for ordinary database
2022-09-13 14:55:11 +03:00
Alexander Tokmakov
da4fbb552d
Merge branch 'master' into fix_parts_removal
2022-09-12 17:49:29 +03:00
alesapin
21ab379026
Fix comment
2022-09-11 13:47:04 +02:00
alesapin
ac53830000
Fix hung check
2022-09-11 13:37:39 +02:00
alesapin
e3fc78aa5f
Fix bad warnings in part fetches
2022-09-09 12:14:42 +02:00
ianton-ru
39e1fc7a0f
Merge branch 'master' into fix-ordinary-system-unfreeze
2022-09-05 17:10:59 +03:00
Luck-Chang
1ac8e739c9
fix some typos and clang-tidy warnings
2022-09-05 09:50:24 +08:00
Alexander Tokmakov
eae2667a1c
Merge pull request #40148 from ClickHouse/replicate-tables-readonly
...
`ReplicatedMergeTree` tables as readonly until attach is finished
2022-08-25 19:39:19 +03:00
Antonio Andelic
797e21761a
Address PR comments
2022-08-24 18:32:03 +00:00
Antonio Andelic
da0d6056f4
Merge branch 'master' into replicate-tables-readonly
2022-08-24 17:29:33 +00:00
alesapin
eeb0a3584e
Merge branch 'master' into data_source_description
2022-08-24 15:26:18 +02:00
Alexander Tokmakov
f9f85a0e8b
Revert "Parallel distributed insert select from *Cluster table functions ( #39107 )"
...
This reverts commit d3cc234986
2022-08-24 15:17:15 +03:00
alesapin
d8664c3227
Add shortcut for backups
2022-08-19 16:58:30 +02:00
Antonio Andelic
ee3df77fda
Simplify logic
2022-08-19 11:12:20 +00:00
Antonio Andelic
5e2c8869a9
Merge branch 'master' into replicate-tables-readonly
2022-08-19 09:41:11 +00:00
Antonio Andelic
a8f3b62bf6
Use global zookeeper
2022-08-19 08:49:51 +00:00
Alexander Tokmakov
832d7c6e7c
Merge branch 'master' into replicated_database_improvements
2022-08-16 15:49:13 +02:00
Antonio Andelic
2ed42bca7e
Merge branch 'master' into replicate-tables-readonly
2022-08-16 08:17:53 +00:00
Nikita Mikhaylov
d3cc234986
Parallel distributed insert select from *Cluster table functions ( #39107 )
2022-08-15 12:41:17 +02:00
Antonio Andelic
c6b3a897d2
Test replicated table without ZK, small fixes
2022-08-12 11:56:46 +00:00
Antonio Andelic
1c0f85ccdb
Attach table in background
2022-08-12 09:32:13 +00:00
Alexander Tokmakov
6f88065cc1
check metadata consistency
2022-07-29 18:33:16 +02:00
jianmei zhang
c1834d183b
ReplicatedMergeTree support lightweight delete
2022-07-26 11:50:09 +08:00
Vadim Volodin
85bf022258
review zookeeper and partition matching
2022-07-21 16:57:28 +03:00
Vitaly Baranov
7689e0c36f
Improve gathering metadata for backup - part 6.
2022-06-30 08:37:17 +02:00
Vitaly Baranov
461a31f237
Improve gathering metadata for backup - part 2.
2022-06-30 08:37:17 +02:00
Vitaly Baranov
64b51a3772
Improve gathering metadata for backup.
2022-06-30 08:37:17 +02:00
Vadim Volodin
3296ba2532
Fix SYSTEM UNFREEZE for ordinary database
2022-06-24 20:46:16 +03:00
Nikolai Kochetov
7452d04e3a
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-20 15:31:02 +02:00
Vitaly Baranov
c2c35fad82
Refactoring of the code getting create table queries for backup.
2022-06-15 20:32:35 +02:00
Vitaly Baranov
592f568f83
Move backup/restore code to storages and databases - part 2.
2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57
Move backup/restore code to storages and databases - part 1.
2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21
Rework collecting replicated parts.
2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982
Restore parts of MergeTree in correct order.
2022-06-15 20:28:40 +02:00
Nikolai Kochetov
2a9a63ac7b
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-15 15:35:26 +02:00
Vadim Volodin
637d293fbd
Add SYSTEM UNFREEZE query
2022-06-08 15:21:14 +03:00
Nikolai Kochetov
678d978acf
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-07 15:23:00 +00:00
Nikolai Kochetov
89c5855d20
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-02 12:19:07 +02:00
Nikolai Kochetov
c71256ea38
Remove some commented code.
2022-05-30 13:18:20 +00:00
Nikolai Kochetov
1b85f2c1d6
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-25 16:27:40 +02:00
Nikolai Kochetov
fd97a9d885
Move some resources
2022-05-23 19:47:32 +00:00
Alexander Tokmakov
db66efb1a7
fix logical error on truncate table
2022-05-06 16:12:31 +02:00
Nikolai Kochetov
35095191eb
Merge branch 'master' into refactor-something-in-part-volumes
2022-05-03 17:51:47 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable
...
... IStorage has deleted copy ctor / assignment already
2022-05-03 09:07:08 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying
...
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
previously allowed.
Hence, this change
- removes shared_ptr_helper and as a result all inherited create() methods,
- instead, Storage objects are now created using make_shared<>() by the
caller (for that to work, many constructors had to be made public), and
- all Storage classes were marked as noncopyable using boost::noncopyable.
In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
2022-05-02 08:46:52 +02:00
Nikolai Kochetov
e44af67fee
Merge branch 'master' into refactor-something-in-part-volumes
2022-04-26 21:08:00 +02:00
Vitaly Baranov
68a020ecea
Implement BACKUP/RESTORE ON CLUSTER.
2022-04-25 16:34:33 +02:00
Vitaly Baranov
acd28d8a1d
Implement RESTORE for replicated tables.
2022-04-25 16:34:33 +02:00
alesapin
ebdd53ef8f
Fix shared fetch
2022-04-22 19:18:18 +02:00
Nikolai Kochetov
8c00692844
Part 8
2022-04-22 16:58:09 +00:00
Nikolai Kochetov
9133e398b8
Part 7
2022-04-21 19:19:13 +00:00
alesapin
cbfe00bb47
Fix tidy
2022-04-21 00:30:13 +02:00
alesapin
c14e2e0b96
Fix more
2022-04-20 21:08:26 +02:00
alesapin
40c15222f8
Merge branch 'master' into fix_trash
2022-04-20 12:45:49 +02:00
tavplubix
2c54381520
Merge pull request #36200 from ClickHouse/fix_create_as_replicated_default_args
...
Fix CREATE AS for ReplicatedMergeTree with default arguments
2022-04-20 12:01:13 +03:00
alesapin
7cb7c120cc
Less ugly
2022-04-19 15:53:10 +02:00
alesapin
cc06bc3d99
Add some clarifications
2022-04-19 14:01:30 +02:00
alesapin
bd7b3847c1
Some code
2022-04-19 01:09:09 +02:00
alesapin
1706ae9e15
Some trash implementation
2022-04-15 18:36:23 +02:00
alesapin
5a8419a48e
Remove more trash
2022-04-15 17:05:17 +02:00
Alexander Tokmakov
40fea2b833
do not unfold {uuid} macro
2022-04-13 16:51:59 +02:00
Alexander Tokmakov
457a9e9691
fixes for ReplicatedMergeTree
2022-04-12 14:14:26 +02:00
Alexander Tokmakov
37a06eec1a
fixes
2022-04-05 17:36:53 +02:00
Alexander Tokmakov
7cba97aeab
Merge branch 'master' into resubmit_21474
2022-03-21 12:09:00 +01:00
alesapin
f2c5e2d3a0
Don't spam logs in zero copy replication
2022-03-19 17:31:33 +01:00
Anton Popov
0ba78c3c3a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-16 15:28:09 +00:00
Maksim Kita
2fdcf53a76
Fix clang-tidy warnings in Server, Storages folders
2022-03-14 18:17:35 +00:00
Anton Popov
fcdebea925
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-25 13:41:30 +03:00
alesapin
b75d551281
Fix clang tidy and add check for master
2022-02-14 14:37:41 +03:00
alesapin
b2886a429b
Fix lock during fetch
2022-02-14 12:20:27 +03:00
alesapin
3af06b23f8
POC
2022-02-10 22:45:52 +03:00
Anton Popov
18940b8637
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-09 23:38:38 +03:00
Alexander Tokmakov
897e94c16c
make restarting thread less bad
2022-02-03 23:29:24 +03:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
Amos Bird
ec7d367814
DiskLocal checker
...
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
2022-02-01 05:55:27 +08:00
Anton Popov
78b9f15abb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-30 03:24:37 +03:00
tavplubix
da9a38655b
Merge pull request #33847 from ClickHouse/fix_33806
...
Fix logical error on RESTORE REPLICA
2022-01-24 13:28:37 +03:00
tavplubix
766786b570
Update StorageReplicatedMergeTree.h
2022-01-24 12:16:13 +03:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
Alexander Tokmakov
8921725d71
fix logical error on RESTORE REPLICA
2022-01-21 14:32:19 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts
...
Fix flushing of in-memory parts
2022-01-21 02:18:03 +03:00
alesapin
4caef03e6a
Add ephemeral lock in zookeeper
2022-01-17 14:52:51 +03:00
alesapin
bf803472b1
Add a tests for a TTL bug in zero copy replication
2022-01-14 18:44:10 +03:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474
2022-01-07 10:12:25 +00:00
Alexander Tokmakov
187c43eba8
rename Committed state to Active
2021-12-30 23:45:38 +03:00
alesapin
91e1ac437e
Tiny improvements
2021-12-30 12:57:38 +03:00
alesapin
cbdba89d65
Merge branch 'master' into ianton-ru-MDB-15474
2021-12-30 11:21:19 +03:00
alesapin
8b331cd9bb
Remove method from IStorage
2021-12-29 22:41:47 +03:00
avogar
8112a71233
Implement schema inference for most input formats
2021-12-29 12:18:56 +03:00
Anton Ivashkin
0465aef15d
Fixes by code review responces
2021-12-27 19:27:06 +03:00
Anton Popov
a83832c03e
fix flushing of in-memory parts
2021-12-27 18:54:28 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474
2021-12-21 17:38:36 +03:00
Anton Ivashkin
33cbfc89ef
Move logic for replicated part to StorageReplicatedMergeTree class
2021-12-21 17:29:50 +03:00
Anton Ivashkin
0c0bf66334
Merge master
2021-12-21 17:27:54 +03:00
Anton Ivashkin
c724b074ae
Remove zero-copy version converter
2021-12-20 20:23:25 +03:00
Anton Ivashkin
f0b9a4327a
Use table UUID in zero-copy shared label in ZooKeeper
2021-12-20 16:19:00 +03:00
Anton Popov
99ebabd822
Merge remote-tracking branch 'upstream/master' into HEAD
2021-12-17 19:02:29 +03:00
Alexander Tokmakov
1b02136941
fix skipping of some mutations
2021-12-15 21:19:29 +03:00
tavplubix
4f46ac6b30
Remove LeaderElection ( #32140 )
...
* remove LeaderElection
* try fix tests
* Update test.py
* Update test.py
2021-12-07 19:55:55 +03:00
Anton Ivashkin
80ab73c691
Fix Zero-Copy replication lost locks, fix remove used remote data in DROP DETACHED PART
2021-12-01 16:11:26 +03:00
Anton Ivashkin
0f9038ebed
Zero-copy: move shared mark outside table node in ZooKeeper
2021-11-29 19:05:31 +03:00
Anton Popov
a20922b2d3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-11-09 15:36:25 +03:00
alesapin
2af950d4d5
Merge pull request #30658 from ClickHouse/std_mutex_alter_lock
...
Change `alter_lock` from `RWLock` to `std::timed_mutex`
2021-10-27 11:14:11 +03:00
mergify[bot]
ce88a84e88
Merge branch 'master' into nv/parts-uuid-move-shard-kill
2021-10-26 11:09:19 +00:00
Alexander Tokmakov
2e7e195e77
change alter_lock to std::timed_mutex
2021-10-26 13:37:00 +03:00
Nikolai Kochetov
a08c98d760
Move some files.
2021-10-16 17:03:50 +03:00
mergify[bot]
3b1f49aac7
Merge branch 'master' into nv/parts-uuid-move-shard-kill
2021-10-05 12:36:02 +00:00
Alexey Milovidov
fe6b7c77c7
Rename "common" to "base"
2021-10-02 10:13:14 +03:00
Nicolae Vartolomei
9a02061d9c
Rewrite part movement between shards logic and add kill support
...
See https://github.com/ClickHouse/ClickHouse/pull/24585#issuecomment-857735081
for extra context about the current implementation.
2021-09-17 18:11:32 +01:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask ( #25165 )
...
Introduced MergeTask and MutateTask
2021-09-17 00:19:58 +03:00
Anton Popov
4c388e3d84
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-09-09 14:10:16 +03:00
Nikita Mikhaylov
6062dd0021
Better
2021-09-08 00:21:21 +00:00
Nikita Mikhaylov
ea0fbf81af
Renaming
2021-09-06 12:01:16 +00:00
Nikita Mikhaylov
292a24abe8
Merge upstream/master into global-merge-executor (using imerge)
2021-09-03 00:34:24 +00:00
Nikita Mikhaylov
cc7c221fad
Own PriorityQueue + prettifying the code
2021-09-02 21:31:32 +00:00
mergify[bot]
5d299fbdee
Merge branch 'master' into remove_outdated_settings
2021-09-01 14:07:48 +00:00
Nikita Mikhaylov
dbc950caa4
added a test
2021-08-31 14:54:24 +00:00
alesapin
7223a32aac
Followup
2021-08-31 15:29:33 +03:00
alesapin
921e51e061
Remove some obsolete settings for replicated fetches
2021-08-31 15:22:56 +03:00
Nikita Mikhaylov
f8d4f04294
Merge upstream/master into global-merge-executor (using imerge)
2021-08-31 11:52:11 +00:00
Nikita Mikhaylov
c4416906c8
done
2021-08-30 19:37:03 +00:00
Alexey Milovidov
79e0433ba7
Merge branch 'master' of github.com:yandex/ClickHouse into async-reads
2021-08-28 01:19:16 +03:00
tavplubix
703101fe4d
Merge pull request #27931 from ClickHouse/wait_for_all_replicas_timeouts
...
Avoid too long waiting for inactive replicas
2021-08-27 14:31:36 +03:00
Alexey Milovidov
7c1d0a3baf
Progress on development
2021-08-25 01:24:47 +03:00
Alexander Tokmakov
cc9c2fd63b
make code better
2021-08-23 15:57:50 +03:00
Anton Popov
61239343e3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-08-20 16:33:30 +03:00
Alexander Tokmakov
59eb3aa9a9
avoid too long waiting for inactive replicas
2021-08-20 15:59:57 +03:00
mergify[bot]
f11e396151
Merge branch 'master' into nv/last-queue-update-exception
2021-08-18 07:00:50 +00:00
Nicolae Vartolomei
3f291b024a
Use plain mutex instead of MultiVersion
2021-08-09 13:58:23 +01:00
Maksim Kita
3f48c85722
StorageSystemReplicas added column replica_is_active
2021-08-04 16:19:42 +03:00
Anton Popov
e36736b50c
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-08-02 22:52:02 +03:00
Nicolae Vartolomei
8b07a7f180
Store exception generated when we tried to update the queue last time
...
The use case is to alert when queue contains broken entries. Especially
important when ClickHouse breaks backwards compatibility between
versions and log entries written by newer versions aren't parseable by
old versions.
```
Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected 'quorum: ' before: 'merge_type: 2\n'
```
2021-07-27 15:42:40 +01:00
Nikolai Kochetov
9c92f43359
Update storages.
2021-07-23 22:33:59 +03:00
Anton Popov
f99374cca6
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-07-20 18:20:21 +03:00
Zhichang Yu
5047c758f4
fix per review
2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1
hdfs zero copy
2021-07-13 07:19:12 +00:00
Anton Popov
3ed7f5a6cc
dynamic subcolumns: add snapshot for storage
2021-07-09 06:15:41 +03:00
alesapin
0d8844c510
Merge pull request #25884 from ClickHouse/fix_drop_part_in_queue
...
Relax `DROP PART` guarantees and turn on checks in ReplicationQueue.
2021-07-07 10:48:48 +03:00
Anton Popov
9071ecd428
fix alter of settings in MergeTree
2021-07-05 15:44:58 +03:00
alesapin
89a401a84f
Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue
2021-06-30 18:53:17 +03:00
alesapin
6a73c8b49e
Review fixes
2021-06-30 18:24:51 +03:00
alesapin
21f15be990
Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue
2021-06-30 15:30:32 +03:00
alesapin
2eb27540b2
Some test version
2021-06-30 15:29:09 +03:00
alesapin
5822f0ba29
Replace lost parts with empty parts instead of hacking replication queue
2021-06-29 18:14:44 +03:00
Raúl Marín
bfc122df64
Fix some typos in Storage classes
2021-06-28 19:03:56 +02:00
Nikita Mikhaylov
c66a3b22b5
done
2021-06-22 23:24:47 +00:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] ( #13652 )
...
* initial commit: add setting and stub
* typo
* added test stub
* fix
* wip merging new integration test and code proto
* adding steps interpreters
* adding firstly proposed solution (moving parts etc)
* added checking zookeeper path existence
* fixing the include
* fixing and sorting includes
* fixing outdated struct
* fix the name
* added ast ptr as level of indirection
* fix ref
* updating the changes
* working on test stub
* fix iterator -> reference
* revert rocksdb submodule update
* fixed show privileges test
* updated the test stub
* replaced rand() with thread_local_rng(), updated the tests
updated the test
fixed test config path
test fix
removed error messages
fixed the test
updated the test
fixed string literal
fixed literal
typo: =
* fixed the empty replica error message
* updated the test and the code with logs
* updated the possible test cases, updated
* added the code/test milestone comments
* updated the test (added more testcases)
* replaced native assert with CH one
* individual replicas recursive delete fix
* updated the AS db.name AST
* two small logging fixes
* manually generated AST fixes
* Updated the test, added the possible algo change
* Some thoughts about optimizing the solution:
ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH
* fix
* Removed the replica sync in test as it's invalid
* Some test tweaks
* tmp
* Rewrote the algo by using the executeQuery instead of
hand-crafting the ASTPtr.
Two questions still active.
* tr: logging active parts
* Extracted the parts moving algo into a separate helper function
* Fixed the test data and the queries slightly
* Replaced query to system.parts to direct invocation,
started building the test that breaks on various parts.
* Added the case for tables when at least one replica is alive
* Updated the test to test replicas restoration by detaching/attaching
* Altered the test to check restoration without replica restart
* Added the tables swap in the start if the server failed last time
* Hotfix when only /replicas/replica... path was deleted
* Restore ZK paths while creating a replicated MergeTree table
* Updated the docs, fixed the algo for individual replicas restoration case
* Initial parts table storage fix, tests sync fix
* Reverted individual replica restoration to general algo
* Slightly optimised getDataParts
* Trying another solution with parts detaching
* Rewrote algo without any steps, added ON CLUSTER support
* Attaching parts from other replica on restoration
* Getting part checksums from ZK
* Removed ON CLUSTER, finished working solution
* Multiple small changes after review
* Fixing parallel test
* Supporting rewritten form on cluster
* Test fix
* Moar logging
* Using source replica as checksum provider
* improve test, remove some code from parser
* Trying solution with move to detached + forget
* Moving all parts (not only Committed) to detached
* Edited docs for RESTORE REPLICA
* Re-merging
* minor fixes
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
2021-06-20 11:24:43 +03:00
Maksim Kita
67e9b85951
Merge ext into common
2021-06-16 23:28:41 +03:00
Alexander Tokmakov
3ade38df82
remove copypaste
2021-06-08 22:17:45 +03:00
alesapin
4a1bf51ad1
Merge branch 'master' into limited_fetches
2021-06-02 23:01:47 +03:00
alexey-milovidov
c7bd3b8674
Merge pull request #23315 from CurtizJ/empty-parts-fix
...
Fix waiting while dropping empty parts
2021-06-01 13:52:49 +03:00
Nikolai Kochetov
afc1fe7f3d
Make ContextPtr const by default.
2021-05-31 17:49:02 +03:00
Anton Popov
179f172c4c
fix build
2021-05-28 02:10:44 +03:00
Anton Popov
336abfa475
Merge remote-tracking branch 'upstream/master' into HEAD
2021-05-28 01:58:30 +03:00
alesapin
2985def91d
Add tests
2021-05-27 15:54:47 +03:00
alesapin
42b2383e01
Basic throttler implementation
2021-05-26 23:37:44 +03:00
Anton Popov
7d1431f6b6
better code near dropPartition
2021-05-25 20:25:00 +03:00
alesapin
20a34e951a
Split fetch part into two stages, don't use replicated block output stream
2021-05-21 12:30:49 +03:00
Anton Popov
e27acc26be
better dropPart and dropPartition
2021-05-17 17:26:36 +03:00
Anton Popov
e043cc45bf
Merge remote-tracking branch 'upstream/master' into HEAD
2021-05-17 16:11:12 +03:00
alesapin
17f229857c
Merge branch 'master' into nvartolomei-parts-move
2021-05-17 13:52:48 +03:00
Alexander Tokmakov
970ad94851
Merge branch 'master' into fix_intersecting_parts
2021-05-14 13:17:02 +03:00
Alexander Tokmakov
2da12f0b59
fix intersection of drop range and merge
2021-05-13 17:04:36 +03:00
Alexander Tokmakov
e114c7eb8b
fix virtual parts in REPLACE_RANGE
2021-05-13 14:29:59 +03:00
alesapin
46e136b5c4
Merge branch 'master' into nv/parts-uuid-move-shard
2021-05-11 15:36:40 +03:00
Alexander Tokmakov
11124f7015
fix
2021-05-07 20:09:39 +03:00
Nicolae Vartolomei
1fa5871ff7
Fix bad rebase and introduce part_moves_between_shards_enable setting
2021-04-27 14:20:13 +01:00
Nicolae Vartolomei
53d57ffb52
Part movement between shards
...
Integrate query deduplication from #17348
2021-04-27 14:20:12 +01:00
mergify[bot]
b9e229d19d
Merge branch 'master' into empty-parts-fix
2021-04-21 13:19:54 +00:00
Anton Popov
42a0416a2b
better drop of empty parts
2021-04-20 05:31:08 +03:00
Alexander Tokmakov
ecc625692e
fix build
2021-04-19 13:40:20 +03:00
Alexander Tokmakov
9bc98771f7
fix race on replica creation
2021-04-19 11:21:42 +03:00
songenjie
f731739fab
fix suggests
2021-04-14 10:05:41 +08:00
songenjie
7c32cc1e18
fix case style for local variable
2021-04-13 17:34:04 +08:00
songenjie
564136ec46
[clickhouse][server][dll][alter]support fetch part
2021-04-13 12:40:33 +08:00
Azat Khuzhin
dfe3699e77
Respect max_part_removal_threads for ReplicatedMergeTree
2021-04-11 11:33:45 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Mike Kot
c947280dfc
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-04-01 21:38:51 +03:00
alexey-milovidov
1512ff832e
Merge branch 'master' into nv/replicated-fetches-timeouts
2021-03-29 19:25:13 +03:00
Mike Kot
406d037ebb
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-03-15 18:41:47 +03:00
Anton Ivashkin
aff13c0c52
Make method StorageReplicatedMergeTree::fetchExistsPart
2021-03-09 20:49:50 +03:00
alesapin
5b3161e0b5
Get rid of const_cast
2021-03-05 20:24:06 +03:00
Anton Ivashkin
d08b481660
Fixes by review responces
2021-03-05 19:20:38 +03:00
Nicolae Vartolomei
1d83c596f8
RFC: Throw exception if removing parts from ZooKeeper fails.
...
This is used for removing part metadata from ZooKeeper when executing
queue events like `DROP_RANGE` triggered when a user tries to drop a
part or a partition. There are other uses but I'll focus only on this
one.
Before this change the method was giving up silently if it was unable to
remove parts from ZooKeeper and this behaviour seems to be problematic.
It could lead to operation being reported as successful at first but
data reappearing later (very rarely) or "stuck" events in replication
queue.
Here is one particular scenario which I think we've hit:
* Execute a DETACH PARTITION
* DROP_RANGE event put in the queue
* Replicas try to execute dropRange but some of them get disconnected
from ZK and 5 retries aren't enough (ZK is miss-behaving), return code
(false) is ignored and log pointer advances.
* One of the replica where dropRange failed is restarted.
* checkParts is executed and it finds parts that weren't removed from
ZK, logs `Removing locally missing part from ZooKeeper and queueing a
fetch` and puts GET_PART on the queue.
* Few things can happen from here:
* There is a lagging replica that din't execute DROP_RANGE yet: part will be
fetched. The other replica will execute DROP_RANGE later and we'll
get diverging set of parts on replicas.
* Another replica also silently failed to remove parts from ZK: both
of them are left with GET_PART in the queue and none of them can
make progress, logging: `No active replica has part ... or covering
part`.
2021-03-05 09:50:26 +00:00
Mike Kot
4e1ac185b5
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-03-01 20:58:07 +03:00
Anton Ivashkin
3c11d44494
Add description for getUniqueId method, fix typos
2021-03-01 13:31:36 +03:00
Anton Ivashkin
5b267b7eec
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication
2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557
Fixes by review response
2021-02-26 12:48:57 +03:00
Mike Kot
2b3b335eda
Exit fixes
2021-02-25 21:41:09 +03:00
Mike Kot
b2c898f58c
Adding the part when found
2021-02-25 21:25:55 +03:00
Mike Kot
ca83775711
Multiple small hotfixes
...
Small fixes
Some fix for old bug
Another old code fix
2021-02-16 16:39:18 +03:00
Mike Kot
73f4740be5
Trying to re-write the solution by adding new command type
...
ATTACH_PART into the replicated log.
The LogEntry now also has the pre-calculated part checksum for this
entry type, which is later used while searching in the detached/ folder
2021-02-15 18:06:48 +03:00
