mateng0915
eae0f9957c
Feature: Optimize the replica delay api logic
...
Description:
============
Currently if in the Whole CK instance has one table is read only status
then the API /replicas_status will throw error, xxx table is read only
For make this monitor can work in prod env, we can catch the read only
status instead of directly throw error
Solution:
=========
Return other normal table's delay value even if the CK instance has readonly Replicatedxxx table
Please enter the commit message for your changes. Lines starting
2023-01-13 14:44:27 +08:00
Raúl Marín
9f121d5d9c
Fix background_fetches_pool_size config reload ( #45189 )
2023-01-12 23:41:25 +01:00
Han Fei
bef6168c05
support cache for async inserts block ids
2023-01-10 13:19:12 +01:00
alesapin
616e37ba43
Merge pull request #44847 from ClickHouse/fix_rare_race
...
Fix rare race which can lead to queue hang
2023-01-10 12:09:10 +01:00
Alexander Gololobov
742887ce53
Merge pull request #44718 from ClickHouse/lwd_mutation_always_sync
...
Make lightweight deletes always synchronous
2023-01-09 15:58:02 +01:00
Nikita Mikhaylov
857799fbca
Parallel distributed insert select with s3Cluster [3] ( #44955 )
...
* Revert "Revert "Resurrect parallel distributed insert select with s3Cluster (#41535 )""
This reverts commit b8d9066004
2023-01-09 13:30:32 +01:00
alesapin
da83e97b4c
Fix startup issue
2023-01-06 15:36:53 +01:00
alesapin
2a5b4067e4
Review fixes
2023-01-04 15:15:14 +01:00
Alexander Gololobov
2c64f0042a
Added force_wait parameter to IStorage::mutate()
2022-12-30 16:23:03 +01:00
alesapin
5dfefb43a3
Fix deadlock in attach thread
2022-12-29 18:52:20 +01:00
alesapin
fa467b3398
Minor code polishing
2022-12-22 14:31:42 +01:00
Han Fei
3c11aca040
address comments
2022-12-07 23:40:52 +01:00
Han Fei
28564df6f1
Merge branch 'master' into hanfei/aysnc-insert
2022-11-24 09:12:43 +01:00
Han Fei
32c7fbec55
use another dedup path for async inserts
2022-11-22 15:12:00 +01:00
Han Fei
b80a2d6c89
address comments
2022-11-18 17:22:05 +01:00
Han Fei
14701670b5
Merge branch 'master' into hanfei/aysnc-insert
2022-11-16 17:24:36 +01:00
Alexander Tokmakov
c64109a2de
Merge branch 'master' into fetch-remove-covered
2022-11-14 18:23:16 +03:00
Han Fei
11f93dec3f
tmp commit
2022-11-14 15:53:45 +01:00
Igor Nikonov
f8c907c463
Revert "Revert " Keeper retries during insert (clean)""
2022-11-10 13:14:04 +01:00
Alexander Tokmakov
fb8c527a2f
Revert " Keeper retries during insert (clean)"
2022-11-10 14:12:24 +03:00
Igor Nikonov
379113f8f1
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-11-09 13:03:32 +00:00
Azat Khuzhin
2bd4a71150
Merge branch 'master' into fetch-remove-covered
2022-11-08 00:55:00 +01:00
Alexander Tokmakov
e67d05604c
Merge branch 'master' into fix_intersecting_parts2
2022-11-02 19:30:27 +01:00
Alexander Tokmakov
00c9e50ee3
fix race between INSERT and ALTER PARTITION
2022-11-02 16:25:26 +01:00
Igor Nikonov
9cc64a0fcc
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-10-31 18:21:21 +00:00
Alexander Tokmakov
c68ab231f9
fix accessing part in Deleting state
2022-10-30 17:30:51 +01:00
Igor Nikonov
3682913172
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-10-27 22:01:53 +00:00
Azat Khuzhin
8db31beee6
Remove covered parts for fetched part
...
Here is an example that I found on production, simplified.
Consider the following queue (nothing of this had been processed on this
replica):
- GET_PART all_0_0_0 (queue-0000000001)
- GET_PART all_1_1_0 (queue-0000000002)
...
- GET_PART all_0_1_1 (queue-0000000003)
- GET_PART all_2_2_0 (queue-0000000004)
...
- MERGE_PARTS from [all_0_1_1, all_2_2_0] to all_0_2_2 (queue-0000000005)
And now queue-0000000005 started to executing (either because
of reording, or because at that time GET_PART fails), and it
does not have any required parts, so it will fetch them, but
not all_0_0_0 and all_1_1_0, so this replica delay will set to
the time of min(queue-0000000001, queue-0000000002), while it
is not true, since it already have parts that covers those
parts.
and since MERGE_PARTS takes 30min, it increased the replica delay
eventually to 30min, for the time range of 30min, which is pretty huge.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-26 11:10:50 +03:00
Anton Popov
c8199bc125
Merge branch 'master' into refactor-data-part
2022-10-25 14:31:05 +02:00
Igor Nikonov
54013009d3
Code migration from #41734 to align diff and git blame.
...
Initial implementation was different and it changed the entire
ReplicatedMergeTreeSink::commitPart() which change history provided by git blame.
Then RetriesControl.retryLoop() was introduced later which significantly reduces
the diff since it's like while() used before.
So, check outing the current version will keep more original history in
git blame, which is useful here
2022-10-24 08:20:11 +00:00
Anton Popov
b40d9200d2
better semantic of constsness of DataPartStorage
2022-10-23 15:24:20 +00:00
Igor Nikonov
c0d1be9cab
Merge pull request #42560 from ClickHouse/igor/replicated_merge_tree_some_polishing
...
Some polishing: replicated merge tree
2022-10-23 15:15:24 +02:00
Anton Popov
56e5daba0c
remove DataPartStorageBuilder
2022-10-23 00:23:15 +00:00
Igor Nikonov
851f556d65
Remove unused parameter
2022-10-21 14:35:37 +02:00
Igor Nikonov
13d31eefbc
Replicated merge tree polishing
...
+ make read-only members const
+ reduce memory allocations
2022-10-21 14:02:52 +02:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Alexander Tokmakov
b8d9066004
Revert "Resurrect parallel distributed insert select with s3Cluster ( #41535 )"
...
This reverts commit 860e34e760
2022-10-07 15:53:30 +02:00
alesapin
3373480ede
Merge pull request #42036 from ClickHouse/fix_one_more_no_such_key
...
Fix No such key during table drop
2022-10-06 14:50:02 +02:00
Nikita Mikhaylov
860e34e760
Resurrect parallel distributed insert select with s3Cluster ( #41535 )
2022-10-06 13:47:32 +02:00
alesapin
210882b9c4
Better fix
2022-10-03 23:30:50 +02:00
alesapin
3114ddcad2
Fix failure during table drop
2022-10-03 19:06:12 +02:00
Nikita Mikhaylov
dbcba1490c
Try to make MergeTreeRestartingThread more reactive in case of expired sessions ( #41092 )
2022-09-23 14:40:04 +02:00
alesapin
acde29e314
Nasty code
2022-09-20 17:55:06 +02:00
alesapin
cfe398cbbc
Fix lost table shared id during server restart
2022-09-20 14:51:16 +02:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze
...
Fix SYSTEM UNFREEZE for ordinary database
2022-09-13 14:55:11 +03:00
Alexander Tokmakov
da4fbb552d
Merge branch 'master' into fix_parts_removal
2022-09-12 17:49:29 +03:00
alesapin
21ab379026
Fix comment
2022-09-11 13:47:04 +02:00
alesapin
ac53830000
Fix hung check
2022-09-11 13:37:39 +02:00
alesapin
e3fc78aa5f
Fix bad warnings in part fetches
2022-09-09 12:14:42 +02:00
ianton-ru
39e1fc7a0f
Merge branch 'master' into fix-ordinary-system-unfreeze
2022-09-05 17:10:59 +03:00
Luck-Chang
1ac8e739c9
fix some typos and clang-tidy warnings
2022-09-05 09:50:24 +08:00
Alexander Tokmakov
eae2667a1c
Merge pull request #40148 from ClickHouse/replicate-tables-readonly
...
`ReplicatedMergeTree` tables as readonly until attach is finished
2022-08-25 19:39:19 +03:00
Antonio Andelic
797e21761a
Address PR comments
2022-08-24 18:32:03 +00:00
Antonio Andelic
da0d6056f4
Merge branch 'master' into replicate-tables-readonly
2022-08-24 17:29:33 +00:00
alesapin
eeb0a3584e
Merge branch 'master' into data_source_description
2022-08-24 15:26:18 +02:00
Alexander Tokmakov
f9f85a0e8b
Revert "Parallel distributed insert select from *Cluster table functions ( #39107 )"
...
This reverts commit d3cc234986
2022-08-24 15:17:15 +03:00
alesapin
d8664c3227
Add shortcut for backups
2022-08-19 16:58:30 +02:00
Antonio Andelic
ee3df77fda
Simplify logic
2022-08-19 11:12:20 +00:00
Antonio Andelic
5e2c8869a9
Merge branch 'master' into replicate-tables-readonly
2022-08-19 09:41:11 +00:00
Antonio Andelic
a8f3b62bf6
Use global zookeeper
2022-08-19 08:49:51 +00:00
Alexander Tokmakov
832d7c6e7c
Merge branch 'master' into replicated_database_improvements
2022-08-16 15:49:13 +02:00
Antonio Andelic
2ed42bca7e
Merge branch 'master' into replicate-tables-readonly
2022-08-16 08:17:53 +00:00
Nikita Mikhaylov
d3cc234986
Parallel distributed insert select from *Cluster table functions ( #39107 )
2022-08-15 12:41:17 +02:00
Antonio Andelic
c6b3a897d2
Test replicated table without ZK, small fixes
2022-08-12 11:56:46 +00:00
Antonio Andelic
1c0f85ccdb
Attach table in background
2022-08-12 09:32:13 +00:00
Alexander Tokmakov
6f88065cc1
check metadata consistency
2022-07-29 18:33:16 +02:00
jianmei zhang
c1834d183b
ReplicatedMergeTree support lightweight delete
2022-07-26 11:50:09 +08:00
Vadim Volodin
85bf022258
review zookeeper and partition matching
2022-07-21 16:57:28 +03:00
Vitaly Baranov
7689e0c36f
Improve gathering metadata for backup - part 6.
2022-06-30 08:37:17 +02:00
Vitaly Baranov
461a31f237
Improve gathering metadata for backup - part 2.
2022-06-30 08:37:17 +02:00
Vitaly Baranov
64b51a3772
Improve gathering metadata for backup.
2022-06-30 08:37:17 +02:00
Vadim Volodin
3296ba2532
Fix SYSTEM UNFREEZE for ordinary database
2022-06-24 20:46:16 +03:00
Nikolai Kochetov
7452d04e3a
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-20 15:31:02 +02:00
Vitaly Baranov
c2c35fad82
Refactoring of the code getting create table queries for backup.
2022-06-15 20:32:35 +02:00
Vitaly Baranov
592f568f83
Move backup/restore code to storages and databases - part 2.
2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57
Move backup/restore code to storages and databases - part 1.
2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21
Rework collecting replicated parts.
2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982
Restore parts of MergeTree in correct order.
2022-06-15 20:28:40 +02:00
Nikolai Kochetov
2a9a63ac7b
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-15 15:35:26 +02:00
Vadim Volodin
637d293fbd
Add SYSTEM UNFREEZE query
2022-06-08 15:21:14 +03:00
Nikolai Kochetov
678d978acf
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-07 15:23:00 +00:00
Nikolai Kochetov
89c5855d20
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-02 12:19:07 +02:00
Nikolai Kochetov
c71256ea38
Remove some commented code.
2022-05-30 13:18:20 +00:00
Nikolai Kochetov
1b85f2c1d6
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-25 16:27:40 +02:00
Nikolai Kochetov
fd97a9d885
Move some resources
2022-05-23 19:47:32 +00:00
Alexander Tokmakov
db66efb1a7
fix logical error on truncate table
2022-05-06 16:12:31 +02:00
Nikolai Kochetov
35095191eb
Merge branch 'master' into refactor-something-in-part-volumes
2022-05-03 17:51:47 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable
...
... IStorage has deleted copy ctor / assignment already
2022-05-03 09:07:08 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying
...
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
previously allowed.
Hence, this change
- removes shared_ptr_helper and as a result all inherited create() methods,
- instead, Storage objects are now created using make_shared<>() by the
caller (for that to work, many constructors had to be made public), and
- all Storage classes were marked as noncopyable using boost::noncopyable.
In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
2022-05-02 08:46:52 +02:00
Nikolai Kochetov
e44af67fee
Merge branch 'master' into refactor-something-in-part-volumes
2022-04-26 21:08:00 +02:00
Vitaly Baranov
68a020ecea
Implement BACKUP/RESTORE ON CLUSTER.
2022-04-25 16:34:33 +02:00
Vitaly Baranov
acd28d8a1d
Implement RESTORE for replicated tables.
2022-04-25 16:34:33 +02:00
alesapin
ebdd53ef8f
Fix shared fetch
2022-04-22 19:18:18 +02:00
Nikolai Kochetov
8c00692844
Part 8
2022-04-22 16:58:09 +00:00
Nikolai Kochetov
9133e398b8
Part 7
2022-04-21 19:19:13 +00:00
alesapin
cbfe00bb47
Fix tidy
2022-04-21 00:30:13 +02:00
alesapin
c14e2e0b96
Fix more
2022-04-20 21:08:26 +02:00
alesapin
40c15222f8
Merge branch 'master' into fix_trash
2022-04-20 12:45:49 +02:00
tavplubix
2c54381520
Merge pull request #36200 from ClickHouse/fix_create_as_replicated_default_args
...
Fix CREATE AS for ReplicatedMergeTree with default arguments
2022-04-20 12:01:13 +03:00
alesapin
7cb7c120cc
Less ugly
2022-04-19 15:53:10 +02:00
alesapin
cc06bc3d99
Add some clarifications
2022-04-19 14:01:30 +02:00
alesapin
bd7b3847c1
Some code
2022-04-19 01:09:09 +02:00
alesapin
1706ae9e15
Some trash implementation
2022-04-15 18:36:23 +02:00
alesapin
5a8419a48e
Remove more trash
2022-04-15 17:05:17 +02:00
Alexander Tokmakov
40fea2b833
do not unfold {uuid} macro
2022-04-13 16:51:59 +02:00
Alexander Tokmakov
457a9e9691
fixes for ReplicatedMergeTree
2022-04-12 14:14:26 +02:00
Alexander Tokmakov
37a06eec1a
fixes
2022-04-05 17:36:53 +02:00
Alexander Tokmakov
7cba97aeab
Merge branch 'master' into resubmit_21474
2022-03-21 12:09:00 +01:00
alesapin
f2c5e2d3a0
Don't spam logs in zero copy replication
2022-03-19 17:31:33 +01:00
Anton Popov
0ba78c3c3a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-16 15:28:09 +00:00
Maksim Kita
2fdcf53a76
Fix clang-tidy warnings in Server, Storages folders
2022-03-14 18:17:35 +00:00
Anton Popov
fcdebea925
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-25 13:41:30 +03:00
alesapin
b75d551281
Fix clang tidy and add check for master
2022-02-14 14:37:41 +03:00
alesapin
b2886a429b
Fix lock during fetch
2022-02-14 12:20:27 +03:00
alesapin
3af06b23f8
POC
2022-02-10 22:45:52 +03:00
Anton Popov
18940b8637
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-09 23:38:38 +03:00
Alexander Tokmakov
897e94c16c
make restarting thread less bad
2022-02-03 23:29:24 +03:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
Amos Bird
ec7d367814
DiskLocal checker
...
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
2022-02-01 05:55:27 +08:00
Anton Popov
78b9f15abb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-30 03:24:37 +03:00
tavplubix
da9a38655b
Merge pull request #33847 from ClickHouse/fix_33806
...
Fix logical error on RESTORE REPLICA
2022-01-24 13:28:37 +03:00
tavplubix
766786b570
Update StorageReplicatedMergeTree.h
2022-01-24 12:16:13 +03:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
Alexander Tokmakov
8921725d71
fix logical error on RESTORE REPLICA
2022-01-21 14:32:19 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts
...
Fix flushing of in-memory parts
2022-01-21 02:18:03 +03:00
alesapin
4caef03e6a
Add ephemeral lock in zookeeper
2022-01-17 14:52:51 +03:00
alesapin
bf803472b1
Add a tests for a TTL bug in zero copy replication
2022-01-14 18:44:10 +03:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474
2022-01-07 10:12:25 +00:00
Alexander Tokmakov
187c43eba8
rename Committed state to Active
2021-12-30 23:45:38 +03:00
alesapin
91e1ac437e
Tiny improvements
2021-12-30 12:57:38 +03:00
alesapin
cbdba89d65
Merge branch 'master' into ianton-ru-MDB-15474
2021-12-30 11:21:19 +03:00
alesapin
8b331cd9bb
Remove method from IStorage
2021-12-29 22:41:47 +03:00
avogar
8112a71233
Implement schema inference for most input formats
2021-12-29 12:18:56 +03:00
Anton Ivashkin
0465aef15d
Fixes by code review responces
2021-12-27 19:27:06 +03:00
Anton Popov
a83832c03e
fix flushing of in-memory parts
2021-12-27 18:54:28 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474
2021-12-21 17:38:36 +03:00
Anton Ivashkin
33cbfc89ef
Move logic for replicated part to StorageReplicatedMergeTree class
2021-12-21 17:29:50 +03:00
Anton Ivashkin
0c0bf66334
Merge master
2021-12-21 17:27:54 +03:00
Anton Ivashkin
c724b074ae
Remove zero-copy version converter
2021-12-20 20:23:25 +03:00
Anton Ivashkin
f0b9a4327a
Use table UUID in zero-copy shared label in ZooKeeper
2021-12-20 16:19:00 +03:00
Anton Popov
99ebabd822
Merge remote-tracking branch 'upstream/master' into HEAD
2021-12-17 19:02:29 +03:00
Alexander Tokmakov
1b02136941
fix skipping of some mutations
2021-12-15 21:19:29 +03:00
tavplubix
4f46ac6b30
Remove LeaderElection ( #32140 )
...
* remove LeaderElection
* try fix tests
* Update test.py
* Update test.py
2021-12-07 19:55:55 +03:00
Anton Ivashkin
80ab73c691
Fix Zero-Copy replication lost locks, fix remove used remote data in DROP DETACHED PART
2021-12-01 16:11:26 +03:00
Anton Ivashkin
0f9038ebed
Zero-copy: move shared mark outside table node in ZooKeeper
2021-11-29 19:05:31 +03:00
Anton Popov
a20922b2d3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-11-09 15:36:25 +03:00
alesapin
2af950d4d5
Merge pull request #30658 from ClickHouse/std_mutex_alter_lock
...
Change `alter_lock` from `RWLock` to `std::timed_mutex`
2021-10-27 11:14:11 +03:00
mergify[bot]
ce88a84e88
Merge branch 'master' into nv/parts-uuid-move-shard-kill
2021-10-26 11:09:19 +00:00
Alexander Tokmakov
2e7e195e77
change alter_lock to std::timed_mutex
2021-10-26 13:37:00 +03:00
Nikolai Kochetov
a08c98d760
Move some files.
2021-10-16 17:03:50 +03:00
