thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-07 06:52:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
109,548 Commits 9,200 Branches 1,782 Tags 2 GiB
3de905bb7c
Commit Graph

396 Commits

Author SHA1 Message Date
Igor Nikonov
3682913172 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-27 22:01:53 +00:00
Azat Khuzhin
8db31beee6 Remove covered parts for fetched part 
Here is an example that I found on production, simplified.

Consider the following queue (nothing of this had been processed on this
replica):

- GET_PART all_0_0_0 (queue-0000000001)
- GET_PART all_1_1_0 (queue-0000000002)
...
- GET_PART all_0_1_1 (queue-0000000003)
- GET_PART all_2_2_0 (queue-0000000004)
...
- MERGE_PARTS from [all_0_1_1, all_2_2_0] to all_0_2_2 (queue-0000000005)

And now queue-0000000005 started to executing (either because
of reording, or because at that time GET_PART fails), and it
does not have any required parts, so it will fetch them, but
not all_0_0_0 and all_1_1_0, so this replica delay will set to
the time of min(queue-0000000001, queue-0000000002), while it
is not true, since it already have parts that covers those
parts.

and since MERGE_PARTS takes 30min, it increased the replica delay
eventually to 30min, for the time range of 30min, which is pretty huge.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-26 11:10:50 +03:00
Anton Popov
c8199bc125
Merge branch 'master' into refactor-data-part 2022-10-25 14:31:05 +02:00
Igor Nikonov
54013009d3 Code migration from #41734 to align diff and git blame. 
Initial implementation was different and it changed the entire
ReplicatedMergeTreeSink::commitPart() which change history provided by git blame.
Then RetriesControl.retryLoop() was introduced later which significantly reduces
the diff since it's like while() used before.
So, check outing the current version will keep more original history in
git blame, which is useful here
 2022-10-24 08:20:11 +00:00
Anton Popov
b40d9200d2 better semantic of constsness of DataPartStorage 2022-10-23 15:24:20 +00:00
Igor Nikonov
c0d1be9cab
Merge pull request #42560 from ClickHouse/igor/replicated_merge_tree_some_polishing 
Some polishing: replicated merge tree
 2022-10-23 15:15:24 +02:00
Anton Popov
56e5daba0c remove DataPartStorageBuilder 2022-10-23 00:23:15 +00:00
Igor Nikonov
851f556d65 Remove unused parameter 2022-10-21 14:35:37 +02:00
Igor Nikonov
13d31eefbc Replicated merge tree polishing 
+ make read-only members const
+ reduce memory allocations
 2022-10-21 14:02:52 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Alexander Tokmakov
b8d9066004 Revert "Resurrect parallel distributed insert select with s3Cluster (#41535)" 
This reverts commit 860e34e760.
 2022-10-07 15:53:30 +02:00
alesapin
3373480ede
Merge pull request #42036 from ClickHouse/fix_one_more_no_such_key 
Fix No such key during table drop
 2022-10-06 14:50:02 +02:00
Nikita Mikhaylov
860e34e760
Resurrect parallel distributed insert select with s3Cluster (#41535) 2022-10-06 13:47:32 +02:00
alesapin
210882b9c4 Better fix 2022-10-03 23:30:50 +02:00
alesapin
3114ddcad2 Fix failure during table drop 2022-10-03 19:06:12 +02:00
Nikita Mikhaylov
dbcba1490c
Try to make MergeTreeRestartingThread more reactive in case of expired sessions (#41092) 2022-09-23 14:40:04 +02:00
alesapin
acde29e314 Nasty code 2022-09-20 17:55:06 +02:00
alesapin
cfe398cbbc Fix lost table shared id during server restart 2022-09-20 14:51:16 +02:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze 
Fix SYSTEM UNFREEZE for ordinary database
 2022-09-13 14:55:11 +03:00
Alexander Tokmakov
da4fbb552d
Merge branch 'master' into fix_parts_removal 2022-09-12 17:49:29 +03:00
alesapin
21ab379026 Fix comment 2022-09-11 13:47:04 +02:00
alesapin
ac53830000 Fix hung check 2022-09-11 13:37:39 +02:00
alesapin
e3fc78aa5f Fix bad warnings in part fetches 2022-09-09 12:14:42 +02:00
ianton-ru
39e1fc7a0f
Merge branch 'master' into fix-ordinary-system-unfreeze 2022-09-05 17:10:59 +03:00
Luck-Chang
1ac8e739c9 fix some typos and clang-tidy warnings 2022-09-05 09:50:24 +08:00
Alexander Tokmakov
eae2667a1c
Merge pull request #40148 from ClickHouse/replicate-tables-readonly 
`ReplicatedMergeTree` tables as readonly until attach is finished
 2022-08-25 19:39:19 +03:00
Antonio Andelic
797e21761a Address PR comments 2022-08-24 18:32:03 +00:00
Antonio Andelic
da0d6056f4 Merge branch 'master' into replicate-tables-readonly 2022-08-24 17:29:33 +00:00
alesapin
eeb0a3584e Merge branch 'master' into data_source_description 2022-08-24 15:26:18 +02:00
Alexander Tokmakov
f9f85a0e8b Revert "Parallel distributed insert select from *Cluster table functions (#39107)" 
This reverts commit d3cc234986.
 2022-08-24 15:17:15 +03:00
alesapin
d8664c3227 Add shortcut for backups 2022-08-19 16:58:30 +02:00
Antonio Andelic
ee3df77fda Simplify logic 2022-08-19 11:12:20 +00:00
Antonio Andelic
5e2c8869a9 Merge branch 'master' into replicate-tables-readonly 2022-08-19 09:41:11 +00:00
Antonio Andelic
a8f3b62bf6 Use global zookeeper 2022-08-19 08:49:51 +00:00
Alexander Tokmakov
832d7c6e7c Merge branch 'master' into replicated_database_improvements 2022-08-16 15:49:13 +02:00
Antonio Andelic
2ed42bca7e Merge branch 'master' into replicate-tables-readonly 2022-08-16 08:17:53 +00:00
Nikita Mikhaylov
d3cc234986
Parallel distributed insert select from *Cluster table functions (#39107) 2022-08-15 12:41:17 +02:00
Antonio Andelic
c6b3a897d2 Test replicated table without ZK, small fixes 2022-08-12 11:56:46 +00:00
Antonio Andelic
1c0f85ccdb Attach table in background 2022-08-12 09:32:13 +00:00
Alexander Tokmakov
6f88065cc1 check metadata consistency 2022-07-29 18:33:16 +02:00
jianmei zhang
c1834d183b ReplicatedMergeTree support lightweight delete 2022-07-26 11:50:09 +08:00
Vadim Volodin
85bf022258 review zookeeper and partition matching 2022-07-21 16:57:28 +03:00
Vitaly Baranov
7689e0c36f Improve gathering metadata for backup - part 6. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
461a31f237 Improve gathering metadata for backup - part 2. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
64b51a3772 Improve gathering metadata for backup. 2022-06-30 08:37:17 +02:00
Vadim Volodin
3296ba2532 Fix SYSTEM UNFREEZE for ordinary database 2022-06-24 20:46:16 +03:00
Nikolai Kochetov
7452d04e3a Merge branch 'master' into refactor-something-in-part-volumes 2022-06-20 15:31:02 +02:00
Vitaly Baranov
c2c35fad82 Refactoring of the code getting create table queries for backup. 2022-06-15 20:32:35 +02:00
Vitaly Baranov
592f568f83 Move backup/restore code to storages and databases - part 2. 2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57 Move backup/restore code to storages and databases - part 1. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21 Rework collecting replicated parts. 2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982 Restore parts of MergeTree in correct order. 2022-06-15 20:28:40 +02:00
Nikolai Kochetov
2a9a63ac7b Merge branch 'master' into refactor-something-in-part-volumes 2022-06-15 15:35:26 +02:00
Vadim Volodin
637d293fbd Add SYSTEM UNFREEZE query 2022-06-08 15:21:14 +03:00
Nikolai Kochetov
678d978acf Merge branch 'master' into refactor-something-in-part-volumes 2022-06-07 15:23:00 +00:00
Nikolai Kochetov
89c5855d20 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-02 12:19:07 +02:00
Nikolai Kochetov
c71256ea38 Remove some commented code. 2022-05-30 13:18:20 +00:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
Nikolai Kochetov
fd97a9d885 Move some resources 2022-05-23 19:47:32 +00:00
Alexander Tokmakov
db66efb1a7 fix logical error on truncate table 2022-05-06 16:12:31 +02:00
Nikolai Kochetov
35095191eb Merge branch 'master' into refactor-something-in-part-volumes 2022-05-03 17:51:47 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable 
... IStorage has deleted copy ctor / assignment already
 2022-05-03 09:07:08 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Nikolai Kochetov
e44af67fee Merge branch 'master' into refactor-something-in-part-volumes 2022-04-26 21:08:00 +02:00
Vitaly Baranov
68a020ecea Implement BACKUP/RESTORE ON CLUSTER. 2022-04-25 16:34:33 +02:00
Vitaly Baranov
acd28d8a1d Implement RESTORE for replicated tables. 2022-04-25 16:34:33 +02:00
alesapin
ebdd53ef8f Fix shared fetch 2022-04-22 19:18:18 +02:00
Nikolai Kochetov
8c00692844 Part 8 2022-04-22 16:58:09 +00:00
Nikolai Kochetov
9133e398b8 Part 7 2022-04-21 19:19:13 +00:00
alesapin
cbfe00bb47 Fix tidy 2022-04-21 00:30:13 +02:00
alesapin
c14e2e0b96 Fix more 2022-04-20 21:08:26 +02:00
alesapin
40c15222f8 Merge branch 'master' into fix_trash 2022-04-20 12:45:49 +02:00
tavplubix
2c54381520
Merge pull request #36200 from ClickHouse/fix_create_as_replicated_default_args 
Fix CREATE AS for ReplicatedMergeTree with default arguments
 2022-04-20 12:01:13 +03:00
alesapin
7cb7c120cc Less ugly 2022-04-19 15:53:10 +02:00
alesapin
cc06bc3d99 Add some clarifications 2022-04-19 14:01:30 +02:00
alesapin
bd7b3847c1 Some code 2022-04-19 01:09:09 +02:00
alesapin
1706ae9e15 Some trash implementation 2022-04-15 18:36:23 +02:00
alesapin
5a8419a48e Remove more trash 2022-04-15 17:05:17 +02:00
Alexander Tokmakov
40fea2b833 do not unfold {uuid} macro 2022-04-13 16:51:59 +02:00
Alexander Tokmakov
457a9e9691 fixes for ReplicatedMergeTree 2022-04-12 14:14:26 +02:00
Alexander Tokmakov
37a06eec1a fixes 2022-04-05 17:36:53 +02:00
Alexander Tokmakov
7cba97aeab Merge branch 'master' into resubmit_21474 2022-03-21 12:09:00 +01:00
alesapin
f2c5e2d3a0 Don't spam logs in zero copy replication 2022-03-19 17:31:33 +01:00
Anton Popov
0ba78c3c3a Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-16 15:28:09 +00:00
Maksim Kita
2fdcf53a76 Fix clang-tidy warnings in Server, Storages folders 2022-03-14 18:17:35 +00:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
alesapin
b75d551281 Fix clang tidy and add check for master 2022-02-14 14:37:41 +03:00
alesapin
b2886a429b Fix lock during fetch 2022-02-14 12:20:27 +03:00
alesapin
3af06b23f8 POC 2022-02-10 22:45:52 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
Alexander Tokmakov
897e94c16c make restarting thread less bad 2022-02-03 23:29:24 +03:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Amos Bird
ec7d367814
DiskLocal checker 
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
 2022-02-01 05:55:27 +08:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
tavplubix
da9a38655b
Merge pull request #33847 from ClickHouse/fix_33806 
Fix logical error on RESTORE REPLICA
 2022-01-24 13:28:37 +03:00
tavplubix
766786b570
Update StorageReplicatedMergeTree.h 2022-01-24 12:16:13 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
8921725d71 fix logical error on RESTORE REPLICA 2022-01-21 14:32:19 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts 
Fix flushing of in-memory parts
 2022-01-21 02:18:03 +03:00
alesapin
4caef03e6a Add ephemeral lock in zookeeper 2022-01-17 14:52:51 +03:00