thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-09 16:02:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
109,548 Commits 9,209 Branches 1,782 Tags 2 GiB
3de905bb7c
Commit Graph

396 Commits

Author SHA1 Message Date
Vitaly Baranov
73b1894a21 Rework collecting replicated parts. 2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982 Restore parts of MergeTree in correct order. 2022-06-15 20:28:40 +02:00
Nikolai Kochetov
2a9a63ac7b Merge branch 'master' into refactor-something-in-part-volumes 2022-06-15 15:35:26 +02:00
Vadim Volodin
637d293fbd Add SYSTEM UNFREEZE query 2022-06-08 15:21:14 +03:00
Nikolai Kochetov
678d978acf Merge branch 'master' into refactor-something-in-part-volumes 2022-06-07 15:23:00 +00:00
Nikolai Kochetov
89c5855d20 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-02 12:19:07 +02:00
Nikolai Kochetov
c71256ea38 Remove some commented code. 2022-05-30 13:18:20 +00:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
Nikolai Kochetov
fd97a9d885 Move some resources 2022-05-23 19:47:32 +00:00
Alexander Tokmakov
db66efb1a7 fix logical error on truncate table 2022-05-06 16:12:31 +02:00
Nikolai Kochetov
35095191eb Merge branch 'master' into refactor-something-in-part-volumes 2022-05-03 17:51:47 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable 
... IStorage has deleted copy ctor / assignment already
 2022-05-03 09:07:08 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Nikolai Kochetov
e44af67fee Merge branch 'master' into refactor-something-in-part-volumes 2022-04-26 21:08:00 +02:00
Vitaly Baranov
68a020ecea Implement BACKUP/RESTORE ON CLUSTER. 2022-04-25 16:34:33 +02:00
Vitaly Baranov
acd28d8a1d Implement RESTORE for replicated tables. 2022-04-25 16:34:33 +02:00
alesapin
ebdd53ef8f Fix shared fetch 2022-04-22 19:18:18 +02:00
Nikolai Kochetov
8c00692844 Part 8 2022-04-22 16:58:09 +00:00
Nikolai Kochetov
9133e398b8 Part 7 2022-04-21 19:19:13 +00:00
alesapin
cbfe00bb47 Fix tidy 2022-04-21 00:30:13 +02:00
alesapin
c14e2e0b96 Fix more 2022-04-20 21:08:26 +02:00
alesapin
40c15222f8 Merge branch 'master' into fix_trash 2022-04-20 12:45:49 +02:00
tavplubix
2c54381520
Merge pull request #36200 from ClickHouse/fix_create_as_replicated_default_args 
Fix CREATE AS for ReplicatedMergeTree with default arguments
 2022-04-20 12:01:13 +03:00
alesapin
7cb7c120cc Less ugly 2022-04-19 15:53:10 +02:00
alesapin
cc06bc3d99 Add some clarifications 2022-04-19 14:01:30 +02:00
alesapin
bd7b3847c1 Some code 2022-04-19 01:09:09 +02:00
alesapin
1706ae9e15 Some trash implementation 2022-04-15 18:36:23 +02:00
alesapin
5a8419a48e Remove more trash 2022-04-15 17:05:17 +02:00
Alexander Tokmakov
40fea2b833 do not unfold {uuid} macro 2022-04-13 16:51:59 +02:00
Alexander Tokmakov
457a9e9691 fixes for ReplicatedMergeTree 2022-04-12 14:14:26 +02:00
Alexander Tokmakov
37a06eec1a fixes 2022-04-05 17:36:53 +02:00
Alexander Tokmakov
7cba97aeab Merge branch 'master' into resubmit_21474 2022-03-21 12:09:00 +01:00
alesapin
f2c5e2d3a0 Don't spam logs in zero copy replication 2022-03-19 17:31:33 +01:00
Anton Popov
0ba78c3c3a Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-16 15:28:09 +00:00
Maksim Kita
2fdcf53a76 Fix clang-tidy warnings in Server, Storages folders 2022-03-14 18:17:35 +00:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
alesapin
b75d551281 Fix clang tidy and add check for master 2022-02-14 14:37:41 +03:00
alesapin
b2886a429b Fix lock during fetch 2022-02-14 12:20:27 +03:00
alesapin
3af06b23f8 POC 2022-02-10 22:45:52 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
Alexander Tokmakov
897e94c16c make restarting thread less bad 2022-02-03 23:29:24 +03:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Amos Bird
ec7d367814
DiskLocal checker 
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
 2022-02-01 05:55:27 +08:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
tavplubix
da9a38655b
Merge pull request #33847 from ClickHouse/fix_33806 
Fix logical error on RESTORE REPLICA
 2022-01-24 13:28:37 +03:00
tavplubix
766786b570
Update StorageReplicatedMergeTree.h 2022-01-24 12:16:13 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
8921725d71 fix logical error on RESTORE REPLICA 2022-01-21 14:32:19 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts 
Fix flushing of in-memory parts
 2022-01-21 02:18:03 +03:00
alesapin
4caef03e6a Add ephemeral lock in zookeeper 2022-01-17 14:52:51 +03:00
alesapin
bf803472b1 Add a tests for a TTL bug in zero copy replication 2022-01-14 18:44:10 +03:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474 2022-01-07 10:12:25 +00:00
Alexander Tokmakov
187c43eba8 rename Committed state to Active 2021-12-30 23:45:38 +03:00
alesapin
91e1ac437e Tiny improvements 2021-12-30 12:57:38 +03:00
alesapin
cbdba89d65 Merge branch 'master' into ianton-ru-MDB-15474 2021-12-30 11:21:19 +03:00
alesapin
8b331cd9bb Remove method from IStorage 2021-12-29 22:41:47 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
Anton Ivashkin
0465aef15d Fixes by code review responces 2021-12-27 19:27:06 +03:00
Anton Popov
a83832c03e fix flushing of in-memory parts 2021-12-27 18:54:28 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474 2021-12-21 17:38:36 +03:00
Anton Ivashkin
33cbfc89ef Move logic for replicated part to StorageReplicatedMergeTree class 2021-12-21 17:29:50 +03:00
Anton Ivashkin
0c0bf66334 Merge master 2021-12-21 17:27:54 +03:00
Anton Ivashkin
c724b074ae Remove zero-copy version converter 2021-12-20 20:23:25 +03:00
Anton Ivashkin
f0b9a4327a Use table UUID in zero-copy shared label in ZooKeeper 2021-12-20 16:19:00 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
Alexander Tokmakov
1b02136941 fix skipping of some mutations 2021-12-15 21:19:29 +03:00
tavplubix
4f46ac6b30
Remove LeaderElection (#32140) 
* remove LeaderElection

* try fix tests

* Update test.py

* Update test.py
 2021-12-07 19:55:55 +03:00
Anton Ivashkin
80ab73c691 Fix Zero-Copy replication lost locks, fix remove used remote data in DROP DETACHED PART 2021-12-01 16:11:26 +03:00
Anton Ivashkin
0f9038ebed Zero-copy: move shared mark outside table node in ZooKeeper 2021-11-29 19:05:31 +03:00
Anton Popov
a20922b2d3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-11-09 15:36:25 +03:00
alesapin
2af950d4d5
Merge pull request #30658 from ClickHouse/std_mutex_alter_lock 
Change `alter_lock` from `RWLock` to `std::timed_mutex`
 2021-10-27 11:14:11 +03:00
mergify[bot]
ce88a84e88
Merge branch 'master' into nv/parts-uuid-move-shard-kill 2021-10-26 11:09:19 +00:00
Alexander Tokmakov
2e7e195e77 change alter_lock to std::timed_mutex 2021-10-26 13:37:00 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
mergify[bot]
3b1f49aac7
Merge branch 'master' into nv/parts-uuid-move-shard-kill 2021-10-05 12:36:02 +00:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Nicolae Vartolomei
9a02061d9c
Rewrite part movement between shards logic and add kill support 
See https://github.com/ClickHouse/ClickHouse/pull/24585#issuecomment-857735081
for extra context about the current implementation.
 2021-09-17 18:11:32 +01:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask (#25165) 
Introduced MergeTask and MutateTask
 2021-09-17 00:19:58 +03:00
Anton Popov
4c388e3d84 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-09-09 14:10:16 +03:00
Nikita Mikhaylov
6062dd0021 Better 2021-09-08 00:21:21 +00:00
Nikita Mikhaylov
ea0fbf81af Renaming 2021-09-06 12:01:16 +00:00
Nikita Mikhaylov
292a24abe8 Merge upstream/master into global-merge-executor (using imerge) 2021-09-03 00:34:24 +00:00
Nikita Mikhaylov
cc7c221fad Own PriorityQueue + prettifying the code 2021-09-02 21:31:32 +00:00
mergify[bot]
5d299fbdee
Merge branch 'master' into remove_outdated_settings 2021-09-01 14:07:48 +00:00
Nikita Mikhaylov
dbc950caa4 added a test 2021-08-31 14:54:24 +00:00
alesapin
7223a32aac Followup 2021-08-31 15:29:33 +03:00
alesapin
921e51e061 Remove some obsolete settings for replicated fetches 2021-08-31 15:22:56 +03:00
Nikita Mikhaylov
f8d4f04294 Merge upstream/master into global-merge-executor (using imerge) 2021-08-31 11:52:11 +00:00
Nikita Mikhaylov
c4416906c8 done 2021-08-30 19:37:03 +00:00
Alexey Milovidov
79e0433ba7 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-28 01:19:16 +03:00
tavplubix
703101fe4d
Merge pull request #27931 from ClickHouse/wait_for_all_replicas_timeouts 
Avoid too long waiting for inactive replicas
 2021-08-27 14:31:36 +03:00
Alexey Milovidov
7c1d0a3baf Progress on development 2021-08-25 01:24:47 +03:00
Alexander Tokmakov
cc9c2fd63b make code better 2021-08-23 15:57:50 +03:00
Anton Popov
61239343e3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-20 16:33:30 +03:00
Alexander Tokmakov
59eb3aa9a9 avoid too long waiting for inactive replicas 2021-08-20 15:59:57 +03:00
mergify[bot]
f11e396151
Merge branch 'master' into nv/last-queue-update-exception 2021-08-18 07:00:50 +00:00
Nicolae Vartolomei
3f291b024a
Use plain mutex instead of MultiVersion 2021-08-09 13:58:23 +01:00
Maksim Kita
3f48c85722 StorageSystemReplicas added column replica_is_active 2021-08-04 16:19:42 +03:00
Anton Popov
e36736b50c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-02 22:52:02 +03:00
Nicolae Vartolomei
8b07a7f180 Store exception generated when we tried to update the queue last time 
The use case is to alert when queue contains broken entries. Especially
important when ClickHouse breaks backwards compatibility between
versions and log entries written by newer versions aren't parseable by
old versions.

```
Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected 'quorum: ' before: 'merge_type: 2\n'
```
 2021-07-27 15:42:40 +01:00
Nikolai Kochetov
9c92f43359 Update storages. 2021-07-23 22:33:59 +03:00
Anton Popov
f99374cca6 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-07-20 18:20:21 +03:00
Zhichang Yu
5047c758f4 fix per review 2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1 hdfs zero copy 2021-07-13 07:19:12 +00:00
Anton Popov
3ed7f5a6cc dynamic subcolumns: add snapshot for storage 2021-07-09 06:15:41 +03:00
alesapin
0d8844c510
Merge pull request #25884 from ClickHouse/fix_drop_part_in_queue 
Relax `DROP PART` guarantees and turn on checks in ReplicationQueue.
 2021-07-07 10:48:48 +03:00
Anton Popov
9071ecd428 fix alter of settings in MergeTree 2021-07-05 15:44:58 +03:00
alesapin
89a401a84f Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 18:53:17 +03:00
alesapin
6a73c8b49e Review fixes 2021-06-30 18:24:51 +03:00
alesapin
21f15be990 Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 15:30:32 +03:00
alesapin
2eb27540b2 Some test version 2021-06-30 15:29:09 +03:00
alesapin
5822f0ba29 Replace lost parts with empty parts instead of hacking replication queue 2021-06-29 18:14:44 +03:00
Raúl Marín
bfc122df64 Fix some typos in Storage classes 2021-06-28 19:03:56 +02:00
Nikita Mikhaylov
c66a3b22b5 done 2021-06-22 23:24:47 +00:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652) 
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
 2021-06-20 11:24:43 +03:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Alexander Tokmakov
3ade38df82 remove copypaste 2021-06-08 22:17:45 +03:00
alesapin
4a1bf51ad1
Merge branch 'master' into limited_fetches 2021-06-02 23:01:47 +03:00
alexey-milovidov
c7bd3b8674
Merge pull request #23315 from CurtizJ/empty-parts-fix 
Fix waiting while dropping empty parts
 2021-06-01 13:52:49 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
Anton Popov
179f172c4c fix build 2021-05-28 02:10:44 +03:00
Anton Popov
336abfa475 Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-28 01:58:30 +03:00
alesapin
2985def91d Add tests 2021-05-27 15:54:47 +03:00
alesapin
42b2383e01 Basic throttler implementation 2021-05-26 23:37:44 +03:00
Anton Popov
7d1431f6b6 better code near dropPartition 2021-05-25 20:25:00 +03:00
alesapin
20a34e951a Split fetch part into two stages, don't use replicated block output stream 2021-05-21 12:30:49 +03:00
Anton Popov
e27acc26be better dropPart and dropPartition 2021-05-17 17:26:36 +03:00
Anton Popov
e043cc45bf Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-17 16:11:12 +03:00
alesapin
17f229857c Merge branch 'master' into nvartolomei-parts-move 2021-05-17 13:52:48 +03:00
Alexander Tokmakov
970ad94851 Merge branch 'master' into fix_intersecting_parts 2021-05-14 13:17:02 +03:00
Alexander Tokmakov
2da12f0b59 fix intersection of drop range and merge 2021-05-13 17:04:36 +03:00
Alexander Tokmakov
e114c7eb8b fix virtual parts in REPLACE_RANGE 2021-05-13 14:29:59 +03:00
alesapin
46e136b5c4
Merge branch 'master' into nv/parts-uuid-move-shard 2021-05-11 15:36:40 +03:00
Alexander Tokmakov
11124f7015 fix 2021-05-07 20:09:39 +03:00
Nicolae Vartolomei
1fa5871ff7 Fix bad rebase and introduce part_moves_between_shards_enable setting 2021-04-27 14:20:13 +01:00
Nicolae Vartolomei
53d57ffb52 Part movement between shards 
Integrate query deduplication from #17348
 2021-04-27 14:20:12 +01:00
mergify[bot]
b9e229d19d
Merge branch 'master' into empty-parts-fix 2021-04-21 13:19:54 +00:00
Anton Popov
42a0416a2b better drop of empty parts 2021-04-20 05:31:08 +03:00
Alexander Tokmakov
ecc625692e fix build 2021-04-19 13:40:20 +03:00
Alexander Tokmakov
9bc98771f7 fix race on replica creation 2021-04-19 11:21:42 +03:00
songenjie
f731739fab fix suggests 2021-04-14 10:05:41 +08:00
songenjie
7c32cc1e18 fix case style for local variable 2021-04-13 17:34:04 +08:00
songenjie
564136ec46 [clickhouse][server][dll][alter]support fetch part 2021-04-13 12:40:33 +08:00
Azat Khuzhin
dfe3699e77 Respect max_part_removal_threads for ReplicatedMergeTree 2021-04-11 11:33:45 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
Mike Kot
c947280dfc Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-04-01 21:38:51 +03:00
alexey-milovidov
1512ff832e
Merge branch 'master' into nv/replicated-fetches-timeouts 2021-03-29 19:25:13 +03:00
Mike Kot
406d037ebb Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-15 18:41:47 +03:00
Anton Ivashkin
aff13c0c52 Make method StorageReplicatedMergeTree::fetchExistsPart 2021-03-09 20:49:50 +03:00
alesapin
5b3161e0b5 Get rid of const_cast 2021-03-05 20:24:06 +03:00