songenjie
564136ec46
[clickhouse][server][dll][alter]support fetch part
2021-04-13 12:40:33 +08:00
Azat Khuzhin
dfe3699e77
Respect max_part_removal_threads for ReplicatedMergeTree
2021-04-11 11:33:45 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Mike Kot
c947280dfc
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-04-01 21:38:51 +03:00
alexey-milovidov
1512ff832e
Merge branch 'master' into nv/replicated-fetches-timeouts
2021-03-29 19:25:13 +03:00
Mike Kot
406d037ebb
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-03-15 18:41:47 +03:00
Anton Ivashkin
aff13c0c52
Make method StorageReplicatedMergeTree::fetchExistsPart
2021-03-09 20:49:50 +03:00
alesapin
5b3161e0b5
Get rid of const_cast
2021-03-05 20:24:06 +03:00
Anton Ivashkin
d08b481660
Fixes by review responces
2021-03-05 19:20:38 +03:00
Nicolae Vartolomei
1d83c596f8
RFC: Throw exception if removing parts from ZooKeeper fails.
...
This is used for removing part metadata from ZooKeeper when executing
queue events like `DROP_RANGE` triggered when a user tries to drop a
part or a partition. There are other uses but I'll focus only on this
one.
Before this change the method was giving up silently if it was unable to
remove parts from ZooKeeper and this behaviour seems to be problematic.
It could lead to operation being reported as successful at first but
data reappearing later (very rarely) or "stuck" events in replication
queue.
Here is one particular scenario which I think we've hit:
* Execute a DETACH PARTITION
* DROP_RANGE event put in the queue
* Replicas try to execute dropRange but some of them get disconnected
from ZK and 5 retries aren't enough (ZK is miss-behaving), return code
(false) is ignored and log pointer advances.
* One of the replica where dropRange failed is restarted.
* checkParts is executed and it finds parts that weren't removed from
ZK, logs `Removing locally missing part from ZooKeeper and queueing a
fetch` and puts GET_PART on the queue.
* Few things can happen from here:
* There is a lagging replica that din't execute DROP_RANGE yet: part will be
fetched. The other replica will execute DROP_RANGE later and we'll
get diverging set of parts on replicas.
* Another replica also silently failed to remove parts from ZK: both
of them are left with GET_PART in the queue and none of them can
make progress, logging: `No active replica has part ... or covering
part`.
2021-03-05 09:50:26 +00:00
Mike Kot
4e1ac185b5
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-03-01 20:58:07 +03:00
Anton Ivashkin
3c11d44494
Add description for getUniqueId method, fix typos
2021-03-01 13:31:36 +03:00
Anton Ivashkin
5b267b7eec
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication
2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557
Fixes by review response
2021-02-26 12:48:57 +03:00
Mike Kot
2b3b335eda
Exit fixes
2021-02-25 21:41:09 +03:00
Mike Kot
b2c898f58c
Adding the part when found
2021-02-25 21:25:55 +03:00
Mike Kot
ca83775711
Multiple small hotfixes
...
Small fixes
Some fix for old bug
Another old code fix
2021-02-16 16:39:18 +03:00
Mike Kot
73f4740be5
Trying to re-write the solution by adding new command type
...
ATTACH_PART into the replicated log.
The LogEntry now also has the pre-calculated part checksum for this
entry type, which is later used while searching in the detached/ folder
2021-02-15 18:06:48 +03:00
Mike Kot
cd32803709
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-02-15 15:42:17 +03:00
Nicolae Vartolomei
b153e8c190
Add support for custom fetchPart timeouts
2021-02-08 19:44:02 +00:00
Alexander Tokmakov
78c1d69b8c
better code
2021-02-08 22:36:17 +03:00
Anton Ivashkin
e64c63c611
Merge master
2021-02-05 20:10:06 +03:00
alesapin
01c8b9e1b1
Fix rare bug when some replicated operations (like mutation) cannot process some parts after data corruption
2021-01-27 13:07:18 +03:00
Anton Ivashkin
357d98eb36
Merge master
2021-01-20 12:23:03 +03:00
Anton Ivashkin
eba98b04b0
Zero copy replication over S3: Hybrid storage support
2021-01-18 19:16:45 +03:00
Mike Kot
6b949109f1
marked places to edit
2021-01-12 21:46:03 +03:00
Alexey Milovidov
9be5fa9ef2
Merge branch 'master' into Enmk-Optimize_deduplicate
2020-12-20 09:57:10 +03:00
Alexey Milovidov
ca9ccf4850
Comment update
2020-12-16 06:03:43 +03:00
Vasily Nemkov
70ea507dae
OPTIMIZE DEDUPLICATE BY columns
...
Extended OPTIMIZE ... DEDUPLICATE syntax to allow explicit (or implicit with asterisk/column transformers) list of columns to check for duplicates on.
Following syntax variants are now supported:
OPTIMIZE TABLE table DEDUPLICATE; -- the old one
OPTIMIZE TABLE table DEDUPLICATE BY *;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT (colX, colY);
OPTIMIZE TABLE table DEDUPLICATE BY col1,col2,col3;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex');
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT (colX, colY);
Note that * behaves just like in SELECT: MATERIALIZED, and ALIAS columns are not used for expansion.
Also, it is an error to specify empty list of columns, or write an expression that results in an empty list of columns, or deduplicate by an ALIAS column.
Column transformers other than EXCEPT are not supported.
2020-12-07 09:44:07 +03:00
Nicolae Vartolomei
796aee032d
Update StorageReplicatedMergeTree::waitForAllReplicasToProcessLogEntry to support waiting on foreign shards / tables
...
This is not used anywhere yet but needed for an upcoming PR for part movement between shards.
2020-12-04 13:01:12 +00:00
alexey-milovidov
f4a61ac3c3
Merge pull request #17527 from ucasFL/spelling
...
fix spelling errors
2020-11-29 13:45:42 +03:00
feng lv
7e3524caa1
fix spelling errors
2020-11-28 08:17:20 +00:00
alexey-milovidov
dfae1efbbd
Merge pull request #17070 from fastio/master
...
Support multiple ZooKeeper clusters
2020-11-27 10:38:01 +03:00
nikitamikhaylov
72c7cd6693
replace Context& to Settings&
2020-11-25 16:47:32 +03:00
nikitamikhaylov
68bef22fda
Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-sequential-consistency
2020-11-23 16:28:35 +03:00
Nicolae Vartolomei
94293ca3ce
Assign UUIDs to parts only when configured to do so
...
Avoid breaking backwards compatibility by default for now.
2020-11-20 13:49:17 +00:00
Peng Jian
3c86c8b3c9
fix test cases
2020-11-19 15:44:47 +08:00
Peng Jian
091f7065cd
fix build
2020-11-19 15:44:47 +08:00
Peng Jian
2df0c3e0b0
Add helper function for extracting zookeeper name and path.
2020-11-19 15:44:47 +08:00
Peng Jian
a0683ce460
Support mulitple ZooKeeper clusters
2020-11-19 15:44:47 +08:00
Mikhail Filimonov
97fef77ed1
execute_merges_on_single_replica
2020-11-18 08:45:33 +01:00
alesapin
8d3858fc22
Merge pull request #16895 from CurtizJ/remove-empty-parts
...
Remove empty parts after they were pruned by TTL [2]
2020-11-13 10:00:18 +03:00
alesapin
bbd85c3b2d
Merge pull request #16390 from ClickHouse/add_separate_pool_for_fetches
...
Add separate pool for fetches
2020-11-12 22:42:28 +03:00
Anton Popov
4cecd89e84
fix dropping of empty parts
2020-11-12 20:37:54 +03:00
alesapin
2cac5e5d97
Merge pull request #16875 from CurtizJ/refactor-alter-partition
...
Avoid code duplication in alterPartition
2020-11-12 13:19:59 +03:00
alesapin
a8d466a645
Merge branch 'master' into add_separate_pool_for_fetches
2020-11-12 12:51:02 +03:00
Anton Popov
6f0dc08ba7
avoid code duplication in alterPartition
2020-11-11 16:34:07 +03:00
Nikolai Kochetov
195c941c4e
Merge branch 'master' into storage-read-query-plan
2020-11-10 15:02:22 +03:00
alesapin
2e357516a6
Merge branch 'master' into add_separate_pool_for_fetches
2020-11-10 13:53:08 +03:00
Vladimir Chebotarev
059357d51e
ALTER UPDATE/DELETE ... IN PARTITION
with partition pruning in ReplicatedMergeTree
(#13403 )
...
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-11-10 13:23:46 +03:00
alesapin
e347f4b72f
Add some comments
2020-11-09 12:14:20 +03:00
alesapin
72046313d4
Merge branch 'master' into nvartolomei-drop-part
2020-11-09 10:02:02 +03:00
alesapin
4e846d2197
Merge branch 'master' into add_separate_pool_for_fetches
2020-11-09 10:00:21 +03:00
Alexey Milovidov
5314185e25
Merge branch 'master' into azat-optimize_skip_unused_shards-optimization
2020-11-08 00:17:59 +03:00
Nikolai Kochetov
c10f733587
Merge branch 'master' into storage-read-query-plan
2020-11-06 15:43:46 +03:00
alesapin
da8ac948bf
Disallow to drop parts with parallel quorum
2020-11-03 12:24:10 +03:00
alesapin
dac6705995
DROP PART and clear blocks in a single zookeeper op
2020-11-02 20:30:59 +03:00
alesapin
67d22b3e57
Merge branch 'master' into nvartolomei-drop-part
2020-11-02 19:09:09 +03:00
alesapin
47ebc9f74e
Add separate pool for fetches
2020-10-26 14:02:47 +03:00
hchen9
feaab260d8
Fix conflict
2020-10-23 12:54:33 -07:00
alesapin
39e47c5338
Less strange lambdas
2020-10-23 11:54:00 +03:00
alesapin
7a63b373f9
Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more
2020-10-22 15:47:11 +03:00
Nikolai Kochetov
7fa045cff8
Merge branch 'master' into storage-read-query-plan
2020-10-22 13:31:10 +03:00
alexey-milovidov
adeba6bdd8
Merge pull request #15074 from amosbird/btc
...
Extend trivial count optimization.
2020-10-22 02:50:57 +03:00
alesapin
7edbe4d276
Add some comments
2020-10-16 13:12:31 +03:00
alesapin
4c1f5cab13
Sleep less in background task
2020-10-15 19:10:22 +03:00
alesapin
483893cdd4
Slightly refactored code
2020-10-14 17:56:42 +03:00
alesapin
0b14a31ba9
Get rid of background processing pool
2020-10-14 15:44:10 +03:00
alesapin
4014e0f08d
Something very similar to working code
2020-10-14 15:32:35 +03:00
alesapin
efd3126b5d
Moving pool
2020-10-14 10:22:48 +03:00
alesapin
fd35368c59
First simple implementation
2020-10-13 17:25:42 +03:00
alesapin
cb7db6d232
Split select and process merges
...
(cherry picked from commit 1889eb0ff1
)
2020-10-13 15:04:16 +03:00
Nikolai Kochetov
7e58f99f64
Merge branch 'master' into storage-read-query-plan
2020-10-12 13:12:39 +03:00
alexey-milovidov
0fd007ad9e
Merge pull request #15601 from alexelex/alexelex-master
...
Improvement of Quorum Inserts in ClickHouse
2020-10-09 22:43:53 +03:00
Amos Bird
867216103f
Extend trivial count optimization.
2020-10-08 18:08:17 +08:00
hchen9
ed2ce4a6fc
Resolve conflict in StorageMemory
2020-10-07 17:19:59 -07:00
Latysheva Alexandra
6c26f77380
Merge pull request #11 from alexelex/quorum_inserts_2
...
changing the condition for updateQuorum, specialized it and reduced t…
2020-10-07 17:49:20 +06:00
Alexandra Latysheva
a43cac2c1a
changing the condition for updateQuorum, specialized it and reduced the number of calls to the zk
2020-10-07 11:28:48 +00:00
Latysheva Alexandra
85bcc07c0b
Merge pull request #9 from alexelex/quorum_inserts_2
...
Quorum inserts [version 3]
2020-10-07 04:04:41 +06:00
Alexandra Latysheva
f549ecf9d1
insert_quorum_parallel: status nodes in /quorum/parallel and allow merges
2020-10-06 21:49:48 +00:00
Alexander Tokmakov
30a18f7567
make it better
2020-10-05 16:52:03 +03:00
Latysheva Alexandra
a63084adaf
Merge pull request #3 from alexelex/quorum_inserts_2
...
Quorum inserts 2
2020-10-05 03:28:58 +06:00
Alexandra Latysheva
8263e62298
working copy (with some debug info)
2020-10-04 19:55:39 +00:00
Azat Khuzhin
b838214a35
Pass non-const SelectQueryInfo (and drop mutable qualifiers)
2020-10-02 22:42:35 +03:00
hchen9
b3949db00f
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into trivial_count_seq_consistency
2020-10-01 21:15:26 -07:00
Nikolai Kochetov
ec64def384
Use QueryPlan while reading from MergeTree.
2020-10-01 20:34:22 +03:00
Nicolae Vartolomei
7197a0c803
Possibly fix the issue with quorum
2020-10-01 11:38:50 +01:00
hchen9
b4494578d7
Support select_sequential_consistency in StorageReplicatedMergeTree.totalRows and totalBytes
2020-09-30 16:50:58 -07:00
Alexandra Latysheva
e89a56969f
first part
2020-09-30 23:16:27 +00:00
Alexander Tokmakov
34addcf61f
add review suggestions
2020-09-26 22:18:28 +03:00
Nicolae Vartolomei
a62e96c7aa
nit: fix argument names for clang-10
2020-09-08 10:56:10 +01:00
Nicolae Vartolomei
97d0b5ab23
Remove unused query argument
2020-09-08 10:56:10 +01:00
Nicolae Vartolomei
6060a1ab57
ALTER TABLE ... DROP|DETACH PART for ReplicatedMergeTree
2020-09-07 17:01:45 +01:00
alesapin
f4c7ff0376
Add fixed size of Merge TTLS
2020-09-03 16:00:13 +03:00
alesapin
5b0822902b
Merge pull request #14155 from amosbird/fzk
...
Allow FETCH PARTITION from other zookeepers
2020-08-31 09:54:51 +03:00
alesapin
2d33a4029b
Merge pull request #14220 from ClickHouse/remove_redundant_flag
...
Disable force TTL on optimize final
2020-08-29 10:40:36 +03:00
alesapin
232c264d7d
Disable force TTL on optimise
2020-08-28 16:45:42 +03:00
Amos Bird
078b14610d
ALTER MODIFY SAMPLE BY
2020-08-27 22:31:30 +08:00
Amos Bird
0c1cf22c00
Allow FETCH PARTITION from other zookeepers
2020-08-27 22:19:18 +08:00
Nikolai Kochetov
9b67cd9faf
Merge branch 'master' into refactor-pipes-3
2020-08-10 10:50:17 +03:00
Alexey Milovidov
edd89a8610
Fix half of typos
2020-08-08 03:47:03 +03:00
Nikolai Kochetov
2cca4d5fcf
Refactor Pipe [part 2].
2020-08-03 16:54:14 +03:00
alesapin
c75830e878
Alter table freeze with verbose output
2020-07-28 18:10:36 +03:00
alesapin
4a53264a86
Remove redundant and duplicated code
2020-07-13 19:19:08 +03:00
Anton Popov
4422df2e37
Merge remote-tracking branch 'upstream/master' into HEAD
2020-07-02 20:18:21 +03:00
Vladimir Chebotarev
bb8da71eff
Moves task shall be started if new storage policy needs them.
2020-06-30 15:36:44 +03:00
Alexey Milovidov
cc597bb30b
Implement for ReplicatedMergeTree
2020-06-28 22:39:31 +03:00
alesapin
6f1824f0ea
Correct merge with master
2020-06-26 14:30:23 +03:00
tavplubix
2401bc9143
Merge pull request #11896 from ClickHouse/merging_drop_replica
...
Merging #10679
2020-06-24 11:42:39 +03:00
Alexander Tokmakov
7b5de16da9
minor improvements
2020-06-23 22:15:01 +03:00
amudong
2a51286527
Optimize code
2020-06-23 16:22:53 +08:00
amudong
0101947403
rewrite static StorageReplicatedMergeTree::dropReplica
2020-06-23 12:12:30 +08:00
amudong
7723dc4935
add drop relica database and the whole replica
...
fix removeReplicaByZKPath
fix bug: add zkpath empty judge
fix: rewrite code
delete useless code.
fix:ast fromat
fix bug
add test_drop_replica
add drop_replica doc
add drop databse checkAccess
refactor dropReplica
update tests
add static method StorageReplicatedMergeTree::dropReplicaByZkPath
update doc and delete useless code
fix conflict
fix doc
fix doc
fix StorageReplicatedMergeTree::dropReplica
fix bug
delete useless code
2020-06-23 12:12:30 +08:00
amudong
2e6a3eff8e
fix: can not drop local table and replace ALTER with SYSTEM
...
fix test_drop_replica
fix drop replica '/path/to/zk/' ending in '/' and update doc
2020-06-23 12:12:30 +08:00
sundy-li
15ad830290
only drop inactive replica
...
update doc
Increase timeout to release the zookeeper Ephemeral nodes
Fix code comment
use PartitionManager
make integrations test passed
2020-06-23 12:12:30 +08:00
sundy-li
34df59baf8
Add tests and docs
...
Fix integration-tests
2020-06-23 12:12:30 +08:00
sundy-li
d4aa3cea4b
fix merge state
...
Fix style && build
2020-06-23 12:12:30 +08:00
sundy-li
906a43e4a8
Add drop replica alter support
2020-06-23 12:12:30 +08:00
alesapin
f262c4f652
Better comments
2020-06-22 12:49:21 +03:00
alesapin
b9e74f4e82
Merge branch 'master' into atomic_metadata5
2020-06-22 12:03:53 +03:00
Alexey Milovidov
468eaa8d93
Remove unused method
2020-06-19 17:21:26 +03:00
Alexey Milovidov
5211a42c04
Remove leader election, step 3
2020-06-19 17:18:58 +03:00
alesapin
d79982f497
Better locks in Storages
2020-06-18 19:10:47 +03:00
alesapin
aab4ce6394
Truncate with metadata
2020-06-18 13:29:13 +03:00
alesapin
ed8f3b2fc4
TTL in storage in memory metadata
2020-06-17 16:39:26 +03:00
alesapin
2fac3290f0
Merge branch 'master' into atomic_metadata5
2020-06-17 11:14:03 +03:00
alexey-milovidov
598fd75caf
Merge pull request #11639 from ClickHouse/remove-leader-election-2
...
Remove leader election, step 2: allow multiple leaders
2020-06-17 09:25:23 +03:00
alesapin
ccc2bda666
getConstraints() in StorageInMemoryMetadata (suspicious commit, but pretend to work)
2020-06-16 19:55:04 +03:00
alesapin
1ddeb3d149
Buildable getSampleBlock in StorageInMemoryMetadata
2020-06-16 18:51:29 +03:00
alesapin
36ba0192df
Metadata in read and write methods of IStorage
2020-06-15 22:08:58 +03:00
alesapin
bc9a3c0458
Merge branch 'master' into consistent_metadata4
2020-06-15 11:20:30 +03:00
Alexey Milovidov
d2c66f9688
Added comments
2020-06-15 05:14:59 +03:00
Alexey Milovidov
bbe5f4c909
Revert "Remove "current_password" because it is harmful"
...
This reverts commit 66ccb2f6b1
.
2020-06-15 05:13:41 +03:00
Alexey Milovidov
66ccb2f6b1
Remove "current_password" because it is harmful
2020-06-15 04:12:01 +03:00
Alexey Milovidov
b596831342
Merge branch 'master' into remove-leader-election-2
2020-06-15 03:16:41 +03:00
Alexey Milovidov
400e9fb64f
Allow to drop table if there is no metadata in ZooKeeper; allow to rename
2020-06-14 04:23:53 +03:00
alesapin
5d3de747cd
Merge remote-tracking branch 'origin/master' into consistent_metadata4
2020-06-13 11:50:27 +03:00
Alexey Milovidov
b8b55a5b99
More LeaderElection to Storage/MergeTree
2020-06-13 04:35:49 +03:00
Alexey Milovidov
6869c122a8
Step 1 towards removing leader election: check and modify version of the "log" node
2020-06-13 04:34:05 +03:00
alesapin
a39d010f94
Merge branch 'master' into consistent_metadata4
2020-06-11 18:47:41 +03:00
Alexey Milovidov
b72a095e99
Fix race conditions in CREATE/DROP of different replicas of ReplicatedMergeTree
2020-06-11 06:24:52 +03:00
alesapin
a5a9048109
At least fixed gcc-9 build
2020-06-08 21:23:26 +03:00
Alexey Milovidov
a0fd9b021a
Fix MSan failure in MergeTree
2020-06-04 01:11:06 +03:00
Alexander Tokmakov
032197b015
fix alter and optimize hangs when replica becomes inactive
2020-05-13 10:06:57 +03:00
Alexander Tokmakov
04d6b59ac0
Merge branch 'master' into database_atomic
2020-04-23 17:31:37 +03:00
Nikita Mikhaylov
05206df88c
better
2020-04-20 13:56:59 +03:00
Nikita Mikhaylov
441bb13d4a
better
2020-04-13 18:21:05 +03:00
Nikita Mikhaylov
a8d72d01c3
mvp fix
2020-04-11 00:29:54 +03:00
Alexander Tokmakov
140cd88c20
fixes
2020-04-09 02:59:39 +03:00
Alexander Tokmakov
dd1590830b
Merge branch 'master' into database_atomic
2020-04-08 22:00:46 +03:00
Alexander Tokmakov
4c48b7dd80
better rename
2020-04-07 18:31:33 +03:00
Alexander Tokmakov
08bae4668d
Merge branch 'master' into database_atomic
2020-04-06 16:18:07 +03:00
alesapin
1cb072d58a
Merge branch 'master' into alter_rename_column
2020-04-06 11:40:27 +03:00
Ivan Lezhankin
06446b4f08
dbms/ → src/
2020-04-03 18:14:31 +03:00