thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 19:32:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
34,518 Commits 9,319 Branches 1,783 Tags 2.1 GiB
01ff1c65e2
Commit Graph

150 Commits

Author SHA1 Message Date
VadimPE
75900f494f CLICKHOUSE-3847 add "is_lost". load() in Queue can check duplicate records 2018-08-09 18:06:39 +03:00
VadimPE
3b9002dfc9 CLICKHOUSE-3847 fix 2018-08-08 17:31:56 +03:00
VadimPE
01e9d15914 CLICKHOUSE-3847 Del restartLostReplica() 2018-08-07 18:28:12 +03:00
VadimPE
32b6965cf2 CLICKHOUSE-3847 Update. Add new settings. Del is_lost 2018-08-07 18:21:42 +03:00
alesapin
425918afe1 CLICKHOUSE-3880: Fix replace partition logic inside replicated merge tree 2018-08-06 15:59:51 +03:00
Alexey Zatelepin
b6b54eea52 old mutations cleaner for ReplicatedMergeTree engines [#CLICKHOUSE-3828] 2018-08-03 22:21:44 +03:00
VadimPE
3f6cdab293 CLICKHOUSE-3847 add support lost replica 2018-07-30 19:31:14 +03:00
Alexey Zatelepin
d5f8e4a1a5 load quorum status in merge predicate asynchronously [#CLICKHOUSE-3802] 2018-07-05 20:06:33 +03:00
Alexey Zatelepin
a41ee1f0a7 switch to ephemeral nodes for block number locks [#CLICKHOUSE-3802] 2018-07-05 20:06:33 +03:00
Alexey Zatelepin
e78dc8142c factor out common code [#CLICKHOUSE-3747] 2018-06-21 22:47:44 +03:00
Alexey Zatelepin
a6c61848f1 mutations finalizing task [#CLICKHOUSE-3747] 2018-06-21 22:47:44 +03:00
Alexey Zatelepin
36982b967d use single mutex in ReplicatedMergeTreeQueue for simplicity 2018-06-21 22:47:44 +03:00
Alexey Zatelepin
b2cdfcbaca calculate and update parts_to_do for mutations [#CLICKHOUSE-3747] 2018-06-21 22:47:44 +03:00
alexey-milovidov
08ec751a8c
Merge pull request #2504 from yandex/system-mutations-table 
[WIP] system.mutations table skeleton
 2018-06-13 23:00:30 +03:00
Alexey Zatelepin
b60a2a90fa rename method for clarity 2018-06-13 17:15:36 +03:00
Alexey Zatelepin
4ee581117b system.mutations table skeleton [#CLICKHOUSE-3747] 2018-06-13 17:15:36 +03:00
Alexey Zatelepin
2c61a5940c store mutation entries in shared_ptr 2018-06-13 17:14:12 +03:00
Alexey Zatelepin
c5c601f662 save serialized commands in ReplicatedMergeTreeMutationEntry; split InterpreterAlterQuery::parseAlter() [#CLICKHOUSE-3747] 2018-06-13 17:14:08 +03:00
Alexey Milovidov
d864f39ec7 Merge branch 'master' into zhang2014-feature/support_truncate 2018-06-09 19:10:06 +03:00
Alexey Milovidov
0bc678474e Merge branch 'feature/support_truncate' of https://github.com/zhang2014/ClickHouse into zhang2014-feature/support_truncate 2018-06-09 17:24:50 +03:00
Alexey Zatelepin
863c1c6ed5 add fields related to part mutations to system.replicas table [#CLICKHOUSE-3747] 2018-06-06 16:22:30 +03:00
Alexey Zatelepin
3947f25365 add general comment about the mutation algorithm [#CLICKHOUSE-3747] 2018-06-05 17:55:35 +03:00
Alexey Zatelepin
e55097b866 introduce constants for max level and max block number 2018-06-04 14:23:47 +03:00
Alexey Zatelepin
c9bb986330 Merge branch 'master' into replicated-mergetree-mutations 
Conflicts:
	dbms/src/Storages/MergeTree/AbandonableLockInZooKeeper.h
	dbms/src/Storages/MergeTree/ReplicatedMergeTreePartCheckThread.cpp
	dbms/src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
	dbms/src/Storages/MergeTree/ReplicatedMergeTreeQueue.h
	dbms/src/Storages/StorageReplicatedMergeTree.cpp
 2018-05-31 18:52:42 +03:00
Alexey Zatelepin
dbf6949e0b support skipping mutations (just in case) [#CLICKHOUSE-3747] 2018-05-31 18:21:00 +03:00
Vitaliy Lyudvichenko
dc5de3bbec More optimal conflicts check in future_parts. [#CLICKHOUSE-2] 2018-05-31 17:14:50 +03:00
Vitaliy Lyudvichenko
888f5f4a24 Allow merges after CLEAR COLUMN. [#CLICKHOUSE-3] 
Resolves #2315
 2018-05-31 17:14:50 +03:00
Alexey Zatelepin
d89221c458 easier init and deactivation of BackgroundSchedulePool tasks 2018-05-31 16:18:52 +03:00
Alexey Zatelepin
f6c72f8e56 Merge branch 'master' into replicated-mergetree-mutations 
Conflicts:
	dbms/src/Common/ActionBlocker.h
	dbms/src/Storages/MergeTree/ActiveDataPartSet.cpp
	dbms/src/Storages/MergeTree/ActiveDataPartSet.h
	dbms/src/Storages/MergeTree/MergeTreeData.h
	dbms/src/Storages/MergeTree/MergeTreeDataPart.cpp
	dbms/src/Storages/MergeTree/ReplicatedMergeTreeAlterThread.cpp
	dbms/src/Storages/MergeTree/ReplicatedMergeTreeLogEntry.cpp
	dbms/src/Storages/MergeTree/ReplicatedMergeTreeLogEntry.h
	dbms/src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
	dbms/src/Storages/MergeTree/ReplicatedMergeTreeQueue.h
	dbms/src/Storages/MergeTree/ReplicatedMergeTreeRestartingThread.cpp
	dbms/src/Storages/StorageMergeTree.cpp
	dbms/src/Storages/StorageReplicatedMergeTree.cpp
	dbms/src/Storages/StorageReplicatedMergeTree.h
 2018-05-28 18:37:30 +03:00
Alexey Zatelepin
2b80fbf972 DROP PARTITION now works with mutated parts [#CLICKHOUSE-3747] 2018-05-25 23:13:00 +03:00
Alexey Zatelepin
fdb33d8f3c execute mutation log entries [#CLICKHOUSE-3747] 2018-05-25 23:13:00 +03:00
Alexey Zatelepin
1147e052e8 select and write part mutations to queue [#CLICKHOUSE-3747] 2018-05-25 23:13:00 +03:00
Alexey Zatelepin
2721daf782 don't select merges of parts with different mutation versions [#CLICKHOUSE-3747] 2018-05-25 23:06:28 +03:00
Alexey Zatelepin
4c661df146 notify relevant events directly from queue [#CLICKHOUSE-3747] 2018-05-25 23:06:23 +03:00
Alexey Zatelepin
42850f09b7 load mutations from ZK [#CLICKHOUSE-3747] 2018-05-25 23:06:15 +03:00
Alexey Zatelepin
17437a95bf merge selection based on checking current inserts [#CLICKHOUSE-3732] 2018-05-25 23:03:56 +03:00
Vitaliy Lyudvichenko
8349d2b9a0 Fixed gcc7 compilation. [#CLICKHOUSE-2] 2018-05-21 18:10:50 +03:00
Vitaliy Lyudvichenko
c888903488
Add query ALTER TABLE t1 [REPLACE|ATTACH] PARTITION FROM t2 
* Add ATTACH PARTITION FROM table for MergeTree. [#CLICKHOUSE-3546]

* Implemented replicated case on non-leader replica. [#CLICKHOUSE-3546]

* Disable merges in the dropping range. [#CLICKHOUSE-3546]

* DROP PARTITION is atomic and simpler now. [#CLICKHOUSE-3546]

* Implemented more SYSTEM queries. [#CLICKHOUSE-2931] [#CLICKHOUSE-3546]

SYSTEM queries:
RESTART REPLICAS
SYNC REPLICA db.name
STOP MERGES [db.name]
START MERGES [db.name]
STOP FETCHES [db.name]
START FETCHES [db.name]
STOP REPLICATED SENDS [db.name]
START REPLICATED SENDS [db.name]
STOP REPLICATION QUEUES [db.name]
START REPLICATION QUEUES [db.name]

* Fixed a bunch of bugs in REPLACE PARTITION. [#CLICKHOUSE-3546]

* Add tests for REPLACE PARTITION and SYSTEM. [#CLICKHOUSE-3546]

* Add system.part_log logging. [#CLICKHOUSE-3546]

* Fixed long wait in SYNC REPLICA. [#CLICKHOUSE-3546]

* Add requested changes. [#CLICKHOUSE-3546]

Fixed clickhouse-client bad return code.

* Add requested chenges. [#CLICKHOUSE-3546]

* Add requested chenges. [#CLICKHOUSE-3546]
 2018-05-21 16:49:54 +03:00
zhang2014
fa865636ce ISSUES-2259 add comment & move delete database metadata implementation 2018-05-21 11:38:46 +08:00
zhang2014
6613e56784 ISSUES-2259 support truncate syntax 2018-05-21 11:38:46 +08:00
Silviu Caragea
c547c5a7b0 Merge remote-tracking branch 'origin/background-schedule-pool-fix' 
# Conflicts:
#	dbms/src/Common/ZooKeeper/LeaderElection.h
#	dbms/src/Common/ZooKeeper/ZooKeeper.cpp
#	dbms/src/Storages/MergeTree/ReplicatedMergeTreeAlterThread.cpp
#	dbms/src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.cpp
#	dbms/src/Storages/MergeTree/ReplicatedMergeTreeCleanupThread.h
#	dbms/src/Storages/MergeTree/ReplicatedMergeTreePartCheckThread.cpp
#	dbms/src/Storages/MergeTree/ReplicatedMergeTreeRestartingThread.cpp
#	dbms/src/Storages/StorageReplicatedMergeTree.cpp
 2018-04-10 16:20:14 +03:00
alexey-milovidov
428ee63a04
Merge branch 'master' into libzookeeper-rewrite 2018-04-01 17:31:20 +03:00
Vitaliy Lyudvichenko
701b58b4ff Faster DROP PARTITION. [#CLICKHOUSE-2] 2018-03-28 15:48:40 +03:00
Silviu Caragea
0aa9b9efbe Use consistent names for the task variables (change from next_update_task_handle to merge_selecting_handle) (reverted from commit 50992843c6) 2018-03-26 23:12:17 +03:00
Silviu Caragea
50992843c6 Use consistent names for the task variables (change from next_update_task_handle to merge_selecting_handle) 2018-03-26 22:59:58 +03:00
Silviu Caragea
4361df913b Remove exists and use existsWatch instead for WatchCallbacks 2018-03-26 22:46:14 +03:00
Silviu Caragea
31874ed172 Use consistent names for the task variables 2018-03-26 22:37:47 +03:00
Alex Zatelepin
c5b04f7f17 Add info about total uncompressed part size to system tables (#2052) 
* column size calculating routines now take into account multiple streams [#CLICKHOUSE-2831]

* rename for clarity

* use more appropriate method for calculating table size

* add info about total uncompressed part size to system.parts and system.parts_columns [#CLICKHOUSE-2831]

* rename columns for clarity [#CLICKHOUSE-2831]

* count shared substreams only once [#CLICKHOUSE-2831]
 2018-03-26 17:18:04 +03:00
Alexey Milovidov
522bf01db9 Using different ZooKeeper library (development) [#CLICKHOUSE-2] 2018-03-25 03:15:52 +03:00
Alexey Milovidov
58f32f98d7 Using different ZooKeeper library (development) [#CLICKHOUSE-2] 2018-03-24 23:11:46 +03:00
Alexey Milovidov
10be6ca8f8 Rewriting ZooKeeper library [#CLICKHOUSE-2] 2018-03-24 03:45:04 +03:00
Silviu Caragea
a2dc16a582 Remove the link between TaskHandles and ZooKeeper 2018-03-22 17:21:14 +02:00
Silviu Caragea
b24a4b2111 Add back the buggy BackgroundSchedulePool 2018-03-22 17:07:12 +02:00
Alexey Milovidov
d690c24198 Using different ZooKeeper library (development) [#CLICKHOUSE-2] 2018-03-22 00:40:53 +03:00
Vitaliy Lyudvichenko
8028ba0de9 Refactoring of zkutil::ZooKeeper::multi* functions. [#CLICKHOUSE-2] 2018-03-14 16:29:03 +03:00
Alexey Milovidov
368f46d764 Addition to prev. revision [#CLICKHOUSE-2] 2018-03-03 19:48:24 +03:00
Alexey Milovidov
2e371822ad Changed low level data race to high level data race [#CLICKHOUSE-2] 2018-03-03 19:46:32 +03:00
Alexey Milovidov
3b0d713120 Cleanup semantics [#CLICKHOUSE-2] 2018-03-03 19:26:06 +03:00
proller
d97b2edc7c Make separate libs Common/ZooKeeper and Common/ConfigProcessor (#1749) 
* Make separate libs Common/ZooKeeper and Common/ConfigProcessor
renamed:    dbms/src/Common/StringUtils.h -> dbms/src/Common/StringUtils/StringUtils.h

* Fix space

* Fix data in system.build_options

* merge

* merge
 2018-01-15 22:07:47 +03:00
Alexey Zatelepin
6d5ce3fa9a revert BackgroundSchedulePool usage in Replicated tables 2017-12-22 21:25:54 +03:00
Alexey Zatelepin
c422b5b2e4 take PreCommitted parts into account when calculating total merge size [#CLICKHOUSE-3508] 2017-12-20 20:52:28 +03:00
Alexey Zatelepin
2340e55d0e revert to old behaviour of getPartIfExists() [#CLICKHOUSE-3508] 2017-12-20 20:04:14 +03:00
Alexey Zatelepin
01d42242b1 mark queue as dirty if status of ZK operation is unknown [#CLICKHOUSE-3405] 2017-11-28 17:09:53 +03:00
Alexey Zatelepin
78aaa42ddd add idempotent retries of updating the replica queue in ZooKeeper [#CLICKHOUSE-3405] 2017-11-23 16:19:29 +03:00
alexey-milovidov
fc0a51f1b9
Merge branch 'master' into fix4 2017-11-19 23:57:44 +03:00
Silviu Caragea
64619c614f Basic idea is : 
- We add BackgroundSchedulePool which can execute a function at a specific point in time. Basically all tasks are added in a queue and precessed by worker threads.
- The most important difference between this and BackgroundProcessingPool is that we have the guarantee that the same function is not executed from many workers in the same time
- Each of the following classes instead starting a thread will register a task in BackgroundSchedulePool and when they need to run will call schedule or scheduleAfter(duration) functions

This commit is moving all threads created by ReplicatedMergeTree to BackgroundSchedulePool tasks

NOTE: I did a minimum number of changes to be much simple to review the code
 2017-11-17 10:58:35 +02:00
proller
6da4f5a661 add locks 2017-11-01 22:56:07 +03:00
proller
4ffbca2d8c Fix warnings (#1406) 
* Log query id in executeQuery; Better type mismatch error; change format in report tool

* Better log query_id

* fix message

* Use c++11 thread_local instaed of gcc's __thread

* lock mutex before notifying waiting thread in sync insertion into distributed [#CLICKHOUSE-3379]

* Cmake: fix build without downloaded submodules (#1379)

* fix

* ZooKeeper: fixed stack smashing with tryGet()

The tryGet() operation creates a 1MB buffer on stack. This may or
may not work depending on the default stack size for threads,
whether the stack protector is enabled or not, recursion depth,
and the actual value size.

This is probably going to slow down some ZK operations, but I don't
see how else this could work reliably with the existing API.

* increased timeout for test_insertion_sync_fails_with_timeout

* Update CHANGELOG_RU.md

* Update ZooKeeper.cpp

* Fix warnings

* Fixes

* Dont strip debug info from  asan, tsan and other builds except releases

* Fix asan error causd by test 00144

* Fix empty log message (#CLICKHOUSE-3378)
 2017-10-25 21:39:10 +03:00
Vitaliy Lyudvichenko
62ea1133dd Disable part sendings and fetches before ALTER. [#CLICKHOUSE-3343] 2017-10-12 23:26:54 +03:00
Alexey Zatelepin
2f80c150db save MergeTree data format version and parse part names depending on it [#CLICKHOUSE-3000] 2017-09-26 20:40:02 +03:00
proller
5054d50307 Remove too noisy message (#CLICKHOUSE-3304) 2017-09-11 21:26:43 +03:00
Alexey Zatelepin
f25f0cd759 reduce number of places where partitioning by month is assumed [#CLICKHOUSE-3000] 2017-08-15 22:30:52 +03:00
Vitaliy Lyudvichenko
10c14cfccb Small code enhancements according to clang-tidy. [#CLICKHOUSE-2931] 2017-08-04 17:35:36 +03:00
Vitaliy Lyudvichenko
eff3588f28 Rename DROP COLUMN FROM PARTITION -> CLEAR COLUMN IN PARTITION. [#CLICKHOUSE-2896] 2017-06-22 16:18:19 +03:00
Vitaliy Lyudvichenko
53adb891b3 Add replicated DROP COLUMN FROM partition for optimistic case. [#CLICKHOUSE-2896] 2017-06-22 00:56:38 +03:00
Vitaliy Lyudvichenko
1d00f45dd3 Detect race condition between fetches and merges of the same part. [#CLICKHOUSE-2989] 2017-05-17 21:44:16 +04:00
Alexey Zatelepin
ba254a082e Merge branch 'master' into lagging-replicas-failover 
Conflicts:
	dbms/src/Core/ErrorCodes.cpp
 2017-04-19 21:31:29 +03:00
Alexey Milovidov
0b5645db1a Better code for parsing part names [#CLICKHOUSE-2]. 2017-04-18 23:38:07 +03:00
Alexey Milovidov
502f42a572 Fixed translation error [#CLICKHOUSE-3]. 2017-04-18 22:13:12 +03:00
Alexey Zatelepin
2bd089d589 start counting absolute replica delay when there are problems with queue update [#CLICKHOUSE-2141] 2017-04-17 18:06:12 +03:00
Alexey Milovidov
310736cd7e Moved headers and sources to same place [#CLICKHOUSE-3]. 2017-04-01 12:22:42 +03:00
Alexey Milovidov
137ad95929 Changed tabs to spaces in code [#CLICKHOUSE-3]. 2017-04-01 11:35:09 +03:00
Alexey Milovidov
f9b4b2012f Removed zero-width spaces [#CLICKHOUSE-2]. 2017-03-13 21:02:45 +03:00
Alexey Milovidov
26788e3938 Formatting [#CLICKHOUSE-2]. 2017-03-13 21:01:46 +03:00
f1yegor
401425bb89 translate comments 2017-03-12 20:18:07 +01:00
Alexey Milovidov
31c4636538 Using internal boost library [#METR-2944]. 2016-12-12 07:59:19 +03:00
Alexey Milovidov
d605bfdc9d Using internal boost library [#METR-2944]. 2016-12-12 07:57:09 +03:00
Alexey Milovidov
573a84c89a MergeSelector: development [#METR-21841]. 2016-10-31 22:45:22 +03:00
Alexey Milovidov
bee00bfcfc MergeSelector: development [#METR-21841]. 2016-10-30 14:05:45 +03:00
Alexey Milovidov
74c3b30622 dbms: Simplified logic of selecting merges [#METR-21841]. 2016-10-28 01:50:02 +03:00
Alexey Milovidov
2dd43be6e6 Don't hold shared_ptr to zookeeper for long time, because it makes very difficult to remove ephemeral nodes in case of temporary communication errors and thus breaks leader election [#METR-23272]. 2016-10-24 15:34:08 +03:00
Alexey Milovidov
55ed2904f4 Allowed for certain aggregate functions to have case insensitive names for compatibility [#METR-22087]. 2016-07-14 08:22:09 +03:00
Alexey Milovidov
cff1004f04 Lowered batch size in pullLogsToQueue [#METR-21013]. 2016-04-21 00:24:12 +03:00
Alexey Milovidov
48a1fe182d dbms: attempt to fix an error [#METR-19866]. 2016-02-03 00:30:27 +03:00
Alexey Milovidov
a08281d2eb dbms: fixed error when replica has too large delay [#METR-19586]. 2016-01-21 22:43:00 +03:00
Alexey Milovidov
82a96d22b1 dbms: tracking replicas delays [#METR-17573]. 2016-01-17 16:00:42 +03:00
Alexey Milovidov
a30532594b dbms: fixed error [#METR-19586]. 2016-01-17 08:22:22 +03:00
Alexey Milovidov
d8e6b1dcf1 dbms: fixed error in prev. revision [#METR-19586]. 2016-01-13 05:36:11 +03:00
Alexey Milovidov
949ecc2070 dbms: optimized function pullLogsToQueue [#METR-19586]. 2016-01-12 22:45:51 +03:00
Alexey Milovidov
1f7f7cd682 dbms: better [#METR-19586]. 2016-01-10 07:44:12 +03:00