Anton Ivashkin
0f9038ebed
Zero-copy: move shared mark outside table node in ZooKeeper
2021-11-29 19:05:31 +03:00
Alexander Tokmakov
1ab8b5d4df
fix segfault on attach parititon
2021-11-25 13:04:39 +03:00
Anton Popov
ccd78e3838
Merge remote-tracking branch 'upstream/master' into HEAD
2021-11-22 17:19:35 +03:00
Nikolai Kochetov
b814f54f50
Merge pull request #31474 from amosbird/projection-improve6
...
Allow partition column to be used for minmax_count projection
2021-11-22 13:01:57 +03:00
Amos Bird
c47ec31617
fix
2021-11-19 11:51:05 +08:00
Alexander Tokmakov
0a4647f927
support alter partition
2021-11-17 21:14:14 +03:00
Alexander Tokmakov
672157b817
Merge branch 'master' into mvcc_prototype
2021-11-11 22:01:34 +03:00
Anton Popov
a20922b2d3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-11-09 15:36:25 +03:00
Alexander Tokmakov
51bd98294e
better merge selecting backoff
2021-11-09 15:26:51 +03:00
Alexander Tokmakov
92eec74ad7
Merge branch 'master' into mvcc_prototype
2021-11-06 21:08:36 +03:00
Vitaly Baranov
a1f29d31ea
Support BACKUP & RESTORE for log family.
2021-11-01 12:07:17 +03:00
Alexander Tokmakov
2e7e195e77
change alter_lock to std::timed_mutex
2021-10-26 13:37:00 +03:00
Nikita Mikhaylov
75487be899
White list of storages that supports final
2021-10-19 12:16:33 +00:00
Nikita Mikhaylov
4fbd332bf1
Done
2021-10-19 11:53:55 +00:00
Nikolai Kochetov
2a95d7fc87
Merge pull request #29918 from amosbird/projection-improve5
...
Add primary key to minmax_count_projection
2021-10-17 14:04:18 +03:00
alexey-milovidov
4f11cfa59d
Merge pull request #28760 from azat/mutator-forbid-cleaner
...
Forbid cleaning of tmp directories that can be used by an active mutation/merge.
2021-10-17 03:18:01 +03:00
Amos Bird
77c4a5fa18
Better
2021-10-16 18:13:41 +08:00
Amos Bird
77a2022cf5
Add primary key to minmax_count_projection
2021-10-16 18:13:41 +08:00
Azat Khuzhin
07e8b2b3c7
Do not try to remove temporary paths that is currently in written by merge/mutation
...
v2: rebase against MergeTask
v3: rebase due to conflicts in src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
v4:
- rebase due to conflicts in src/Storages/MergeTree/MergeTask.cpp
- drop common/scope_guard_safe.h (not used)
2021-10-16 00:43:52 +03:00
Amos Bird
23602f4607
Speed up part loading for JBOD
2021-10-13 12:33:54 +08:00
Nikolai Kochetov
c36bb9206e
Merge pull request #29879 from amosbird/queryidholderfix
...
Fix potential leak of query_id_holder
2021-10-12 15:42:02 +03:00
Maksim Kita
2d069acc22
System table data skipping indices added size
2021-10-11 11:39:50 +03:00
Amos Bird
ebfb013ea1
Fix potential leak of query_id_holder
2021-10-11 15:56:00 +08:00
Alexander Tokmakov
72b1b2e360
Merge branch 'master' into mvcc_prototype
2021-09-23 22:53:27 +03:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask ( #25165 )
...
Introduced MergeTask and MutateTask
2021-09-17 00:19:58 +03:00
Anton Popov
4c388e3d84
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-09-09 14:10:16 +03:00
Nikita Mikhaylov
6062dd0021
Better
2021-09-08 00:21:21 +00:00
Nikita Mikhaylov
ea0fbf81af
Renaming
2021-09-06 12:01:16 +00:00
Nikita Mikhaylov
292a24abe8
Merge upstream/master into global-merge-executor (using imerge)
2021-09-03 00:34:24 +00:00
Nikita Mikhaylov
cc7c221fad
Own PriorityQueue + prettifying the code
2021-09-02 21:31:32 +00:00
alesapin
b592400e54
Merge pull request #28404 from ClickHouse/remove_outdated_settings
...
Remove obsolete settings for replicated fetches.
2021-09-02 14:03:12 +03:00
Nikolai Kochetov
5f0b1bc4e3
Merge pull request #26286 from amosbird/projection-improve2
...
Virtual projection for min max indices.
2021-09-02 13:03:30 +03:00
mergify[bot]
5d299fbdee
Merge branch 'master' into remove_outdated_settings
2021-09-01 14:07:48 +00:00
alesapin
921e51e061
Remove some obsolete settings for replicated fetches
2021-08-31 15:22:56 +03:00
Nikita Mikhaylov
f8d4f04294
Merge upstream/master into global-merge-executor (using imerge)
2021-08-31 11:52:11 +00:00
Nikita Mikhaylov
c4416906c8
done
2021-08-30 19:37:03 +00:00
Amos Bird
b68857d086
Simplify projection, add minmax_count projection.
2021-08-28 11:25:37 +08:00
Alexey Milovidov
79e0433ba7
Merge branch 'master' of github.com:yandex/ClickHouse into async-reads
2021-08-28 01:19:16 +03:00
Amos Bird
0169fce78e
Projection bug fixes and refactoring.
2021-08-26 19:09:31 +08:00
Alexey Milovidov
fd27a2faef
Remove imperfect code
2021-08-25 02:05:55 +03:00
Alexey Milovidov
7c1d0a3baf
Progress on development
2021-08-25 01:24:47 +03:00
Vitaly Baranov
63e8bc1f20
Added new commands BACKUP and RESTORE.
2021-08-23 23:41:45 +03:00
Alexander Tokmakov
c74bfbf991
Merge branch 'master' into mvcc_prototype
2021-07-28 22:21:48 +03:00
Anton Popov
2b58f39c10
dynamic columns: support missed columns in distributed
2021-07-23 19:34:23 +03:00
Anton Popov
f99374cca6
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-07-20 18:20:21 +03:00
Zhichang Yu
5047c758f4
fix per review
2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1
hdfs zero copy
2021-07-13 07:19:12 +00:00
Alexey Milovidov
1ce40d6629
Fix bad code (default function argument)
2021-07-11 04:01:23 +03:00
Anton Popov
3ed7f5a6cc
dynamic subcolumns: add snapshot for storage
2021-07-09 06:15:41 +03:00
Anton Popov
072e65b728
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-07-07 17:20:38 +03:00
Anton Popov
9071ecd428
fix alter of settings in MergeTree
2021-07-05 15:44:58 +03:00
Nikita Mikhaylov
c66a3b22b5
done
2021-06-22 23:24:47 +00:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] ( #13652 )
...
* initial commit: add setting and stub
* typo
* added test stub
* fix
* wip merging new integration test and code proto
* adding steps interpreters
* adding firstly proposed solution (moving parts etc)
* added checking zookeeper path existence
* fixing the include
* fixing and sorting includes
* fixing outdated struct
* fix the name
* added ast ptr as level of indirection
* fix ref
* updating the changes
* working on test stub
* fix iterator -> reference
* revert rocksdb submodule update
* fixed show privileges test
* updated the test stub
* replaced rand() with thread_local_rng(), updated the tests
updated the test
fixed test config path
test fix
removed error messages
fixed the test
updated the test
fixed string literal
fixed literal
typo: =
* fixed the empty replica error message
* updated the test and the code with logs
* updated the possible test cases, updated
* added the code/test milestone comments
* updated the test (added more testcases)
* replaced native assert with CH one
* individual replicas recursive delete fix
* updated the AS db.name AST
* two small logging fixes
* manually generated AST fixes
* Updated the test, added the possible algo change
* Some thoughts about optimizing the solution:
ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH
* fix
* Removed the replica sync in test as it's invalid
* Some test tweaks
* tmp
* Rewrote the algo by using the executeQuery instead of
hand-crafting the ASTPtr.
Two questions still active.
* tr: logging active parts
* Extracted the parts moving algo into a separate helper function
* Fixed the test data and the queries slightly
* Replaced query to system.parts to direct invocation,
started building the test that breaks on various parts.
* Added the case for tables when at least one replica is alive
* Updated the test to test replicas restoration by detaching/attaching
* Altered the test to check restoration without replica restart
* Added the tables swap in the start if the server failed last time
* Hotfix when only /replicas/replica... path was deleted
* Restore ZK paths while creating a replicated MergeTree table
* Updated the docs, fixed the algo for individual replicas restoration case
* Initial parts table storage fix, tests sync fix
* Reverted individual replica restoration to general algo
* Slightly optimised getDataParts
* Trying another solution with parts detaching
* Rewrote algo without any steps, added ON CLUSTER support
* Attaching parts from other replica on restoration
* Getting part checksums from ZK
* Removed ON CLUSTER, finished working solution
* Multiple small changes after review
* Fixing parallel test
* Supporting rewritten form on cluster
* Test fix
* Moar logging
* Using source replica as checksum provider
* improve test, remove some code from parser
* Trying solution with move to detached + forget
* Moving all parts (not only Committed) to detached
* Edited docs for RESTORE REPLICA
* Re-merging
* minor fixes
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
2021-06-20 11:24:43 +03:00
Alexander Tokmakov
9a88b9dacc
set maxtid on drop part
2021-06-08 21:17:18 +03:00
Alexander Tokmakov
881889ef22
parts cleanup, fixes
2021-06-07 21:22:58 +03:00
Anton Popov
205a23282b
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-06-07 19:38:22 +03:00
Alexander Tokmakov
9915bd0a8b
Merge branch 'master' into mvcc_prototype
2021-06-03 00:52:57 +03:00
alexey-milovidov
c7bd3b8674
Merge pull request #23315 from CurtizJ/empty-parts-fix
...
Fix waiting while dropping empty parts
2021-06-01 13:52:49 +03:00
Anton Popov
9f52362b81
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-05-31 23:08:45 +03:00
Alexander Tokmakov
5692db736c
Merge branch 'master' into mvcc_prototype
2021-05-31 21:11:22 +03:00
Nikolai Kochetov
afc1fe7f3d
Make ContextPtr const by default.
2021-05-31 17:49:02 +03:00
Anton Popov
24707e6604
dynamic subcolumns: better handling of incompatible types
2021-05-29 04:19:20 +03:00
Anton Popov
336abfa475
Merge remote-tracking branch 'upstream/master' into HEAD
2021-05-28 01:58:30 +03:00
Anton Popov
0bdf9d207c
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-05-26 00:25:47 +03:00
Alexander Tokmakov
f2ab5a05c6
Merge branch 'master' into mvcc_prototype
2021-05-18 13:36:38 +03:00
Alexander Tokmakov
1fd6142c1a
try enable merges
2021-05-18 13:35:48 +03:00
Anton Popov
e27acc26be
better dropPart and dropPartition
2021-05-17 17:26:36 +03:00
Anton Popov
e043cc45bf
Merge remote-tracking branch 'upstream/master' into HEAD
2021-05-17 16:11:12 +03:00
alesapin
17f229857c
Merge branch 'master' into nvartolomei-parts-move
2021-05-17 13:52:48 +03:00
Alexander Tokmakov
970ad94851
Merge branch 'master' into fix_intersecting_parts
2021-05-14 13:17:02 +03:00
Alexander Tokmakov
df2849be90
Merge branch 'master' into mvcc_prototype
2021-05-13 21:48:36 +03:00
Alexander Tokmakov
2da12f0b59
fix intersection of drop range and merge
2021-05-13 17:04:36 +03:00
alesapin
46e136b5c4
Merge branch 'master' into nv/parts-uuid-move-shard
2021-05-11 15:36:40 +03:00
Amos Bird
9c069ebdbf
support prewhere, row_filter, read_in_order and decent projection selection
...
TODO set index analysis in projection
2021-05-11 18:12:27 +08:00
Amos Bird
35961c0c5d
Use ActionsDAG to rewrite projection queries
2021-05-11 18:12:27 +08:00
Nikolai Kochetov
3296c9292f
Try to merge projectons faster.
2021-05-11 18:12:26 +08:00
Amos Bird
264cff6415
Projections
...
TODO (suggested by Nikolai)
1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies
3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)
Also need to figure out how prewhere works for projections, and
row_filter_policies.
wip
2021-05-11 18:12:23 +08:00
Anton Popov
e44706911e
dynamic columns: better getting of sample block
2021-05-05 02:02:54 +03:00
Anton Popov
3bc2a0820c
dynamic subcolumns: support merges
2021-05-03 03:56:19 +03:00
Nicolae Vartolomei
1fa5871ff7
Fix bad rebase and introduce part_moves_between_shards_enable setting
2021-04-27 14:20:13 +01:00
Nicolae Vartolomei
53d57ffb52
Part movement between shards
...
Integrate query deduplication from #17348
2021-04-27 14:20:12 +01:00
Amos Bird
8a3b5c1fab
Add _partition_value virtual column
2021-04-27 16:15:59 +08:00
Anton Popov
644df6be7d
dynamic subcolumns: wip
2021-04-24 07:09:01 +03:00
Anton Popov
42a0416a2b
better drop of empty parts
2021-04-20 05:31:08 +03:00
songenjie
564136ec46
[clickhouse][server][dll][alter]support fetch part
2021-04-13 12:40:33 +08:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Alexander Tokmakov
3422bd1742
check parts visibility for select
2021-04-08 20:20:45 +03:00
alesapin
0204f5dd35
Merge branch 'master' into merge_tree_deduplication
2021-04-03 15:24:26 +03:00
alesapin
759dd79c76
Add tests
2021-04-02 19:45:18 +03:00
Mike Kot
c947280dfc
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-04-01 21:38:51 +03:00
Alexey Milovidov
230b2930b7
Merge branch 'master' into virtualutil
2021-03-30 00:43:00 +03:00
Mike Kot
285af08949
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-03-24 22:34:20 +03:00
alesapin
36898bdc4a
Merge pull request #21142 from Jokser/unfreeze-partitions
...
Add ALTER TABLE UNFREEZE command.
2021-03-23 14:05:38 +03:00
Nikolai Kochetov
c3c393a7aa
Merge branch 'master' into refactor-actions-dag
2021-03-18 14:33:07 +03:00
Mike Kot
e309dd275a
Trying to investigate the errors after attaching the part.
...
+ A new entry is added to the part_log.
2021-03-17 18:49:04 +03:00
Pavel Kovalenko
5627ae9067
Make MergeTreeData::getPartitionMatcher const
2021-03-17 12:12:55 +03:00
Pavel Kovalenko
39f3b17041
Merge remote-tracking branch 'origin/master' into unfreeze-partitions
2021-03-16 10:56:24 +03:00
alexey-milovidov
378e10f929
Merge pull request #21351 from fuwhu/refine-load-data-parts
...
Refine MergeTreeData::loadDataParts to not parse format version file and detached directory
2021-03-15 22:38:30 +03:00
Anton Ivashkin
ace628d565
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication
2021-03-12 12:58:32 +03:00
fuwhu
69f516cd47
Refine MergeTreeData::loadDataParts to not parse format version file and detached directory
2021-03-10 10:10:25 +08:00
Amos Bird
2ec20c5d23
update and add tests
2021-03-08 17:38:07 +08:00
Amos Bird
e6522e1ebe
JBOD data balancer
2021-03-08 11:10:35 +08:00
alesapin
5b3161e0b5
Get rid of const_cast
2021-03-05 20:24:06 +03:00
Anton Ivashkin
d08b481660
Fixes by review responces
2021-03-05 19:20:38 +03:00
Nikolai Kochetov
a669f7d641
Merge branch 'master' into refactor-actions-dag
2021-03-05 18:21:14 +03:00
Amos Bird
93b661ad5a
partition id pruning
2021-03-04 19:43:03 +08:00
Anton Ivashkin
e69124a0a6
Merge master
2021-03-04 13:26:40 +03:00
Pavel Kovalenko
182ba7b5b4
Fixed argument name.
2021-03-03 10:35:31 +03:00
alesapin
6b1005aea5
Merge pull request #21334 from ClickHouse/fix_alter_partition_key
...
Fix alter modify query for partition key and other metadata fields
2021-03-03 10:15:11 +03:00
Pavel Kovalenko
a3c0c35dff
ALTER TABLE UNFREEZE rework.
2021-03-02 23:28:42 +03:00
alesapin
c29d7c7f49
Shutup clang tidy
2021-03-02 19:13:36 +03:00
alesapin
4e33587043
Comments
2021-03-02 13:57:09 +03:00
alesapin
9ebf1b4fad
Get rid of separate minmax index fields
2021-03-02 13:33:54 +03:00
Anton Ivashkin
3c11d44494
Add description for getUniqueId method, fix typos
2021-03-01 13:31:36 +03:00
feng lv
51021c1164
forbid to drop a column if it's referenced by materialized view
2021-02-28 05:24:39 +00:00
Anton Ivashkin
5b267b7eec
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication
2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557
Fixes by review response
2021-02-26 12:48:57 +03:00
alesapin
653d962cdd
Fix style and add low-level check
2021-02-25 17:43:58 +03:00
alesapin
5c6c318737
Restrict mutations for engines which doesn't support them
2021-02-25 13:07:48 +03:00
Pavel Kovalenko
078dfce038
Add ALTER TABLE UNFREEZE command.
2021-02-24 17:28:30 +03:00
alexey-milovidov
7b55eb8c58
Merge pull request #20178 from amosbird/delayinactive
...
Delay or throw insertion when too many inactive parts
2021-02-11 20:52:04 +03:00
Amos Bird
afb5846a24
refactor
2021-02-10 11:22:24 +08:00
Nicolae Vartolomei
c336cac306
Remove dead code
2021-02-08 17:36:58 +00:00
Nicolae Vartolomei
a8f37e7704
Remove dead code
2021-02-08 17:19:35 +00:00
Amos Bird
ab5c7b75a4
Delay or throw insertion when too many inactive parts
2021-02-07 19:03:55 +08:00
Anton Ivashkin
e64c63c611
Merge master
2021-02-05 20:10:06 +03:00
Amos Bird
66fe97d8bd
Per MergeTree table query limit
2021-01-26 14:03:31 +08:00
Anton Ivashkin
357d98eb36
Merge master
2021-01-20 12:23:03 +03:00
Anton Ivashkin
eba98b04b0
Zero copy replication over S3: Hybrid storage support
2021-01-18 19:16:45 +03:00
Alexey Milovidov
24c8e53440
Merge branch 'master' into multiple-nested
2021-01-16 16:28:40 +03:00
Alexey Milovidov
6a2a5e53ed
Slightly better code of IMergeTreeDataPart #18955
2021-01-15 15:15:13 +03:00
Anton Popov
36ae0e4d35
Merge remote-tracking branch 'upstream/master' into HEAD
2021-01-11 13:51:12 +03:00
Azat Khuzhin
64cb6405ac
Drop IStorage::supportsSettings() (replaced with StorageFeatures::supports_settings)
2021-01-08 14:36:26 +03:00
Anton Popov
1be39fddac
fix subcolumns with some storages
2020-12-22 19:42:37 +03:00
flynn
4e580f7b7c
Add asynchronous metrics on MergeTree tables data volume ( #17639 )
...
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-12-22 13:34:35 +03:00
Alexey Milovidov
9a017b77df
Merge branch 'master' into relax-too-many-parts-1
2020-12-17 04:40:39 +03:00
Anton Ivashkin
0f0500ca0c
Merge master
2020-12-16 18:31:13 +03:00
Alexander Kuzmenkov
d757f32300
Remove some redundant includes to speed up build
2020-12-11 02:58:11 +03:00
Alexey Milovidov
2e6bedce91
Relax "Too many parts" threshold, part 1
2020-11-29 18:08:02 +03:00
feng lv
7e3524caa1
fix spelling errors
2020-11-28 08:17:20 +00:00
Nicolae Vartolomei
040aba9f85
Add uuid.txt to checksums for parts stored on disk
...
We are breaking backwards compatibility anyway (but agted by a setting)
2020-11-20 13:49:17 +00:00
Anton Popov
4cecd89e84
fix dropping of empty parts
2020-11-12 20:37:54 +03:00
Anton Popov
0aa35d53c7
Merge remote-tracking branch 'upstream/master' into HEAD
2020-11-12 14:31:58 +03:00
Anton Popov
0a6d93d929
Update MergeTreeData.h
2020-11-11 21:17:41 +03:00
Anton Popov
2497598a49
Remove empty parts after they were pruned by TTL
2020-11-11 19:19:43 +03:00
Anton Popov
c01870076d
Update MergeTreeData.h
2020-11-11 19:09:01 +03:00
Anton Popov
6f0dc08ba7
avoid code duplication in alterPartition
2020-11-11 16:34:07 +03:00
alesapin
72046313d4
Merge branch 'master' into nvartolomei-drop-part
2020-11-09 10:02:02 +03:00
alexey-milovidov
ee3e289ec2
Merge pull request #15983 from ClickHouse/no_background_pool_no_more
...
Replace BackgroundProcessingPool with SchedulePool task and ThreadPool.
2020-11-05 21:38:22 +03:00
alesapin
20043e77bb
Merge branch 'master' into nvartolomei-drop-part
2020-11-04 10:22:23 +03:00
