alexey-milovidov
f6684dbc62
Merge pull request #32304 from devcrafter/deduplication_token_7461
...
insert_deduplication_token setting for INSERT statement
2022-01-28 13:03:55 +03:00
zhongyuankai
a6254516e0
Fix Alter ttl modification unsupported table engine
2022-01-24 21:48:52 +08:00
taiyang-li
73def8b483
merge master and solve conflict
2022-01-24 11:01:43 +08:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
Alexander Tokmakov
0ea0a8ccc7
Merge branch 'master' into mvcc_prototype
2022-01-20 17:05:55 +03:00
alexey-milovidov
994584df09
Merge pull request #33753 from azat/mt-fix-count-race
...
Fix tiny race between count() and INSERT/merges/... in MergeTree
2022-01-20 06:33:54 +03:00
Alexander Tokmakov
e9a5a64a71
Merge branch 'master' into mvcc_prototype
2022-01-19 21:41:23 +03:00
alexey-milovidov
d222cb9771
Merge pull request #33695 from amosbird/buildimprove
...
Some build improvement
2022-01-19 03:19:22 +03:00
Azat Khuzhin
c9e93b21c2
Fix tiny race between count() and INSERT/merges/... in MergeTree
...
Before it was possible to return stale counter from
StorageMergeTree::totalRows() (that is used for
optimize_trivial_count_query) since the accounting is done in two steps:
- subtract old number of rows
<-- here the number can be zero, even though there are rows -->
- add new number of rows
This was found by CI [1] in 01615_random_one_shard_insertion test:
Here you can see that INSERT went to both tables:
<details>
2022.01.16 09:07:34.288252 [ 154369 ] {a1905be0-93da-460c-8c6f-9b5adace72a0} <Debug> DistributedBlockOutputStream: It took 0.035197041 sec. to insert 100
blocks, 2841.1479249065287 rows per second. Insertion status:
Wrote 54 blocks and 54 rows on shard 0 replica 0, localhost:9000 (average 0 ms per block, the slowest block 1 ms)
Wrote 46 blocks and 46 rows on shard 1 replica 0, localhost:9000 (average 0 ms per block, the slowest block 1 ms)
</details>
But the test fails, since select from shard1.tbl returns 0, and the
problem was concurrent merge:
<details>
2022.01.16 09:07:34.289470 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> executeQuery: (from [::1]:42082) (comment: 01615_random_one_shard_insertion.sql) select count() != 0 from shard_0.tbl;
2022.01.16 09:07:34.289564 [ 375 ] {c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab::all_1_54_9} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 54 rows, containing 1 columns (1 merged, 0 gathered) in 0.00171193 sec., 31543.345814373253 rows/sec., 246.43 KiB>
2022.01.16 09:07:34.289810 [ 375 ] {c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab::all_1_54_9} <Trace> shard_0.tbl (c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab): Renaming temporary part tmp_merge_all_1_54_9 to all_1_54_9.
2022.01.16 09:07:34.289858 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> ContextAccess (default): Access granted: SELECT(number) ON shard_0.tbl
2022.01.16 09:07:34.289897 [ 375 ] {c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab::all_1_54_9} <Trace> shard_0.tbl (c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab) (MergerMutator): Merged 6 parts: from all_1_49_8 to all_54_54_0
2022.01.16 09:07:34.289920 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> InterpreterSelectQuery: WithMergeableState -> Complete
2022.01.16 09:07:34.289987 [ 375 ] {} <Debug> MemoryTracker: Peak memory usage Mutate/Merge: 3.12 MiB.
2022.01.16 09:07:34.290305 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> MergingAggregatedTransform: Reading blocks of partially aggregated data.
2022.01.16 09:07:34.290332 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> MergingAggregatedTransform: Read 1 blocks of partially aggregated data, total 1 rows.
2022.01.16 09:07:34.290343 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> Aggregator: Merging partially aggregated single-level data.
2022.01.16 09:07:34.290358 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> Aggregator: Merged partially aggregated single-level data.
2022.01.16 09:07:34.290366 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> Aggregator: Converting aggregated data to blocks
2022.01.16 09:07:34.290391 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> Aggregator: Converted aggregated data to blocks. 1 rows, 8.00 B in 1.0939e-05 sec. (91416.034 rows/sec., 714.19 KiB/sec.)
2022.01.16 09:07:34.290709 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Information> executeQuery: Read 1 rows, 4.01 KiB in 0.001187722 sec., 841 rows/sec., 3.30 MiB/sec.
2022.01.16 09:07:34.290774 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.
</details>
[1]: https://s3.amazonaws.com/clickhouse-test-reports/33675/7848ea7d609e4c720e8e4494eb6207c0751f5aea/stateless_tests__ubsan__actions_.html
This also fixes a race between DROP TABLE check and INSERT/merges.
v0: use Active parts instead.
v2: fix total counters accounting instead.
2022-01-18 23:58:02 +03:00
Amos Bird
6d62060e16
Build improvement
2022-01-17 22:36:27 +08:00
alesapin
4caef03e6a
Add ephemeral lock in zookeeper
2022-01-17 14:52:51 +03:00
Alexander Tokmakov
bf2b6c2c37
Merge branch 'master' into mvcc_prototype
2022-01-10 19:53:50 +03:00
taiyang-li
2cae8d552c
remove useless code
2022-01-10 19:17:12 +08:00
taiyang-li
1b4b727d79
fix conflict
2022-01-10 10:43:13 +08:00
Igor Nikonov
102b9cdcfb
Merge remote-tracking branch 'origin/master' into deduplication_token_7461
2022-01-09 19:15:33 +01:00
Igor Nikonov
0857a8d061
insert_deduplication_token support in non-replicated MergeTree
2022-01-09 19:06:09 +01:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474
2022-01-07 10:12:25 +00:00
taiyang-li
c0a9c2b916
refactor metadatacache to reduce using of USE_ROCKSDB
2022-01-05 19:51:50 +08:00
taiyang-li
3f6d830536
Merge remote-tracking branch 'origin/master' into rocksdb_metacache
2022-01-04 10:00:34 +08:00
Alexander Tokmakov
187c43eba8
rename Committed state to Active
2021-12-30 23:45:38 +03:00
taiyang-li
98f37afc80
fix building
2021-12-28 19:29:01 +08:00
Anton Ivashkin
0465aef15d
Fixes by code review responces
2021-12-27 19:27:06 +03:00
Anton Ivashkin
e88b97dafb
Fix typos
2021-12-21 19:56:29 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474
2021-12-21 17:38:36 +03:00
Anton Ivashkin
33cbfc89ef
Move logic for replicated part to StorageReplicatedMergeTree class
2021-12-21 17:29:50 +03:00
Anton Ivashkin
0c0bf66334
Merge master
2021-12-21 17:27:54 +03:00
Anton Ivashkin
f0b9a4327a
Use table UUID in zero-copy shared label in ZooKeeper
2021-12-20 16:19:00 +03:00
liyang
37ba8004ff
Speep up mergetree starting up process
2021-12-18 16:39:59 +08:00
Anton Popov
99ebabd822
Merge remote-tracking branch 'upstream/master' into HEAD
2021-12-17 19:02:29 +03:00
Alexander Tokmakov
32e62ed5c2
Merge branch 'master' into mvcc_prototype
2021-12-17 10:43:46 +03:00
Alexander Tokmakov
e185ad260b
fix skipping of some mutations
2021-12-15 21:28:24 +03:00
Alexander Tokmakov
1b02136941
fix skipping of some mutations
2021-12-15 21:19:29 +03:00
Alexander Tokmakov
6e1c16c2e7
add support for mutations
2021-12-14 23:06:34 +03:00
nautaa
0da3881ff4
flush all InMemoryDataParts when wal is not enabled
2021-12-14 16:31:17 +08:00
Alexander Tokmakov
7fcb79ae72
Merge branch 'master' into mvcc_prototype
2021-12-07 14:39:29 +03:00
Anton Ivashkin
98bae1b313
Fix tests
2021-12-02 16:25:45 +03:00
Anton Ivashkin
e0a16a483e
Merge master
2021-12-01 19:03:31 +03:00
Anton Ivashkin
80ab73c691
Fix Zero-Copy replication lost locks, fix remove used remote data in DROP DETACHED PART
2021-12-01 16:11:26 +03:00
Anton Popov
6f4d9a53b2
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-12-01 15:54:33 +03:00
Alexander Tokmakov
2cf4b51029
fix
2021-11-29 21:54:46 +03:00
Anton Ivashkin
0f9038ebed
Zero-copy: move shared mark outside table node in ZooKeeper
2021-11-29 19:05:31 +03:00
Alexander Tokmakov
1ab8b5d4df
fix segfault on attach parititon
2021-11-25 13:04:39 +03:00
Anton Popov
ccd78e3838
Merge remote-tracking branch 'upstream/master' into HEAD
2021-11-22 17:19:35 +03:00
Nikolai Kochetov
b814f54f50
Merge pull request #31474 from amosbird/projection-improve6
...
Allow partition column to be used for minmax_count projection
2021-11-22 13:01:57 +03:00
Amos Bird
c47ec31617
fix
2021-11-19 11:51:05 +08:00
Alexander Tokmakov
0a4647f927
support alter partition
2021-11-17 21:14:14 +03:00
Alexander Tokmakov
672157b817
Merge branch 'master' into mvcc_prototype
2021-11-11 22:01:34 +03:00
Anton Popov
a20922b2d3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-11-09 15:36:25 +03:00
Alexander Tokmakov
51bd98294e
better merge selecting backoff
2021-11-09 15:26:51 +03:00
Alexander Tokmakov
92eec74ad7
Merge branch 'master' into mvcc_prototype
2021-11-06 21:08:36 +03:00
Vitaly Baranov
a1f29d31ea
Support BACKUP & RESTORE for log family.
2021-11-01 12:07:17 +03:00
Alexander Tokmakov
2e7e195e77
change alter_lock to std::timed_mutex
2021-10-26 13:37:00 +03:00
Nikita Mikhaylov
75487be899
White list of storages that supports final
2021-10-19 12:16:33 +00:00
Nikita Mikhaylov
4fbd332bf1
Done
2021-10-19 11:53:55 +00:00
Nikolai Kochetov
2a95d7fc87
Merge pull request #29918 from amosbird/projection-improve5
...
Add primary key to minmax_count_projection
2021-10-17 14:04:18 +03:00
alexey-milovidov
4f11cfa59d
Merge pull request #28760 from azat/mutator-forbid-cleaner
...
Forbid cleaning of tmp directories that can be used by an active mutation/merge.
2021-10-17 03:18:01 +03:00
Amos Bird
77c4a5fa18
Better
2021-10-16 18:13:41 +08:00
Amos Bird
77a2022cf5
Add primary key to minmax_count_projection
2021-10-16 18:13:41 +08:00
Azat Khuzhin
07e8b2b3c7
Do not try to remove temporary paths that is currently in written by merge/mutation
...
v2: rebase against MergeTask
v3: rebase due to conflicts in src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
v4:
- rebase due to conflicts in src/Storages/MergeTree/MergeTask.cpp
- drop common/scope_guard_safe.h (not used)
2021-10-16 00:43:52 +03:00
Amos Bird
23602f4607
Speed up part loading for JBOD
2021-10-13 12:33:54 +08:00
Nikolai Kochetov
c36bb9206e
Merge pull request #29879 from amosbird/queryidholderfix
...
Fix potential leak of query_id_holder
2021-10-12 15:42:02 +03:00
Maksim Kita
2d069acc22
System table data skipping indices added size
2021-10-11 11:39:50 +03:00
Amos Bird
ebfb013ea1
Fix potential leak of query_id_holder
2021-10-11 15:56:00 +08:00
Alexander Tokmakov
72b1b2e360
Merge branch 'master' into mvcc_prototype
2021-09-23 22:53:27 +03:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask ( #25165 )
...
Introduced MergeTask and MutateTask
2021-09-17 00:19:58 +03:00
Anton Popov
4c388e3d84
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-09-09 14:10:16 +03:00
Nikita Mikhaylov
6062dd0021
Better
2021-09-08 00:21:21 +00:00
Nikita Mikhaylov
ea0fbf81af
Renaming
2021-09-06 12:01:16 +00:00
Nikita Mikhaylov
292a24abe8
Merge upstream/master into global-merge-executor (using imerge)
2021-09-03 00:34:24 +00:00
Nikita Mikhaylov
cc7c221fad
Own PriorityQueue + prettifying the code
2021-09-02 21:31:32 +00:00
alesapin
b592400e54
Merge pull request #28404 from ClickHouse/remove_outdated_settings
...
Remove obsolete settings for replicated fetches.
2021-09-02 14:03:12 +03:00
Nikolai Kochetov
5f0b1bc4e3
Merge pull request #26286 from amosbird/projection-improve2
...
Virtual projection for min max indices.
2021-09-02 13:03:30 +03:00
mergify[bot]
5d299fbdee
Merge branch 'master' into remove_outdated_settings
2021-09-01 14:07:48 +00:00
alesapin
921e51e061
Remove some obsolete settings for replicated fetches
2021-08-31 15:22:56 +03:00
Nikita Mikhaylov
f8d4f04294
Merge upstream/master into global-merge-executor (using imerge)
2021-08-31 11:52:11 +00:00
Nikita Mikhaylov
c4416906c8
done
2021-08-30 19:37:03 +00:00
Amos Bird
b68857d086
Simplify projection, add minmax_count projection.
2021-08-28 11:25:37 +08:00
Alexey Milovidov
79e0433ba7
Merge branch 'master' of github.com:yandex/ClickHouse into async-reads
2021-08-28 01:19:16 +03:00
Amos Bird
0169fce78e
Projection bug fixes and refactoring.
2021-08-26 19:09:31 +08:00
Alexey Milovidov
fd27a2faef
Remove imperfect code
2021-08-25 02:05:55 +03:00
Alexey Milovidov
7c1d0a3baf
Progress on development
2021-08-25 01:24:47 +03:00
Vitaly Baranov
63e8bc1f20
Added new commands BACKUP and RESTORE.
2021-08-23 23:41:45 +03:00
Alexander Tokmakov
c74bfbf991
Merge branch 'master' into mvcc_prototype
2021-07-28 22:21:48 +03:00
Anton Popov
2b58f39c10
dynamic columns: support missed columns in distributed
2021-07-23 19:34:23 +03:00
Anton Popov
f99374cca6
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-07-20 18:20:21 +03:00
Zhichang Yu
5047c758f4
fix per review
2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1
hdfs zero copy
2021-07-13 07:19:12 +00:00
Alexey Milovidov
1ce40d6629
Fix bad code (default function argument)
2021-07-11 04:01:23 +03:00
Anton Popov
3ed7f5a6cc
dynamic subcolumns: add snapshot for storage
2021-07-09 06:15:41 +03:00
Anton Popov
072e65b728
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-07-07 17:20:38 +03:00
Anton Popov
9071ecd428
fix alter of settings in MergeTree
2021-07-05 15:44:58 +03:00
Nikita Mikhaylov
c66a3b22b5
done
2021-06-22 23:24:47 +00:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] ( #13652 )
...
* initial commit: add setting and stub
* typo
* added test stub
* fix
* wip merging new integration test and code proto
* adding steps interpreters
* adding firstly proposed solution (moving parts etc)
* added checking zookeeper path existence
* fixing the include
* fixing and sorting includes
* fixing outdated struct
* fix the name
* added ast ptr as level of indirection
* fix ref
* updating the changes
* working on test stub
* fix iterator -> reference
* revert rocksdb submodule update
* fixed show privileges test
* updated the test stub
* replaced rand() with thread_local_rng(), updated the tests
updated the test
fixed test config path
test fix
removed error messages
fixed the test
updated the test
fixed string literal
fixed literal
typo: =
* fixed the empty replica error message
* updated the test and the code with logs
* updated the possible test cases, updated
* added the code/test milestone comments
* updated the test (added more testcases)
* replaced native assert with CH one
* individual replicas recursive delete fix
* updated the AS db.name AST
* two small logging fixes
* manually generated AST fixes
* Updated the test, added the possible algo change
* Some thoughts about optimizing the solution:
ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH
* fix
* Removed the replica sync in test as it's invalid
* Some test tweaks
* tmp
* Rewrote the algo by using the executeQuery instead of
hand-crafting the ASTPtr.
Two questions still active.
* tr: logging active parts
* Extracted the parts moving algo into a separate helper function
* Fixed the test data and the queries slightly
* Replaced query to system.parts to direct invocation,
started building the test that breaks on various parts.
* Added the case for tables when at least one replica is alive
* Updated the test to test replicas restoration by detaching/attaching
* Altered the test to check restoration without replica restart
* Added the tables swap in the start if the server failed last time
* Hotfix when only /replicas/replica... path was deleted
* Restore ZK paths while creating a replicated MergeTree table
* Updated the docs, fixed the algo for individual replicas restoration case
* Initial parts table storage fix, tests sync fix
* Reverted individual replica restoration to general algo
* Slightly optimised getDataParts
* Trying another solution with parts detaching
* Rewrote algo without any steps, added ON CLUSTER support
* Attaching parts from other replica on restoration
* Getting part checksums from ZK
* Removed ON CLUSTER, finished working solution
* Multiple small changes after review
* Fixing parallel test
* Supporting rewritten form on cluster
* Test fix
* Moar logging
* Using source replica as checksum provider
* improve test, remove some code from parser
* Trying solution with move to detached + forget
* Moving all parts (not only Committed) to detached
* Edited docs for RESTORE REPLICA
* Re-merging
* minor fixes
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
2021-06-20 11:24:43 +03:00
Alexander Tokmakov
9a88b9dacc
set maxtid on drop part
2021-06-08 21:17:18 +03:00
Alexander Tokmakov
881889ef22
parts cleanup, fixes
2021-06-07 21:22:58 +03:00
Anton Popov
205a23282b
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-06-07 19:38:22 +03:00
Alexander Tokmakov
9915bd0a8b
Merge branch 'master' into mvcc_prototype
2021-06-03 00:52:57 +03:00
alexey-milovidov
c7bd3b8674
Merge pull request #23315 from CurtizJ/empty-parts-fix
...
Fix waiting while dropping empty parts
2021-06-01 13:52:49 +03:00
Anton Popov
9f52362b81
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-05-31 23:08:45 +03:00
Alexander Tokmakov
5692db736c
Merge branch 'master' into mvcc_prototype
2021-05-31 21:11:22 +03:00
Nikolai Kochetov
afc1fe7f3d
Make ContextPtr const by default.
2021-05-31 17:49:02 +03:00
Anton Popov
24707e6604
dynamic subcolumns: better handling of incompatible types
2021-05-29 04:19:20 +03:00
Anton Popov
336abfa475
Merge remote-tracking branch 'upstream/master' into HEAD
2021-05-28 01:58:30 +03:00
Anton Popov
0bdf9d207c
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-05-26 00:25:47 +03:00
Alexander Tokmakov
f2ab5a05c6
Merge branch 'master' into mvcc_prototype
2021-05-18 13:36:38 +03:00
Alexander Tokmakov
1fd6142c1a
try enable merges
2021-05-18 13:35:48 +03:00
Anton Popov
e27acc26be
better dropPart and dropPartition
2021-05-17 17:26:36 +03:00
Anton Popov
e043cc45bf
Merge remote-tracking branch 'upstream/master' into HEAD
2021-05-17 16:11:12 +03:00
alesapin
17f229857c
Merge branch 'master' into nvartolomei-parts-move
2021-05-17 13:52:48 +03:00
Alexander Tokmakov
970ad94851
Merge branch 'master' into fix_intersecting_parts
2021-05-14 13:17:02 +03:00
Alexander Tokmakov
df2849be90
Merge branch 'master' into mvcc_prototype
2021-05-13 21:48:36 +03:00
Alexander Tokmakov
2da12f0b59
fix intersection of drop range and merge
2021-05-13 17:04:36 +03:00
alesapin
46e136b5c4
Merge branch 'master' into nv/parts-uuid-move-shard
2021-05-11 15:36:40 +03:00
Amos Bird
9c069ebdbf
support prewhere, row_filter, read_in_order and decent projection selection
...
TODO set index analysis in projection
2021-05-11 18:12:27 +08:00
Amos Bird
35961c0c5d
Use ActionsDAG to rewrite projection queries
2021-05-11 18:12:27 +08:00
Nikolai Kochetov
3296c9292f
Try to merge projectons faster.
2021-05-11 18:12:26 +08:00
Amos Bird
264cff6415
Projections
...
TODO (suggested by Nikolai)
1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies
3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)
Also need to figure out how prewhere works for projections, and
row_filter_policies.
wip
2021-05-11 18:12:23 +08:00
Anton Popov
e44706911e
dynamic columns: better getting of sample block
2021-05-05 02:02:54 +03:00
Anton Popov
3bc2a0820c
dynamic subcolumns: support merges
2021-05-03 03:56:19 +03:00
Nicolae Vartolomei
1fa5871ff7
Fix bad rebase and introduce part_moves_between_shards_enable setting
2021-04-27 14:20:13 +01:00
Nicolae Vartolomei
53d57ffb52
Part movement between shards
...
Integrate query deduplication from #17348
2021-04-27 14:20:12 +01:00
Amos Bird
8a3b5c1fab
Add _partition_value virtual column
2021-04-27 16:15:59 +08:00
Anton Popov
644df6be7d
dynamic subcolumns: wip
2021-04-24 07:09:01 +03:00
Anton Popov
42a0416a2b
better drop of empty parts
2021-04-20 05:31:08 +03:00
songenjie
564136ec46
[clickhouse][server][dll][alter]support fetch part
2021-04-13 12:40:33 +08:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Alexander Tokmakov
3422bd1742
check parts visibility for select
2021-04-08 20:20:45 +03:00
alesapin
0204f5dd35
Merge branch 'master' into merge_tree_deduplication
2021-04-03 15:24:26 +03:00
alesapin
759dd79c76
Add tests
2021-04-02 19:45:18 +03:00
Mike Kot
c947280dfc
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-04-01 21:38:51 +03:00
Alexey Milovidov
230b2930b7
Merge branch 'master' into virtualutil
2021-03-30 00:43:00 +03:00
Mike Kot
285af08949
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-03-24 22:34:20 +03:00
alesapin
36898bdc4a
Merge pull request #21142 from Jokser/unfreeze-partitions
...
Add ALTER TABLE UNFREEZE command.
2021-03-23 14:05:38 +03:00
Nikolai Kochetov
c3c393a7aa
Merge branch 'master' into refactor-actions-dag
2021-03-18 14:33:07 +03:00
Mike Kot
e309dd275a
Trying to investigate the errors after attaching the part.
...
+ A new entry is added to the part_log.
2021-03-17 18:49:04 +03:00
Pavel Kovalenko
5627ae9067
Make MergeTreeData::getPartitionMatcher const
2021-03-17 12:12:55 +03:00
Pavel Kovalenko
39f3b17041
Merge remote-tracking branch 'origin/master' into unfreeze-partitions
2021-03-16 10:56:24 +03:00
alexey-milovidov
378e10f929
Merge pull request #21351 from fuwhu/refine-load-data-parts
...
Refine MergeTreeData::loadDataParts to not parse format version file and detached directory
2021-03-15 22:38:30 +03:00
Anton Ivashkin
ace628d565
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication
2021-03-12 12:58:32 +03:00
fuwhu
69f516cd47
Refine MergeTreeData::loadDataParts to not parse format version file and detached directory
2021-03-10 10:10:25 +08:00
Amos Bird
2ec20c5d23
update and add tests
2021-03-08 17:38:07 +08:00
Amos Bird
e6522e1ebe
JBOD data balancer
2021-03-08 11:10:35 +08:00
alesapin
5b3161e0b5
Get rid of const_cast
2021-03-05 20:24:06 +03:00
Anton Ivashkin
d08b481660
Fixes by review responces
2021-03-05 19:20:38 +03:00
Nikolai Kochetov
a669f7d641
Merge branch 'master' into refactor-actions-dag
2021-03-05 18:21:14 +03:00
Amos Bird
93b661ad5a
partition id pruning
2021-03-04 19:43:03 +08:00
Anton Ivashkin
e69124a0a6
Merge master
2021-03-04 13:26:40 +03:00
Pavel Kovalenko
182ba7b5b4
Fixed argument name.
2021-03-03 10:35:31 +03:00
alesapin
6b1005aea5
Merge pull request #21334 from ClickHouse/fix_alter_partition_key
...
Fix alter modify query for partition key and other metadata fields
2021-03-03 10:15:11 +03:00
Pavel Kovalenko
a3c0c35dff
ALTER TABLE UNFREEZE rework.
2021-03-02 23:28:42 +03:00
alesapin
c29d7c7f49
Shutup clang tidy
2021-03-02 19:13:36 +03:00
alesapin
4e33587043
Comments
2021-03-02 13:57:09 +03:00
alesapin
9ebf1b4fad
Get rid of separate minmax index fields
2021-03-02 13:33:54 +03:00
Anton Ivashkin
3c11d44494
Add description for getUniqueId method, fix typos
2021-03-01 13:31:36 +03:00
feng lv
51021c1164
forbid to drop a column if it's referenced by materialized view
2021-02-28 05:24:39 +00:00
Anton Ivashkin
5b267b7eec
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication
2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557
Fixes by review response
2021-02-26 12:48:57 +03:00
alesapin
653d962cdd
Fix style and add low-level check
2021-02-25 17:43:58 +03:00
alesapin
5c6c318737
Restrict mutations for engines which doesn't support them
2021-02-25 13:07:48 +03:00
Pavel Kovalenko
078dfce038
Add ALTER TABLE UNFREEZE command.
2021-02-24 17:28:30 +03:00
alexey-milovidov
7b55eb8c58
Merge pull request #20178 from amosbird/delayinactive
...
Delay or throw insertion when too many inactive parts
2021-02-11 20:52:04 +03:00
Amos Bird
afb5846a24
refactor
2021-02-10 11:22:24 +08:00
Nicolae Vartolomei
c336cac306
Remove dead code
2021-02-08 17:36:58 +00:00
Nicolae Vartolomei
a8f37e7704
Remove dead code
2021-02-08 17:19:35 +00:00
Amos Bird
ab5c7b75a4
Delay or throw insertion when too many inactive parts
2021-02-07 19:03:55 +08:00
Anton Ivashkin
e64c63c611
Merge master
2021-02-05 20:10:06 +03:00
Amos Bird
66fe97d8bd
Per MergeTree table query limit
2021-01-26 14:03:31 +08:00
Anton Ivashkin
357d98eb36
Merge master
2021-01-20 12:23:03 +03:00
Anton Ivashkin
eba98b04b0
Zero copy replication over S3: Hybrid storage support
2021-01-18 19:16:45 +03:00
Alexey Milovidov
24c8e53440
Merge branch 'master' into multiple-nested
2021-01-16 16:28:40 +03:00
Alexey Milovidov
6a2a5e53ed
Slightly better code of IMergeTreeDataPart #18955
2021-01-15 15:15:13 +03:00
Anton Popov
36ae0e4d35
Merge remote-tracking branch 'upstream/master' into HEAD
2021-01-11 13:51:12 +03:00
Azat Khuzhin
64cb6405ac
Drop IStorage::supportsSettings() (replaced with StorageFeatures::supports_settings)
2021-01-08 14:36:26 +03:00
Anton Popov
1be39fddac
fix subcolumns with some storages
2020-12-22 19:42:37 +03:00
flynn
4e580f7b7c
Add asynchronous metrics on MergeTree tables data volume ( #17639 )
...
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-12-22 13:34:35 +03:00
Alexey Milovidov
9a017b77df
Merge branch 'master' into relax-too-many-parts-1
2020-12-17 04:40:39 +03:00
Anton Ivashkin
0f0500ca0c
Merge master
2020-12-16 18:31:13 +03:00
Alexander Kuzmenkov
d757f32300
Remove some redundant includes to speed up build
2020-12-11 02:58:11 +03:00
Alexey Milovidov
2e6bedce91
Relax "Too many parts" threshold, part 1
2020-11-29 18:08:02 +03:00
feng lv
7e3524caa1
fix spelling errors
2020-11-28 08:17:20 +00:00
Nicolae Vartolomei
040aba9f85
Add uuid.txt to checksums for parts stored on disk
...
We are breaking backwards compatibility anyway (but agted by a setting)
2020-11-20 13:49:17 +00:00
Anton Popov
4cecd89e84
fix dropping of empty parts
2020-11-12 20:37:54 +03:00
Anton Popov
0aa35d53c7
Merge remote-tracking branch 'upstream/master' into HEAD
2020-11-12 14:31:58 +03:00
Anton Popov
0a6d93d929
Update MergeTreeData.h
2020-11-11 21:17:41 +03:00
Anton Popov
2497598a49
Remove empty parts after they were pruned by TTL
2020-11-11 19:19:43 +03:00
Anton Popov
c01870076d
Update MergeTreeData.h
2020-11-11 19:09:01 +03:00
Anton Popov
6f0dc08ba7
avoid code duplication in alterPartition
2020-11-11 16:34:07 +03:00
alesapin
72046313d4
Merge branch 'master' into nvartolomei-drop-part
2020-11-09 10:02:02 +03:00
alexey-milovidov
ee3e289ec2
Merge pull request #15983 from ClickHouse/no_background_pool_no_more
...
Replace BackgroundProcessingPool with SchedulePool task and ThreadPool.
2020-11-05 21:38:22 +03:00
alesapin
20043e77bb
Merge branch 'master' into nvartolomei-drop-part
2020-11-04 10:22:23 +03:00
alexey-milovidov
11fc6fd8c9
Merge pull request #16584 from ClickHouse/drop_table_task_better_scheduling
...
Better scheduling of drop table task
2020-11-03 14:22:51 +03:00
Anton Ivashkin
1742fb3256
Merge master
2020-11-03 12:27:16 +03:00
alesapin
dac6705995
DROP PART and clear blocks in a single zookeeper op
2020-11-02 20:30:59 +03:00
Alexander Tokmakov
ac32809b6a
fix #16482
2020-11-02 19:40:39 +03:00
alesapin
101aa9e931
Merge branch 'master' into no_background_pool_no_more
2020-11-02 10:49:19 +03:00
alesapin
9ed4668dbb
Refactor common part of background list
2020-10-26 15:40:55 +03:00
alesapin
7edbe4d276
Add some comments
2020-10-16 13:12:31 +03:00
Anton Ivashkin
b877459cf7
Zero copy replication over S3: check s3 storage
2020-10-14 18:05:59 +03:00
alesapin
4014e0f08d
Something very similar to working code
2020-10-14 15:32:35 +03:00
alesapin
efd3126b5d
Moving pool
2020-10-14 10:22:48 +03:00
alesapin
fd35368c59
First simple implementation
2020-10-13 17:25:42 +03:00
alesapin
8ec58c17f3
More consistent metadata usage
2020-10-05 19:41:46 +03:00
alesapin
f5c38b34ae
Fix race condition during write ahead log cleanup and rename
2020-09-25 15:07:38 +03:00
alexey-milovidov
1fcebce926
Merge pull request #15000 from Jokser/disable-ttl-move-on-insert
...
Option to disable TTL move on data part insert
2020-09-19 13:45:02 +03:00
Pavel Kovalenko
6b4b5898a7
Renaming.
2020-09-18 18:41:14 +03:00
Pavel Kovalenko
0da19ab46d
Add option to disable ttl move on data part insert
2020-09-18 18:30:00 +03:00
alesapin
dc677b93fb
Comments and fix test
2020-09-17 22:30:17 +03:00
alesapin
b07c00ef7c
Ugly, but probably working implementation
2020-09-17 18:33:50 +03:00
alesapin
40b2f203b6
Merge branch 'master' into manual-write-duplicate-parts-to-replicas
2020-09-17 13:21:00 +03:00
alesapin
f274ffc9d1
Add comments
2020-09-07 10:59:14 +03:00
alesapin
4079636f81
Merge
2020-09-04 13:52:51 +03:00
alesapin
82c56349a5
Some comments
2020-09-04 13:08:09 +03:00
alesapin
128cb7ce22
Don't select already selected parts
2020-09-02 15:16:12 +03:00
alesapin
46f833b7df
Some changes
2020-08-31 22:50:42 +03:00
alesapin
2fc80189af
Add default compression codec to merge tree data part
2020-08-26 18:29:46 +03:00
Alexey Milovidov
5059d1f441
Merge branch 'master' into manual-write-duplicate-parts-to-replicas
2020-08-16 15:58:57 +03:00
Alexey Milovidov
edd89a8610
Fix half of typos
2020-08-08 03:47:03 +03:00
alesapin
c75830e878
Alter table freeze with verbose output
2020-07-28 18:10:36 +03:00
alesapin
014bb070ec
Fix tests
2020-07-14 11:19:39 +03:00
alesapin
1f576ee039
Some intermediate solution
2020-07-13 20:27:52 +03:00
alesapin
4a53264a86
Remove redundant and duplicated code
2020-07-13 19:19:08 +03:00
alexey-milovidov
ae7eff98ed
Merge pull request #12433 from amosbird/np
...
Allow nullable key in MergeTree
2020-07-13 04:36:00 +03:00
Amos Bird
cac5a89169
Allow nullable key in MergeTree
2020-07-12 22:21:51 +08:00
Alexey Milovidov
4b86f36d37
Check arguments of bloom filter index
2020-07-10 11:13:21 +03:00
alesapin
84f8bf1e14
Merge pull request #10697 from CurtizJ/polymorphic-parts
...
Polymorphic parts (in-memory format)
2020-07-08 13:25:45 +03:00
Anton Popov
416d9648c2
remove questionable functionality
2020-07-07 12:58:14 +03:00
Nikita Mikhaylov
d31ed58f01
done
2020-07-06 17:33:31 +03:00
Anton Popov
4422df2e37
Merge remote-tracking branch 'upstream/master' into HEAD
2020-07-02 20:18:21 +03:00
Anton Popov
d45697a1b5
in-memory-parts: lazy creation of WAL
2020-06-30 21:47:12 +03:00
Vladimir Chebotarev
bb8da71eff
Moves task shall be started if new storage policy needs them.
2020-06-30 15:36:44 +03:00
Alexey Milovidov
cc597bb30b
Implement for ReplicatedMergeTree
2020-06-28 22:39:31 +03:00
Alexey Milovidov
1462a66d1e
Fix typos
2020-06-27 22:05:00 +03:00
alesapin
6f1824f0ea
Correct merge with master
2020-06-26 14:30:23 +03:00
alesapin
e9c47dc89c
Merge branch 'master' into CurtizJ-polymorphic-parts
2020-06-26 14:27:19 +03:00
alesapin
292016e045
Add missed lock
2020-06-23 11:04:43 +03:00
Alexey Milovidov
0e6fb617aa
Merge branch 'master' into manual-write-duplicate-parts-to-replicas
2020-06-22 01:54:03 +03:00
alesapin
a2e7e9f232
Remove unused constructor and rename method
2020-06-19 18:21:48 +03:00
alesapin
3c47faa9da
Move partition key initialization into registerMergeTree
2020-06-19 18:14:08 +03:00
alesapin
e888dafdc2
Remove unused method
2020-06-18 20:03:42 +03:00
alesapin
d79982f497
Better locks in Storages
2020-06-18 19:10:47 +03:00
alesapin
ed8f3b2fc4
TTL in storage in memory metadata
2020-06-17 16:39:26 +03:00
alesapin
1afdebeebd
Primary key in storage metadata
2020-06-17 15:39:20 +03:00
alesapin
ab61abccc1
Partition key in StorageInMemoryMetadata
2020-06-17 13:34:23 +03:00
alesapin
62f2c17a66
Secondary indices in StorageInMemoryMetadata
2020-06-17 12:38:47 +03:00
alesapin
eae901f58d
Merge branch 'master' into atomic_metadata5
2020-06-16 14:44:45 +03:00
alesapin
605c904fc6
Add method for compatibility
2020-06-16 11:39:12 +03:00
Anton Popov
b19d48a11c
Merge remote-tracking branch 'upstream/master' into HEAD
2020-06-16 06:37:55 +03:00
Anton Popov
a3ac224ae4
in-memory parts: sync insert to replicated
2020-06-15 22:57:12 +03:00
Alexey Milovidov
e44e1ad0d4
Don't ignore duplicate parts written to replicas
2020-06-15 21:57:38 +03:00
alesapin
e667eb57b2
Working set*Keys methods
2020-06-15 20:24:49 +03:00
alesapin
18b58e8483
Revert too strict lock
2020-06-13 12:12:45 +03:00
alesapin
465c4b65b7
Slightly better interfaces and comments
2020-06-12 12:37:52 +03:00
alesapin
a39d010f94
Merge branch 'master' into consistent_metadata4
2020-06-11 18:47:41 +03:00
Alexey Milovidov
b72a095e99
Fix race conditions in CREATE/DROP of different replicas of ReplicatedMergeTree
2020-06-11 06:24:52 +03:00
alesapin
8be957ecb5
Better checks around metadata
2020-06-10 14:16:31 +03:00
alesapin
ec933d9d03
Better naming
2020-06-10 12:09:51 +03:00
alesapin
d2fcf5aea5
Fixes for gcc
2020-06-09 20:28:29 +03:00
alesapin
4c2e0561f5
Fix primary key update
2020-06-09 20:21:38 +03:00
alesapin
abaf47f0cd
Make metadata single structure
2020-06-05 20:29:40 +03:00
alesapin
5bc7f67e61
Better metadata for select query and renames
2020-06-05 14:54:54 +03:00
Anton Popov
1ce09e1faa
Merge remote-tracking branch 'upstream/master' into polymorphic-parts
2020-06-03 16:27:54 +03:00
Anton Popov
caf1e4e8cc
in-memory-parts: fixes
2020-06-03 12:51:23 +03:00
alesapin
e0e77194f4
Remove redundant parts from MergeTree
2020-06-01 15:39:20 +03:00
alesapin
d9a817f578
Less virtual methods in IStorage
2020-06-01 14:17:18 +03:00
alesapin
3847ea892d
Merge branch 'master' into consistent_metadata3
2020-06-01 13:17:59 +03:00
alexey-milovidov
9984e989c1
Merge pull request #11317 from ClickHouse/logger-fmt-2
...
Preparation for structured logging
2020-06-01 00:11:29 +03:00
alexey-milovidov
8accde79b5
Merge pull request #11162 from azat/data-skip-index-merging-params-fix
...
[RFC] Fix data skipping indexes for columns with additional actions during merge
2020-05-31 19:10:53 +03:00
Alexey Milovidov
25f941020b
Remove namespace pollution
2020-05-31 00:57:37 +03:00
alesapin
860dc2c814
Merge with consistent_metadata2
2020-05-28 19:45:51 +03:00
alesapin
61aff798d3
Compileable code
2020-05-28 18:33:44 +03:00
Anton Popov
d8342e5b12
in-memory parts: better restore and clear stale wal files
2020-05-27 23:05:55 +03:00
alesapin
380c60a33f
Intermediate stage (doesn't compile)
2020-05-27 21:38:34 +03:00
Nikolai Sorokin
eeaf608aa9
Refactor code; support prefix of PK as GROUP BY key
2020-05-27 12:53:15 +03:00
Nikolai Sorokin
61974e0047
TTL Expressions WHERE and GROUP BY draft
2020-05-27 12:53:15 +03:00
alesapin
7bbb064b34
Fix metadata update
2020-05-26 11:58:32 +03:00
alesapin
c3a6571036
Compilable code
2020-05-25 20:22:20 +03:00
alesapin
9b3cc9e525
Almost working ttl in IStorage
2020-05-25 20:07:14 +03:00
alesapin
6281dd6893
Merge pull request #11115 from ClickHouse/consistent_metadata
...
Refactoring in storage metadata (more consistent keys)
2020-05-25 13:09:56 +03:00
Azat Khuzhin
c36708caf9
Split data skipping indexes from the order by to evaluate order by before merge
2020-05-24 16:12:08 +03:00
alesapin
f026fa7de1
Rename PartDestinationType to DataDestionationType, because there is no part-specific information
2020-05-22 16:29:33 +03:00
alesapin
09034157cd
Remove comments
2020-05-21 22:53:46 +03:00
Alexander Kuzmenkov
f98ffdbc4c
Merge pull request #11087 from azat/context-fwd-decl
...
[RFC] Forward declaration for Context as much as possible.
2020-05-21 19:43:29 +03:00
alesapin
87eab514fb
Merge branch 'master' into consistent_metadata
2020-05-21 18:54:52 +03:00
BohuTANG
c9d315654c
Fix Storages/MergeTree typo
2020-05-21 17:00:44 +08:00
Azat Khuzhin
d93b9a57f6
Forward declaration for Context as much as possible.
...
Now after changing Context.h 488 modules will be recompiled instead of 582.
2020-05-21 01:53:18 +03:00
alesapin
616902a995
Sorting and primary key (broken)
2020-05-20 21:11:38 +03:00
alesapin
9fb28f5ac0
Add sampling key
2020-05-20 18:16:39 +03:00
alesapin
8bc527eecd
Compile partition key
2020-05-20 15:16:55 +03:00
Anton Popov
4fb6492b08
in-memory parts: delay merges
2020-05-20 15:02:02 +03:00
Gleb Novikov
1a25ac6e1f
Merge branch 'master' into refactor-reservations
2020-05-16 23:34:45 +03:00
Anton Popov
14e1a46a8c
Merge remote-tracking branch 'upstream/master' into polymorphic-parts
2020-05-15 03:48:27 +03:00
alesapin
f209052519
Better comments
2020-05-13 14:04:27 +03:00
Gleb Novikov
390b39b272
VolumePtr instead of DiskPtr in MergeTreeData*
2020-05-10 00:24:15 +03:00
Anton Popov
387041a97e
Merge remote-tracking branch 'upstream/master' into polymorphic-parts
2020-05-06 19:22:07 +03:00
Anton Popov
4878c91d07
in-memory parts: better restore from wal
2020-05-06 14:57:38 +03:00
Gleb Novikov
c637d99e07
Volumes and storages refactoring:
...
1. Moved Volume to separate file
2. Created IVolume interface and implemented current behaviour in implementation of new interface — VolumeJBOD
3. Replaced all old volume usages with new VolumeJBOD. Where it is unnecessary to have JBOD — left just IVolume.
4. Removed old Volume completely
5. Moved StoragePolicy to separated files
6. Moved DiskSelector to separated files
7. Removed DiskSpaceMonitor file
2020-05-04 23:15:38 +03:00
alexey-milovidov
db4c2352e1
Merge pull request #10569 from zhang2014/fix/ISSUES-10551
...
ISSUES-10551 add backward compatibility for create bloom filter index
2020-05-02 22:25:29 +03:00
Anton Popov
aa723d245f
Merge remote-tracking branch 'upstream/master' into polymorphic-parts
2020-04-30 18:27:39 +03:00
zhang2014
8ac709b5d3
ISSUES-10551 add backward compatibility for create bloom filter index
2020-04-29 11:57:28 +08:00
alesapin
f981649213
Fix pushing to views stream and refactor virtuals
2020-04-28 13:38:57 +03:00