Alexander Tokmakov
ca989e9212
less runtime format strings
2022-12-23 19:50:34 +01:00
Anton Popov
0722020cae
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-23 18:15:55 +00:00
alesapin
2fecf420cb
Merge pull request #44513 from ClickHouse/small_polishing
...
Minor code polishing
2022-12-23 13:47:26 +01:00
alesapin
a32fab90d5
Merge pull request #44436 from ClickHouse/fix_bug_with_max_task_count_update
...
Fix bug with merge/mutate pool size increase
2022-12-22 23:48:54 +01:00
alesapin
fa467b3398
Minor code polishing
2022-12-22 14:31:42 +01:00
alesapin
2818520572
Fix bug with merge/mutate pool size increase
2022-12-20 14:51:11 +01:00
alesapin
c8d9ccf3cd
Lock table for share during startup for database ordinary
2022-12-19 18:51:46 +01:00
Alexander Tokmakov
f4c8281ccc
avoid outdating part twice
2022-12-13 21:24:26 +01:00
Anton Popov
0b396bab14
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-12 18:08:27 +00:00
Sema Checherinda
77d4f052ea
do not clear old parts at startup also
2022-12-05 16:05:14 +01:00
Sema Checherinda
dc52a40c04
decrease stop max-tries back to default, do not clear old parts at shutdown
2022-12-05 16:05:14 +01:00
Sema Checherinda
fd2c7657be
do not read part status just for logging
2022-12-01 16:08:05 +01:00
Anton Popov
df6d2f2163
Merge remote-tracking branch 'upstream/master' into HEAD
2022-11-29 23:21:23 +00:00
Anton Popov
c65be55da1
Merge remote-tracking branch 'upstream/master' into HEAD
2022-11-24 14:30:23 +00:00
Sema Checherinda
eccb62e4c3
work with the latest remarks
2022-11-23 15:16:09 +00:00
Sema Checherinda
e54ca534b0
test 01167_isolation_hermitage uses fallback to recreate table
2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208
fix isolation breach, droping outdated part should't be ignored
2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25
fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part
2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104
less pedantic checks in operations, rely on existed tx functionality
2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0
no groupping parts for cover with empty
2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a
fix several critical races, fix tests
2022-11-23 15:16:08 +00:00
Sema Checherinda
f2b9f80f2f
fix tidy build
2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070
address the review comments
2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9
Delete parts over replacing them with empty parts
2022-11-23 15:16:08 +00:00
Anton Popov
75c807609b
add some comments
2022-11-22 16:11:01 +00:00
Anton Popov
e3db29aebf
better waiting for outdated data parts
2022-11-22 15:15:20 +00:00
Anton Popov
92ac9385b9
wait loading of outdated parts for partition commands
2022-11-21 17:01:46 +00:00
Anton Popov
f94182384a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-11-16 03:30:29 +00:00
Alexey Milovidov
4966556f47
Miscellaneous changes
2022-11-14 06:41:51 +01:00
Alexander Tokmakov
e415fc6e95
Revert "Optimize TTL merge, completely expired parts can be removed in time"
2022-11-10 19:38:55 +03:00
zhongyuankai
c1e2b2af74
Optimize TTL merge, completely expired parts can be removed in time
2022-11-01 22:58:15 +08:00
Anton Popov
c4d4f2dbed
better interface
2022-10-25 22:14:06 +00:00
Anton Popov
c8199bc125
Merge branch 'master' into refactor-data-part
2022-10-25 14:31:05 +02:00
Anton Popov
cf375c9732
better interface
2022-10-23 22:29:24 +00:00
Anton Popov
b40d9200d2
better semantic of constsness of DataPartStorage
2022-10-23 15:24:20 +00:00
Anton Popov
56e5daba0c
remove DataPartStorageBuilder
2022-10-23 00:23:15 +00:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Alexander Tokmakov
0d8a814d80
Merge pull request #42254 from xinhuitian/optimize-selectPartsToMutate
...
Early return of corner cases in selectPartsToMutate function
2022-10-20 18:13:37 +03:00
Tian Xinhui
45c6974f7d
Use alias for std::shared_ptr<MergeMutateSelectedEntry> ( #42211 )
...
* use alias for MergeMutateSelectedEntry share ptr
* fix StorageMergeTree.cpp
2022-10-18 13:35:34 +02:00
Anton Popov
95fdb0a877
remove outdated parts asynchronously
2022-10-14 14:52:57 +00:00
xinhuitian
daaf01c344
early return
2022-10-12 12:18:17 +08:00
alesapin
695133cc5b
Better mutations
2022-09-27 15:23:02 +02:00
Anton Popov
ee54451741
fix adding of columns of type Object
2022-09-13 22:43:59 +00:00
Vitaly Baranov
122009a2bd
Use table lock if database is ordinary and zero-copy-replication is enabled.
2022-09-08 13:54:59 +02:00
Alexey Milovidov
940a53e519
Merge pull request #40984 from Lucky-Chang/typo_fix
...
Fix some typos and clang-tidy warnings
2022-09-06 02:37:50 +03:00
Luck-Chang
1ac8e739c9
fix some typos and clang-tidy warnings
2022-09-05 09:50:24 +08:00
Antonio Andelic
e64436fef3
Fix typos with new codespell
2022-09-02 08:54:48 +00:00
Alexander Tokmakov
0aea4c7184
fix
2022-08-25 14:22:27 +02:00
Alexander Tokmakov
3cd26aafe4
remove wrong code for skipping mutations in MergeTree
2022-08-24 20:58:59 +02:00
Alexander Tokmakov
fe572104aa
fix old tmp dirs cleanup
2022-08-09 18:44:51 +02:00
Alexander Gololobov
25deba2c1b
Merge branch 'master' into feature/sql-standard-delete
2022-07-25 22:13:20 +02:00
Alexander Gololobov
1ea9f143ff
Leave only _row_exists-based implementation of lightweight delete
2022-07-21 11:26:13 +02:00
Raúl Marín
aea045f297
Improve logging around replicated merges
2022-07-15 15:48:35 +02:00
jianmei zhang
4268ac8b0e
Add supportLightweightDeleteMutate() in IMergeTreeDataPart to disable LWD for part with projections
2022-07-15 12:32:42 +08:00
jianmei zhang
198674e994
Consider lightweight delete as ordinary for compact part in selectPartsToMutate()
2022-07-15 12:32:42 +08:00
jianmei zhang
780cdfb8f0
Code changes based on review opinions. Make ordinary single alter delete lightwight by default
2022-07-15 12:32:42 +08:00
jianmei zhang
b4a37e1e22
Disable optimizations for count() when lightweight delete exists, add hasLightweightDelete() function in IMergeTreeDataPart
2022-07-15 12:32:41 +08:00
jianmei zhang
8696319d62
Support lightweight delete execution using string as deleted rows mask,also part of select can handle LWD
2022-07-15 12:32:41 +08:00
jianmei zhang
8df7b7a030
Support new mutation type for lightweight
2022-07-15 12:32:41 +08:00
Amos Bird
0795cdd36a
Slightly better interface of waitForMutation
2022-07-13 12:34:12 +08:00
Vitaly Baranov
ed27987646
Merge pull request #38861 from vitlibar/backup-improvements-9
...
Backup Improvements 9
2022-07-07 02:24:47 +02:00
Vitaly Baranov
7f84cf3968
Fix style.
2022-07-06 16:36:59 +02:00
Vitaly Baranov
43d35eec1b
Write unfinished mutations to backup.
2022-07-05 14:51:09 +02:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder
2022-07-02 15:39:44 +02:00
alesapin
0d7298f3a9
Fix typos
2022-07-01 15:26:27 +02:00
alesapin
57284e6a9d
Fix possible deadlocks with MergeTreeData::Transaction
2022-07-01 15:16:32 +02:00
alesapin
d88cdcd5c1
AAA
2022-06-27 21:41:29 +02:00
alesapin
80d27f6381
Merge pull request #38380 from ClickHouse/simplify_parts_commit_merge_tree
...
Simplify parts commit methods
2022-06-25 00:12:52 +02:00
alesapin
011d58d7a0
Simplify more
2022-06-24 17:19:59 +02:00
alesapin
37310dc9df
Simpler
2022-06-24 14:10:15 +02:00
alesapin
36b810b076
Remove unused parameter
2022-06-24 13:42:36 +02:00
alesapin
af1a9d18ab
Remove transaction argument
2022-06-24 13:34:00 +02:00
alesapin
9910395823
Simplify method signature
2022-06-24 13:19:29 +02:00
alesapin
0f8eed98c4
Merge branch 'master' into better-support-gcp
2022-06-22 12:46:01 +02:00
Nikolai Kochetov
cc6fdfe0eb
Merge pull request #36555 from ClickHouse/refactor-something-in-part-volumes
...
Separate data storage abstraction for MergeTree
2022-06-22 11:13:36 +02:00
Anton Popov
d5d568e235
Merge pull request #37913 from azat/parts-refcnt
...
Fix refcnt for unused MergeTree parts in SELECT queries
2022-06-21 22:56:26 +02:00
Nikolai Kochetov
854d148d73
Add some comments. Remove some commented code.
2022-06-21 12:31:02 +00:00
Nikolai Kochetov
1e8c9ecd4c
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-21 12:37:21 +02:00
Nikolai Kochetov
dccf90b1ea
Cleanup.
2022-06-20 18:18:17 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions
2022-06-20 18:11:30 +00:00
Nikolai Kochetov
7452d04e3a
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-20 15:31:02 +02:00
Alexander Tokmakov
83adf56383
fix race
2022-06-17 18:13:57 +02:00
mergify[bot]
7f24574609
Merge branch 'master' into better-support-gcp
2022-06-15 23:18:20 +00:00
Vitaly Baranov
5cabdbd982
Restore parts of MergeTree in correct order.
2022-06-15 20:28:40 +02:00
Nikolai Kochetov
2a9a63ac7b
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-15 15:35:26 +02:00
Anton Popov
a59be0fd5d
better support of GCP storage
2022-06-15 00:23:45 +00:00
kssenii
4859e11c27
Review fixes
2022-06-12 01:15:10 +02:00
kssenii
fd6c84b831
Review fixes
2022-06-10 14:23:13 +02:00
kssenii
7a2676c7ab
Clean up broken detached parts with timeout
2022-06-10 12:27:57 +02:00
Nikolai Kochetov
5bc9b32025
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-08 11:10:06 +00:00
Azat Khuzhin
61a2613127
Fix refcnt for unused MergeTree parts in SELECT queries
...
Before this patch SELECT queries hold parts even if they were not
required by select (had been eliminated by partition pruning).
This defers removing parts if you have long running queries.
This had been introduced in #23932 , with introduction of
StorageSnapshotPtr.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-08 07:04:53 +03:00
alesapin
e32d36d790
Proper fix
2022-06-07 17:58:32 +02:00
Nikolai Kochetov
678d978acf
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-07 15:23:00 +00:00
Nikolai Kochetov
89c5855d20
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-02 12:19:07 +02:00
Nikolai Kochetov
edac3d6714
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-06-02 09:36:20 +00:00
lthaooo
6632616733
Fix TTL merge scheduling bug ( #36387 )
2022-06-01 21:09:53 +02:00
Nikolai Kochetov
86fbb74703
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-31 18:07:47 +00:00
Nikolai Kochetov
c71256ea38
Remove some commented code.
2022-05-30 13:18:20 +00:00
Nikolai Kochetov
5b4658aa5e
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-30 09:47:35 +00:00
alesapin
be1c3c132b
Fix some trash
2022-05-27 16:08:49 +02:00
Alexander Tokmakov
eb71dd4c78
Merge pull request #37547 from ClickHouse/followup_37398
...
Follow-up to #37398
2022-05-26 20:29:41 +03:00
Alexander Tokmakov
e8f33fb0d9
fix flaky tests
2022-05-26 14:17:05 +02:00
Azat Khuzhin
dc9ca3d70c
Fix LOGICAL_ERROR in getMaxSourcePartsSizeForMerge during merges ( #37413 )
2022-05-26 14:14:58 +02:00
Nikolai Kochetov
fd97a9d885
Move some resources
2022-05-23 19:47:32 +00:00
Nikolai Kochetov
fc3645a483
Fix some tests.
2022-05-03 19:32:24 +00:00
Nikolai Kochetov
35095191eb
Merge branch 'master' into refactor-something-in-part-volumes
2022-05-03 17:51:47 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying
...
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
previously allowed.
Hence, this change
- removes shared_ptr_helper and as a result all inherited create() methods,
- instead, Storage objects are now created using make_shared<>() by the
caller (for that to work, many constructors had to be made public), and
- all Storage classes were marked as noncopyable using boost::noncopyable.
In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
2022-05-02 08:46:52 +02:00
Nikolai Kochetov
e44af67fee
Merge branch 'master' into refactor-something-in-part-volumes
2022-04-26 21:08:00 +02:00
Vitaly Baranov
68a020ecea
Implement BACKUP/RESTORE ON CLUSTER.
2022-04-25 16:34:33 +02:00
Nikolai Kochetov
8c00692844
Part 8
2022-04-22 16:58:09 +00:00
Nikolai Kochetov
9133e398b8
Part 7
2022-04-21 19:19:13 +00:00
alesapin
5465415751
Fix replace/move partition with zero copy replication
2022-04-21 14:39:12 +02:00
alesapin
40c15222f8
Merge branch 'master' into fix_trash
2022-04-20 12:45:49 +02:00
alesapin
7cb7c120cc
Less ugly
2022-04-19 15:53:10 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains"
...
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
2022-04-18 23:53:11 +02:00
Alexander Tokmakov
66fdf35dfd
remove outdated parts immediately on drop partition
2022-04-13 18:01:22 +02:00
Alexander Tokmakov
49c35f3261
Merge branch 'master' into mvcc_prototype
2022-04-08 13:34:40 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements
...
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
2022-04-08 12:06:26 +02:00
Alexander Tokmakov
6a46da93ae
Merge branch 'master' into mvcc_prototype
2022-04-07 23:22:19 +02:00
jewisliu
77edd41b2e
[Improvement] improvement in PARTITION ALL
...
1. ASTPartition::formatImpl should output ALL while executing ALTER TABLE t DETACH PARTITION ALL
2. prohibit PARTITION ALL excepte DETACH PARTITION ALL
2022-04-07 17:37:01 +08:00
Alexander Tokmakov
1fe50ad201
Merge branch 'master' into mvcc_prototype
2022-04-05 14:38:02 +02:00
Vladimir C
54b4049ea5
Merge pull request #35794 from awakeljw/fork_chmaster
2022-04-05 09:44:45 +02:00
kssenii
6c8e073a61
Merge master
2022-04-01 16:48:29 +02:00
jewisliu
9a4cc78dfb
support ALTER TABLE t DETACH PARTITION ALL syntax
2022-04-01 13:13:07 +08:00
Alexander Tokmakov
287d858fda
Merge branch 'master' into mvcc_prototype
2022-03-29 16:24:12 +02:00
kssenii
d2a3cfe5dc
Cache on all write operations
2022-03-23 19:14:33 +01:00
kssenii
c2c7f365ce
Merge master
2022-03-23 16:58:35 +01:00
kssenii
8c1be8950e
Improve test
2022-03-23 16:55:18 +01:00
taiyang-li
8dbf1c60e7
merge master and fix conflict
2022-03-23 11:36:50 +08:00
Alexander Tokmakov
3c762f566d
Merge branch 'master' into mvcc_prototype
2022-03-21 20:16:29 +01:00
Vitaly Baranov
ce25afb2e9
Storages and databases are hollow by default now.
2022-03-20 20:02:15 +01:00
Vitaly Baranov
7f89b98308
Rework BackupSettings and RestoreSettings a little, pass StorageRestoreSettings to storages.
2022-03-20 20:02:15 +01:00
Vitaly Baranov
258a472001
Shorter names: rename IRestoreFromBackupTask -> IRestoreTask.
2022-03-20 20:02:15 +01:00
Vitaly Baranov
7a63feb3f7
Make restore tasks explicit.
2022-03-20 20:01:31 +01:00
Alexander Tokmakov
418d52044a
try fix merge assignment
2022-03-18 20:31:44 +01:00
Alexander Tokmakov
c28008a2df
more comments, minor fixes
2022-03-18 12:01:26 +01:00
Alexander Tokmakov
07d952b728
use snapshots for semistructured data, durability fixes
2022-03-17 18:26:18 +01:00
Alexander Tokmakov
d04dc03fa4
Merge branch 'master' into mvcc_prototype
2022-03-17 15:24:32 +01:00
Alexander Tokmakov
c2ac8d4a5c
review fixes
2022-03-16 21:05:34 +01:00
Alexander Tokmakov
278d779a01
log cleanup, more comments
2022-03-14 21:43:34 +01:00
Anton Popov
36ec379aeb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-14 16:28:35 +00:00
Alexander Tokmakov
061fa6a6f2
Merge branch 'master' into mvcc_prototype
2022-03-10 13:13:04 +01:00
Azat Khuzhin
caffc144b5
Fix possible "Part directory doesn't exist" during INSERT
...
In #33291 final part commit had been defered, and now it can take
significantly more time, that may lead to "Part directory doesn't exist"
error during INSERT:
2022.02.21 18:18:06.979881 [ 11329 ] {insert} <Debug> executeQuery: (from 127.1:24572, user: default) INSERT INTO db.table (...) VALUES
2022.02.21 20:58:03.933593 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18044_18044_0 to 20220214_270654_270654_0.
2022.02.21 21:16:50.961917 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18197_18197_0 to 20220214_270689_270689_0.
...
2022.02.22 21:16:57.632221 [ 64878 ] {} <Warning> db.table: Removing temporary directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/
...
2022.02.23 12:23:56.277480 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18232_18232_0 to 20220214_273459_273459_0.
2022.02.23 12:23:56.299218 [ 11329 ] {insert} <Error> executeQuery: Code: 107. DB::Exception: Part directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/ doesn't exist. Most likely it is a logical error. (FILE_DOESNT_EXIST) (version 22.2.1.1) (from 127.1:24572) (in query: INSERT INTO db.table (...) VALUES), Stack trace (when copying this message, always include the lines below):
Follow-up for: #28760
Refs: #33291
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-08 07:44:11 +03:00
taiyang-li
b4174b0bef
merge master and fix conflicts
2022-03-08 11:39:25 +08:00
Alexander Tokmakov
8acfb8d27f
Merge branch 'master' into mvcc_prototype
2022-03-07 17:40:15 +01:00
Anton Popov
df3b07fe7c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-03 22:25:28 +00:00
Amos Bird
d4cdf04683
Add missing locks ( #34025 )
2022-03-02 16:23:29 +01:00
Anton Popov
fcdebea925
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-25 13:41:30 +03:00
Alexander Tokmakov
dae044f86b
Merge branch 'master' into mvcc_prototype
2022-02-17 13:49:37 +03:00
Alexander Tokmakov
e37ef4560c
fix
2022-02-15 18:00:45 +03:00
Alexander Tokmakov
ae5aa8c12d
write part version before other files
2022-02-15 02:24:51 +03:00
Anton Popov
18940b8637
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-09 23:38:38 +03:00
taiyang-li
b6132d490f
merge master and solve conflict
2022-02-08 15:24:59 +08:00
Alexander Tokmakov
fe30e0f162
fixes
2022-02-03 21:57:09 +03:00
Alexander Tokmakov
ca5f951558
Merge branch 'master' into mvcc_prototype
2022-02-03 18:56:44 +03:00
alesapin
25375bc76b
Remove unused param
2022-02-03 11:21:19 +03:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
Azat Khuzhin
bedf208cbd
Use fmt::runtime() for LOG_* for non constexpr
...
Here is oneliner:
$ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'
Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:
$ cat fmt.cocci
@@
expression log;
expression var;
@@
-LOG_DEBUG(log, var)
+LOG_DEBUG(log, fmt::runtime(var))
I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-01 14:30:03 +03:00
Alexander Tokmakov
2e4ae37d98
Merge branch 'master' into mvcc_prototype
2022-02-01 13:20:03 +03:00
Alexander Tokmakov
5fad3fdffc
throw exception on non-transactional queries
2022-02-01 01:27:55 +03:00
Maksim Kita
5ef83deaa6
Update sort to pdqsort
2022-01-30 19:49:48 +00:00
Alexander Tokmakov
e0304c2a58
review fixes, write tid into mutation entry
2022-01-28 20:47:37 +03:00
taiyang-li
73def8b483
merge master and solve conflict
2022-01-24 11:01:43 +08:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001
Merge branch 'master' into mvcc_prototype
2022-01-21 18:59:01 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts
...
Fix flushing of in-memory parts
2022-01-21 02:18:03 +03:00
Alexander Tokmakov
e9a5a64a71
Merge branch 'master' into mvcc_prototype
2022-01-19 21:41:23 +03:00
Azat Khuzhin
aee034a597
Use explicit template instantiation for SystemLog
...
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h
v2: rebase
v3: squash move into module part with explicit template instantiation
(to make each commit self compilable after rebase)
2022-01-10 22:01:41 +03:00
Alexander Tokmakov
158fbaaa29
fix
2022-01-10 20:07:01 +03:00
Alexander Tokmakov
bf2b6c2c37
Merge branch 'master' into mvcc_prototype
2022-01-10 19:53:50 +03:00
taiyang-li
cf413f16a8
remove function checkPartMetadataCache
2022-01-07 18:37:08 +08:00
Alexander Tokmakov
187c43eba8
rename Committed state to Active
2021-12-30 23:45:38 +03:00
Anton Popov
4ebf61b809
proper checking of atomic flags
2021-12-29 01:03:55 +03:00
Anton Popov
a83832c03e
fix flushing of in-memory parts
2021-12-27 18:54:28 +03:00
Alexander Tokmakov
df12fdf612
fix tests
2021-12-22 19:34:02 +03:00
Alexander Tokmakov
a15444031c
add test
2021-12-20 21:53:05 +03:00
Anton Popov
99ebabd822
Merge remote-tracking branch 'upstream/master' into HEAD
2021-12-17 19:02:29 +03:00
Alexander Tokmakov
32e62ed5c2
Merge branch 'master' into mvcc_prototype
2021-12-17 10:43:46 +03:00
Alexander Tokmakov
e185ad260b
fix skipping of some mutations
2021-12-15 21:28:24 +03:00
Alexander Tokmakov
1b02136941
fix skipping of some mutations
2021-12-15 21:19:29 +03:00
Alexander Tokmakov
d7ad72838c
Merge branch 'master' into mvcc_prototype
2021-12-14 23:07:52 +03:00
Alexander Tokmakov
6e1c16c2e7
add support for mutations
2021-12-14 23:06:34 +03:00
nautaa
0da3881ff4
flush all InMemoryDataParts when wal is not enabled
2021-12-14 16:31:17 +08:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas ( #29279 )
2021-12-09 13:39:28 +03:00
Alexander Tokmakov
7fcb79ae72
Merge branch 'master' into mvcc_prototype
2021-12-07 14:39:29 +03:00
Anton Popov
6f4d9a53b2
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-12-01 15:54:33 +03:00
Vladimir Chebotarev
e8e3411ebc
Minor fixes for StorageMergeTree
.
2021-12-01 11:18:07 +03:00
tavplubix
78224ef273
Merge pull request #31738 from ClickHouse/fix_segfault_on_attach_partition
...
Fix segfault on attach parititon
2021-11-30 14:32:50 +03:00
Vladimir Chebotarev
100e61ab0e
Rebase fix.
2021-11-29 10:15:00 +03:00
Vladimir Chebotarev
bb2f21b1ad
Rebase fix.
2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
e414fa01e0
Rebase fix.
2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
c6e73d7b60
Synced clearOldMutations()
with rest of changes.
2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
ec1f909d74
Fixed wrong lock order.
2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
12f36206cf
Deadlock fix.
2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
84ffcb479b
Additional comments.
2021-11-25 17:11:52 +03:00
Vladimir Chebotarev
e0a453aadf
Minor fix.
2021-11-25 17:11:29 +03:00
Vladimir Chebotarev
863f3565ad
Test fixes.
2021-11-25 17:11:28 +03:00
Vladimir Chebotarev
390416b24d
Rebase fix.
2021-11-25 17:11:28 +03:00
Vladimir Chebotarev
30cd133418
Fixed duplication commands bug.
2021-11-25 17:11:28 +03:00
Vladimir Chebotarev
aa3eecbd6d
Fixed bug with frozen mutations.
2021-11-25 17:11:27 +03:00
Vladimir Chebotarev
6599124dba
Attempt to skip mutations of different partitions in StorageMergeTree
.
2021-11-25 17:10:11 +03:00
Alexander Tokmakov
1ab8b5d4df
fix segfault on attach parititon
2021-11-25 13:04:39 +03:00
Anton Popov
ccd78e3838
Merge remote-tracking branch 'upstream/master' into HEAD
2021-11-22 17:19:35 +03:00
tavplubix
67753e5b83
Update StorageMergeTree.cpp
2021-11-19 20:05:45 +03:00
Alexander Tokmakov
8d7413b0d6
remove strange multimap from mutations
2021-11-19 19:54:43 +03:00
Alexander Tokmakov
52885db5d7
remove strange multimap from mutations
2021-11-19 19:51:03 +03:00
Alexander Tokmakov
0a4647f927
support alter partition
2021-11-17 21:14:14 +03:00
Alexander Tokmakov
672157b817
Merge branch 'master' into mvcc_prototype
2021-11-11 22:01:34 +03:00
Anton Popov
a20922b2d3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-11-09 15:36:25 +03:00
Alexander Tokmakov
51bd98294e
better merge selecting backoff
2021-11-09 15:26:51 +03:00
Alexander Tokmakov
92eec74ad7
Merge branch 'master' into mvcc_prototype
2021-11-06 21:08:36 +03:00
tavplubix
ac4a9bcf23
Fix 00975_move_partition_merge_tree ( #30717 )
...
* Revert "Fix race between MOVE PARTITION and merges/mutations for MergeTree"
This reverts commit e9ce859b02
.
* fix
* Update StorageMergeTree.cpp
Co-authored-by: alesapin <alesapin@gmail.com>
2021-10-27 18:19:41 +03:00
Alexander Tokmakov
2e7e195e77
change alter_lock to std::timed_mutex
2021-10-26 13:37:00 +03:00
alexey-milovidov
7beffb04ad
Merge pull request #29495 from ClickHouse/improvement_for_strange_mutations
...
Minor improvement for strange mutations
2021-10-17 20:57:06 +03:00
Nikolai Kochetov
bfcbf5abe0
Merge branch 'master' into removing-data-streams-folder
2021-10-17 10:42:37 +03:00
Nikolai Kochetov
a08c98d760
Move some files.
2021-10-16 17:03:50 +03:00
Azat Khuzhin
07e8b2b3c7
Do not try to remove temporary paths that is currently in written by merge/mutation
...
v2: rebase against MergeTask
v3: rebase due to conflicts in src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
v4:
- rebase due to conflicts in src/Storages/MergeTree/MergeTask.cpp
- drop common/scope_guard_safe.h (not used)
2021-10-16 00:43:52 +03:00
Azat Khuzhin
e9ce859b02
Fix race between MOVE PARTITION and merges/mutations for MergeTree
...
From time to time 00975_move_partition_merge_tree test got failed, the
reason is that there is merge in progress that writes data to the source
table again:
<details>
2021.10.12 13:20:40.243839 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> executeQuery: (from [::1]:38060) (comment: 00975_move_partition_merge_tree.sql) ALTER TABLE test_move_partition_src MOVE PARTITION 1 TO TABLE test_move_partition_dest;
2021.10.12 13:20:40.244482 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_2_2_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_21_21_0
...
2021.10.12 13:20:40.373487 [ 378 ] {} <Trace> test_t1tzb2.test_move_partition_src (467d7145-b47e-444e-867d-7145b47ea44e) (MergerMutator): Merged 6 parts: from 1_2_2_0 to 1_11_11_0
...
2021.10.12 13:20:40.379750 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_15_15_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_28_28_0
</details>
And also remove cleaning of mutations since this will cause deadlock
after doing MOVE PARTITION under
currently_processing_in_background_mutex.
CI: https://clickhouse-test-reports.s3.yandex.net/0/a59c6b1c8eb47ebf77189a491ee0c3980b38e91c/functional_stateless_tests_
Fixes: 00975_move_partition_merge_tree
2021-10-13 00:14:24 +03:00
Alexander Tokmakov
5365cc686f
Merge branch 'master' into improvement_for_strange_mutations
2021-10-07 13:26:58 +03:00
Alexander Tokmakov
bb32432943
more fixes
2021-10-05 00:13:18 +03:00
Azat Khuzhin
117e9e77c8
Apply max_untracked_memory/memory_profiler_{step,sample_probability} during mutate/merge
2021-10-03 17:39:07 +03:00
Nikita Mikhaylov
9756b8dc33
Added an ability to execute more merges and mutations than threads, added new settings ( #29140 )
2021-10-01 00:26:24 +03:00
Nikolai Kochetov
236d71ea94
Merge pull request #28582 from ClickHouse/rewrite-pushing-to-views
...
Rewrite PushingToViews
2021-09-27 21:19:11 +03:00
Amos Bird
34017c942e
More fixes
2021-09-25 13:38:56 +08:00
Alexander Tokmakov
4114b82937
hotfixes for merge task
2021-09-23 23:15:49 +03:00
Alexander Tokmakov
72b1b2e360
Merge branch 'master' into mvcc_prototype
2021-09-23 22:53:27 +03:00
Nikolai Kochetov
a790d391c0
Merge branch 'master' into rewrite-pushing-to-views
2021-09-20 19:43:15 +03:00
Vladimir Chebotarev
db516e8c91
Removed sub-optimal mutation notifications in StorageMergeTree
when merges are still going ( #27552 )
...
* Removed redundand mutation notifications in `StorageMergeTree` when merges are still going.
2021-09-17 16:19:36 +03:00
Nikolai Kochetov
a8c3b02598
Merge branch 'master' into rewrite-pushing-to-views
2021-09-17 15:38:11 +03:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask ( #25165 )
...
Introduced MergeTask and MutateTask
2021-09-17 00:19:58 +03:00
Nikolai Kochetov
f569a3e3f7
Merge branch 'master' into rewrite-pushing-to-views
2021-09-09 20:30:23 +03:00
kssenii
0c013ed91b
Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-web-disk-2
2021-09-09 15:26:36 +03:00
Anton Popov
4c388e3d84
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-09-09 14:10:16 +03:00
Nikolai Kochetov
999a4fe831
Fix other tests.
2021-09-08 21:29:38 +03:00
Nikita Mikhaylov
6062dd0021
Better
2021-09-08 00:21:21 +00:00
Nikita Mikhaylov
ea0fbf81af
Renaming
2021-09-06 12:01:16 +00:00
kssenii
97e0a0310a
Slightly better
2021-09-04 12:02:07 +03:00
Nikita Mikhaylov
cc7c221fad
Own PriorityQueue + prettifying the code
2021-09-02 21:31:32 +00:00
Nikita Mikhaylov
c4416906c8
done
2021-08-30 19:37:03 +00:00
kssenii
b536653851
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-24 07:29:50 +00:00
kssenii
5307d31924
Throw for alter and silence for drop
2021-08-23 23:48:53 +00:00
Vitaly Baranov
63e8bc1f20
Added new commands BACKUP and RESTORE.
2021-08-23 23:41:45 +03:00
Anton Popov
61239343e3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-08-20 16:33:30 +03:00
alexey-milovidov
54d8282209
Merge pull request #27248 from azat/mt-mutation-stuck-fix
...
Fix mutation stuck on invalid partitions in non-replicated MergeTree
2021-08-11 06:10:15 +03:00
Azat Khuzhin
91d7f3daa7
Fix lock-order-inversion while notifying about finished mutations
...
TSAN reports [1]:
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=36)
Cycle in lock order graph: M16388 (0x7b7400011d68) => M1030334152907497744 (0x000000000000) => M16388
Mutex M1030334152907497744 acquired here while holding mutex M16388 in thread T4:
0 pthread_mutex_lock <null> (clickhouse+0x967d536)
1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse+0x1b25c7d9)
2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x1b25c7d9)
3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x15220cd9)
4 DB::StorageMergeTree::getIncompleteMutationsStatus(long, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std:>
5 DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0::operator()() const obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:464:36 (clickhouse+0x1521e2b1)
6 void std::__1::condition_variable::wait<DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0>(std::__1::unique_lock<std::__1::mutex>&, DB::StorageMergeTree::waitForMutation(lon>
7 DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:469:29 (clickhouse+0x1521e2b1)
8 DB::StorageMergeTree::mutate(DB::MutationCommands const&, std::__1::shared_ptr<DB::Context const>) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:496:9 (clickhouse+0x15221738)
9 DB::InterpreterAlterQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterAlterQuery.cpp:113:16 (clickhouse+0x141182f6)
10 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:560:32 (clickhouse+0x149152f6)
11 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:909:30 (clickhous>
12 DB::TCPHandler::runImpl() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:313:24 (clickhouse+0x153270af)
Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
Mutex M16388 acquired here while holding mutex M1030334152907497744 in thread T59:
0 pthread_mutex_lock <null> (clickhouse+0x967d536)
1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse+0x1b25c7d9)
2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x1b25c7d9)
3 std::__1::unique_lock<std::__1::mutex>::unique_lock(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:119:61 (clickhouse+0x15226c31)
4 DB::StorageMergeTree::selectPartsToMutate(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&) obj-x86_64-linux->
5 DB::StorageMergeTree::scheduleDataProcessingJob(DB::IBackgroundJobExecutor&) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:1060:28 (clickhouse+0x15228b10)
6 DB::BackgroundJobsExecutor::scheduleJob() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:229:17 (clickhouse+0x14f441dc)
7 DB::IBackgroundJobExecutor::backgroundTaskFunction() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:185:10 (clickhouse+0x14f438ed)
[1]: https://clickhouse-test-reports.s3.yandex.net/27248/4f3b80ff33c846465983aa2bc9ae9490e1118b15/fuzzer_tsan/report.htmlfail1
2021-08-08 19:28:29 +03:00
Azat Khuzhin
7964355ecf
Fix mutation stuck on invalid partitions in non-replicated MergeTree
...
v2: Do not try to process empty mutations
Found with flaky check [1].
[1]: https://clickhouse-test-reports.s3.yandex.net/27248/66e8c0833392c20ba8dba3780f2b0d5c18f8194e/functional_stateless_tests_flaky_check_(address).html#fail1
2021-08-08 19:28:15 +03:00
mergify[bot]
c696817a79
Merge branch 'master' into control_execution_period_of_clearOldTemporaryDirectories
2021-08-06 16:16:22 +00:00
Anton Popov
e36736b50c
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-08-02 22:52:02 +03:00
Alexander Tokmakov
c74bfbf991
Merge branch 'master' into mvcc_prototype
2021-07-28 22:21:48 +03:00
Nikolai Kochetov
61d8f880cd
Rename some files.
2021-07-26 19:48:25 +03:00