thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
132,860 Commits 9,328 Branches 1,783 Tags 2.1 GiB
97e0e5fc9c
Commit Graph

712 Commits

Author SHA1 Message Date
Anton Popov
2adf0e9db3 do not wait loading parts at shutdown 2022-12-28 16:07:14 +00:00
Alexander Tokmakov
ca989e9212 less runtime format strings 2022-12-23 19:50:34 +01:00
Anton Popov
0722020cae Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-23 18:15:55 +00:00
alesapin
2fecf420cb
Merge pull request #44513 from ClickHouse/small_polishing 
Minor code polishing
 2022-12-23 13:47:26 +01:00
alesapin
a32fab90d5
Merge pull request #44436 from ClickHouse/fix_bug_with_max_task_count_update 
Fix bug with merge/mutate pool size increase
 2022-12-22 23:48:54 +01:00
alesapin
fa467b3398 Minor code polishing 2022-12-22 14:31:42 +01:00
alesapin
2818520572 Fix bug with merge/mutate pool size increase 2022-12-20 14:51:11 +01:00
alesapin
c8d9ccf3cd Lock table for share during startup for database ordinary 2022-12-19 18:51:46 +01:00
Alexander Tokmakov
f4c8281ccc avoid outdating part twice 2022-12-13 21:24:26 +01:00
Anton Popov
0b396bab14 Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-12 18:08:27 +00:00
Sema Checherinda
77d4f052ea do not clear old parts at startup also 2022-12-05 16:05:14 +01:00
Sema Checherinda
dc52a40c04 decrease stop max-tries back to default, do not clear old parts at shutdown 2022-12-05 16:05:14 +01:00
Sema Checherinda
fd2c7657be do not read part status just for logging 2022-12-01 16:08:05 +01:00
Anton Popov
df6d2f2163 Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-29 23:21:23 +00:00
Anton Popov
c65be55da1 Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-24 14:30:23 +00:00
Sema Checherinda
eccb62e4c3 work with the latest remarks 2022-11-23 15:16:09 +00:00
Sema Checherinda
e54ca534b0 test 01167_isolation_hermitage uses fallback to recreate table 2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208 fix isolation breach, droping outdated part should't be ignored 2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25 fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part 2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104 less pedantic checks in operations, rely on existed tx functionality 2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0 no groupping parts for cover with empty 2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a fix several critical races, fix tests 2022-11-23 15:16:08 +00:00
Sema Checherinda
f2b9f80f2f fix tidy build 2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070 address the review comments 2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9 Delete parts over replacing them with empty parts 2022-11-23 15:16:08 +00:00
Anton Popov
75c807609b add some comments 2022-11-22 16:11:01 +00:00
Anton Popov
e3db29aebf better waiting for outdated data parts 2022-11-22 15:15:20 +00:00
Anton Popov
92ac9385b9 wait loading of outdated parts for partition commands 2022-11-21 17:01:46 +00:00
Anton Popov
f94182384a Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-16 03:30:29 +00:00
Alexey Milovidov
4966556f47 Miscellaneous changes 2022-11-14 06:41:51 +01:00
Alexander Tokmakov
e415fc6e95
Revert "Optimize TTL merge, completely expired parts can be removed in time" 2022-11-10 19:38:55 +03:00
zhongyuankai
c1e2b2af74 Optimize TTL merge, completely expired parts can be removed in time 2022-11-01 22:58:15 +08:00
Anton Popov
c4d4f2dbed better interface 2022-10-25 22:14:06 +00:00
Anton Popov
c8199bc125
Merge branch 'master' into refactor-data-part 2022-10-25 14:31:05 +02:00
Anton Popov
cf375c9732 better interface 2022-10-23 22:29:24 +00:00
Anton Popov
b40d9200d2 better semantic of constsness of DataPartStorage 2022-10-23 15:24:20 +00:00
Anton Popov
56e5daba0c remove DataPartStorageBuilder 2022-10-23 00:23:15 +00:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Alexander Tokmakov
0d8a814d80
Merge pull request #42254 from xinhuitian/optimize-selectPartsToMutate 
Early return of corner cases in selectPartsToMutate function
 2022-10-20 18:13:37 +03:00
Tian Xinhui
45c6974f7d
Use alias for std::shared_ptr<MergeMutateSelectedEntry> (#42211) 
* use alias for MergeMutateSelectedEntry share ptr

* fix StorageMergeTree.cpp
 2022-10-18 13:35:34 +02:00
Anton Popov
95fdb0a877 remove outdated parts asynchronously 2022-10-14 14:52:57 +00:00
xinhuitian
daaf01c344 early return 2022-10-12 12:18:17 +08:00
alesapin
695133cc5b Better mutations 2022-09-27 15:23:02 +02:00
Anton Popov
ee54451741 fix adding of columns of type Object 2022-09-13 22:43:59 +00:00
Vitaly Baranov
122009a2bd Use table lock if database is ordinary and zero-copy-replication is enabled. 2022-09-08 13:54:59 +02:00
Alexey Milovidov
940a53e519
Merge pull request #40984 from Lucky-Chang/typo_fix 
Fix some typos and clang-tidy warnings
 2022-09-06 02:37:50 +03:00
Luck-Chang
1ac8e739c9 fix some typos and clang-tidy warnings 2022-09-05 09:50:24 +08:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Alexander Tokmakov
0aea4c7184 fix 2022-08-25 14:22:27 +02:00
Alexander Tokmakov
3cd26aafe4 remove wrong code for skipping mutations in MergeTree 2022-08-24 20:58:59 +02:00
Alexander Tokmakov
fe572104aa fix old tmp dirs cleanup 2022-08-09 18:44:51 +02:00
Alexander Gololobov
25deba2c1b
Merge branch 'master' into feature/sql-standard-delete 2022-07-25 22:13:20 +02:00
Alexander Gololobov
1ea9f143ff Leave only _row_exists-based implementation of lightweight delete 2022-07-21 11:26:13 +02:00
Raúl Marín
aea045f297 Improve logging around replicated merges 2022-07-15 15:48:35 +02:00
jianmei zhang
4268ac8b0e Add supportLightweightDeleteMutate() in IMergeTreeDataPart to disable LWD for part with projections 2022-07-15 12:32:42 +08:00
jianmei zhang
198674e994 Consider lightweight delete as ordinary for compact part in selectPartsToMutate() 2022-07-15 12:32:42 +08:00
jianmei zhang
780cdfb8f0 Code changes based on review opinions. Make ordinary single alter delete lightwight by default 2022-07-15 12:32:42 +08:00
jianmei zhang
b4a37e1e22 Disable optimizations for count() when lightweight delete exists, add hasLightweightDelete() function in IMergeTreeDataPart 2022-07-15 12:32:41 +08:00
jianmei zhang
8696319d62 Support lightweight delete execution using string as deleted rows mask,also part of select can handle LWD 2022-07-15 12:32:41 +08:00
jianmei zhang
8df7b7a030 Support new mutation type for lightweight 2022-07-15 12:32:41 +08:00
Amos Bird
0795cdd36a
Slightly better interface of waitForMutation 2022-07-13 12:34:12 +08:00
Vitaly Baranov
ed27987646
Merge pull request #38861 from vitlibar/backup-improvements-9 
Backup Improvements 9
 2022-07-07 02:24:47 +02:00
Vitaly Baranov
7f84cf3968 Fix style. 2022-07-06 16:36:59 +02:00
Vitaly Baranov
43d35eec1b Write unfinished mutations to backup. 2022-07-05 14:51:09 +02:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder 2022-07-02 15:39:44 +02:00
alesapin
0d7298f3a9 Fix typos 2022-07-01 15:26:27 +02:00
alesapin
57284e6a9d Fix possible deadlocks with MergeTreeData::Transaction 2022-07-01 15:16:32 +02:00
alesapin
d88cdcd5c1 AAA 2022-06-27 21:41:29 +02:00
alesapin
80d27f6381
Merge pull request #38380 from ClickHouse/simplify_parts_commit_merge_tree 
Simplify parts commit methods
 2022-06-25 00:12:52 +02:00
alesapin
011d58d7a0 Simplify more 2022-06-24 17:19:59 +02:00
alesapin
37310dc9df Simpler 2022-06-24 14:10:15 +02:00
alesapin
36b810b076 Remove unused parameter 2022-06-24 13:42:36 +02:00
alesapin
af1a9d18ab Remove transaction argument 2022-06-24 13:34:00 +02:00
alesapin
9910395823 Simplify method signature 2022-06-24 13:19:29 +02:00
alesapin
0f8eed98c4
Merge branch 'master' into better-support-gcp 2022-06-22 12:46:01 +02:00
Nikolai Kochetov
cc6fdfe0eb
Merge pull request #36555 from ClickHouse/refactor-something-in-part-volumes 
Separate data storage abstraction for MergeTree
 2022-06-22 11:13:36 +02:00
Anton Popov
d5d568e235
Merge pull request #37913 from azat/parts-refcnt 
Fix refcnt for unused MergeTree parts in SELECT queries
 2022-06-21 22:56:26 +02:00
Nikolai Kochetov
854d148d73 Add some comments. Remove some commented code. 2022-06-21 12:31:02 +00:00
Nikolai Kochetov
1e8c9ecd4c Merge branch 'master' into refactor-something-in-part-volumes 2022-06-21 12:37:21 +02:00
Nikolai Kochetov
dccf90b1ea Cleanup. 2022-06-20 18:18:17 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 18:11:30 +00:00
Nikolai Kochetov
7452d04e3a Merge branch 'master' into refactor-something-in-part-volumes 2022-06-20 15:31:02 +02:00
Alexander Tokmakov
83adf56383 fix race 2022-06-17 18:13:57 +02:00
mergify[bot]
7f24574609
Merge branch 'master' into better-support-gcp 2022-06-15 23:18:20 +00:00
Vitaly Baranov
5cabdbd982 Restore parts of MergeTree in correct order. 2022-06-15 20:28:40 +02:00
Nikolai Kochetov
2a9a63ac7b Merge branch 'master' into refactor-something-in-part-volumes 2022-06-15 15:35:26 +02:00
Anton Popov
a59be0fd5d better support of GCP storage 2022-06-15 00:23:45 +00:00
kssenii
4859e11c27 Review fixes 2022-06-12 01:15:10 +02:00
kssenii
fd6c84b831 Review fixes 2022-06-10 14:23:13 +02:00
kssenii
7a2676c7ab Clean up broken detached parts with timeout 2022-06-10 12:27:57 +02:00
Nikolai Kochetov
5bc9b32025 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-08 11:10:06 +00:00
Azat Khuzhin
61a2613127 Fix refcnt for unused MergeTree parts in SELECT queries 
Before this patch SELECT queries hold parts even if they were not
required by select (had been eliminated by partition pruning).

This defers removing parts if you have long running queries.

This had been introduced in #23932, with introduction of
StorageSnapshotPtr.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-08 07:04:53 +03:00
alesapin
e32d36d790 Proper fix 2022-06-07 17:58:32 +02:00
Nikolai Kochetov
678d978acf Merge branch 'master' into refactor-something-in-part-volumes 2022-06-07 15:23:00 +00:00
Nikolai Kochetov
89c5855d20 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-02 12:19:07 +02:00
Nikolai Kochetov
edac3d6714 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 09:36:20 +00:00
lthaooo
6632616733
Fix TTL merge scheduling bug (#36387) 2022-06-01 21:09:53 +02:00
Nikolai Kochetov
86fbb74703 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-31 18:07:47 +00:00
Nikolai Kochetov
c71256ea38 Remove some commented code. 2022-05-30 13:18:20 +00:00
Nikolai Kochetov
5b4658aa5e Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-30 09:47:35 +00:00
alesapin
be1c3c132b Fix some trash 2022-05-27 16:08:49 +02:00
Alexander Tokmakov
eb71dd4c78
Merge pull request #37547 from ClickHouse/followup_37398 
Follow-up to #37398
 2022-05-26 20:29:41 +03:00
Alexander Tokmakov
e8f33fb0d9 fix flaky tests 2022-05-26 14:17:05 +02:00
Azat Khuzhin
dc9ca3d70c
Fix LOGICAL_ERROR in getMaxSourcePartsSizeForMerge during merges (#37413) 2022-05-26 14:14:58 +02:00
Nikolai Kochetov
fd97a9d885 Move some resources 2022-05-23 19:47:32 +00:00
Nikolai Kochetov
fc3645a483 Fix some tests. 2022-05-03 19:32:24 +00:00
Nikolai Kochetov
35095191eb Merge branch 'master' into refactor-something-in-part-volumes 2022-05-03 17:51:47 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Nikolai Kochetov
e44af67fee Merge branch 'master' into refactor-something-in-part-volumes 2022-04-26 21:08:00 +02:00
Vitaly Baranov
68a020ecea Implement BACKUP/RESTORE ON CLUSTER. 2022-04-25 16:34:33 +02:00
Nikolai Kochetov
8c00692844 Part 8 2022-04-22 16:58:09 +00:00
Nikolai Kochetov
9133e398b8 Part 7 2022-04-21 19:19:13 +00:00
alesapin
5465415751 Fix replace/move partition with zero copy replication 2022-04-21 14:39:12 +02:00
alesapin
40c15222f8 Merge branch 'master' into fix_trash 2022-04-20 12:45:49 +02:00
alesapin
7cb7c120cc Less ugly 2022-04-19 15:53:10 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
Alexander Tokmakov
66fdf35dfd remove outdated parts immediately on drop partition 2022-04-13 18:01:22 +02:00
Alexander Tokmakov
49c35f3261 Merge branch 'master' into mvcc_prototype 2022-04-08 13:34:40 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements 
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
 2022-04-08 12:06:26 +02:00
Alexander Tokmakov
6a46da93ae Merge branch 'master' into mvcc_prototype 2022-04-07 23:22:19 +02:00
jewisliu
77edd41b2e [Improvement] improvement in PARTITION ALL 
1. ASTPartition::formatImpl should output ALL while executing ALTER TABLE t DETACH PARTITION ALL
2. prohibit PARTITION ALL excepte DETACH PARTITION ALL
 2022-04-07 17:37:01 +08:00
Alexander Tokmakov
1fe50ad201 Merge branch 'master' into mvcc_prototype 2022-04-05 14:38:02 +02:00
Vladimir C
54b4049ea5
Merge pull request #35794 from awakeljw/fork_chmaster 2022-04-05 09:44:45 +02:00
kssenii
6c8e073a61 Merge master 2022-04-01 16:48:29 +02:00
jewisliu
9a4cc78dfb support ALTER TABLE t DETACH PARTITION ALL syntax 2022-04-01 13:13:07 +08:00
Alexander Tokmakov
287d858fda Merge branch 'master' into mvcc_prototype 2022-03-29 16:24:12 +02:00
kssenii
d2a3cfe5dc Cache on all write operations 2022-03-23 19:14:33 +01:00
kssenii
c2c7f365ce Merge master 2022-03-23 16:58:35 +01:00
kssenii
8c1be8950e Improve test 2022-03-23 16:55:18 +01:00
taiyang-li
8dbf1c60e7 merge master and fix conflict 2022-03-23 11:36:50 +08:00
Alexander Tokmakov
3c762f566d Merge branch 'master' into mvcc_prototype 2022-03-21 20:16:29 +01:00
Vitaly Baranov
ce25afb2e9 Storages and databases are hollow by default now. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
7f89b98308 Rework BackupSettings and RestoreSettings a little, pass StorageRestoreSettings to storages. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
258a472001 Shorter names: rename IRestoreFromBackupTask -> IRestoreTask. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
7a63feb3f7 Make restore tasks explicit. 2022-03-20 20:01:31 +01:00
Alexander Tokmakov
418d52044a try fix merge assignment 2022-03-18 20:31:44 +01:00
Alexander Tokmakov
c28008a2df more comments, minor fixes 2022-03-18 12:01:26 +01:00
Alexander Tokmakov
07d952b728 use snapshots for semistructured data, durability fixes 2022-03-17 18:26:18 +01:00
Alexander Tokmakov
d04dc03fa4 Merge branch 'master' into mvcc_prototype 2022-03-17 15:24:32 +01:00
Alexander Tokmakov
c2ac8d4a5c review fixes 2022-03-16 21:05:34 +01:00
Alexander Tokmakov
278d779a01 log cleanup, more comments 2022-03-14 21:43:34 +01:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Alexander Tokmakov
061fa6a6f2 Merge branch 'master' into mvcc_prototype 2022-03-10 13:13:04 +01:00
Azat Khuzhin
caffc144b5 Fix possible "Part directory doesn't exist" during INSERT 
In #33291 final part commit had been defered, and now it can take
significantly more time, that may lead to "Part directory doesn't exist"
error during INSERT:

    2022.02.21 18:18:06.979881 [ 11329 ] {insert} <Debug> executeQuery: (from 127.1:24572, user: default) INSERT INTO db.table (...) VALUES
    2022.02.21 20:58:03.933593 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18044_18044_0 to 20220214_270654_270654_0.
    2022.02.21 21:16:50.961917 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18197_18197_0 to 20220214_270689_270689_0.
    ...
    2022.02.22 21:16:57.632221 [ 64878 ] {} <Warning> db.table: Removing temporary directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/
    ...
    2022.02.23 12:23:56.277480 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18232_18232_0 to 20220214_273459_273459_0.
    2022.02.23 12:23:56.299218 [ 11329 ] {insert} <Error> executeQuery: Code: 107. DB::Exception: Part directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/ doesn't exist. Most likely it is a logical error. (FILE_DOESNT_EXIST) (version 22.2.1.1) (from 127.1:24572) (in query: INSERT INTO db.table (...) VALUES), Stack trace (when copying this message, always include the lines below):

Follow-up for: #28760
Refs: #33291

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 07:44:11 +03:00
taiyang-li
b4174b0bef merge master and fix conflicts 2022-03-08 11:39:25 +08:00
Alexander Tokmakov
8acfb8d27f Merge branch 'master' into mvcc_prototype 2022-03-07 17:40:15 +01:00
Anton Popov
df3b07fe7c Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-03 22:25:28 +00:00
Amos Bird
d4cdf04683
Add missing locks (#34025) 2022-03-02 16:23:29 +01:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
Alexander Tokmakov
e37ef4560c fix 2022-02-15 18:00:45 +03:00
Alexander Tokmakov
ae5aa8c12d write part version before other files 2022-02-15 02:24:51 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
taiyang-li
b6132d490f merge master and solve conflict 2022-02-08 15:24:59 +08:00
Alexander Tokmakov
fe30e0f162 fixes 2022-02-03 21:57:09 +03:00
Alexander Tokmakov
ca5f951558 Merge branch 'master' into mvcc_prototype 2022-02-03 18:56:44 +03:00
alesapin
25375bc76b Remove unused param 2022-02-03 11:21:19 +03:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
Alexander Tokmakov
5fad3fdffc throw exception on non-transactional queries 2022-02-01 01:27:55 +03:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
Alexander Tokmakov
e0304c2a58 review fixes, write tid into mutation entry 2022-01-28 20:47:37 +03:00
taiyang-li
73def8b483 merge master and solve conflict 2022-01-24 11:01:43 +08:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001 Merge branch 'master' into mvcc_prototype 2022-01-21 18:59:01 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts 
Fix flushing of in-memory parts
 2022-01-21 02:18:03 +03:00
Alexander Tokmakov
e9a5a64a71 Merge branch 'master' into mvcc_prototype 2022-01-19 21:41:23 +03:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog 
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
 2022-01-10 22:01:41 +03:00
Alexander Tokmakov
158fbaaa29 fix 2022-01-10 20:07:01 +03:00
Alexander Tokmakov
bf2b6c2c37 Merge branch 'master' into mvcc_prototype 2022-01-10 19:53:50 +03:00
taiyang-li
cf413f16a8 remove function checkPartMetadataCache 2022-01-07 18:37:08 +08:00
Alexander Tokmakov
187c43eba8 rename Committed state to Active 2021-12-30 23:45:38 +03:00
Anton Popov
4ebf61b809 proper checking of atomic flags 2021-12-29 01:03:55 +03:00
Anton Popov
a83832c03e fix flushing of in-memory parts 2021-12-27 18:54:28 +03:00
Alexander Tokmakov
df12fdf612 fix tests 2021-12-22 19:34:02 +03:00
Alexander Tokmakov
a15444031c add test 2021-12-20 21:53:05 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
Alexander Tokmakov
32e62ed5c2 Merge branch 'master' into mvcc_prototype 2021-12-17 10:43:46 +03:00
Alexander Tokmakov
e185ad260b fix skipping of some mutations 2021-12-15 21:28:24 +03:00
Alexander Tokmakov
1b02136941 fix skipping of some mutations 2021-12-15 21:19:29 +03:00
Alexander Tokmakov
d7ad72838c Merge branch 'master' into mvcc_prototype 2021-12-14 23:07:52 +03:00
Alexander Tokmakov
6e1c16c2e7 add support for mutations 2021-12-14 23:06:34 +03:00
nautaa
0da3881ff4 flush all InMemoryDataParts when wal is not enabled 2021-12-14 16:31:17 +08:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
Alexander Tokmakov
7fcb79ae72 Merge branch 'master' into mvcc_prototype 2021-12-07 14:39:29 +03:00
Anton Popov
6f4d9a53b2 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-12-01 15:54:33 +03:00
Vladimir Chebotarev
e8e3411ebc Minor fixes for StorageMergeTree. 2021-12-01 11:18:07 +03:00
tavplubix
78224ef273
Merge pull request #31738 from ClickHouse/fix_segfault_on_attach_partition 
Fix segfault on attach parititon
 2021-11-30 14:32:50 +03:00
Vladimir Chebotarev
100e61ab0e Rebase fix. 2021-11-29 10:15:00 +03:00
Vladimir Chebotarev
bb2f21b1ad Rebase fix. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
e414fa01e0 Rebase fix. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
c6e73d7b60 Synced clearOldMutations() with rest of changes. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
ec1f909d74 Fixed wrong lock order. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
12f36206cf Deadlock fix. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
84ffcb479b Additional comments. 2021-11-25 17:11:52 +03:00
Vladimir Chebotarev
e0a453aadf Minor fix. 2021-11-25 17:11:29 +03:00
Vladimir Chebotarev
863f3565ad Test fixes. 2021-11-25 17:11:28 +03:00
Vladimir Chebotarev
390416b24d Rebase fix. 2021-11-25 17:11:28 +03:00
Vladimir Chebotarev
30cd133418 Fixed duplication commands bug. 2021-11-25 17:11:28 +03:00
Vladimir Chebotarev
aa3eecbd6d Fixed bug with frozen mutations. 2021-11-25 17:11:27 +03:00
Vladimir Chebotarev
6599124dba Attempt to skip mutations of different partitions in StorageMergeTree. 2021-11-25 17:10:11 +03:00
Alexander Tokmakov
1ab8b5d4df fix segfault on attach parititon 2021-11-25 13:04:39 +03:00
Anton Popov
ccd78e3838 Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-22 17:19:35 +03:00
tavplubix
67753e5b83
Update StorageMergeTree.cpp 2021-11-19 20:05:45 +03:00
Alexander Tokmakov
8d7413b0d6 remove strange multimap from mutations 2021-11-19 19:54:43 +03:00
Alexander Tokmakov
52885db5d7 remove strange multimap from mutations 2021-11-19 19:51:03 +03:00
Alexander Tokmakov
0a4647f927 support alter partition 2021-11-17 21:14:14 +03:00
Alexander Tokmakov
672157b817 Merge branch 'master' into mvcc_prototype 2021-11-11 22:01:34 +03:00
Anton Popov
a20922b2d3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-11-09 15:36:25 +03:00
Alexander Tokmakov
51bd98294e better merge selecting backoff 2021-11-09 15:26:51 +03:00
Alexander Tokmakov
92eec74ad7 Merge branch 'master' into mvcc_prototype 2021-11-06 21:08:36 +03:00
tavplubix
ac4a9bcf23
Fix 00975_move_partition_merge_tree (#30717) 
* Revert "Fix race between MOVE PARTITION and merges/mutations for MergeTree"

This reverts commit e9ce859b02.

* fix

* Update StorageMergeTree.cpp

Co-authored-by: alesapin <alesapin@gmail.com>
 2021-10-27 18:19:41 +03:00
Alexander Tokmakov
2e7e195e77 change alter_lock to std::timed_mutex 2021-10-26 13:37:00 +03:00
alexey-milovidov
7beffb04ad
Merge pull request #29495 from ClickHouse/improvement_for_strange_mutations 
Minor improvement for strange mutations
 2021-10-17 20:57:06 +03:00
Nikolai Kochetov
bfcbf5abe0 Merge branch 'master' into removing-data-streams-folder 2021-10-17 10:42:37 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
Azat Khuzhin
07e8b2b3c7 Do not try to remove temporary paths that is currently in written by merge/mutation 
v2: rebase against MergeTask
v3: rebase due to conflicts in src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
v4:
- rebase due to conflicts in src/Storages/MergeTree/MergeTask.cpp
- drop common/scope_guard_safe.h (not used)
 2021-10-16 00:43:52 +03:00
Azat Khuzhin
e9ce859b02 Fix race between MOVE PARTITION and merges/mutations for MergeTree 
From time to time 00975_move_partition_merge_tree test got failed, the
reason is that there is merge in progress that writes data to the source
table again:

<details>

    2021.10.12 13:20:40.243839 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> executeQuery: (from [::1]:38060) (comment: 00975_move_partition_merge_tree.sql) ALTER TABLE test_move_partition_src MOVE PARTITION 1 TO TABLE test_move_partition_dest;
    2021.10.12 13:20:40.244482 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_2_2_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_21_21_0
    ...
    2021.10.12 13:20:40.373487 [ 378 ] {} <Trace> test_t1tzb2.test_move_partition_src (467d7145-b47e-444e-867d-7145b47ea44e) (MergerMutator): Merged 6 parts: from 1_2_2_0 to 1_11_11_0
    ...
    2021.10.12 13:20:40.379750 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_15_15_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_28_28_0

</details>

And also remove cleaning of mutations since this will cause deadlock
after doing MOVE PARTITION under
currently_processing_in_background_mutex.

CI: https://clickhouse-test-reports.s3.yandex.net/0/a59c6b1c8eb47ebf77189a491ee0c3980b38e91c/functional_stateless_tests_
Fixes: 00975_move_partition_merge_tree
 2021-10-13 00:14:24 +03:00
Alexander Tokmakov
5365cc686f Merge branch 'master' into improvement_for_strange_mutations 2021-10-07 13:26:58 +03:00
Alexander Tokmakov
bb32432943 more fixes 2021-10-05 00:13:18 +03:00
Azat Khuzhin
117e9e77c8 Apply max_untracked_memory/memory_profiler_{step,sample_probability} during mutate/merge 2021-10-03 17:39:07 +03:00
Nikita Mikhaylov
9756b8dc33
Added an ability to execute more merges and mutations than threads, added new settings (#29140) 2021-10-01 00:26:24 +03:00
Nikolai Kochetov
236d71ea94
Merge pull request #28582 from ClickHouse/rewrite-pushing-to-views 
Rewrite PushingToViews
 2021-09-27 21:19:11 +03:00
Amos Bird
34017c942e
More fixes 2021-09-25 13:38:56 +08:00
Alexander Tokmakov
4114b82937 hotfixes for merge task 2021-09-23 23:15:49 +03:00
Alexander Tokmakov
72b1b2e360 Merge branch 'master' into mvcc_prototype 2021-09-23 22:53:27 +03:00
Nikolai Kochetov
a790d391c0 Merge branch 'master' into rewrite-pushing-to-views 2021-09-20 19:43:15 +03:00
Vladimir Chebotarev
db516e8c91
Removed sub-optimal mutation notifications in StorageMergeTree when merges are still going (#27552) 
* Removed redundand mutation notifications in `StorageMergeTree` when merges are still going.
 2021-09-17 16:19:36 +03:00
Nikolai Kochetov
a8c3b02598 Merge branch 'master' into rewrite-pushing-to-views 2021-09-17 15:38:11 +03:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask (#25165) 
Introduced MergeTask and MutateTask
 2021-09-17 00:19:58 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
kssenii
0c013ed91b Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-web-disk-2 2021-09-09 15:26:36 +03:00
Anton Popov
4c388e3d84 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-09-09 14:10:16 +03:00
Nikolai Kochetov
999a4fe831 Fix other tests. 2021-09-08 21:29:38 +03:00
Nikita Mikhaylov
6062dd0021 Better 2021-09-08 00:21:21 +00:00
Nikita Mikhaylov
ea0fbf81af Renaming 2021-09-06 12:01:16 +00:00
kssenii
97e0a0310a Slightly better 2021-09-04 12:02:07 +03:00
Nikita Mikhaylov
cc7c221fad Own PriorityQueue + prettifying the code 2021-09-02 21:31:32 +00:00
Nikita Mikhaylov
c4416906c8 done 2021-08-30 19:37:03 +00:00
kssenii
b536653851 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-24 07:29:50 +00:00
kssenii
5307d31924 Throw for alter and silence for drop 2021-08-23 23:48:53 +00:00
Vitaly Baranov
63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00
Anton Popov
61239343e3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-20 16:33:30 +03:00
alexey-milovidov
54d8282209
Merge pull request #27248 from azat/mt-mutation-stuck-fix 
Fix mutation stuck on invalid partitions in non-replicated MergeTree
 2021-08-11 06:10:15 +03:00
Azat Khuzhin
91d7f3daa7 Fix lock-order-inversion while notifying about finished mutations 
TSAN reports [1]:

  WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=36)
    Cycle in lock order graph: M16388 (0x7b7400011d68) => M1030334152907497744 (0x000000000000) => M16388

    Mutex M1030334152907497744 acquired here while holding mutex M16388 in thread T4:
      0 pthread_mutex_lock <null> (clickhouse+0x967d536)
      1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse+0x1b25c7d9)
      2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x1b25c7d9)
      3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x15220cd9)
      4 DB::StorageMergeTree::getIncompleteMutationsStatus(long, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std:>
      5 DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0::operator()() const obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:464:36 (clickhouse+0x1521e2b1)
      6 void std::__1::condition_variable::wait<DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0>(std::__1::unique_lock<std::__1::mutex>&, DB::StorageMergeTree::waitForMutation(lon>
      7 DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:469:29 (clickhouse+0x1521e2b1)
      8 DB::StorageMergeTree::mutate(DB::MutationCommands const&, std::__1::shared_ptr<DB::Context const>) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:496:9 (clickhouse+0x15221738)
      9 DB::InterpreterAlterQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterAlterQuery.cpp:113:16 (clickhouse+0x141182f6)
      10 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:560:32 (clickhouse+0x149152f6)
      11 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:909:30 (clickhous>
      12 DB::TCPHandler::runImpl() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:313:24 (clickhouse+0x153270af)

      Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

    Mutex M16388 acquired here while holding mutex M1030334152907497744 in thread T59:
      0 pthread_mutex_lock <null> (clickhouse+0x967d536)
      1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse+0x1b25c7d9)
      2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x1b25c7d9)
      3 std::__1::unique_lock<std::__1::mutex>::unique_lock(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:119:61 (clickhouse+0x15226c31)
      4 DB::StorageMergeTree::selectPartsToMutate(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&) obj-x86_64-linux->
      5 DB::StorageMergeTree::scheduleDataProcessingJob(DB::IBackgroundJobExecutor&) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:1060:28 (clickhouse+0x15228b10)
      6 DB::BackgroundJobsExecutor::scheduleJob() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:229:17 (clickhouse+0x14f441dc)
      7 DB::IBackgroundJobExecutor::backgroundTaskFunction() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:185:10 (clickhouse+0x14f438ed)

  [1]: https://clickhouse-test-reports.s3.yandex.net/27248/4f3b80ff33c846465983aa2bc9ae9490e1118b15/fuzzer_tsan/report.htmlfail1
 2021-08-08 19:28:29 +03:00
Azat Khuzhin
7964355ecf Fix mutation stuck on invalid partitions in non-replicated MergeTree 
v2: Do not try to process empty mutations
    Found with flaky check [1].

      [1]: https://clickhouse-test-reports.s3.yandex.net/27248/66e8c0833392c20ba8dba3780f2b0d5c18f8194e/functional_stateless_tests_flaky_check_(address).html#fail1
 2021-08-08 19:28:15 +03:00
mergify[bot]
c696817a79
Merge branch 'master' into control_execution_period_of_clearOldTemporaryDirectories 2021-08-06 16:16:22 +00:00
Anton Popov
e36736b50c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-02 22:52:02 +03:00
Alexander Tokmakov
c74bfbf991 Merge branch 'master' into mvcc_prototype 2021-07-28 22:21:48 +03:00