thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 18:32:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
114,056 Commits 9,328 Branches 1,783 Tags 2.1 GiB
f3f6cfdcbb
Commit Graph

580 Commits

Author SHA1 Message Date
Anton Popov
c4d4f2dbed better interface 2022-10-25 22:14:06 +00:00
Anton Popov
c8199bc125
Merge branch 'master' into refactor-data-part 2022-10-25 14:31:05 +02:00
Anton Popov
cf375c9732 better interface 2022-10-23 22:29:24 +00:00
Anton Popov
b40d9200d2 better semantic of constsness of DataPartStorage 2022-10-23 15:24:20 +00:00
Anton Popov
56e5daba0c remove DataPartStorageBuilder 2022-10-23 00:23:15 +00:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Alexander Tokmakov
0d8a814d80
Merge pull request #42254 from xinhuitian/optimize-selectPartsToMutate 
Early return of corner cases in selectPartsToMutate function
 2022-10-20 18:13:37 +03:00
Tian Xinhui
45c6974f7d
Use alias for std::shared_ptr<MergeMutateSelectedEntry> (#42211) 
* use alias for MergeMutateSelectedEntry share ptr

* fix StorageMergeTree.cpp
 2022-10-18 13:35:34 +02:00
Anton Popov
95fdb0a877 remove outdated parts asynchronously 2022-10-14 14:52:57 +00:00
xinhuitian
daaf01c344 early return 2022-10-12 12:18:17 +08:00
alesapin
695133cc5b Better mutations 2022-09-27 15:23:02 +02:00
Anton Popov
ee54451741 fix adding of columns of type Object 2022-09-13 22:43:59 +00:00
Vitaly Baranov
122009a2bd Use table lock if database is ordinary and zero-copy-replication is enabled. 2022-09-08 13:54:59 +02:00
Alexey Milovidov
940a53e519
Merge pull request #40984 from Lucky-Chang/typo_fix 
Fix some typos and clang-tidy warnings
 2022-09-06 02:37:50 +03:00
Luck-Chang
1ac8e739c9 fix some typos and clang-tidy warnings 2022-09-05 09:50:24 +08:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Alexander Tokmakov
0aea4c7184 fix 2022-08-25 14:22:27 +02:00
Alexander Tokmakov
3cd26aafe4 remove wrong code for skipping mutations in MergeTree 2022-08-24 20:58:59 +02:00
Alexander Tokmakov
fe572104aa fix old tmp dirs cleanup 2022-08-09 18:44:51 +02:00
Alexander Gololobov
25deba2c1b
Merge branch 'master' into feature/sql-standard-delete 2022-07-25 22:13:20 +02:00
Alexander Gololobov
1ea9f143ff Leave only _row_exists-based implementation of lightweight delete 2022-07-21 11:26:13 +02:00
Raúl Marín
aea045f297 Improve logging around replicated merges 2022-07-15 15:48:35 +02:00
jianmei zhang
4268ac8b0e Add supportLightweightDeleteMutate() in IMergeTreeDataPart to disable LWD for part with projections 2022-07-15 12:32:42 +08:00
jianmei zhang
198674e994 Consider lightweight delete as ordinary for compact part in selectPartsToMutate() 2022-07-15 12:32:42 +08:00
jianmei zhang
780cdfb8f0 Code changes based on review opinions. Make ordinary single alter delete lightwight by default 2022-07-15 12:32:42 +08:00
jianmei zhang
b4a37e1e22 Disable optimizations for count() when lightweight delete exists, add hasLightweightDelete() function in IMergeTreeDataPart 2022-07-15 12:32:41 +08:00
jianmei zhang
8696319d62 Support lightweight delete execution using string as deleted rows mask,also part of select can handle LWD 2022-07-15 12:32:41 +08:00
jianmei zhang
8df7b7a030 Support new mutation type for lightweight 2022-07-15 12:32:41 +08:00
Amos Bird
0795cdd36a
Slightly better interface of waitForMutation 2022-07-13 12:34:12 +08:00
Vitaly Baranov
ed27987646
Merge pull request #38861 from vitlibar/backup-improvements-9 
Backup Improvements 9
 2022-07-07 02:24:47 +02:00
Vitaly Baranov
7f84cf3968 Fix style. 2022-07-06 16:36:59 +02:00
Vitaly Baranov
43d35eec1b Write unfinished mutations to backup. 2022-07-05 14:51:09 +02:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder 2022-07-02 15:39:44 +02:00
alesapin
0d7298f3a9 Fix typos 2022-07-01 15:26:27 +02:00
alesapin
57284e6a9d Fix possible deadlocks with MergeTreeData::Transaction 2022-07-01 15:16:32 +02:00
alesapin
d88cdcd5c1 AAA 2022-06-27 21:41:29 +02:00
alesapin
80d27f6381
Merge pull request #38380 from ClickHouse/simplify_parts_commit_merge_tree 
Simplify parts commit methods
 2022-06-25 00:12:52 +02:00
alesapin
011d58d7a0 Simplify more 2022-06-24 17:19:59 +02:00
alesapin
37310dc9df Simpler 2022-06-24 14:10:15 +02:00
alesapin
36b810b076 Remove unused parameter 2022-06-24 13:42:36 +02:00
alesapin
af1a9d18ab Remove transaction argument 2022-06-24 13:34:00 +02:00
alesapin
9910395823 Simplify method signature 2022-06-24 13:19:29 +02:00
alesapin
0f8eed98c4
Merge branch 'master' into better-support-gcp 2022-06-22 12:46:01 +02:00
Nikolai Kochetov
cc6fdfe0eb
Merge pull request #36555 from ClickHouse/refactor-something-in-part-volumes 
Separate data storage abstraction for MergeTree
 2022-06-22 11:13:36 +02:00
Anton Popov
d5d568e235
Merge pull request #37913 from azat/parts-refcnt 
Fix refcnt for unused MergeTree parts in SELECT queries
 2022-06-21 22:56:26 +02:00
Nikolai Kochetov
854d148d73 Add some comments. Remove some commented code. 2022-06-21 12:31:02 +00:00
Nikolai Kochetov
1e8c9ecd4c Merge branch 'master' into refactor-something-in-part-volumes 2022-06-21 12:37:21 +02:00
Nikolai Kochetov
dccf90b1ea Cleanup. 2022-06-20 18:18:17 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 18:11:30 +00:00
Nikolai Kochetov
7452d04e3a Merge branch 'master' into refactor-something-in-part-volumes 2022-06-20 15:31:02 +02:00
Alexander Tokmakov
83adf56383 fix race 2022-06-17 18:13:57 +02:00
mergify[bot]
7f24574609
Merge branch 'master' into better-support-gcp 2022-06-15 23:18:20 +00:00
Vitaly Baranov
5cabdbd982 Restore parts of MergeTree in correct order. 2022-06-15 20:28:40 +02:00
Nikolai Kochetov
2a9a63ac7b Merge branch 'master' into refactor-something-in-part-volumes 2022-06-15 15:35:26 +02:00
Anton Popov
a59be0fd5d better support of GCP storage 2022-06-15 00:23:45 +00:00
kssenii
4859e11c27 Review fixes 2022-06-12 01:15:10 +02:00
kssenii
fd6c84b831 Review fixes 2022-06-10 14:23:13 +02:00
kssenii
7a2676c7ab Clean up broken detached parts with timeout 2022-06-10 12:27:57 +02:00
Nikolai Kochetov
5bc9b32025 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-08 11:10:06 +00:00
Azat Khuzhin
61a2613127 Fix refcnt for unused MergeTree parts in SELECT queries 
Before this patch SELECT queries hold parts even if they were not
required by select (had been eliminated by partition pruning).

This defers removing parts if you have long running queries.

This had been introduced in #23932, with introduction of
StorageSnapshotPtr.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-08 07:04:53 +03:00
alesapin
e32d36d790 Proper fix 2022-06-07 17:58:32 +02:00
Nikolai Kochetov
678d978acf Merge branch 'master' into refactor-something-in-part-volumes 2022-06-07 15:23:00 +00:00
Nikolai Kochetov
89c5855d20 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-02 12:19:07 +02:00
Nikolai Kochetov
edac3d6714 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 09:36:20 +00:00
lthaooo
6632616733
Fix TTL merge scheduling bug (#36387) 2022-06-01 21:09:53 +02:00
Nikolai Kochetov
86fbb74703 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-31 18:07:47 +00:00
Nikolai Kochetov
c71256ea38 Remove some commented code. 2022-05-30 13:18:20 +00:00
Nikolai Kochetov
5b4658aa5e Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-30 09:47:35 +00:00
alesapin
be1c3c132b Fix some trash 2022-05-27 16:08:49 +02:00
Alexander Tokmakov
eb71dd4c78
Merge pull request #37547 from ClickHouse/followup_37398 
Follow-up to #37398
 2022-05-26 20:29:41 +03:00
Alexander Tokmakov
e8f33fb0d9 fix flaky tests 2022-05-26 14:17:05 +02:00
Azat Khuzhin
dc9ca3d70c
Fix LOGICAL_ERROR in getMaxSourcePartsSizeForMerge during merges (#37413) 2022-05-26 14:14:58 +02:00
Nikolai Kochetov
fd97a9d885 Move some resources 2022-05-23 19:47:32 +00:00
Nikolai Kochetov
fc3645a483 Fix some tests. 2022-05-03 19:32:24 +00:00
Nikolai Kochetov
35095191eb Merge branch 'master' into refactor-something-in-part-volumes 2022-05-03 17:51:47 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Nikolai Kochetov
e44af67fee Merge branch 'master' into refactor-something-in-part-volumes 2022-04-26 21:08:00 +02:00
Vitaly Baranov
68a020ecea Implement BACKUP/RESTORE ON CLUSTER. 2022-04-25 16:34:33 +02:00
Nikolai Kochetov
8c00692844 Part 8 2022-04-22 16:58:09 +00:00
Nikolai Kochetov
9133e398b8 Part 7 2022-04-21 19:19:13 +00:00
alesapin
5465415751 Fix replace/move partition with zero copy replication 2022-04-21 14:39:12 +02:00
alesapin
40c15222f8 Merge branch 'master' into fix_trash 2022-04-20 12:45:49 +02:00
alesapin
7cb7c120cc Less ugly 2022-04-19 15:53:10 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
Alexander Tokmakov
66fdf35dfd remove outdated parts immediately on drop partition 2022-04-13 18:01:22 +02:00
Alexander Tokmakov
49c35f3261 Merge branch 'master' into mvcc_prototype 2022-04-08 13:34:40 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements 
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
 2022-04-08 12:06:26 +02:00
Alexander Tokmakov
6a46da93ae Merge branch 'master' into mvcc_prototype 2022-04-07 23:22:19 +02:00
jewisliu
77edd41b2e [Improvement] improvement in PARTITION ALL 
1. ASTPartition::formatImpl should output ALL while executing ALTER TABLE t DETACH PARTITION ALL
2. prohibit PARTITION ALL excepte DETACH PARTITION ALL
 2022-04-07 17:37:01 +08:00
Alexander Tokmakov
1fe50ad201 Merge branch 'master' into mvcc_prototype 2022-04-05 14:38:02 +02:00
Vladimir C
54b4049ea5
Merge pull request #35794 from awakeljw/fork_chmaster 2022-04-05 09:44:45 +02:00
kssenii
6c8e073a61 Merge master 2022-04-01 16:48:29 +02:00
jewisliu
9a4cc78dfb support ALTER TABLE t DETACH PARTITION ALL syntax 2022-04-01 13:13:07 +08:00
Alexander Tokmakov
287d858fda Merge branch 'master' into mvcc_prototype 2022-03-29 16:24:12 +02:00
kssenii
d2a3cfe5dc Cache on all write operations 2022-03-23 19:14:33 +01:00
kssenii
c2c7f365ce Merge master 2022-03-23 16:58:35 +01:00
kssenii
8c1be8950e Improve test 2022-03-23 16:55:18 +01:00
taiyang-li
8dbf1c60e7 merge master and fix conflict 2022-03-23 11:36:50 +08:00
Alexander Tokmakov
3c762f566d Merge branch 'master' into mvcc_prototype 2022-03-21 20:16:29 +01:00
Vitaly Baranov
ce25afb2e9 Storages and databases are hollow by default now. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
7f89b98308 Rework BackupSettings and RestoreSettings a little, pass StorageRestoreSettings to storages. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
258a472001 Shorter names: rename IRestoreFromBackupTask -> IRestoreTask. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
7a63feb3f7 Make restore tasks explicit. 2022-03-20 20:01:31 +01:00
Alexander Tokmakov
418d52044a try fix merge assignment 2022-03-18 20:31:44 +01:00
Alexander Tokmakov
c28008a2df more comments, minor fixes 2022-03-18 12:01:26 +01:00
Alexander Tokmakov
07d952b728 use snapshots for semistructured data, durability fixes 2022-03-17 18:26:18 +01:00
Alexander Tokmakov
d04dc03fa4 Merge branch 'master' into mvcc_prototype 2022-03-17 15:24:32 +01:00
Alexander Tokmakov
c2ac8d4a5c review fixes 2022-03-16 21:05:34 +01:00
Alexander Tokmakov
278d779a01 log cleanup, more comments 2022-03-14 21:43:34 +01:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Alexander Tokmakov
061fa6a6f2 Merge branch 'master' into mvcc_prototype 2022-03-10 13:13:04 +01:00
Azat Khuzhin
caffc144b5 Fix possible "Part directory doesn't exist" during INSERT 
In #33291 final part commit had been defered, and now it can take
significantly more time, that may lead to "Part directory doesn't exist"
error during INSERT:

    2022.02.21 18:18:06.979881 [ 11329 ] {insert} <Debug> executeQuery: (from 127.1:24572, user: default) INSERT INTO db.table (...) VALUES
    2022.02.21 20:58:03.933593 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18044_18044_0 to 20220214_270654_270654_0.
    2022.02.21 21:16:50.961917 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18197_18197_0 to 20220214_270689_270689_0.
    ...
    2022.02.22 21:16:57.632221 [ 64878 ] {} <Warning> db.table: Removing temporary directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/
    ...
    2022.02.23 12:23:56.277480 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18232_18232_0 to 20220214_273459_273459_0.
    2022.02.23 12:23:56.299218 [ 11329 ] {insert} <Error> executeQuery: Code: 107. DB::Exception: Part directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/ doesn't exist. Most likely it is a logical error. (FILE_DOESNT_EXIST) (version 22.2.1.1) (from 127.1:24572) (in query: INSERT INTO db.table (...) VALUES), Stack trace (when copying this message, always include the lines below):

Follow-up for: #28760
Refs: #33291

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 07:44:11 +03:00
taiyang-li
b4174b0bef merge master and fix conflicts 2022-03-08 11:39:25 +08:00
Alexander Tokmakov
8acfb8d27f Merge branch 'master' into mvcc_prototype 2022-03-07 17:40:15 +01:00
Anton Popov
df3b07fe7c Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-03 22:25:28 +00:00
Amos Bird
d4cdf04683
Add missing locks (#34025) 2022-03-02 16:23:29 +01:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
Alexander Tokmakov
e37ef4560c fix 2022-02-15 18:00:45 +03:00
Alexander Tokmakov
ae5aa8c12d write part version before other files 2022-02-15 02:24:51 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
taiyang-li
b6132d490f merge master and solve conflict 2022-02-08 15:24:59 +08:00
Alexander Tokmakov
fe30e0f162 fixes 2022-02-03 21:57:09 +03:00
Alexander Tokmakov
ca5f951558 Merge branch 'master' into mvcc_prototype 2022-02-03 18:56:44 +03:00
alesapin
25375bc76b Remove unused param 2022-02-03 11:21:19 +03:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
Alexander Tokmakov
5fad3fdffc throw exception on non-transactional queries 2022-02-01 01:27:55 +03:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
Alexander Tokmakov
e0304c2a58 review fixes, write tid into mutation entry 2022-01-28 20:47:37 +03:00
taiyang-li
73def8b483 merge master and solve conflict 2022-01-24 11:01:43 +08:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001 Merge branch 'master' into mvcc_prototype 2022-01-21 18:59:01 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts 
Fix flushing of in-memory parts
 2022-01-21 02:18:03 +03:00
Alexander Tokmakov
e9a5a64a71 Merge branch 'master' into mvcc_prototype 2022-01-19 21:41:23 +03:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog 
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
 2022-01-10 22:01:41 +03:00
Alexander Tokmakov
158fbaaa29 fix 2022-01-10 20:07:01 +03:00
Alexander Tokmakov
bf2b6c2c37 Merge branch 'master' into mvcc_prototype 2022-01-10 19:53:50 +03:00
taiyang-li
cf413f16a8 remove function checkPartMetadataCache 2022-01-07 18:37:08 +08:00
Alexander Tokmakov
187c43eba8 rename Committed state to Active 2021-12-30 23:45:38 +03:00
Anton Popov
4ebf61b809 proper checking of atomic flags 2021-12-29 01:03:55 +03:00
Anton Popov
a83832c03e fix flushing of in-memory parts 2021-12-27 18:54:28 +03:00
Alexander Tokmakov
df12fdf612 fix tests 2021-12-22 19:34:02 +03:00
Alexander Tokmakov
a15444031c add test 2021-12-20 21:53:05 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
Alexander Tokmakov
32e62ed5c2 Merge branch 'master' into mvcc_prototype 2021-12-17 10:43:46 +03:00
Alexander Tokmakov
e185ad260b fix skipping of some mutations 2021-12-15 21:28:24 +03:00
Alexander Tokmakov
1b02136941 fix skipping of some mutations 2021-12-15 21:19:29 +03:00
Alexander Tokmakov
d7ad72838c Merge branch 'master' into mvcc_prototype 2021-12-14 23:07:52 +03:00