Commit Graph

618 Commits

Author SHA1 Message Date
Azat Khuzhin
bac7def719 Fix UB (signed integer overflow) in StorageMergeTree::backupData()
UBsan report:

    /build/src/Storages/StorageMergeTree.cpp:2088:80: runtime error: signed integer overflow: 9223372036854775807 + 1 cannot be represented in type 'long'
        0 0x30c2ae7b in DB::StorageMergeTree::backupData(DB::BackupEntriesCollector&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<absl::lts_20211102::InlinedVector<std::__1::shared_ptr<DB::IAST>, 7ul, std::__1::allocator<std::__1::shared_ptr<DB::IAST>>>> const&) build_docker/./src/Storages/StorageMergeTree.cpp:2088:80
        1 0x2dd641b5 in DB::BackupEntriesCollector::makeBackupEntriesForTableData(DB::QualifiedTableName const&) build_docker/./src/Backups/BackupEntriesCollector.cpp:703:18
        2 0x2dd5c2a7 in DB::BackupEntriesCollector::makeBackupEntriesForTablesData() build_docker/./src/Backups/BackupEntriesCollector.cpp:676:9
        3 0x2dd5848d in DB::BackupEntriesCollector::run() build_docker/./src/Backups/BackupEntriesCollector.cpp:119:5
        4 0x2dd84da3 in DB::BackupsWorker::doBackup(std::__1::shared_ptr<DB::ASTBackupQuery> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::BackupInfo const&, DB::BackupSettings, std::__1::shared_ptr<DB::IBackupCoordination>, std::__1::shared_ptr<DB::Context const> const&, std::__1::shared_ptr<DB::Context>, bool) build_docker/./src/Backups/BackupsWorker.cpp:359:59
        5 0x2dd82405 in DB::BackupsWorker::startMakingBackup(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const> const&) build_docker/./src/Backups/BackupsWorker.cpp:248:13
        6 0x2dd81d0a in DB::BackupsWorker::start(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context>) build_docker/./src/Backups/BackupsWorker.cpp:179:16
        7 0x2f4d0d5f in DB::InterpreterBackupQuery::execute() build_docker/./src/Interpreters/InterpreterBackupQuery.cpp:39:30

    SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /build/src/Storages/StorageMergeTree.cpp:2088:80 in

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-03-31 12:29:31 +03:00
Nikita Mikhaylov
301e26b23d
Parallel replicas. Part 2.5 (#47858) 2023-03-22 13:05:38 +01:00
alesapin
8db0d26519
Merge branch 'master' into revert-46909-revert-45911-mutations_rename_hang 2023-03-13 14:48:20 +01:00
Maksim Kita
a762112e15 Analyzer support distributed JOINS and subqueries in IN functions 2023-03-11 11:51:54 +01:00
Dmitry Novik
4f24a7bd11 Merge remote-tracking branch 'origin/master' into background-memory-tracker 2023-03-10 18:57:45 +00:00
alesapin
4423be8332
Merge branch 'master' into revert-46909-revert-45911-mutations_rename_hang 2023-03-08 12:59:07 +01:00
Robert Schulze
740aeaba1f
Apply some CTAD 2023-03-02 13:36:47 +00:00
alesapin
db0f9b6578 Fix one more bug 2023-03-02 14:26:07 +01:00
alesapin
e3961c118a
Revert "Revert "Fix several RENAME COLUMN bugs."" 2023-02-27 12:27:57 +01:00
Alexey Milovidov
8f40bbd5b4
Revert "Fix several RENAME COLUMN bugs." 2023-02-26 05:47:05 +03:00
Alexey Milovidov
eb89ab471f
Merge pull request #45911 from ClickHouse/mutations_rename_hang
Fix several `RENAME COLUMN` bugs.
2023-02-26 01:56:40 +03:00
Vladimir C
a791250a4a
Merge pull request #38614 from bharatnc/ncb/profile-events-partlog 2023-02-21 11:06:44 +01:00
Alexey Milovidov
d8cda3dbb8 Remove PVS-Studio 2023-02-19 23:30:05 +01:00
Dmitry Novik
0336f0d8cc
Merge branch 'master' into background-memory-tracker 2023-02-16 20:57:44 +01:00
youennL-cs
6526c2a8ab
[RFC] Replacing merge tree new engine (#41005)
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add keyword 'CLEANUP' when OPTIMIZE

* Cleanup uniquely when it's a replacingMergeTree

* Propagate CLEANUP information and change from 'with_cleanup' to 'cleanup'

* Cleanup data flagged as 'is_deleted'

* Fix merge when optimize and add a test

* Fix OPTIMIZE and INSERT + add tests

* New fix for cleanup at the merge

* Cleanup debug logs

* Add the SETTINGS option 'clean_deleted_rows' that can be 'never' or 'always'

* Fix regression bug; Now REplicatedMergeTree can be called as before without 'is_deleted'

* Add Replicated tests

* Disable tag 'long' for our test and cleanup some white spaces

* Update tests

* Fix tests and remove additional useless whitespace

* Fix replica test

* Style clean && add condition check for is_deleted values

* clean_deleted_rows settings is nom an enum

* Add valid default value to the clean_deleted_rows settings

* Update cleanup checkers to use the enum and fix typos in the test

* Fix submodule contrib/AMQP-CPP pointer

* Add missing messages in test reference and remove a print with non derterministic order

* fix replica test reference

* Fix edge case

* Fix a typo for the spell checker

* Fix reference

* Fix a condition to raise an error if is_deleted differ from 0/1 and cleanup

* Change tests file name and update number

* This should fix the ReplacingMergeTree parameter set

* Fix replicated parameters

* Disable allow_deprecated_syntax_for_merge_tree for our new column

* Fix a test

* Remove non deterministic order print in the test

* Test on replicas

* Remove a condition, when checking optional parameters, that should not be sueful since we disabled the deprected_syntaxe

* Revert "Remove a condition, when checking optional parameters, that should not be useful since we disabled the deprected_syntaxe"

This reverts commit b65d64c05e.

* Fix replica management and limit the number of argument to two maximum, due to the possiblity of deprecated table create/attach failing otherwise

* Test a fix for replicated log information error

* Try to add sync to have consistent results

* Change path of replicas that should cause one issue and add few prints in case it's not that

* Get cleanup info on replicas only if information found

* Fix style issues

* Try to avoid replication error 'cannot select parts...' and and replica read/write field order

* Cleanup according to PR reviews
 and add tests on error raised.

* Update src/Storages/MergeTree/registerStorageMergeTree.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* Select ... FINAL don't show rows with is_deleted = true

* Update and fix SELECT ... FINAL merge parameter

* Remove is_deleted rows only on the version inserted when merge

* Fix (master) updates issues

* Revert changes that should not be commited

* Add changes according to review

* Revert changes that should not be commited - part 2

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-02-16 16:03:16 +03:00
Dmitry Novik
e7a94a28ee
Merge branch 'master' into background-memory-tracker 2023-02-15 19:39:28 +01:00
vdimir
4c1aeb7972
Use overload of PartLog::addNewPart with profile_events 2023-02-14 14:11:12 +00:00
alesapin
808a939ad2 Small rehacktoing 2023-02-10 14:25:19 +01:00
alesapin
fd52979991 Fix snapshot for ordinary merge tree 2023-02-09 14:56:13 +01:00
alesapin
bc58fd156b Merge remote-tracking branch 'origin' into mutations_rename_hang 2023-02-07 20:53:04 +01:00
Dmitry Novik
d43d17f59a Address the review 2023-02-07 16:05:02 +00:00
Dmitry Novik
1e065b32f3 Add MemoryTracker for the background tasks 2023-02-06 17:25:58 +00:00
Alexander Tokmakov
af655eb7dc
Merge pull request #45889 from CheSema/release-merge-blocker-drop-part
do not block merges when old parts are droping in drop queries
2023-02-06 19:09:09 +03:00
alesapin
a02e6f3d5b fix bug ordinary merge tree 2023-02-06 14:57:21 +01:00
alesapin
b265300a19 Merge branch 'master' into mutations_rename_hang 2023-02-06 12:25:29 +01:00
alesapin
7a0d0f0c53 Some code which doesn't work 2023-02-03 19:14:49 +01:00
Alexander Tokmakov
3f11948bb0
Merge branch 'master' into stack_trace_in_part_log 2023-02-03 20:05:00 +03:00
Nikita Mikhaylov
33877b5e00
Parallel replicas. Part [2] (#43772) 2023-02-03 14:34:18 +01:00
alesapin
15ddcdc4a6 Fix clang tidy 2023-02-03 12:34:47 +01:00
Frank Chen
d38adfab30 Merge two overridden functions as one 2023-02-03 15:27:45 +08:00
Frank Chen
d3a05a11da Merge remote-tracking branch 'remotes/github/master' into stack_trace_in_part_log 2023-02-03 11:39:43 +08:00
alesapin
b8b0f1c3d3 Make RENAME COLUMN barrier 2023-02-02 21:11:19 +01:00
alesapin
bdc530dead Fix style 2023-02-02 17:30:38 +01:00
alesapin
dea46e58fb Fixes for ordinary merge tree 2023-02-02 17:03:24 +01:00
Sema Checherinda
44be0f41dd
Merge branch 'master' into release-merge-blocker-drop-part 2023-02-02 14:12:40 +01:00
Sema Checherinda
db2aa48c91 do not block merges when old parts are droping in drop queries 2023-02-01 13:30:56 +01:00
Alexander Gololobov
863548114d force_wait flag is no longer needed 2023-01-30 18:38:28 +01:00
Anton Popov
8e3698c91f refactoring of code near merge tree parts 2023-01-25 17:34:09 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449)
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
2023-01-24 00:13:58 +03:00
Sema Checherinda
962894afc8
Merge pull request #44909 from CheSema/intersect-prev-part
Do not merge over a gap with outdated undeleted parts
2023-01-21 11:51:21 +01:00
Sema Checherinda
09f3a5c599 add a comment, add a check, fix test 2023-01-20 12:10:31 +01:00
Frank Chen
f867c4c206 Add an overriden fromCurrentException to simply the code 2023-01-20 16:43:10 +08:00
Frank Chen
38799eb852 Merge two methods as one 2023-01-20 16:35:28 +08:00
Frank Chen
be8ceeb4bb Save stack trace in part_log 2023-01-20 16:22:40 +08:00
Sema Checherinda
a344b526a6
Update src/Storages/StorageMergeTree.cpp
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-01-18 15:16:18 +01:00
Alexander Tokmakov
5cd90c1a3e Merge branch 'master' into exception_message_patterns 2023-01-17 20:04:04 +01:00
Alexander Tokmakov
72e8615bec formatting of some exception messages 2023-01-17 20:03:56 +01:00
Alexander Tokmakov
8b13b85ea0
Merge pull request #44543 from ClickHouse/text_log_add_pattern
Add a column with a message pattern to text_log
2023-01-17 20:19:32 +03:00
Sema Checherinda
dbe89cd5d8 fix that optimize final waits for currently running merges 2023-01-16 16:47:12 +01:00
Sema Checherinda
90fa1ecd49 make that old_parts_lifetime=0 deletes files instantly at drop/truncate 2023-01-16 16:47:12 +01:00
Sema Checherinda
8f660afab3 style fix 2023-01-16 16:47:12 +01:00
Sema Checherinda
c51f4d7be1 do not merge over a gap with uotdate parts, delete empty parts with respect to old_parts_lifetime 2023-01-16 16:47:11 +01:00
Alexander Tokmakov
2d7773fccc Merge branch 'master' into text_log_add_pattern 2023-01-13 20:33:46 +01:00
Anton Popov
6c4f53bdca Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-11 01:00:57 +00:00
Anton Popov
fcdbc8123d better loading of outdated parts 2023-01-06 11:59:54 +00:00
Alexander Gololobov
2c64f0042a Added force_wait parameter to IStorage::mutate() 2022-12-30 16:23:03 +01:00
Anton Popov
2adf0e9db3 do not wait loading parts at shutdown 2022-12-28 16:07:14 +00:00
Alexander Tokmakov
ca989e9212 less runtime format strings 2022-12-23 19:50:34 +01:00
Anton Popov
0722020cae Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-23 18:15:55 +00:00
alesapin
2fecf420cb
Merge pull request #44513 from ClickHouse/small_polishing
Minor code polishing
2022-12-23 13:47:26 +01:00
alesapin
a32fab90d5
Merge pull request #44436 from ClickHouse/fix_bug_with_max_task_count_update
Fix bug with merge/mutate pool size increase
2022-12-22 23:48:54 +01:00
alesapin
fa467b3398 Minor code polishing 2022-12-22 14:31:42 +01:00
alesapin
2818520572 Fix bug with merge/mutate pool size increase 2022-12-20 14:51:11 +01:00
alesapin
c8d9ccf3cd Lock table for share during startup for database ordinary 2022-12-19 18:51:46 +01:00
Alexander Tokmakov
f4c8281ccc avoid outdating part twice 2022-12-13 21:24:26 +01:00
Anton Popov
0b396bab14 Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-12 18:08:27 +00:00
Sema Checherinda
77d4f052ea do not clear old parts at startup also 2022-12-05 16:05:14 +01:00
Sema Checherinda
dc52a40c04 decrease stop max-tries back to default, do not clear old parts at shutdown 2022-12-05 16:05:14 +01:00
Sema Checherinda
fd2c7657be do not read part status just for logging 2022-12-01 16:08:05 +01:00
Anton Popov
df6d2f2163 Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-29 23:21:23 +00:00
Anton Popov
c65be55da1 Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-24 14:30:23 +00:00
Sema Checherinda
eccb62e4c3 work with the latest remarks 2022-11-23 15:16:09 +00:00
Sema Checherinda
e54ca534b0 test 01167_isolation_hermitage uses fallback to recreate table 2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208 fix isolation breach, droping outdated part should't be ignored 2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25 fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part 2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104 less pedantic checks in operations, rely on existed tx functionality 2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0 no groupping parts for cover with empty 2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a fix several critical races, fix tests 2022-11-23 15:16:08 +00:00
Sema Checherinda
f2b9f80f2f fix tidy build 2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070 address the review comments 2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9 Delete parts over replacing them with empty parts 2022-11-23 15:16:08 +00:00
Anton Popov
75c807609b add some comments 2022-11-22 16:11:01 +00:00
Anton Popov
e3db29aebf better waiting for outdated data parts 2022-11-22 15:15:20 +00:00
Anton Popov
92ac9385b9 wait loading of outdated parts for partition commands 2022-11-21 17:01:46 +00:00
Anton Popov
f94182384a Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-16 03:30:29 +00:00
Alexey Milovidov
4966556f47 Miscellaneous changes 2022-11-14 06:41:51 +01:00
Alexander Tokmakov
e415fc6e95
Revert "Optimize TTL merge, completely expired parts can be removed in time" 2022-11-10 19:38:55 +03:00
zhongyuankai
c1e2b2af74 Optimize TTL merge, completely expired parts can be removed in time 2022-11-01 22:58:15 +08:00
Anton Popov
c4d4f2dbed better interface 2022-10-25 22:14:06 +00:00
Anton Popov
c8199bc125
Merge branch 'master' into refactor-data-part 2022-10-25 14:31:05 +02:00
Anton Popov
cf375c9732 better interface 2022-10-23 22:29:24 +00:00
Anton Popov
b40d9200d2 better semantic of constsness of DataPartStorage 2022-10-23 15:24:20 +00:00
Anton Popov
56e5daba0c remove DataPartStorageBuilder 2022-10-23 00:23:15 +00:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Alexander Tokmakov
0d8a814d80
Merge pull request #42254 from xinhuitian/optimize-selectPartsToMutate
Early return of corner cases in selectPartsToMutate function
2022-10-20 18:13:37 +03:00
Tian Xinhui
45c6974f7d
Use alias for std::shared_ptr<MergeMutateSelectedEntry> (#42211)
* use alias for MergeMutateSelectedEntry share ptr

* fix StorageMergeTree.cpp
2022-10-18 13:35:34 +02:00
Anton Popov
95fdb0a877 remove outdated parts asynchronously 2022-10-14 14:52:57 +00:00
xinhuitian
daaf01c344 early return 2022-10-12 12:18:17 +08:00
alesapin
695133cc5b Better mutations 2022-09-27 15:23:02 +02:00
Anton Popov
ee54451741 fix adding of columns of type Object 2022-09-13 22:43:59 +00:00