Han Fei
4e7745f757
Merge branch 'master' into hanfei/statistic
2023-09-26 21:43:53 +02:00
Anton Popov
0ebc22e851
Merge branch 'master' into long-column-names
2023-09-25 16:55:14 +02:00
Nikolai Kochetov
cf40723179
Merge pull request #54895 from amosbird/allow-nullable-primary-keys-for-projections
...
Always allow nullable keys for projections
2023-09-22 19:09:24 +02:00
Amos Bird
973cd5e972
Always allow nullable keys for projections
2023-09-21 22:46:16 +08:00
Anton Popov
4d7ecf39ac
check file collisions on all alters
2023-09-21 14:16:03 +00:00
Azat Khuzhin
c706101891
Fix throttling of BACKUPs from/to S3 (in case native copy was not used)
...
In some cases native copy is not possible, and such requests should be
throttled.
v0: copyS3FileNativeWithFallback
v2: revert v0 and pass write_settings
v3: pass read_settings to copyFile()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-09-20 18:28:43 +02:00
Han Fei
17243b84ab
Merge branch 'master' into hanfei/statistic
2023-09-19 17:09:17 +02:00
Anton Popov
624782bd8f
Merge remote-tracking branch 'upstream/master' into HEAD
2023-09-18 23:13:20 +00:00
Anton Popov
d94c1b1ec9
add check for collisions in filenames
2023-09-18 22:27:04 +00:00
alesapin
f3729da64b
Fix rare case of CHECKSUM_DOESNT_MATCH error
2023-09-12 19:55:56 +02:00
Vitaly Baranov
8060407c70
Add backup setting "check_parts" and restore setting "restore_broken_parts_as_detached".
2023-09-04 18:54:03 +02:00
Han Fei
f60dad0598
Merge branch 'master' into hanfei/statistic
2023-09-04 16:17:07 +02:00
Han Fei
2330a28f7b
support estimate by stat
2023-09-04 15:51:00 +02:00
Alexey Milovidov
7cb4d52d92
Deprecate metadata cache, part 2
2023-09-03 03:56:11 +02:00
Azat Khuzhin
51e8b16916
Fix IO throttling during copying whole directories
...
v2: pass local context during ALTER TABLE MOVE PARTITION TO VOLUME/DISK as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-29 23:13:46 +02:00
Raúl Marín
45e8185f9b
Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation
2023-08-29 11:17:34 +02:00
alesapin
3a289534cb
Fxi
2023-08-24 20:48:10 +02:00
alesapin
415b3826e0
Add event for tracking data parts lock
2023-08-24 15:28:38 +02:00
Alexander Tokmakov
376202f739
fix creation of empty parts
2023-08-15 00:26:17 +02:00
Vitaly Baranov
e127d27453
Disable using fs cache for backup/restore #2 .
2023-08-03 13:52:35 +02:00
Raúl Marín
974f72800f
Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation
2023-07-26 10:35:40 +02:00
Azat Khuzhin
d7f7f16fbc
Introduce IStorage::supportsTrivialCountOptimization()
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-24 05:50:44 +02:00
Igor Nikonov
2d33661594
Merge branch 'master' into fix-potentially-bad-code
2023-07-22 22:48:07 +02:00
Nikolai Kochetov
b5cf644668
Check projection metadata the same way we check ordinary metadata. ( #52361 )
...
* Check projection metadata the same way we check ordinary metadata.
* Allow aggregate projection to have empty PK
---------
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2023-07-21 16:23:04 +03:00
Alexey Milovidov
8cd2e7c7d6
Merge branch 'master' into fix-potentially-bad-code
2023-07-18 22:18:22 +02:00
Raúl Marín
d5c332327d
Start cleaning up things
2023-07-11 13:57:51 +02:00
Raúl Marín
f5e0c1bd25
Interpreter: Control number of parallel replicas based on row estimation
2023-07-11 13:57:51 +02:00
Alexey Milovidov
e2c8cea381
Merge pull request #51801 from ClickHouse/fix_broken_parts_handling
...
Fix broken parts handling in `ReplicatedMergeTree`
2023-07-09 05:32:51 +03:00
Alexander Tokmakov
fdbd59ec6c
Merge pull request #51135 from ClickHouse/fix_trash_in_disks
...
Try to fix some trash in Disks and part moves
2023-07-07 02:02:17 +03:00
Alexey Milovidov
89788f3d5b
Merge branch 'master' into fix-potentially-bad-code
2023-07-06 02:23:28 +03:00
Alexey Milovidov
69d8923c4c
Merge branch 'master' into fix-potentially-bad-code
2023-07-05 00:46:24 +02:00
Alexey Milovidov
3371e038cf
Merge pull request #51498 from ClickHouse/vdimir/test_alter_moving_garbage_51342
...
Fix `test_alter_moving_garbage`: lock between getActiveContainingPart and swapActivePart in parts mover
2023-07-05 00:32:21 +03:00
Alexander Tokmakov
42222662e6
Merge branch 'master' into fix_broken_parts_handling
2023-07-04 22:46:35 +02:00
Alexander Tokmakov
d987b94ed4
fix the way how broken parts are detached
2023-07-04 20:51:15 +02:00
Alexander Tokmakov
594ec09edf
fix missing metadata_version.txt
2023-06-28 18:19:16 +02:00
vdimir
0b6d367bdb
Parts mover: lock between getActiveContainingPart and swapActivePart
2023-06-27 17:14:49 +00:00
Alexey Milovidov
27f41869a9
Remove code that I don't like
2023-06-25 09:11:42 +02:00
Alexander Tokmakov
220520c516
fix
2023-06-22 21:45:10 +02:00
Alexander Tokmakov
a00842498e
Merge branch 'master' into rmt_better_background_tasks_scheduling
2023-06-19 12:37:17 +02:00
Alexander Tokmakov
b4e455430c
Merge branch 'master' into rmt_better_background_tasks_scheduling
2023-06-17 22:19:21 +03:00
Alexey Milovidov
688d5d59af
Merge branch 'master' into relax-too-many-parts
2023-06-16 23:08:29 +02:00
Nikolai Kochetov
df1ea0b572
Merge pull request #50489 from ClickHouse/vdimir/alter_moving_garbage
...
Cleanup moving parts
2023-06-16 15:39:32 +02:00
Alexander Tokmakov
9260a1bf2e
Merge pull request #51006 from ClickHouse/followup_50448
...
Follow-up to #50448
2023-06-16 15:32:12 +03:00
Alexander Tokmakov
a018d9ca11
try to fix false-positive 'part is lost forever'
2023-06-15 12:25:16 +02:00
Vitaly Baranov
6366940a37
Use hash_of_all_files from system.parts to check identity of parts during on-cluster backups.
2023-06-14 20:39:50 +02:00
Anton Popov
32caf87163
Merge pull request #50104 from amosbird/fix_43107_47549
...
Proper mutation of skip indices and projections
2023-06-12 15:09:18 +02:00
Alexey Milovidov
598501011f
Relax "too many parts" further
2023-06-11 17:51:54 +02:00
vdimir
44c68ffdab
Add config param allow_remove_stale_moving_parts
2023-06-08 10:17:02 +00:00
Alexander Tokmakov
46caffd249
Merge branch 'master' into rmt_better_background_tasks_scheduling
2023-06-07 23:30:02 +03:00
Nikita Mikhaylov
e1ceb01ad0
Parallel replicas for short queries ( #50639 )
2023-06-07 19:45:27 +02:00
vdimir
db1c03d6db
Cleanup moving parts
2023-06-07 13:57:22 +00:00
Alexander Tokmakov
d2aa1779ed
Merge branch 'master' into rmt_better_background_tasks_scheduling
2023-06-07 14:58:50 +02:00
Nikita Mikhaylov
e87348010d
Rework loading and removing of data parts for MergeTree tables. ( #49474 )
...
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
2023-06-06 14:42:56 +02:00
Amos Bird
97bd3f0483
Proper mutation of skip indices and projections
2023-06-04 18:32:08 +08:00
Alexander Tokmakov
e148c60d5a
Fixes for MergeTree with readonly disks ( #50244 )
...
* fixes for MergeTree with readonly disks
* Automatic style fix
* Update test.py
* Automatic style fix
* Update test.py
* Update test.py
* Automatic style fix
* Update test.py
---------
Co-authored-by: robot-clickhouse <robot-clickhouse@users.noreply.github.com>
Co-authored-by: alesapin <alesapin@clickhouse.com>
2023-06-02 16:18:32 +03:00
Alexander Tokmakov
82e1d5ae67
Merge branch 'master' into rmt_better_background_tasks_scheduling
2023-05-30 17:40:13 +02:00
Anton Popov
612173e734
refactoring near alter conversions
2023-05-25 22:54:54 +00:00
Alexander Tokmakov
b8305503d8
more flexible cleanup thread scheduling
2023-05-22 19:07:18 +02:00
Amos Bird
0a3d986e42
Fix reporting projection broken part
2023-05-21 20:58:58 +08:00
Vitaly Baranov
943707963f
Add backup setting "decrypt_files_from_encrypted_disks"
2023-05-16 14:27:27 +02:00
Alexey Milovidov
75a2589a42
Merge branch 'master' into deprecate-in-memory-parts
2023-05-06 00:49:59 +03:00
Anton Popov
e61d7baa54
Merge remote-tracking branch 'upstream/master' into HEAD
2023-05-04 14:03:53 +00:00
Alexey Milovidov
261907c143
Merge branch 'master' into deprecate-in-memory-parts
2023-05-04 01:07:38 +02:00
Nikita Mikhaylov
954e3b724c
Speedup outdated parts loading ( #49317 )
2023-05-03 18:56:45 +02:00
Alexey Milovidov
f449df85b6
Deprecate in-memory parts
2023-05-03 00:31:09 +02:00
Anton Popov
bcdd2307fb
add comments
2023-05-02 11:30:50 +00:00
Anton Popov
c7db58be66
add MergeTree setting async_insert
2023-04-25 01:11:58 +00:00
Anton Popov
38622d0770
add settings to delay or throw in case of too many mutations
2023-04-24 18:21:49 +00:00
凌涛
08e81d0a52
Merge branch 'master' into feature/support_settings_allow_suspicious_indices
2023-04-23 10:08:00 +08:00
凌涛
de18bf72b4
modify code
2023-04-21 17:27:26 +08:00
凌涛
a975211185
add more tests
2023-04-18 10:26:35 +08:00
凌涛
7ed4c413e3
add settings allow_suspicious_indices
2023-04-17 20:23:19 +08:00
alesapin
a72ee0b85e
Merge remote-tracking branch 'origin/master' into revert-46909-revert-45911-mutations_rename_hang
2023-04-11 20:14:43 +02:00
Anton Popov
5b8db57fe8
Merge branch 'master' into retry-loading-of-parts
2023-04-11 15:51:40 +02:00
Anton Popov
4fb7e0ec3c
add retries to loading of data parts
2023-04-05 21:23:57 +00:00
Azat Khuzhin
218b1f9c29
Add ability to throttle BACKUPs on per-server/backup basis
...
Server settings:
- backup_read_bandwidth_for_server
- backup_write_bandwidth_for_server
Query settings:
- backup_read_bandwidth
- backup_write_bandwidth
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-05 09:39:48 +02:00
Alexander Tokmakov
c114361879
Update MergeTreeData.h
2023-03-21 23:15:59 +03:00
alesapin
fc913649cf
Check only for interactive moves
2023-03-15 22:51:53 +01:00
alesapin
ef7508469c
Fix wait for zero-copy replication move during manual move
2023-03-15 22:29:24 +01:00
alesapin
915b4ea56a
Merge branch 'master' into revert-46909-revert-45911-mutations_rename_hang
2023-03-02 14:29:21 +01:00
Alexey Milovidov
97c0f8a0e0
Merge pull request #47027 from amosbird/clang16
...
Prepare to build with clang 16
2023-03-02 02:10:03 +03:00
alesapin
46f25d53b3
Merge branch 'master' into revert-46909-revert-45911-mutations_rename_hang
2023-03-01 20:09:12 +01:00
Amos Bird
c0fb4ddc3e
Make thread analyzer happy
2023-03-01 11:07:36 +08:00
alesapin
abd6500ee3
Better exception message when move was unseccessful
2023-02-28 17:49:16 +01:00
alesapin
e3961c118a
Revert "Revert "Fix several RENAME COLUMN bugs.""
2023-02-27 12:27:57 +01:00
Alexey Milovidov
8f40bbd5b4
Revert "Fix several RENAME COLUMN bugs."
2023-02-26 05:47:05 +03:00
Alexey Milovidov
eb89ab471f
Merge pull request #45911 from ClickHouse/mutations_rename_hang
...
Fix several `RENAME COLUMN` bugs.
2023-02-26 01:56:40 +03:00
Vladimir C
a791250a4a
Merge pull request #38614 from bharatnc/ncb/profile-events-partlog
2023-02-21 11:06:44 +01:00
youennL-cs
6526c2a8ab
[RFC] Replacing merge tree new engine ( #41005 )
...
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree
* Add new test for the new ReplacingMergeTree engine
* Limit sign value to -1/1
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree
* Add new test for the new ReplacingMergeTree engine
* Limit sign value to -1/1
* Replace sign column(Int8) by is_deleted(UInt8)
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree
* Add new test for the new ReplacingMergeTree engine
* Limit sign value to -1/1
* Replace sign column(Int8) by is_deleted(UInt8)
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree
* Add new test for the new ReplacingMergeTree engine
* Limit sign value to -1/1
* Replace sign column(Int8) by is_deleted(UInt8)
* Add keyword 'CLEANUP' when OPTIMIZE
* Cleanup uniquely when it's a replacingMergeTree
* Propagate CLEANUP information and change from 'with_cleanup' to 'cleanup'
* Cleanup data flagged as 'is_deleted'
* Fix merge when optimize and add a test
* Fix OPTIMIZE and INSERT + add tests
* New fix for cleanup at the merge
* Cleanup debug logs
* Add the SETTINGS option 'clean_deleted_rows' that can be 'never' or 'always'
* Fix regression bug; Now REplicatedMergeTree can be called as before without 'is_deleted'
* Add Replicated tests
* Disable tag 'long' for our test and cleanup some white spaces
* Update tests
* Fix tests and remove additional useless whitespace
* Fix replica test
* Style clean && add condition check for is_deleted values
* clean_deleted_rows settings is nom an enum
* Add valid default value to the clean_deleted_rows settings
* Update cleanup checkers to use the enum and fix typos in the test
* Fix submodule contrib/AMQP-CPP pointer
* Add missing messages in test reference and remove a print with non derterministic order
* fix replica test reference
* Fix edge case
* Fix a typo for the spell checker
* Fix reference
* Fix a condition to raise an error if is_deleted differ from 0/1 and cleanup
* Change tests file name and update number
* This should fix the ReplacingMergeTree parameter set
* Fix replicated parameters
* Disable allow_deprecated_syntax_for_merge_tree for our new column
* Fix a test
* Remove non deterministic order print in the test
* Test on replicas
* Remove a condition, when checking optional parameters, that should not be sueful since we disabled the deprected_syntaxe
* Revert "Remove a condition, when checking optional parameters, that should not be useful since we disabled the deprected_syntaxe"
This reverts commit b65d64c05e
2023-02-16 16:03:16 +03:00
Alexey Milovidov
f554ff9d99
Remove trash
2023-02-14 14:11:08 +00:00
John Skopis
f7604cc686
[feat] Add ProfileEvents map to PartLog
...
closes #10316
2023-02-14 14:11:02 +00:00
alesapin
1683cd5d6f
Merge branch 'master' into mutations_rename_hang
2023-02-12 13:06:48 +01:00
KitKatKKK
fa73b00a66
Update src/Storages/MergeTree/MergeTreeData.h
...
Co-authored-by: flynn <fenglv15@mails.ucas.ac.cn>
2023-02-10 16:46:57 +08:00
attack204
7f41173a1f
rename API
2023-02-10 16:24:10 +08:00
attack204
93e92dab3b
rename API
2023-02-10 16:05:50 +08:00
attack204
52960d4e7d
add API to MergeTreeData
2023-02-10 11:00:11 +08:00
alesapin
b265300a19
Merge branch 'master' into mutations_rename_hang
2023-02-06 12:25:29 +01:00
alesapin
7a0d0f0c53
Some code which doesn't work
2023-02-03 19:14:49 +01:00
Anton Popov
8ca602a148
check dynamic columns of part before its commit
2023-02-03 02:58:28 +00:00
alesapin
bdc530dead
Fix style
2023-02-02 17:30:38 +01:00
Anton Popov
8e3698c91f
refactoring of code near merge tree parts
2023-01-25 17:34:09 +00:00
Sergei Trifonov
0d1ea05ff6
Merge pull request #45007 from ClickHouse/cancellable-mutex-integration
...
Fast shared mutex integration
2023-01-25 11:15:46 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Sergei Trifonov
0fbfa17863
Merge branch 'master' into cancellable-mutex-integration
2023-01-23 12:44:09 +01:00
Anton Popov
6863cd152f
Merge pull request #42181 from CurtizJ/optimize-loading-parts
...
Do not load inactive parts at startup
2023-01-16 14:38:50 +01:00
serxa
693489a8ad
review fixes
2023-01-12 15:51:04 +00:00
Sergei Trifonov
12d8543578
Merge branch 'master' into cancellable-mutex-integration
2023-01-12 16:03:49 +01:00
Igor Nikonov
edbcb24da0
Merge pull request #44954 from ClickHouse/fix-insert-delay
...
Fix: insert delay calculation
2023-01-12 09:43:00 +01:00
Nikolai Kochetov
4673b3fe1d
Revert "Revert "Custom reading for mutation""
2023-01-10 16:31:01 +01:00
Alexander Tokmakov
c8ec130be4
Revert "Custom reading for mutation"
2023-01-10 17:51:30 +03:00
Anton Popov
b9bf92fa5d
check for network errors while loading parts
2023-01-10 11:55:08 +00:00
Nikolai Kochetov
3e430b1b6e
Add comments. Remove unused flag from Context.
2023-01-09 19:52:11 +00:00
serxa
15bb127b01
replace every std::shared_mutex with DB::FastSharedMutex
2023-01-06 23:35:38 +00:00
Igor Nikonov
a67afdff6a
Fix: insert delay calculation
2023-01-05 22:43:41 +00:00
Anton Popov
8d2788445b
fix style check
2022-12-28 19:29:56 +00:00
Anton Popov
2adf0e9db3
do not wait loading parts at shutdown
2022-12-28 16:07:14 +00:00
Anton Popov
0722020cae
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-23 18:15:55 +00:00
alesapin
fa467b3398
Minor code polishing
2022-12-22 14:31:42 +01:00
Anton Popov
b7e76e3539
do not hold data parts during insert
2022-12-15 23:57:10 +00:00
Alexander Tokmakov
f4c8281ccc
avoid outdating part twice
2022-12-13 21:24:26 +01:00
Anton Popov
a2fee110b8
add query for waiting of loading parts and fix test_transactions
2022-12-12 14:33:42 +00:00
Anton Popov
df6d2f2163
Merge remote-tracking branch 'upstream/master' into HEAD
2022-11-29 23:21:23 +00:00
Alexander Tokmakov
5cc99312d7
provide hint for loading uncommitted blocks in merge predicate
2022-11-25 16:41:20 +01:00
Sema Checherinda
a29437e179
delete verbose logging about cleaning old parts
2022-11-23 15:16:09 +00:00
Sema Checherinda
3a8024002d
fix race in read full path from deleting part, add more logs in grabOldParts
2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208
fix isolation breach, droping outdated part should't be ignored
2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25
fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part
2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104
less pedantic checks in operations, rely on existed tx functionality
2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0
no groupping parts for cover with empty
2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a
fix several critical races, fix tests
2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070
address the review comments
2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9
Delete parts over replacing them with empty parts
2022-11-23 15:16:08 +00:00
Anton Popov
75c807609b
add some comments
2022-11-22 16:11:01 +00:00
Anton Popov
e3db29aebf
better waiting for outdated data parts
2022-11-22 15:15:20 +00:00
Anton Popov
92ac9385b9
wait loading of outdated parts for partition commands
2022-11-21 17:01:46 +00:00
Anton Popov
d96ed50610
fix race
2022-11-16 13:21:07 +00:00
Anton Popov
f94182384a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-11-16 03:30:29 +00:00
Anton Popov
2ae3cfa9e0
Merge branch 'master' into dynamic-columns-14
2022-10-31 16:15:19 +01:00
Alexander Tokmakov
599ccb9939
Update MergeTreeData.h
2022-10-31 16:18:17 +03:00
Alexander Tokmakov
b2b9479afa
Update src/Storages/MergeTree/MergeTreeData.h
...
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
2022-10-31 16:15:35 +03:00
Alexander Tokmakov
f53df7870c
fix race between drop and failed insert
2022-10-30 17:59:47 +01:00
Alexander Tokmakov
c68ab231f9
fix accessing part in Deleting state
2022-10-30 17:30:51 +01:00
Anton Popov
b40d9200d2
better semantic of constsness of DataPartStorage
2022-10-23 15:24:20 +00:00
Anton Popov
56e5daba0c
remove DataPartStorageBuilder
2022-10-23 00:23:15 +00:00
Anton Popov
95fdb0a877
remove outdated parts asynchronously
2022-10-14 14:52:57 +00:00
Anton Popov
71a223c1c9
Merge remote-tracking branch 'upstream/master' into optimize-loading-parts
2022-10-10 11:31:47 +00:00
Anton Popov
286da7ee19
fix loading of parts with old syntax
2022-10-09 15:39:20 +00:00
Anton Popov
9f1104c2ad
do not load inactive parts
2022-10-08 01:39:46 +00:00
Alexey Milovidov
169c505603
Relax "too many parts" thresholds
2022-10-08 00:24:51 +02:00
Anton Popov
6e61cf92f5
Merge remote-tracking branch 'upstream/master' into HEAD
2022-10-03 13:16:57 +00:00
