thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-09-20 08:40:50 +00:00
Code Issues Packages Projects Releases Wiki Activity
122,427 Commits 8,927 Branches 1,765 Tags 2 GiB
bf40767f10
Commit Graph

412 Commits

Author SHA1 Message Date
Vitaly Baranov
6af6247f8a
Merge pull request #52970 from arenadata/ADQM-985 
Add `storage_policy` parameter in the `SETTINGS` clause of `RESTORE` statements
 2023-08-07 19:01:08 +02:00
Victor Krasnov
a2f48e5bb3 Merge branch 'master' into ADQM-985 2023-08-04 19:02:18 +03:00
Daniel Pozo Escalona
ed43d6f780
Merge branch 'master' into s3_request_throttling_on_backups 2023-08-04 16:57:47 +02:00
Vitaly Baranov
6ac61b1fdd Fix tests. 2023-08-03 13:52:35 +02:00
Vitaly Baranov
556ecf5c1c Renamed backup setting "read_from_filesystem_cache_if_exists_otherwise_bypass_cache" -> "read_from_filesystem_cache". 2023-08-03 13:52:35 +02:00
Vitaly Baranov
e127d27453 Disable using fs cache for backup/restore #2. 2023-08-03 13:52:35 +02:00
Vitaly Baranov
f427c779eb Disable using fs cache for backup/restore. 2023-08-03 13:52:32 +02:00
Dani Pozo
fc7f47d083 Make it possible to throttle S3 requests on backups 2023-08-03 11:06:24 +02:00
Victor Krasnov
d8792bf429 Merge branch 'master' into ADQM-985 
# Conflicts:
#	docs/en/operations/backup.md
 2023-08-02 20:17:59 +03:00
Victor Krasnov
35941ddb92 Refine RestorerFromBackup::applyCustomStoragePolicy infinitesimally 2023-08-02 11:57:38 +03:00
robot-ch-test-poll4
5b67cff6b2
Merge pull request #52829 from ClickHouse/throw-s3-exception-whenever-possible 
Throw S3Exception whenever possible.
 2023-08-01 04:35:27 +02:00
Nikolai Kochetov
c4a00b8c68 Throw S3Exception whenever possible. 2023-07-31 15:39:20 +00:00
Victor Krasnov
04ad661968 Introduce clean-up of the source storage policy by setting it to an empty string 2023-07-27 18:17:47 +03:00
Victor Krasnov
eb689708a8 Fix typo 2023-07-26 15:26:38 +03:00
Victor Krasnov
0b5b0df58f Apply custom storage policy to the restored table 2023-07-26 15:24:35 +03:00
Victor Krasnov
06b782d37a Add storage_policy to RestoreSettings in order to allow it in the SETTINGS clause of RESTORE queries 2023-07-26 14:40:50 +03:00
Roman Vasin
2cf79ac592 Merge branch 'master' of github.com:ClickHouse/ClickHouse into ADQM-984 2023-07-26 09:58:26 +00:00
Roman Vasin
19c1f39286 Refactor code to support storage class only via settings 2023-07-21 05:59:25 +00:00
Roman Vasin
a3a576cd86 Add usage of in s3_storage_class in SETTINGS 2023-07-14 17:35:31 +00:00
Roman Vasin
c27f0691a8 Clean extra whitespace 2023-07-14 05:38:55 +00:00
Roman Vasin
6c3b7f76d4 Add S3 storage class as 4th parameter 2023-07-13 16:00:01 +00:00
Azat Khuzhin
1844ac37d7 Remove create_read_buffer argument for copyS3File() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-09 08:20:03 +02:00
Azat Khuzhin
559d328178 Rename BACKUP setting native_copy to allow_s3_native_copy 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-09 08:20:03 +02:00
Azat Khuzhin
5835e72fd6 More generic approach to disable native copy 
Previous patch implements this only for BACKUP/RESTORE, but it can be
useful for regular disks as well, so add allow_native_copy for disks.

Note, that there is s3_allow_native_copy query setting, since it looks
redundant, since it make sense only for S3 disks, and not on a per query
basis.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-09 08:20:02 +02:00
Azat Khuzhin
00cf66ab7a Add ability to disable native copy for BACKUP/RESTORE 
Native copy uses CopyObject for S3/GCS, but in GCS the CopyObject is
buggy - it does not always works.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-09 08:19:10 +02:00
Alexey Milovidov
b1d1918607
Merge branch 'master' into retry 2023-07-08 03:08:48 +03:00
Alexander Tokmakov
fdbd59ec6c
Merge pull request #51135 from ClickHouse/fix_trash_in_disks 
Try to fix some trash in Disks and part moves
 2023-07-07 02:02:17 +03:00
Alexey Milovidov
cca3f9937c
Merge branch 'master' into retry 2023-06-24 22:29:17 +03:00
Vitaly Baranov
1ec5304957
Fix multiple writing of same file to backup (#51299) 
* Fix writing the same file multiple times to backup.

* Add test.

---------

Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-06-23 12:11:18 +02:00
Michael Kolupaev
4a570a05c9 Decrease default timeouts for S3 and HTTP requests 2023-06-21 18:08:50 +00:00
Alexander Tokmakov
60a0843f58 fix 2023-06-20 17:47:25 +02:00
Vitaly Baranov
f7638de213
Fix crash in BackupCoordinationStageSync::setError(). (#51012) 2023-06-16 15:32:30 +02:00
Vitaly Baranov
6366940a37 Use hash_of_all_files from system.parts to check identity of parts during on-cluster backups. 2023-06-14 20:39:50 +02:00
Vitaly Baranov
5aa0566767 Fix checking the lock file too often while writing a backup. 2023-06-12 18:13:26 +02:00
Nikita Mikhaylov
e87348010d
Rework loading and removing of data parts for MergeTree tables. (#49474) 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-06-06 14:42:56 +02:00
Vitaly Baranov
6d45d0c374
Use fingerprints instead of key IDs in encrypted disks (#49882) 
* Use fingerprints instead of key IDs to find keys in encrypted disks.
Always use little endian in the headers of encryption files.

* Add tests.

* Fix copying binary files to test containers.

* Fix ownership for copied files in test containers.

* Add comments after review.

---------

Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-05-31 13:11:10 +02:00
Vitaly Baranov
0e2156c776
Fix checksum calculation for backup entries (#50264) 
* Fix possible deadlock in calculating backup entries' checksums.

* Fix data type used to store checksums.

* Allow ReadBufferFromMemory::seek() to go to the end of the buffer
(seek(0, SEEK_SET) must be allowed for an empty buffer)

* Add tests.

---------

Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-05-27 14:35:37 +02:00
kssenii
295fe3b228 Merge remote-tracking branch 'upstream/master' into abstract-async-prefetched-buffer 2023-05-22 20:01:38 +02:00
kssenii
c4d862a16f Make async reader work with any impl 2023-05-22 19:54:04 +02:00
kssenii
3e42ee7f2b Get rid of finalize callback in object storages 2023-05-19 17:29:37 +02:00
Alexander Tokmakov
1e529263d0
Merge branch 'master' into Follow_up_Backup_Restore_concurrency_check_node_2 2023-05-17 13:57:50 +03:00
Vitaly Baranov
f4ac4c3f9d Corrections after review. 2023-05-17 03:23:16 +02:00
Vitaly Baranov
b068f0b619 Fix build. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
943707963f Add backup setting "decrypt_files_from_encrypted_disks" 2023-05-16 14:27:27 +02:00
Vitaly Baranov
019493efa3 Fix throttling in backups. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
5198997fd8 Remove ReadSettings from backup entries. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
7cea264230 Fix whitespaces. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
c48c20fac8 Use combined checksums for encrypted immutable files. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
517e119e03 Move checksum calculation to IBackupEntry. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
002fd19cb7 Move the common part of BackupIO_* to BackupIO_Default. 2023-05-16 14:27:23 +02:00