MikhailBurdukov
6f19e8ebd1
Merge branch 'master' into native_copy_for_s3_disks
2023-11-27 14:25:36 +03:00
MikhailBurdukov
c10c30832c
Review fix
2023-11-27 10:58:30 +00:00
vdimir
15234474d7
Implement system table blob_storage_log
2023-11-21 09:18:25 +00:00
MikhailBurdukov
1aea8186c7
Enable s3 native copy for scenario: from s3 to s3.
2023-11-14 13:53:48 +00:00
Vitaly Baranov
6ac61b1fdd
Fix tests.
2023-08-03 13:52:35 +02:00
Vitaly Baranov
556ecf5c1c
Renamed backup setting "read_from_filesystem_cache_if_exists_otherwise_bypass_cache" -> "read_from_filesystem_cache".
2023-08-03 13:52:35 +02:00
Vitaly Baranov
1cd7902079
Add test.
2023-08-03 13:52:35 +02:00
Azat Khuzhin
29dc9abfca
Fix test_backup_restore_s3 after logging for native copying changed
...
Check profile events instead of some odd logs.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-09 08:20:03 +02:00
Azat Khuzhin
84c720b33e
Beatify test_backup_restore_s3 (using per-query profile events)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-09 08:20:03 +02:00
AVMusorin
7df4820af7
Fix metrics WriteBufferFromS3Bytes, WriteBufferFromS3Microseconds and WriteBufferFromS3RequestsErrors
...
Ref: https://github.com/ClickHouse/ClickHouse/pull/45188
2023-05-17 14:50:38 +02:00
Vitaly Baranov
c48c20fac8
Use combined checksums for encrypted immutable files.
2023-05-16 14:27:27 +02:00
Vitaly Baranov
101aa6eff0
Add function copyS3FileFromDisk().
2023-05-16 14:25:37 +02:00
Vitaly Baranov
fd2731845c
Simplify interface of IBackupWriter: Remove supportNativeCopy() function.
2023-05-16 14:25:36 +02:00
Vitaly Baranov
478140f8b4
Fix LimitSeekableReadBuffer.
2023-05-11 22:58:05 +02:00
Alexander Tokmakov
83b12ec5ca
replace NO DELAY with SYNC in tests
2023-05-03 20:08:49 +02:00
Nikita Mikhaylov
fa5b2bd4a5
Added Keeper retries for backups operations ( #47224 )
2023-04-04 18:12:08 +02:00
Vitaly Baranov
25356786ea
Simplify the implementation, create new utility function copyS3FileToDisk().
2023-03-14 23:34:44 +01:00
Vitaly Baranov
1cf1ce07fe
Use server-side copy during restore from S3 to S3.
2023-03-13 23:50:13 +01:00
Vitaly Baranov
5ceb64accc
Use new copy s3 functions in S3ObjectStorage.
2023-01-21 15:47:58 +01:00
Vitaly Baranov
14a7ee8e26
Copy files to S3 during backup directly without using WriteBufferFromS3 to decrease memory consumption.
2023-01-17 09:35:41 +01:00
Aleksandr Musorin
5cb69d8a22
changed type name for S3_Plain storage
...
renamed a disk for S3PlainObjectStorage in system.disks table from s3 to s3_plain
2022-11-04 17:35:51 +01:00
Vitaly Baranov
b9f2f17331
Add test and logging.
2022-11-01 12:23:20 +01:00
Vitaly Baranov
1365105bc4
Implement backup to S3
2022-10-19 00:04:41 +02:00
Azat Khuzhin
8830f0608d
Support BACKUP to S3 with as-is path/data structure
...
Right now backup to S3 does not make a lot of sense, since:
- it has random names, and to decoding them
- requires metadata from local disk (/var/lib/disks/DISK/BACKUP_NAME)
- or send_metadata (but it is also tricky even with it)
So this patch adds simpler interface for S3, it is only suitable for
BACKUP/RESTORE, so don't try to use it for MergeTree engine.
It is done by adding separate disk - `s3_plain` for this, that:
- does not support any extended features, like renames/hardlinks/attrs/...
(so basically everything that MergeTree requires)
- only write/read/unlink/list files
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-14 12:59:25 +02:00
