Commit Graph

13 Commits

Author SHA1 Message Date
Antonio Andelic
cc31b837f2 User specific S3 endpoint backup/restore ON CLUSTER 2024-04-03 08:55:56 +02:00
Antonio Andelic
8d6088c2e6 Fix test 2024-02-21 09:48:34 +01:00
Antonio Andelic
175a1db787 Support specifying users for s3 settings 2024-02-19 16:43:27 +01:00
MikhailBurdukov
6f19e8ebd1
Merge branch 'master' into native_copy_for_s3_disks 2023-11-27 14:25:36 +03:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
MikhailBurdukov
1aea8186c7 Enable s3 native copy for scenario: from s3 to s3. 2023-11-14 13:53:48 +00:00
alesapin
54c48df486 Fix test 2023-09-29 14:03:58 +02:00
Vitaly Baranov
1cd7902079 Add test. 2023-08-03 13:52:35 +02:00
Nikita Mikhaylov
fa5b2bd4a5
Added Keeper retries for backups operations (#47224) 2023-04-04 18:12:08 +02:00
Vitaly Baranov
14a7ee8e26 Copy files to S3 during backup directly without using WriteBufferFromS3 to decrease memory consumption. 2023-01-17 09:35:41 +01:00
Vitaly Baranov
b9f2f17331 Add test and logging. 2022-11-01 12:23:20 +01:00
Vitaly Baranov
1365105bc4 Implement backup to S3 2022-10-19 00:04:41 +02:00
Azat Khuzhin
8830f0608d Support BACKUP to S3 with as-is path/data structure
Right now backup to S3 does not make a lot of sense, since:
- it has random names, and to decoding them
- requires metadata from local disk (/var/lib/disks/DISK/BACKUP_NAME)
- or send_metadata (but it is also tricky even with it)

So this patch adds simpler interface for S3, it is only suitable for
BACKUP/RESTORE, so don't try to use it for MergeTree engine.

It is done by adding separate disk - `s3_plain` for this, that:
- does not support any extended features, like renames/hardlinks/attrs/...
  (so basically everything that MergeTree requires)
- only write/read/unlink/list files

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-14 12:59:25 +02:00