Antonio Andelic
cc31b837f2
User specific S3 endpoint backup/restore ON CLUSTER
2024-04-03 08:55:56 +02:00
Kseniia Sumarokova
19d8256fa8
Update test.py
2024-03-11 14:44:19 +01:00
Kseniia Sumarokova
90b27432a2
Update test.py
2024-03-11 12:18:58 +01:00
robot-clickhouse
be98c95f58
Automatic style fix
2024-03-06 09:12:26 +00:00
kssenii
6d4514c045
Fix test
2024-03-06 16:55:48 +08:00
Joshua Hildred
ca9a6f809f
add support for .tar.zst and tar.xz
2024-02-29 05:20:53 -08:00
Joshua Hildred
92b446a552
Add support for tar archive compression
2024-02-29 05:20:52 -08:00
Joshua Hildred
c53021f55e
Fix formating
2024-02-29 05:20:51 -08:00
Joshua Hildred
eb4ec0e871
Add support for reading and writing backups as a
...
tar archive using libarchive.
2024-02-29 05:20:51 -08:00
Alexey Milovidov
330043af30
Fix integration test_backup_restore_s3
2024-02-22 10:58:05 +01:00
Antonio Andelic
8d6088c2e6
Fix test
2024-02-21 09:48:34 +01:00
Antonio Andelic
175a1db787
Support specifying users for s3 settings
2024-02-19 16:43:27 +01:00
Vitaly Baranov
e1a136b791
Explicit finalize() function in ZipArchiveWriter.
...
Simplify too complicated code in ZipArchiveWriter.
2023-12-24 00:33:59 +01:00
MikhailBurdukov
6f19e8ebd1
Merge branch 'master' into native_copy_for_s3_disks
2023-11-27 14:25:36 +03:00
MikhailBurdukov
c10c30832c
Review fix
2023-11-27 10:58:30 +00:00
vdimir
15234474d7
Implement system table blob_storage_log
2023-11-21 09:18:25 +00:00
MikhailBurdukov
1aea8186c7
Enable s3 native copy for scenario: from s3 to s3.
2023-11-14 13:53:48 +00:00
alesapin
54c48df486
Fix test
2023-09-29 14:03:58 +02:00
Vitaly Baranov
6ac61b1fdd
Fix tests.
2023-08-03 13:52:35 +02:00
Vitaly Baranov
556ecf5c1c
Renamed backup setting "read_from_filesystem_cache_if_exists_otherwise_bypass_cache" -> "read_from_filesystem_cache".
2023-08-03 13:52:35 +02:00
Vitaly Baranov
1cd7902079
Add test.
2023-08-03 13:52:35 +02:00
Azat Khuzhin
29dc9abfca
Fix test_backup_restore_s3 after logging for native copying changed
...
Check profile events instead of some odd logs.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-09 08:20:03 +02:00
Azat Khuzhin
84c720b33e
Beatify test_backup_restore_s3 (using per-query profile events)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-09 08:20:03 +02:00
AVMusorin
7df4820af7
Fix metrics WriteBufferFromS3Bytes, WriteBufferFromS3Microseconds and WriteBufferFromS3RequestsErrors
...
Ref: https://github.com/ClickHouse/ClickHouse/pull/45188
2023-05-17 14:50:38 +02:00
Vitaly Baranov
c48c20fac8
Use combined checksums for encrypted immutable files.
2023-05-16 14:27:27 +02:00
Vitaly Baranov
101aa6eff0
Add function copyS3FileFromDisk().
2023-05-16 14:25:37 +02:00
Vitaly Baranov
fd2731845c
Simplify interface of IBackupWriter: Remove supportNativeCopy() function.
2023-05-16 14:25:36 +02:00
Vitaly Baranov
478140f8b4
Fix LimitSeekableReadBuffer.
2023-05-11 22:58:05 +02:00
Alexander Tokmakov
83b12ec5ca
replace NO DELAY with SYNC in tests
2023-05-03 20:08:49 +02:00
Nikita Mikhaylov
fa5b2bd4a5
Added Keeper retries for backups operations ( #47224 )
2023-04-04 18:12:08 +02:00
Vitaly Baranov
25356786ea
Simplify the implementation, create new utility function copyS3FileToDisk().
2023-03-14 23:34:44 +01:00
Vitaly Baranov
1cf1ce07fe
Use server-side copy during restore from S3 to S3.
2023-03-13 23:50:13 +01:00
Vitaly Baranov
5ceb64accc
Use new copy s3 functions in S3ObjectStorage.
2023-01-21 15:47:58 +01:00
Vitaly Baranov
14a7ee8e26
Copy files to S3 during backup directly without using WriteBufferFromS3 to decrease memory consumption.
2023-01-17 09:35:41 +01:00
Aleksandr Musorin
5cb69d8a22
changed type name for S3_Plain storage
...
renamed a disk for S3PlainObjectStorage in system.disks table from s3 to s3_plain
2022-11-04 17:35:51 +01:00
Vitaly Baranov
b9f2f17331
Add test and logging.
2022-11-01 12:23:20 +01:00
Vitaly Baranov
1365105bc4
Implement backup to S3
2022-10-19 00:04:41 +02:00
Azat Khuzhin
8830f0608d
Support BACKUP to S3 with as-is path/data structure
...
Right now backup to S3 does not make a lot of sense, since:
- it has random names, and to decoding them
- requires metadata from local disk (/var/lib/disks/DISK/BACKUP_NAME)
- or send_metadata (but it is also tricky even with it)
So this patch adds simpler interface for S3, it is only suitable for
BACKUP/RESTORE, so don't try to use it for MergeTree engine.
It is done by adding separate disk - `s3_plain` for this, that:
- does not support any extended features, like renames/hardlinks/attrs/...
(so basically everything that MergeTree requires)
- only write/read/unlink/list files
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-14 12:59:25 +02:00
