Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
Alexander Tokmakov
6eb557b2ba
Merge branch 'master' into exception_message_patterns4
2023-01-25 13:49:17 +01:00
Alexander Tokmakov
3f6594f4c6
forbid old ctor of Exception
2023-01-23 22:18:05 +01:00
Vitaly Baranov
5ceb64accc
Use new copy s3 functions in S3ObjectStorage.
2023-01-21 15:47:58 +01:00
Vitaly Baranov
14a7ee8e26
Copy files to S3 during backup directly without using WriteBufferFromS3 to decrease memory consumption.
2023-01-17 09:35:41 +01:00
Vitaly Baranov
21b8aaeb8b
Stop using HeadObject requests in S3
...
because they don't work well with endpoints without explicit region.
2023-01-15 20:28:11 +01:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
Azat Khuzhin
7d81c39207
backups: ignore file not found error for S3 (similar to Disk)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-27 15:56:05 +01:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
Nikolai Kochetov
b2355a2212
Fixing tests.
2022-12-17 16:02:34 +00:00
Nikolai Kochetov
29c6caaeaf
Validate s3 part upload settings.
2022-12-17 14:18:42 +00:00
Nikolai Kochetov
62ff98344e
Validate s3 part upload settings.
2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Anton Popov
cce3257f39
Merge branch 'master' into optimize-storage-s3
2022-12-13 21:35:12 +01:00
Vitaly Baranov
d7eccb4581
Merge pull request #43940 from azat/backups/gcs
...
Fix BACKUP TO S3 for Google Cloud Storage
2022-12-13 19:04:52 +01:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
Vitaly Baranov
b91af1b650
Fix initialization of s3 request settings.
2022-12-10 05:43:51 +01:00
Vitaly Baranov
0ba4870a18
Fix race in S3 multipart upload.
2022-12-09 03:02:39 +01:00
Azat Khuzhin
3d8ea48103
Fix BACKUP TO S3 for Google Cloud Storage (no batch delete support)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-08 20:46:54 +01:00
Anton Ivashkin
d6ca97c8d0
Fix multipart upload for large S3 object
2022-11-30 11:58:04 +02:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
serxa
ad377b357f
fix backup tests
2022-11-11 13:24:43 +00:00
serxa
2de26daa56
fix build
2022-11-08 14:31:29 +00:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Vitaly Baranov
32194c1200
Add max limitation for the size of an uploaded part.
2022-11-02 17:53:54 +01:00
Vitaly Baranov
b9f2f17331
Add test and logging.
2022-11-01 12:23:20 +01:00
Vitaly Baranov
914ab51992
Increase the size of upload part exponentially for backup to S3.
2022-10-31 17:54:41 +01:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Vitaly Baranov
1365105bc4
Implement backup to S3
2022-10-19 00:04:41 +02:00
