Azat Khuzhin
c332d290d8
Keep only one throttler for BACKUPs IO (instead of separate read/write)
...
There is no need in separate read/write throttling, because you cannot
write faster then read anyway, and plus this makes the code less cleaner
(and also it will allow avoid implementing throttling backups to S3,
since it does not use common S3 writer).
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-05 09:40:17 +02:00
Azat Khuzhin
218b1f9c29
Add ability to throttle BACKUPs on per-server/backup basis
...
Server settings:
- backup_read_bandwidth_for_server
- backup_write_bandwidth_for_server
Query settings:
- backup_read_bandwidth
- backup_write_bandwidth
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-05 09:39:48 +02:00
Antonio Andelic
e982f2a67a
Merge branch 'master' into enable-env-credentials-default
2023-03-31 09:11:01 +00:00
Antonio Andelic
160aa186bb
Add support for NOSIGN keyword and no_sign_request config
2023-03-28 07:05:35 +00:00
Antonio Andelic
a0582a14b9
Merge pull request #47423 from ClickHouse/add-expiration-window-s3
...
Add expiration window for S3 credentials
2023-03-18 10:11:29 +01:00
Vitaly Baranov
25356786ea
Simplify the implementation, create new utility function copyS3FileToDisk().
2023-03-14 23:34:44 +01:00
Vitaly Baranov
1cf1ce07fe
Use server-side copy during restore from S3 to S3.
2023-03-13 23:50:13 +01:00
Antonio Andelic
a170a909a4
Add expiration window for S3 credentials
2023-03-10 10:06:32 +00:00
Antonio Andelic
5bc21538e5
Enable use_environment_credentials by default
2023-03-09 10:31:55 +00:00
Nikita Mikhaylov
5c4da5aa4a
Use separate thread pool for IO operations for backups ( #47174 )
2023-03-03 20:05:42 +01:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
Alexander Tokmakov
6eb557b2ba
Merge branch 'master' into exception_message_patterns4
2023-01-25 13:49:17 +01:00
Alexander Tokmakov
3f6594f4c6
forbid old ctor of Exception
2023-01-23 22:18:05 +01:00
Vitaly Baranov
5ceb64accc
Use new copy s3 functions in S3ObjectStorage.
2023-01-21 15:47:58 +01:00
Vitaly Baranov
14a7ee8e26
Copy files to S3 during backup directly without using WriteBufferFromS3 to decrease memory consumption.
2023-01-17 09:35:41 +01:00
Vitaly Baranov
21b8aaeb8b
Stop using HeadObject requests in S3
...
because they don't work well with endpoints without explicit region.
2023-01-15 20:28:11 +01:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
Azat Khuzhin
7d81c39207
backups: ignore file not found error for S3 (similar to Disk)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-27 15:56:05 +01:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
Nikolai Kochetov
b2355a2212
Fixing tests.
2022-12-17 16:02:34 +00:00
Nikolai Kochetov
29c6caaeaf
Validate s3 part upload settings.
2022-12-17 14:18:42 +00:00
Nikolai Kochetov
62ff98344e
Validate s3 part upload settings.
2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Anton Popov
cce3257f39
Merge branch 'master' into optimize-storage-s3
2022-12-13 21:35:12 +01:00
Vitaly Baranov
d7eccb4581
Merge pull request #43940 from azat/backups/gcs
...
Fix BACKUP TO S3 for Google Cloud Storage
2022-12-13 19:04:52 +01:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
Vitaly Baranov
b91af1b650
Fix initialization of s3 request settings.
2022-12-10 05:43:51 +01:00
Vitaly Baranov
0ba4870a18
Fix race in S3 multipart upload.
2022-12-09 03:02:39 +01:00
Azat Khuzhin
3d8ea48103
Fix BACKUP TO S3 for Google Cloud Storage (no batch delete support)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-08 20:46:54 +01:00
Anton Ivashkin
d6ca97c8d0
Fix multipart upload for large S3 object
2022-11-30 11:58:04 +02:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
serxa
ad377b357f
fix backup tests
2022-11-11 13:24:43 +00:00
serxa
2de26daa56
fix build
2022-11-08 14:31:29 +00:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Vitaly Baranov
32194c1200
Add max limitation for the size of an uploaded part.
2022-11-02 17:53:54 +01:00
Vitaly Baranov
b9f2f17331
Add test and logging.
2022-11-01 12:23:20 +01:00
Vitaly Baranov
914ab51992
Increase the size of upload part exponentially for backup to S3.
2022-10-31 17:54:41 +01:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Vitaly Baranov
1365105bc4
Implement backup to S3
2022-10-19 00:04:41 +02:00
