Robert Schulze
62c12cbfca
Replace boost::algorithm::starts_with() by std::string::starts_with()
2024-02-27 10:42:47 +00:00
Antonio Andelic
175a1db787
Support specifying users for s3 settings
2024-02-19 16:43:27 +01:00
Azat Khuzhin
b9233f6d4f
Move Allocator code into module part
...
This should reduce amount of code that should be recompiled on
Exception.h changes (and everything else that had been included there).
This will actually not help a lot, because it is also included into
PODArray.h and ThreadPool.h at least... Sigh.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-27 15:42:08 +01:00
Azat Khuzhin
5835e72fd6
More generic approach to disable native copy
...
Previous patch implements this only for BACKUP/RESTORE, but it can be
useful for regular disks as well, so add allow_native_copy for disks.
Note, that there is s3_allow_native_copy query setting, since it looks
redundant, since it make sense only for S3 disks, and not on a per query
basis.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-09 08:20:02 +02:00
Michael Kolupaev
4a570a05c9
Decrease default timeouts for S3 and HTTP requests
2023-06-21 18:08:50 +00:00
Sema Checherinda
206efee9b7
add test with merge tree table
2023-05-26 18:59:21 +02:00
Sema Checherinda
4249bda449
s3_max_inflight_parts_for_one_file with tests
2023-05-26 18:59:11 +02:00
Azat Khuzhin
c8ec152f57
Add ability to use strict parts size for S3 (compatibility with R2)
...
The are some implementations that does not allow to use parts of
different size for multi-part upload:
- CouldFlare R2 [1] and also got help on discord
- AWS S3 Glacier [2]
- and I also know of one bug in Google Cloud Storage (GCS), but it
should be fixed in a couple of months.
[1]: https://developers.cloudflare.com/r2/reference/changelog/#2022-10-28
> Multipart upload part sizes are always expected to be of the
> same size, but this enforcement is now done when you complete
> an upload instead of being done very time you upload a part
[2]: https://docs.aws.amazon.com/amazonglacier/latest/dev/api-multipart-initiate-upload.html
On top of this I'm going to add auto detection of this issue for disks,
but not for S3 function, since those checks are quite costly, and they
need to write at least 5MB*3, which may be questionable.
v2: mark test as long [1]
[1]: https://s3.amazonaws.com/clickhouse-test-reports/48492/d4a90880e947ab3a17b3eddeea274afed8ebceb4/stateless_tests_flaky_check__asan_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-28 11:01:56 +02:00
Azat Khuzhin
f5c3f3a4ab
Remove quirk for s3_max_single_part_upload_size (when it exceed INT_MAX)
...
Reverts: #48816
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-17 10:49:55 +02:00
Alexey Milovidov
8e0b1eae69
Update StorageS3Settings.cpp
2023-04-16 08:53:52 +03:00
Alexey Milovidov
1174f4dcbe
Prevent a bug in AWS S3
2023-04-15 23:56:49 +02:00
Antonio Andelic
98b6944d37
Merge branch 'master' into s3-client-polish
2023-02-08 10:07:00 +00:00
Antonio Andelic
4cd361747d
Address PR comment
2023-02-06 10:11:48 +00:00
xiedeyantu
f13eedd644
change settings name
2023-02-04 22:11:14 +08:00
xiedeyantu
562642ab7f
add settings s3_allow_throw_if_mismatch_files
2023-02-03 12:27:13 +08:00
alesapin
631c8fb155
Merge branch 'master' into add-setting-allow-head-object-request
2023-01-29 14:58:27 +01:00
Vitaly Baranov
d02b255b55
Add new setting "allow_head_object_request" to S3::RequestSettings.
2023-01-27 15:10:34 +01:00
Alexander Tokmakov
bb4c8e169f
check number of parameters in format string
2023-01-23 23:16:16 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Anton Popov
41a199e175
Fix crash when ListObjects request fails ( #45371 )
2023-01-20 20:10:23 +01:00
Kseniia Sumarokova
9ae5ad202a
Merge pull request #44707 from xiedeyantu/s3-add-config
...
S3 support upload object using storage class INTELLIGENT_TIERING
2023-01-09 13:28:53 +01:00
xiedeyantu
2f59b1d002
better
2023-01-05 00:27:07 +08:00
chen
72200273da
Update StorageS3Settings.cpp
2022-12-30 22:47:48 +08:00
xiedeyantu
c636b3e363
S3 support upload object using storage class INTELLIGENT_TIERING
2022-12-30 21:05:01 +08:00
kssenii
1d75f740d7
Fix tests
2022-12-20 22:33:54 +01:00
Nikolai Kochetov
a3226b2daf
Remove extra compare operator. Review fixes.
2022-12-19 13:12:59 +00:00
Nikolai Kochetov
4032ee8203
Try to fix another one test
2022-12-18 12:31:17 +00:00
Nikolai Kochetov
2b999ce715
Fixing testx.
2022-12-17 21:23:08 +00:00
Nikolai Kochetov
f7c308077d
Fixing build.
2022-12-17 17:12:04 +00:00
Nikolai Kochetov
b3278211bb
Fixing tests.
2022-12-17 16:06:42 +00:00
Nikolai Kochetov
b2355a2212
Fixing tests.
2022-12-17 16:02:34 +00:00
Nikolai Kochetov
98ebef7914
Fixing special build.
2022-12-17 15:27:01 +00:00
Nikolai Kochetov
29c6caaeaf
Validate s3 part upload settings.
2022-12-17 14:18:42 +00:00
Nikolai Kochetov
62ff98344e
Validate s3 part upload settings.
2022-12-17 14:09:53 +00:00
Vitaly Baranov
b91af1b650
Fix initialization of s3 request settings.
2022-12-10 05:43:51 +01:00
Vitaly Baranov
0ba4870a18
Fix race in S3 multipart upload.
2022-12-09 03:02:39 +01:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Vitaly Baranov
32194c1200
Add max limitation for the size of an uploaded part.
2022-11-02 17:53:54 +01:00
Vitaly Baranov
b9f2f17331
Add test and logging.
2022-11-01 12:23:20 +01:00
Antonio Andelic
fc6c1faa57
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-21 06:57:58 +00:00
alesapin
d9c3549cc8
Revert s3 prefixes + add retries on write requests unexpected errors
2022-09-16 14:43:43 +02:00
Antonio Andelic
32a297339b
Fix build
2022-09-15 13:36:16 +00:00
Antonio Andelic
9fdd2f2e61
Basic implementation for S3 snapshot upload
2022-09-15 09:11:55 +00:00
alesapin
07286dd042
Add paranoid option to check blobs to S3 after upload
2022-08-23 15:12:09 +02:00
Anton Popov
df6882d2b9
Revert "Fix errors of CheckTriviallyCopyableMove type"
2022-06-07 13:53:10 +02:00
HeenaBansal2009
b7eb6bbd38
Fixed clang-tidy-CheckTriviallyCopyableMove-errors
2022-05-30 11:09:03 -07:00
