Antonio Andelic
175a1db787
Support specifying users for s3 settings
2024-02-19 16:43:27 +01:00
kssenii
4a28f10c3d
Minor cache changes
2023-12-04 19:02:37 +01:00
Sema Checherinda
f999337dae
Revert "Revert "s3 adaptive timeouts""
2023-11-20 14:53:22 +01:00
Alexander Tokmakov
5031f239c3
Revert "s3 adaptive timeouts"
2023-11-20 14:28:59 +01:00
Sema Checherinda
8d36fd6e54
get rid off of client_with_long_timeout_ptr
2023-11-14 11:34:12 +01:00
Roman Vasin
2cf79ac592
Merge branch 'master' of github.com:ClickHouse/ClickHouse into ADQM-984
2023-07-26 09:58:26 +00:00
Roman Vasin
6c3b7f76d4
Add S3 storage class as 4th parameter
2023-07-13 16:00:01 +00:00
Azat Khuzhin
5835e72fd6
More generic approach to disable native copy
...
Previous patch implements this only for BACKUP/RESTORE, but it can be
useful for regular disks as well, so add allow_native_copy for disks.
Note, that there is s3_allow_native_copy query setting, since it looks
redundant, since it make sense only for S3 disks, and not on a per query
basis.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-09 08:20:02 +02:00
Michael Kolupaev
4a570a05c9
Decrease default timeouts for S3 and HTTP requests
2023-06-21 18:08:50 +00:00
Sema Checherinda
206efee9b7
add test with merge tree table
2023-05-26 18:59:21 +02:00
Sema Checherinda
4249bda449
s3_max_inflight_parts_for_one_file with tests
2023-05-26 18:59:11 +02:00
Azat Khuzhin
c8ec152f57
Add ability to use strict parts size for S3 (compatibility with R2)
...
The are some implementations that does not allow to use parts of
different size for multi-part upload:
- CouldFlare R2 [1] and also got help on discord
- AWS S3 Glacier [2]
- and I also know of one bug in Google Cloud Storage (GCS), but it
should be fixed in a couple of months.
[1]: https://developers.cloudflare.com/r2/reference/changelog/#2022-10-28
> Multipart upload part sizes are always expected to be of the
> same size, but this enforcement is now done when you complete
> an upload instead of being done very time you upload a part
[2]: https://docs.aws.amazon.com/amazonglacier/latest/dev/api-multipart-initiate-upload.html
On top of this I'm going to add auto detection of this issue for disks,
but not for S3 function, since those checks are quite costly, and they
need to write at least 5MB*3, which may be questionable.
v2: mark test as long [1]
[1]: https://s3.amazonaws.com/clickhouse-test-reports/48492/d4a90880e947ab3a17b3eddeea274afed8ebceb4/stateless_tests_flaky_check__asan_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-28 11:01:56 +02:00
Antonio Andelic
98b6944d37
Merge branch 'master' into s3-client-polish
2023-02-08 10:07:00 +00:00
Antonio Andelic
4cd361747d
Address PR comment
2023-02-06 10:11:48 +00:00
xiedeyantu
f13eedd644
change settings name
2023-02-04 22:11:14 +08:00
xiedeyantu
562642ab7f
add settings s3_allow_throw_if_mismatch_files
2023-02-03 12:27:13 +08:00
Vitaly Baranov
d02b255b55
Add new setting "allow_head_object_request" to S3::RequestSettings.
2023-01-27 15:10:34 +01:00
Vitaly Baranov
a8304525ed
Move getObjectInfo() to a separate header.
2023-01-27 15:09:38 +01:00
Anton Popov
41a199e175
Fix crash when ListObjects request fails ( #45371 )
2023-01-20 20:10:23 +01:00
Kseniia Sumarokova
9ae5ad202a
Merge pull request #44707 from xiedeyantu/s3-add-config
...
S3 support upload object using storage class INTELLIGENT_TIERING
2023-01-09 13:28:53 +01:00
chen
741dbb5a75
Update StorageS3Settings.h
2023-01-05 00:41:58 +08:00
xiedeyantu
2f59b1d002
better
2023-01-05 00:27:07 +08:00
xiedeyantu
c636b3e363
S3 support upload object using storage class INTELLIGENT_TIERING
2022-12-30 21:05:01 +08:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
Nikolai Kochetov
a3226b2daf
Remove extra compare operator. Review fixes.
2022-12-19 13:12:59 +00:00
Nikolai Kochetov
4032ee8203
Try to fix another one test
2022-12-18 12:31:17 +00:00
Nikolai Kochetov
b2355a2212
Fixing tests.
2022-12-17 16:02:34 +00:00
Nikolai Kochetov
62ff98344e
Validate s3 part upload settings.
2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Vitaly Baranov
b91af1b650
Fix initialization of s3 request settings.
2022-12-10 05:43:51 +01:00
Vitaly Baranov
0ba4870a18
Fix race in S3 multipart upload.
2022-12-09 03:02:39 +01:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Vitaly Baranov
32194c1200
Add max limitation for the size of an uploaded part.
2022-11-02 17:53:54 +01:00
Vitaly Baranov
b9f2f17331
Add test and logging.
2022-11-01 12:23:20 +01:00
Antonio Andelic
655bde8e3f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-27 07:28:26 +00:00
kssenii
e34101456a
Fix
2022-09-21 17:11:37 +02:00
kssenii
f917b268b3
Merge remote-tracking branch 'upstream/master' into s3-header-auth
2022-09-21 12:56:25 +02:00
Antonio Andelic
fc6c1faa57
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-21 06:57:58 +00:00
alesapin
d9c3549cc8
Revert s3 prefixes + add retries on write requests unexpected errors
2022-09-16 14:43:43 +02:00
Antonio Andelic
9fdd2f2e61
Basic implementation for S3 snapshot upload
2022-09-15 09:11:55 +00:00
kssenii
420ac4eb43
s3 header auth in ast
2022-09-13 15:13:28 +02:00
alesapin
07286dd042
Add paranoid option to check blobs to S3 after upload
2022-08-23 15:12:09 +02:00
kssenii
ef2b238444
Fix
2022-04-08 12:30:24 +02:00
kssenii
4e50da722b
Better s3 settings
2022-04-04 16:14:56 +02:00
Alexey Milovidov
fe6b7c77c7
Rename "common" to "base"
2021-10-02 10:13:14 +03:00
Vladimir Chebotarev
a6ab040b07
Added region parameter for S3 storage and disk.
2021-05-01 20:45:35 +03:00
