thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
117,803 Commits 9,328 Branches 1,783 Tags 2.1 GiB
71678f64b1
Commit Graph

49 Commits

Author SHA1 Message Date
Sema Checherinda
206efee9b7 add test with merge tree table 2023-05-26 18:59:21 +02:00
Sema Checherinda
4249bda449 s3_max_inflight_parts_for_one_file with tests 2023-05-26 18:59:11 +02:00
Azat Khuzhin
c8ec152f57 Add ability to use strict parts size for S3 (compatibility with R2) 
The are some implementations that does not allow to use parts of
different size for multi-part upload:

- CouldFlare R2 [1] and also got help on discord
- AWS S3 Glacier [2]
- and I also know of one bug in Google Cloud Storage (GCS), but it
  should be fixed in a couple of months.

  [1]: https://developers.cloudflare.com/r2/reference/changelog/#2022-10-28
       > Multipart upload part sizes are always expected to be of the
       > same size, but this enforcement is now done when you complete
       > an upload instead of being done very time you upload a part

  [2]: https://docs.aws.amazon.com/amazonglacier/latest/dev/api-multipart-initiate-upload.html

On top of this I'm going to add auto detection of this issue for disks,
but not for S3 function, since those checks are quite costly, and they
need to write at least 5MB*3, which may be questionable.

v2: mark test as long [1]
    [1]: https://s3.amazonaws.com/clickhouse-test-reports/48492/d4a90880e947ab3a17b3eddeea274afed8ebceb4/stateless_tests_flaky_check__asan_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-28 11:01:56 +02:00
Antonio Andelic
98b6944d37 Merge branch 'master' into s3-client-polish 2023-02-08 10:07:00 +00:00
Antonio Andelic
4cd361747d Address PR comment 2023-02-06 10:11:48 +00:00
xiedeyantu
f13eedd644 change settings name 2023-02-04 22:11:14 +08:00
xiedeyantu
562642ab7f add settings s3_allow_throw_if_mismatch_files 2023-02-03 12:27:13 +08:00
Vitaly Baranov
d02b255b55 Add new setting "allow_head_object_request" to S3::RequestSettings. 2023-01-27 15:10:34 +01:00
Vitaly Baranov
a8304525ed Move getObjectInfo() to a separate header. 2023-01-27 15:09:38 +01:00
Anton Popov
41a199e175
Fix crash when ListObjects request fails (#45371) 2023-01-20 20:10:23 +01:00
Kseniia Sumarokova
9ae5ad202a
Merge pull request #44707 from xiedeyantu/s3-add-config 
S3 support upload object using storage class INTELLIGENT_TIERING
 2023-01-09 13:28:53 +01:00
chen
741dbb5a75
Update StorageS3Settings.h 2023-01-05 00:41:58 +08:00
xiedeyantu
2f59b1d002 better 2023-01-05 00:27:07 +08:00
xiedeyantu
c636b3e363 S3 support upload object using storage class INTELLIGENT_TIERING 2022-12-30 21:05:01 +08:00
kssenii
6bd4f8c029 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2022-12-20 21:17:28 +01:00
Nikolai Kochetov
a3226b2daf Remove extra compare operator. Review fixes. 2022-12-19 13:12:59 +00:00
Nikolai Kochetov
4032ee8203 Try to fix another one test 2022-12-18 12:31:17 +00:00
Nikolai Kochetov
b2355a2212 Fixing tests. 2022-12-17 16:02:34 +00:00
Nikolai Kochetov
62ff98344e Validate s3 part upload settings. 2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd Replace old named collections code for url 2022-12-17 00:24:05 +01:00
Vitaly Baranov
b91af1b650 Fix initialization of s3 request settings. 2022-12-10 05:43:51 +01:00
Vitaly Baranov
0ba4870a18 Fix race in S3 multipart upload. 2022-12-09 03:02:39 +01:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling"" 2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling" 2022-11-16 22:33:48 +03:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler 2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421 rename ReadWriteSettings -> RequestSettings 2022-11-08 13:48:23 +00:00
serxa
2daec0b45e S3 request per second rate throttling + refactoring 2022-11-07 18:05:40 +00:00
Vitaly Baranov
32194c1200 Add max limitation for the size of an uploaded part. 2022-11-02 17:53:54 +01:00
Vitaly Baranov
b9f2f17331 Add test and logging. 2022-11-01 12:23:20 +01:00
Antonio Andelic
655bde8e3f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-27 07:28:26 +00:00
kssenii
e34101456a Fix 2022-09-21 17:11:37 +02:00
kssenii
f917b268b3 Merge remote-tracking branch 'upstream/master' into s3-header-auth 2022-09-21 12:56:25 +02:00
Antonio Andelic
fc6c1faa57 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-21 06:57:58 +00:00
alesapin
d9c3549cc8 Revert s3 prefixes + add retries on write requests unexpected errors 2022-09-16 14:43:43 +02:00
Antonio Andelic
9fdd2f2e61 Basic implementation for S3 snapshot upload 2022-09-15 09:11:55 +00:00
kssenii
420ac4eb43 s3 header auth in ast 2022-09-13 15:13:28 +02:00
alesapin
07286dd042 Add paranoid option to check blobs to S3 after upload 2022-08-23 15:12:09 +02:00
kssenii
ef2b238444 Fix 2022-04-08 12:30:24 +02:00
kssenii
4e50da722b Better s3 settings 2022-04-04 16:14:56 +02:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Vladimir Chebotarev
a6ab040b07 Added region parameter for S3 storage and disk. 2021-05-01 20:45:35 +03:00
Vladimir Chebotarev
c0ea3ce08a Removed STS client. 2021-04-15 10:01:43 +03:00
Vladimir Chebotarev
3a467ffc50 Attempt to make credentials provider which uses internal credentials provider chain. 2021-04-15 10:01:13 +03:00
Vladimir Chebotarev
037a4de854 Added STS GetCallerIdentity credentials provider. 2021-04-15 10:00:18 +03:00
Pervakov Grigorii
eb87afc4d4 recreate S3 client if credentials changed 2021-03-10 13:36:57 +03:00
Vladimir Chebotarev
579f8da573 Added SSE-C support in S3 client. 2021-01-28 09:32:41 +03:00
Vladimir Chebotarev
39d867e56d
Added proper authentication for S3 client (#16856) 2020-12-10 12:19:42 +03:00
Artem Zuikov
51ba12c2c3
Try speedup build (#14809) 2020-09-15 12:55:57 +03:00
Pervakov Grigorii
bc9381406f Add default credentials and custom headers for s3 table functions. 2020-06-02 20:48:49 +03:00