thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,457 Commits 9,328 Branches 1,783 Tags 2.1 GiB
1a93fd7f7d
Commit Graph

60 Commits

Author SHA1 Message Date
Azat Khuzhin
5835e72fd6 More generic approach to disable native copy 
Previous patch implements this only for BACKUP/RESTORE, but it can be
useful for regular disks as well, so add allow_native_copy for disks.

Note, that there is s3_allow_native_copy query setting, since it looks
redundant, since it make sense only for S3 disks, and not on a per query
basis.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-09 08:20:02 +02:00
Michael Kolupaev
4a570a05c9 Decrease default timeouts for S3 and HTTP requests 2023-06-21 18:08:50 +00:00
Sema Checherinda
206efee9b7 add test with merge tree table 2023-05-26 18:59:21 +02:00
Sema Checherinda
4249bda449 s3_max_inflight_parts_for_one_file with tests 2023-05-26 18:59:11 +02:00
Azat Khuzhin
c8ec152f57 Add ability to use strict parts size for S3 (compatibility with R2) 
The are some implementations that does not allow to use parts of
different size for multi-part upload:

- CouldFlare R2 [1] and also got help on discord
- AWS S3 Glacier [2]
- and I also know of one bug in Google Cloud Storage (GCS), but it
  should be fixed in a couple of months.

  [1]: https://developers.cloudflare.com/r2/reference/changelog/#2022-10-28
       > Multipart upload part sizes are always expected to be of the
       > same size, but this enforcement is now done when you complete
       > an upload instead of being done very time you upload a part

  [2]: https://docs.aws.amazon.com/amazonglacier/latest/dev/api-multipart-initiate-upload.html

On top of this I'm going to add auto detection of this issue for disks,
but not for S3 function, since those checks are quite costly, and they
need to write at least 5MB*3, which may be questionable.

v2: mark test as long [1]
    [1]: https://s3.amazonaws.com/clickhouse-test-reports/48492/d4a90880e947ab3a17b3eddeea274afed8ebceb4/stateless_tests_flaky_check__asan_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-28 11:01:56 +02:00
Azat Khuzhin
f5c3f3a4ab Remove quirk for s3_max_single_part_upload_size (when it exceed INT_MAX) 
Reverts: #48816
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-17 10:49:55 +02:00
Alexey Milovidov
8e0b1eae69
Update StorageS3Settings.cpp 2023-04-16 08:53:52 +03:00
Alexey Milovidov
1174f4dcbe Prevent a bug in AWS S3 2023-04-15 23:56:49 +02:00
Antonio Andelic
98b6944d37 Merge branch 'master' into s3-client-polish 2023-02-08 10:07:00 +00:00
Antonio Andelic
4cd361747d Address PR comment 2023-02-06 10:11:48 +00:00
xiedeyantu
f13eedd644 change settings name 2023-02-04 22:11:14 +08:00
xiedeyantu
562642ab7f add settings s3_allow_throw_if_mismatch_files 2023-02-03 12:27:13 +08:00
alesapin
631c8fb155
Merge branch 'master' into add-setting-allow-head-object-request 2023-01-29 14:58:27 +01:00
Vitaly Baranov
d02b255b55 Add new setting "allow_head_object_request" to S3::RequestSettings. 2023-01-27 15:10:34 +01:00
Alexander Tokmakov
bb4c8e169f check number of parameters in format string 2023-01-23 23:16:16 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Anton Popov
41a199e175
Fix crash when ListObjects request fails (#45371) 2023-01-20 20:10:23 +01:00
Kseniia Sumarokova
9ae5ad202a
Merge pull request #44707 from xiedeyantu/s3-add-config 
S3 support upload object using storage class INTELLIGENT_TIERING
 2023-01-09 13:28:53 +01:00
xiedeyantu
2f59b1d002 better 2023-01-05 00:27:07 +08:00
chen
72200273da
Update StorageS3Settings.cpp 2022-12-30 22:47:48 +08:00
xiedeyantu
c636b3e363 S3 support upload object using storage class INTELLIGENT_TIERING 2022-12-30 21:05:01 +08:00
kssenii
1d75f740d7 Fix tests 2022-12-20 22:33:54 +01:00
Nikolai Kochetov
a3226b2daf Remove extra compare operator. Review fixes. 2022-12-19 13:12:59 +00:00
Nikolai Kochetov
4032ee8203 Try to fix another one test 2022-12-18 12:31:17 +00:00
Nikolai Kochetov
2b999ce715 Fixing testx. 2022-12-17 21:23:08 +00:00
Nikolai Kochetov
f7c308077d Fixing build. 2022-12-17 17:12:04 +00:00
Nikolai Kochetov
b3278211bb Fixing tests. 2022-12-17 16:06:42 +00:00
Nikolai Kochetov
b2355a2212 Fixing tests. 2022-12-17 16:02:34 +00:00
Nikolai Kochetov
98ebef7914 Fixing special build. 2022-12-17 15:27:01 +00:00
Nikolai Kochetov
29c6caaeaf Validate s3 part upload settings. 2022-12-17 14:18:42 +00:00
Nikolai Kochetov
62ff98344e Validate s3 part upload settings. 2022-12-17 14:09:53 +00:00
Vitaly Baranov
b91af1b650 Fix initialization of s3 request settings. 2022-12-10 05:43:51 +01:00
Vitaly Baranov
0ba4870a18 Fix race in S3 multipart upload. 2022-12-09 03:02:39 +01:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling"" 2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling" 2022-11-16 22:33:48 +03:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler 2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421 rename ReadWriteSettings -> RequestSettings 2022-11-08 13:48:23 +00:00
serxa
2daec0b45e S3 request per second rate throttling + refactoring 2022-11-07 18:05:40 +00:00
Vitaly Baranov
32194c1200 Add max limitation for the size of an uploaded part. 2022-11-02 17:53:54 +01:00
Vitaly Baranov
b9f2f17331 Add test and logging. 2022-11-01 12:23:20 +01:00
Antonio Andelic
fc6c1faa57 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-21 06:57:58 +00:00
alesapin
d9c3549cc8 Revert s3 prefixes + add retries on write requests unexpected errors 2022-09-16 14:43:43 +02:00
Antonio Andelic
32a297339b Fix build 2022-09-15 13:36:16 +00:00
Antonio Andelic
9fdd2f2e61 Basic implementation for S3 snapshot upload 2022-09-15 09:11:55 +00:00
alesapin
07286dd042 Add paranoid option to check blobs to S3 after upload 2022-08-23 15:12:09 +02:00
Anton Popov
df6882d2b9
Revert "Fix errors of CheckTriviallyCopyableMove type" 2022-06-07 13:53:10 +02:00
HeenaBansal2009
b7eb6bbd38 Fixed clang-tidy-CheckTriviallyCopyableMove-errors 2022-05-30 11:09:03 -07:00
kssenii
0e05e1fa2f Fix 2022-04-19 11:31:27 +02:00
kssenii
ef2b238444 Fix 2022-04-08 12:30:24 +02:00
kssenii
4e50da722b Better s3 settings 2022-04-04 16:14:56 +02:00