Sema Checherinda
8bc9a32d19
fix special build
2023-05-10 18:45:59 +00:00
johanngan
731823b873
Add support for SSE-KMS configuration with S3
...
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html
Similar to the server_side_encryption_customer_key_base64 option for
configuring SSE-C with S3, add the following settings to configure
SSE-KMS on a per-endpoint/disk basis:
- server_side_encryption_kms_key_id
- server_side_encryption_kms_encryption_context
- server_side_encryption_kms_bucket_key_enabled
2023-05-03 21:35:38 -05:00
johanngan
1bc52977a7
Support "header" setting in S3 disk config
...
The S3 table engine supports specifying extra HTTP headers in S3
requests to certain endpoints, via the "headers" setting. This commit
adds the same setting to S3 disk config.
2023-05-03 21:32:58 -05:00
Antonio Andelic
160aa186bb
Add support for NOSIGN keyword and no_sign_request config
2023-03-28 07:05:35 +00:00
Antonio Andelic
a170a909a4
Add expiration window for S3 credentials
2023-03-10 10:06:32 +00:00
Antonio Andelic
ae905b5762
Split S3Common into multiple files
2023-02-13 12:28:04 +00:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
Vitaly Baranov
a8304525ed
Move getObjectInfo() to a separate header.
2023-01-27 15:09:38 +01:00
Vitaly Baranov
9a52087989
More complex logic: GetObjectAttributes requests will be used
...
only if the endpoint is "*.amazonaws.com", otherwise HeadObject requests will be used.
2023-01-16 20:14:39 +01:00
Vitaly Baranov
16a20cd06e
Use std::string_view instead of const std::string_view &
2023-01-16 10:18:04 +01:00
Vitaly Baranov
e435edb4ab
Make checkObjectExists() easier.
2023-01-16 10:06:20 +01:00
Vitaly Baranov
a955504043
Move throw_on_error parameter to the end.
2023-01-15 20:28:16 +01:00
Vitaly Baranov
21b8aaeb8b
Stop using HeadObject requests in S3
...
because they don't work well with endpoints without explicit region.
2023-01-15 20:28:11 +01:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
kssenii
f689fa5964
Fix style check
2022-12-18 20:16:28 +01:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
kssenii
c7429d19e7
Merge remote-tracking branch 'upstream/master' into fix-progress-from-s3
2022-12-05 18:32:47 +01:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
Kseniia Sumarokova
5c90d5aa7e
Merge pull request #43253 from xiedeyantu/fix-s3-glob
...
fix s3 support question mark wildcard
2022-11-22 14:26:56 +01:00
kssenii
5e01441f61
Show progress bar while reading from s3 table function
2022-11-21 17:56:02 +01:00
xiedeyantu
c258d3ac8b
fix s3 support question mark wildcard
2022-11-18 12:11:22 +08:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Antonio Andelic
7c7395575f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-10-17 11:02:36 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Antonio Andelic
655bde8e3f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-27 07:28:26 +00:00
kssenii
f917b268b3
Merge remote-tracking branch 'upstream/master' into s3-header-auth
2022-09-21 12:56:25 +02:00
Antonio Andelic
fc6c1faa57
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-21 06:57:58 +00:00
serxa
2ef696ffe1
fix issues
2022-09-19 18:40:32 +00:00
Antonio Andelic
ad7864cb5f
enable S3 if client is built
2022-09-15 14:06:22 +00:00
Antonio Andelic
32a297339b
Fix build
2022-09-15 13:36:16 +00:00
Antonio Andelic
9fdd2f2e61
Basic implementation for S3 snapshot upload
2022-09-15 09:11:55 +00:00
alesapin
ad279940d6
Trying print less scary messages in case No such key for outdated parts
2022-09-14 17:25:58 +02:00
kssenii
420ac4eb43
s3 header auth in ast
2022-09-13 15:13:28 +02:00
alesapin
43493389b9
Merge branch 'master' into one_more_logging
2022-09-06 19:40:24 +02:00
alesapin
422b1658eb
Review fix
2022-09-06 14:42:48 +02:00
alesapin
6ea7f1e011
Better exception handling for ReadBufferFromS3
2022-09-06 13:59:55 +02:00
Sergei Trifonov
bcb6475c4a
add separate s3 profile events for disk s3
2022-09-01 18:30:55 +02:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
Vladimir Chebotarev
5fcf840156
Typo.
2022-06-02 08:43:44 +03:00
alesapin
4f1d4dde22
Disable s3 logging by default
2022-05-16 00:16:00 +02:00
alesapin
5dcd25be23
Initial implementation
2022-05-12 00:04:54 +02:00
Sergei Trifonov
9c2804b924
merge from master + resolve conflicts
2022-04-07 11:22:01 +02:00
Antonio Andelic
04e6e6ab5e
Add ParallelReadBuffer for S3
2022-03-21 14:52:26 +00:00
Saad Ur Rahman
ec28af055f
[IO] S3 URI data struct contains Version ID
...
Added Version ID field to S3::URI struct.
TODO: Extract version id if present.
2022-03-14 22:11:18 -04:00
Maksim Kita
e30117a3d6
Fix clang-tidy warnings in Interpreters, IO folders
2022-03-14 18:17:35 +00:00
Alexey Milovidov
8b4a6a2416
Remove cruft
2021-10-28 02:10:39 +03:00
Alexey Milovidov
fe6b7c77c7
Rename "common" to "base"
2021-10-02 10:13:14 +03:00
