Michael Kolupaev
eb3b774ad0
Better control over Parquet row group size
2023-05-04 14:59:55 -07:00
kssenii
1433f5ffc9
Merge remote-tracking branch 'upstream/master' into remove-dependency-from-context
2023-05-04 13:24:02 +02:00
Antonio Andelic
a68a023ca7
Merge pull request #48724 from johanngan/sse-kms
...
Support SSE-KMS configuration with S3 client
2023-05-04 13:20:54 +02:00
alesapin
412b161104
Merge pull request #48791 from kssenii/better-local-object-storage
...
Make local object storage work consistently with s3 object storage, fix problem with append, make it configurable as independent storage
2023-05-04 11:47:43 +02:00
johanngan
731823b873
Add support for SSE-KMS configuration with S3
...
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html
Similar to the server_side_encryption_customer_key_base64 option for
configuring SSE-C with S3, add the following settings to configure
SSE-KMS on a per-endpoint/disk basis:
- server_side_encryption_kms_key_id
- server_side_encryption_kms_encryption_context
- server_side_encryption_kms_bucket_key_enabled
2023-05-03 21:35:38 -05:00
Nikita Mikhaylov
954e3b724c
Speedup outdated parts loading ( #49317 )
2023-05-03 18:56:45 +02:00
kssenii
189f276ff5
Fix
2023-05-03 13:16:08 +02:00
kssenii
ecfbf1e304
Remove dependency from DB::Context in readers
2023-05-02 21:45:27 +02:00
Kseniia Sumarokova
45e2d296f9
Merge branch 'master' into better-local-object-storage
2023-04-27 14:54:04 +02:00
avogar
c503f6532c
Add more finalize() to avoid terminate
2023-04-24 15:11:36 +00:00
Alexey Milovidov
67de39c2d9
Merge pull request #48727 from ClickHouse/parallel-processing-from-storages
...
Parallelize query processing right after reading FROM ...
2023-04-23 23:10:32 +03:00
kssenii
d8023806a9
Merge remote-tracking branch 'upstream/master' into better-local-object-storage
2023-04-23 12:39:34 +02:00
Kseniia Sumarokova
bd748045ad
Fix typo
2023-04-21 18:54:23 +02:00
Igor Nikonov
d5eb65b5ea
Remove redundant narrowPipe()
2023-04-18 22:41:28 +00:00
kssenii
16b027ed0c
Merge remote-tracking branch 'upstream/master' into better-local-object-storage
2023-04-18 16:25:08 +02:00
kssenii
b77e9c1ef0
Merge remote-tracking branch 'upstream/master' into better-local-object-storage
2023-04-17 16:44:10 +02:00
kssenii
d2c73a5522
Better
2023-04-17 16:41:21 +02:00
Michael Kolupaev
473f212c82
Hopefully fix assertion failure in CachedOnDiskReadBufferFromFile
2023-04-17 04:58:32 +00:00
Michael Kolupaev
87be78e6de
Better
2023-04-17 04:58:32 +00:00
Michael Kolupaev
2d4fe85513
Something
2023-04-17 04:58:32 +00:00
kssenii
3fb4cd0f52
Fix s3 test
2023-04-05 14:13:46 +02:00
kssenii
a3d69694f4
Fix build
2023-04-04 23:13:17 +02:00
kssenii
f44c53b97a
Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes
2023-04-04 22:41:22 +02:00
Antonio Andelic
a329d80bfa
Merge pull request #47397 from ClickHouse/enable-env-credentials-default
...
Enable `use_environment_credentials` by default
2023-04-04 10:00:03 +02:00
kssenii
9b3d0ec86d
Adjustments after conflicts
2023-04-03 19:53:34 +02:00
kssenii
8915f49b7d
Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes
2023-04-03 17:43:42 +02:00
kssenii
5578cb08ad
Fix s3 cluster
2023-04-03 14:40:04 +02:00
Anton Popov
f715bd95f1
fix writing to StorageS3
2023-03-31 14:08:28 +00:00
Antonio Andelic
e982f2a67a
Merge branch 'master' into enable-env-credentials-default
2023-03-31 09:11:01 +00:00
Anton Popov
5ceb855e7f
Merge branch 'master' into fix-race-storage-s3
2023-03-31 04:16:35 +02:00
Anton Popov
38389d878c
fix one more race in StorageS3
2023-03-30 21:06:53 +00:00
kssenii
319417062f
Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes
2023-03-30 18:29:46 +02:00
Antonio Andelic
80cb121d2a
Merge pull request #48092 from ClickHouse/nosign-keyword-for-s3
...
Add support for `NOSIGN` keyword and `no_sign_request` config for S3
2023-03-30 18:10:56 +02:00
Anton Popov
e72472e71b
Merge branch 'master' into fix-race-storage-s3
2023-03-30 16:19:57 +02:00
kssenii
539414554f
Fix s3
2023-03-30 15:32:38 +02:00
Antonio Andelic
9db58532f4
Clang-tidy fix
2023-03-30 08:41:14 +02:00
Anton Popov
ed29c141fb
fix race in StorageS3
2023-03-29 22:13:45 +00:00
Antonio Andelic
7b1ad221b2
Address PR comments
2023-03-29 11:08:44 +00:00
Azat Khuzhin
f38a7aeabe
ThreadPool metrics introspection
...
There are lots of thread pools and simple local-vs-global is not enough
already, it is good to know which one in particular uses threads.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-03-29 10:46:59 +02:00
kssenii
82b642c9c6
Fix style check
2023-03-28 21:57:14 +02:00
kssenii
13f29a7242
Better
2023-03-28 18:57:24 +02:00
Antonio Andelic
160aa186bb
Add support for NOSIGN keyword and no_sign_request config
2023-03-28 07:05:35 +00:00
kssenii
36cc6fee51
Rewrite data lakes (part 1)
2023-03-24 22:35:12 +01:00
kssenii
cae3b335d6
Merge remote-tracking branch 'upstream/master' into named-collections-finish
2023-03-20 11:23:22 +01:00
Antonio Andelic
1b7401b58a
Update src/Storages/StorageS3.cpp
...
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
2023-03-17 15:46:15 +01:00
Antonio Andelic
a170a909a4
Add expiration window for S3 credentials
2023-03-10 10:06:32 +00:00
Antonio Andelic
5bc21538e5
Enable use_environment_credentials by default
2023-03-09 10:31:55 +00:00
kssenii
8f2d75cef8
Fix tests
2023-03-05 12:56:00 +01:00
flynn
b3a9468661
fix
2023-02-17 12:42:24 +00:00
flynn
a39f6f419b
refactor
2023-02-17 08:27:52 +00:00
flynn
ecc39978d7
fix conflict
2023-02-16 02:23:55 +00:00
Kruglov Pavel
4f380370a9
Fix s3Cluster schema inference in parallel distributed insert select ( #46381 )
...
* Fix s3Cluster schema inference in parallel distributed insert select
* Try fix flaky test
* Try SYSTEM SYNC REPLICA to avoid test flakiness
2023-02-15 15:30:43 +01:00
flynn
289c5c60d3
fix
2023-02-15 11:24:06 +00:00
Antonio Andelic
5ab24285fc
Fix arg parsing
2023-02-14 08:33:59 +00:00
Antonio Andelic
3a6ea861d8
Extract common argument parsing logic
2023-02-13 12:27:49 +00:00
kssenii
3067c1d723
Merge remote-tracking branch 'upstream/master' into resubmit-prefetches
2023-02-11 11:36:23 +01:00
kssenii
b0b865c32e
Resubmit prefetches
2023-02-08 21:26:24 +01:00
kssenii
9485873a2f
Fix integration test
2023-02-07 12:45:23 +01:00
kssenii
ab0dedf0c8
Simplify code around storage s3 configuration
2023-02-06 16:23:17 +01:00
Kseniia Sumarokova
38c001ca42
Merge pull request #45957 from xiedeyantu/s3_file_not_found
...
Throw an error on no files satisfying S3 wildcard
2023-02-06 12:32:12 +01:00
xiedeyantu
f13eedd644
change settings name
2023-02-04 22:11:14 +08:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
xiedeyantu
562642ab7f
add settings s3_allow_throw_if_mismatch_files
2023-02-03 12:27:13 +08:00
xiedeyantu
e22cc0eb98
Throw an error on no files satisfying S3 wildcard
2023-02-02 19:13:34 +08:00
Vitaly Baranov
af74c008c4
Use one request to implement S3ObjectStorage::getObjectMetadata instead of two ones.
2023-01-27 18:42:37 +01:00
Vitaly Baranov
aea9ccdb60
Pass request settings to S3::getObjectInfo().
2023-01-27 15:10:09 +01:00
Anton Popov
b58b73b0e7
Merge pull request #45529 from CurtizJ/fix-storage-s3-race
...
Try to fix test `test_storage_s3/test.py::test_wrong_s3_syntax` (race in `StorageS3`)
2023-01-26 14:21:32 +01:00
Anton Popov
8a1ea62aec
fix race on cancelation of query in StorageS3
2023-01-24 01:12:01 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Anton Popov
41a199e175
Fix crash when ListObjects
request fails ( #45371 )
2023-01-20 20:10:23 +01:00
Anton Popov
f40fd7a151
Add checks for compilation of regexps ( #45356 )
2023-01-17 23:46:04 +01:00
Vitaly Baranov
a955504043
Move throw_on_error
parameter to the end.
2023-01-15 20:28:16 +01:00
Maksim Kita
4571c74fdd
Fixed build
2023-01-10 16:49:55 +01:00
Anton Popov
79e89cf69c
Merge pull request #44939 from ClickHouse/revert-44493-s3_optimize
...
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
2023-01-10 10:42:18 +01:00
Anton Popov
4447afb14d
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
2023-01-05 16:38:20 +01:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
Antonio Andelic
e0b8fd528d
Merge pull request #44842 from ClickHouse/fix-data-race-storage-s3
...
Fix data race in StorageS3
2023-01-03 09:20:08 +01:00
Kruglov Pavel
966f57ef68
Merge pull request #42777 from Avogar/improve-streaming-engines
...
Refactor and Improve streaming engines Kafka/RabbitMQ/NATS and data formats
2023-01-02 15:59:06 +01:00
Antonio Andelic
e07d820156
Fix race on total_size
2023-01-02 14:28:27 +00:00
Kruglov Pavel
0844fe7089
Merge pull request #44493 from xiedeyantu/s3_optimize
...
If user only need virtual columns, we don't need to initialize ReadBufferFromS3
2022-12-30 15:44:20 +01:00
xiedeyantu
d6a92fbd63
better
2022-12-28 10:11:33 +08:00
xiedeyantu
2dd809e403
fix
2022-12-23 19:45:26 +08:00
xiedeyantu
68aeb39892
fix
2022-12-23 19:33:08 +08:00
xiedeyantu
d0eb22a1cd
fix
2022-12-23 19:25:14 +08:00
xiedeyantu
b5fd23358f
fixed
2022-12-23 19:11:51 +08:00
chen
5c8fb627b3
Update StorageS3.cpp
2022-12-22 08:28:09 +08:00
chen
9fc1020855
Update StorageS3.cpp
2022-12-22 08:15:50 +08:00
xiedeyantu
6a9ec7efb1
If user only need virtual columns, we don't need to initialize ReadBufferFromS3
2022-12-21 23:43:56 +08:00
Kruglov Pavel
5e01a3d74e
Merge branch 'master' into improve-streaming-engines
2022-12-21 10:51:50 +01:00
kssenii
1d75f740d7
Fix tests
2022-12-20 22:33:54 +01:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
Raúl Marín
45d27f461b
Merge branch 'master' into perf_experiment
2022-12-20 09:07:48 +00:00
Nikolai Kochetov
f7c308077d
Fixing build.
2022-12-17 17:12:04 +00:00
Nikolai Kochetov
b3278211bb
Fixing tests.
2022-12-17 16:06:42 +00:00
Nikolai Kochetov
b2355a2212
Fixing tests.
2022-12-17 16:02:34 +00:00
Nikolai Kochetov
98ebef7914
Fixing special build.
2022-12-17 15:27:01 +00:00
Nikolai Kochetov
62ff98344e
Validate s3 part upload settings.
2022-12-17 14:09:53 +00:00
kssenii
2ce5af421e
Replace old named collections code for mongo
2022-12-17 00:50:25 +01:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Anton Popov
8b9b8b083c
Merge pull request #43726 from CurtizJ/optimize-storage-s3
...
Improve performance of storage `S3` with large number of small files
2022-12-16 14:38:10 +01:00