kssenii
ab0dedf0c8
Simplify code around storage s3 configuration
2023-02-06 16:23:17 +01:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
Vitaly Baranov
aea9ccdb60
Pass request settings to S3::getObjectInfo().
2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed
Move getObjectInfo() to a separate header.
2023-01-27 15:09:38 +01:00
Anton Popov
8a1ea62aec
fix race on cancelation of query in StorageS3
2023-01-24 01:12:01 +00:00
Kruglov Pavel
04d95f4877
Fix
2023-01-16 16:47:04 +01:00
avogar
6cb7c4d175
Better commit, mark noexcept
2023-01-13 16:33:11 +00:00
avogar
76c89c6d20
Fix heap-use-after-free in reading from s3
2023-01-13 16:31:30 +00:00
Anton Popov
79e89cf69c
Merge pull request #44939 from ClickHouse/revert-44493-s3_optimize
...
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
2023-01-10 10:42:18 +01:00
Anton Popov
4447afb14d
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
2023-01-05 16:38:20 +01:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
xiedeyantu
d6a92fbd63
better
2022-12-28 10:11:33 +08:00
xiedeyantu
b5fd23358f
fixed
2022-12-23 19:11:51 +08:00
xiedeyantu
6a9ec7efb1
If user only need virtual columns, we don't need to initialize ReadBufferFromS3
2022-12-21 23:43:56 +08:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Anton Popov
8b9b8b083c
Merge pull request #43726 from CurtizJ/optimize-storage-s3
...
Improve performance of storage `S3` with large number of small files
2022-12-16 14:38:10 +01:00
kssenii
9aa6d31bce
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code
2022-12-13 22:25:10 +01:00
kssenii
fae817863c
Apply new code of named collections to s3
2022-12-13 22:19:09 +01:00
Anton Popov
6cd606ffeb
better saving of object info in iterator
2022-12-13 17:18:17 +00:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
kssenii
c7429d19e7
Merge remote-tracking branch 'upstream/master' into fix-progress-from-s3
2022-12-05 18:32:47 +01:00
Anton Popov
fe5fff0347
Merge pull request #43329 from xiedeyantu/support_nested_column
...
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-29 22:27:19 +01:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
xiedeyantu
304b6ebf3a
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-23 23:36:12 +08:00
kssenii
5e01441f61
Show progress bar while reading from s3 table function
2022-11-21 17:56:02 +01:00
Antonio Andelic
9a8d82f0fc
Merge branch 'master' into fix-describe-delta-lake-hudi
2022-11-21 08:23:33 +00:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Daniil Rubin
99d57325bb
Merge branch 'master' into hudi
2022-11-03 19:29:23 +00:00
Kruglov Pavel
21d50f76ea
Merge pull request #41979 from Avogar/s3-cluster-schema-inference
...
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-11-01 14:00:21 +01:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Antonio Andelic
7c7395575f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-10-17 11:02:36 +00:00
Kruglov Pavel
3d9f46a1e7
Merge branch 'master' into s3-cluster-schema-inference
2022-10-14 22:07:54 +02:00
avogar
c74b5c8126
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-09-30 16:59:17 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Daniil Rubin
95b5d2d29d
Merge branch 'master' into hudi
2022-09-28 14:24:55 +03:00
Daniil Rubin
13e50c771c
Fix bug, use static method
2022-09-28 11:03:43 +00:00
Antonio Andelic
655bde8e3f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-27 07:28:26 +00:00
Kseniia Sumarokova
9e97ca1df7
Update src/Storages/StorageS3.h
...
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
2022-09-22 17:59:26 +02:00
Antonio Andelic
9fdd2f2e61
Basic implementation for S3 snapshot upload
2022-09-15 09:11:55 +00:00
Kseniia Sumarokova
2cd6ceb203
Update StorageS3.h
2022-09-13 16:30:28 +02:00
kssenii
420ac4eb43
s3 header auth in ast
2022-09-13 15:13:28 +02:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
Nikolai Kochetov
5b4658aa5e
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-30 09:47:35 +00:00
