flynn
c49a293a9f
refactor and get rid of s3
2023-02-13 12:39:54 +00:00
flynn
d3dd9421da
refactor and get rid of s3
2023-02-13 08:29:22 +00:00
flynn
db15634a01
fix conflict
2023-02-10 08:41:04 +00:00
kssenii
ab0dedf0c8
Simplify code around storage s3 configuration
2023-02-06 16:23:17 +01:00
flynn
0dd8a61a8e
fix conflict
2023-02-06 03:25:12 +00:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
flynn
fd1ee98183
fix style
2023-01-31 14:40:48 +00:00
flynn
ffddc0dcce
fix conflict
2023-01-31 10:28:58 +00:00
flynn
fc2ce9e8e2
refactor and unify storage data lake
2023-01-29 14:53:56 +00:00
flynn
9b517cdc76
fix conflict
2023-01-29 08:34:56 +00:00
Vitaly Baranov
aea9ccdb60
Pass request settings to S3::getObjectInfo().
2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed
Move getObjectInfo() to a separate header.
2023-01-27 15:09:38 +01:00
Anton Popov
8a1ea62aec
fix race on cancelation of query in StorageS3
2023-01-24 01:12:01 +00:00
flynn
2fb32dc56c
fix and add test
2023-01-18 08:33:55 +00:00
Kruglov Pavel
04d95f4877
Fix
2023-01-16 16:47:04 +01:00
avogar
6cb7c4d175
Better commit, mark noexcept
2023-01-13 16:33:11 +00:00
avogar
76c89c6d20
Fix heap-use-after-free in reading from s3
2023-01-13 16:31:30 +00:00
Anton Popov
79e89cf69c
Merge pull request #44939 from ClickHouse/revert-44493-s3_optimize
...
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
2023-01-10 10:42:18 +01:00
Anton Popov
4447afb14d
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
2023-01-05 16:38:20 +01:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
xiedeyantu
d6a92fbd63
better
2022-12-28 10:11:33 +08:00
xiedeyantu
b5fd23358f
fixed
2022-12-23 19:11:51 +08:00
xiedeyantu
6a9ec7efb1
If user only need virtual columns, we don't need to initialize ReadBufferFromS3
2022-12-21 23:43:56 +08:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Anton Popov
8b9b8b083c
Merge pull request #43726 from CurtizJ/optimize-storage-s3
...
Improve performance of storage `S3` with large number of small files
2022-12-16 14:38:10 +01:00
kssenii
9aa6d31bce
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code
2022-12-13 22:25:10 +01:00
kssenii
fae817863c
Apply new code of named collections to s3
2022-12-13 22:19:09 +01:00
Anton Popov
6cd606ffeb
better saving of object info in iterator
2022-12-13 17:18:17 +00:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
kssenii
c7429d19e7
Merge remote-tracking branch 'upstream/master' into fix-progress-from-s3
2022-12-05 18:32:47 +01:00
Anton Popov
fe5fff0347
Merge pull request #43329 from xiedeyantu/support_nested_column
...
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-29 22:27:19 +01:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
xiedeyantu
304b6ebf3a
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-23 23:36:12 +08:00
kssenii
5e01441f61
Show progress bar while reading from s3 table function
2022-11-21 17:56:02 +01:00
Antonio Andelic
9a8d82f0fc
Merge branch 'master' into fix-describe-delta-lake-hudi
2022-11-21 08:23:33 +00:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Daniil Rubin
99d57325bb
Merge branch 'master' into hudi
2022-11-03 19:29:23 +00:00
Kruglov Pavel
21d50f76ea
Merge pull request #41979 from Avogar/s3-cluster-schema-inference
...
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-11-01 14:00:21 +01:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Antonio Andelic
7c7395575f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-10-17 11:02:36 +00:00
Kruglov Pavel
3d9f46a1e7
Merge branch 'master' into s3-cluster-schema-inference
2022-10-14 22:07:54 +02:00
avogar
c74b5c8126
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-09-30 16:59:17 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Daniil Rubin
95b5d2d29d
Merge branch 'master' into hudi
2022-09-28 14:24:55 +03:00
Daniil Rubin
13e50c771c
Fix bug, use static method
2022-09-28 11:03:43 +00:00
Antonio Andelic
655bde8e3f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-27 07:28:26 +00:00
Kseniia Sumarokova
9e97ca1df7
Update src/Storages/StorageS3.h
...
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
2022-09-22 17:59:26 +02:00
Antonio Andelic
9fdd2f2e61
Basic implementation for S3 snapshot upload
2022-09-15 09:11:55 +00:00
Kseniia Sumarokova
2cd6ceb203
Update StorageS3.h
2022-09-13 16:30:28 +02:00
kssenii
420ac4eb43
s3 header auth in ast
2022-09-13 15:13:28 +02:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
Nikolai Kochetov
5b4658aa5e
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-30 09:47:35 +00:00
alesapin
c7b16065e1
Merge with master
2022-05-25 21:47:05 +02:00
alesapin
6f5c86e55e
Merge branch 'master' into i_object_storage
2022-05-25 20:49:01 +02:00
Nikolai Kochetov
1b85f2c1d6
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-25 16:27:40 +02:00
Kseniia Sumarokova
b50d4549c9
Merge pull request #37356 from amosbird/partition-prune-for-s3
...
"Partition pruning" for s3
2022-05-25 11:03:07 +02:00
Amos Bird
093d315756
partition pruning for s3
2022-05-24 18:57:55 +08:00
Nikolai Kochetov
56feef01e7
Move some resources
2022-05-20 19:49:31 +00:00
avogar
68bb07d166
Better naming
2022-05-13 18:39:19 +00:00
avogar
b17fec659a
Improve performance and memory usage for select of subset of columns for some formats
2022-05-13 13:51:28 +00:00
alesapin
5dcd25be23
Initial implementation
2022-05-12 00:04:54 +02:00
Anton Popov
e911900054
remove last mentions of data streams
2022-05-09 19:15:24 +00:00
Kruglov Pavel
c6e26f6128
Merge pull request #36434 from Avogar/fix-s3-cluster
...
Fix bug in s3Cluster schema inference
2022-05-05 13:08:49 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable
...
... IStorage has deleted copy ctor / assignment already
2022-05-03 09:07:08 +02:00
Kruglov Pavel
c3570b9f7e
Merge branch 'master' into fix-s3-cluster
2022-05-02 14:15:21 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying
...
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
previously allowed.
Hence, this change
- removes shared_ptr_helper and as a result all inherited create() methods,
- instead, Storage objects are now created using make_shared<>() by the
caller (for that to work, many constructors had to be made public), and
- all Storage classes were marked as noncopyable using boost::noncopyable.
In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
2022-05-02 08:46:52 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common
2022-04-29 10:26:35 +08:00
Kruglov Pavel
7fb0f2284d
Merge branch 'master' into fix-s3-cluster
2022-04-21 13:39:52 +02:00
Sergei Trifonov
34842f209c
merge master and fix conflicts
2022-04-20 18:35:56 +02:00
avogar
9a81b67ba5
Fix bug in s3Cluster schema inference
2022-04-19 18:23:04 +00:00
kssenii
ef2b238444
Fix
2022-04-08 12:30:24 +02:00
Sergei Trifonov
01425df758
fix build
2022-04-07 16:07:12 +02:00
Sergei Trifonov
9c2804b924
merge from master + resolve conflicts
2022-04-07 11:22:01 +02:00
kssenii
4e50da722b
Better s3 settings
2022-04-04 16:14:56 +02:00
kssenii
98e85dd68e
Fix build
2022-03-30 20:27:04 +02:00
kssenii
099fa0e668
Merge master
2022-03-30 17:33:32 +02:00
kssenii
a2013ebe2b
Better
2022-03-28 22:40:27 +02:00
Antonio Andelic
d9d826c813
Address PR review
2022-03-28 08:19:23 +00:00
Antonio Andelic
93ad209910
Attach memory tracker
2022-03-23 11:52:31 +00:00
Antonio Andelic
131b3a091c
Refactor StorageS3
2022-03-23 08:57:01 +00:00
Anton Popov
0ba78c3c3a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-16 15:28:09 +00:00
Saad Ur Rahman
679abaafdc
[Storages] bugfix
...
Extracting Version Id for S3::URI.
2022-03-14 22:25:40 -04:00
Saad Ur Rahman
171260f40b
[Storages] Added Version ID parameter for S3.
2022-03-14 22:03:58 -04:00
Maksim Kita
2fdcf53a76
Fix clang-tidy warnings in Server, Storages folders
2022-03-14 18:17:35 +00:00
Anton Popov
df3b07fe7c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-03 22:25:28 +00:00
kssenii
9b64a8fe39
Fix odbc bridge
2022-02-28 14:29:05 +01:00
kssenii
003b807b00
Fix
2022-02-23 20:33:05 +01:00
Anton Popov
18940b8637
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-09 23:38:38 +03:00
alesapin
f6e7cea1f8
Add settings for S3 multipart upload parts size
2022-02-08 19:38:04 +03:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
alesapin
1adc7538fe
Fix benign race condition for storage HDFS, S3, URL
2022-01-31 13:50:28 +03:00
Anton Popov
78b9f15abb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-30 03:24:37 +03:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
