flynn
2fb32dc56c
fix and add test
2023-01-18 08:33:55 +00:00
Kruglov Pavel
04d95f4877
Fix
2023-01-16 16:47:04 +01:00
avogar
6cb7c4d175
Better commit, mark noexcept
2023-01-13 16:33:11 +00:00
avogar
76c89c6d20
Fix heap-use-after-free in reading from s3
2023-01-13 16:31:30 +00:00
Anton Popov
79e89cf69c
Merge pull request #44939 from ClickHouse/revert-44493-s3_optimize
...
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
2023-01-10 10:42:18 +01:00
Anton Popov
4447afb14d
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
2023-01-05 16:38:20 +01:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
xiedeyantu
d6a92fbd63
better
2022-12-28 10:11:33 +08:00
xiedeyantu
b5fd23358f
fixed
2022-12-23 19:11:51 +08:00
xiedeyantu
6a9ec7efb1
If user only need virtual columns, we don't need to initialize ReadBufferFromS3
2022-12-21 23:43:56 +08:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Anton Popov
8b9b8b083c
Merge pull request #43726 from CurtizJ/optimize-storage-s3
...
Improve performance of storage `S3` with large number of small files
2022-12-16 14:38:10 +01:00
kssenii
9aa6d31bce
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code
2022-12-13 22:25:10 +01:00
kssenii
fae817863c
Apply new code of named collections to s3
2022-12-13 22:19:09 +01:00
Anton Popov
6cd606ffeb
better saving of object info in iterator
2022-12-13 17:18:17 +00:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
kssenii
c7429d19e7
Merge remote-tracking branch 'upstream/master' into fix-progress-from-s3
2022-12-05 18:32:47 +01:00
Anton Popov
fe5fff0347
Merge pull request #43329 from xiedeyantu/support_nested_column
...
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-29 22:27:19 +01:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
xiedeyantu
304b6ebf3a
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-23 23:36:12 +08:00
kssenii
5e01441f61
Show progress bar while reading from s3 table function
2022-11-21 17:56:02 +01:00
Antonio Andelic
9a8d82f0fc
Merge branch 'master' into fix-describe-delta-lake-hudi
2022-11-21 08:23:33 +00:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Daniil Rubin
99d57325bb
Merge branch 'master' into hudi
2022-11-03 19:29:23 +00:00
Kruglov Pavel
21d50f76ea
Merge pull request #41979 from Avogar/s3-cluster-schema-inference
...
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-11-01 14:00:21 +01:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Antonio Andelic
7c7395575f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-10-17 11:02:36 +00:00
Kruglov Pavel
3d9f46a1e7
Merge branch 'master' into s3-cluster-schema-inference
2022-10-14 22:07:54 +02:00
avogar
c74b5c8126
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-09-30 16:59:17 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Daniil Rubin
95b5d2d29d
Merge branch 'master' into hudi
2022-09-28 14:24:55 +03:00
Daniil Rubin
13e50c771c
Fix bug, use static method
2022-09-28 11:03:43 +00:00
Antonio Andelic
655bde8e3f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-27 07:28:26 +00:00
Kseniia Sumarokova
9e97ca1df7
Update src/Storages/StorageS3.h
...
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
2022-09-22 17:59:26 +02:00
Antonio Andelic
9fdd2f2e61
Basic implementation for S3 snapshot upload
2022-09-15 09:11:55 +00:00
Kseniia Sumarokova
2cd6ceb203
Update StorageS3.h
2022-09-13 16:30:28 +02:00
kssenii
420ac4eb43
s3 header auth in ast
2022-09-13 15:13:28 +02:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
Nikolai Kochetov
5b4658aa5e
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-30 09:47:35 +00:00
alesapin
c7b16065e1
Merge with master
2022-05-25 21:47:05 +02:00
alesapin
6f5c86e55e
Merge branch 'master' into i_object_storage
2022-05-25 20:49:01 +02:00
Nikolai Kochetov
1b85f2c1d6
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-25 16:27:40 +02:00
Kseniia Sumarokova
b50d4549c9
Merge pull request #37356 from amosbird/partition-prune-for-s3
...
"Partition pruning" for s3
2022-05-25 11:03:07 +02:00
Amos Bird
093d315756
partition pruning for s3
2022-05-24 18:57:55 +08:00
Nikolai Kochetov
56feef01e7
Move some resources
2022-05-20 19:49:31 +00:00
avogar
68bb07d166
Better naming
2022-05-13 18:39:19 +00:00
avogar
b17fec659a
Improve performance and memory usage for select of subset of columns for some formats
2022-05-13 13:51:28 +00:00
alesapin
5dcd25be23
Initial implementation
2022-05-12 00:04:54 +02:00
Anton Popov
e911900054
remove last mentions of data streams
2022-05-09 19:15:24 +00:00
Kruglov Pavel
c6e26f6128
Merge pull request #36434 from Avogar/fix-s3-cluster
...
Fix bug in s3Cluster schema inference
2022-05-05 13:08:49 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable
...
... IStorage has deleted copy ctor / assignment already
2022-05-03 09:07:08 +02:00
Kruglov Pavel
c3570b9f7e
Merge branch 'master' into fix-s3-cluster
2022-05-02 14:15:21 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying
...
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
previously allowed.
Hence, this change
- removes shared_ptr_helper and as a result all inherited create() methods,
- instead, Storage objects are now created using make_shared<>() by the
caller (for that to work, many constructors had to be made public), and
- all Storage classes were marked as noncopyable using boost::noncopyable.
In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
2022-05-02 08:46:52 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common
2022-04-29 10:26:35 +08:00
Kruglov Pavel
7fb0f2284d
Merge branch 'master' into fix-s3-cluster
2022-04-21 13:39:52 +02:00
Sergei Trifonov
34842f209c
merge master and fix conflicts
2022-04-20 18:35:56 +02:00
avogar
9a81b67ba5
Fix bug in s3Cluster schema inference
2022-04-19 18:23:04 +00:00
kssenii
ef2b238444
Fix
2022-04-08 12:30:24 +02:00
Sergei Trifonov
01425df758
fix build
2022-04-07 16:07:12 +02:00
Sergei Trifonov
9c2804b924
merge from master + resolve conflicts
2022-04-07 11:22:01 +02:00
kssenii
4e50da722b
Better s3 settings
2022-04-04 16:14:56 +02:00
kssenii
98e85dd68e
Fix build
2022-03-30 20:27:04 +02:00
kssenii
099fa0e668
Merge master
2022-03-30 17:33:32 +02:00
kssenii
a2013ebe2b
Better
2022-03-28 22:40:27 +02:00
Antonio Andelic
d9d826c813
Address PR review
2022-03-28 08:19:23 +00:00
Antonio Andelic
93ad209910
Attach memory tracker
2022-03-23 11:52:31 +00:00
Antonio Andelic
131b3a091c
Refactor StorageS3
2022-03-23 08:57:01 +00:00
Anton Popov
0ba78c3c3a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-16 15:28:09 +00:00
Saad Ur Rahman
679abaafdc
[Storages] bugfix
...
Extracting Version Id for S3::URI.
2022-03-14 22:25:40 -04:00
Saad Ur Rahman
171260f40b
[Storages] Added Version ID parameter for S3.
2022-03-14 22:03:58 -04:00
Maksim Kita
2fdcf53a76
Fix clang-tidy warnings in Server, Storages folders
2022-03-14 18:17:35 +00:00
Anton Popov
df3b07fe7c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-03 22:25:28 +00:00
kssenii
9b64a8fe39
Fix odbc bridge
2022-02-28 14:29:05 +01:00
kssenii
003b807b00
Fix
2022-02-23 20:33:05 +01:00
Anton Popov
18940b8637
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-09 23:38:38 +03:00
alesapin
f6e7cea1f8
Add settings for S3 multipart upload parts size
2022-02-08 19:38:04 +03:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
alesapin
1adc7538fe
Fix benign race condition for storage HDFS, S3, URL
2022-01-31 13:50:28 +03:00
Anton Popov
78b9f15abb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-30 03:24:37 +03:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
avogar
97788b9c21
Allow to create new files on insert for File/S3/HDFS engines
2021-12-29 21:19:13 +03:00
avogar
8112a71233
Implement schema inference for most input formats
2021-12-29 12:18:56 +03:00
kssenii
1f6ca619b7
Allow some killing
2021-12-27 22:42:56 +03:00
Anton Popov
a20922b2d3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-11-09 15:36:25 +03:00
kssenii
20772fde04
For storage
2021-10-27 10:04:17 +03:00
Alexey Milovidov
fe6b7c77c7
Rename "common" to "base"
2021-10-02 10:13:14 +03:00
Anton Popov
4c388e3d84
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-09-09 14:10:16 +03:00
kssenii
3deb9a0ecb
s3, url, http headers
2021-09-07 14:17:25 +03:00
Nikita Mikhaylov
ae241fc129
done
2021-08-31 12:54:06 +00:00
Vladimir Chebotarev
d1efe3afa7
Review fixes.
2021-08-23 09:22:13 +03:00
Anton Popov
e36736b50c
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-08-02 22:52:02 +03:00
Nikolai Kochetov
9b5a816b43
Merge branch 'master' into output-streams-to-processors
2021-07-26 18:03:11 +03:00
Nikolai Kochetov
9c92f43359
Update storages.
2021-07-23 22:33:59 +03:00
Nikolai Kochetov
179ec05a72
Remove some streams.
2021-07-20 21:18:43 +03:00
Anton Popov
3ed7f5a6cc
dynamic subcolumns: add snapshot for storage
2021-07-09 06:15:41 +03:00
Kseniia Sumarokova
ed687f45a1
Merge branch 'master' into truncate
2021-06-30 20:08:42 +03:00
Raúl Marín
bfc122df64
Fix some typos in Storage classes
2021-06-28 19:03:56 +02:00
kssenii
ac0f86cdbf
Truncate for s3
2021-06-21 15:44:36 +00:00
Maksim Kita
67e9b85951
Merge ext into common
2021-06-16 23:28:41 +03:00
Nikita Mikhaylov
0055a924f6
better
2021-06-01 14:23:46 +00:00
Nikita Mikhaylov
f33b70c54a
better
2021-05-20 00:42:25 +03:00
Vladimir Chebotarev
bdf03387ab
Better S3 read retries. Renamed s3_max_single_read_retries -> s3_single_read_retry_attempts, added sleeps before next attempts.
2021-05-17 10:38:32 +03:00
feng lv
0f5c05854c
fix conflict
2021-04-23 12:22:51 +00:00
feng lv
4ffe199d39
Implement table comments
2021-04-23 12:18:23 +00:00
Vladimir Chebotarev
9d4295f261
Removed infinite retries.
2021-04-19 11:02:36 +03:00
Vladimir Chebotarev
95866e6e8d
Rebase fix.
2021-04-15 10:57:08 +03:00
Vladimir Chebotarev
10bf56e540
Attempt to add retries to S3 reads.
2021-04-15 09:47:19 +03:00
Nikita Mikhaylov
ec35a878d3
simplify storages3cluster
2021-04-13 23:17:25 +03:00
Nikita Mikhaylov
024374a2ec
review fixes
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
09a62e713a
rename to s3Cluster
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
a743442a17
build fixes
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
c333c3dedb
review fixes
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
7276b40556
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
587fbdd10d
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
cef9e19eb2
better
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
64b4cd0e63
save
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
44ca65a9a4
save
2021-04-13 22:39:41 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Pervakov Grigorii
eb87afc4d4
recreate S3 client if credentials changed
2021-03-10 13:36:57 +03:00
Vladimir Chebotarev
5cbc25c647
Added global setting s3_max_connections.
2021-01-05 13:34:55 +03:00
Pavel Kovalenko
db7a87089b
Adaptive choose of single/multi part upload in WriteBufferFromS3.
2020-12-09 17:09:04 +03:00
Amos Bird
1d9d586e20
Make global_context consistent.
2020-11-20 18:23:14 +08:00
Azat Khuzhin
b838214a35
Pass non-const SelectQueryInfo (and drop mutable qualifiers)
2020-10-02 22:42:35 +03:00
Vladimir Chebotarev
a38c7ff2a8
Somehow uncompressed PUT works and gzipped PUT doesn't, in S3 storage.
2020-09-30 15:04:42 +03:00
Nikolai Kochetov
2cca4d5fcf
Refactor Pipe [part 2].
2020-08-03 16:54:14 +03:00
Peng Jian
7579d65ebc
Registering StorageS3 and TableFunctionS3 under two different names (S3 and COSN).
2020-07-13 22:13:30 +08:00
Peng Jian
958bb1f3aa
COS (Tencent Cloud Object Storage) integration
...
To allow import and export to COS.
2020-07-10 15:26:55 +08:00
alesapin
71f99a274d
Compileable getSampleBlockWithColumns in StorageInMemoryMetadata
2020-06-16 17:25:08 +03:00
alesapin
36ba0192df
Metadata in read and write methods of IStorage
2020-06-15 22:08:58 +03:00
alesapin
f981649213
Fix pushing to views stream and refactor virtuals
2020-04-28 13:38:57 +03:00
alesapin
18c550df15
Better virtuals logic
2020-04-27 16:55:30 +03:00
Ivan Lezhankin
06446b4f08
dbms/ → src/
2020-04-03 18:14:31 +03:00
