thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-11 08:52:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
91,484 Commits 9,280 Branches 1,782 Tags 2.1 GiB
b2776e3dcd
Commit Graph

267 Commits

Author SHA1 Message Date
kssenii
1f6ca619b7 Allow some killing 2021-12-27 22:42:56 +03:00
avogar
58966e90e9 Don't allow to write into S3 if path contains globs 2021-12-24 17:13:35 +03:00
Anton Popov
ccd78e3838 Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-22 17:19:35 +03:00
avogar
e5661b9b62 Minor change 2021-11-11 21:09:21 +03:00
Anton Popov
a20922b2d3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-11-09 15:36:25 +03:00
avogar
2dd4393ca1 Small refactoring in formats 2021-11-03 20:07:05 +03:00
Kseniia Sumarokova
065106c299
Merge pull request #30690 from kssenii/partitioned-write 
Partitioned write for file, hdfs, url storages and table functions
 2021-11-02 22:58:16 +03:00
kssenii
20772fde04 For storage 2021-10-27 10:04:17 +03:00
kssenii
2ba3ee830e Partitioned write part 2 2021-10-26 14:00:41 +03:00
kssenii
52296e8b04 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-25 00:22:32 +03:00
kssenii
e04e37916e Review fixes 2021-10-23 18:20:31 +03:00
kssenii
36d0f67b38 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remote-connection 2021-10-23 17:12:46 +03:00
kssenii
39b9e9c258 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-22 15:40:41 +03:00
kssenii
4a0bdd296d Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-16 18:35:48 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
kssenii
7b81766168 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into remote-connection 2021-10-15 12:36:44 +00:00
kssenii
881ae8617e Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-15 15:09:56 +03:00
Nikolai Kochetov
ab28c6c855 Remove BlockInputStream interfaces. 2021-10-14 13:25:43 +03:00
Nikolai Kochetov
2957971ee3 Remove some last streams. 2021-10-13 21:22:02 +03:00
Nikolai Kochetov
ec18340351 Remove streams from formats. 2021-10-11 19:11:50 +03:00
kssenii
448d63a0fc Range read with upper bound offset 2021-10-11 01:53:51 +03:00
kssenii
5a87b6eb4f Predefined configuration for table function remote 2021-10-06 05:55:09 +00:00
Nikolai Kochetov
998d29ebc7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-23 13:10:27 +03:00
Nikolai Kochetov
341553febd Fix build. 2021-09-16 20:40:42 +03:00
kssenii
1650c92407 Better 2021-09-16 11:38:26 +03:00
kssenii
221c09589c Review fixes 2021-09-15 21:11:49 +03:00
Nikolai Kochetov
b997214620 Rename QueryPipeline to QueryPipelineBuilder. 2021-09-14 20:48:18 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
Anton Popov
4c388e3d84 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-09-09 14:10:16 +03:00
kssenii
e9721804b6 Fix tests 2021-09-09 12:34:47 +03:00
kssenii
3deb9a0ecb s3, url, http headers 2021-09-07 14:17:25 +03:00
Nikolai Kochetov
66a76ab70f Rewrite PushingToViewsBlockOutputStream part 6 2021-09-03 20:29:36 +03:00
Nikita Mikhaylov
e41d1cff1d Fix test 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
e5593f594d After rebase 2021-08-31 12:54:06 +00:00
Nikita Mikhaylov
ae241fc129 done 2021-08-31 12:54:06 +00:00
vdimir
6fe63a80bc Fix validateKey/Bucket for S3 2021-08-23 09:22:15 +03:00
vdimir
6ba6577978 Do not allow slashes in bucket formatted from PARTITION BY 2021-08-23 09:22:15 +03:00
vdimir
c4c42b61c4 Validate uft8 in partition key from PARTITION BY for s3 2021-08-23 09:22:15 +03:00
Vladimir Chebotarev
f6cb82aba5 Minor fix. 2021-08-23 09:22:15 +03:00
Vladimir Chebotarev
e203c330c2 Validation. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
6e10d28c78 Fix. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
ea67849fd9 Typo. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
119172543b Better fix. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
0b75c66378 Revert "Update src/Storages/StorageS3.cpp " 
This reverts commit bbbfe1891bfdc868878cfc8d52c254babd4ca181.
 2021-08-23 09:22:14 +03:00
Vladimir C
067cc763cc Update src/Storages/StorageS3.cpp 
StringRef in fmt::format throws `fmt::v7::format_error: cannot switch from manual to automatic argument indexing`
 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
d8bc601dbf Minor. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
4b1cc38056 Fixes. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
cebe1e7123 Style fix. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
b34f60ad46 Fixed find. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
be343766a6 Update src/Storages/StorageS3.cpp 
Co-authored-by: Vladimir C <vdimir@yandex-team.ru>
 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
24fcf152ed Minor fix. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
5ce9e266b4 Less strict validation. 2021-08-23 09:22:13 +03:00
vdimir
4c9fb7ff58 Fix error code in StorageS3.cpp 2021-08-23 09:22:13 +03:00
vdimir
eb31b194b9 Add validation to insert partition by key to s3 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
d1efe3afa7 Review fixes. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
e99433e094 Better functionality. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
328213f5d9 Fixes. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
1146efacff Partitioned writes for S3. 2021-08-23 09:22:13 +03:00
Anton Popov
61239343e3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-20 16:33:30 +03:00
Vladimir Chebotarev
c2410920d3 Safer ReadBufferFromS3 for merges and backports. 2021-08-04 09:14:20 +03:00
Anton Popov
e36736b50c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-02 22:52:02 +03:00
Nikolai Kochetov
9b5a816b43 Merge branch 'master' into output-streams-to-processors 2021-07-26 18:03:11 +03:00
Nikolai Kochetov
9c92f43359 Update storages. 2021-07-23 22:33:59 +03:00
Nikolai Kochetov
f38de35b14 Rename some constants. 2021-07-21 19:13:17 +03:00
Nikolai Kochetov
179ec05a72 Remove some streams. 2021-07-20 21:18:43 +03:00
Anton Popov
3ed7f5a6cc dynamic subcolumns: add snapshot for storage 2021-07-09 06:15:41 +03:00
Kseniia Sumarokova
55ce7de248
Remove trailing spaces -- style check fix 2021-07-02 22:39:21 +03:00
Kseniia Sumarokova
ed687f45a1
Merge branch 'master' into truncate 2021-06-30 20:08:42 +03:00
Raúl Marín
bfc122df64 Fix some typos in Storage classes 2021-06-28 19:03:56 +02:00
kssenii
ac0f86cdbf Truncate for s3 2021-06-21 15:44:36 +00:00
Nikita Mikhaylov
0055a924f6 better 2021-06-01 14:23:46 +00:00
Nikita Mikhaylov
3f18bde246 Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-s3-backoff 2021-06-01 13:39:09 +00:00
Nikita Mikhaylov
f33b70c54a better 2021-05-20 00:42:25 +03:00
kssenii
9b8df78fdd Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-17 17:42:05 +03:00
Vladimir Chebotarev
bdf03387ab Better S3 read retries. Renamed s3_max_single_read_retries -> s3_single_read_retry_attempts, added sleeps before next attempts. 2021-05-17 10:38:32 +03:00
feng lv
9035e3a643 fix test and conflict 2021-05-11 16:22:24 +00:00
kssenii
393b026f2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-11 16:49:23 +03:00
Vladimir
d02d14fa00
Merge pull request #23846 from excitoon-favorites/s3regions 2021-05-10 12:14:08 +03:00
kssenii
266567e04c Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-08 17:49:05 +03:00
kssenii
02288359c5 Less manual concatenation of paths 2021-05-08 13:59:55 +03:00
Azat Khuzhin
e398635cb3 Make StorageS3.cpp self compilable (missing AbstractConfiguration.h) 2021-05-07 10:07:44 +03:00
feng lv
39f68bf5ff fix conflict 2021-05-02 16:33:45 +00:00
Vladimir Chebotarev
a6ab040b07 Added region parameter for S3 storage and disk. 2021-05-01 20:45:35 +03:00
Azat Khuzhin
a4036a8199 Correctly stop ParallelFormattingOutputFormat from StorageS3BlockOutputStream 2021-04-25 12:49:57 +03:00
feng lv
0f5c05854c fix conflict 2021-04-23 12:22:51 +00:00
feng lv
4ffe199d39 Implement table comments 2021-04-23 12:18:23 +00:00
Nikita Mikhaylov
f55eacee7d
Merge pull request #22988 from excitoon-favorites/s3connectionresetretries 
Retries on HTTP connection drops during reads from S3
 2021-04-20 14:59:21 +03:00
Vladimir Chebotarev
9d4295f261 Removed infinite retries. 2021-04-19 11:02:36 +03:00
Vladimir Chebotarev
f59d35efd8 Rebase fix. 2021-04-15 10:53:00 +03:00
Vladimir Chebotarev
c0ea3ce08a Removed STS client. 2021-04-15 10:01:43 +03:00
Vladimir Chebotarev
3a467ffc50 Attempt to make credentials provider which uses internal credentials provider chain. 2021-04-15 10:01:13 +03:00
Vladimir Chebotarev
037a4de854 Added STS GetCallerIdentity credentials provider. 2021-04-15 10:00:18 +03:00
Vladimir Chebotarev
10bf56e540 Attempt to add retries to S3 reads. 2021-04-15 09:47:19 +03:00
Nikita Mikhaylov
2a86d76ccd style 2021-04-13 23:19:04 +03:00
Nikita Mikhaylov
ec35a878d3 simplify storages3cluster 2021-04-13 23:17:25 +03:00
Nikita Mikhaylov
024374a2ec review fixes 2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
a743442a17 build fixes 2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
507cb8514a review fixes 2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
7a68820342 style 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
7b95ff579d fix S3 test 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
4465a0627f better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
c333c3dedb review fixes 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
36a8419f60 style 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
7276b40556 better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
587fbdd10d better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
3ab17233cf cleanup 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
b3094412b1 better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
64b4cd0e63 save 2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
44ca65a9a4 save 2021-04-13 22:39:41 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
Nikita Mikhaylov
7d7a2bc724
Merge pull request #21630 from nikitamikhaylov/pf-local 
Parallel formatting for clickhouse-local
 2021-03-16 14:37:40 +03:00
Nikita Mikhaylov
1f92c8ce58 parallel formatting everywhere 2021-03-15 14:51:24 +03:00
vdimir
1194d50e48
Try to fix test_storage_s3: crash in WriteBufferFromS3 2021-03-11 15:21:13 +03:00
Pervakov Grigorii
eb87afc4d4 recreate S3 client if credentials changed 2021-03-10 13:36:57 +03:00
alexey-milovidov
a299790522
Merge pull request #19748 from excitoon-favorites/s3ssec 
Added Server Side Encryption Customer Keys support in S3 client
 2021-02-15 15:23:14 +03:00
Vladimir Chebotarev
d729aacb09 Fixed table function S3 auto compression mode. 2021-01-29 07:54:52 +03:00
Vladimir Chebotarev
579f8da573 Added SSE-C support in S3 client. 2021-01-28 09:32:41 +03:00
alexey-milovidov
7c240d0ca7
Merge pull request #17870 from excitoon-favorites/s3testiostream 
Update of AWS C++ SDK
 2021-01-28 07:12:52 +03:00
Vladimir Chebotarev
afdea5eff0 Changed handling of PocoHTTPClientConfiguration in order to ensure AWS SDK is initialized. 2021-01-27 09:38:35 +03:00
Vladimir Chebotarev
8ba8e8efd5 Added prefix-based S3 endpoint settings. 2021-01-25 06:40:57 +03:00
Vladimir Chebotarev
5cbc25c647 Added global setting s3_max_connections. 2021-01-05 13:34:55 +03:00
Nikita Mikhailov
c5f92e5096 better formatfactory 2020-12-30 06:07:30 +03:00
Nikita Mikhailov
d293ae5e8e style 2020-12-28 20:18:53 +03:00
Nikita Mikhaylov
e0addac6fc save changes 2020-12-15 00:56:46 +03:00
Pavel Kovalenko
59a5e63417 Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload 
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
#	src/Disks/S3/DiskS3.h
#	src/Disks/S3/registerDiskS3.cpp
#	src/IO/WriteBufferFromS3.cpp
#	src/IO/WriteBufferFromS3.h
 2020-12-11 18:28:41 +03:00
Vladimir Chebotarev
39d867e56d
Added proper authentication for S3 client (#16856) 2020-12-10 12:19:42 +03:00
Pavel Kovalenko
db7a87089b Adaptive choose of single/multi part upload in WriteBufferFromS3. 2020-12-09 17:09:04 +03:00
alesapin
45f09fe293
Merge pull request #17220 from ianton-ru/s3_max_redirects 
S3 max redirects
 2020-11-24 12:13:10 +03:00
Anton Ivashkin
aa208a9d9f Use only 's3_max_redirect' in params instead of all settings 2020-11-23 14:02:17 +03:00
Amos Bird
1d9d586e20
Make global_context consistent. 2020-11-20 18:23:14 +08:00
Anton Ivashkin
15b5d3a55a Merge master 2020-11-13 19:32:35 +03:00
Anton Ivashkin
026f7e0a27 Add 's3_max_redirects' setting 2020-11-13 19:31:51 +03:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
Alexander Tokmakov
5cdfcfb307 remove other stringstreams 2020-11-09 22:12:44 +03:00
alexey-milovidov
f4ba5f1f9a
Merge pull request #16772 from ClickHouse/fix-stringstream 
Fix "server failed to start" error
 2020-11-08 14:27:08 +03:00
Alexey Milovidov
5314185e25 Merge branch 'master' into azat-optimize_skip_unused_shards-optimization 2020-11-08 00:17:59 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Amos Bird
47fcd8bffb
Fix some problems introduced by refactoring. 2020-10-12 17:58:09 +08:00
Amos Bird
d2dcfc3f0d
Refactor processors. 2020-10-12 17:30:05 +08:00
Azat Khuzhin
b838214a35 Pass non-const SelectQueryInfo (and drop mutable qualifiers) 2020-10-02 22:42:35 +03:00
alesapin
17b86dec96 Recursive defaults 2020-10-02 15:38:50 +03:00
Vladimir Chebotarev
a38c7ff2a8 Somehow uncompressed PUT works and gzipped PUT doesn't, in S3 storage. 2020-09-30 15:04:42 +03:00
Vladimir Chebotarev
698768add0 Fixed format argument in S3 storage when compression method is selected. 2020-09-30 13:55:44 +03:00
alexey-milovidov
eeefcd6493
Merge pull request #13337 from ClickHouse/refactor-pipes-3 
Refactor Pipe
 2020-08-14 04:12:22 +03:00
Vladimir Chebotarev
7371cf6bd1
Proper remote host checking in S3 redirects (#13404) 
* Proper remote host checking in S3 redirects.

Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-08-11 09:09:08 +03:00
Nikolai Kochetov
2cca4d5fcf Refactor Pipe [part 2]. 2020-08-03 16:54:14 +03:00
Peng Jian
7579d65ebc Registering StorageS3 and TableFunctionS3 under two different names (S3 and COSN). 2020-07-13 22:13:30 +08:00
Peng Jian
958bb1f3aa COS (Tencent Cloud Object Storage) integration 
To allow import and export to COS.
 2020-07-10 15:26:55 +08:00
Vladimir Chebotarev
faf6be6576
Implemented single part uploads for DiskS3 (#12026) 
* Implemented single part uploads for DiskS3.
* Added `min_multi_part_upload_size` to disk configuration.
 2020-07-09 17:09:17 +03:00