thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 10:22:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
130,906 Commits 9,328 Branches 1,783 Tags 2.1 GiB
679a0e1300
Commit Graph

570 Commits

Author SHA1 Message Date
avogar
97788b9c21 Allow to create new files on insert for File/S3/HDFS engines 2021-12-29 21:19:13 +03:00
avogar
aaf9f85c67 Add more tests and fixes 2021-12-29 12:18:56 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
kssenii
1f6ca619b7 Allow some killing 2021-12-27 22:42:56 +03:00
avogar
58966e90e9 Don't allow to write into S3 if path contains globs 2021-12-24 17:13:35 +03:00
Anton Popov
ccd78e3838 Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-22 17:19:35 +03:00
avogar
e5661b9b62 Minor change 2021-11-11 21:09:21 +03:00
Anton Popov
a20922b2d3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-11-09 15:36:25 +03:00
avogar
2dd4393ca1 Small refactoring in formats 2021-11-03 20:07:05 +03:00
Kseniia Sumarokova
065106c299
Merge pull request #30690 from kssenii/partitioned-write 
Partitioned write for file, hdfs, url storages and table functions
 2021-11-02 22:58:16 +03:00
kssenii
20772fde04 For storage 2021-10-27 10:04:17 +03:00
kssenii
2ba3ee830e Partitioned write part 2 2021-10-26 14:00:41 +03:00
kssenii
52296e8b04 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-25 00:22:32 +03:00
kssenii
e04e37916e Review fixes 2021-10-23 18:20:31 +03:00
kssenii
36d0f67b38 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remote-connection 2021-10-23 17:12:46 +03:00
kssenii
39b9e9c258 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-22 15:40:41 +03:00
kssenii
4a0bdd296d Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-16 18:35:48 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
kssenii
7b81766168 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into remote-connection 2021-10-15 12:36:44 +00:00
kssenii
881ae8617e Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-15 15:09:56 +03:00
Nikolai Kochetov
ab28c6c855 Remove BlockInputStream interfaces. 2021-10-14 13:25:43 +03:00
Nikolai Kochetov
2957971ee3 Remove some last streams. 2021-10-13 21:22:02 +03:00
Nikolai Kochetov
ec18340351 Remove streams from formats. 2021-10-11 19:11:50 +03:00
kssenii
448d63a0fc Range read with upper bound offset 2021-10-11 01:53:51 +03:00
kssenii
5a87b6eb4f Predefined configuration for table function remote 2021-10-06 05:55:09 +00:00
Nikolai Kochetov
998d29ebc7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-23 13:10:27 +03:00
Nikolai Kochetov
341553febd Fix build. 2021-09-16 20:40:42 +03:00
kssenii
1650c92407 Better 2021-09-16 11:38:26 +03:00
kssenii
221c09589c Review fixes 2021-09-15 21:11:49 +03:00
Nikolai Kochetov
b997214620 Rename QueryPipeline to QueryPipelineBuilder. 2021-09-14 20:48:18 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
Anton Popov
4c388e3d84 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-09-09 14:10:16 +03:00
kssenii
e9721804b6 Fix tests 2021-09-09 12:34:47 +03:00
kssenii
3deb9a0ecb s3, url, http headers 2021-09-07 14:17:25 +03:00
Nikolai Kochetov
66a76ab70f Rewrite PushingToViewsBlockOutputStream part 6 2021-09-03 20:29:36 +03:00
Nikita Mikhaylov
e41d1cff1d Fix test 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
e5593f594d After rebase 2021-08-31 12:54:06 +00:00
Nikita Mikhaylov
ae241fc129 done 2021-08-31 12:54:06 +00:00
vdimir
6fe63a80bc Fix validateKey/Bucket for S3 2021-08-23 09:22:15 +03:00
vdimir
6ba6577978 Do not allow slashes in bucket formatted from PARTITION BY 2021-08-23 09:22:15 +03:00
vdimir
c4c42b61c4 Validate uft8 in partition key from PARTITION BY for s3 2021-08-23 09:22:15 +03:00
Vladimir Chebotarev
f6cb82aba5 Minor fix. 2021-08-23 09:22:15 +03:00
Vladimir Chebotarev
e203c330c2 Validation. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
6e10d28c78 Fix. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
ea67849fd9 Typo. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
119172543b Better fix. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
0b75c66378 Revert "Update src/Storages/StorageS3.cpp " 
This reverts commit bbbfe1891bfdc868878cfc8d52c254babd4ca181.
 2021-08-23 09:22:14 +03:00
Vladimir C
067cc763cc Update src/Storages/StorageS3.cpp 
StringRef in fmt::format throws `fmt::v7::format_error: cannot switch from manual to automatic argument indexing`
 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
d8bc601dbf Minor. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
4b1cc38056 Fixes. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
cebe1e7123 Style fix. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
b34f60ad46 Fixed find. 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
be343766a6 Update src/Storages/StorageS3.cpp 
Co-authored-by: Vladimir C <vdimir@yandex-team.ru>
 2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
24fcf152ed Minor fix. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
5ce9e266b4 Less strict validation. 2021-08-23 09:22:13 +03:00
vdimir
4c9fb7ff58 Fix error code in StorageS3.cpp 2021-08-23 09:22:13 +03:00
vdimir
eb31b194b9 Add validation to insert partition by key to s3 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
d1efe3afa7 Review fixes. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
e99433e094 Better functionality. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
328213f5d9 Fixes. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
1146efacff Partitioned writes for S3. 2021-08-23 09:22:13 +03:00
Anton Popov
61239343e3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-20 16:33:30 +03:00
Vladimir Chebotarev
c2410920d3 Safer ReadBufferFromS3 for merges and backports. 2021-08-04 09:14:20 +03:00
Anton Popov
e36736b50c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-02 22:52:02 +03:00
Nikolai Kochetov
9b5a816b43 Merge branch 'master' into output-streams-to-processors 2021-07-26 18:03:11 +03:00
Nikolai Kochetov
9c92f43359 Update storages. 2021-07-23 22:33:59 +03:00
Nikolai Kochetov
f38de35b14 Rename some constants. 2021-07-21 19:13:17 +03:00
Nikolai Kochetov
179ec05a72 Remove some streams. 2021-07-20 21:18:43 +03:00
Anton Popov
3ed7f5a6cc dynamic subcolumns: add snapshot for storage 2021-07-09 06:15:41 +03:00
Kseniia Sumarokova
55ce7de248
Remove trailing spaces -- style check fix 2021-07-02 22:39:21 +03:00
Kseniia Sumarokova
ed687f45a1
Merge branch 'master' into truncate 2021-06-30 20:08:42 +03:00
Raúl Marín
bfc122df64 Fix some typos in Storage classes 2021-06-28 19:03:56 +02:00
kssenii
ac0f86cdbf Truncate for s3 2021-06-21 15:44:36 +00:00
Nikita Mikhaylov
0055a924f6 better 2021-06-01 14:23:46 +00:00
Nikita Mikhaylov
3f18bde246 Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-s3-backoff 2021-06-01 13:39:09 +00:00
Nikita Mikhaylov
f33b70c54a better 2021-05-20 00:42:25 +03:00
kssenii
9b8df78fdd Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-17 17:42:05 +03:00
Vladimir Chebotarev
bdf03387ab Better S3 read retries. Renamed s3_max_single_read_retries -> s3_single_read_retry_attempts, added sleeps before next attempts. 2021-05-17 10:38:32 +03:00
feng lv
9035e3a643 fix test and conflict 2021-05-11 16:22:24 +00:00
kssenii
393b026f2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-11 16:49:23 +03:00
Vladimir
d02d14fa00
Merge pull request #23846 from excitoon-favorites/s3regions 2021-05-10 12:14:08 +03:00
kssenii
266567e04c Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-08 17:49:05 +03:00
kssenii
02288359c5 Less manual concatenation of paths 2021-05-08 13:59:55 +03:00
Azat Khuzhin
e398635cb3 Make StorageS3.cpp self compilable (missing AbstractConfiguration.h) 2021-05-07 10:07:44 +03:00
feng lv
39f68bf5ff fix conflict 2021-05-02 16:33:45 +00:00
Vladimir Chebotarev
a6ab040b07 Added region parameter for S3 storage and disk. 2021-05-01 20:45:35 +03:00
Azat Khuzhin
a4036a8199 Correctly stop ParallelFormattingOutputFormat from StorageS3BlockOutputStream 2021-04-25 12:49:57 +03:00
feng lv
0f5c05854c fix conflict 2021-04-23 12:22:51 +00:00
feng lv
4ffe199d39 Implement table comments 2021-04-23 12:18:23 +00:00
Nikita Mikhaylov
f55eacee7d
Merge pull request #22988 from excitoon-favorites/s3connectionresetretries 
Retries on HTTP connection drops during reads from S3
 2021-04-20 14:59:21 +03:00
Vladimir Chebotarev
9d4295f261 Removed infinite retries. 2021-04-19 11:02:36 +03:00
Vladimir Chebotarev
f59d35efd8 Rebase fix. 2021-04-15 10:53:00 +03:00
Vladimir Chebotarev
c0ea3ce08a Removed STS client. 2021-04-15 10:01:43 +03:00
Vladimir Chebotarev
3a467ffc50 Attempt to make credentials provider which uses internal credentials provider chain. 2021-04-15 10:01:13 +03:00
Vladimir Chebotarev
037a4de854 Added STS GetCallerIdentity credentials provider. 2021-04-15 10:00:18 +03:00
Vladimir Chebotarev
10bf56e540 Attempt to add retries to S3 reads. 2021-04-15 09:47:19 +03:00
Nikita Mikhaylov
2a86d76ccd style 2021-04-13 23:19:04 +03:00
Nikita Mikhaylov
ec35a878d3 simplify storages3cluster 2021-04-13 23:17:25 +03:00
Nikita Mikhaylov
024374a2ec review fixes 2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
a743442a17 build fixes 2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
507cb8514a review fixes 2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
7a68820342 style 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
7b95ff579d fix S3 test 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
4465a0627f better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
c333c3dedb review fixes 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
36a8419f60 style 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
7276b40556 better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
587fbdd10d better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
3ab17233cf cleanup 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
b3094412b1 better 2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
64b4cd0e63 save 2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
44ca65a9a4 save 2021-04-13 22:39:41 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
Nikita Mikhaylov
7d7a2bc724
Merge pull request #21630 from nikitamikhaylov/pf-local 
Parallel formatting for clickhouse-local
 2021-03-16 14:37:40 +03:00
Nikita Mikhaylov
1f92c8ce58 parallel formatting everywhere 2021-03-15 14:51:24 +03:00
vdimir
1194d50e48
Try to fix test_storage_s3: crash in WriteBufferFromS3 2021-03-11 15:21:13 +03:00
Pervakov Grigorii
eb87afc4d4 recreate S3 client if credentials changed 2021-03-10 13:36:57 +03:00
alexey-milovidov
a299790522
Merge pull request #19748 from excitoon-favorites/s3ssec 
Added Server Side Encryption Customer Keys support in S3 client
 2021-02-15 15:23:14 +03:00
Vladimir Chebotarev
d729aacb09 Fixed table function S3 auto compression mode. 2021-01-29 07:54:52 +03:00
Vladimir Chebotarev
579f8da573 Added SSE-C support in S3 client. 2021-01-28 09:32:41 +03:00
alexey-milovidov
7c240d0ca7
Merge pull request #17870 from excitoon-favorites/s3testiostream 
Update of AWS C++ SDK
 2021-01-28 07:12:52 +03:00
Vladimir Chebotarev
afdea5eff0 Changed handling of PocoHTTPClientConfiguration in order to ensure AWS SDK is initialized. 2021-01-27 09:38:35 +03:00
Vladimir Chebotarev
8ba8e8efd5 Added prefix-based S3 endpoint settings. 2021-01-25 06:40:57 +03:00
Vladimir Chebotarev
5cbc25c647 Added global setting s3_max_connections. 2021-01-05 13:34:55 +03:00
Nikita Mikhailov
c5f92e5096 better formatfactory 2020-12-30 06:07:30 +03:00
Nikita Mikhailov
d293ae5e8e style 2020-12-28 20:18:53 +03:00
Nikita Mikhaylov
e0addac6fc save changes 2020-12-15 00:56:46 +03:00
Pavel Kovalenko
59a5e63417 Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload 
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
#	src/Disks/S3/DiskS3.h
#	src/Disks/S3/registerDiskS3.cpp
#	src/IO/WriteBufferFromS3.cpp
#	src/IO/WriteBufferFromS3.h
 2020-12-11 18:28:41 +03:00
Vladimir Chebotarev
39d867e56d
Added proper authentication for S3 client (#16856) 2020-12-10 12:19:42 +03:00
Pavel Kovalenko
db7a87089b Adaptive choose of single/multi part upload in WriteBufferFromS3. 2020-12-09 17:09:04 +03:00
alesapin
45f09fe293
Merge pull request #17220 from ianton-ru/s3_max_redirects 
S3 max redirects
 2020-11-24 12:13:10 +03:00
Anton Ivashkin
aa208a9d9f Use only 's3_max_redirect' in params instead of all settings 2020-11-23 14:02:17 +03:00
Amos Bird
1d9d586e20
Make global_context consistent. 2020-11-20 18:23:14 +08:00
Anton Ivashkin
15b5d3a55a Merge master 2020-11-13 19:32:35 +03:00
Anton Ivashkin
026f7e0a27 Add 's3_max_redirects' setting 2020-11-13 19:31:51 +03:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
Alexander Tokmakov
5cdfcfb307 remove other stringstreams 2020-11-09 22:12:44 +03:00
alexey-milovidov
f4ba5f1f9a
Merge pull request #16772 from ClickHouse/fix-stringstream 
Fix "server failed to start" error
 2020-11-08 14:27:08 +03:00
Alexey Milovidov
5314185e25 Merge branch 'master' into azat-optimize_skip_unused_shards-optimization 2020-11-08 00:17:59 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Amos Bird
47fcd8bffb
Fix some problems introduced by refactoring. 2020-10-12 17:58:09 +08:00
Amos Bird
d2dcfc3f0d
Refactor processors. 2020-10-12 17:30:05 +08:00
Azat Khuzhin
b838214a35 Pass non-const SelectQueryInfo (and drop mutable qualifiers) 2020-10-02 22:42:35 +03:00
alesapin
17b86dec96 Recursive defaults 2020-10-02 15:38:50 +03:00
Vladimir Chebotarev
a38c7ff2a8 Somehow uncompressed PUT works and gzipped PUT doesn't, in S3 storage. 2020-09-30 15:04:42 +03:00
Vladimir Chebotarev
698768add0 Fixed format argument in S3 storage when compression method is selected. 2020-09-30 13:55:44 +03:00
alexey-milovidov
eeefcd6493
Merge pull request #13337 from ClickHouse/refactor-pipes-3 
Refactor Pipe
 2020-08-14 04:12:22 +03:00
Vladimir Chebotarev
7371cf6bd1
Proper remote host checking in S3 redirects (#13404) 
* Proper remote host checking in S3 redirects.

Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-08-11 09:09:08 +03:00
Nikolai Kochetov
2cca4d5fcf Refactor Pipe [part 2]. 2020-08-03 16:54:14 +03:00
Peng Jian
7579d65ebc Registering StorageS3 and TableFunctionS3 under two different names (S3 and COSN). 2020-07-13 22:13:30 +08:00
Peng Jian
958bb1f3aa COS (Tencent Cloud Object Storage) integration 
To allow import and export to COS.
 2020-07-10 15:26:55 +08:00
Vladimir Chebotarev
faf6be6576
Implemented single part uploads for DiskS3 (#12026) 
* Implemented single part uploads for DiskS3.
* Added `min_multi_part_upload_size` to disk configuration.
 2020-07-09 17:09:17 +03:00
alesapin
c9fa5d2ec3 Better naming 2020-06-19 18:39:41 +03:00
alesapin
71f99a274d Compileable getSampleBlockWithColumns in StorageInMemoryMetadata 2020-06-16 17:25:08 +03:00
alesapin
36ba0192df Metadata in read and write methods of IStorage 2020-06-15 22:08:58 +03:00
alesapin
af2fe2ba55 Compilable setColumns, setConstraints, setIndices 2020-06-15 19:55:33 +03:00
alexey-milovidov
e3c528e661
Merge pull request #11174 from excitoon-favorites/s3tests 
Added tests for improved S3 URL parsing
 2020-06-07 02:44:40 +03:00
Vladimir Chebotarev
d62e85f875 Improved logging of globs in S3. 2020-06-04 16:12:08 +03:00
Pervakov Grigorii
bc9381406f Add default credentials and custom headers for s3 table functions. 2020-06-02 20:48:49 +03:00
alexey-milovidov
256f84ddc2
Merge pull request #11230 from Jokser/s3-poco-http-client 
S3 Poco Http Client
 2020-06-01 01:20:21 +03:00
Pavel Kovalenko
677cb56a06 S3 Poco HTTP Client WIP. 2020-05-27 22:41:49 +03:00
Vladimir Chebotarev
7f9a257c31 Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. 2020-05-25 15:57:32 +03:00
Vladimir Chebotarev
98f31374d5 Added improved S3 URL parsing to DiskS3. 2020-05-22 11:18:34 +03:00
Vladimir Chebotarev
02cb08ea2f Fixed parsing of S3 URL. 2020-05-22 11:17:10 +03:00
alesapin
f981649213 Fix pushing to views stream and refactor virtuals 2020-04-28 13:38:57 +03:00
alesapin
4badd0fd28 Better code 2020-04-27 20:46:51 +03:00
alesapin
18c550df15 Better virtuals logic 2020-04-27 16:55:30 +03:00
Vitaly Baranov
e573549945 Rework access rights for table functions. 2020-04-07 23:31:59 +03:00
Ivan Lezhankin
06446b4f08 dbms/ → src/ 2020-04-03 18:14:31 +03:00