319 Commits

Author	SHA1	Message	Date
Sergei Trifonov	755d5c55f9	resolve conflict	2022-04-07 11:57:45 +02:00
Sergei Trifonov	9c2804b924	merge from master + resolve conflicts	2022-04-07 11:22:01 +02:00
kssenii	cbd7f4acfb	Fix	2022-04-05 19:24:10 +02:00
kssenii	4e50da722b	Better s3 settings	2022-04-04 16:14:56 +02:00
Alexey Milovidov	5c6b84c398	Merge pull request #35586 from kssenii/fix-virtual-columns-s3 ... Fix s3 engine getting virtual columns	2022-04-01 04:43:32 +03:00
kssenii	099fa0e668	Merge master	2022-03-30 17:33:32 +02:00
Antonio Andelic	a3ad99be03	Create CallbackRunner in Storage	2022-03-30 11:24:50 +00:00
Antonio Andelic	1173eab15e	Use same IOThreadPool for URL and S3	2022-03-30 08:15:20 +00:00
kssenii	4374415bd7	Fix clang-tidy, style check	2022-03-29 14:20:21 +02:00
kssenii	a2013ebe2b	Better	2022-03-28 22:40:27 +02:00
Antonio Andelic	d9d826c813	Address PR review	2022-03-28 08:19:23 +00:00
kssenii	c6706b8bfa	Better	2022-03-25 19:30:13 +01:00
Antonio Andelic	394682b983	Merge branch 'master' into parallel-s3-downloading	2022-03-25 13:00:44 +00:00
kssenii	e91e30aaa4	Fix	2022-03-24 17:10:27 +01:00
Antonio Andelic	62a495a5fc	Small refactor	2022-03-24 09:45:30 +00:00
Sergei Trifonov	bf9c6be6c6	move threadPoolCallbackRunner from Common into Interpreters folder to fix split build issue	2022-03-23 17:57:19 +01:00
Antonio Andelic	1a5f5c32bb	Remove commented out code	2022-03-23 14:36:25 +00:00
Antonio Andelic	93ad209910	Attach memory tracker	2022-03-23 11:52:31 +00:00
Antonio Andelic	131b3a091c	Refactor StorageS3	2022-03-23 08:57:01 +00:00
Antonio Andelic	6785ad165a	Fix issue for mutliple download threads	2022-03-22 14:06:30 +00:00
Antonio Andelic	04e6e6ab5e	Add ParallelReadBuffer for S3	2022-03-21 14:52:26 +00:00
mergify[bot]	7ac606fa65	Merge branch 'master' into parallel-multipart-upload-for-s3storage	2022-03-20 18:25:28 +00:00
Sergei Trifonov	bbe657ba24	allow parallelization of multipart S3 storage upload fixes #34350	2022-03-16 19:41:41 +01:00
Anton Popov	0ba78c3c3a	Merge remote-tracking branch 'upstream/master' into HEAD	2022-03-16 15:28:09 +00:00
Saad Ur Rahman	679abaafdc	[Storages] bugfix ... Extracting Version Id for S3::URI.	2022-03-14 22:25:40 -04:00
Saad Ur Rahman	fb450b7068	[Storages] bugfix ... Corrected order of parameters by swapping bucket and versionId.	2022-03-14 22:20:02 -04:00
Saad Ur Rahman	171260f40b	[Storages] Added Version ID parameter for S3.	2022-03-14 22:03:58 -04:00
Maksim Kita	2fdcf53a76	Fix clang-tidy warnings in Server, Storages folders	2022-03-14 18:17:35 +00:00
Anton Popov	df3b07fe7c	Merge remote-tracking branch 'upstream/master' into HEAD	2022-03-03 22:25:28 +00:00
Anton Popov	fcdebea925	Merge remote-tracking branch 'upstream/master' into HEAD	2022-02-25 13:41:30 +03:00
kssenii	cfad79bf74	Remove redundant	2022-02-25 09:37:48 +01:00
kssenii	003b807b00	Fix	2022-02-23 20:33:05 +01:00
Kruglov Pavel	dd863ca2a0	Merge branch 'master' into fix-url-globs	2022-02-16 12:45:31 +03:00
Dmitry Novik	50131fd732	Fix cancelation for S3 and HDFS	2022-02-11 18:14:55 +00:00
Anton Popov	18940b8637	Merge remote-tracking branch 'upstream/master' into HEAD	2022-02-09 23:38:38 +03:00
avogar	84087c1554	Improve schema inference with globs	2022-02-09 19:16:15 +03:00
alesapin	f6e7cea1f8	Add settings for S3 multipart upload parts size	2022-02-08 19:38:04 +03:00
feng lv	6325d4d9b0	continue of #34317 ... fix fix	2022-02-06 08:59:17 +00:00
Anton Popov	836a348a9c	Merge remote-tracking branch 'upstream/master' into HEAD	2022-02-01 15:23:07 +03:00
alesapin	5230c59f05	Smaller scope	2022-01-31 14:03:06 +03:00
alesapin	1adc7538fe	Fix benign race condition for storage HDFS, S3, URL	2022-01-31 13:50:28 +03:00
Anton Popov	78b9f15abb	Merge remote-tracking branch 'upstream/master' into HEAD	2022-01-30 03:24:37 +03:00
Anton Popov	e8ce091e68	Merge remote-tracking branch 'upstream/master' into HEAD	2022-01-21 20:11:18 +03:00
avogar	4efadfad3c	Fix tests	2022-01-18 22:26:13 +03:00
avogar	af54ca4866	Fix partition write	2022-01-14 21:21:36 +03:00
Kruglov Pavel	a7df9cd53a	Merge branch 'master' into formats-with-suffixes	2022-01-14 21:03:49 +03:00
avogar	89a181bd19	Make better	2022-01-14 18:16:18 +03:00
avogar	8884c4a3fe	Fix tests and build	2022-01-14 16:27:57 +03:00
avogar	2d7b1bfa5e	Detect format in S3/HDFS/URL table engines	2022-01-13 16:14:18 +03:00
avogar	97788b9c21	Allow to create new files on insert for File/S3/HDFS engines	2021-12-29 21:19:13 +03:00
avogar	aaf9f85c67	Add more tests and fixes	2021-12-29 12:18:56 +03:00
avogar	8112a71233	Implement schema inference for most input formats	2021-12-29 12:18:56 +03:00
kssenii	1f6ca619b7	Allow some killing	2021-12-27 22:42:56 +03:00
avogar	58966e90e9	Don't allow to write into S3 if path contains globs	2021-12-24 17:13:35 +03:00
Anton Popov	ccd78e3838	Merge remote-tracking branch 'upstream/master' into HEAD	2021-11-22 17:19:35 +03:00
avogar	e5661b9b62	Minor change	2021-11-11 21:09:21 +03:00
Anton Popov	a20922b2d3	Merge remote-tracking branch 'origin/sparse-serialization' into HEAD	2021-11-09 15:36:25 +03:00
avogar	2dd4393ca1	Small refactoring in formats	2021-11-03 20:07:05 +03:00
Kseniia Sumarokova	065106c299	Merge pull request #30690 from kssenii/partitioned-write ... Partitioned write for file, hdfs, url storages and table functions	2021-11-02 22:58:16 +03:00
kssenii	20772fde04	For storage	2021-10-27 10:04:17 +03:00
kssenii	2ba3ee830e	Partitioned write part 2	2021-10-26 14:00:41 +03:00
kssenii	52296e8b04	Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read	2021-10-25 00:22:32 +03:00
kssenii	e04e37916e	Review fixes	2021-10-23 18:20:31 +03:00
kssenii	36d0f67b38	Merge branch 'master' of github.com:ClickHouse/ClickHouse into remote-connection	2021-10-23 17:12:46 +03:00
kssenii	39b9e9c258	Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read	2021-10-22 15:40:41 +03:00
kssenii	4a0bdd296d	Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read	2021-10-16 18:35:48 +03:00
Nikolai Kochetov	a08c98d760	Move some files.	2021-10-16 17:03:50 +03:00
Nikolai Kochetov	fd14faeae2	Remove DataStreams folder.	2021-10-15 23:18:20 +03:00
kssenii	7b81766168	Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into remote-connection	2021-10-15 12:36:44 +00:00
kssenii	881ae8617e	Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read	2021-10-15 15:09:56 +03:00
Nikolai Kochetov	ab28c6c855	Remove BlockInputStream interfaces.	2021-10-14 13:25:43 +03:00
Nikolai Kochetov	2957971ee3	Remove some last streams.	2021-10-13 21:22:02 +03:00
Nikolai Kochetov	ec18340351	Remove streams from formats.	2021-10-11 19:11:50 +03:00
kssenii	448d63a0fc	Range read with upper bound offset	2021-10-11 01:53:51 +03:00
kssenii	5a87b6eb4f	Predefined configuration for table function remote	2021-10-06 05:55:09 +00:00
Nikolai Kochetov	998d29ebc7	Merge branch 'master' into rewrite-pushing-to-views	2021-09-23 13:10:27 +03:00
Nikolai Kochetov	341553febd	Fix build.	2021-09-16 20:40:42 +03:00
kssenii	1650c92407	Better	2021-09-16 11:38:26 +03:00
kssenii	221c09589c	Review fixes	2021-09-15 21:11:49 +03:00
Nikolai Kochetov	b997214620	Rename QueryPipeline to QueryPipelineBuilder.	2021-09-14 20:48:18 +03:00
Nikolai Kochetov	f569a3e3f7	Merge branch 'master' into rewrite-pushing-to-views	2021-09-09 20:30:23 +03:00
Anton Popov	4c388e3d84	Merge remote-tracking branch 'origin/sparse-serialization' into HEAD	2021-09-09 14:10:16 +03:00
kssenii	e9721804b6	Fix tests	2021-09-09 12:34:47 +03:00
kssenii	3deb9a0ecb	s3, url, http headers	2021-09-07 14:17:25 +03:00
Nikolai Kochetov	66a76ab70f	Rewrite PushingToViewsBlockOutputStream part 6	2021-09-03 20:29:36 +03:00
Nikita Mikhaylov	e41d1cff1d	Fix test	2021-08-31 12:54:45 +00:00
Nikita Mikhaylov	e5593f594d	After rebase	2021-08-31 12:54:06 +00:00
Nikita Mikhaylov	ae241fc129	done	2021-08-31 12:54:06 +00:00
vdimir	6fe63a80bc	Fix validateKey/Bucket for S3	2021-08-23 09:22:15 +03:00
vdimir	6ba6577978	Do not allow slashes in bucket formatted from PARTITION BY	2021-08-23 09:22:15 +03:00
vdimir	c4c42b61c4	Validate uft8 in partition key from PARTITION BY for s3	2021-08-23 09:22:15 +03:00
Vladimir Chebotarev	f6cb82aba5	Minor fix.	2021-08-23 09:22:15 +03:00
Vladimir Chebotarev	e203c330c2	Validation.	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	6e10d28c78	Fix.	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	ea67849fd9	Typo.	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	119172543b	Better fix.	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	0b75c66378	Revert "Update src/Storages/StorageS3.cpp " ... This reverts commit bbbfe1891bfdc868878cfc8d52c254babd4ca181.	2021-08-23 09:22:14 +03:00
Vladimir C	067cc763cc	Update src/Storages/StorageS3.cpp ... StringRef in fmt::format throws `fmt::v7::format_error: cannot switch from manual to automatic argument indexing`	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	d8bc601dbf	Minor.	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	4b1cc38056	Fixes.	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	cebe1e7123	Style fix.	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	b34f60ad46	Fixed find.	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	be343766a6	Update src/Storages/StorageS3.cpp ... Co-authored-by: Vladimir C <vdimir@yandex-team.ru>	2021-08-23 09:22:14 +03:00
Vladimir Chebotarev	24fcf152ed	Minor fix.	2021-08-23 09:22:13 +03:00
Vladimir Chebotarev	5ce9e266b4	Less strict validation.	2021-08-23 09:22:13 +03:00
vdimir	4c9fb7ff58	Fix error code in StorageS3.cpp	2021-08-23 09:22:13 +03:00
vdimir	eb31b194b9	Add validation to insert partition by key to s3	2021-08-23 09:22:13 +03:00
Vladimir Chebotarev	d1efe3afa7	Review fixes.	2021-08-23 09:22:13 +03:00
Vladimir Chebotarev	e99433e094	Better functionality.	2021-08-23 09:22:13 +03:00
Vladimir Chebotarev	328213f5d9	Fixes.	2021-08-23 09:22:13 +03:00
Vladimir Chebotarev	1146efacff	Partitioned writes for S3.	2021-08-23 09:22:13 +03:00
Anton Popov	61239343e3	Merge remote-tracking branch 'origin/sparse-serialization' into HEAD	2021-08-20 16:33:30 +03:00
Vladimir Chebotarev	c2410920d3	Safer ReadBufferFromS3 for merges and backports.	2021-08-04 09:14:20 +03:00
Anton Popov	e36736b50c	Merge remote-tracking branch 'origin/sparse-serialization' into HEAD	2021-08-02 22:52:02 +03:00
Nikolai Kochetov	9b5a816b43	Merge branch 'master' into output-streams-to-processors	2021-07-26 18:03:11 +03:00
Nikolai Kochetov	9c92f43359	Update storages.	2021-07-23 22:33:59 +03:00
Nikolai Kochetov	f38de35b14	Rename some constants.	2021-07-21 19:13:17 +03:00
Nikolai Kochetov	179ec05a72	Remove some streams.	2021-07-20 21:18:43 +03:00
Anton Popov	3ed7f5a6cc	dynamic subcolumns: add snapshot for storage	2021-07-09 06:15:41 +03:00
Kseniia Sumarokova	55ce7de248	Remove trailing spaces -- style check fix	2021-07-02 22:39:21 +03:00
Kseniia Sumarokova	ed687f45a1	Merge branch 'master' into truncate	2021-06-30 20:08:42 +03:00
Raúl Marín	bfc122df64	Fix some typos in Storage classes	2021-06-28 19:03:56 +02:00
kssenii	ac0f86cdbf	Truncate for s3	2021-06-21 15:44:36 +00:00
Nikita Mikhaylov	0055a924f6	better	2021-06-01 14:23:46 +00:00
Nikita Mikhaylov	3f18bde246	Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-s3-backoff	2021-06-01 13:39:09 +00:00
Nikita Mikhaylov	f33b70c54a	better	2021-05-20 00:42:25 +03:00
kssenii	9b8df78fdd	Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs	2021-05-17 17:42:05 +03:00
Vladimir Chebotarev	bdf03387ab	Better S3 read retries. Renamed s3_max_single_read_retries -> s3_single_read_retry_attempts, added sleeps before next attempts.	2021-05-17 10:38:32 +03:00
feng lv	9035e3a643	fix test and conflict	2021-05-11 16:22:24 +00:00
kssenii	393b026f2e	Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs	2021-05-11 16:49:23 +03:00
Vladimir	d02d14fa00	Merge pull request #23846 from excitoon-favorites/s3regions	2021-05-10 12:14:08 +03:00
kssenii	266567e04c	Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs	2021-05-08 17:49:05 +03:00
kssenii	02288359c5	Less manual concatenation of paths	2021-05-08 13:59:55 +03:00
Azat Khuzhin	e398635cb3	Make StorageS3.cpp self compilable (missing AbstractConfiguration.h)	2021-05-07 10:07:44 +03:00
feng lv	39f68bf5ff	fix conflict	2021-05-02 16:33:45 +00:00
Vladimir Chebotarev	a6ab040b07	Added region parameter for S3 storage and disk.	2021-05-01 20:45:35 +03:00
Azat Khuzhin	a4036a8199	Correctly stop ParallelFormattingOutputFormat from StorageS3BlockOutputStream	2021-04-25 12:49:57 +03:00
feng lv	0f5c05854c	fix conflict	2021-04-23 12:22:51 +00:00
feng lv	4ffe199d39	Implement table comments	2021-04-23 12:18:23 +00:00
Nikita Mikhaylov	f55eacee7d	Merge pull request #22988 from excitoon-favorites/s3connectionresetretries ... Retries on HTTP connection drops during reads from S3	2021-04-20 14:59:21 +03:00
Vladimir Chebotarev	9d4295f261	Removed infinite retries.	2021-04-19 11:02:36 +03:00
Vladimir Chebotarev	f59d35efd8	Rebase fix.	2021-04-15 10:53:00 +03:00
Vladimir Chebotarev	c0ea3ce08a	Removed STS client.	2021-04-15 10:01:43 +03:00
Vladimir Chebotarev	3a467ffc50	Attempt to make credentials provider which uses internal credentials provider chain.	2021-04-15 10:01:13 +03:00
Vladimir Chebotarev	037a4de854	Added STS GetCallerIdentity credentials provider.	2021-04-15 10:00:18 +03:00
Vladimir Chebotarev	10bf56e540	Attempt to add retries to S3 reads.	2021-04-15 09:47:19 +03:00
Nikita Mikhaylov	2a86d76ccd	style	2021-04-13 23:19:04 +03:00
Nikita Mikhaylov	ec35a878d3	simplify storages3cluster	2021-04-13 23:17:25 +03:00
Nikita Mikhaylov	024374a2ec	review fixes	2021-04-13 22:39:43 +03:00
Nikita Mikhaylov	a743442a17	build fixes	2021-04-13 22:39:43 +03:00