thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-11 08:52:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
135,560 Commits 9,280 Branches 1,782 Tags 2.1 GiB
1a1e78cf63
Commit Graph

600 Commits

Author SHA1 Message Date
avogar
7f9e81d504 Clean up 2023-08-22 12:55:00 +00:00
Kruglov Pavel
c0bdd0e00b
Merge branch 'master' into cache-count 2023-08-22 14:42:22 +02:00
avogar
b4145aeddc Cache number of rows in files for count in file/s3/url/hdfs/azure functions 2023-08-22 11:59:59 +00:00
pufit
9d454d9afc Merge branch 'master' into pufit/fix_s3_threads 
# Conflicts:
#	src/Storages/StorageS3.cpp
#	src/Storages/StorageS3.h
#	src/Storages/StorageURL.cpp
#	src/Storages/StorageURL.h
 2023-08-21 21:32:15 -04:00
pufit
98a701e2c1 Limiting number of parsing threads for S3 source 2023-08-21 21:21:03 -04:00
Michael Kolupaev
2f4d433e69 Parquet filter pushdown 2023-08-21 14:15:52 -07:00
avogar
47304bf7aa Optimize count from files in most input formats 2023-08-21 12:30:52 +00:00
avogar
8e445b5461 Fixes 2023-08-18 17:49:40 +00:00
avogar
4c32097df3 Use filter by file/path before reading in url/file/hdfs table functions, reduce code duplication 2023-08-17 16:54:43 +00:00
Anton Popov
ff137773e7
Merge branch 'master' into formats-with-subcolumns 2023-08-02 15:24:56 +02:00
Kseniia Sumarokova
8893670d98
Merge branch 'master' into s3queue 2023-08-01 18:42:32 +02:00
kssenii
870a506a0b Some fixes 2023-07-31 20:07:23 +02:00
Nikolai Kochetov
c4a00b8c68 Throw S3Exception whenever possible. 2023-07-31 15:39:20 +00:00
Kruglov Pavel
0d34e97dbe
Merge branch 'master' into formats-with-subcolumns 2023-07-26 13:30:35 +02:00
Kruglov Pavel
fec5675cd4
Merge branch 'master' into better-progress-bar-2 2023-07-24 19:59:38 +02:00
avogar
3682254342 Fix hdfs tests 2023-07-18 13:49:23 +00:00
Kruglov Pavel
64e88cde21
Merge branch 'master' into better-progress-bar-2 2023-07-18 13:37:53 +02:00
Kruglov Pavel
1dd05319b5
Merge branch 'master' into formats-with-subcolumns 2023-07-17 19:13:42 +02:00
Nikita Mikhaylov
ee0bbc0e54
Merge branch 'master' into headers-blacklist 2023-07-17 19:08:52 +02:00
avogar
d11cd0dc30 Fix tests 2023-07-05 17:56:03 +00:00
avogar
98aa6b317f Support reading subcolumns from file/s3/hdfs/url/azureBlobStorage table functions 2023-07-04 21:17:26 +00:00
Nikolay Degterinsky
8dfa773f44
Merge branch 'master' into headers-blacklist 2023-06-30 23:40:17 +02:00
avogar
4eeb431003 Merge branch 'master' of github.com:ClickHouse/ClickHouse into better-progress-bar-2 2023-06-28 18:53:08 +00:00
Alexey Milovidov
cca3f9937c
Merge branch 'master' into retry 2023-06-24 22:29:17 +03:00
avogar
c679dd400e Make better 2023-06-23 13:43:40 +00:00
avogar
24fab7bfde Remove old includes 2023-06-22 18:48:15 +00:00
avogar
cf082f2f9a Use read_bytes/total_bytes_to_read for progress bar in s3/file/url/... table functions 2023-06-22 17:24:43 +00:00
Sema Checherinda
95349a405b release buffers with exception context 2023-06-22 13:00:13 +02:00
Sema Checherinda
c2fa7a1030
Merge branch 'master' into memory-leak 2023-06-21 21:15:01 +02:00
Michael Kolupaev
4a570a05c9 Decrease default timeouts for S3 and HTTP requests 2023-06-21 18:08:50 +00:00
avogar
3209ebe34b Improve progress bar for file/s3/hdfs/url table functions. Step 1 2023-06-16 15:51:18 +00:00
Kruglov Pavel
f8ddfb1fd8
Merge branch 'master' into allow-skip-empty-files 2023-06-16 13:23:41 +02:00
Nikolay Degterinsky
9a25958be8 Add HTTP header filtering 2023-06-15 13:49:49 +00:00
avogar
870f3d1270 Fix comments 2023-06-15 12:59:46 +00:00
Kseniia Sumarokova
c786fbf8bd
Add comment 2023-06-15 10:22:02 +02:00
kssenii
827ac17dc2 Fix 2023-06-14 12:59:06 +02:00
avogar
2e1f56ae33 Address comments 2023-06-13 14:43:50 +00:00
Sema Checherinda
2c018f5261 do not call finalize after exception 2023-06-13 10:48:31 +02:00
Kruglov Pavel
e4838725e3
Merge branch 'master' into allow-skip-empty-files 2023-06-12 20:03:23 +02:00
kssenii
1cbcd2f2ef Refactor reading from object storages 2023-06-08 14:33:59 +02:00
Kruglov Pavel
bf28074d32
Merge branch 'master' into allow-skip-empty-files 2023-06-08 12:36:18 +02:00
Antonio Andelic
b11f744252
Correctly disable async insert with deduplication when it's not needed (#50663) 
* Correctly disable async insert when it's not used

* Better

* Add comment

* Better

* Fix tests

---------

Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-06-07 20:33:08 +02:00
Nikita Mikhaylov
e87348010d
Rework loading and removing of data parts for MergeTree tables. (#49474) 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-06-06 14:42:56 +02:00
Michael Kolupaev
b51064a508 Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead 2023-06-01 18:48:30 -07:00
Alexey Milovidov
52ffaa4630
Merge pull request #50203 from Avogar/head-requests-on-shcema-inference 
Don't send head request for all keys in Iceberg schema inference
 2023-06-02 01:28:00 +03:00
Kruglov Pavel
d9113a3b75
Style 2023-06-01 16:57:43 +02:00
Kruglov Pavel
0beca0336d
Merge pull request #49112 from ClickHouse/Avogar-patch-3 
Fix possible terminate called for uncaught exception in some places
 2023-05-31 16:55:43 +02:00
avogar
d4efbbfbd3 Allow to skip empty files in file/s3/url/hdfs table functions 2023-05-30 19:32:24 +00:00
Sema Checherinda
206efee9b7 add test with merge tree table 2023-05-26 18:59:21 +02:00
Sergei Trifonov
0d1f2e297b
Unify priorities: lower value means higher priority (#50205) 2023-05-26 15:55:30 +02:00
Kseniia Sumarokova
f1a3c9cfd5
Merge pull request #50109 from kssenii/abstract-async-prefetched-buffer 
Make async prefetched buffer work with arbitrary impl
 2023-05-25 15:06:44 +02:00
avogar
bc527c7588 Don't send head request for all keys in Iceberg schema inference 2023-05-24 17:07:31 +00:00
avogar
88e4c93abc Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster 2023-05-22 19:19:57 +00:00
kssenii
295fe3b228 Merge remote-tracking branch 'upstream/master' into abstract-async-prefetched-buffer 2023-05-22 20:01:38 +02:00
kssenii
c4d862a16f Make async reader work with any impl 2023-05-22 19:54:04 +02:00
kssenii
3e42ee7f2b Get rid of finalize callback in object storages 2023-05-19 17:29:37 +02:00
Sema Checherinda
7fbf87be17 rework WriteBufferFromS3, squashed 2023-05-10 18:31:47 +00:00
Kseniia Sumarokova
336bb41c5d
Merge branch 'master' into remove-dependency-from-context 2023-05-08 12:46:10 +02:00
Michael Kolupaev
3bd1489f18 Propagate input_format_parquet_preserve_order to parallelizeOutputAfterReading() 2023-05-05 04:20:27 +00:00
Michael Kolupaev
eb3b774ad0 Better control over Parquet row group size 2023-05-04 14:59:55 -07:00
kssenii
1433f5ffc9 Merge remote-tracking branch 'upstream/master' into remove-dependency-from-context 2023-05-04 13:24:02 +02:00
Antonio Andelic
a68a023ca7
Merge pull request #48724 from johanngan/sse-kms 
Support SSE-KMS configuration with S3 client
 2023-05-04 13:20:54 +02:00
alesapin
412b161104
Merge pull request #48791 from kssenii/better-local-object-storage 
Make local object storage work consistently with s3 object storage, fix problem with append, make it configurable as independent storage
 2023-05-04 11:47:43 +02:00
johanngan
731823b873 Add support for SSE-KMS configuration with S3 
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html

Similar to the server_side_encryption_customer_key_base64 option for
configuring SSE-C with S3, add the following settings to configure
SSE-KMS on a per-endpoint/disk basis:
  - server_side_encryption_kms_key_id
  - server_side_encryption_kms_encryption_context
  - server_side_encryption_kms_bucket_key_enabled
 2023-05-03 21:35:38 -05:00
Nikita Mikhaylov
954e3b724c
Speedup outdated parts loading (#49317) 2023-05-03 18:56:45 +02:00
kssenii
189f276ff5 Fix 2023-05-03 13:16:08 +02:00
kssenii
ecfbf1e304 Remove dependency from DB::Context in readers 2023-05-02 21:45:27 +02:00
Kseniia Sumarokova
45e2d296f9
Merge branch 'master' into better-local-object-storage 2023-04-27 14:54:04 +02:00
avogar
c503f6532c Add more finalize() to avoid terminate 2023-04-24 15:11:36 +00:00
Alexey Milovidov
67de39c2d9
Merge pull request #48727 from ClickHouse/parallel-processing-from-storages 
Parallelize query processing right after reading FROM ...
 2023-04-23 23:10:32 +03:00
kssenii
d8023806a9 Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-23 12:39:34 +02:00
Kseniia Sumarokova
bd748045ad
Fix typo 2023-04-21 18:54:23 +02:00
Igor Nikonov
d5eb65b5ea Remove redundant narrowPipe() 2023-04-18 22:41:28 +00:00
kssenii
16b027ed0c Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-18 16:25:08 +02:00
kssenii
b77e9c1ef0 Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-17 16:44:10 +02:00
kssenii
d2c73a5522 Better 2023-04-17 16:41:21 +02:00
Michael Kolupaev
473f212c82 Hopefully fix assertion failure in CachedOnDiskReadBufferFromFile 2023-04-17 04:58:32 +00:00
Michael Kolupaev
87be78e6de Better 2023-04-17 04:58:32 +00:00
Michael Kolupaev
2d4fe85513 Something 2023-04-17 04:58:32 +00:00
kssenii
3fb4cd0f52 Fix s3 test 2023-04-05 14:13:46 +02:00
kssenii
a3d69694f4 Fix build 2023-04-04 23:13:17 +02:00
kssenii
f44c53b97a Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes 2023-04-04 22:41:22 +02:00
Antonio Andelic
a329d80bfa
Merge pull request #47397 from ClickHouse/enable-env-credentials-default 
Enable `use_environment_credentials` by default
 2023-04-04 10:00:03 +02:00
kssenii
9b3d0ec86d Adjustments after conflicts 2023-04-03 19:53:34 +02:00
kssenii
8915f49b7d Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes 2023-04-03 17:43:42 +02:00
kssenii
5578cb08ad Fix s3 cluster 2023-04-03 14:40:04 +02:00
Anton Popov
f715bd95f1 fix writing to StorageS3 2023-03-31 14:08:28 +00:00
Antonio Andelic
e982f2a67a Merge branch 'master' into enable-env-credentials-default 2023-03-31 09:11:01 +00:00
Anton Popov
5ceb855e7f
Merge branch 'master' into fix-race-storage-s3 2023-03-31 04:16:35 +02:00
Anton Popov
38389d878c fix one more race in StorageS3 2023-03-30 21:06:53 +00:00
kssenii
319417062f Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes 2023-03-30 18:29:46 +02:00
Antonio Andelic
80cb121d2a
Merge pull request #48092 from ClickHouse/nosign-keyword-for-s3 
Add support for `NOSIGN` keyword and `no_sign_request` config for S3
 2023-03-30 18:10:56 +02:00
Anton Popov
e72472e71b
Merge branch 'master' into fix-race-storage-s3 2023-03-30 16:19:57 +02:00
kssenii
539414554f Fix s3 2023-03-30 15:32:38 +02:00
Antonio Andelic
9db58532f4
Clang-tidy fix 2023-03-30 08:41:14 +02:00
Anton Popov
ed29c141fb fix race in StorageS3 2023-03-29 22:13:45 +00:00
Antonio Andelic
7b1ad221b2 Address PR comments 2023-03-29 11:08:44 +00:00
Azat Khuzhin
f38a7aeabe ThreadPool metrics introspection 
There are lots of thread pools and simple local-vs-global is not enough
already, it is good to know which one in particular uses threads.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-29 10:46:59 +02:00
kssenii
82b642c9c6 Fix style check 2023-03-28 21:57:14 +02:00
kssenii
13f29a7242 Better 2023-03-28 18:57:24 +02:00
Antonio Andelic
160aa186bb Add support for NOSIGN keyword and no_sign_request config 2023-03-28 07:05:35 +00:00
kssenii
36cc6fee51 Rewrite data lakes (part 1) 2023-03-24 22:35:12 +01:00
kssenii
cae3b335d6 Merge remote-tracking branch 'upstream/master' into named-collections-finish 2023-03-20 11:23:22 +01:00
Antonio Andelic
1b7401b58a
Update src/Storages/StorageS3.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-03-17 15:46:15 +01:00
Antonio Andelic
a170a909a4 Add expiration window for S3 credentials 2023-03-10 10:06:32 +00:00
Antonio Andelic
5bc21538e5 Enable use_environment_credentials by default 2023-03-09 10:31:55 +00:00
kssenii
8f2d75cef8 Fix tests 2023-03-05 12:56:00 +01:00
flynn
b3a9468661 fix 2023-02-17 12:42:24 +00:00
flynn
a39f6f419b refactor 2023-02-17 08:27:52 +00:00
flynn
ecc39978d7 fix conflict 2023-02-16 02:23:55 +00:00
Kruglov Pavel
4f380370a9
Fix s3Cluster schema inference in parallel distributed insert select (#46381) 
* Fix s3Cluster schema inference in parallel distributed insert select
* Try fix flaky test
* Try SYSTEM SYNC REPLICA to avoid test flakiness
 2023-02-15 15:30:43 +01:00
flynn
289c5c60d3 fix 2023-02-15 11:24:06 +00:00
Antonio Andelic
5ab24285fc Fix arg parsing 2023-02-14 08:33:59 +00:00
Antonio Andelic
3a6ea861d8 Extract common argument parsing logic 2023-02-13 12:27:49 +00:00
kssenii
3067c1d723 Merge remote-tracking branch 'upstream/master' into resubmit-prefetches 2023-02-11 11:36:23 +01:00
kssenii
b0b865c32e Resubmit prefetches 2023-02-08 21:26:24 +01:00
kssenii
9485873a2f Fix integration test 2023-02-07 12:45:23 +01:00
kssenii
ab0dedf0c8 Simplify code around storage s3 configuration 2023-02-06 16:23:17 +01:00
Kseniia Sumarokova
38c001ca42
Merge pull request #45957 from xiedeyantu/s3_file_not_found 
Throw an error on no files satisfying S3 wildcard
 2023-02-06 12:32:12 +01:00
xiedeyantu
f13eedd644 change settings name 2023-02-04 22:11:14 +08:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution (#45783) 
* Update aws

* Define S3 client with bucket and endpoint resolution

* Add defines for ErrorCodes

* Use S3Client everywhere

* Remove unused errorcode

* Add DROP S3 CLIENT CACHE query

* Add a comment

* Fix style

* Update aws

* Update reference files

* Add missing include

* Fix unit test

* Remove unneeded declarations

* Correctly use RetryStrategy

* Rename S3Client to Client

* Fix retry count

* fix clang-tidy warnings
 2023-02-03 14:30:52 +01:00
xiedeyantu
562642ab7f add settings s3_allow_throw_if_mismatch_files 2023-02-03 12:27:13 +08:00
xiedeyantu
e22cc0eb98 Throw an error on no files satisfying S3 wildcard 2023-02-02 19:13:34 +08:00
Vitaly Baranov
af74c008c4 Use one request to implement S3ObjectStorage::getObjectMetadata instead of two ones. 2023-01-27 18:42:37 +01:00
Vitaly Baranov
aea9ccdb60 Pass request settings to S3::getObjectInfo(). 2023-01-27 15:10:09 +01:00
Anton Popov
b58b73b0e7
Merge pull request #45529 from CurtizJ/fix-storage-s3-race 
Try to fix test `test_storage_s3/test.py::test_wrong_s3_syntax` (race in `StorageS3`)
 2023-01-26 14:21:32 +01:00
Anton Popov
8a1ea62aec fix race on cancelation of query in StorageS3 2023-01-24 01:12:01 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Anton Popov
41a199e175
Fix crash when ListObjects request fails (#45371) 2023-01-20 20:10:23 +01:00
Anton Popov
f40fd7a151
Add checks for compilation of regexps (#45356) 2023-01-17 23:46:04 +01:00
Vitaly Baranov
a955504043 Move throw_on_error parameter to the end. 2023-01-15 20:28:16 +01:00
Maksim Kita
4571c74fdd Fixed build 2023-01-10 16:49:55 +01:00
Anton Popov
79e89cf69c
Merge pull request #44939 from ClickHouse/revert-44493-s3_optimize 
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
 2023-01-10 10:42:18 +01:00
Anton Popov
4447afb14d
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3" 2023-01-05 16:38:20 +01:00
kssenii
67509aa2d5 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2023-01-03 16:41:30 +01:00
Antonio Andelic
e0b8fd528d
Merge pull request #44842 from ClickHouse/fix-data-race-storage-s3 
Fix data race in StorageS3
 2023-01-03 09:20:08 +01:00
Kruglov Pavel
966f57ef68
Merge pull request #42777 from Avogar/improve-streaming-engines 
Refactor and Improve streaming engines Kafka/RabbitMQ/NATS and data formats
 2023-01-02 15:59:06 +01:00
Antonio Andelic
e07d820156 Fix race on total_size 2023-01-02 14:28:27 +00:00
Kruglov Pavel
0844fe7089
Merge pull request #44493 from xiedeyantu/s3_optimize 
If user only need virtual columns, we don't need to initialize ReadBufferFromS3
 2022-12-30 15:44:20 +01:00
xiedeyantu
d6a92fbd63 better 2022-12-28 10:11:33 +08:00
xiedeyantu
2dd809e403 fix 2022-12-23 19:45:26 +08:00
xiedeyantu
68aeb39892 fix 2022-12-23 19:33:08 +08:00
xiedeyantu
d0eb22a1cd fix 2022-12-23 19:25:14 +08:00
xiedeyantu
b5fd23358f fixed 2022-12-23 19:11:51 +08:00
chen
5c8fb627b3
Update StorageS3.cpp 2022-12-22 08:28:09 +08:00
chen
9fc1020855
Update StorageS3.cpp 2022-12-22 08:15:50 +08:00
xiedeyantu
6a9ec7efb1 If user only need virtual columns, we don't need to initialize ReadBufferFromS3 2022-12-21 23:43:56 +08:00
Kruglov Pavel
5e01a3d74e
Merge branch 'master' into improve-streaming-engines 2022-12-21 10:51:50 +01:00
kssenii
1d75f740d7 Fix tests 2022-12-20 22:33:54 +01:00
kssenii
6bd4f8c029 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2022-12-20 21:17:28 +01:00