thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
135,292 Commits 9,328 Branches 1,783 Tags 2.1 GiB
f944faec6f
Commit Graph

595 Commits

Author SHA1 Message Date
Michael Kolupaev
2f4d433e69 Parquet filter pushdown 2023-08-21 14:15:52 -07:00
avogar
47304bf7aa Optimize count from files in most input formats 2023-08-21 12:30:52 +00:00
avogar
8e445b5461 Fixes 2023-08-18 17:49:40 +00:00
avogar
4c32097df3 Use filter by file/path before reading in url/file/hdfs table functions, reduce code duplication 2023-08-17 16:54:43 +00:00
Anton Popov
ff137773e7
Merge branch 'master' into formats-with-subcolumns 2023-08-02 15:24:56 +02:00
Kseniia Sumarokova
8893670d98
Merge branch 'master' into s3queue 2023-08-01 18:42:32 +02:00
kssenii
870a506a0b Some fixes 2023-07-31 20:07:23 +02:00
Nikolai Kochetov
c4a00b8c68 Throw S3Exception whenever possible. 2023-07-31 15:39:20 +00:00
Kruglov Pavel
0d34e97dbe
Merge branch 'master' into formats-with-subcolumns 2023-07-26 13:30:35 +02:00
Kruglov Pavel
fec5675cd4
Merge branch 'master' into better-progress-bar-2 2023-07-24 19:59:38 +02:00
avogar
3682254342 Fix hdfs tests 2023-07-18 13:49:23 +00:00
Kruglov Pavel
64e88cde21
Merge branch 'master' into better-progress-bar-2 2023-07-18 13:37:53 +02:00
Kruglov Pavel
1dd05319b5
Merge branch 'master' into formats-with-subcolumns 2023-07-17 19:13:42 +02:00
Nikita Mikhaylov
ee0bbc0e54
Merge branch 'master' into headers-blacklist 2023-07-17 19:08:52 +02:00
avogar
d11cd0dc30 Fix tests 2023-07-05 17:56:03 +00:00
avogar
98aa6b317f Support reading subcolumns from file/s3/hdfs/url/azureBlobStorage table functions 2023-07-04 21:17:26 +00:00
Nikolay Degterinsky
8dfa773f44
Merge branch 'master' into headers-blacklist 2023-06-30 23:40:17 +02:00
avogar
4eeb431003 Merge branch 'master' of github.com:ClickHouse/ClickHouse into better-progress-bar-2 2023-06-28 18:53:08 +00:00
Alexey Milovidov
cca3f9937c
Merge branch 'master' into retry 2023-06-24 22:29:17 +03:00
avogar
c679dd400e Make better 2023-06-23 13:43:40 +00:00
avogar
24fab7bfde Remove old includes 2023-06-22 18:48:15 +00:00
avogar
cf082f2f9a Use read_bytes/total_bytes_to_read for progress bar in s3/file/url/... table functions 2023-06-22 17:24:43 +00:00
Sema Checherinda
95349a405b release buffers with exception context 2023-06-22 13:00:13 +02:00
Sema Checherinda
c2fa7a1030
Merge branch 'master' into memory-leak 2023-06-21 21:15:01 +02:00
Michael Kolupaev
4a570a05c9 Decrease default timeouts for S3 and HTTP requests 2023-06-21 18:08:50 +00:00
avogar
3209ebe34b Improve progress bar for file/s3/hdfs/url table functions. Step 1 2023-06-16 15:51:18 +00:00
Kruglov Pavel
f8ddfb1fd8
Merge branch 'master' into allow-skip-empty-files 2023-06-16 13:23:41 +02:00
Nikolay Degterinsky
9a25958be8 Add HTTP header filtering 2023-06-15 13:49:49 +00:00
avogar
870f3d1270 Fix comments 2023-06-15 12:59:46 +00:00
Kseniia Sumarokova
c786fbf8bd
Add comment 2023-06-15 10:22:02 +02:00
kssenii
827ac17dc2 Fix 2023-06-14 12:59:06 +02:00
avogar
2e1f56ae33 Address comments 2023-06-13 14:43:50 +00:00
Sema Checherinda
2c018f5261 do not call finalize after exception 2023-06-13 10:48:31 +02:00
Kruglov Pavel
e4838725e3
Merge branch 'master' into allow-skip-empty-files 2023-06-12 20:03:23 +02:00
kssenii
1cbcd2f2ef Refactor reading from object storages 2023-06-08 14:33:59 +02:00
Kruglov Pavel
bf28074d32
Merge branch 'master' into allow-skip-empty-files 2023-06-08 12:36:18 +02:00
Antonio Andelic
b11f744252
Correctly disable async insert with deduplication when it's not needed (#50663) 
* Correctly disable async insert when it's not used

* Better

* Add comment

* Better

* Fix tests

---------

Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-06-07 20:33:08 +02:00
Nikita Mikhaylov
e87348010d
Rework loading and removing of data parts for MergeTree tables. (#49474) 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-06-06 14:42:56 +02:00
Michael Kolupaev
b51064a508 Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead 2023-06-01 18:48:30 -07:00
Alexey Milovidov
52ffaa4630
Merge pull request #50203 from Avogar/head-requests-on-shcema-inference 
Don't send head request for all keys in Iceberg schema inference
 2023-06-02 01:28:00 +03:00
Kruglov Pavel
d9113a3b75
Style 2023-06-01 16:57:43 +02:00
Kruglov Pavel
0beca0336d
Merge pull request #49112 from ClickHouse/Avogar-patch-3 
Fix possible terminate called for uncaught exception in some places
 2023-05-31 16:55:43 +02:00
avogar
d4efbbfbd3 Allow to skip empty files in file/s3/url/hdfs table functions 2023-05-30 19:32:24 +00:00
Sema Checherinda
206efee9b7 add test with merge tree table 2023-05-26 18:59:21 +02:00
Sergei Trifonov
0d1f2e297b
Unify priorities: lower value means higher priority (#50205) 2023-05-26 15:55:30 +02:00
Kseniia Sumarokova
f1a3c9cfd5
Merge pull request #50109 from kssenii/abstract-async-prefetched-buffer 
Make async prefetched buffer work with arbitrary impl
 2023-05-25 15:06:44 +02:00
avogar
bc527c7588 Don't send head request for all keys in Iceberg schema inference 2023-05-24 17:07:31 +00:00
avogar
88e4c93abc Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster 2023-05-22 19:19:57 +00:00
kssenii
295fe3b228 Merge remote-tracking branch 'upstream/master' into abstract-async-prefetched-buffer 2023-05-22 20:01:38 +02:00
kssenii
c4d862a16f Make async reader work with any impl 2023-05-22 19:54:04 +02:00
kssenii
3e42ee7f2b Get rid of finalize callback in object storages 2023-05-19 17:29:37 +02:00
Sema Checherinda
7fbf87be17 rework WriteBufferFromS3, squashed 2023-05-10 18:31:47 +00:00
Kseniia Sumarokova
336bb41c5d
Merge branch 'master' into remove-dependency-from-context 2023-05-08 12:46:10 +02:00
Michael Kolupaev
3bd1489f18 Propagate input_format_parquet_preserve_order to parallelizeOutputAfterReading() 2023-05-05 04:20:27 +00:00
Michael Kolupaev
eb3b774ad0 Better control over Parquet row group size 2023-05-04 14:59:55 -07:00
kssenii
1433f5ffc9 Merge remote-tracking branch 'upstream/master' into remove-dependency-from-context 2023-05-04 13:24:02 +02:00
Antonio Andelic
a68a023ca7
Merge pull request #48724 from johanngan/sse-kms 
Support SSE-KMS configuration with S3 client
 2023-05-04 13:20:54 +02:00
alesapin
412b161104
Merge pull request #48791 from kssenii/better-local-object-storage 
Make local object storage work consistently with s3 object storage, fix problem with append, make it configurable as independent storage
 2023-05-04 11:47:43 +02:00
johanngan
731823b873 Add support for SSE-KMS configuration with S3 
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html

Similar to the server_side_encryption_customer_key_base64 option for
configuring SSE-C with S3, add the following settings to configure
SSE-KMS on a per-endpoint/disk basis:
  - server_side_encryption_kms_key_id
  - server_side_encryption_kms_encryption_context
  - server_side_encryption_kms_bucket_key_enabled
 2023-05-03 21:35:38 -05:00
Nikita Mikhaylov
954e3b724c
Speedup outdated parts loading (#49317) 2023-05-03 18:56:45 +02:00
kssenii
189f276ff5 Fix 2023-05-03 13:16:08 +02:00
kssenii
ecfbf1e304 Remove dependency from DB::Context in readers 2023-05-02 21:45:27 +02:00
Kseniia Sumarokova
45e2d296f9
Merge branch 'master' into better-local-object-storage 2023-04-27 14:54:04 +02:00
avogar
c503f6532c Add more finalize() to avoid terminate 2023-04-24 15:11:36 +00:00
Alexey Milovidov
67de39c2d9
Merge pull request #48727 from ClickHouse/parallel-processing-from-storages 
Parallelize query processing right after reading FROM ...
 2023-04-23 23:10:32 +03:00
kssenii
d8023806a9 Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-23 12:39:34 +02:00
Kseniia Sumarokova
bd748045ad
Fix typo 2023-04-21 18:54:23 +02:00
Igor Nikonov
d5eb65b5ea Remove redundant narrowPipe() 2023-04-18 22:41:28 +00:00
kssenii
16b027ed0c Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-18 16:25:08 +02:00
kssenii
b77e9c1ef0 Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-17 16:44:10 +02:00
kssenii
d2c73a5522 Better 2023-04-17 16:41:21 +02:00
Michael Kolupaev
473f212c82 Hopefully fix assertion failure in CachedOnDiskReadBufferFromFile 2023-04-17 04:58:32 +00:00
Michael Kolupaev
87be78e6de Better 2023-04-17 04:58:32 +00:00
Michael Kolupaev
2d4fe85513 Something 2023-04-17 04:58:32 +00:00
kssenii
3fb4cd0f52 Fix s3 test 2023-04-05 14:13:46 +02:00
kssenii
a3d69694f4 Fix build 2023-04-04 23:13:17 +02:00
kssenii
f44c53b97a Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes 2023-04-04 22:41:22 +02:00
Antonio Andelic
a329d80bfa
Merge pull request #47397 from ClickHouse/enable-env-credentials-default 
Enable `use_environment_credentials` by default
 2023-04-04 10:00:03 +02:00
kssenii
9b3d0ec86d Adjustments after conflicts 2023-04-03 19:53:34 +02:00
kssenii
8915f49b7d Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes 2023-04-03 17:43:42 +02:00
kssenii
5578cb08ad Fix s3 cluster 2023-04-03 14:40:04 +02:00
Anton Popov
f715bd95f1 fix writing to StorageS3 2023-03-31 14:08:28 +00:00
Antonio Andelic
e982f2a67a Merge branch 'master' into enable-env-credentials-default 2023-03-31 09:11:01 +00:00
Anton Popov
5ceb855e7f
Merge branch 'master' into fix-race-storage-s3 2023-03-31 04:16:35 +02:00
Anton Popov
38389d878c fix one more race in StorageS3 2023-03-30 21:06:53 +00:00
kssenii
319417062f Merge remote-tracking branch 'upstream/master' into better-tests-for-data-lakes 2023-03-30 18:29:46 +02:00
Antonio Andelic
80cb121d2a
Merge pull request #48092 from ClickHouse/nosign-keyword-for-s3 
Add support for `NOSIGN` keyword and `no_sign_request` config for S3
 2023-03-30 18:10:56 +02:00
Anton Popov
e72472e71b
Merge branch 'master' into fix-race-storage-s3 2023-03-30 16:19:57 +02:00
kssenii
539414554f Fix s3 2023-03-30 15:32:38 +02:00
Antonio Andelic
9db58532f4
Clang-tidy fix 2023-03-30 08:41:14 +02:00
Anton Popov
ed29c141fb fix race in StorageS3 2023-03-29 22:13:45 +00:00
Antonio Andelic
7b1ad221b2 Address PR comments 2023-03-29 11:08:44 +00:00
Azat Khuzhin
f38a7aeabe ThreadPool metrics introspection 
There are lots of thread pools and simple local-vs-global is not enough
already, it is good to know which one in particular uses threads.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-29 10:46:59 +02:00
kssenii
82b642c9c6 Fix style check 2023-03-28 21:57:14 +02:00
kssenii
13f29a7242 Better 2023-03-28 18:57:24 +02:00
Antonio Andelic
160aa186bb Add support for NOSIGN keyword and no_sign_request config 2023-03-28 07:05:35 +00:00
kssenii
36cc6fee51 Rewrite data lakes (part 1) 2023-03-24 22:35:12 +01:00
kssenii
cae3b335d6 Merge remote-tracking branch 'upstream/master' into named-collections-finish 2023-03-20 11:23:22 +01:00
Antonio Andelic
1b7401b58a
Update src/Storages/StorageS3.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-03-17 15:46:15 +01:00
Antonio Andelic
a170a909a4 Add expiration window for S3 credentials 2023-03-10 10:06:32 +00:00
Antonio Andelic
5bc21538e5 Enable use_environment_credentials by default 2023-03-09 10:31:55 +00:00
kssenii
8f2d75cef8 Fix tests 2023-03-05 12:56:00 +01:00
flynn
b3a9468661 fix 2023-02-17 12:42:24 +00:00
flynn
a39f6f419b refactor 2023-02-17 08:27:52 +00:00
flynn
ecc39978d7 fix conflict 2023-02-16 02:23:55 +00:00
Kruglov Pavel
4f380370a9
Fix s3Cluster schema inference in parallel distributed insert select (#46381) 
* Fix s3Cluster schema inference in parallel distributed insert select
* Try fix flaky test
* Try SYSTEM SYNC REPLICA to avoid test flakiness
 2023-02-15 15:30:43 +01:00
flynn
289c5c60d3 fix 2023-02-15 11:24:06 +00:00
Antonio Andelic
5ab24285fc Fix arg parsing 2023-02-14 08:33:59 +00:00
Antonio Andelic
3a6ea861d8 Extract common argument parsing logic 2023-02-13 12:27:49 +00:00
kssenii
3067c1d723 Merge remote-tracking branch 'upstream/master' into resubmit-prefetches 2023-02-11 11:36:23 +01:00
kssenii
b0b865c32e Resubmit prefetches 2023-02-08 21:26:24 +01:00
kssenii
9485873a2f Fix integration test 2023-02-07 12:45:23 +01:00
kssenii
ab0dedf0c8 Simplify code around storage s3 configuration 2023-02-06 16:23:17 +01:00
Kseniia Sumarokova
38c001ca42
Merge pull request #45957 from xiedeyantu/s3_file_not_found 
Throw an error on no files satisfying S3 wildcard
 2023-02-06 12:32:12 +01:00
xiedeyantu
f13eedd644 change settings name 2023-02-04 22:11:14 +08:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution (#45783) 
* Update aws

* Define S3 client with bucket and endpoint resolution

* Add defines for ErrorCodes

* Use S3Client everywhere

* Remove unused errorcode

* Add DROP S3 CLIENT CACHE query

* Add a comment

* Fix style

* Update aws

* Update reference files

* Add missing include

* Fix unit test

* Remove unneeded declarations

* Correctly use RetryStrategy

* Rename S3Client to Client

* Fix retry count

* fix clang-tidy warnings
 2023-02-03 14:30:52 +01:00
xiedeyantu
562642ab7f add settings s3_allow_throw_if_mismatch_files 2023-02-03 12:27:13 +08:00
xiedeyantu
e22cc0eb98 Throw an error on no files satisfying S3 wildcard 2023-02-02 19:13:34 +08:00
Vitaly Baranov
af74c008c4 Use one request to implement S3ObjectStorage::getObjectMetadata instead of two ones. 2023-01-27 18:42:37 +01:00
Vitaly Baranov
aea9ccdb60 Pass request settings to S3::getObjectInfo(). 2023-01-27 15:10:09 +01:00
Anton Popov
b58b73b0e7
Merge pull request #45529 from CurtizJ/fix-storage-s3-race 
Try to fix test `test_storage_s3/test.py::test_wrong_s3_syntax` (race in `StorageS3`)
 2023-01-26 14:21:32 +01:00
Anton Popov
8a1ea62aec fix race on cancelation of query in StorageS3 2023-01-24 01:12:01 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Anton Popov
41a199e175
Fix crash when ListObjects request fails (#45371) 2023-01-20 20:10:23 +01:00
Anton Popov
f40fd7a151
Add checks for compilation of regexps (#45356) 2023-01-17 23:46:04 +01:00
Vitaly Baranov
a955504043 Move throw_on_error parameter to the end. 2023-01-15 20:28:16 +01:00
Maksim Kita
4571c74fdd Fixed build 2023-01-10 16:49:55 +01:00
Anton Popov
79e89cf69c
Merge pull request #44939 from ClickHouse/revert-44493-s3_optimize 
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
 2023-01-10 10:42:18 +01:00
Anton Popov
4447afb14d
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3" 2023-01-05 16:38:20 +01:00
kssenii
67509aa2d5 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2023-01-03 16:41:30 +01:00
Antonio Andelic
e0b8fd528d
Merge pull request #44842 from ClickHouse/fix-data-race-storage-s3 
Fix data race in StorageS3
 2023-01-03 09:20:08 +01:00
Kruglov Pavel
966f57ef68
Merge pull request #42777 from Avogar/improve-streaming-engines 
Refactor and Improve streaming engines Kafka/RabbitMQ/NATS and data formats
 2023-01-02 15:59:06 +01:00
Antonio Andelic
e07d820156 Fix race on total_size 2023-01-02 14:28:27 +00:00
Kruglov Pavel
0844fe7089
Merge pull request #44493 from xiedeyantu/s3_optimize 
If user only need virtual columns, we don't need to initialize ReadBufferFromS3
 2022-12-30 15:44:20 +01:00
xiedeyantu
d6a92fbd63 better 2022-12-28 10:11:33 +08:00
xiedeyantu
2dd809e403 fix 2022-12-23 19:45:26 +08:00
xiedeyantu
68aeb39892 fix 2022-12-23 19:33:08 +08:00
xiedeyantu
d0eb22a1cd fix 2022-12-23 19:25:14 +08:00
xiedeyantu
b5fd23358f fixed 2022-12-23 19:11:51 +08:00
chen
5c8fb627b3
Update StorageS3.cpp 2022-12-22 08:28:09 +08:00
chen
9fc1020855
Update StorageS3.cpp 2022-12-22 08:15:50 +08:00
xiedeyantu
6a9ec7efb1 If user only need virtual columns, we don't need to initialize ReadBufferFromS3 2022-12-21 23:43:56 +08:00
Kruglov Pavel
5e01a3d74e
Merge branch 'master' into improve-streaming-engines 2022-12-21 10:51:50 +01:00
kssenii
1d75f740d7 Fix tests 2022-12-20 22:33:54 +01:00
kssenii
6bd4f8c029 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2022-12-20 21:17:28 +01:00
Raúl Marín
45d27f461b
Merge branch 'master' into perf_experiment 2022-12-20 09:07:48 +00:00
Nikolai Kochetov
f7c308077d Fixing build. 2022-12-17 17:12:04 +00:00
Nikolai Kochetov
b3278211bb Fixing tests. 2022-12-17 16:06:42 +00:00
Nikolai Kochetov
b2355a2212 Fixing tests. 2022-12-17 16:02:34 +00:00
Nikolai Kochetov
98ebef7914 Fixing special build. 2022-12-17 15:27:01 +00:00