Commit Graph

2148 Commits

Author SHA1 Message Date
Robert Schulze
9fff447716
Re-enable clang-tidy checks 2023-09-26 09:34:12 +00:00
Azat Khuzhin
c706101891 Fix throttling of BACKUPs from/to S3 (in case native copy was not used)
In some cases native copy is not possible, and such requests should be
throttled.

v0: copyS3FileNativeWithFallback
v2: revert v0 and pass write_settings
v3: pass read_settings to copyFile()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-09-20 18:28:43 +02:00
Sema Checherinda
e7550523c8
Merge pull request #54651 from CheSema/limit_backoff_timeout
limit the delay before next try in S3
2023-09-18 19:41:26 +02:00
Robert Schulze
f5e8028bb1
Merge pull request #54642 from rschu1ze/broken-re2st
Remove broken lockless variant of re2
2023-09-17 15:30:57 +02:00
Kseniia Sumarokova
79697d4102
Update WebObjectStorage.cpp 2023-09-15 20:18:36 +02:00
Sema Checherinda
d9e15c00c9 limit the delay before next try in S3 2023-09-14 19:45:07 +02:00
Robert Schulze
7b378dbad3
Remove broken lockless variant of re2 2023-09-14 16:40:42 +00:00
Sema Checherinda
0ea0799118
Merge pull request #54430 from CheSema/remove-recursive-with-projection
reproduce and fix the bug in removeSharedRecursive
2023-09-12 00:18:55 +02:00
Sema Checherinda
d5268f3fb0 rename a member 2023-09-11 19:57:16 +02:00
Sergei Trifonov
08bad4d440
Merge pull request #47009 from ClickHouse/disks-and-io-sheduler 2023-09-11 07:22:14 +02:00
serxa
d326554caa fix keeper build 2023-09-10 14:55:40 +00:00
Sema Checherinda
1298dba0c6 fix the bug, add useful logs 2023-09-09 15:54:28 +02:00
SmitaRKulkarni
f0146c0ff0
Increase retries for test_merge_tree_azure_blob_storage (#54069)
* Increase max_single_download_retries & max_single_read_retries for test_merge_tree_azure_blob_storage

* Updated logs to log exception leading to detaching part

* Updated to catch RequestFailedException azure exception as its the base exception

* Updated isRetryableExcepion to take std::exception_ptr as input

* Fixed style check fails

* Fix build issue & moved NetException & TimeoutException to retryable exceptions

* Fixed style check with catch
2023-09-06 19:57:59 +02:00
Yakov Olkhovskiy
0dd6928a13
Merge pull request #53068 from nickitat/unify_keep_alive_setting
Unify setting http keep-alive timeout, increase default to 30s
2023-09-05 22:05:23 -04:00
Sergei Trifonov
258da91b5f
Merge branch 'master' into disks-and-io-sheduler 2023-09-04 15:14:28 +02:00
Kseniia Sumarokova
b3319f7908
Minor changes (#54171) 2023-09-03 15:47:52 +02:00
Michael Kolupaev
8505b3c940
Merge pull request #54134 from kssenii/refactor-reading-from-remote-fs
Small refactoring for read from object storage
2023-09-02 11:13:03 -07:00
kssenii
82948ac843 Review fixes 2023-09-02 14:00:07 +02:00
Sergei Trifonov
cf5ea46713
Merge pull request #53338 from azat/throttling-fixes
Fix IO throttling during copying whole directories
2023-09-01 18:29:15 +02:00
serxa
ef4dcf909f fix build 2023-09-01 16:23:15 +00:00
kssenii
9eb1dfcd12 Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
kssenii
3f20d1f578 Rename 2023-09-01 12:55:58 +02:00
serxa
9c92d413b3 Merge branch 'master' into disks-and-io-sheduler 2023-08-30 11:18:49 +00:00
Azat Khuzhin
51e8b16916 Fix IO throttling during copying whole directories
v2: pass local context during ALTER TABLE MOVE PARTITION TO VOLUME/DISK as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-29 23:13:46 +02:00
xuelei
ebc7dbb628 The configuration of S3 append '/' automatically if needed 2023-08-28 09:39:22 -03:00
xuelei
636e7ddae7 deal with conflict 2023-08-28 06:05:21 -03:00
xiaolei565
853551bd3e
Update registerDiskS3.cpp 2023-08-28 12:38:33 +08:00
xiaolei565
ef6e1fd2d6
Update diskSettings.cpp 2023-08-28 12:38:08 +08:00
xiaolei565
58a1160095
Update diskSettings.cpp 2023-08-28 12:37:54 +08:00
xiaolei565
8d2c03ea26
Update registerDiskS3.cpp 2023-08-28 11:41:53 +08:00
xiaolei565
b9e872ddf7
Update diskSettings.cpp 2023-08-28 11:28:31 +08:00
xiaolei565
3c457c47c2
Update diskSettings.cpp 2023-08-28 11:05:30 +08:00
xuelei
643dd499b8 deal with conflict 2023-08-27 23:46:53 -03:00
xuelei
bb12f7decc deal with conflict 2023-08-27 23:41:23 -03:00
alesapin
0e601b856f
Merge pull request #53751 from Algunenano/clang18
Support clang-18 (Wmissing-field-initializers)
2023-08-25 23:27:06 +02:00
Kseniia Sumarokova
34ac113af6
Merge pull request #53121 from kssenii/fix-cache-logical-error
Fix possible error from cache "Read unexpected size"
2023-08-25 12:16:24 +02:00
Raúl Marín
6490213207 Merge remote-tracking branch 'blessed/master' into clang18 2023-08-24 17:35:26 +02:00
Kruglov Pavel
44db5fa992
Merge branch 'master' into cache-count 2023-08-24 17:21:18 +02:00
Arthur Passos
2bade7db08
Add global proxy setting (#51749)
* initial impl

* fix env ut

* move ut directory

* make sure no null proxy resolver is returned by ProxyConfigurationResolverProvider

* minor adjustment

* add a few tests, still incomplete

* add proxy support for url table function

* use proxy for select from url as well

* remove optional from return type, just returns empty config

* fix style

* style

* black

* ohg boy

* rm in progress file

* god pls don't let me kill anyone

* ...

* add use_aws guards

* remove hard coded s3 proxy resolver

* add concurrency-mt-unsafe

* aa

* black

* add logging back

* revert change

* imrpove code a bit

* helper functions and separate tests

* for some reason, this env test is not working..

* formatting

* :)

* clangtidy

* lint

* revert some stupid things

* small test adjusmtments

* simplify tests

* rename test

* remove extra line

* freaking style change

* simplify a bit

* fix segfault & remove an extra call

* tightly couple proxy provider with context..

* remove useless include

* rename config prefix parameter

* simplify provider a bit

* organize provider a bit

* add a few comments

* comment out proxy env tests

* fix nullptr in unit tests

* make sure old storage proxy config is properly covered without global context instance

* move a few functions from class to anonymous namespace

* fix no fallback for specific storage conf

* change API to accept http method instead of bool

* implement http/https distinction in listresolver, any still not implemented

* implement http/https distinction in remote resolver

* progress on code, improve tests and add url function working test

* use protcol instead of method for http and https

* small fix

* few more adjustments

* fix style

* black

* move enum to proxyconfiguration

* wip

* fix build

* fix ut

* delete atomicroundrobin class

* remove stale include

* add some tests.. need to spend some more time on the design..

* change design a bit

* progress

* use existing context for tests

* rename aux function and fix ut

* ..

* rename test

* try to simplify tests a bit

* simplify tests a bit more

* attempt to fix tests, accept more than one remote resolver

* use proper log id

* try waiting for resolver

* proper wait logic

* black

* empty

* address a few comments

* refactor tests

* remove old tests

* baclk

* use RAII to set/unset env

* black

* clang tidy

* fix env proxy not respecting any

* use log trace

* fix wrong logic in getRemoteREsolver

* fix wrong logic in getRemoteREsolver

* fix test

* remove unwanted code

* remove ClientConfigurationperRequest and auxilary classes

* remove unwanted code

* remove adapter test

* few adjustments and add test for s3 storage conf  with new proxy settings

* black

* use chassert for context

* Add getenv comment
2023-08-24 16:07:26 +03:00
Kseniia Sumarokova
a4f39fe134
Merge branch 'master' into fix-cache-logical-error 2023-08-24 12:00:18 +02:00
Raúl Marín
93dac0c880 Support clang-18 (Wmissing-field-initializers) 2023-08-23 15:53:45 +02:00
Kruglov Pavel
592fa77987
Merge branch 'master' into cache-count 2023-08-23 15:18:02 +02:00
kssenii
57b5ad854d Fxi 2023-08-23 13:43:05 +02:00
Sema Checherinda
60103c577d
Merge pull request #53651 from CheSema/read_beyon_last_offset
fix Logical Error in AsynchronousBoundedReadBuffer
2023-08-23 11:21:47 +02:00
avogar
cd228754c8 Apply suggestions 2023-08-22 17:32:35 +00:00
Kruglov Pavel
c0bdd0e00b
Merge branch 'master' into cache-count 2023-08-22 14:42:22 +02:00
avogar
b4145aeddc Cache number of rows in files for count in file/s3/url/hdfs/azure functions 2023-08-22 11:59:59 +00:00
Sema Checherinda
81577e041b fix Logical Error in AsynchronousBoundedReadBuffer 2023-08-21 19:30:04 +02:00
Kseniia Sumarokova
404fb0588a
Merge branch 'master' into fix-cache-logical-error 2023-08-20 16:28:13 +02:00
kssenii
5382442da0 Merge remote-tracking branch 'upstream/master' into allowed-caches-dir-for-dynamic-disks 2023-08-19 08:44:36 +04:00