Commit Graph

50 Commits

Author SHA1 Message Date
vdimir
38d8a9a956
Do not log HTTP body 2024-10-09 08:51:09 +00:00
flynn
e4aceed36a Add virtual column _headers for url table engine 2024-08-25 07:49:30 +00:00
Alexey Milovidov
487c6a097e Merge branch 'master' into revert-65361-revert-65164-ecs 2024-08-18 06:19:27 +02:00
Alexey Milovidov
414ebf035d Fix error 2024-07-26 15:32:05 +02:00
Alexey Milovidov
9c6026965d Fix error 2024-07-26 04:55:53 +02:00
Alexey Milovidov
57a6d28100 Fix error 2024-07-24 11:17:43 +02:00
Alexey Milovidov
dc601dc745 Fix error 2024-07-22 07:37:15 +02:00
Alexey Milovidov
32f624eeba Fix error 2024-07-22 07:35:10 +02:00
Alexey Milovidov
db549c93a1 Fix error 2024-07-22 07:34:34 +02:00
Alexey Milovidov
4ef9cb6d7a Fix style 2024-07-18 23:13:32 +02:00
Alexey Milovidov
c7be25f0a1 Fix everything 2024-07-18 04:54:36 +02:00
Michael Kolupaev
b98c123f3a Remove logging, if this happens again we'll have to guess again 2024-07-04 18:41:20 +00:00
Michael Kolupaev
9c25a1f69b Exclude some more HTTP codes in ReadWriteBufferFromHTTP::getFileInfo() 2024-06-25 02:16:06 +00:00
Kruglov Pavel
4817657375
Revert "Revert "Fix AWS ECS"" 2024-06-17 23:43:03 +02:00
Alexander Tokmakov
b150b0f5fa
Revert "Fix AWS ECS" 2024-06-17 23:11:59 +02:00
Alexey Milovidov
8b1f1947f7 Slightly better user-agent 2024-06-16 17:39:36 +02:00
Alexander Gololobov
b2bcfaf344 Reduce log levels for ReadWriteBufferFromHTTP retries 2024-04-05 17:18:22 +02:00
Sema Checherinda
9a9f0161bb add test 2024-04-01 01:33:40 +02:00
Sema Checherinda
1ea058a702 ReadWriteBufferFromHTTP set right header host when redireced 2024-03-28 19:33:23 +01:00
Michael Kolupaev
899b856ad6 Add profile events for number of HTTP requests and bytes read 2024-03-26 09:14:55 +00:00
Michael Kolupaev
1fb902c5b7 Fix sending a new request every 2 MB 2024-03-26 09:14:55 +00:00
kssenii
296f7a1da2 Fix 2024-03-12 19:02:58 +01:00
Sema Checherinda
5c41727725 http connections pools 2024-03-10 10:00:18 +01:00
Raúl Marín
b12245fdc5 Decrease logging level for http retriable errors to Info 2024-02-28 19:47:42 +01:00
Azat Khuzhin
4228eaf44a Decrease logging level for http retriable errors to Warning
Messages like this:

    2024-02-12 21:25:24 [5c66f008cd40] 2024.02.12 14:25:01.596769 [ 2530 ] {e71fcb6d-356d-4962-95bd-ef8b8c504e11} <Error> ReadWriteBufferFromHTTP: HTTP request to `https://clickhouse-datasets-web.s3.us-east-1.amazonaws.com/store/78e/78ebf6a1-d987-4579-b3ec-00c1a087b1f3/201403_1_1_2/UserAgent.bin` failed at try 1/10 with bytes read: 311149/378695. Error: DB::HTTPException: Received error from remote server /store/78e/78ebf6a1-d987-4579-b3ec-00c1a087b1f3/201403_1_1_2/UserAgent.bin. HTTP status code: 500 Internal Server Error, body: <?xml version="1.0" encoding="UTF-8"?>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-02-13 08:31:28 +01:00
Maksim Kita
2a327107b6 Updated implementation 2024-01-25 14:31:49 +03:00
Kruglov Pavel
6c0615d493
Merge pull request #59099 from skyoct/fix/fix_mult_redirect
Disable "Too many redirects" error  retry
2024-01-24 12:22:01 +01:00
Kruglov Pavel
718064bb43
Minor change to restart CI 2024-01-24 12:19:10 +01:00
skyoct
95829e074e Opt code 2024-01-24 03:49:21 +00:00
skyoct
fdf8008804 Fix redirect retry 2024-01-22 10:27:35 +00:00
kssenii
4c8ced6af8 Fix log level 2024-01-19 18:04:58 +01:00
Sema Checherinda
802ad4ec47
Merge pull request #57523 from ClickHouse/remove-the-limit-for-connections-per-endpoint
Remove the limit for the number of connections per endpoint for background fetches + small refactoring
2023-12-19 19:16:01 +01:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
vdimir
1044e3366b
Update http_make_head_request doc 2023-11-24 09:32:20 +00:00
Tim Windelschmidt
51e3899f73
Add setting http_make_head_request
Clickhouse always does a HEAD request before actually executing the GET request,
this adds a settings to skip that request.
2023-11-24 09:32:16 +00:00
Arthur Passos
b6e205dcdf
Add ClickHouse setting to disable tunneling for HTTPS requests over HTTP proxy (#55033)
* initial commit. integ tests passing, need to re-run unit & my own personal tests

* partial refactoring to remove Protocol::ANY

* improve naming

* remove all usages of ProxyConfiguration::Protocol::ANY

* fix ut

* blabla

* support url functions as well

* support for HTTPS requests over HTTP proxy with tunneling off

* remove gtestabc

* fix silly mistake

* ...

* remove usages of httpclientsession::proxyconfig in src/

* got you

* remove stale comment

* it seems like I need reasonable defaults

* fix ut

* add some comments

* remove no longer needed header

* matrix out

* add https over http proxy with no tunneling

* soem docs

* partial refactoring

* rename to use_tunneling_for_https_requests_over_http_proxy

* improve docs

* use shorter version

* remove useless test

* rename the setting

* update

* fix typo

* fix setting docs typo

* move ); up

* move ) up
2023-11-04 13:47:52 -04:00
Michael Kolupaev
df71dcd94d Fix data race in copyFromIStreamWithProgressCallback 2023-09-11 21:04:37 +00:00
Kruglov Pavel
44db5fa992
Merge branch 'master' into cache-count 2023-08-24 17:21:18 +02:00
Arthur Passos
2bade7db08
Add global proxy setting (#51749)
* initial impl

* fix env ut

* move ut directory

* make sure no null proxy resolver is returned by ProxyConfigurationResolverProvider

* minor adjustment

* add a few tests, still incomplete

* add proxy support for url table function

* use proxy for select from url as well

* remove optional from return type, just returns empty config

* fix style

* style

* black

* ohg boy

* rm in progress file

* god pls don't let me kill anyone

* ...

* add use_aws guards

* remove hard coded s3 proxy resolver

* add concurrency-mt-unsafe

* aa

* black

* add logging back

* revert change

* imrpove code a bit

* helper functions and separate tests

* for some reason, this env test is not working..

* formatting

* :)

* clangtidy

* lint

* revert some stupid things

* small test adjusmtments

* simplify tests

* rename test

* remove extra line

* freaking style change

* simplify a bit

* fix segfault & remove an extra call

* tightly couple proxy provider with context..

* remove useless include

* rename config prefix parameter

* simplify provider a bit

* organize provider a bit

* add a few comments

* comment out proxy env tests

* fix nullptr in unit tests

* make sure old storage proxy config is properly covered without global context instance

* move a few functions from class to anonymous namespace

* fix no fallback for specific storage conf

* change API to accept http method instead of bool

* implement http/https distinction in listresolver, any still not implemented

* implement http/https distinction in remote resolver

* progress on code, improve tests and add url function working test

* use protcol instead of method for http and https

* small fix

* few more adjustments

* fix style

* black

* move enum to proxyconfiguration

* wip

* fix build

* fix ut

* delete atomicroundrobin class

* remove stale include

* add some tests.. need to spend some more time on the design..

* change design a bit

* progress

* use existing context for tests

* rename aux function and fix ut

* ..

* rename test

* try to simplify tests a bit

* simplify tests a bit more

* attempt to fix tests, accept more than one remote resolver

* use proper log id

* try waiting for resolver

* proper wait logic

* black

* empty

* address a few comments

* refactor tests

* remove old tests

* baclk

* use RAII to set/unset env

* black

* clang tidy

* fix env proxy not respecting any

* use log trace

* fix wrong logic in getRemoteREsolver

* fix wrong logic in getRemoteREsolver

* fix test

* remove unwanted code

* remove ClientConfigurationperRequest and auxilary classes

* remove unwanted code

* remove adapter test

* few adjustments and add test for s3 storage conf  with new proxy settings

* black

* use chassert for context

* Add getenv comment
2023-08-24 16:07:26 +03:00
avogar
894513f6cd Fix tests 2023-08-23 18:43:08 +00:00
Kruglov Pavel
c0bdd0e00b
Merge branch 'master' into cache-count 2023-08-22 14:42:22 +02:00
avogar
b4145aeddc Cache number of rows in files for count in file/s3/url/hdfs/azure functions 2023-08-22 11:59:59 +00:00
Alexey Milovidov
d133cafd9c Better documentation about redirects 2023-08-20 00:38:18 +02:00
avogar
2b8e4ebd4c Allow to disable decoding/encoding path in uri in URL engine 2023-07-19 19:48:39 +00:00
Nikita Taranov
63b9c1ac06 add test 2023-07-06 18:58:13 +02:00
Nikita Taranov
aec7205636 rework pool usage 2023-07-06 14:41:09 +02:00
Michael Kolupaev
b51064a508 Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead 2023-06-01 18:48:30 -07:00
Michael Kolupaev
fd9c9bb25b Split ReadWriteBufferFromHTTP.h into .h and .cpp file 2023-05-06 21:11:06 -07:00
Alexey Milovidov
eb0e3a83d0 Support SNI in https connections to remote resources 2020-10-22 05:02:26 +03:00
Ivan Lezhankin
06446b4f08 dbms/ → src/ 2020-04-03 18:14:31 +03:00