Commit Graph

132 Commits

Author SHA1 Message Date
flynn
e4aceed36a Add virtual column _headers for url table engine 2024-08-25 07:49:30 +00:00
Alexey Milovidov
c7be25f0a1 Fix everything 2024-07-18 04:54:36 +02:00
Sema Checherinda
1ea058a702 ReadWriteBufferFromHTTP set right header host when redireced 2024-03-28 19:33:23 +01:00
Robert Schulze
c17536c3b3
Enable clang-tidy in headers 2024-03-18 08:00:09 +00:00
kssenii
296f7a1da2 Fix 2024-03-12 19:02:58 +01:00
Sema Checherinda
77c5de700f fix how web disk read empty directories 2024-03-10 10:00:18 +01:00
Sema Checherinda
5c41727725 http connections pools 2024-03-10 10:00:18 +01:00
Maksim Kita
2a327107b6 Updated implementation 2024-01-25 14:31:49 +03:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Alexey Milovidov
82c461031e Fix build 2023-11-13 10:09:23 +01:00
Arthur Passos
b6e205dcdf
Add ClickHouse setting to disable tunneling for HTTPS requests over HTTP proxy (#55033)
* initial commit. integ tests passing, need to re-run unit & my own personal tests

* partial refactoring to remove Protocol::ANY

* improve naming

* remove all usages of ProxyConfiguration::Protocol::ANY

* fix ut

* blabla

* support url functions as well

* support for HTTPS requests over HTTP proxy with tunneling off

* remove gtestabc

* fix silly mistake

* ...

* remove usages of httpclientsession::proxyconfig in src/

* got you

* remove stale comment

* it seems like I need reasonable defaults

* fix ut

* add some comments

* remove no longer needed header

* matrix out

* add https over http proxy with no tunneling

* soem docs

* partial refactoring

* rename to use_tunneling_for_https_requests_over_http_proxy

* improve docs

* use shorter version

* remove useless test

* rename the setting

* update

* fix typo

* fix setting docs typo

* move ); up

* move ) up
2023-11-04 13:47:52 -04:00
Kruglov Pavel
44db5fa992
Merge branch 'master' into cache-count 2023-08-24 17:21:18 +02:00
Arthur Passos
2bade7db08
Add global proxy setting (#51749)
* initial impl

* fix env ut

* move ut directory

* make sure no null proxy resolver is returned by ProxyConfigurationResolverProvider

* minor adjustment

* add a few tests, still incomplete

* add proxy support for url table function

* use proxy for select from url as well

* remove optional from return type, just returns empty config

* fix style

* style

* black

* ohg boy

* rm in progress file

* god pls don't let me kill anyone

* ...

* add use_aws guards

* remove hard coded s3 proxy resolver

* add concurrency-mt-unsafe

* aa

* black

* add logging back

* revert change

* imrpove code a bit

* helper functions and separate tests

* for some reason, this env test is not working..

* formatting

* :)

* clangtidy

* lint

* revert some stupid things

* small test adjusmtments

* simplify tests

* rename test

* remove extra line

* freaking style change

* simplify a bit

* fix segfault & remove an extra call

* tightly couple proxy provider with context..

* remove useless include

* rename config prefix parameter

* simplify provider a bit

* organize provider a bit

* add a few comments

* comment out proxy env tests

* fix nullptr in unit tests

* make sure old storage proxy config is properly covered without global context instance

* move a few functions from class to anonymous namespace

* fix no fallback for specific storage conf

* change API to accept http method instead of bool

* implement http/https distinction in listresolver, any still not implemented

* implement http/https distinction in remote resolver

* progress on code, improve tests and add url function working test

* use protcol instead of method for http and https

* small fix

* few more adjustments

* fix style

* black

* move enum to proxyconfiguration

* wip

* fix build

* fix ut

* delete atomicroundrobin class

* remove stale include

* add some tests.. need to spend some more time on the design..

* change design a bit

* progress

* use existing context for tests

* rename aux function and fix ut

* ..

* rename test

* try to simplify tests a bit

* simplify tests a bit more

* attempt to fix tests, accept more than one remote resolver

* use proper log id

* try waiting for resolver

* proper wait logic

* black

* empty

* address a few comments

* refactor tests

* remove old tests

* baclk

* use RAII to set/unset env

* black

* clang tidy

* fix env proxy not respecting any

* use log trace

* fix wrong logic in getRemoteREsolver

* fix wrong logic in getRemoteREsolver

* fix test

* remove unwanted code

* remove ClientConfigurationperRequest and auxilary classes

* remove unwanted code

* remove adapter test

* few adjustments and add test for s3 storage conf  with new proxy settings

* black

* use chassert for context

* Add getenv comment
2023-08-24 16:07:26 +03:00
avogar
894513f6cd Fix tests 2023-08-23 18:43:08 +00:00
Michael Kolupaev
b51064a508 Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead 2023-06-01 18:48:30 -07:00
Michael Kolupaev
fd9c9bb25b Split ReadWriteBufferFromHTTP.h into .h and .cpp file 2023-05-06 21:11:06 -07:00
Michael Kolupaev
bd426a7d6d Unbreak reading from web servers that don't support HEAD requests 2023-04-17 04:58:32 +00:00
Michael Kolupaev
87be78e6de Better 2023-04-17 04:58:32 +00:00
Michael Kolupaev
e133633359 Parallel decoding with one row group per thread 2023-04-17 04:58:32 +00:00
Michael Kolupaev
683077890f Highly questionable refactoring (getInputMultistream() nonsense) 2023-04-17 04:58:32 +00:00
Michael Kolupaev
2d4fe85513 Something 2023-04-17 04:58:32 +00:00
Alexey Milovidov
09ea79aaf7 Add support for {server_uuid} macro 2023-04-09 03:04:26 +02:00
Alexander Tokmakov
9bf4f0108e set content-length for empty POST requests 2023-03-23 18:02:31 +01:00
Antonio Andelic
a70ca31884 Merge branch 'master' into fix-url-progress-bar 2023-03-09 10:17:33 +00:00
Antonio Andelic
9722d8a403 Rename variable 2023-03-02 14:17:48 +00:00
Antonio Andelic
f540f7f6f9 Fix some tests 2023-03-01 12:45:00 +00:00
Antonio Andelic
919122948f More fixes and add tests 2023-02-28 14:22:44 +00:00
Antonio Andelic
25353bab43 Merge branch 'master' into fix-single-thread-parquet 2023-02-28 08:25:19 +00:00
Alexey Milovidov
cde1bdad10 Spelling 2023-02-25 08:45:53 +01:00
Alexey Milovidov
f9faec7ba0 Do not backoff reading from URL in case of too many opened files 2023-02-25 03:21:34 +01:00
Antonio Andelic
c66d5bc687 better 2023-02-24 12:51:36 +00:00
Antonio Andelic
a587bcbb8d Use initial range for fetching file size 2023-02-24 12:21:42 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449)
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
2023-01-24 00:13:58 +03:00
kssenii
30547d2dcd Replace old named collections code for url 2022-12-17 00:24:05 +01:00
kssenii
0a702d9520 Fix 2022-10-10 17:23:09 +02:00
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH}
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
avogar
d8cbd51333 Support relative path in Location header after http redirect 2022-09-09 19:26:14 +00:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache 2022-07-27 11:35:36 +02:00
Kseniia Sumarokova
9abbb35cda
Fix integration test 2022-07-25 23:40:23 +03:00
kssenii
a06fe4e21b Fix 2022-07-15 13:53:20 +02:00
kssenii
03897589b9 Fix 2022-07-14 21:31:17 +02:00
Kruglov Pavel
86e8f31ad4
Merge branch 'master' into schema-inference-cache 2022-06-24 16:10:25 +02:00
avogar
c14364e3d9 Check last modification time for URL function too 2022-06-21 17:18:14 +00:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
kssenii
9d364cdce2 Refactor 2022-04-26 15:33:53 +02:00
Anton Popov
2de6668b3f show names of erroneous files 2022-04-16 00:10:47 +00:00
Antonio Andelic
04e6e6ab5e Add ParallelReadBuffer for S3 2022-03-21 14:52:26 +00:00
Antonio Andelic
0979155f2f Address PR comments 2022-03-16 15:45:17 +00:00