Commit Graph

105 Commits

Author SHA1 Message Date
Vitaly Baranov
aea9ccdb60 Pass request settings to S3::getObjectInfo(). 2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed Move getObjectInfo() to a separate header. 2023-01-27 15:09:38 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449)
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
2023-01-24 00:13:58 +03:00
Alexander Gololobov
91fb1eab8c
Merge pull request #45373 from ClickHouse/throttler-metrics
Add detailed profile events for throttling
2023-01-22 11:20:45 +01:00
serxa
ce7e22b87b add detailed profile events for throttling 2023-01-17 18:29:24 +00:00
Vitaly Baranov
a955504043 Move throw_on_error parameter to the end. 2023-01-15 20:28:16 +01:00
Anton Popov
0c87031e80 Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-13 16:33:21 +00:00
Anton Popov
65a78bcd91 improve performance of storage S3 2022-11-26 15:24:01 +00:00
kssenii
5e01441f61 Show progress bar while reading from s3 table function 2022-11-21 17:56:02 +01:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling"" 2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling" 2022-11-16 22:33:48 +03:00
serxa
6d5d9ff421 rename ReadWriteSettings -> RequestSettings 2022-11-08 13:48:23 +00:00
serxa
2daec0b45e S3 request per second rate throttling + refactoring 2022-11-07 18:05:40 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
serxa
2ef696ffe1 fix issues 2022-09-19 18:40:32 +00:00
serxa
7a1d4a404a add DiskS3* profile events per S3 API calls 2022-09-19 18:10:47 +00:00
serxa
f8aa738511 more conventional profile events names 2022-09-19 17:23:22 +00:00
Sergei Trifonov
ff2e1ff8d8
Merge branch 'master' into support_s3_api_event_metrics 2022-09-19 14:35:45 +02:00
alesapin
ad279940d6 Trying print less scary messages in case No such key for outdated parts 2022-09-14 17:25:58 +02:00
mateng0915
f7f976e94e remove extra space 2022-09-14 19:19:47 +08:00
mateng0915
5badb1b186 resolve the review comments 2022-09-14 19:19:40 +08:00
teng.ma
a85c917488 fix master conflict 2022-09-14 19:18:31 +08:00
teng.ma
ebea9f9359 Feature: Support Event Tracing when calling Aws S3 API
Description:
============
In  Computing && Storage Architecture,  using S3 as remote / Shared storage, the Method to access S3 is using AWS S3 API
There is a gap between ClickHouse DB with Ozone Operation
In ClickHouse, operation is SQL and background task
In S3 , the operation should be AWS S3 API
And one sql maybe can mapped to multiple  API

Solution:
=========
Added Calling API as event into system.events table
2022-09-14 19:18:31 +08:00
alesapin
6da37b92e1 Improve logging one time 2022-09-08 13:11:53 +02:00
alesapin
09e97a6381 Fix style 2022-09-06 18:38:34 +02:00
alesapin
ceed9f418b Return better errors handling 2022-09-06 18:22:44 +02:00
alesapin
6ea7f1e011 Better exception handling for ReadBufferFromS3 2022-09-06 13:59:55 +02:00
kssenii
3134bdeb13 Fix 2022-09-01 16:35:28 +02:00
Sergei Trifonov
b2317ff710 renaming to be consistent 2022-07-14 17:33:22 +02:00
Sergei Trifonov
f854507729 merge master and resolve conflicts 2022-07-13 19:07:28 +02:00
Sergei Trifonov
60c6a5b23c fix s3 read/write throttling instant 2022-07-13 18:36:08 +02:00
Sergei Trifonov
43779ec280 add max_remote_{read,write}_network_bandwidth_for_server settings 2022-07-11 14:59:39 +02:00
kssenii
bf1c99daab Refactoring of object storages, add LocalObjectStorage 2022-07-09 12:39:31 +02:00
kssenii
a772a09149 Fix 2022-07-04 00:14:45 +02:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
Azat Khuzhin
71285edfbd Fix reading of empty S3 files
It is possible for ReadBufferFromS3::nextImpl() called even after eof(),
at least once, and in this case, if the file was empty, then local
working_buffer will be null, while impl.working_buffer will be empty,
but not null, and so local position() after impl->position() =
position() will be incorrect.

I found this with test_storage_s3/test.py::test_empty_file in debug
build, assertion catched this, so maybe it worth get back debug
integration build...

v2: fix test_log_family_s3 failures
    https://s3.amazonaws.com/clickhouse-test-reports/37801/b5e6e2ddae94d6a7eac551309cb67003dff97df1/integration_tests__asan__actions__[2/3].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-07 20:41:26 +03:00
alesapin
5dcd25be23 Initial implementation 2022-05-12 00:04:54 +02:00
Alexey Milovidov
ef9633499c Merge branch 'master' into fix-trash-s3-metrics 2022-05-02 03:55:10 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
Sergei Trifonov
11f40376ae
Merge pull request #34571 from surahman/CH-31221-AWS-S3-object-version-id
[CH-31221] AWS S3 Object Version Specification
2022-04-28 15:45:33 +02:00
kssenii
9d364cdce2 Refactor 2022-04-26 15:33:53 +02:00
Alexey Milovidov
6de4f2bbff Everything related to S3 metrics was a trash 2022-04-24 03:42:18 +02:00
Sergei Trifonov
b371251e80 Merge branch 'master' into surahman-CH-31221-AWS-S3-object-version-id 2022-04-22 08:39:02 +02:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
2022-04-20 10:29:05 +02:00
Sergei Trifonov
01425df758 fix build 2022-04-07 16:07:12 +02:00
Sergei Trifonov
9c2804b924 merge from master + resolve conflicts 2022-04-07 11:22:01 +02:00
kssenii
6c6fb5c3e8 Fix race in cached buffer 2022-04-04 16:32:37 +02:00
Antonio Andelic
d9d826c813 Address PR review 2022-03-28 08:19:23 +00:00
Antonio Andelic
62a495a5fc Small refactor 2022-03-24 09:45:30 +00:00
Antonio Andelic
f32ef2a556 Small polishing for S3 reader 2022-03-23 08:15:18 +00:00