Commit Graph

91 Commits

Author SHA1 Message Date
serxa
2ef696ffe1 fix issues 2022-09-19 18:40:32 +00:00
serxa
7a1d4a404a add DiskS3* profile events per S3 API calls 2022-09-19 18:10:47 +00:00
serxa
f8aa738511 more conventional profile events names 2022-09-19 17:23:22 +00:00
Sergei Trifonov
ff2e1ff8d8
Merge branch 'master' into support_s3_api_event_metrics 2022-09-19 14:35:45 +02:00
alesapin
ad279940d6 Trying print less scary messages in case No such key for outdated parts 2022-09-14 17:25:58 +02:00
mateng0915
f7f976e94e remove extra space 2022-09-14 19:19:47 +08:00
mateng0915
5badb1b186 resolve the review comments 2022-09-14 19:19:40 +08:00
teng.ma
a85c917488 fix master conflict 2022-09-14 19:18:31 +08:00
teng.ma
ebea9f9359 Feature: Support Event Tracing when calling Aws S3 API
Description:
============
In  Computing && Storage Architecture,  using S3 as remote / Shared storage, the Method to access S3 is using AWS S3 API
There is a gap between ClickHouse DB with Ozone Operation
In ClickHouse, operation is SQL and background task
In S3 , the operation should be AWS S3 API
And one sql maybe can mapped to multiple  API

Solution:
=========
Added Calling API as event into system.events table
2022-09-14 19:18:31 +08:00
alesapin
6da37b92e1 Improve logging one time 2022-09-08 13:11:53 +02:00
alesapin
09e97a6381 Fix style 2022-09-06 18:38:34 +02:00
alesapin
ceed9f418b Return better errors handling 2022-09-06 18:22:44 +02:00
alesapin
6ea7f1e011 Better exception handling for ReadBufferFromS3 2022-09-06 13:59:55 +02:00
kssenii
3134bdeb13 Fix 2022-09-01 16:35:28 +02:00
Sergei Trifonov
b2317ff710 renaming to be consistent 2022-07-14 17:33:22 +02:00
Sergei Trifonov
f854507729 merge master and resolve conflicts 2022-07-13 19:07:28 +02:00
Sergei Trifonov
60c6a5b23c fix s3 read/write throttling instant 2022-07-13 18:36:08 +02:00
Sergei Trifonov
43779ec280 add max_remote_{read,write}_network_bandwidth_for_server settings 2022-07-11 14:59:39 +02:00
kssenii
bf1c99daab Refactoring of object storages, add LocalObjectStorage 2022-07-09 12:39:31 +02:00
kssenii
a772a09149 Fix 2022-07-04 00:14:45 +02:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
Azat Khuzhin
71285edfbd Fix reading of empty S3 files
It is possible for ReadBufferFromS3::nextImpl() called even after eof(),
at least once, and in this case, if the file was empty, then local
working_buffer will be null, while impl.working_buffer will be empty,
but not null, and so local position() after impl->position() =
position() will be incorrect.

I found this with test_storage_s3/test.py::test_empty_file in debug
build, assertion catched this, so maybe it worth get back debug
integration build...

v2: fix test_log_family_s3 failures
    https://s3.amazonaws.com/clickhouse-test-reports/37801/b5e6e2ddae94d6a7eac551309cb67003dff97df1/integration_tests__asan__actions__[2/3].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-07 20:41:26 +03:00
alesapin
5dcd25be23 Initial implementation 2022-05-12 00:04:54 +02:00
Alexey Milovidov
ef9633499c Merge branch 'master' into fix-trash-s3-metrics 2022-05-02 03:55:10 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
Sergei Trifonov
11f40376ae
Merge pull request #34571 from surahman/CH-31221-AWS-S3-object-version-id
[CH-31221] AWS S3 Object Version Specification
2022-04-28 15:45:33 +02:00
kssenii
9d364cdce2 Refactor 2022-04-26 15:33:53 +02:00
Alexey Milovidov
6de4f2bbff Everything related to S3 metrics was a trash 2022-04-24 03:42:18 +02:00
Sergei Trifonov
b371251e80 Merge branch 'master' into surahman-CH-31221-AWS-S3-object-version-id 2022-04-22 08:39:02 +02:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
2022-04-20 10:29:05 +02:00
Sergei Trifonov
01425df758 fix build 2022-04-07 16:07:12 +02:00
Sergei Trifonov
9c2804b924 merge from master + resolve conflicts 2022-04-07 11:22:01 +02:00
kssenii
6c6fb5c3e8 Fix race in cached buffer 2022-04-04 16:32:37 +02:00
Antonio Andelic
d9d826c813 Address PR review 2022-03-28 08:19:23 +00:00
Antonio Andelic
62a495a5fc Small refactor 2022-03-24 09:45:30 +00:00
Antonio Andelic
f32ef2a556 Small polishing for S3 reader 2022-03-23 08:15:18 +00:00
Saad Ur Rahman
c7402c2a39
[IO] Version ID provided during buffer read
The Version ID, if provided, is to be set in the request to read the buffer from S3. If not provided the latest version of the object must be retrieved.

Version ID set to last parameter to allow defaulting and backward compatibility with calls.
2022-03-14 20:10:14 -04:00
kssenii
f3cba6f270 Remove already fixed crutch 2022-03-08 14:30:21 +01:00
kssenii
2dcfe66236 Fix race 2022-03-06 20:33:07 +01:00
kssenii
fad7c042c1 Fix cache with non-async reads 2022-02-21 19:50:27 +01:00
kssenii
e8b2e47e93 Style 2022-02-19 20:38:23 +01:00
kssenii
acf9a092a9 Review fixes 2022-02-18 19:09:54 +01:00
kssenii
47f94120da Merge master 2022-02-16 14:43:28 +01:00
Anton Popov
1b16db72c3 fix consecutive backward seeks in seekable read buffers 2022-02-07 17:20:26 +03:00
kssenii
aeec749173 Fix predownload case part 1 2022-02-02 17:25:25 +03:00
kssenii
5cba11428a Fixes 2022-02-02 00:25:16 +03:00
kssenii
6b03c577d6 Make downloader's term resticted to one nextImpl 2022-01-29 23:58:53 +03:00
kssenii
c91b86e220 Better 2022-01-26 21:56:08 +03:00
kssenii
a566099759 Local cache for remote filesystem 2022-01-19 00:42:11 +03:00
kssenii
d8ed4db6d4 Fix 2021-12-22 11:42:23 +03:00