serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
serxa
2ef696ffe1
fix issues
2022-09-19 18:40:32 +00:00
serxa
7a1d4a404a
add DiskS3* profile events per S3 API calls
2022-09-19 18:10:47 +00:00
serxa
f8aa738511
more conventional profile events names
2022-09-19 17:23:22 +00:00
Sergei Trifonov
ff2e1ff8d8
Merge branch 'master' into support_s3_api_event_metrics
2022-09-19 14:35:45 +02:00
alesapin
ad279940d6
Trying print less scary messages in case No such key for outdated parts
2022-09-14 17:25:58 +02:00
mateng0915
f7f976e94e
remove extra space
2022-09-14 19:19:47 +08:00
mateng0915
5badb1b186
resolve the review comments
2022-09-14 19:19:40 +08:00
teng.ma
a85c917488
fix master conflict
2022-09-14 19:18:31 +08:00
teng.ma
ebea9f9359
Feature: Support Event Tracing when calling Aws S3 API
...
Description:
============
In Computing && Storage Architecture, using S3 as remote / Shared storage, the Method to access S3 is using AWS S3 API
There is a gap between ClickHouse DB with Ozone Operation
In ClickHouse, operation is SQL and background task
In S3 , the operation should be AWS S3 API
And one sql maybe can mapped to multiple API
Solution:
=========
Added Calling API as event into system.events table
2022-09-14 19:18:31 +08:00
alesapin
6da37b92e1
Improve logging one time
2022-09-08 13:11:53 +02:00
alesapin
09e97a6381
Fix style
2022-09-06 18:38:34 +02:00
alesapin
ceed9f418b
Return better errors handling
2022-09-06 18:22:44 +02:00
alesapin
6ea7f1e011
Better exception handling for ReadBufferFromS3
2022-09-06 13:59:55 +02:00
kssenii
3134bdeb13
Fix
2022-09-01 16:35:28 +02:00
Sergei Trifonov
b2317ff710
renaming to be consistent
2022-07-14 17:33:22 +02:00
Sergei Trifonov
f854507729
merge master and resolve conflicts
2022-07-13 19:07:28 +02:00
Sergei Trifonov
60c6a5b23c
fix s3 read/write throttling instant
2022-07-13 18:36:08 +02:00
Sergei Trifonov
43779ec280
add max_remote_{read,write}_network_bandwidth_for_server settings
2022-07-11 14:59:39 +02:00
kssenii
bf1c99daab
Refactoring of object storages, add LocalObjectStorage
2022-07-09 12:39:31 +02:00
kssenii
a772a09149
Fix
2022-07-04 00:14:45 +02:00
kssenii
5dd1bb2fd8
improvements for getFileSize
2022-06-20 15:22:56 +02:00
Azat Khuzhin
71285edfbd
Fix reading of empty S3 files
...
It is possible for ReadBufferFromS3::nextImpl() called even after eof(),
at least once, and in this case, if the file was empty, then local
working_buffer will be null, while impl.working_buffer will be empty,
but not null, and so local position() after impl->position() =
position() will be incorrect.
I found this with test_storage_s3/test.py::test_empty_file in debug
build, assertion catched this, so maybe it worth get back debug
integration build...
v2: fix test_log_family_s3 failures
https://s3.amazonaws.com/clickhouse-test-reports/37801/b5e6e2ddae94d6a7eac551309cb67003dff97df1/integration_tests__asan__actions__[2/3].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-07 20:41:26 +03:00
alesapin
5dcd25be23
Initial implementation
2022-05-12 00:04:54 +02:00
Alexey Milovidov
ef9633499c
Merge branch 'master' into fix-trash-s3-metrics
2022-05-02 03:55:10 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common
2022-04-29 10:26:35 +08:00
Sergei Trifonov
11f40376ae
Merge pull request #34571 from surahman/CH-31221-AWS-S3-object-version-id
...
[CH-31221] AWS S3 Object Version Specification
2022-04-28 15:45:33 +02:00
kssenii
9d364cdce2
Refactor
2022-04-26 15:33:53 +02:00
Alexey Milovidov
6de4f2bbff
Everything related to S3 metrics was a trash
2022-04-24 03:42:18 +02:00
Sergei Trifonov
b371251e80
Merge branch 'master' into surahman-CH-31221-AWS-S3-object-version-id
2022-04-22 08:39:02 +02:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings
...
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
2022-04-20 10:29:05 +02:00
Sergei Trifonov
01425df758
fix build
2022-04-07 16:07:12 +02:00
Sergei Trifonov
9c2804b924
merge from master + resolve conflicts
2022-04-07 11:22:01 +02:00
kssenii
6c6fb5c3e8
Fix race in cached buffer
2022-04-04 16:32:37 +02:00
Antonio Andelic
d9d826c813
Address PR review
2022-03-28 08:19:23 +00:00
Antonio Andelic
62a495a5fc
Small refactor
2022-03-24 09:45:30 +00:00
Antonio Andelic
f32ef2a556
Small polishing for S3 reader
2022-03-23 08:15:18 +00:00
Saad Ur Rahman
c7402c2a39
[IO] Version ID provided during buffer read
...
The Version ID, if provided, is to be set in the request to read the buffer from S3. If not provided the latest version of the object must be retrieved.
Version ID set to last parameter to allow defaulting and backward compatibility with calls.
2022-03-14 20:10:14 -04:00
kssenii
f3cba6f270
Remove already fixed crutch
2022-03-08 14:30:21 +01:00
kssenii
2dcfe66236
Fix race
2022-03-06 20:33:07 +01:00
kssenii
fad7c042c1
Fix cache with non-async reads
2022-02-21 19:50:27 +01:00
kssenii
e8b2e47e93
Style
2022-02-19 20:38:23 +01:00
kssenii
acf9a092a9
Review fixes
2022-02-18 19:09:54 +01:00
kssenii
47f94120da
Merge master
2022-02-16 14:43:28 +01:00
Anton Popov
1b16db72c3
fix consecutive backward seeks in seekable read buffers
2022-02-07 17:20:26 +03:00
kssenii
aeec749173
Fix predownload case part 1
2022-02-02 17:25:25 +03:00
kssenii
5cba11428a
Fixes
2022-02-02 00:25:16 +03:00
kssenii
6b03c577d6
Make downloader's term resticted to one nextImpl
2022-01-29 23:58:53 +03:00
kssenii
c91b86e220
Better
2022-01-26 21:56:08 +03:00
kssenii
a566099759
Local cache for remote filesystem
2022-01-19 00:42:11 +03:00
kssenii
d8ed4db6d4
Fix
2021-12-22 11:42:23 +03:00
Nikolai Kochetov
7dea7b7f76
Decrease log level for some s3 messages.
2021-12-20 18:18:54 +03:00
kssenii
f18dcd2287
Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers
2021-11-13 14:38:57 +03:00
kssenii
618c332f9c
Rename read_threadpool to threadpool
2021-11-10 07:46:18 +00:00
kssenii
45ea820297
Reduce memory usage for some formats
2021-11-03 14:30:03 +03:00
Kseniia Sumarokova
b49797ad51
Update ReadBufferFromS3.cpp
2021-10-29 15:54:40 +03:00
kssenii
4f3433b4e1
Some clean up before merging
2021-10-29 01:04:52 +03:00
kssenii
e4e157688d
Review fixes
2021-10-21 01:57:43 +03:00
kssenii
edfb1a5521
Better performance for smaller number of threads
2021-10-15 15:07:39 +03:00
kssenii
c1c574e9ca
Better
2021-10-15 13:14:57 +03:00
kssenii
3995506d37
Adjust range reader for remote fs reads
2021-10-15 11:36:26 +03:00
kssenii
448d63a0fc
Range read with upper bound offset
2021-10-11 01:53:51 +03:00
kssenii
d6e1d0503d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-07 16:21:00 +03:00
Alexey Milovidov
fe6b7c77c7
Rename "common" to "base"
2021-10-02 10:13:14 +03:00
kssenii
55972db866
Small fix and more comments
2021-10-01 11:38:58 +03:00
kssenii
e4bc81ac78
Better
2021-09-30 21:45:57 +03:00
kssenii
cc9ce13bac
Finally all fixed
2021-09-30 15:35:59 +03:00
kssenii
ebca30694d
Fix
2021-09-27 18:25:30 +03:00
kssenii
6219d541a5
Finally fixed
2021-09-24 15:29:04 +03:00
kssenii
c1ea44b9c2
Fixes
2021-09-24 13:38:08 +03:00
Alexey Milovidov
c03dc92dd1
Fix excessive logging in ReadBufferFromS3
2021-08-25 03:28:15 +03:00
Vladimir Chebotarev
21d4d26f20
Final fix ReadBufferFromS3.
2021-08-23 05:05:49 +00:00
Vitaly Baranov
ec7f2d74b1
Fix ReadBufferFromS3
2021-07-12 17:36:36 +03:00
Pavel Kovalenko
ce8e2f7f28
Avoid double initializing ReadBufferFromS3
2021-06-17 22:31:50 +03:00
Pavel Kovalenko
47252e07f3
Remove trailing whitespaces
2021-06-12 15:57:14 +03:00
Pavel Kovalenko
0c82d45dea
Missed initialize in ReadBufferFromS3
2021-06-12 14:35:34 +03:00
mergify[bot]
bfbc787b08
Merge branch 'master' into disk-s3-read-error-fix
2021-06-09 06:56:19 +00:00
Pavel Kovalenko
6ba40c475a
Code cleanup + comments.
2021-06-07 13:49:34 +03:00
Nikita Mikhaylov
8076848d62
Update ReadBufferFromS3.cpp
2021-06-04 02:46:21 +03:00
Pavel Kovalenko
2bd12a7612
Fix DiskS3 read error due to broken getPosition() method in ReadBufferFromS3.cpp
2021-06-02 18:03:25 +03:00
Nikita Mikhaylov
f33b70c54a
better
2021-05-20 00:42:25 +03:00
Vladimir Chebotarev
fa7308d2cb
Minor fix.
2021-05-17 10:38:34 +03:00
Vladimir Chebotarev
bdf03387ab
Better S3 read retries. Renamed s3_max_single_read_retries -> s3_single_read_retry_attempts, added sleeps before next attempts.
2021-05-17 10:38:32 +03:00
Vladimir Chebotarev
9d4295f261
Removed infinite retries.
2021-04-19 11:02:36 +03:00
Vladimir Chebotarev
817c5cec25
Correct S3 read retries.
2021-04-15 09:47:22 +03:00
Vladimir Chebotarev
cf903d526c
Better logging.
2021-04-15 09:47:22 +03:00
Vladimir Chebotarev
10bf56e540
Attempt to add retries to S3 reads.
2021-04-15 09:47:19 +03:00
Anton Ivashkin
fbd80e8d67
Fix codestyle
2020-07-15 14:15:12 +03:00
Anton Ivashkin
26fca949bd
add profiling events from S3 request to system.events and system.query_log
2020-07-13 11:27:11 +03:00
Alexey Milovidov
7e1813825b
Return old names of macros
2020-05-24 01:24:01 +03:00
Alexey Milovidov
57f6dc66d3
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" \+ [^+]+ \+ "[^"]+" \+ [^+]+ \+ "[^"]+" \+ [^+]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" \+ ([^+]+) \+ "([^"]+)" \+ ([^+]+) \+ "([^"]+)" \+ ([^+]+)\);/\1_FORMATTED(\2, "\3{}\5{}\7{}", \4, \6, \8);/'
2020-05-23 20:11:09 +03:00
Ivan Lezhankin
06446b4f08
dbms/ → src/
2020-04-03 18:14:31 +03:00
