Commit Graph

17 Commits

Author SHA1 Message Date
Nikita Taranov
a86bda5208 use local thread pool reader for cache segments 2023-10-24 14:06:13 +02:00
Sergei Trifonov
0d1f2e297b
Unify priorities: lower value means higher priority (#50205) 2023-05-26 15:55:30 +02:00
Azat Khuzhin
a25dd1d348 Add ability to throttle local IO on per-query/server basis
Server settings:
- max_local_read_bandwidth_for_server
- max_local_write_bandwidth_for_server

Query settings:
- max_local_read_bandwidth
- max_local_write_bandwidth

This is the preparation for adding ability to throttle BACKUPs

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-05 09:39:15 +02:00
kssenii
b0b865c32e Resubmit prefetches 2023-02-08 21:26:24 +01:00
kssenii
30726721ad Fix threadpool reader (for local fs) 2022-09-23 19:35:16 +02:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
taiyang-li
b36d9f8143 refactor readinto 2022-06-06 12:58:22 +08:00
Amos Bird
01652d9892
Add more assertion 2022-02-21 22:42:43 +08:00
Amos Bird
84d111cde9
Better ReadBuffer asynchronicity 2022-02-20 01:08:21 +08:00
Azat Khuzhin
71a99ab27c Accept real file size in createReadBufferFromFileBase()
Right now streams relies on correct file size not the number of bytes
that will be read from the stream, to overcome one bug in the linux
kernel that may return EIINVAL for pread() with offset pass the EOF.

v2: Swap read_hint and file_size (since it is easy to miss something)

Before the first argument to readFile()/createReadBufferFromFileBase()
was read_hint, not the file_size, and let's preserve the order, since
it is easy to miss something

This will also fix 02051_read_settings test automatically because now
MergeTreeReaderStream will pass estimated_sum_mark_range_bytes to
read_hint not file_size, previously it cause on of the following errors:
- Attempt to read after EOF w/ O_DIRECT
- and LOGICAL_ERROR while adjusting granulas w/o O_DIRECT

This will also improve zero-length reads guard (via
ReadBufferFromEmptyFile), that had been added in #30190

v3: fix for other storages that wasn't enabled in fast-test
v4: ignore ENOENT/ENOTSUP in readFile
2022-01-04 10:53:17 +03:00
Azat Khuzhin
e548dae8d9 Pass size of the file to the readers
v2: s/size/file_size/g
2022-01-04 10:53:17 +03:00
Azat Khuzhin
34810456e1 Fix alignment for prefetch in AsynchronousReadBufferFromFileDescriptor 2021-10-17 23:54:57 +03:00
Alexey Milovidov
5243315227 Improve performance in case without prefetch by avoiding swapping of memory buffers 2021-08-25 03:13:05 +03:00
Alexey Milovidov
d184b79bba Progress on async reads. 2021-08-16 03:00:32 +03:00
Alexey Milovidov
91173b8934 Improve performance 2021-08-04 03:07:04 +03:00
Alexey Milovidov
46723a2c3c Fix error 2021-07-28 02:47:28 +03:00
Alexey Milovidov
5c3f5cb35a Experiment with asynchronous readers 2021-07-26 03:34:36 +03:00