ClickHouse/src/IO
Alexey Milovidov 10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads
Add ability to disable checksums for S3 to avoid excessive input file read
2023-12-05 06:34:21 +01:00
..
Archives Re-enable clang-tidy checks 2023-09-26 09:34:12 +00:00
examples Ditch tons of garbage 2023-08-09 02:19:02 +02:00
Resource fix start time in test 2023-09-19 10:13:05 +00:00
S3 Merge pull request #55559 from azat/s3-fix-excessive-reads 2023-12-05 06:34:21 +01:00
tests Add ability to disable checksums for S3 to avoid excessive input file read 2023-11-26 19:20:19 +01:00
AIO.cpp
AIO.h
AsynchronousReadBufferFromFile.cpp Unify priorities: lower value means higher priority (#50205) 2023-05-26 15:55:30 +02:00
AsynchronousReadBufferFromFile.h Merge branch 'master' into optimize_reading3 2023-10-28 00:04:20 +02:00
AsynchronousReadBufferFromFileDescriptor.cpp better 2023-11-07 22:04:13 +01:00
AsynchronousReadBufferFromFileDescriptor.h use local thread pool reader for cache segments 2023-10-24 14:06:13 +02:00
AsynchronousReader.h Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
AsyncReadCounters.cpp
AsyncReadCounters.h Make async reader work with any impl 2023-05-22 19:54:04 +02:00
BitHelpers.h Minor changes 2023-04-23 05:49:39 +02:00
BoundedReadBuffer.cpp Something 2023-04-17 04:58:32 +00:00
BoundedReadBuffer.h Fix race 2023-06-27 15:54:06 +02:00
BrotliReadBuffer.cpp
BrotliReadBuffer.h
BrotliWriteBuffer.cpp no finalize at d-tor in compression 2023-06-16 16:38:18 +02:00
BrotliWriteBuffer.h test compressed write to S3 2023-06-14 14:28:57 +02:00
BufferBase.h Update BufferBase.h 2022-02-07 17:49:53 +03:00
BufferWithOwnMemory.h finalize tmp_out 2023-11-18 17:30:49 +01:00
Bzip2ReadBuffer.cpp
Bzip2ReadBuffer.h
Bzip2WriteBuffer.cpp no finalize at d-tor in compression 2023-06-16 16:38:18 +02:00
Bzip2WriteBuffer.h
CascadeWriteBuffer.cpp fix finalization of cascade write buffer 2023-06-21 12:27:39 +02:00
CascadeWriteBuffer.h work with comment on the PR 2023-06-20 20:02:04 +02:00
CMakeLists.txt
CompressedReadBufferWrapper.h Unify priorities: lower value means higher priority (#50205) 2023-05-26 15:55:30 +02:00
CompressionMethod.cpp
CompressionMethod.h
ConcatReadBuffer.h
ConcatSeekableReadBuffer.cpp
ConcatSeekableReadBuffer.h
ConnectionTimeouts.cpp Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
ConnectionTimeouts.h Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
copyData.cpp
copyData.h
DoubleConverter.cpp
DoubleConverter.h
EmptyReadBuffer.h
FileEncryptionCommon.cpp Generate safe IVs 2023-06-16 17:09:08 +02:00
FileEncryptionCommon.h Use fingerprints instead of key IDs in encrypted disks (#49882) 2023-05-31 13:11:10 +02:00
ForkWriteBuffer.cpp remove reminder comments 2023-05-30 12:26:20 +02:00
ForkWriteBuffer.h
HadoopSnappyReadBuffer.cpp
HadoopSnappyReadBuffer.h
HashingReadBuffer.h
HashingWriteBuffer.cpp
HashingWriteBuffer.h do abort/cancel/deletion merge/mutate task with exception context 2023-06-16 16:48:21 +02:00
HTTPChunkedReadBuffer.cpp change some exception codes 2023-11-03 20:19:24 +01:00
HTTPChunkedReadBuffer.h
HTTPCommon.cpp Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
HTTPCommon.h Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
HTTPHeaderEntries.h
IReadableWriteBuffer.h MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
IResourceManager.h fix build 2023-09-09 09:26:18 +00:00
ISchedulerConstraint.h fix build 2023-09-03 22:40:00 +00:00
ISchedulerNode.h Merge branch 'io-scheduling-throttler' of github.com:ClickHouse/ClickHouse into io-scheduling-throttler 2023-09-19 10:10:35 +00:00
ISchedulerQueue.h add system.scheduler table for introspection 2023-09-03 21:48:33 +00:00
LimitReadBuffer.cpp
LimitReadBuffer.h
LimitSeekableReadBuffer.cpp
LimitSeekableReadBuffer.h
Lz4DeflatingWriteBuffer.cpp Update src/IO/Lz4DeflatingWriteBuffer.cpp 2023-11-20 14:12:52 +01:00
Lz4DeflatingWriteBuffer.h fix in_data pointer 2023-11-19 00:44:39 +01:00
Lz4InflatingReadBuffer.cpp
Lz4InflatingReadBuffer.h
LZMADeflatingWriteBuffer.cpp Avoid possible memory leaks in case of missing buffer finalization 2023-10-08 08:51:45 +02:00
LZMADeflatingWriteBuffer.h fix build 2023-06-21 22:32:33 +02:00
LZMAInflatingReadBuffer.cpp
LZMAInflatingReadBuffer.h
MemoryReadWriteBuffer.cpp MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
MemoryReadWriteBuffer.h MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
MMappedFile.cpp
MMappedFile.h
MMappedFileCache.h Implement endianness-independent SipHash and MergeTree checksum serialization 2023-07-19 10:01:55 -07:00
MMappedFileDescriptor.cpp
MMappedFileDescriptor.h
MMapReadBufferFromFile.cpp DWARF input format (#55450) 2023-10-16 17:00:07 -07:00
MMapReadBufferFromFile.h DWARF input format (#55450) 2023-10-16 17:00:07 -07:00
MMapReadBufferFromFileDescriptor.cpp Better 2023-06-23 14:03:59 +00:00
MMapReadBufferFromFileDescriptor.h Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead 2023-06-01 18:48:30 -07:00
MMapReadBufferFromFileWithCache.cpp
MMapReadBufferFromFileWithCache.h DWARF input format (#55450) 2023-10-16 17:00:07 -07:00
MySQLBinlogEventReadBuffer.cpp
MySQLBinlogEventReadBuffer.h
MySQLPacketPayloadReadBuffer.cpp MaterializedMySQL: Fix the infinite loop in ReadBuffer::read 2023-08-04 11:34:31 +02:00
MySQLPacketPayloadReadBuffer.h
MySQLPacketPayloadWriteBuffer.cpp
MySQLPacketPayloadWriteBuffer.h
NullWriteBuffer.cpp do call finalize for all buffers 2023-06-16 16:38:18 +02:00
NullWriteBuffer.h do call finalize for all buffers 2023-06-16 16:38:18 +02:00
OpenedFile.cpp
OpenedFile.h Improve performance of file descriptor cache by narrowing mutex scopes (#36682) 2022-04-28 14:56:53 +02:00
OpenedFileCache.h Shard OpenedFileCache to avoid lock contention (#51341) 2023-07-24 15:58:21 +02:00
Operators.h
ParallelReadBuffer.cpp Don't report LOGICAL_ERROR if a file got truncated during read 2023-07-31 15:04:30 +00:00
ParallelReadBuffer.h Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead 2023-06-01 18:48:30 -07:00
parseDateTimeBestEffort.cpp More clang-tidy fixes 2023-09-21 14:40:57 +00:00
parseDateTimeBestEffort.h
PeekableReadBuffer.cpp
PeekableReadBuffer.h Unify priorities: lower value means higher priority (#50205) 2023-05-26 15:55:30 +02:00
PeekableWriteBuffer.cpp Fix wrong query result when http_write_exception_in_output_format=1 2023-10-30 18:09:29 +00:00
PeekableWriteBuffer.h Output valid JSON/XML on excetpion during HTTP query execution 2023-08-01 10:06:56 +00:00
Progress.cpp Review #51946 and partially revert it 2023-10-08 18:00:59 +02:00
Progress.h Review #51946 and partially revert it 2023-10-08 15:17:41 +02:00
ReadBuffer.cpp
ReadBuffer.h Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
ReadBufferFromEmptyFile.h
ReadBufferFromEncryptedFile.cpp
ReadBufferFromEncryptedFile.h
ReadBufferFromFile.cpp
ReadBufferFromFile.h DWARF input format (#55450) 2023-10-16 17:00:07 -07:00
ReadBufferFromFileBase.cpp Use read_bytes/total_bytes_to_read for progress bar in s3/file/url/... table functions 2023-06-22 17:24:43 +00:00
ReadBufferFromFileBase.h DWARF input format (#55450) 2023-10-16 17:00:07 -07:00
ReadBufferFromFileDecorator.cpp
ReadBufferFromFileDecorator.h Make async reader work with any impl 2023-05-22 19:54:04 +02:00
ReadBufferFromFileDescriptor.cpp Rename TaskStatsInfoGetter into NetlinkMetricsProvider 2023-07-21 06:48:18 +02:00
ReadBufferFromFileDescriptor.h Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead 2023-06-01 18:48:30 -07:00
ReadBufferFromIStream.cpp
ReadBufferFromIStream.h Ditch tons of garbage 2023-08-09 02:19:02 +02:00
ReadBufferFromMemory.cpp Fix checksum calculation for backup entries (#50264) 2023-05-27 14:35:37 +02:00
ReadBufferFromMemory.h
ReadBufferFromPocoSocket.cpp Fix bugs in Poco, use true non-blocking IO 2023-05-26 23:11:57 +00:00
ReadBufferFromPocoSocket.h Fix bugs in Poco, use true non-blocking IO 2023-05-26 23:11:57 +00:00
ReadBufferFromS3.cpp Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
ReadBufferFromS3.h Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
ReadBufferFromString.h fix tests 2023-08-15 12:16:22 +08:00
readDecimalText.h
readFloatText.cpp update as per reviews 2023-07-31 16:26:18 +02:00
readFloatText.h Update readFloatText.h 2023-10-27 11:42:33 +08:00
ReadHelpers.cpp ci fix 2023-11-07 09:45:46 +08:00
ReadHelpers.h Fix Date text parsing in optimistic path 2023-11-14 18:58:00 +00:00
ReadHelpersArena.h
ReadSettings.h Review fix 2023-11-16 17:47:51 +01:00
ReadWriteBufferFromHTTP.cpp Add ClickHouse setting to disable tunneling for HTTPS requests over HTTP proxy (#55033) 2023-11-04 13:47:52 -04:00
ReadWriteBufferFromHTTP.h Fix build 2023-11-13 10:09:23 +01:00
ResourceBudget.h add system.scheduler table for introspection 2023-09-03 21:48:33 +00:00
ResourceGuard.h
ResourceLink.h
ResourceManagerFactory.h IO scheduling subsystem 2022-09-27 13:26:41 +00:00
ResourceRequest.h
S3Common.cpp
S3Common.h
SchedulerNodeFactory.h
SchedulerRoot.h fix typos 2023-09-14 09:21:23 +00:00
SeekableReadBuffer.cpp Fix data race in copyFromIStreamWithProgressCallback 2023-09-11 21:04:37 +00:00
SeekableReadBuffer.h better 2023-11-07 22:04:13 +01:00
SharedThreadPools.cpp Fix style 2023-11-18 20:19:24 +01:00
SharedThreadPools.h Add metrics for the number of queued jobs, which is useful for the IO thread pool 2023-11-18 19:07:59 +01:00
SnappyReadBuffer.cpp
SnappyReadBuffer.h
SnappyWriteBuffer.cpp remove reminder comments 2023-05-30 12:26:20 +02:00
SnappyWriteBuffer.h
StdIStreamFromMemory.cpp work with comments on PR 2023-05-15 14:41:51 +02:00
StdIStreamFromMemory.h work with comments on PR 2023-05-15 14:41:51 +02:00
StdStreamBufFromReadBuffer.cpp Fix using std::ios_base::end in StdStreamFromReadBuffer::seekg(). 2023-01-17 17:56:14 +01:00
StdStreamBufFromReadBuffer.h
StdStreamFromReadBuffer.h Ditch tons of garbage 2023-08-09 02:19:02 +02:00
SwapHelper.cpp
SwapHelper.h
SynchronousReader.cpp set ignore metric in all readers 2023-10-20 22:24:56 +02:00
SynchronousReader.h Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
TimeoutSetter.cpp Change default value of handshake_timeout to 10 sec, fix possible use-after-free 2023-05-23 11:39:40 +00:00
TimeoutSetter.h Change default value of handshake_timeout to 10 sec, fix possible use-after-free 2023-05-23 11:39:40 +00:00
UncompressedCache.h Better parameter name 2023-08-22 15:43:13 +00:00
UseSSL.cpp
UseSSL.h
VarInt.cpp Allow var-int encoded 64-bit integers with MSB=1 2023-07-06 20:23:23 +00:00
VarInt.h Ditch tons of garbage 2023-08-09 02:19:02 +02:00
WithFileName.cpp Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead 2023-06-01 18:48:30 -07:00
WithFileName.h
WithFileSize.cpp Improve progress bar for file/s3/hdfs/url table functions. Step 1 2023-06-16 15:51:18 +00:00
WithFileSize.h Improve progress bar for file/s3/hdfs/url table functions. Step 1 2023-06-16 15:51:18 +00:00
WriteBuffer.cpp do abort/cancel/deletion merge/mutate task with exception context 2023-06-16 16:48:21 +02:00
WriteBuffer.h add comments 2023-11-20 12:13:24 +01:00
WriteBufferDecorator.h
WriteBufferFromArena.h fix 02428_combinators_with_over_statement 2023-06-16 16:38:18 +02:00
WriteBufferFromEncryptedFile.cpp remove reminder comments 2023-05-30 12:26:20 +02:00
WriteBufferFromEncryptedFile.h
WriteBufferFromFile.cpp fix build and tests 2023-06-16 19:52:52 +02:00
WriteBufferFromFile.h
WriteBufferFromFileBase.cpp
WriteBufferFromFileBase.h do abort/cancel/deletion merge/mutate task with exception context 2023-06-16 16:48:21 +02:00
WriteBufferFromFileDecorator.cpp remove reminder comments 2023-05-30 12:26:20 +02:00
WriteBufferFromFileDecorator.h
WriteBufferFromFileDescriptor.cpp remove reminder comments 2023-05-30 12:26:20 +02:00
WriteBufferFromFileDescriptor.h
WriteBufferFromFileDescriptorDiscardOnFailure.cpp
WriteBufferFromFileDescriptorDiscardOnFailure.h
WriteBufferFromHTTP.cpp Add ClickHouse setting to disable tunneling for HTTPS requests over HTTP proxy (#55033) 2023-11-04 13:47:52 -04:00
WriteBufferFromHTTP.h Add ClickHouse setting to disable tunneling for HTTPS requests over HTTP proxy (#55033) 2023-11-04 13:47:52 -04:00
WriteBufferFromOStream.cpp no finalize in d-tor WriteBufferFromOStream 2023-06-26 12:02:09 +02:00
WriteBufferFromOStream.h Ditch tons of garbage 2023-08-09 02:19:02 +02:00
WriteBufferFromPocoSocket.cpp Merge branch 'master' into fix-secure-async-read-write 2023-06-01 16:24:54 +02:00
WriteBufferFromPocoSocket.h Fix bugs in Poco, use true non-blocking IO 2023-05-26 23:11:57 +00:00
WriteBufferFromS3.cpp Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
WriteBufferFromS3.h Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
WriteBufferFromS3BufferAllocationPolicy.cpp work with comments on PR 2023-05-15 14:41:51 +02:00
WriteBufferFromS3TaskTracker.cpp introduce LogSeriesLimiterPtrt 2023-06-23 23:06:32 +02:00
WriteBufferFromS3TaskTracker.h introduce LogSeriesLimiterPtrt 2023-06-23 23:06:32 +02:00
WriteBufferFromString.h
WriteBufferFromVector.h use old_size 2023-06-05 08:43:55 +08:00
WriteBufferValidUTF8.cpp AARCH64 Neon memequal wide 2023-08-20 12:45:18 +03:00
WriteBufferValidUTF8.h
WriteHelpers.cpp Add setting to print pretty deep nested Tuples 2023-09-28 22:17:44 +00:00
WriteHelpers.h Remove useless header files 2023-11-06 03:49:55 +01:00
WriteIntText.h
WriteSettings.h Delete feature 2023-06-21 20:48:12 +02:00
ZlibDeflatingWriteBuffer.cpp Avoid possible memory leaks in case of missing buffer finalization 2023-10-08 08:51:45 +02:00
ZlibDeflatingWriteBuffer.h fix build 2023-06-21 22:32:33 +02:00
ZlibInflatingReadBuffer.cpp
ZlibInflatingReadBuffer.h
ZstdDeflatingAppendableWriteBuffer.cpp Revert changes from https://github.com/ClickHouse/ClickHouse/pull/53064 2023-08-07 12:21:50 +00:00
ZstdDeflatingAppendableWriteBuffer.h Revert changes from https://github.com/ClickHouse/ClickHouse/pull/53064 2023-08-07 12:21:50 +00:00
ZstdDeflatingWriteBuffer.cpp Avoid possible memory leaks in case of missing buffer finalization 2023-10-08 08:51:45 +02:00
ZstdDeflatingWriteBuffer.h Fix ZstdDeflatingWriteBuffer truncating the output sometimes 2023-08-04 20:47:45 +00:00
ZstdInflatingReadBuffer.cpp
ZstdInflatingReadBuffer.h