ClickHouse/src/IO
Antonio Andelic 3e6314675c
Merge pull request #49930 from AVMusorin/write-buffer-from-s3
Fix metrics `WriteBufferFromS3Bytes`, `WriteBufferFromS3Microseconds` and `WriteBufferFromS3RequestsErrors`
2023-05-23 12:26:05 +02:00
..
Archives Preparation for libcxx(abi), llvm, clang-tidy 16 2023-03-19 17:25:41 +00:00
examples Merge pull request #48628 from ClickHouse/rs/var_uint_max 2023-04-13 10:17:55 +02:00
Resource Geo types are production ready 2023-05-19 12:43:55 +02:00
S3 Merge pull request #49930 from AVMusorin/write-buffer-from-s3 2023-05-23 12:26:05 +02:00
tests Fix build 2023-05-20 13:31:27 +02:00
AIO.cpp Fix -Wshorten-64-to-32 on FreeBSD 2022-12-12 08:05:28 +01:00
AIO.h Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
AsynchronousReadBufferFromFile.cpp Check return value of ::close() 2023-02-07 11:28:22 +01:00
AsynchronousReadBufferFromFile.h Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
AsynchronousReadBufferFromFileDescriptor.cpp Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
AsynchronousReadBufferFromFileDescriptor.h Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
AsynchronousReader.h Resubmit prefetches 2023-02-08 21:26:24 +01:00
AsyncReadCounters.cpp Resubmit prefetches 2023-02-08 21:26:24 +01:00
AsyncReadCounters.h Resubmit prefetches 2023-02-08 21:26:24 +01:00
BitHelpers.h Minor changes 2023-04-23 05:49:39 +02:00
BoundedReadBuffer.cpp Something 2023-04-17 04:58:32 +00:00
BoundedReadBuffer.h Something 2023-04-17 04:58:32 +00:00
BrotliReadBuffer.cpp Add file name to exception raised during decompression 2023-04-27 02:52:23 +00:00
BrotliReadBuffer.h show names of erroneous files 2022-04-16 00:10:47 +00:00
BrotliWriteBuffer.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
BrotliWriteBuffer.h Fix comments 2021-11-22 14:19:26 +03:00
BufferBase.h Update BufferBase.h 2022-02-07 17:49:53 +03:00
BufferWithOwnMemory.h Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
Bzip2ReadBuffer.cpp Add file name to exception raised during decompression 2023-04-27 02:52:23 +00:00
Bzip2ReadBuffer.h show names of erroneous files 2022-04-16 00:10:47 +00:00
Bzip2WriteBuffer.cpp Fixes for -Wshorten-64-to-32 2022-10-21 13:25:19 +02:00
Bzip2WriteBuffer.h Fix comments 2021-11-22 14:19:26 +03:00
CascadeWriteBuffer.cpp Don't use CURRENT_WRITE_BUFFER_IS_EXHAUSTED for expected behaviour 2023-03-31 14:09:21 +02:00
CascadeWriteBuffer.h Don't use CURRENT_WRITE_BUFFER_IS_EXHAUSTED for expected behaviour 2023-03-31 14:09:21 +02:00
CMakeLists.txt fix unit test and build error 2022-01-06 11:41:24 +08:00
CompressedReadBufferWrapper.h Resubmit prefetches 2023-02-08 21:26:24 +01:00
CompressionMethod.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
CompressionMethod.h Add methods to the tests 2022-10-04 19:04:03 +00:00
ConcatReadBuffer.h Use BackupEntryFromAppendOnlyFile for Log engine, 2022-03-20 20:02:15 +01:00
ConcatSeekableReadBuffer.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
ConcatSeekableReadBuffer.h Fix incremental backups for Log family. 2022-08-31 12:57:28 +02:00
ConnectionTimeouts.cpp Cosmetics 2023-02-07 11:42:31 +00:00
ConnectionTimeouts.h Cosmetics 2023-02-07 11:42:31 +00:00
copyData.cpp use http request body in predefined http handlers 2023-03-02 12:51:48 +01:00
copyData.h use http request body in predefined http handlers 2023-03-02 12:51:48 +01:00
DoubleConverter.cpp
DoubleConverter.h abort instead of __builtin_unreachable in debug builds 2022-10-07 21:49:08 +02:00
EmptyReadBuffer.h
FileEncryptionCommon.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
FileEncryptionCommon.h Generate config.h into ${CONFIG_INCLUDE_PATH} 2022-09-28 12:48:26 +00:00
ForkWriteBuffer.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
ForkWriteBuffer.h Addressed review comments 2022-07-20 21:02:54 +02:00
HadoopSnappyReadBuffer.cpp Better 2023-05-01 13:01:23 +00:00
HadoopSnappyReadBuffer.h Better 2023-05-01 13:01:23 +00:00
HashingReadBuffer.h Fix incremental backups for Log family. 2022-08-31 12:57:28 +02:00
HashingWriteBuffer.cpp
HashingWriteBuffer.h Allow LowCardinality in GenerateRandom 2023-01-26 15:35:20 +01:00
HTTPChunkedReadBuffer.cpp fix some bad error messages, suppress others 2023-05-02 17:55:42 +02:00
HTTPChunkedReadBuffer.h fix heap buffer overflow by limiting http chunk size 2022-08-18 14:30:52 +02:00
HTTPCommon.cpp Remove duplicate code 2023-04-15 02:40:21 +08:00
HTTPCommon.h Remove duplicate code 2023-04-15 02:40:21 +08:00
HTTPHeaderEntries.h Replace old named collections code for url 2022-12-17 00:24:05 +01:00
IReadableWriteBuffer.h
IResourceManager.h integrate IO scheduler with buffers 2023-01-27 18:47:22 +00:00
ISchedulerConstraint.h fix typos 2022-09-27 15:20:43 +00:00
ISchedulerNode.h fix build 2022-11-24 18:06:10 +00:00
ISchedulerQueue.h fix typos 2023-01-27 22:48:02 +00:00
LimitReadBuffer.cpp fix multipart requests 2023-02-22 17:54:35 +01:00
LimitReadBuffer.h fix multipart requests 2023-02-22 17:54:35 +01:00
LimitSeekableReadBuffer.cpp Fix LimitSeekableReadBuffer. 2023-05-11 22:58:05 +02:00
LimitSeekableReadBuffer.h Copy files to S3 during backup directly without using WriteBufferFromS3 to decrease memory consumption. 2023-01-17 09:35:41 +01:00
Lz4DeflatingWriteBuffer.cpp check number of parameters in format string 2023-01-23 23:16:16 +01:00
Lz4DeflatingWriteBuffer.h Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
Lz4InflatingReadBuffer.cpp Add file name to exception raised during decompression 2023-04-27 02:52:23 +00:00
Lz4InflatingReadBuffer.h show names of erroneous files 2022-04-16 00:10:47 +00:00
LZMADeflatingWriteBuffer.cpp Fix comments 2021-11-22 14:19:26 +03:00
LZMADeflatingWriteBuffer.h Fix comments 2021-11-22 14:19:26 +03:00
LZMAInflatingReadBuffer.cpp Add file name to exception raised during decompression 2023-04-27 02:52:23 +00:00
LZMAInflatingReadBuffer.h show names of erroneous files 2022-04-16 00:10:47 +00:00
MemoryReadWriteBuffer.cpp Don't use CURRENT_WRITE_BUFFER_IS_EXHAUSTED for expected behaviour 2023-03-31 14:09:21 +02:00
MemoryReadWriteBuffer.h Don't use CURRENT_WRITE_BUFFER_IS_EXHAUSTED for expected behaviour 2023-03-31 14:09:21 +02:00
MMappedFile.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
MMappedFile.h
MMappedFileCache.h Change class LRUCache to class CachBase. Check running CacheBase with default pcahce policy SLRU 2022-08-07 19:59:30 +00:00
MMappedFileDescriptor.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
MMappedFileDescriptor.h Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
MMapReadBufferFromFile.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
MMapReadBufferFromFile.h
MMapReadBufferFromFileDescriptor.cpp Adapt internal data structures to 512-bit era (#42564) 2022-10-25 13:56:28 +02:00
MMapReadBufferFromFileDescriptor.h improvements for getFileSize 2022-06-20 15:22:56 +02:00
MMapReadBufferFromFileWithCache.cpp Rewrite StorageFile buffer creation with createReadBufferFromFileBase. 2022-12-07 22:31:32 -05:00
MMapReadBufferFromFileWithCache.h
MySQLBinlogEventReadBuffer.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
MySQLBinlogEventReadBuffer.h
MySQLPacketPayloadReadBuffer.cpp Enforce checking read output 2022-11-11 10:56:18 +01:00
MySQLPacketPayloadReadBuffer.h
MySQLPacketPayloadWriteBuffer.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
MySQLPacketPayloadWriteBuffer.h Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
NullWriteBuffer.cpp
NullWriteBuffer.h Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
OpenedFile.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
OpenedFile.h Improve performance of file descriptor cache by narrowing mutex scopes (#36682) 2022-04-28 14:56:53 +02:00
OpenedFileCache.h Fix OpenedFileCacheHits/OpenedFileCacheMisses 2021-11-10 21:15:27 +03:00
Operators.h Use string concatenation for XML serialization (#47251) 2023-03-05 18:19:33 +01:00
ParallelReadBuffer.cpp Fixes for clang-17 2023-05-13 02:57:31 +02:00
ParallelReadBuffer.h Parallel decoding with one row group per thread 2023-04-17 04:58:32 +00:00
parseDateTimeBestEffort.cpp Dec 15, 2021 support for parseDateTimeBestEffort function 2023-03-01 21:05:58 +08:00
parseDateTimeBestEffort.h initial changes to close #37492 2022-08-08 19:45:00 -07:00
PeekableReadBuffer.cpp fix 2023-01-24 17:10:54 +01:00
PeekableReadBuffer.h Resubmit prefetches 2023-02-08 21:26:24 +01:00
Progress.cpp Better handling of values too large for VarInt encoding 2023-04-11 07:47:33 +00:00
Progress.h Fix error 2022-08-10 09:31:24 +02:00
ReadBuffer.cpp Copy files to S3 during backup directly without using WriteBufferFromS3 to decrease memory consumption. 2023-01-17 09:35:41 +01:00
ReadBuffer.h Hopefully fix assertion failure in CachedOnDiskReadBufferFromFile 2023-04-17 04:58:32 +00:00
ReadBufferFromEmptyFile.h improvements for getFileSize 2022-06-20 15:22:56 +02:00
ReadBufferFromEncryptedFile.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
ReadBufferFromEncryptedFile.h Generate config.h into ${CONFIG_INCLUDE_PATH} 2022-09-28 12:48:26 +00:00
ReadBufferFromFile.cpp Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
ReadBufferFromFile.h Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
ReadBufferFromFileBase.cpp Use forward declaration of ThreadPool 2023-04-07 11:25:35 +02:00
ReadBufferFromFileBase.h Fix builds 2023-01-10 19:22:18 +00:00
ReadBufferFromFileDecorator.cpp Refactoring of object storages, add LocalObjectStorage 2022-07-09 12:39:31 +02:00
ReadBufferFromFileDecorator.h improvements for getFileSize 2022-06-20 15:22:56 +02:00
ReadBufferFromFileDescriptor.cpp Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
ReadBufferFromFileDescriptor.h Something 2023-04-17 04:58:32 +00:00
ReadBufferFromIStream.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
ReadBufferFromIStream.h
ReadBufferFromMemory.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
ReadBufferFromMemory.h Add convenience ctor to ReadBufferFromMemory 2022-10-27 10:26:55 +02:00
ReadBufferFromPocoSocket.cpp Improve async reading from socket 2023-03-15 12:18:08 +00:00
ReadBufferFromPocoSocket.h Improve async reading from socket 2023-03-15 12:18:08 +00:00
ReadBufferFromS3.cpp Merge remote-tracking branch 'upstream/master' into cache-better-locks 2023-04-17 19:47:43 +02:00
ReadBufferFromS3.h Better 2023-04-17 04:58:32 +00:00
ReadBufferFromString.h refactor metadatacache to reduce using of USE_ROCKSDB 2022-01-05 19:51:50 +08:00
readDecimalText.h reformat ParsingException 2023-01-24 23:21:29 +01:00
readFloatText.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
readFloatText.h Remove PVS-Studio 2023-02-19 23:30:05 +01:00
ReadHelpers.cpp address comments and update docs 2023-04-21 11:31:59 -03:00
ReadHelpers.h Merge pull request #48745 from Algunenano/arenita 2023-04-25 00:53:11 +03:00
ReadHelpersArena.h Style 2023-04-14 10:35:33 +02:00
ReadSettings.h Merge pull request #48791 from kssenii/better-local-object-storage 2023-05-04 11:47:43 +02:00
ReadWriteBufferFromHTTP.cpp Split ReadWriteBufferFromHTTP.h into .h and .cpp file 2023-05-06 21:11:06 -07:00
ReadWriteBufferFromHTTP.h Split ReadWriteBufferFromHTTP.h into .h and .cpp file 2023-05-06 21:11:06 -07:00
ResourceBudget.h fix typos 2023-01-27 22:48:02 +00:00
ResourceGuard.h fix typos 2023-01-27 22:48:02 +00:00
ResourceLink.h integrate IO scheduler with buffers 2023-01-27 18:47:22 +00:00
ResourceManagerFactory.h IO scheduling subsystem 2022-09-27 13:26:41 +00:00
ResourceRequest.h integrate IO scheduler with buffers 2023-01-27 18:47:22 +00:00
S3Common.cpp Add support for SSE-KMS configuration with S3 2023-05-03 21:35:38 -05:00
S3Common.h fix special build 2023-05-10 18:45:59 +00:00
SchedulerNodeFactory.h IO scheduling subsystem 2022-09-27 13:26:41 +00:00
SchedulerRoot.h more fixes 2022-12-16 13:25:26 +00:00
SeekableReadBuffer.cpp Copy files to S3 during backup directly without using WriteBufferFromS3 to decrease memory consumption. 2023-01-17 09:35:41 +01:00
SeekableReadBuffer.h Unbreak reading from web servers that don't support HEAD requests 2023-04-17 04:58:32 +00:00
SeekAvoidingReadBuffer.cpp Drop unused headers 2021-09-01 02:47:52 +03:00
SeekAvoidingReadBuffer.h Resubmit prefetches 2023-02-08 21:26:24 +01:00
SharedThreadPools.cpp Speedup outdated parts loading (#49317) 2023-05-03 18:56:45 +02:00
SharedThreadPools.h Speedup outdated parts loading (#49317) 2023-05-03 18:56:45 +02:00
SnappyReadBuffer.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
SnappyReadBuffer.h Generate config.h into ${CONFIG_INCLUDE_PATH} 2022-09-28 12:48:26 +00:00
SnappyWriteBuffer.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
SnappyWriteBuffer.h Generate config.h into ${CONFIG_INCLUDE_PATH} 2022-09-28 12:48:26 +00:00
StdIStreamFromMemory.cpp work with comments on PR 2023-05-15 14:41:51 +02:00
StdIStreamFromMemory.h work with comments on PR 2023-05-15 14:41:51 +02:00
StdStreamBufFromReadBuffer.cpp Fix using std::ios_base::end in StdStreamFromReadBuffer::seekg(). 2023-01-17 17:56:14 +01:00
StdStreamBufFromReadBuffer.h Split upload into parts of the same size for smooth uploading. 2023-01-17 09:35:43 +01:00
StdStreamFromReadBuffer.h Split upload into parts of the same size for smooth uploading. 2023-01-17 09:35:43 +01:00
SwapHelper.cpp rework WriteBufferFromS3, squashed 2023-05-10 18:31:47 +00:00
SwapHelper.h rework WriteBufferFromS3, squashed 2023-05-10 18:31:47 +00:00
SynchronousReader.cpp Better ReadBuffer asynchronicity 2022-02-20 01:08:21 +08:00
SynchronousReader.h Get rid of static threadpools 2022-09-07 17:48:11 +02:00
TimeoutSetter.cpp Fix error on OS X regarding resetting timeouts. 2023-05-13 22:55:27 -07:00
TimeoutSetter.h
UncompressedCache.h Cleanup: Untangle CacheBase's constructors 2023-03-13 15:11:12 +00:00
UseSSL.cpp Generate config.h into ${CONFIG_INCLUDE_PATH} 2022-09-28 12:48:26 +00:00
UseSSL.h
VarInt.cpp VarInt coding: Always perform sanity check 2023-04-13 09:44:35 +00:00
VarInt.h VarInt coding: Always perform sanity check 2023-04-13 09:44:35 +00:00
WithFileName.cpp Update WithFileName.cpp 2023-05-02 16:05:18 +02:00
WithFileName.h Add file name to exception raised during decompression 2023-04-27 02:52:23 +00:00
WithFileSize.cpp Fix 2022-07-04 16:05:08 +02:00
WithFileSize.h improvements for getFileSize 2022-06-20 15:22:56 +02:00
WriteBuffer.h rework WriteBufferFromS3, squashed 2023-05-10 18:31:47 +00:00
WriteBufferDecorator.h Fix comments 2021-11-22 14:19:26 +03:00
WriteBufferFromArena.h Small refactoring of WriteBiffer-s 2021-11-11 02:11:18 +03:00
WriteBufferFromEncryptedFile.cpp Fix comments 2021-11-22 14:19:26 +03:00
WriteBufferFromEncryptedFile.h Generate config.h into ${CONFIG_INCLUDE_PATH} 2022-09-28 12:48:26 +00:00
WriteBufferFromFile.cpp Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
WriteBufferFromFile.h Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
WriteBufferFromFileBase.cpp
WriteBufferFromFileBase.h
WriteBufferFromFileDecorator.cpp rework WriteBufferFromS3, squashed 2023-05-10 18:31:47 +00:00
WriteBufferFromFileDecorator.h rework WriteBufferFromS3, squashed 2023-05-10 18:31:47 +00:00
WriteBufferFromFileDescriptor.cpp Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
WriteBufferFromFileDescriptor.h Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
WriteBufferFromFileDescriptorDiscardOnFailure.cpp fix tracing of profile events 2022-12-08 15:13:53 +00:00
WriteBufferFromFileDescriptorDiscardOnFailure.h Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
WriteBufferFromHTTP.cpp Pass headers from StorageURL to WriteBufferFromHTTP (#46996) 2023-03-03 13:55:52 +01:00
WriteBufferFromHTTP.h Pass headers from StorageURL to WriteBufferFromHTTP (#46996) 2023-03-03 13:55:52 +01:00
WriteBufferFromOStream.cpp Better formatting for exception messages (#45449) 2023-01-24 00:13:58 +03:00
WriteBufferFromOStream.h Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
WriteBufferFromPocoSocket.cpp Improve async reading from socket 2023-03-15 12:18:08 +00:00
WriteBufferFromPocoSocket.h Improve async reading from socket 2023-03-15 12:18:08 +00:00
WriteBufferFromS3.cpp Get rid of finalize callback in object storages 2023-05-19 17:29:37 +02:00
WriteBufferFromS3.h Get rid of finalize callback in object storages 2023-05-19 17:29:37 +02:00
WriteBufferFromS3BufferAllocationPolicy.cpp work with comments on PR 2023-05-15 14:41:51 +02:00
WriteBufferFromS3TaskTracker.cpp work with comments on PR 2023-05-15 14:41:51 +02:00
WriteBufferFromS3TaskTracker.h work with comments on PR 2023-05-15 14:41:51 +02:00
WriteBufferFromString.h Less usage of StringRef 2022-07-24 18:33:52 +00:00
WriteBufferFromVector.h Preallocate more space 2023-01-25 16:26:14 +00:00
WriteBufferValidUTF8.cpp Remove cruft from build 2023-03-17 13:44:04 +00:00
WriteBufferValidUTF8.h Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
WriteHelpers.cpp use std::lerp, constexpr hex.h 2023-03-07 22:50:17 +00:00
WriteHelpers.h Fix writeAnyEscapedString if quote_character is a meta character 2023-05-05 12:40:48 +00:00
WriteIntText.h Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
WriteSettings.h Add ability to throttle local IO on per-query/server basis 2023-04-05 09:39:15 +02:00
ZlibDeflatingWriteBuffer.cpp Clean up GCC warning pragmas 2023-04-11 18:21:08 +00:00
ZlibDeflatingWriteBuffer.h Fix comments 2021-11-22 14:19:26 +03:00
ZlibInflatingReadBuffer.cpp Add file name to exception raised during decompression 2023-04-27 02:52:23 +00:00
ZlibInflatingReadBuffer.h fix reading from storage File compressed files with zlib and gzip compression 2023-03-20 20:56:43 +00:00
ZstdDeflatingAppendableWriteBuffer.cpp Better exception message for ZSTD 2023-04-08 00:54:05 +02:00
ZstdDeflatingAppendableWriteBuffer.h Preallocate more space 2023-01-25 16:26:14 +00:00
ZstdDeflatingWriteBuffer.cpp Better exception message for ZSTD 2023-04-08 00:54:05 +02:00
ZstdDeflatingWriteBuffer.h Fix comments 2021-11-22 14:19:26 +03:00
ZstdInflatingReadBuffer.cpp Add file name to exception raised during decompression 2023-04-27 02:52:23 +00:00
ZstdInflatingReadBuffer.h Revert "Revert "Add a setting to use more memory for zstd decompression"" 2022-06-18 15:55:35 +03:00