Commit Graph

79148 Commits

Author SHA1 Message Date
Jakub Kuklis
b3c4a92cc0 Code cleanup 2021-12-10 22:04:37 +01:00
Jakub Kuklis
c4691affb2 Adding a new test for read with offset 2021-12-10 22:04:37 +01:00
Jakub Kuklis
9a00acfb72 Fixing read size error plus minor improvements 2021-12-10 22:04:37 +01:00
Jakub Kuklis
fd8ebe0d9f Maximum read 2021-12-10 22:04:37 +01:00
Jakub Kuklis
c9f3164cba Bringing back old single part upload limit 2021-12-10 22:04:37 +01:00
Jakub Kuklis
0d7a4b10c3 Using unique_ptr for Blob Client in Read Buffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
12924b3659 Minor improvements 2021-12-10 22:04:37 +01:00
Jakub Kuklis
fa2261dee6 Changing offset in ReadBuffer to size_t 2021-12-10 22:04:37 +01:00
Jakub Kuklis
5e2b2b1eee Better search for blobs with a given name 2021-12-10 22:04:37 +01:00
Jakub Kuklis
33247db3f4 Adding finalize() to WriteBuffer, always using block blob uploads, code cleanup 2021-12-10 22:04:37 +01:00
Jakub Kuklis
bf2476a076 Shared pointer for Azure client and fixing path reference in ReadBuffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
60dd22dd4e Checking for success of file removal operation 2021-12-10 22:04:37 +01:00
Jakub Kuklis
486f267b3b Further code cleanup and error codes improvement 2021-12-10 22:04:37 +01:00
Jakub Kuklis
6755c5e897 File deletion prototype 2021-12-10 22:04:37 +01:00
Jakub Kuklis
a0cfafa387 checkUniqueId implementation using ListBlobs 2021-12-10 22:04:37 +01:00
Jakub Kuklis
956348073b ReadBufferFromBlobStorage operating like a proper stream, rather than a single download-read buffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
95816875f4 Change false to true in supportZeroCopyReplication 2021-12-10 22:04:37 +01:00
Jakub Kuklis
b41ab197e8 Blob Storage endpoint url verification using re2 2021-12-10 22:04:37 +01:00
Jakub Kuklis
63ec8d4244 Adding multipart upload to WriteBuffer, retries in ReadBuffer, additional settings for buffers 2021-12-10 22:04:37 +01:00
Jakub Kuklis
02060791e3 Adding settings getter and applyNewSettings 2021-12-10 22:04:37 +01:00
Jakub Kuklis
3f28f2c62b Enabling local cache for DiskBlobStorage 2021-12-10 22:04:37 +01:00
Jakub Kuklis
e3daf8aa49 Increase default thread pool size 2021-12-10 22:04:37 +01:00
Jakub Kuklis
dbb98e3636 Code cleanup, leaving some debug information printing under #ifdef VERBOSE_DEBUG_MODE 2021-12-10 22:04:37 +01:00
Jakub Kuklis
8230f8039c Two column MergeTree table with Blob Storage disk work! Fixed buffer in ReadBufferFromBlobStorage to be permanent (class attribute) 2021-12-10 22:04:37 +01:00
Jakub Kuklis
2b4d3665ab Basic inserts and selects work for a MergeTree table with a Blob Disk, based on a single column table 2021-12-10 22:04:37 +01:00
Jakub Kuklis
4b51ddee1b Initial check with write and read works for the Blob Storage Disk 2021-12-10 22:04:37 +01:00
Youenn Lebras
bf4fca0351 Fix compilation issues with Azure in order to follow the main git and not a patched one. 2021-12-10 22:04:37 +01:00
Jakub Kuklis
4ddc3abfe3 Patch for fixing warnings triggered by Azure SDK, cd to contrib/azure and use 'git apply ../azure-cmake/azure_patch 2021-12-10 22:04:37 +01:00
Jakub Kuklis
c6412f24d8 Add azure-sdk-for-cpp as submodule in contrib/azure, linking the main repo here instead of a fork, needs to be updated eventually according to https://clickhouse.com/docs/en/development/contrib/#adding-third-party-libraries 2021-12-10 22:04:37 +01:00
Jakub Kuklis
03f85dee41 Managed to read markers file in SELECT, getting error for CODEC version now 2021-12-10 22:04:37 +01:00
Jakub Kuklis
482a10e62e Uploading to Blob Storage on INSERT seems to do something reasonable 2021-12-10 22:04:37 +01:00
Jakub Kuklis
c43b90c58c Progressing with WriteBufferFromBlobStorage 2021-12-10 22:04:37 +01:00
Jakub Kuklis
fb4c1d6686 Starting insertion pipeline 2021-12-10 22:04:37 +01:00
Jakub Kuklis
6436f5b04d Adding an authenticated client to DiskBlobStorage 2021-12-10 22:04:37 +01:00
Jakub Kuklis
ee1a568919 The following query passes without erors: CREATE TABLE ... ENGINE = MergeTree ... storage_policy = 'blob_storage', debug constants present in the code 2021-12-10 22:04:37 +01:00
Jakub Kuklis
cacdac7ba6 Moving implementation to the .cpp file 2021-12-10 22:04:37 +01:00
Jakub Kuklis
4f506c4fc6 Forgot about commiting this file 2021-12-10 22:04:37 +01:00
Jakub Kuklis
88b5feb59d Registering the Blob Storage Disk 2021-12-10 22:04:36 +01:00
Jakub Kuklis
20dda7f666 Giving Disk Blob Storage files proper names 2021-12-10 22:04:36 +01:00
Jakub Kuklis
0cf338e165 Skeleton for DiskBlobStorage 2021-12-10 22:04:36 +01:00
Jakub Kuklis
43f420cba8 Adding a flag to use Azure Blob Storage 2021-12-10 22:04:36 +01:00
Jakub Kuklis
ec4d2a9e03 Removing unused file 2021-12-10 22:04:36 +01:00
Jakub Kuklis
4711be1ab7 Cleaning azure-cmake CMakeLists.txt 2021-12-10 22:04:36 +01:00
Jakub Kuklis
81361640b9 Using internal curl library 2021-12-10 22:04:36 +01:00
Jakub Kuklis
162e2b334b Initial working example of Azure Blob Storage manipulation from within ClickHouse 2021-12-10 22:04:36 +01:00
Vitaly Baranov
b5b195f4e2
Merge branch 'master' into kafka_protobuf_issue26643 2021-12-10 23:22:35 +03:00
Vitaly Baranov
82c2d8dd2c Add synchronization to ProtobufSchemas. 2021-12-10 23:18:47 +03:00
Alexey
750f2cc73b spelling 2021-12-10 20:18:08 +00:00
alesapin
5a542516e6
Merge pull request #32515 from ClickHouse/revert-32514-revert-32496-smaller_checks
Revert "Revert "Split long tests into multiple checks""
2021-12-10 22:30:04 +03:00
Azat Khuzhin
837ff82312 Suppress UBSan errors for avg() function
CI: https://s3.amazonaws.com/clickhouse-test-reports/0/dd2ccd3b17e7c59ed9c8184f00f8dae85ee87d1f/fuzzer_astfuzzerubsan,actions//report.html
2021-12-10 22:23:39 +03:00