thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 02:12:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
141,567 Commits 9,330 Branches 1,783 Tags 2.1 GiB
e2b39db242
Commit Graph

2575 Commits

Author SHA1 Message Date
Kseniia Sumarokova
b6094ac233
Merge branch 'master' into filecache-small-optimization 2024-01-19 11:59:04 +01:00
kssenii
2f34ef6ff8 Follow up 2024-01-19 11:19:59 +01:00
Kseniia Sumarokova
1c74e7fa08
Merge pull request #58902 from ClickHouse/fs-cache-refactorings 
fs cache: minor refactoring
 2024-01-19 10:25:45 +01:00
Alexey Milovidov
8a288055c9
Merge pull request #58949 from MikhailBurdukov/fix_s3_copy 
Using the destination object for  paths generation in S3copy.
 2024-01-19 09:28:13 +01:00
Alexey Milovidov
bcff1bac47
Merge pull request #58874 from ClickHouse/disk-minor-improvement 
DiskWeb minor improvement in loading
 2024-01-19 03:52:05 +01:00
Alexander Gololobov
e9919f47c1 Move paramaters from span name to attributes 2024-01-18 14:26:19 +01:00
MikhailBurdukov
7518241be4 Use dest object storage to native copy for path generation. 2024-01-18 11:46:16 +00:00
kssenii
9b674e640d Fxi 2024-01-18 12:20:40 +01:00
Smita Kulkarni
e71977b724 Merge branch 'master' into Azure_backup 2024-01-18 08:29:01 +01:00
kssenii
ef48b90bca Fix and test 2024-01-17 18:44:09 +01:00
kssenii
dd9e56f08f Better 2024-01-17 14:58:42 +01:00
Anton Popov
245e1cbd5c Merge remote-tracking branch 'upstream/master' into better-read-buffers 2024-01-17 13:46:48 +00:00
kssenii
d5198c1b47 Better 2024-01-17 14:44:53 +01:00
kssenii
a572fda873 Fix 2024-01-17 14:22:34 +01:00
Nikita Taranov
bb124b544c
Merge pull request #58077 from nickitat/revert_rest 
Remove unneeded parameter `use_external_buffer` from `AsynchronousReadBuffer*`
 2024-01-17 13:45:07 +01:00
kssenii
c905937ef6 Refactoring 2024-01-17 13:22:59 +01:00
Anton Popov
299293c088 better interface in read buffers 2024-01-16 23:26:20 +00:00
kssenii
e5d8b9b190 More lazy loading 2024-01-16 16:12:14 +01:00
Kseniia Sumarokova
f2a31db9d2
Merge branch 'master' into filecache-small-optimization 2024-01-16 15:04:02 +01:00
kssenii
8decf5039a Fix for custom cached disk 2024-01-16 14:41:12 +01:00
Kseniia Sumarokova
ae884760db
Merge pull request #58357 from ClickHouse/better-disks-configuration 
Flexible configuration for disks
 2024-01-15 21:34:36 +01:00
kssenii
abd800dfa6 Fix keeper build 2024-01-15 13:25:10 +01:00
Alexey Milovidov
c132e0141c
Merge pull request #58791 from ClickHouse/better-stateful-tests 
Simplify stateful tests
 2024-01-14 21:39:13 +01:00
Nikita Taranov
7a3ceeb54c
Fix usage of small buffers for remote reading (#58768) 
* impl

* add test

* fix test
 2024-01-14 16:06:51 +01:00
Alexey Milovidov
fe70165921 Whitespace 2024-01-14 08:08:32 +01:00
Alexey Milovidov
bbbf455146 Better error message 2024-01-14 00:05:15 +01:00
kssenii
341a9b3b70 Fix keeper build 2024-01-13 12:56:27 +01:00
Alexey Milovidov
d112492c56 Remove some code 2024-01-13 03:48:04 +01:00
kssenii
dc61f68f65 Rename one more method 2024-01-12 17:55:57 +01:00
kssenii
7c92d69820 Rename a method 2024-01-12 17:27:45 +01:00
kssenii
caf267c6ed Merge remote-tracking branch 'origin' into filecache-small-optimization 2024-01-12 13:23:45 +01:00
kssenii
7b674c31c5 Fix for issue 58697 2024-01-12 10:55:27 +01:00
kssenii
df1d4458b4 Fix build without s3 2024-01-11 12:04:54 +01:00
kssenii
0527d18f3f Fix build without s3 2024-01-10 18:28:11 +01:00
Robert Schulze
52d75ab68f
Merge pull request #58458 from rschu1ze/replace-std_regexp-by-re2 
Replace `std::regex` by re2
 2024-01-10 17:46:54 +01:00
kssenii
23659bec8f Fix style check 2024-01-10 17:02:51 +01:00
Vitaly Baranov
dbb19e72b3
Merge pull request #58629 from vitlibar/remove-finalize-from-destructor-of-write-buffer-from-encrypted-file 
Remove finalize() from ~WriteBufferFromEncryptedFile
 2024-01-10 16:36:11 +01:00
kssenii
383d40ea03 Finish changes 2024-01-10 14:18:04 +01:00
kssenii
eb294075a6 Merge remote-tracking branch 'origin' into better-disks-configuration 2024-01-10 13:10:44 +01:00
Robert Schulze
499227b9cf
Merge remote-tracking branch 'rschu1ze/master' into replace-std_regexp-by-re2 2024-01-10 10:02:53 +00:00
Vitaly Baranov
01dfe282ea Remove finalize() from ~WriteBufferFromEncryptedFile. 2024-01-09 17:22:17 +01:00
Robert Schulze
8e804487f3
Some fixups 2024-01-07 22:28:08 +00:00
Sema Checherinda
8ab4aa3423 enable randomization setting 2024-01-05 13:42:30 +00:00
Sema Checherinda
53f36f8b21 object storage key template 2024-01-05 13:42:30 +00:00
Smita Kulkarni
2ee6893312 Updated to return container for getObjectsNamespace 2024-01-05 10:58:04 +01:00
Smita Kulkarni
c14605caa7 Added flag use_native_copy and updated to use StartCopyFromUri for native copy with large files 2024-01-04 18:27:54 +01:00
Smita Kulkarni
91bad5bc39 Updated to use MultiVersion for BlobContainerClient in Backups and updated to get client from disk 2024-01-04 16:06:36 +01:00
Smita Kulkarni
4122de9721 Updated tests and added settings 2024-01-02 20:19:01 +01:00
kssenii
653d48a552 Flexible disks configuration 2023-12-29 19:57:58 +01:00
Azat Khuzhin
8c54380d80 Avoid sending ComposeObject requests after upload to GCS 
This should not be required anymore, but leave it as an option, since
likely this is required for old files.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
f4a7789cd4 Convert various S3::Client settings into separate ClientSettings struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
kssenii
d5b138eee4 Log exception 2023-12-28 13:41:07 +01:00
Alexey Milovidov
30f6d03b37
Merge pull request #57977 from bianpengyuan/add-s3-read-only-setting 
Add a setting to specify s3 disk is read only.
 2023-12-28 01:24:50 +01:00
Alexey Milovidov
8d984df135
Merge pull request #58237 from azat/build/fwd-decl-exception 
Some code refactoring (was an attempt to improve build time, but failed)
 2023-12-28 00:21:09 +01:00
Azat Khuzhin
b9233f6d4f Move Allocator code into module part 
This should reduce amount of code that should be recompiled on
Exception.h changes (and everything else that had been included there).

This will actually not help a lot, because it is also included into
PODArray.h and ThreadPool.h at least... Sigh.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 15:42:08 +01:00
Dani Pozo
2be2486e94 Remove retryStrategy assignments overwritten in ClientFactory::create() 2023-12-22 17:28:43 +01:00
Alexey Milovidov
ee199877d6
Update diskSettings.cpp 2023-12-22 07:27:48 +03:00
Raúl Marín
ced9407cef Improve error messages 2023-12-21 10:29:05 +01:00
Nikita Taranov
354b689800 impl 2023-12-20 16:07:00 +01:00
vinay92-ch
462aeadb6e Style check 2023-12-18 23:07:29 +00:00
vinay92-ch
f3037b502d Merge branch 'master' into add_azure_workload_identity_support_blob 2023-12-18 22:37:09 +00:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
Raúl Marín
2639d0715f Merge remote-tracking branch 'blessed/master' into log_message_string 2023-12-18 10:40:18 +01:00
Pengyuan Bian
32816d7e7e Add a setting to specify s3 disk is read only. 2023-12-18 04:07:54 +00:00
Alexey Milovidov
dbd509417e
Merge pull request #57970 from ClickHouse/nickitat-patch-17 
Always use `pread` for reading cache segments
 2023-12-17 20:53:05 +01:00
Nikita Taranov
9c2ef4eae5
Add profile event for cache lookup in ThreadPoolRemoteFSReader (#57437) 2023-12-17 19:03:49 +01:00
Nikita Taranov
587f829eb8
Always use pread for reading cache segments 2023-12-17 18:37:07 +01:00
vinay92-ch
2ff364cb73 Merge branch 'master' into add_azure_workload_identity_support_blob 2023-12-16 00:33:30 +00:00
Raúl Marín
b269f87f4c Better text_log with ErrnoException 2023-12-15 19:27:56 +01:00
Sema Checherinda
0dfe530a7f
Merge pull request #56744 from MikhailBurdukov/native_copy_for_s3_disks 
Enabled s3 `copyObject` for copy between s3 disks.
 2023-12-15 16:05:16 +01:00
Vinay Suryadevara
d347bec531
[WIP] Add support for Workload identity credentials in AzureBlobStorage 2023-12-14 16:09:14 -08:00
Kseniia Sumarokova
79db3c66df
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-13 23:33:59 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Alexey Milovidov
62b6d1ef5e iMerge branch 'master' of github.com:ClickHouse/ClickHouse into clickbench-ci 2023-12-13 01:41:27 +01:00
Kseniia Sumarokova
91d36ad224
Merge pull request #57076 from ClickHouse/slru-for-filesystem-cache 
Implement SLRU cache policy for filesystem cache
 2023-12-12 10:20:58 +01:00
Alexey Milovidov
9789c2caa2 Review fixes 2023-12-12 05:48:09 +01:00
MikhailBurdukov
119e451967 Merge branch 'master' into native_copy_for_s3_disks 2023-12-11 07:25:20 +00:00
Azat Khuzhin
6ccbc2ea75 Move io_uring reader into the Context from static to make it's thread joinable 
v2: fix for standalone keeper build
CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/72b1052f7c2d453308262924e767ab8dc2206933/stateless_tests__debug__[4_5].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 22:50:48 +01:00
Nikita Mikhaylov
c979124cfe Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-08 16:25:02 +00:00
MikhailBurdukov
d10217af16 style 2023-12-08 13:14:47 +00:00
kssenii
725571461d Merge remote-tracking branch 'origin/master' into slru-for-filesystem-cache 2023-12-07 19:49:03 +01:00
kssenii
8be3c9d218 Merge remote-tracking branch 'origin/master' into allow-to-change-some-cache-settings-without-restart 2023-12-07 12:14:24 +01:00
kssenii
614da21144 Better 2023-12-07 12:12:10 +01:00
kssenii
f44f7c8c28 Allow to change some cache settings without server restart 2023-12-06 19:29:18 +01:00
jsc0218
cdd5280272
Merge pull request #57387 from evillique/better-disks 
Initialize only required disks in clickhouse-disks
 2023-12-05 13:59:04 -05:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
kssenii
4a28f10c3d Minor cache changes 2023-12-04 19:02:37 +01:00
vdimir
a4ae90de0d
Merge pull request #57275 from ClickHouse/vdimir/merge_task_tmp_data 
Background merges correctly use temporary data storage in the cache
 2023-12-04 14:52:20 +01:00
robot-ch-test-poll
1b49463bd2
Merge pull request #55841 from nickitat/optimize_reading3 
Optimize reading from cache
 2023-12-01 17:36:57 +01:00
Nikolay Degterinsky
e8203f8a76 Initialize only required disks 2023-11-30 03:09:55 +00:00
vdimir
b5babe1692
MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
Kseniia Sumarokova
a89bb04e9c
Update comment 2023-11-29 13:30:55 +01:00
kssenii
28d54f0027 Better exception code 2023-11-29 10:57:35 +01:00
Nikita Taranov
a81453cafc fix test 2023-11-28 23:48:52 +01:00
Nikita Taranov
03450d5077 merge fixes 2023-11-28 18:24:05 +01:00
kssenii
4d64cd5d11 Fix 2023-11-28 17:13:08 +01:00
Nikita Taranov
52f644c0df Merge branch 'master' into optimize_reading3 2023-11-28 16:36:38 +01:00
Nikita Taranov
bdab4d5944 handle case with multiple blobs in ReadBufferFromRemoteFSGather 2023-11-28 15:40:35 +01:00
Nikita Taranov
0ad796aa99 add profile event 
add profile event for set&seek

fix

fix

fix
 2023-11-28 15:40:34 +01:00
MikhailBurdukov
e04416b48f Fix 2023-11-27 18:42:06 +00:00
MikhailBurdukov
0b25e5c347 review 2023-11-27 17:45:02 +00:00
MikhailBurdukov
aa3cba1f1c Fix 2023-11-27 13:45:21 +00:00
MikhailBurdukov
6f19e8ebd1
Merge branch 'master' into native_copy_for_s3_disks 2023-11-27 14:25:36 +03:00
MikhailBurdukov
c10c30832c Review fix 2023-11-27 10:58:30 +00:00
Azat Khuzhin
4a02de4674 Add ability to disable checksums for S3 to avoid excessive input file read 
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
  PayloadSigningPolicy::Never)

So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.

By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.

And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.

Here is an example stacktrace of this excessive read:

<details>

<summary>stacktrace</summary>

    (lldb) bt
    * thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
      * frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
        frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
        frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
        frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
        frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
        frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
        frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
        frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
        frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
        frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
        frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
        frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
        frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
        frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
        frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
        frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
        frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
        frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()

</details>

This new behaviour could be enabled with `s3_disable_checksum=true`.

Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
 2023-11-26 19:20:19 +01:00
Aleksei Filatov
1a03f5f7f4 Merge remote-tracking branch 'upstream/master' into add_cancellation_point_for_moving_background_operation 2023-11-23 16:43:33 +03:00
kssenii
00177a8016 Merge remote-tracking branch 'origin/master' into slru-for-filesystem-cache 2023-11-23 12:29:53 +01:00
Kseniia Sumarokova
2880e6437e
Merge pull request #56936 from jrdi/fs-cache-hit-profile-events 
Add CachedReadBufferReadFromCache{Hits,Misses} profile events
 2023-11-22 11:26:00 +01:00
Kseniia Sumarokova
e4f66b8469
Merge pull request #55158 from kssenii/fs-cache-improvement 
fs cache improvement for big reads
 2023-11-21 21:50:00 +01:00
kssenii
42e16bcc49 SLRU for filesystem cache 2023-11-21 21:06:32 +01:00
vdimir
a139ae97eb
Merge pull request #52918 from ClickHouse/vdimir/s3_blob_log 
Add system table with blob storage operations log
 2023-11-21 17:40:42 +01:00
Alexander Gololobov
8cf93fe133
Merge pull request #56905 from ClickHouse/disk_encrypted_create_dirs 
Allow delegate disk to handle retries for createDirectories
 2023-11-21 12:51:05 +01:00
Kseniia Sumarokova
d384762123
Merge branch 'master' into fs-cache-improvement 2023-11-21 11:24:52 +01:00
Jordi Villar
6bd20abb90 Add CachedReadBufferReadFromCache{Hits,Misses} profile events 2023-11-21 10:34:12 +01:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
Sema Checherinda
f999337dae
Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
Alexander Tokmakov
5031f239c3
Revert "s3 adaptive timeouts" 2023-11-20 14:28:59 +01:00
Sema Checherinda
a950595c24
Merge pull request #56314 from CheSema/s3-aggressive-timeouts 
s3 adaptive timeouts
 2023-11-19 14:12:14 +01:00
Alexey Milovidov
d56cbda185 Add metrics for the number of queued jobs, which is useful for the IO thread pool 2023-11-18 19:07:59 +01:00
Alexander Gololobov
6c3793acb0 Allow delegate disk to handle retries for createDirectories 2023-11-17 12:03:00 +01:00
Aleksei Filatov
163b8a814e Fix clang-tidy issues and disable parallel execution of tests 2023-11-17 11:02:07 +03:00
Kseniia Sumarokova
a2ed756eec
Merge branch 'master' into fs-cache-improvement 2023-11-16 17:49:26 +01:00
kssenii
472cfdc86d Review fix 2023-11-16 17:47:51 +01:00
Aleksei Filatov
21a17f8334 Add cancellation hook for moving background operation 2023-11-16 13:37:51 +03:00
Sema Checherinda
6e3e6383ba perf check 2 2023-11-15 19:00:27 +01:00
MikhailBurdukov
f29c072429 Test fix 2023-11-15 09:33:40 +00:00
Sema Checherinda
b9cc1580e0
Merge pull request #56367 from canhld94/ch_table_reinit_new_disk 
Adding new disk to storage configuration without restart
 2023-11-14 15:54:22 +01:00
MikhailBurdukov
1aea8186c7 Enable s3 native copy for scenario: from s3 to s3. 2023-11-14 13:53:48 +00:00
Sema Checherinda
8d36fd6e54 get rid off of client_with_long_timeout_ptr 2023-11-14 11:34:12 +01:00
Sema Checherinda
27fb25d056 alter the naming, fix client_with_long_timeout in s3 storage 2023-11-14 11:34:12 +01:00
Sema Checherinda
be01a5cd3e turn off agressive timeouts for heavy requests 2023-11-14 11:34:12 +01:00
Sema Checherinda
ab2594154e add tests 2023-11-14 11:34:12 +01:00
Kseniia Sumarokova
e428e4ed7a
Merge branch 'master' into cache-config-improvement 2023-11-13 10:09:58 +01:00
Sema Checherinda
3c15a5e714 hide VERSION_INLINE_DATA under storage_metadata_write_full_object_key feature 2023-11-10 17:24:50 +01:00
kssenii
c28c1ac4a2 Allow getting cache configuration from named collection 2023-11-09 18:39:16 +01:00
kssenii
dbea50738b Merge remote-tracking branch 'origin/master' into fs-cache-improvement 2023-11-08 11:53:54 +01:00
Nikita Taranov
c52571223e better 2023-11-07 22:04:13 +01:00
Sema Checherinda
8216a24abe move storage_metadata_write_full_object_key setting to the server scope 2023-11-07 17:14:46 +01:00
Nikita Taranov
1f2f51844f smart fix 2023-11-06 17:32:11 +01:00
Duc Canh Le
4c21ba7b6f tables auto initialize new disks without restart 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-11-06 10:26:48 +00:00
Nikita Taranov
5040a3b6d3 do seek before calling isContentCached() 2023-11-03 22:52:12 +01:00
Nikita Taranov
7046832056 change signatures 2023-11-03 22:50:03 +01:00
Sema Checherinda
8958861f99 review notes 2023-11-03 22:43:40 +01:00
Sema Checherinda
a108f9f764
Update src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-11-03 14:54:48 +01:00
Sema Checherinda
890cc8bbca
Update src/Disks/ObjectStorages/DiskObjectStorageMetadata.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-11-03 14:54:40 +01:00
Sema Checherinda
bab867f383
Update src/Disks/ObjectStorages/DiskObjectStorageMetadata.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-11-03 14:54:10 +01:00
Sema Checherinda
90b64bcdb9 backward compatibility and implementation feature storage_metadata_write_full_object_key 2023-11-02 15:20:55 +01:00
Kseniia Sumarokova
e5efc42633
Merge branch 'master' into fs-cache-improvement 2023-11-01 10:47:36 +01:00
Nikita Taranov
1d58cedce5 better 2023-10-31 22:48:20 +01:00
Azat Khuzhin
c7f392500e Fix incorrect free space accounting for least_used JBOD policy 
Before least_used fails to detect when the disk started to have more
space, it works only when the disk starts to have less space.

The reason for this is that it uses priority_queue, and once the disk
goes at the bottom of the queue, free space will not be updated for it
until it will be selected again.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-29 17:36:31 +01:00
Nikita Taranov
213ce63ffb
Merge branch 'master' into optimize_reading3 2023-10-28 00:04:20 +02:00
Nikita Taranov
c97bfbeb92 fix tests 2023-10-27 13:47:13 +02:00
Nikita Taranov
a86bda5208 use local thread pool reader for cache segments 2023-10-24 14:06:13 +02:00
Nikita Taranov
9b13c796e3 set ignore metric in all readers 2023-10-20 22:24:56 +02:00
Nikita Taranov
2a6a97c1fa read sync from cache 2023-10-20 22:24:56 +02:00
Alexander Tokmakov
f899254e2c
Merge pull request #55430 from arthurpassos/properly_split_remote_proxy_http_https 
Properly split remote proxy http https
 2023-10-20 21:15:34 +02:00
kssenii
7515853ad4 Fix build 2023-10-17 11:43:51 +02:00
kssenii
43c5e173de Merge remote-tracking branch 'upstream/master' into fs-cache-improvement 2023-10-16 13:06:28 +02:00
alesapin
3b02748cb6 Fix some typos 2023-10-15 15:43:02 +02:00
alesapin
21e0cb0b18
Merge pull request #55468 from ClickHouse/fix_out_of_bound_disk_web 
Fix out of bound error in system.remote_data_paths + disk web
 2023-10-11 12:54:00 +02:00
Alexey Milovidov
5ad055df98
Merge pull request #55372 from azat/web-disk-fix-race 
Fix data-race in web disk
 2023-10-11 01:03:48 +02:00
Azat Khuzhin
8021d8d747 Fix data-race in web disk 
TSan report:

    WARNING: ThreadSanitizer: data race (pid=8)
     Write of size 8 at 0x7b54002b0400 by thread T310:
       0 DB::ReadBufferFromWebServer::seek(long, int) build_docker/./src/Disks/IO/ReadBufferFromWebServer.cpp:159:12 (clickhouse+0x18a6a45b) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       1 DB::CachedOnDiskReadBufferFromFile::getImplementationBuffer(DB::FileSegment&) build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:471:47 (clickhouse+0x17f1aace) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       2 DB::CachedOnDiskReadBufferFromFile::nextImplStep() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:844:33 (clickhouse+0x17f1f5af) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       3 DB::CachedOnDiskReadBufferFromFile::nextImpl() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:774:16 (clickhouse+0x17f1f33a) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       4 DB::ReadBuffer::next() build_docker/./src/IO/ReadBuffer.h:70:20 (clickhouse+0x17f1307f) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       5 DB::ReadBufferFromRemoteFSGather::readImpl() build_docker/./src/Disks/IO/ReadBufferFromRemoteFSGather.cpp:188:32 (clickhouse+0x17f1307f)
       6 DB::ReadBufferFromRemoteFSGather::nextImpl() build_docker/./src/Disks/IO/ReadBufferFromRemoteFSGather.cpp:160:9 (clickhouse+0x17f12f89) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       7 DB::ReadBuffer::next() build_docker/./src/IO/ReadBuffer.h:70:20 (clickhouse+0x17d695ca) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       8 DB::ThreadPoolRemoteFSReader::execute(DB::IAsynchronousReader::Request) build_docker/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:95:25 (clickhouse+0x17d695ca)
       9 DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0::operator()() const build_docker/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:70:80 (clickhouse+0x17d6ab48) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)

     Previous read of size 8 at 0x7b54002b0400 by thread T297:
       0 DB::ReadBufferFromWebServer::getFileOffsetOfBufferEnd() const build_docker/./src/Disks/IO/ReadBufferFromWebServer.h:36:63 (clickhouse+0x18a6ac9d) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       1 DB::CachedOnDiskReadBufferFromFile::getInfoForLog() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:1223:71 (clickhouse+0x17f24835) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       2 DB::CachedOnDiskReadBufferFromFile::nextImplStep()::$_0::operator()() const build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:799:5 (clickhouse+0x17f2347c) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       3 BasicScopeGuard<DB::CachedOnDiskReadBufferFromFile::nextImplStep()::$_0>::invoke() build_docker/./base/base/../base/scope_guard.h:99:9 (clickhouse+0x17f2347c)
       4 BasicScopeGuard<DB::CachedOnDiskReadBufferFromFile::nextImplStep()::$_0>::~BasicScopeGuard() build_docker/./base/base/../base/scope_guard.h:48:26 (clickhouse+0x17f2347c)
       5 DB::CachedOnDiskReadBufferFromFile::nextImplStep() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:1071:1 (clickhouse+0x17f22480) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       6 DB::CachedOnDiskReadBufferFromFile::nextImpl() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:774:16 (clickhouse+0x17f1f33a) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       7 DB::ReadBuffer::next() build_docker/./src/IO/ReadBuffer.h:70:20 (clickhouse+0x17f1307f) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       8 DB::ReadBufferFromRemoteFSGather::readImpl() build_docker/./src/Disks/IO/ReadBufferFromRemoteFSGather.cpp:188:32 (clickhouse+0x17f1307f)
       9 DB::ReadBufferFromRemoteFSGather::nextImpl() build_docker/./src/Disks/IO/ReadBufferFromRemoteFSGather.cpp:160:9 (clickhouse+0x17f12f89) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       10 DB::ReadBuffer::next() build_docker/./src/IO/ReadBuffer.h:70:20 (clickhouse+0x17d695ca) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       11 DB::ThreadPoolRemoteFSReader::execute(DB::IAsynchronousReader::Request) build_docker/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:95:25 (clickhouse+0x17d695ca)
       12 DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0::operator()() const build_docker/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:70:80 (clickhouse+0x17d6ab48) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)

CI: https://s3.amazonaws.com/clickhouse-test-reports/55261/de503f75dcbc5a4d0e7fbb2e6b08c2106d62848a/integration_tests__tsan__[5_6].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-10 21:11:57 +02:00
alesapin
3e2c9c662c Fix tabulation 2023-10-10 17:09:26 +02:00
alesapin
37f5c445b4 Fix out of bound error in disk web 2023-10-10 15:44:27 +02:00
alesapin
666c690b4f
Merge pull request #55309 from ClickHouse/fix_redundant_check 
Fix bug with inability to drop detached partition in replicated merge tree on top of S3 without zero copy
 2023-10-10 11:48:00 +02:00
Arthur Passos
63562cbb20 Merge branch 'master' into properly_split_remote_proxy_http_https 2023-10-09 17:58:29 -03:00
alesapin
89294c0765 Review fixes: make check consistent in all write functions 2023-10-09 12:06:35 +02:00
alesapin
08f4621b48 Fix for write-once storages 2023-10-08 12:20:31 +02:00
Alexey Milovidov
f5b0fb27a4
Update DiskObjectStorageMetadata.cpp 2023-10-08 00:53:33 +03:00
alesapin
aa130b74e5 Fix offsets bug 2023-10-07 21:18:23 +02:00
kssenii
74a34a73de Merge remote-tracking branch 'upstream/master' into fs-cache-improvement 2023-10-07 13:09:42 +02:00
Arthur Passos
d54186df30 tmp 2023-10-04 17:21:41 -03:00
Sema Checherinda
a02bf94812 remove the bloker to grow the metadata file version 2023-10-03 17:37:56 +02:00
Nikita Taranov
97e705325b
Optimise memory consumption when external group by generates ton of temporary files (#54798) 
* impl

* fix

* fix test
 2023-10-02 13:09:40 +02:00
Alexey Milovidov
5ebd8ef41d Remove Meilisearch 2023-09-30 06:02:38 +02:00
kssenii
a72541d93f Improvement for big reads 2023-09-29 19:03:00 +02:00
Robert Schulze
9fff447716
Re-enable clang-tidy checks 2023-09-26 09:34:12 +00:00
Azat Khuzhin
c706101891 Fix throttling of BACKUPs from/to S3 (in case native copy was not used) 
In some cases native copy is not possible, and such requests should be
throttled.

v0: copyS3FileNativeWithFallback
v2: revert v0 and pass write_settings
v3: pass read_settings to copyFile()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-20 18:28:43 +02:00
Sema Checherinda
e7550523c8
Merge pull request #54651 from CheSema/limit_backoff_timeout 
limit the delay before next try in S3
 2023-09-18 19:41:26 +02:00
Robert Schulze
f5e8028bb1
Merge pull request #54642 from rschu1ze/broken-re2st 
Remove broken lockless variant of re2
 2023-09-17 15:30:57 +02:00
Kseniia Sumarokova
79697d4102
Update WebObjectStorage.cpp 2023-09-15 20:18:36 +02:00
Sema Checherinda
d9e15c00c9 limit the delay before next try in S3 2023-09-14 19:45:07 +02:00
Robert Schulze
7b378dbad3
Remove broken lockless variant of re2 2023-09-14 16:40:42 +00:00
Sema Checherinda
0ea0799118
Merge pull request #54430 from CheSema/remove-recursive-with-projection 
reproduce and fix the bug in removeSharedRecursive
 2023-09-12 00:18:55 +02:00
Sema Checherinda
d5268f3fb0 rename a member 2023-09-11 19:57:16 +02:00
Sergei Trifonov
08bad4d440
Merge pull request #47009 from ClickHouse/disks-and-io-sheduler 2023-09-11 07:22:14 +02:00
serxa
d326554caa fix keeper build 2023-09-10 14:55:40 +00:00
Sema Checherinda
1298dba0c6 fix the bug, add useful logs 2023-09-09 15:54:28 +02:00
SmitaRKulkarni
f0146c0ff0
Increase retries for test_merge_tree_azure_blob_storage (#54069) 
* Increase max_single_download_retries & max_single_read_retries for test_merge_tree_azure_blob_storage

* Updated logs to log exception leading to detaching part

* Updated to catch RequestFailedException azure exception as its the base exception

* Updated isRetryableExcepion to take std::exception_ptr as input

* Fixed style check fails

* Fix build issue & moved NetException & TimeoutException to retryable exceptions

* Fixed style check with catch
 2023-09-06 19:57:59 +02:00
Yakov Olkhovskiy
0dd6928a13
Merge pull request #53068 from nickitat/unify_keep_alive_setting 
Unify setting http keep-alive timeout, increase default to 30s
 2023-09-05 22:05:23 -04:00
Sergei Trifonov
258da91b5f
Merge branch 'master' into disks-and-io-sheduler 2023-09-04 15:14:28 +02:00
Kseniia Sumarokova
b3319f7908
Minor changes (#54171) 2023-09-03 15:47:52 +02:00
Michael Kolupaev
8505b3c940
Merge pull request #54134 from kssenii/refactor-reading-from-remote-fs 
Small refactoring for read from object storage
 2023-09-02 11:13:03 -07:00
kssenii
82948ac843 Review fixes 2023-09-02 14:00:07 +02:00
Sergei Trifonov
cf5ea46713
Merge pull request #53338 from azat/throttling-fixes 
Fix IO throttling during copying whole directories
 2023-09-01 18:29:15 +02:00
serxa
ef4dcf909f fix build 2023-09-01 16:23:15 +00:00
kssenii
9eb1dfcd12 Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
kssenii
3f20d1f578 Rename 2023-09-01 12:55:58 +02:00
serxa
9c92d413b3 Merge branch 'master' into disks-and-io-sheduler 2023-08-30 11:18:49 +00:00
Azat Khuzhin
51e8b16916 Fix IO throttling during copying whole directories 
v2: pass local context during ALTER TABLE MOVE PARTITION TO VOLUME/DISK as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-29 23:13:46 +02:00
xuelei
ebc7dbb628 The configuration of S3 append '/' automatically if needed 2023-08-28 09:39:22 -03:00
xuelei
636e7ddae7 deal with conflict 2023-08-28 06:05:21 -03:00
xiaolei565
853551bd3e
Update registerDiskS3.cpp 2023-08-28 12:38:33 +08:00
xiaolei565
ef6e1fd2d6
Update diskSettings.cpp 2023-08-28 12:38:08 +08:00
xiaolei565
58a1160095
Update diskSettings.cpp 2023-08-28 12:37:54 +08:00
xiaolei565
8d2c03ea26
Update registerDiskS3.cpp 2023-08-28 11:41:53 +08:00
xiaolei565
b9e872ddf7
Update diskSettings.cpp 2023-08-28 11:28:31 +08:00
xiaolei565
3c457c47c2
Update diskSettings.cpp 2023-08-28 11:05:30 +08:00
xuelei
643dd499b8 deal with conflict 2023-08-27 23:46:53 -03:00
xuelei
bb12f7decc deal with conflict 2023-08-27 23:41:23 -03:00
alesapin
0e601b856f
Merge pull request #53751 from Algunenano/clang18 
Support clang-18 (Wmissing-field-initializers)
 2023-08-25 23:27:06 +02:00
Kseniia Sumarokova
34ac113af6
Merge pull request #53121 from kssenii/fix-cache-logical-error 
Fix possible error from cache "Read unexpected size"
 2023-08-25 12:16:24 +02:00
Raúl Marín
6490213207 Merge remote-tracking branch 'blessed/master' into clang18 2023-08-24 17:35:26 +02:00
Kruglov Pavel
44db5fa992
Merge branch 'master' into cache-count 2023-08-24 17:21:18 +02:00
Arthur Passos
2bade7db08
Add global proxy setting (#51749) 
* initial impl

* fix env ut

* move ut directory

* make sure no null proxy resolver is returned by ProxyConfigurationResolverProvider

* minor adjustment

* add a few tests, still incomplete

* add proxy support for url table function

* use proxy for select from url as well

* remove optional from return type, just returns empty config

* fix style

* style

* black

* ohg boy

* rm in progress file

* god pls don't let me kill anyone

* ...

* add use_aws guards

* remove hard coded s3 proxy resolver

* add concurrency-mt-unsafe

* aa

* black

* add logging back

* revert change

* imrpove code a bit

* helper functions and separate tests

* for some reason, this env test is not working..

* formatting

* :)

* clangtidy

* lint

* revert some stupid things

* small test adjusmtments

* simplify tests

* rename test

* remove extra line

* freaking style change

* simplify a bit

* fix segfault & remove an extra call

* tightly couple proxy provider with context..

* remove useless include

* rename config prefix parameter

* simplify provider a bit

* organize provider a bit

* add a few comments

* comment out proxy env tests

* fix nullptr in unit tests

* make sure old storage proxy config is properly covered without global context instance

* move a few functions from class to anonymous namespace

* fix no fallback for specific storage conf

* change API to accept http method instead of bool

* implement http/https distinction in listresolver, any still not implemented

* implement http/https distinction in remote resolver

* progress on code, improve tests and add url function working test

* use protcol instead of method for http and https

* small fix

* few more adjustments

* fix style

* black

* move enum to proxyconfiguration

* wip

* fix build

* fix ut

* delete atomicroundrobin class

* remove stale include

* add some tests.. need to spend some more time on the design..

* change design a bit

* progress

* use existing context for tests

* rename aux function and fix ut

* ..

* rename test

* try to simplify tests a bit

* simplify tests a bit more

* attempt to fix tests, accept more than one remote resolver

* use proper log id

* try waiting for resolver

* proper wait logic

* black

* empty

* address a few comments

* refactor tests

* remove old tests

* baclk

* use RAII to set/unset env

* black

* clang tidy

* fix env proxy not respecting any

* use log trace

* fix wrong logic in getRemoteREsolver

* fix wrong logic in getRemoteREsolver

* fix test

* remove unwanted code

* remove ClientConfigurationperRequest and auxilary classes

* remove unwanted code

* remove adapter test

* few adjustments and add test for s3 storage conf  with new proxy settings

* black

* use chassert for context

* Add getenv comment
 2023-08-24 16:07:26 +03:00
Kseniia Sumarokova
a4f39fe134
Merge branch 'master' into fix-cache-logical-error 2023-08-24 12:00:18 +02:00
Raúl Marín
93dac0c880 Support clang-18 (Wmissing-field-initializers) 2023-08-23 15:53:45 +02:00
Kruglov Pavel
592fa77987
Merge branch 'master' into cache-count 2023-08-23 15:18:02 +02:00
kssenii
57b5ad854d Fxi 2023-08-23 13:43:05 +02:00
Sema Checherinda
60103c577d
Merge pull request #53651 from CheSema/read_beyon_last_offset 
fix Logical Error in AsynchronousBoundedReadBuffer
 2023-08-23 11:21:47 +02:00
avogar
cd228754c8 Apply suggestions 2023-08-22 17:32:35 +00:00
Kruglov Pavel
c0bdd0e00b
Merge branch 'master' into cache-count 2023-08-22 14:42:22 +02:00
avogar
b4145aeddc Cache number of rows in files for count in file/s3/url/hdfs/azure functions 2023-08-22 11:59:59 +00:00
Sema Checherinda
81577e041b fix Logical Error in AsynchronousBoundedReadBuffer 2023-08-21 19:30:04 +02:00
Kseniia Sumarokova
404fb0588a
Merge branch 'master' into fix-cache-logical-error 2023-08-20 16:28:13 +02:00
kssenii
5382442da0 Merge remote-tracking branch 'upstream/master' into allowed-caches-dir-for-dynamic-disks 2023-08-19 08:44:36 +04:00
Yakov Olkhovskiy
c73037a602 Merge remote-tracking branch 'ch/master' into unify_keep_alive_setting 2023-08-17 20:40:18 +00:00
Michael Kolupaev
557bfea4d8 Use long timeout for S3 copy requests 2023-08-17 19:36:45 +00:00
Kseniia Sumarokova
99f7a83385
Merge branch 'master' into allowed-caches-dir-for-dynamic-disks 2023-08-10 22:26:33 +02:00
Alexey Milovidov
bb57caa83e
Merge pull request #53180 from ClickHouse/ditch-tons-of-garbage 
Ditch tons of garbage
 2023-08-10 16:59:51 +03:00
Kseniia Sumarokova
e79f1ad724
Merge branch 'master' into allowed-caches-dir-for-dynamic-disks 2023-08-10 11:02:11 +02:00
Kseniia Sumarokova
c4c76f60a7
Merge branch 'master' into fix-cache-logical-error 2023-08-10 10:55:22 +02:00
Alexey Milovidov
a807ccec6c Merge remote-tracking branch 'origin/Retry_for_TransportException_Azure' into remove-gargabe-tests 2023-08-10 02:21:25 +02:00
Smita Kulkarni
901901a762 Fixed typo 2023-08-09 16:50:40 +02:00
Smita Kulkarni
9953050b62 Fixed logs for download 2023-08-09 14:57:17 +02:00
Smita Kulkarni
dc4be33895 Updated logs to include retries 2023-08-09 14:55:44 +02:00
Kseniia Sumarokova
944e85d4e5
Merge branch 'master' into fix-cache-logical-error 2023-08-09 12:08:53 +02:00
kssenii
2cfb4c0264 Merge remote-tracking branch 'upstream/master' into allowed-caches-dir-for-dynamic-disks 2023-08-09 11:09:34 +02:00
Alexey Milovidov
aa757490bd Ditch tons of garbage 2023-08-09 02:19:02 +02:00
Kseniia Sumarokova
f4553c2aa7
Update registerDiskCache.cpp 2023-08-08 17:06:41 +02:00
Smita Kulkarni
473fb60e6f Added retry for TransportException for azure read & download 2023-08-07 18:56:18 +02:00
kssenii
0c174a7ec1 Allow to restrict fs caches paths 2023-08-07 18:00:26 +02:00
Vitaly Baranov
7082cbf5c6 Use more unique name for TemporaryFileOnDisk. 2023-08-07 17:56:30 +02:00
kssenii
9cdd886fde Fix 2023-08-07 16:46:42 +02:00
Nikita Taranov
37d382aba4 impl 2023-08-05 20:30:49 +02:00
Kseniia Sumarokova
eabca22909
Merge branch 'master' into revert-revert-drop-cache 2023-08-01 23:19:38 +02:00
Vitaly Baranov
65bd8f5e81
Merge branch 'master' into backups-fix-tmp-files 2023-08-01 16:37:06 +02:00
Alexey Milovidov
c284696e0d
Merge branch 'master' into fix-dynamic-disk-conf 2023-08-01 08:22:52 +03:00
Nikolai Kochetov
c4a00b8c68 Throw S3Exception whenever possible. 2023-07-31 15:39:20 +00:00
kssenii
b683477b20 Better dynamic disk configuration 2023-07-31 14:15:22 +02:00
Alexander Gololobov
24bfd41680
Merge branch 'master' into read_empty_file 2023-07-31 09:41:37 +02:00
kssenii
6ae6268ad0 Revert "Merge pull request #51917 from ClickHouse/revert-system-drop-filesystem-cache-by-key" 
This reverts commit 520195f7ef, reversing
changes made to e36a638014.
 2023-07-30 13:39:54 +02:00
Alexey Milovidov
caa4590361 Merge branch 'master' into check-for-hiding-cyrillic-characters 2023-07-30 02:16:56 +02:00
Alexander Gololobov
84ab423842 Allow reading empty file with no blobs 2023-07-29 19:30:23 +02:00
robot-clickhouse-ci-1
66dd0faec8
Merge pull request #52733 from ClickHouse/zero 
Allow reading zero objects in CachedObjectStorage::readObjects()
 2023-07-29 17:52:32 +02:00
Nikita Mikhaylov
cecc636449
Update CachedObjectStorage.cpp 2023-07-28 19:35:22 +02:00
Alexander Tokmakov
39200606ec
Merge pull request #52174 from arenadata/ADQM-988 
Added possibility to save logs on crash and options to configure logs buffer
 2023-07-28 16:55:00 +03:00
Alexander Gololobov
bf9504d95d
Move UnlinkMetadataFileOperationOutcome to common header (#52710) 2023-07-28 15:49:55 +02:00
Michael Kolupaev
ff9e85529a Allow reading zero objects in CachedObjectStorage::readObjects() 2023-07-28 13:43:15 +00:00
Alexey Gerasimchuck
63b05da1f2 System logs improvements 2023-07-28 07:23:34 +00:00
Azat Khuzhin
d40dbdee62 Fix usage of temporary directories during RESTORE 
Previously during RESTORE you may get the following warning in logs:

    <Warning> TemporaryFileOnDisk: Temporary path 'tmp/21672kaaaaa' does not exist in './disks/s3_common_disk/'

The reason is that it uses not disks/s3_common_disk/tmp but instead
disks/s3_common_disk/disks/s3_common_disk/tmp path.

Fix this by adding TemporaryFileOnDisk::getRelativePath() and use it
when appropriate.

And now - `find disks`, does not shows any more temporary leftovers.

v2: rename getPath to getAbsolutePath
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-27 19:24:54 +02:00
Alexander Tokmakov
f3dc6dd061
Revert "Added field refcount to system.remote_data_paths table" 2023-07-27 15:23:57 +03:00
Alexey Milovidov
6aab4cc835 Check for unexpected cyrillic 2023-07-27 05:25:40 +02:00
Anton Popov
568afbbec3 added field with refcount to system.remote_data_paths table 2023-07-25 12:33:15 +00:00
Kseniia Sumarokova
a1782ea70e
Merge pull request #52211 from kssenii/fix-race-in-disk-web 
Fix race in Web disk
 2023-07-23 18:07:25 +02:00
Alexey Milovidov
2467eaf32d
Merge pull request #52392 from azat/netlink-provider 
Rename TaskStatsInfoGetter into NetlinkMetricsProvider
 2023-07-22 19:14:15 +03:00
Alexey Milovidov
a092c17313
Merge branch 'master' into fix-race-in-disk-web 2023-07-22 16:53:27 +03:00
robot-clickhouse-ci-2
0db9c79886
Merge pull request #52332 from rschu1ze/better-formatsettings 
Minor: Less awkward IAST::FormatSettings
 2023-07-21 14:49:58 +02:00
Azat Khuzhin
f0e277f94a Rename TaskStatsInfoGetter into NetlinkMetricsProvider 
There is ProcfsMetricsProvider, so by analogy to it.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-21 06:48:18 +02:00
Alexander Tokmakov
a39ba00ec3
Merge pull request #52116 from aalexfvk/http_failover 
Handle multiple resolved IPs for HTTP/HTTPS session connection
 2023-07-20 18:25:45 +03:00
Robert Schulze
bd761c365a
Make serializeAST() more regular 2023-07-20 10:39:26 +00:00
vdimir
b9009f0e44
Merge pull request #52275 from ClickHouse/vdimir/tmp_data_add_test 2023-07-20 11:34:13 +02:00
Antonio Andelic
0c86df519f Fix unspported disks in Keeper 2023-07-19 12:41:25 +00:00
vdimir
25eee81746
Fix temp data deletion on startup, add test 2023-07-18 16:29:21 +00:00
Kseniia Sumarokova
d11ded952e
Update WebObjectStorage.cpp 2023-07-17 16:09:11 +02:00
kssenii
13b493a353 Add comment 2023-07-17 15:53:47 +02:00
kssenii
988b46e3f0 Fix 2023-07-17 15:45:26 +02:00
Aleksei Filatov
08defa36b2 Add code 2023-07-15 08:18:35 +03:00
robot-ch-test-poll4
139f59f51f
Merge pull request #52038 from azat/s3-plain-send_metadata 
Prohibit send_metadata for s3_plain disks
 2023-07-12 00:37:50 +02:00
Azat Khuzhin
326d51c3fb Prohibit send_metadata for s3_plain disks 
send_metadata changes the filenames (includes revision), while
s3_plain do not care about this, and expect that the file name
will not be changed.

So during initialization s3_plain will do some basic checks and will
write file clickhouse_access_check_e0f72fcc-d6f8-44fe-8997-96cb87767515, but instead
r0000000000000000000000000000000000000000000000000000000000000001-file-clickhouse_access_check_e0f72fcc-d6f8-44fe-8997-96cb87767515
will be written, later read will not find the file and it will fail:

    Application: Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: While checking access for disk s3_plain: Cannot attach table `system`.`crash_log` ...

And besides, send_metadata does not make sense for s3_plain.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-11 18:34:23 +02:00
Nikolai Kochetov
ab6356f9bc
Merge branch 'master' into vdimir/fix_race_in_remove_objects 2023-07-11 11:22:04 +02:00
vdimir
bd5f393515
Always remove blobs in MetadataStorageFromPlainObjectStorageTransaction::unlinkMetadata 2023-07-11 05:14:39 +00:00
Alexey Milovidov
3d4800995f
Merge pull request #49732 from nickitat/impr_prefetch 
Improve reading with prefetch
 2023-07-09 06:10:58 +03:00
Alexey Milovidov
b1d1918607
Merge branch 'master' into retry 2023-07-08 03:08:48 +03:00
Alexey Milovidov
03d2606eb9
Merge pull request #49342 from ClickHouse/disks-space-improvement 
Slight improvement in Disks interface
 2023-07-08 02:24:57 +03:00
Alexey Milovidov
520195f7ef
Merge pull request #51917 from ClickHouse/revert-system-drop-filesystem-cache-by-key 
Revert system drop filesystem cache by key
 2023-07-07 19:44:28 +03:00
vdimir
88911e1378
Check refcount in finalize for other RemoveObjectStorageOperations 2023-07-07 16:42:03 +00:00
vdimir
227e415d6d
Check refcount in RemoveManyObjectStorageOperation::finalize instead of execute 2023-07-07 15:08:21 +00:00
Alexey Milovidov
4a6a6fc4f2
Merge branch 'master' into disks-space-improvement 2023-07-07 13:43:04 +03:00
xiaolei565
5f523e8376
Merge branch 'master' into xuelei_dev 2023-07-07 10:21:19 +08:00
Alexey Milovidov
7cece62d26 Revert "Merge pull request #51547 from kssenii/more-flexible-drop-cache" 
This reverts commit 2ce7bcaa3d, reversing
changes made to e897207cd5.
 2023-07-07 04:07:19 +02:00
Nikolay Degterinsky
8da8b79cc4
Merge pull request #51854 from fky2015/master 
fix: correct exception messages on policies comparison
 2023-07-07 03:58:06 +02:00
xiaolei565
dd87269fc2
Merge branch 'master' into xuelei_dev 2023-07-07 09:49:34 +08:00
Alexander Tokmakov
fdbd59ec6c
Merge pull request #51135 from ClickHouse/fix_trash_in_disks 
Try to fix some trash in Disks and part moves
 2023-07-07 02:02:17 +03:00
xiaolei565
0e4192f590
Merge branch 'master' into xuelei_dev 2023-07-06 10:35:22 +08:00
Alexey Milovidov
2e454ef89e
Merge branch 'master' into disks-space-improvement 2023-07-06 03:58:45 +03:00
Feng Kaiyu
f7640ff573
fix: correct exception message on policies comparison 2023-07-06 01:27:20 +08:00
Kseniia Sumarokova
f2d942fbb5
Merge branch 'master' into more-flexible-drop-cache 2023-07-05 11:43:57 +02:00
xiaolei565
2c5fc5ee13
Merge branch 'master' into xuelei_dev 2023-07-05 16:33:42 +08:00