thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 18:32:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,518 Commits 9,328 Branches 1,783 Tags 2.1 GiB
8d984df135
Commit Graph

2265 Commits

Author SHA1 Message Date
kssenii
42e16bcc49 SLRU for filesystem cache 2023-11-21 21:06:32 +01:00
vdimir
a139ae97eb
Merge pull request #52918 from ClickHouse/vdimir/s3_blob_log 
Add system table with blob storage operations log
 2023-11-21 17:40:42 +01:00
Alexander Gololobov
8cf93fe133
Merge pull request #56905 from ClickHouse/disk_encrypted_create_dirs 
Allow delegate disk to handle retries for createDirectories
 2023-11-21 12:51:05 +01:00
Kseniia Sumarokova
d384762123
Merge branch 'master' into fs-cache-improvement 2023-11-21 11:24:52 +01:00
Jordi Villar
6bd20abb90 Add CachedReadBufferReadFromCache{Hits,Misses} profile events 2023-11-21 10:34:12 +01:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
Sema Checherinda
f999337dae
Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
Alexander Tokmakov
5031f239c3
Revert "s3 adaptive timeouts" 2023-11-20 14:28:59 +01:00
Sema Checherinda
a950595c24
Merge pull request #56314 from CheSema/s3-aggressive-timeouts 
s3 adaptive timeouts
 2023-11-19 14:12:14 +01:00
Alexey Milovidov
d56cbda185 Add metrics for the number of queued jobs, which is useful for the IO thread pool 2023-11-18 19:07:59 +01:00
Alexander Gololobov
6c3793acb0 Allow delegate disk to handle retries for createDirectories 2023-11-17 12:03:00 +01:00
Aleksei Filatov
163b8a814e Fix clang-tidy issues and disable parallel execution of tests 2023-11-17 11:02:07 +03:00
Kseniia Sumarokova
a2ed756eec
Merge branch 'master' into fs-cache-improvement 2023-11-16 17:49:26 +01:00
kssenii
472cfdc86d Review fix 2023-11-16 17:47:51 +01:00
Aleksei Filatov
21a17f8334 Add cancellation hook for moving background operation 2023-11-16 13:37:51 +03:00
Sema Checherinda
6e3e6383ba perf check 2 2023-11-15 19:00:27 +01:00
MikhailBurdukov
f29c072429 Test fix 2023-11-15 09:33:40 +00:00
Sema Checherinda
b9cc1580e0
Merge pull request #56367 from canhld94/ch_table_reinit_new_disk 
Adding new disk to storage configuration without restart
 2023-11-14 15:54:22 +01:00
MikhailBurdukov
1aea8186c7 Enable s3 native copy for scenario: from s3 to s3. 2023-11-14 13:53:48 +00:00
Sema Checherinda
8d36fd6e54 get rid off of client_with_long_timeout_ptr 2023-11-14 11:34:12 +01:00
Sema Checherinda
27fb25d056 alter the naming, fix client_with_long_timeout in s3 storage 2023-11-14 11:34:12 +01:00
Sema Checherinda
be01a5cd3e turn off agressive timeouts for heavy requests 2023-11-14 11:34:12 +01:00
Sema Checherinda
ab2594154e add tests 2023-11-14 11:34:12 +01:00
Kseniia Sumarokova
e428e4ed7a
Merge branch 'master' into cache-config-improvement 2023-11-13 10:09:58 +01:00
Sema Checherinda
3c15a5e714 hide VERSION_INLINE_DATA under storage_metadata_write_full_object_key feature 2023-11-10 17:24:50 +01:00
kssenii
c28c1ac4a2 Allow getting cache configuration from named collection 2023-11-09 18:39:16 +01:00
kssenii
dbea50738b Merge remote-tracking branch 'origin/master' into fs-cache-improvement 2023-11-08 11:53:54 +01:00
Nikita Taranov
c52571223e better 2023-11-07 22:04:13 +01:00
Sema Checherinda
8216a24abe move storage_metadata_write_full_object_key setting to the server scope 2023-11-07 17:14:46 +01:00
Nikita Taranov
1f2f51844f smart fix 2023-11-06 17:32:11 +01:00
Duc Canh Le
4c21ba7b6f tables auto initialize new disks without restart 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-11-06 10:26:48 +00:00
Nikita Taranov
5040a3b6d3 do seek before calling isContentCached() 2023-11-03 22:52:12 +01:00
Nikita Taranov
7046832056 change signatures 2023-11-03 22:50:03 +01:00
Sema Checherinda
8958861f99 review notes 2023-11-03 22:43:40 +01:00
Sema Checherinda
a108f9f764
Update src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-11-03 14:54:48 +01:00
Sema Checherinda
890cc8bbca
Update src/Disks/ObjectStorages/DiskObjectStorageMetadata.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-11-03 14:54:40 +01:00
Sema Checherinda
bab867f383
Update src/Disks/ObjectStorages/DiskObjectStorageMetadata.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-11-03 14:54:10 +01:00
Sema Checherinda
90b64bcdb9 backward compatibility and implementation feature storage_metadata_write_full_object_key 2023-11-02 15:20:55 +01:00
Kseniia Sumarokova
e5efc42633
Merge branch 'master' into fs-cache-improvement 2023-11-01 10:47:36 +01:00
Nikita Taranov
1d58cedce5 better 2023-10-31 22:48:20 +01:00
Azat Khuzhin
c7f392500e Fix incorrect free space accounting for least_used JBOD policy 
Before least_used fails to detect when the disk started to have more
space, it works only when the disk starts to have less space.

The reason for this is that it uses priority_queue, and once the disk
goes at the bottom of the queue, free space will not be updated for it
until it will be selected again.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-29 17:36:31 +01:00
Nikita Taranov
213ce63ffb
Merge branch 'master' into optimize_reading3 2023-10-28 00:04:20 +02:00
Nikita Taranov
c97bfbeb92 fix tests 2023-10-27 13:47:13 +02:00
Nikita Taranov
a86bda5208 use local thread pool reader for cache segments 2023-10-24 14:06:13 +02:00
Nikita Taranov
9b13c796e3 set ignore metric in all readers 2023-10-20 22:24:56 +02:00
Nikita Taranov
2a6a97c1fa read sync from cache 2023-10-20 22:24:56 +02:00
Alexander Tokmakov
f899254e2c
Merge pull request #55430 from arthurpassos/properly_split_remote_proxy_http_https 
Properly split remote proxy http https
 2023-10-20 21:15:34 +02:00
kssenii
7515853ad4 Fix build 2023-10-17 11:43:51 +02:00
kssenii
43c5e173de Merge remote-tracking branch 'upstream/master' into fs-cache-improvement 2023-10-16 13:06:28 +02:00
alesapin
3b02748cb6 Fix some typos 2023-10-15 15:43:02 +02:00
alesapin
21e0cb0b18
Merge pull request #55468 from ClickHouse/fix_out_of_bound_disk_web 
Fix out of bound error in system.remote_data_paths + disk web
 2023-10-11 12:54:00 +02:00
Alexey Milovidov
5ad055df98
Merge pull request #55372 from azat/web-disk-fix-race 
Fix data-race in web disk
 2023-10-11 01:03:48 +02:00
Azat Khuzhin
8021d8d747 Fix data-race in web disk 
TSan report:

    WARNING: ThreadSanitizer: data race (pid=8)
     Write of size 8 at 0x7b54002b0400 by thread T310:
       0 DB::ReadBufferFromWebServer::seek(long, int) build_docker/./src/Disks/IO/ReadBufferFromWebServer.cpp:159:12 (clickhouse+0x18a6a45b) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       1 DB::CachedOnDiskReadBufferFromFile::getImplementationBuffer(DB::FileSegment&) build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:471:47 (clickhouse+0x17f1aace) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       2 DB::CachedOnDiskReadBufferFromFile::nextImplStep() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:844:33 (clickhouse+0x17f1f5af) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       3 DB::CachedOnDiskReadBufferFromFile::nextImpl() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:774:16 (clickhouse+0x17f1f33a) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       4 DB::ReadBuffer::next() build_docker/./src/IO/ReadBuffer.h:70:20 (clickhouse+0x17f1307f) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       5 DB::ReadBufferFromRemoteFSGather::readImpl() build_docker/./src/Disks/IO/ReadBufferFromRemoteFSGather.cpp:188:32 (clickhouse+0x17f1307f)
       6 DB::ReadBufferFromRemoteFSGather::nextImpl() build_docker/./src/Disks/IO/ReadBufferFromRemoteFSGather.cpp:160:9 (clickhouse+0x17f12f89) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       7 DB::ReadBuffer::next() build_docker/./src/IO/ReadBuffer.h:70:20 (clickhouse+0x17d695ca) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       8 DB::ThreadPoolRemoteFSReader::execute(DB::IAsynchronousReader::Request) build_docker/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:95:25 (clickhouse+0x17d695ca)
       9 DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0::operator()() const build_docker/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:70:80 (clickhouse+0x17d6ab48) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)

     Previous read of size 8 at 0x7b54002b0400 by thread T297:
       0 DB::ReadBufferFromWebServer::getFileOffsetOfBufferEnd() const build_docker/./src/Disks/IO/ReadBufferFromWebServer.h:36:63 (clickhouse+0x18a6ac9d) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       1 DB::CachedOnDiskReadBufferFromFile::getInfoForLog() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:1223:71 (clickhouse+0x17f24835) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       2 DB::CachedOnDiskReadBufferFromFile::nextImplStep()::$_0::operator()() const build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:799:5 (clickhouse+0x17f2347c) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       3 BasicScopeGuard<DB::CachedOnDiskReadBufferFromFile::nextImplStep()::$_0>::invoke() build_docker/./base/base/../base/scope_guard.h:99:9 (clickhouse+0x17f2347c)
       4 BasicScopeGuard<DB::CachedOnDiskReadBufferFromFile::nextImplStep()::$_0>::~BasicScopeGuard() build_docker/./base/base/../base/scope_guard.h:48:26 (clickhouse+0x17f2347c)
       5 DB::CachedOnDiskReadBufferFromFile::nextImplStep() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:1071:1 (clickhouse+0x17f22480) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       6 DB::CachedOnDiskReadBufferFromFile::nextImpl() build_docker/./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:774:16 (clickhouse+0x17f1f33a) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       7 DB::ReadBuffer::next() build_docker/./src/IO/ReadBuffer.h:70:20 (clickhouse+0x17f1307f) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       8 DB::ReadBufferFromRemoteFSGather::readImpl() build_docker/./src/Disks/IO/ReadBufferFromRemoteFSGather.cpp:188:32 (clickhouse+0x17f1307f)
       9 DB::ReadBufferFromRemoteFSGather::nextImpl() build_docker/./src/Disks/IO/ReadBufferFromRemoteFSGather.cpp:160:9 (clickhouse+0x17f12f89) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       10 DB::ReadBuffer::next() build_docker/./src/IO/ReadBuffer.h:70:20 (clickhouse+0x17d695ca) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)
       11 DB::ThreadPoolRemoteFSReader::execute(DB::IAsynchronousReader::Request) build_docker/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:95:25 (clickhouse+0x17d695ca)
       12 DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0::operator()() const build_docker/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:70:80 (clickhouse+0x17d6ab48) (BuildId: cb41845e0edd51bbbf1abf7c1716e74d12491aae)

CI: https://s3.amazonaws.com/clickhouse-test-reports/55261/de503f75dcbc5a4d0e7fbb2e6b08c2106d62848a/integration_tests__tsan__[5_6].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-10 21:11:57 +02:00
alesapin
3e2c9c662c Fix tabulation 2023-10-10 17:09:26 +02:00
alesapin
37f5c445b4 Fix out of bound error in disk web 2023-10-10 15:44:27 +02:00
alesapin
666c690b4f
Merge pull request #55309 from ClickHouse/fix_redundant_check 
Fix bug with inability to drop detached partition in replicated merge tree on top of S3 without zero copy
 2023-10-10 11:48:00 +02:00
Arthur Passos
63562cbb20 Merge branch 'master' into properly_split_remote_proxy_http_https 2023-10-09 17:58:29 -03:00
alesapin
89294c0765 Review fixes: make check consistent in all write functions 2023-10-09 12:06:35 +02:00
alesapin
08f4621b48 Fix for write-once storages 2023-10-08 12:20:31 +02:00
Alexey Milovidov
f5b0fb27a4
Update DiskObjectStorageMetadata.cpp 2023-10-08 00:53:33 +03:00
alesapin
aa130b74e5 Fix offsets bug 2023-10-07 21:18:23 +02:00
kssenii
74a34a73de Merge remote-tracking branch 'upstream/master' into fs-cache-improvement 2023-10-07 13:09:42 +02:00
Arthur Passos
d54186df30 tmp 2023-10-04 17:21:41 -03:00
Sema Checherinda
a02bf94812 remove the bloker to grow the metadata file version 2023-10-03 17:37:56 +02:00
Nikita Taranov
97e705325b
Optimise memory consumption when external group by generates ton of temporary files (#54798) 
* impl

* fix

* fix test
 2023-10-02 13:09:40 +02:00
Alexey Milovidov
5ebd8ef41d Remove Meilisearch 2023-09-30 06:02:38 +02:00
kssenii
a72541d93f Improvement for big reads 2023-09-29 19:03:00 +02:00
Robert Schulze
9fff447716
Re-enable clang-tidy checks 2023-09-26 09:34:12 +00:00
Azat Khuzhin
c706101891 Fix throttling of BACKUPs from/to S3 (in case native copy was not used) 
In some cases native copy is not possible, and such requests should be
throttled.

v0: copyS3FileNativeWithFallback
v2: revert v0 and pass write_settings
v3: pass read_settings to copyFile()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-20 18:28:43 +02:00
Sema Checherinda
e7550523c8
Merge pull request #54651 from CheSema/limit_backoff_timeout 
limit the delay before next try in S3
 2023-09-18 19:41:26 +02:00
Robert Schulze
f5e8028bb1
Merge pull request #54642 from rschu1ze/broken-re2st 
Remove broken lockless variant of re2
 2023-09-17 15:30:57 +02:00
Kseniia Sumarokova
79697d4102
Update WebObjectStorage.cpp 2023-09-15 20:18:36 +02:00
Sema Checherinda
d9e15c00c9 limit the delay before next try in S3 2023-09-14 19:45:07 +02:00
Robert Schulze
7b378dbad3
Remove broken lockless variant of re2 2023-09-14 16:40:42 +00:00
Sema Checherinda
0ea0799118
Merge pull request #54430 from CheSema/remove-recursive-with-projection 
reproduce and fix the bug in removeSharedRecursive
 2023-09-12 00:18:55 +02:00
Sema Checherinda
d5268f3fb0 rename a member 2023-09-11 19:57:16 +02:00
Sergei Trifonov
08bad4d440
Merge pull request #47009 from ClickHouse/disks-and-io-sheduler 2023-09-11 07:22:14 +02:00
serxa
d326554caa fix keeper build 2023-09-10 14:55:40 +00:00
Sema Checherinda
1298dba0c6 fix the bug, add useful logs 2023-09-09 15:54:28 +02:00
SmitaRKulkarni
f0146c0ff0
Increase retries for test_merge_tree_azure_blob_storage (#54069) 
* Increase max_single_download_retries & max_single_read_retries for test_merge_tree_azure_blob_storage

* Updated logs to log exception leading to detaching part

* Updated to catch RequestFailedException azure exception as its the base exception

* Updated isRetryableExcepion to take std::exception_ptr as input

* Fixed style check fails

* Fix build issue & moved NetException & TimeoutException to retryable exceptions

* Fixed style check with catch
 2023-09-06 19:57:59 +02:00
Yakov Olkhovskiy
0dd6928a13
Merge pull request #53068 from nickitat/unify_keep_alive_setting 
Unify setting http keep-alive timeout, increase default to 30s
 2023-09-05 22:05:23 -04:00
Sergei Trifonov
258da91b5f
Merge branch 'master' into disks-and-io-sheduler 2023-09-04 15:14:28 +02:00
Kseniia Sumarokova
b3319f7908
Minor changes (#54171) 2023-09-03 15:47:52 +02:00
Michael Kolupaev
8505b3c940
Merge pull request #54134 from kssenii/refactor-reading-from-remote-fs 
Small refactoring for read from object storage
 2023-09-02 11:13:03 -07:00
kssenii
82948ac843 Review fixes 2023-09-02 14:00:07 +02:00
Sergei Trifonov
cf5ea46713
Merge pull request #53338 from azat/throttling-fixes 
Fix IO throttling during copying whole directories
 2023-09-01 18:29:15 +02:00
serxa
ef4dcf909f fix build 2023-09-01 16:23:15 +00:00
kssenii
9eb1dfcd12 Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
kssenii
3f20d1f578 Rename 2023-09-01 12:55:58 +02:00
serxa
9c92d413b3 Merge branch 'master' into disks-and-io-sheduler 2023-08-30 11:18:49 +00:00
Azat Khuzhin
51e8b16916 Fix IO throttling during copying whole directories 
v2: pass local context during ALTER TABLE MOVE PARTITION TO VOLUME/DISK as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-29 23:13:46 +02:00
xuelei
ebc7dbb628 The configuration of S3 append '/' automatically if needed 2023-08-28 09:39:22 -03:00
xuelei
636e7ddae7 deal with conflict 2023-08-28 06:05:21 -03:00
xiaolei565
853551bd3e
Update registerDiskS3.cpp 2023-08-28 12:38:33 +08:00
xiaolei565
ef6e1fd2d6
Update diskSettings.cpp 2023-08-28 12:38:08 +08:00
xiaolei565
58a1160095
Update diskSettings.cpp 2023-08-28 12:37:54 +08:00
xiaolei565
8d2c03ea26
Update registerDiskS3.cpp 2023-08-28 11:41:53 +08:00
xiaolei565
b9e872ddf7
Update diskSettings.cpp 2023-08-28 11:28:31 +08:00
xiaolei565
3c457c47c2
Update diskSettings.cpp 2023-08-28 11:05:30 +08:00
xuelei
643dd499b8 deal with conflict 2023-08-27 23:46:53 -03:00