Robert Schulze
5fb8ea4c62
Merge remote-tracking branch 'ClickHouse/master' into qiangxuhui-loongarch64
2024-05-17 07:56:01 +00:00
kssenii
af6f124df0
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-05-16 16:02:35 +02:00
Julia Kartseva
acbe0cab0f
Add endpoint_subpath S3 setting
...
Multiple plain_rewritable disks can share the same
endpoint, e.g., in the cloud system tables configuration.
At startup, a disk traverses the contents of a bucket to construct the
local-to-remote plain_rewritable mapping. A data race occurs when
another disk writes to the same bucket while the first disk is starting
up.
Introduce `endpoint_subpath` S3 setting, that will be unique for each
Pod instance in the cloud scenario.
2024-05-15 03:00:08 +00:00
kssenii
65f404c153
Review fixes
2024-05-14 22:08:50 +02:00
kssenii
0abb2be5eb
Review fixes
2024-05-14 18:40:09 +02:00
qiangxuhui
97a5de9653
Initial support for loongarch64
...
Make ClickHouse compilable and runnable on loongarch64
So far only basic functionality was tested (on real hw),
clickhouse server runs, exceptions works, client works,
simple tests works.
2024-05-14 07:45:26 +00:00
kssenii
904800afc8
Apply recent changes to storages3/hdfs/azure
2024-05-13 17:44:14 +02:00
kssenii
9ce465bb88
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-05-13 16:14:58 +02:00
MikhailBurdukov
e2b39db242
Fix
2024-05-13 14:14:35 +00:00
MikhailBurdukov
cfa0aa772e
Merge branch 'master' into truncate_for_obj_storage_disks
2024-05-13 16:05:44 +03:00
MikhailBurdukov
7d0214018c
Enable truncate operation for object storage disks.
2024-05-13 12:46:42 +00:00
Alexey Milovidov
dd58af7d4f
Merge branch 'master' of github.com:ClickHouse/ClickHouse into clang-18-ci
2024-05-10 07:17:39 +02:00
Alexey Milovidov
c17a3bb944
Useless changes
2024-05-10 03:31:40 +02:00
tomershafir
b97bf456c6
try fix build
2024-05-09 20:53:36 +03:00
Antonio Andelic
bc9d50cf16
Merge pull request #63523 from ClickHouse/leftover-threads-copy
...
Correctly stop `asyncCopy` tasks in all cases
2024-05-09 06:50:51 +00:00
Alexey Milovidov
c198bbcafb
Useless changes
2024-05-09 04:20:54 +02:00
Alexey Milovidov
224e1940ae
Useless changes
2024-05-09 03:58:34 +02:00
Alexey Milovidov
0d14a2c67e
Useless changes
2024-05-09 03:11:02 +02:00
Alexey Milovidov
95f12ef274
Useless changes
2024-05-09 01:08:33 +02:00
Antonio Andelic
3d5c8db1e0
Use ThreadPoolCallbackRunnerLocal
2024-05-08 15:12:52 +02:00
Antonio Andelic
16eb12a321
Correctly stop copy tasks when exception thrown
2024-05-08 14:58:21 +02:00
Kseniia Sumarokova
35d96f98f4
Merge pull request #63484 from ClickHouse/fix-custom-web-disk-with-config-reload
...
Fix logical error when reloading config with customly created web disk broken after #56367
2024-05-08 09:28:54 +00:00
kssenii
836cf150b5
Fix
2024-05-07 19:00:30 +02:00
tomershafir
f57abbd806
add missing include
2024-05-07 17:27:52 +03:00
tomershafir
ed63ad5e61
iouring: refactor get from context
2024-05-07 14:10:49 +03:00
kssenii
cd093ca793
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-05-07 12:50:48 +02:00
Julia Kartseva
0a1d852dfd
Enable plain_rewritable metadata for local and azure
...
Enable plain_rewritable support for local and azure (azure_blob_storage)
metadata type.
- HDFS object storage currently does not support iteration
and does not implement listObjects method. It's a blocker
for enabling plain_rewritable metadata type with HDFS.
- StaticWeb object storage is read-only and works with
its dedicated metadata type.
2024-05-07 03:53:23 +00:00
kssenii
a51786532b
Remove copy
2024-05-06 13:21:11 +02:00
Alexey Milovidov
7ac5821a23
Merge pull request #63074 from ClickHouse/local-less-directories
...
Less directories in clickhouse-local
2024-05-05 00:19:31 +02:00
kssenii
e9d9f2bc25
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-05-03 13:55:12 +02:00
alesapin
15954b5b5c
Merge pull request #63257 from jkartseva/fix-inf-loop-azure-list-objects
...
Fix infinite loop in AzureObjectStorage::listObjects
2024-05-03 09:39:00 +00:00
Julia Kartseva
dd7e88b37e
fix infinite loop in AzureObjectStorage::listObjects
...
While the exit condition is correct for `AzureIteratorAsync`,
listObject may never exit.
This is because BlobContainerClient::LstBlobs sets a page, thus
making a sequential HasPage() condition true.
This issue reproduces when passing an empty path.
Tested with the following integration test:
https://pastila.nl/?01691b54/9ceaf103c91ab9e38ea3541abfcd1ae6#hUtrrz5sFHcBSHCLS9AlVA==
(will be added when plain_rewritable is enabled for all disk types).
2024-05-03 01:56:23 +00:00
Alexey Milovidov
ed1bbd0bd0
Merge branch 'master' into local-less-directories
2024-05-03 02:35:44 +02:00
Smita Kulkarni
37c618289b
Merge branch 'master' into add_azure_workload_identity_support_blob
2024-05-02 14:58:56 +02:00
Sema Checherinda
083bcfc1ce
distinct message for s3 error 'no key' for cases disk and storage
2024-05-02 13:08:25 +02:00
Alexey Milovidov
817570c5e4
Merge branch 'master' into local-less-directories
2024-04-30 04:45:33 +02:00
Julia Kartseva
dc95558962
method rename
2024-04-29 19:48:17 +00:00
Julia Kartseva
3c1207ed4d
remove path normalization
2024-04-29 18:01:21 +00:00
Julia Kartseva
508a42bc8f
use ordered map for path map
2024-04-29 18:01:21 +00:00
Julia Kartseva
24d5abba10
extract plain_rewritable metadata type
...
Make PlainRewritableObjectStorage generic;
Support config type
```
<s3_plain_rewritable>
<type>object_storage</type>
<object_storage_type>s3</object_storage_type>
<metadata_type>plain_rewritable</metadata_type>
<endpoint>https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/ </endpoint>
<use_environment_credentials>1</use_environment_credentials>
</s3_plain_rewritable>
```
2024-04-29 18:01:21 +00:00
Julia Kartseva
d1217af389
address feedback - pt.4
2024-04-29 18:01:21 +00:00
Julia Kartseva
802ee27b1b
address feedback - pt.3
...
non-functional changes
2024-04-29 18:01:21 +00:00
Julia Kartseva
4a7f28f6bd
address feedback - pt.2
2024-04-29 18:01:21 +00:00
Julia Kartseva
36a1cae910
address feedback - pt.1
2024-04-29 18:01:21 +00:00
Julia Kartseva
4f6a3e27b7
Update src/Disks/ObjectStorages/MetadataStorageFromPlainObjectStorageOperations.cpp
...
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
2024-04-29 18:01:21 +00:00
Julia Kartseva
70d55aa618
Update src/Disks/ObjectStorages/CommonPathPrefixKeyGenerator.h
...
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
2024-04-29 18:01:21 +00:00
Julia Kartseva
89f28f3c18
explicitly disallow ALTERs and mutations for plain
2024-04-29 18:01:21 +00:00
Julia Kartseva
01ee500b06
improvements, cleanups, comments
2024-04-29 18:01:21 +00:00
Julia Kartseva
a67a8299ce
Do not list prefix.path in listDirectory
2024-04-29 18:01:21 +00:00
Julia Kartseva
d1e5a09b18
better transaction rollback
2024-04-29 18:01:21 +00:00
Julia Kartseva
b4375131cb
concurrency control
2024-04-29 18:01:21 +00:00
Julia Kartseva
7916792baa
Do not create directory metadata recursively
2024-04-29 18:01:21 +00:00
Julia Kartseva
75147f3ac7
S3-plain based disk supporting directory rename
...
Fixes: https://github.com/ClickHouse/ClickHouse/issues/58347
2024-04-29 18:01:21 +00:00
kssenii
7a416f2e76
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-04-28 12:19:32 +02:00
Alexey Milovidov
64fff163f7
Less directories
2024-04-27 19:54:37 +02:00
Alexey Milovidov
6a083ebed0
Even less directories
2024-04-27 19:13:17 +02:00
kssenii
91014456b5
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-04-26 19:24:07 +02:00
kssenii
434d2d16f1
Cleanuo
2024-04-26 19:23:33 +02:00
Smita Kulkarni
f474c29b4d
Fix build with new master
2024-04-26 15:33:27 +02:00
kssenii
18e4c0f1da
Fix remaining integration test
2024-04-26 13:35:18 +02:00
Smita Kulkarni
3ace11f92d
Merge branch 'master' into Azure_write_buffer_parallel_upload
2024-04-26 10:27:41 +02:00
liuneng
76b989ebbc
fix missing-field-initializers
2024-04-24 17:13:08 +08:00
Raúl Marín
3df253cb0a
Merge pull request #62850 from danipozo/fix-building-endpoint-from-storage-account-url
...
Azure: fix building final url from endpoint object
2024-04-23 17:26:22 +00:00
Dani Pozo
771219a118
Apply suggestion to use pop_back
2024-04-23 12:34:26 +02:00
Dani Pozo
c75574a1d6
Azure: fix building final url from endpoint object
2024-04-22 18:49:46 +02:00
kssenii
a4daf2b454
Fix hdfs race
2024-04-20 14:46:32 +01:00
Smita Kulkarni
065eceecbf
Merge branch 'master' into Azure_write_buffer_parallel_upload
2024-04-19 17:37:15 +02:00
kssenii
e2e6071063
Fix a few more tests
2024-04-19 13:44:07 +01:00
Smita Kulkarni
613dd1a6c0
Updated to use workload identity for backups & disk
2024-04-18 18:26:07 +02:00
kssenii
c8915a16a5
Fix a few mote tests
2024-04-18 17:22:51 +01:00
Smita Kulkarni
5846e2c8ab
Merge branch 'master' into add_azure_workload_identity_support_blob
2024-04-18 18:11:16 +02:00
Anton Popov
98d47e4ef5
Update src/Disks/ObjectStorages/AzureBlobStorage/AzureBlobStorageAuth.cpp
...
Co-authored-by: SmitaRKulkarni <Smita.Kulkarni@clickhouse.com>
2024-04-18 12:12:14 +02:00
Anton Popov
b50117780c
Merge remote-tracking branch 'upstream/master' into HEAD
2024-04-17 13:05:46 +00:00
kssenii
11be538ac8
Fix several tests
2024-04-16 14:09:33 +01:00
kssenii
652796acd6
Fix MergeTree with HDFS
2024-04-15 16:37:38 +01:00
kssenii
27672e1dfc
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-04-15 11:28:14 +01:00
Anton Popov
03916cc13f
fix build and update submodule
2024-04-12 18:26:30 +00:00
Alexander Gololobov
85cd0291a5
Remove getRemotePathsRecursive() from IDisk
2024-04-12 18:34:51 +02:00
Anton Popov
47c653f87c
better retries in azure sdk
2024-04-12 15:39:14 +00:00
Alexander Tokmakov
d510c3ac55
Merge pull request #62266 from ClickHouse/cannot_allocate_thread
...
Add fault injection for "Cannot allocate thread"
2024-04-12 14:40:42 +00:00
Anton Popov
523ee302cb
more profile events
2024-04-11 16:37:54 +00:00
Smita Kulkarni
b8c77a4453
Support parallel write buffer for AzureBlobStorage
2024-04-11 10:13:41 +02:00
Alexander Tokmakov
d8e97b51bf
Merge branch 'master' into cannot_allocate_thread
2024-04-10 21:21:42 +02:00
Anton Popov
8d813b6594
Merge remote-tracking branch 'upstream/master' into azure-profile-events
2024-04-10 14:40:54 +00:00
Anton Popov
4a8c1bf6a3
Merge pull request #62306 from CurtizJ/azure-retryable-errors
...
Better handling of errors from azure storage
2024-04-10 13:38:33 +00:00
Anton Popov
52635d2b8f
add profile events for azure disk
2024-04-10 01:29:09 +00:00
Robert Schulze
4035334634
Merge remote-tracking branch 'rschu1ze/master' into static-instance2
2024-04-09 11:45:27 +00:00
Robert Schulze
97281203b8
Don't access static members through instance, pt. II
...
clang-tidy rightfully complains
(-readability-static-accessed-through-instance) that static member
functions are accessed through non-static members
2024-04-07 11:09:35 +00:00
Anton Popov
fa01f26def
add test
2024-04-05 17:39:12 +00:00
kssenii
357e5e3ae6
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-04-05 16:15:56 +02:00
Sema Checherinda
8b75b6ae77
Merge branch 'master' into chesema-keep-alive-send-and-receive
2024-04-05 01:04:22 +02:00
Sema Checherinda
ae3a199939
support max requests for keep alive
2024-04-05 00:37:29 +02:00
Sema Checherinda
dd852da339
add more debug logs
2024-04-05 00:30:03 +02:00
Alexander Tokmakov
5db9fbed52
cancel tasks on exception
2024-04-04 22:32:57 +02:00
Anton Popov
d7fb851d17
better handling of errors from azure storage
2024-04-04 19:53:58 +00:00
kssenii
1e982a2368
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-04-04 13:26:06 +02:00
kssenii
be30806dbe
Merge remote-tracking branch 'origin/master' into use-iobject-storage-for-table-engines-1
2024-04-04 13:20:12 +02:00
Robert Schulze
de2a0be025
Don't access static members through instance
...
- clang-tidy rightfully complains (-readability-static-accessed-through-instance)
- not going to enable the warning for now to avoid breaking the build
2024-04-03 18:50:33 +00:00
SmitaRKulkarni
9bb185593e
Merge pull request #61988 from ClickHouse/Azure_fix_exception_in_finalize
...
Fix WriteBufferAzureBlobStorage destructor uncaught exception
2024-04-03 18:22:27 +02:00
vdimir
e5f218c4cc
Merge branch 'master' into vdimir/storage_join_load_order
2024-04-01 14:15:01 +02:00
