thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-04 21:42:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
115,051 Commits 9,185 Branches 1,782 Tags 2.3 GiB
9dbe9507e7
Commit Graph

348 Commits

Author SHA1 Message Date
Alexey Milovidov
5a44dc26e7 Fixes for clang-17 2023-05-13 02:57:31 +02:00
Antonio Andelic
a68a023ca7
Merge pull request #48724 from johanngan/sse-kms 
Support SSE-KMS configuration with S3 client
 2023-05-04 13:20:54 +02:00
alesapin
412b161104
Merge pull request #48791 from kssenii/better-local-object-storage 
Make local object storage work consistently with s3 object storage, fix problem with append, make it configurable as independent storage
 2023-05-04 11:47:43 +02:00
johanngan
731823b873 Add support for SSE-KMS configuration with S3 
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html

Similar to the server_side_encryption_customer_key_base64 option for
configuring SSE-C with S3, add the following settings to configure
SSE-KMS on a per-endpoint/disk basis:
  - server_side_encryption_kms_key_id
  - server_side_encryption_kms_encryption_context
  - server_side_encryption_kms_bucket_key_enabled
 2023-05-03 21:35:38 -05:00
Nikita Mikhaylov
954e3b724c
Speedup outdated parts loading (#49317) 2023-05-03 18:56:45 +02:00
kssenii
35f437ac9c Address review comments 2023-05-03 14:37:18 +02:00
Alexey Milovidov
530b764953 Fix IBM 2023-04-21 12:38:45 +02:00
Raúl Marín
45ad555c39
Merge branch 'master' into zk_retry_timeout 2023-04-10 10:04:16 +02:00
Azat Khuzhin
011480924a Use forward declaration of ThreadPool 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 11:25:35 +02:00
robot-ch-test-poll1
9466cec1fc
Merge pull request #48342 from ClickHouse/Backup_Restore_concurrency_check_node_2 
Check node for Backup Restore concurrency
 2023-04-05 23:49:32 +02:00
Raúl Marín
2276d4feb4 Backups have no context and no process list element 2023-04-05 11:19:04 +02:00
Azat Khuzhin
dd9f0f409b Remove knowledge about throttling from IBackupWriter::supportNativeCopy() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:19 +02:00
Azat Khuzhin
c8597fbb9a Do not throttle S3-S3 backups if native copy is possible 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
c332d290d8 Keep only one throttler for BACKUPs IO (instead of separate read/write) 
There is no need in separate read/write throttling, because you cannot
write faster then read anyway, and plus this makes the code less cleaner

(and also it will allow avoid implementing throttling backups to S3,
since it does not use common S3 writer).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:17 +02:00
Azat Khuzhin
218b1f9c29 Add ability to throttle BACKUPs on per-server/backup basis 
Server settings:
- backup_read_bandwidth_for_server
- backup_write_bandwidth_for_server

Query settings:
- backup_read_bandwidth
- backup_write_bandwidth

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:48 +02:00
Smita Kulkarni
a36b81a22d Fixed comment 2023-04-04 19:07:31 +02:00
Smita Kulkarni
f4e2d45fbc Added check for backup/restore when they fail and status is not COMPLETED 2023-04-04 19:05:27 +02:00
Raúl Marín
3cee537e73 Changes for master 2023-04-04 18:58:28 +02:00
Raúl Marín
0e17b3b147 Merge remote-tracking branch 'blessed/master' into zk_retry_timeout 2023-04-04 18:53:55 +02:00
Smita Kulkarni
beb164dd51 Merge branch 'master' into Backup_Restore_concurrency_check_node_2 2023-04-04 18:51:30 +02:00
Nikita Mikhaylov
fa5b2bd4a5
Added Keeper retries for backups operations (#47224) 2023-04-04 18:12:08 +02:00
Antonio Andelic
a329d80bfa
Merge pull request #47397 from ClickHouse/enable-env-credentials-default 
Enable `use_environment_credentials` by default
 2023-04-04 10:00:03 +02:00
Smita Kulkarni
bce8eb7468 Updated to use tryGet instead of get for checking stage of backups/restores in concurrency check 2023-04-03 12:21:16 +02:00
Raúl Marín
8fdf87982c Merge remote-tracking branch 'blessed/master' into zk_retry_timeout 2023-04-03 10:26:18 +02:00
Antonio Andelic
da194f082d
Merge branch 'master' into enable-env-credentials-default 2023-04-03 09:45:03 +02:00
Vitaly Baranov
3f4aadfe7d Add logging for concurrency checks for backups. 2023-03-31 23:50:35 +02:00
Alexey Milovidov
070210a02f
Merge pull request #48271 from vchekan/master 
In messages, put values into quotes
 2023-03-31 15:35:19 +03:00
Antonio Andelic
e982f2a67a Merge branch 'master' into enable-env-credentials-default 2023-03-31 09:11:01 +00:00
Vadym Chekan
0f4c8144a6 In messages, put values into quotes 
Configuration values, such as disk names, backup engine names, etc, may give error message unintended sense, for example, if trying to backup to `disk` instead of `Disk`, the error message will be "Not found backup engine disk", which can be interpreted as "disk of backup engine not found". It might be not clear that the word "disk" comes from the query and is not part of the error message.
 2023-03-30 22:46:18 -07:00
Antonio Andelic
80cb121d2a
Merge pull request #48092 from ClickHouse/nosign-keyword-for-s3 
Add support for `NOSIGN` keyword and `no_sign_request` config for S3
 2023-03-30 18:10:56 +02:00
Alexey Milovidov
e982fb9f1c
Merge pull request #47880 from azat/threadpool-introspection 
ThreadPool metrics introspection
 2023-03-30 01:27:31 +03:00
Vitaly Baranov
481a7a76ac
Simplify backup coordination for file infos (#48095) 
* Remove obsolete code for archive suffixes.

* Simplify backup coordination, stop using it for restoring.

* Build all file infos before writing to backup. Decrease number of znodes.

* Split long values before writing to ZooKeeper.

* Use separate mutexes for unrelated activities.

* Make test test_disallow_concurrency less flaky.

* Add comments and test for backup_keeper_value_max_size.
 2023-03-29 15:19:40 +02:00
Antonio Andelic
44e95aa65f Merge branch 'master' into nosign-keyword-for-s3 2023-03-29 11:10:03 +00:00
Azat Khuzhin
f38a7aeabe ThreadPool metrics introspection 
There are lots of thread pools and simple local-vs-global is not enough
already, it is good to know which one in particular uses threads.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-29 10:46:59 +02:00
Antonio Andelic
160aa186bb Add support for NOSIGN keyword and no_sign_request config 2023-03-28 07:05:35 +00:00
Vitaly Baranov
e43fc77a4e
Merge pull request #46989 from AVMusorin/update-system-backups-periodically 
Dynamic update `system.backups`
 2023-03-27 17:26:47 +02:00
Raúl Marín
83b68caccc Do not continue retrying to connect to ZK if the query is killed or over limits 2023-03-27 16:01:15 +02:00
Vitaly Baranov
1badc3cba0
Move information about current hosts and list of all hosts to BackupCoordination (#47971) 
to simplify the code and help implementing other features.

Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-03-24 17:38:19 +01:00
Alexander Tokmakov
cd7d1fb990
Revert "Revert "Revert "Backup_Restore_concurrency_check_node""" 2023-03-24 04:35:50 +03:00
SmitaRKulkarni
04822a63e1
Merge pull request #47586 from ClickHouse/revert-47581-revert-47216-Backup_Restore_concurrency_check_node 
Revert "Revert "Backup_Restore_concurrency_check_node""
 2023-03-23 10:02:00 +01:00
Azat Khuzhin
1ebbfac721 Use restore_threads (not backup_threads) for RESTORE ASYNC 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-21 20:35:00 +02:00
AVMusorin
038bfb40ab
dynamic update system.backups 2023-03-21 11:42:54 +01:00
Vitaly Baranov
198409e12a
Merge pull request #46085 from aalexfvk/alexfvk/store_udf_in_zookeeper 
Replication of user-defined SQL functions using ZooKeeper
 2023-03-20 13:41:29 +01:00
Antonio Andelic
a0582a14b9
Merge pull request #47423 from ClickHouse/add-expiration-window-s3 
Add expiration window for S3 credentials
 2023-03-18 10:11:29 +01:00
Aleksei Filatov
886b530963 [rev: 1] Fix review remarks 2023-03-17 13:56:05 +03:00
Aleksei Filatov
690d8355ca Add classes for backup/restore UDF 2023-03-17 13:56:04 +03:00
Sema Checherinda
3c6deddd1d work with comments on PR 2023-03-16 19:55:58 +01:00
Vitaly Baranov
25356786ea Simplify the implementation, create new utility function copyS3FileToDisk(). 2023-03-14 23:34:44 +01:00
SmitaRKulkarni
8db4964ebc
Revert "Revert "Backup_Restore_concurrency_check_node"" 2023-03-14 20:23:43 +01:00
Alexander Tokmakov
773cd5a686
Revert "Backup_Restore_concurrency_check_node" 2023-03-14 18:55:08 +03:00