Raúl Marín
2b70e08f23
Don't count unreserved bytes in Arenas as read_bytes
2023-04-13 12:43:24 +02:00
Antonio Andelic
b30aff06d8
Merge pull request #48595 from aalexfvk/alexfvk/fix_create_not_auth_problem
...
Fix bug in Keeper when a node is not created with scheme `auth` in ACL sometimes.
2023-04-11 14:41:37 +02:00
Aleksei Filatov
dd2364361c
Update src/Coordination/tests/gtest_coordination.cpp
...
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2023-04-11 10:45:30 +03:00
Azat Khuzhin
79b83c4fd2
Remove superfluous includes of logger_userful.h from headers
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-10 17:59:30 +02:00
Aleksei Filatov
e8bf96d126
Add unit tests
2023-04-10 13:02:43 +03:00
Aleksei Filatov
000c2b5b91
Correct FixupACL for auth scheme
2023-04-10 13:02:31 +03:00
Azat Khuzhin
011480924a
Use forward declaration of ThreadPool
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-07 11:25:35 +02:00
Antonio Andelic
88f77c686d
Correctly handle concurrent snapshots
2023-04-06 15:40:05 +00:00
Antonio Andelic
e982f2a67a
Merge branch 'master' into enable-env-credentials-default
2023-03-31 09:11:01 +00:00
Antonio Andelic
44e95aa65f
Merge branch 'master' into nosign-keyword-for-s3
2023-03-29 11:10:03 +00:00
Antonio Andelic
160aa186bb
Add support for NOSIGN keyword and no_sign_request config
2023-03-28 07:05:35 +00:00
Antonio Andelic
79e22d13fb
small fix
2023-03-27 07:41:31 +00:00
Antonio Andelic
f355bc5342
Merge branch 'master' into more-batching-keeper
2023-03-27 07:28:17 +00:00
Antonio Andelic
6c3765c8b6
Small fix
2023-03-24 20:40:45 +00:00
Antonio Andelic
f353561204
Avoid breaking batches with read requests
2023-03-24 14:37:40 +00:00
Alexander Gololobov
afe3af230f
Limit keeper request batching by size in bytes
2023-03-24 14:41:09 +01:00
Antonio Andelic
51416da818
Merge branch 'master' into enable-env-credentials-default
2023-03-13 10:20:21 +00:00
Antonio Andelic
a170a909a4
Add expiration window for S3 credentials
2023-03-10 10:06:32 +00:00
Antonio Andelic
83a3243758
Merge pull request #47065 from ClickHouse/debug-keeper
...
Fix Keeper with `force_sync = false`
2023-03-10 09:27:13 +01:00
Antonio Andelic
5bc21538e5
Enable use_environment_credentials by default
2023-03-09 10:31:55 +00:00
Antonio Andelic
0ad436aa2c
Revert debug changes
2023-03-09 08:07:59 +00:00
Antonio Andelic
b4061d8aa4
Add logs
2023-03-08 13:06:10 +00:00
Mike Kot
9920a52c51
use std::lerp, constexpr hex.h
2023-03-07 22:50:17 +00:00
Antonio Andelic
dd4db9132a
Merge branch 'master' into debug-keeper
2023-03-07 13:15:05 +01:00
Antonio Andelic
c38d820d0f
Add 4LW for cleaning resource
2023-03-05 16:57:18 +00:00
Antonio Andelic
cd4100cb19
Merge branch 'master' into fix-data-size
2023-03-03 15:33:43 +01:00
Antonio Andelic
fdcb5b784f
Merge branch 'master' into debug-keeper
2023-03-03 13:45:13 +01:00
Robert Schulze
740aeaba1f
Apply some CTAD
2023-03-02 13:36:47 +00:00
Antonio Andelic
6ebee202bd
Flush buffer when no sync
2023-03-01 14:47:15 +00:00
Antonio Andelic
c797122dbe
Better
2023-03-01 08:45:50 +00:00
Antonio Andelic
6e28406751
Merge branch 'master' into fix-data-size
2023-02-28 20:09:29 +00:00
Alexey Milovidov
d8cda3dbb8
Remove PVS-Studio
2023-02-19 23:30:05 +01:00
Alexander Tokmakov
e96340e183
Merge pull request #46410 from ClickHouse/fix_noisy_log_messages
...
Fix noisy log messages
2023-02-15 01:22:42 +03:00
Alexander Tokmakov
1bf3735ca9
fix noisy log messages
2023-02-14 15:33:27 +01:00
Sema Checherinda
9bef1bec28
Merge pull request #46274 from aiven-sal/aiven-sal/keeper
...
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
2023-02-14 12:44:04 +01:00
Sema Checherinda
5882f3878f
except zero values election_timeout
2023-02-13 12:22:08 +01:00
Salvatore Mesoraca
fff2c09b87
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
2023-02-13 10:09:45 +01:00
Antonio Andelic
f96d480563
Merge branch 'master' into add-support-for-settings-alias
2023-02-09 16:07:45 +01:00
Azat Khuzhin
8cc41b7f41
Check return value of ::close()
...
Note, that according close(2), EINTR should not be retriable for close()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-07 11:28:22 +01:00
Antonio Andelic
b11458f142
Fix erase
2023-02-07 07:51:22 +00:00
Antonio Andelic
9cde46acff
Correctly calculate size on update
2023-02-06 16:28:05 +00:00
Antonio Andelic
aca4f08bf5
Correctly calculate size after snapshot load
2023-02-06 14:59:23 +00:00
Antonio Andelic
8929838fcc
Merge branch 'master' into add-support-for-settings-alias
2023-02-03 17:47:00 +00:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
LiuYangkuan
f29700bd2f
use IDisk to do IO in Keeper's snapshots and logs
2023-02-02 19:47:30 +08:00
Antonio Andelic
f613bfd8d2
Merge branch 'master' into add-support-for-settings-alias
2023-02-02 11:15:39 +00:00
Antonio Andelic
85cfee4bb9
Better alias definition
2023-02-01 13:54:03 +00:00
Antonio Andelic
88c1744975
Better
2023-02-01 14:25:58 +01:00
Antonio Andelic
5de6f86c27
Fix clang-tidy error
2023-02-01 12:28:44 +00:00
Antonio Andelic
ec04cf6876
Merge branch 'master' into add-support-for-settings-alias
2023-02-01 09:01:01 +00:00
Antonio Andelic
ece92fa866
Update src/Coordination/Changelog.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2023-01-31 17:11:18 +01:00
Antonio Andelic
8457d3bf3d
Apply comments
2023-01-31 14:18:54 +00:00
Antonio Andelic
1786237be6
Merge branch 'master' into keeper-use-fallocate
2023-01-31 13:53:47 +00:00
Vitaly Baranov
f9282cfcac
Make version_id an optional parameter again.
2023-01-27 15:10:19 +01:00
Vitaly Baranov
aea9ccdb60
Pass request settings to S3::getObjectInfo().
2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed
Move getObjectInfo() to a separate header.
2023-01-27 15:09:38 +01:00
Antonio Andelic
714fad1529
Add support for settings alias
2023-01-26 14:06:46 +00:00
Antonio Andelic
ebfe7f9991
Fix style
2023-01-26 11:18:07 +00:00
Antonio Andelic
1186718658
Add overallocate size setting
2023-01-26 09:44:37 +00:00
Antonio Andelic
7bfdfa4bf7
Preallocate more space
2023-01-25 16:26:14 +00:00
Antonio Andelic
7421d8f872
Merge branch 'master' into keeper-use-fallocate
2023-01-25 11:36:47 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Alexander Tokmakov
870cfcc36a
less fmt::runtime usages
2023-01-17 00:11:59 +01:00
Antonio Andelic
6a0e481ed7
Merge branch 'master' into keeper-use-fallocate
2023-01-12 13:57:05 +00:00
Antonio Andelic
30510c5eb6
Merge pull request #44937 from ClickHouse/fix-changelog-deadlock
...
Fix deadlock in Keeper's changelog
2023-01-10 08:59:51 +01:00
Antonio Andelic
22679ddf11
Add comment
2023-01-09 14:26:20 +01:00
Antonio Andelic
a39f437306
Typo fix
2023-01-09 13:34:48 +01:00
Antonio Andelic
27b2bae640
make it better
2023-01-09 12:01:22 +00:00
Antonio Andelic
4fcbe14435
Small fixes for Coordination unit tests
2023-01-08 13:44:32 +00:00
Kseniia Sumarokova
573d3283b0
Merge pull request #44327 from kssenii/use-new-named-collections-code-2
...
Replace old named collections code with new (from #43147 ) part 2
2023-01-06 13:06:26 +01:00
Antonio Andelic
9b3314a931
Fix deadlock in Changelog flush
2023-01-05 14:51:05 +00:00
Antonio Andelic
136436958b
Merge pull request #44908 from ClickHouse/fix-keeper-snapshot-on-exit
...
Avoid Keeper crash on shutdown (fix `test_keeper_snapshot_on_exit`)
2023-01-05 08:59:58 +01:00
Antonio Andelic
7c2cdfa65a
Don't throw exception if queue is finished
2023-01-04 15:46:38 +00:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
Smita Kulkarni
8a5fd7f96d
Added comments and addressed review comments - Allow macro substitution in endpoint
2023-01-03 11:44:26 +01:00
Antonio Andelic
f2653c016a
Manually set deleted flag
2023-01-02 14:36:58 +00:00
Antonio Andelic
f32fab5025
Check if file exists before rename
2023-01-02 13:48:13 +00:00
Antonio Andelic
2e93640614
Fix ASAN error
2023-01-02 13:37:40 +00:00
Antonio Andelic
b845f880fb
Polish
2023-01-02 10:42:15 +00:00
Antonio Andelic
14355d8009
Merge branch 'master' into keeper-use-fallocate
2023-01-02 08:22:27 +00:00
Antonio Andelic
55548a6678
style fix
2023-01-02 08:22:00 +00:00
SmitaRKulkarni
f9a3aeb172
Merge branch 'master' into 40951_Allow_macro_substitution_in_endpoint
2022-12-30 10:43:14 +01:00
Alexey Milovidov
f0fa0977ed
Fix wrong log messages
2022-12-26 18:36:07 +01:00
Smita Kulkarni
3d62bc2635
Allow macro substitution in endpoint of disks
...
Implementation:
* Updated to substitute macros in endpoint for all disks (s3, hfs, web).
Testing:
* Added an integration test test/integration/test_endpoint_macro_substitution.
2022-12-23 15:09:24 +01:00
Antonio Andelic
64ade259cf
Fix tests
2022-12-22 10:20:55 +00:00
Antonio Andelic
88d41dea93
Merge branch 'master' into keeper-use-fallocate
2022-12-21 08:59:09 +00:00
Antonio Andelic
9711355cc9
Fix tests
2022-12-21 08:43:55 +00:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
Antonio Andelic
6d83221d4e
Remove unused error code
2022-12-19 12:20:26 +00:00
Antonio Andelic
c2c115d2ba
Add setting for max_log_file_size
2022-12-19 11:51:05 +00:00
Antonio Andelic
f241701c2a
Add checks when fallocate is not supported
2022-12-19 11:48:54 +00:00
Antonio Andelic
9a8b248735
Merge branch 'master' into keeper-use-fallocate
2022-12-19 09:32:05 +00:00
Nikolai Kochetov
62ff98344e
Validate s3 part upload settings.
2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Antonio Andelic
ee900ca7df
Add truncate
2022-12-15 13:09:52 +00:00
Antonio Andelic
bf6f163952
Merge branch 'master' into keeper-use-fallocate
2022-12-15 09:05:40 +00:00
Antonio Andelic
d454e00016
Changelog refactor
2022-12-15 08:12:02 +00:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
Antonio Andelic
662c1f956c
Changelog refactoring
2022-12-13 14:00:27 +00:00
Antonio Andelic
898cf867af
Try with fallocate
2022-12-13 11:08:59 +00:00
Vitaly Baranov
b91af1b650
Fix initialization of s3 request settings.
2022-12-10 05:43:51 +01:00
Antonio Andelic
e1a6a1182e
Throw when port already in use ( #43984 )
2022-12-07 13:27:49 +01:00
Antonio Andelic
200141da13
Use logger in tests
2022-12-02 08:16:18 +00:00
Antonio Andelic
7cd76df918
Merge branch 'master' into parallel-log-appending
2022-12-01 14:44:36 +00:00
Antonio Andelic
554ff1db6f
Small changes
2022-12-01 12:54:29 +00:00
Antonio Andelic
940fdd4abd
Remove comment
2022-12-01 12:24:27 +01:00
Antonio Andelic
25ae52111e
Protect writer with mutex
2022-12-01 11:18:23 +00:00
Antonio Andelic
c12c0cc8af
Merge branch 'master' into parallel-log-appending
2022-12-01 08:53:52 +00:00
Antonio Andelic
2f501d5327
Merge branch 'master' into keeper-prometheus
2022-11-30 13:24:37 +01:00
Antonio Andelic
fbda86ddc8
PR fixes
2022-11-30 13:24:08 +01:00
Antonio Andelic
7f897f1a20
Refactor
2022-11-29 13:00:13 +00:00
Antonio Andelic
c8b2cb5650
Update src/Coordination/CoordinationSettings.h
...
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
2022-11-29 10:08:25 +01:00
Antonio Andelic
678958603b
Merge branch 'master' into keeper-prometheus
2022-11-28 09:48:57 +01:00
Antonio Andelic
fe44ac0c86
Merge branch 'master' into parallel-log-appending
2022-11-28 08:45:20 +00:00
Antonio Andelic
e91314d57f
Use initialized flag
2022-11-28 08:44:46 +00:00
Antonio Andelic
cade21b76e
Fix 4LW test
2022-11-26 19:24:10 +00:00
Antonio Andelic
76eca8bdfb
Merge branch 'master' into more-aggresive-batching-keeper
2022-11-26 18:28:11 +00:00
Antonio Andelic
cd471e9c61
Add setting for quick batch size
2022-11-26 17:33:40 +00:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
Antonio Andelic
7437c719a7
Try batching requests more aggresively
2022-11-25 17:25:28 +00:00
Antonio Andelic
38d99c241f
Fix tests
2022-11-25 13:53:03 +00:00
Antonio Andelic
53f708366c
Fix data race in Keeper snapshot
2022-11-25 12:50:42 +00:00
Antonio Andelic
ce532cd12d
Fix circular deps
2022-11-24 13:43:05 +00:00
Antonio Andelic
a6f38cb1cc
Merge branch 'master' into keeper-prometheus
2022-11-24 12:12:08 +01:00
Antonio Andelic
0cc53995df
Merge branch 'master' into parallel-log-appending
2022-11-22 15:05:08 +01:00
Kseniia Sumarokova
5c90d5aa7e
Merge pull request #43253 from xiedeyantu/fix-s3-glob
...
fix s3 support question mark wildcard
2022-11-22 14:26:56 +01:00
Antonio Andelic
4827bcb6bb
Merge branch 'master' into parallel-log-appending
2022-11-22 14:14:17 +01:00
Antonio Andelic
55ef96dc11
Fix unit tests
2022-11-22 11:37:59 +00:00
Antonio Andelic
aeb0d39b50
Fix shutdown
2022-11-21 15:01:18 +00:00
Antonio Andelic
c9472d67b6
Refactor changelog
2022-11-21 12:22:36 +00:00
Antonio Andelic
ff681d5faa
Merge branch 'master' into parallel-log-appending
2022-11-21 11:59:16 +00:00
Antonio Andelic
3cb202a63b
Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader
...
Keeper support manually assigning leader
2022-11-21 09:30:00 +01:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus
2022-11-21 09:10:10 +01:00
xiedeyantu
c258d3ac8b
fix s3 support question mark wildcard
2022-11-18 12:11:22 +08:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
159743edd6
Merge pull request #43014 from ClickHouse/disk-s3-throttler
...
S3 request per second rate throttling
2022-11-16 18:51:06 +01:00
JackyWoo
f8d71f60a2
Merge branch 'master' into keeper_manually_assign_leader
2022-11-16 09:34:40 +08:00
Antonio Andelic
96838f5be7
Merge branch 'master' into keeper-prometheus
2022-11-15 12:00:02 +00:00
Antonio Andelic
d08dce5ec7
Test parallel append
2022-11-14 13:17:05 +00:00
Raúl Marín
54db7c6520
Enforce checking read output
2022-11-11 10:56:18 +01:00
Antonio Andelic
94aab4276c
Bring back heavy metrics
2022-11-10 12:56:27 +01:00
Antonio Andelic
8aeb406d82
Fix standalone build
2022-11-10 10:49:00 +01:00
Antonio Andelic
36d763011b
Compile standalone keeper
2022-11-10 09:23:52 +01:00
Antonio Andelic
1ed3930809
Add more metrics for Keeper
2022-11-09 15:51:41 +01:00
Antonio Andelic
c92ae5a385
Add suport for AsyncMetrics in standalone Keeper
2022-11-09 13:46:03 +01:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
JackyWoo
9060141f16
revert last change
2022-11-08 18:44:43 +08:00
JackyWoo
e865842098
Add rqld to white list
2022-11-08 16:03:13 +08:00
