Alexander Gololobov
afe3af230f
Limit keeper request batching by size in bytes
2023-03-24 14:41:09 +01:00
Antonio Andelic
51416da818
Merge branch 'master' into enable-env-credentials-default
2023-03-13 10:20:21 +00:00
Antonio Andelic
a170a909a4
Add expiration window for S3 credentials
2023-03-10 10:06:32 +00:00
Antonio Andelic
83a3243758
Merge pull request #47065 from ClickHouse/debug-keeper
...
Fix Keeper with `force_sync = false`
2023-03-10 09:27:13 +01:00
Antonio Andelic
5bc21538e5
Enable use_environment_credentials by default
2023-03-09 10:31:55 +00:00
Antonio Andelic
0ad436aa2c
Revert debug changes
2023-03-09 08:07:59 +00:00
Antonio Andelic
b4061d8aa4
Add logs
2023-03-08 13:06:10 +00:00
Mike Kot
9920a52c51
use std::lerp, constexpr hex.h
2023-03-07 22:50:17 +00:00
Antonio Andelic
dd4db9132a
Merge branch 'master' into debug-keeper
2023-03-07 13:15:05 +01:00
Antonio Andelic
c38d820d0f
Add 4LW for cleaning resource
2023-03-05 16:57:18 +00:00
Antonio Andelic
cd4100cb19
Merge branch 'master' into fix-data-size
2023-03-03 15:33:43 +01:00
Antonio Andelic
fdcb5b784f
Merge branch 'master' into debug-keeper
2023-03-03 13:45:13 +01:00
Robert Schulze
740aeaba1f
Apply some CTAD
2023-03-02 13:36:47 +00:00
Antonio Andelic
6ebee202bd
Flush buffer when no sync
2023-03-01 14:47:15 +00:00
Antonio Andelic
c797122dbe
Better
2023-03-01 08:45:50 +00:00
Antonio Andelic
6e28406751
Merge branch 'master' into fix-data-size
2023-02-28 20:09:29 +00:00
Alexey Milovidov
d8cda3dbb8
Remove PVS-Studio
2023-02-19 23:30:05 +01:00
Alexander Tokmakov
e96340e183
Merge pull request #46410 from ClickHouse/fix_noisy_log_messages
...
Fix noisy log messages
2023-02-15 01:22:42 +03:00
Alexander Tokmakov
1bf3735ca9
fix noisy log messages
2023-02-14 15:33:27 +01:00
Sema Checherinda
9bef1bec28
Merge pull request #46274 from aiven-sal/aiven-sal/keeper
...
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
2023-02-14 12:44:04 +01:00
Sema Checherinda
5882f3878f
except zero values election_timeout
2023-02-13 12:22:08 +01:00
Salvatore Mesoraca
fff2c09b87
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
2023-02-13 10:09:45 +01:00
Antonio Andelic
f96d480563
Merge branch 'master' into add-support-for-settings-alias
2023-02-09 16:07:45 +01:00
Azat Khuzhin
8cc41b7f41
Check return value of ::close()
...
Note, that according close(2), EINTR should not be retriable for close()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-07 11:28:22 +01:00
Antonio Andelic
b11458f142
Fix erase
2023-02-07 07:51:22 +00:00
Antonio Andelic
9cde46acff
Correctly calculate size on update
2023-02-06 16:28:05 +00:00
Antonio Andelic
aca4f08bf5
Correctly calculate size after snapshot load
2023-02-06 14:59:23 +00:00
Antonio Andelic
8929838fcc
Merge branch 'master' into add-support-for-settings-alias
2023-02-03 17:47:00 +00:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
LiuYangkuan
f29700bd2f
use IDisk to do IO in Keeper's snapshots and logs
2023-02-02 19:47:30 +08:00
Antonio Andelic
f613bfd8d2
Merge branch 'master' into add-support-for-settings-alias
2023-02-02 11:15:39 +00:00
Antonio Andelic
85cfee4bb9
Better alias definition
2023-02-01 13:54:03 +00:00
Antonio Andelic
88c1744975
Better
2023-02-01 14:25:58 +01:00
Antonio Andelic
5de6f86c27
Fix clang-tidy error
2023-02-01 12:28:44 +00:00
Antonio Andelic
ec04cf6876
Merge branch 'master' into add-support-for-settings-alias
2023-02-01 09:01:01 +00:00
Antonio Andelic
ece92fa866
Update src/Coordination/Changelog.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2023-01-31 17:11:18 +01:00
Antonio Andelic
8457d3bf3d
Apply comments
2023-01-31 14:18:54 +00:00
Antonio Andelic
1786237be6
Merge branch 'master' into keeper-use-fallocate
2023-01-31 13:53:47 +00:00
Vitaly Baranov
f9282cfcac
Make version_id an optional parameter again.
2023-01-27 15:10:19 +01:00
Vitaly Baranov
aea9ccdb60
Pass request settings to S3::getObjectInfo().
2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed
Move getObjectInfo() to a separate header.
2023-01-27 15:09:38 +01:00
Antonio Andelic
714fad1529
Add support for settings alias
2023-01-26 14:06:46 +00:00
Antonio Andelic
ebfe7f9991
Fix style
2023-01-26 11:18:07 +00:00
Antonio Andelic
1186718658
Add overallocate size setting
2023-01-26 09:44:37 +00:00
Antonio Andelic
7bfdfa4bf7
Preallocate more space
2023-01-25 16:26:14 +00:00
Antonio Andelic
7421d8f872
Merge branch 'master' into keeper-use-fallocate
2023-01-25 11:36:47 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Alexander Tokmakov
870cfcc36a
less fmt::runtime usages
2023-01-17 00:11:59 +01:00
Antonio Andelic
6a0e481ed7
Merge branch 'master' into keeper-use-fallocate
2023-01-12 13:57:05 +00:00
Antonio Andelic
30510c5eb6
Merge pull request #44937 from ClickHouse/fix-changelog-deadlock
...
Fix deadlock in Keeper's changelog
2023-01-10 08:59:51 +01:00
Antonio Andelic
22679ddf11
Add comment
2023-01-09 14:26:20 +01:00
Antonio Andelic
a39f437306
Typo fix
2023-01-09 13:34:48 +01:00
Antonio Andelic
27b2bae640
make it better
2023-01-09 12:01:22 +00:00
Antonio Andelic
4fcbe14435
Small fixes for Coordination unit tests
2023-01-08 13:44:32 +00:00
Kseniia Sumarokova
573d3283b0
Merge pull request #44327 from kssenii/use-new-named-collections-code-2
...
Replace old named collections code with new (from #43147 ) part 2
2023-01-06 13:06:26 +01:00
Antonio Andelic
9b3314a931
Fix deadlock in Changelog flush
2023-01-05 14:51:05 +00:00
Antonio Andelic
136436958b
Merge pull request #44908 from ClickHouse/fix-keeper-snapshot-on-exit
...
Avoid Keeper crash on shutdown (fix `test_keeper_snapshot_on_exit`)
2023-01-05 08:59:58 +01:00
Antonio Andelic
7c2cdfa65a
Don't throw exception if queue is finished
2023-01-04 15:46:38 +00:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
Smita Kulkarni
8a5fd7f96d
Added comments and addressed review comments - Allow macro substitution in endpoint
2023-01-03 11:44:26 +01:00
Antonio Andelic
f2653c016a
Manually set deleted flag
2023-01-02 14:36:58 +00:00
Antonio Andelic
f32fab5025
Check if file exists before rename
2023-01-02 13:48:13 +00:00
Antonio Andelic
2e93640614
Fix ASAN error
2023-01-02 13:37:40 +00:00
Antonio Andelic
b845f880fb
Polish
2023-01-02 10:42:15 +00:00
Antonio Andelic
14355d8009
Merge branch 'master' into keeper-use-fallocate
2023-01-02 08:22:27 +00:00
Antonio Andelic
55548a6678
style fix
2023-01-02 08:22:00 +00:00
SmitaRKulkarni
f9a3aeb172
Merge branch 'master' into 40951_Allow_macro_substitution_in_endpoint
2022-12-30 10:43:14 +01:00
Alexey Milovidov
f0fa0977ed
Fix wrong log messages
2022-12-26 18:36:07 +01:00
Smita Kulkarni
3d62bc2635
Allow macro substitution in endpoint of disks
...
Implementation:
* Updated to substitute macros in endpoint for all disks (s3, hfs, web).
Testing:
* Added an integration test test/integration/test_endpoint_macro_substitution.
2022-12-23 15:09:24 +01:00
Antonio Andelic
64ade259cf
Fix tests
2022-12-22 10:20:55 +00:00
Antonio Andelic
88d41dea93
Merge branch 'master' into keeper-use-fallocate
2022-12-21 08:59:09 +00:00
Antonio Andelic
9711355cc9
Fix tests
2022-12-21 08:43:55 +00:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
Antonio Andelic
6d83221d4e
Remove unused error code
2022-12-19 12:20:26 +00:00
Antonio Andelic
c2c115d2ba
Add setting for max_log_file_size
2022-12-19 11:51:05 +00:00
Antonio Andelic
f241701c2a
Add checks when fallocate is not supported
2022-12-19 11:48:54 +00:00
Antonio Andelic
9a8b248735
Merge branch 'master' into keeper-use-fallocate
2022-12-19 09:32:05 +00:00
Nikolai Kochetov
62ff98344e
Validate s3 part upload settings.
2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Antonio Andelic
ee900ca7df
Add truncate
2022-12-15 13:09:52 +00:00
Antonio Andelic
bf6f163952
Merge branch 'master' into keeper-use-fallocate
2022-12-15 09:05:40 +00:00
Antonio Andelic
d454e00016
Changelog refactor
2022-12-15 08:12:02 +00:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
Antonio Andelic
662c1f956c
Changelog refactoring
2022-12-13 14:00:27 +00:00
Antonio Andelic
898cf867af
Try with fallocate
2022-12-13 11:08:59 +00:00
Vitaly Baranov
b91af1b650
Fix initialization of s3 request settings.
2022-12-10 05:43:51 +01:00
Antonio Andelic
e1a6a1182e
Throw when port already in use ( #43984 )
2022-12-07 13:27:49 +01:00
Antonio Andelic
200141da13
Use logger in tests
2022-12-02 08:16:18 +00:00
Antonio Andelic
7cd76df918
Merge branch 'master' into parallel-log-appending
2022-12-01 14:44:36 +00:00
Antonio Andelic
554ff1db6f
Small changes
2022-12-01 12:54:29 +00:00
Antonio Andelic
940fdd4abd
Remove comment
2022-12-01 12:24:27 +01:00
Antonio Andelic
25ae52111e
Protect writer with mutex
2022-12-01 11:18:23 +00:00
Antonio Andelic
c12c0cc8af
Merge branch 'master' into parallel-log-appending
2022-12-01 08:53:52 +00:00
Antonio Andelic
2f501d5327
Merge branch 'master' into keeper-prometheus
2022-11-30 13:24:37 +01:00
Antonio Andelic
fbda86ddc8
PR fixes
2022-11-30 13:24:08 +01:00
Antonio Andelic
7f897f1a20
Refactor
2022-11-29 13:00:13 +00:00
Antonio Andelic
c8b2cb5650
Update src/Coordination/CoordinationSettings.h
...
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
2022-11-29 10:08:25 +01:00
Antonio Andelic
678958603b
Merge branch 'master' into keeper-prometheus
2022-11-28 09:48:57 +01:00
Antonio Andelic
fe44ac0c86
Merge branch 'master' into parallel-log-appending
2022-11-28 08:45:20 +00:00
Antonio Andelic
e91314d57f
Use initialized flag
2022-11-28 08:44:46 +00:00
Antonio Andelic
cade21b76e
Fix 4LW test
2022-11-26 19:24:10 +00:00
Antonio Andelic
76eca8bdfb
Merge branch 'master' into more-aggresive-batching-keeper
2022-11-26 18:28:11 +00:00
Antonio Andelic
cd471e9c61
Add setting for quick batch size
2022-11-26 17:33:40 +00:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
Antonio Andelic
7437c719a7
Try batching requests more aggresively
2022-11-25 17:25:28 +00:00
Antonio Andelic
38d99c241f
Fix tests
2022-11-25 13:53:03 +00:00
Antonio Andelic
53f708366c
Fix data race in Keeper snapshot
2022-11-25 12:50:42 +00:00
Antonio Andelic
ce532cd12d
Fix circular deps
2022-11-24 13:43:05 +00:00
Antonio Andelic
a6f38cb1cc
Merge branch 'master' into keeper-prometheus
2022-11-24 12:12:08 +01:00
Antonio Andelic
0cc53995df
Merge branch 'master' into parallel-log-appending
2022-11-22 15:05:08 +01:00
Kseniia Sumarokova
5c90d5aa7e
Merge pull request #43253 from xiedeyantu/fix-s3-glob
...
fix s3 support question mark wildcard
2022-11-22 14:26:56 +01:00
Antonio Andelic
4827bcb6bb
Merge branch 'master' into parallel-log-appending
2022-11-22 14:14:17 +01:00
Antonio Andelic
55ef96dc11
Fix unit tests
2022-11-22 11:37:59 +00:00
Antonio Andelic
aeb0d39b50
Fix shutdown
2022-11-21 15:01:18 +00:00
Antonio Andelic
c9472d67b6
Refactor changelog
2022-11-21 12:22:36 +00:00
Antonio Andelic
ff681d5faa
Merge branch 'master' into parallel-log-appending
2022-11-21 11:59:16 +00:00
Antonio Andelic
3cb202a63b
Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader
...
Keeper support manually assigning leader
2022-11-21 09:30:00 +01:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus
2022-11-21 09:10:10 +01:00
xiedeyantu
c258d3ac8b
fix s3 support question mark wildcard
2022-11-18 12:11:22 +08:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
159743edd6
Merge pull request #43014 from ClickHouse/disk-s3-throttler
...
S3 request per second rate throttling
2022-11-16 18:51:06 +01:00
JackyWoo
f8d71f60a2
Merge branch 'master' into keeper_manually_assign_leader
2022-11-16 09:34:40 +08:00
Antonio Andelic
96838f5be7
Merge branch 'master' into keeper-prometheus
2022-11-15 12:00:02 +00:00
Antonio Andelic
d08dce5ec7
Test parallel append
2022-11-14 13:17:05 +00:00
Raúl Marín
54db7c6520
Enforce checking read output
2022-11-11 10:56:18 +01:00
Antonio Andelic
94aab4276c
Bring back heavy metrics
2022-11-10 12:56:27 +01:00
Antonio Andelic
8aeb406d82
Fix standalone build
2022-11-10 10:49:00 +01:00
Antonio Andelic
36d763011b
Compile standalone keeper
2022-11-10 09:23:52 +01:00
Antonio Andelic
1ed3930809
Add more metrics for Keeper
2022-11-09 15:51:41 +01:00
Antonio Andelic
c92ae5a385
Add suport for AsyncMetrics in standalone Keeper
2022-11-09 13:46:03 +01:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
JackyWoo
9060141f16
revert last change
2022-11-08 18:44:43 +08:00
JackyWoo
e865842098
Add rqld to white list
2022-11-08 16:03:13 +08:00
JackyWoo
eeedd7d9ba
Keeper support manually assigning leader
2022-11-08 12:01:53 +08:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Antonio Andelic
9ac829d4c4
Merge pull request #41766 from JackyWoo/manual_snapshot_for_keeper
...
Manually snapshot creation for keeper
2022-11-07 09:18:58 +01:00
JackyWoo
73dfe4acd7
Add to default white list
2022-11-05 17:56:55 +08:00
Antonio Andelic
25b19e6ba0
Merge branch 'master' into manual_snapshot_for_keeper
2022-11-03 08:55:01 +01:00
Antonio Andelic
664690f9a9
Add comments
2022-11-03 07:38:43 +00:00
Antonio Andelic
a6c4204b0c
Don't iterate all cached nodes on commit
2022-11-03 07:36:25 +00:00
JackyWoo
c7a0ebeb05
little fix
2022-10-25 17:46:24 +08:00
JackyWoo
9a36a509fe
fix test
2022-10-25 17:15:49 +08:00
JackyWoo
b5d1c4e657
replace snpd with lgif
2022-10-24 20:08:58 +08:00
JackyWoo
2f30c817bf
little fix
2022-10-24 17:23:47 +08:00
JackyWoo
39c88c74e8
check whether last manually created snapshot is done
2022-10-22 22:31:17 +08:00
JackyWoo
bf291790b7
Merge branch 'master' into manual_snapshot_for_keeper
2022-10-22 16:57:50 +08:00
Azat Khuzhin
905a95e166
Review fixes
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 22:40:13 +02:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Antonio Andelic
540728ab96
Simplify logic a bit
2022-10-18 10:58:25 +00:00
Antonio Andelic
1275c464aa
Fix unit test build
2022-10-18 10:44:22 +00:00
Antonio Andelic
8157fd0595
fix build
2022-10-17 14:16:56 +00:00
Antonio Andelic
238f86d892
Small refactoring
2022-10-17 11:06:33 +00:00
Antonio Andelic
7c7395575f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-10-17 11:02:36 +00:00
Robert Schulze
da5a2e2db0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup
...
Physical merge conflicts:
- src/Common/ZooKeeper/ZooKeeperImpl.cpp
- src/Core/config_core.h.in
- src/Functions/FunctionsAES.h
- src/Functions/config_functions.h.in
- src/configure_config.cmake
Logical merge conflicts:
- Functions/tryDecrypt.cpp
2022-10-06 08:43:25 +00:00
Antonio Andelic
94f1fe39bb
Add support for exists in multiread
2022-09-29 10:39:33 +00:00
Antonio Andelic
3109ce51c6
Merge branch 'master' into keeper-multiread
2022-09-29 06:50:49 +00:00
Antonio Andelic
349bd7fd9a
Dont fail MultiRead on first failed op
2022-09-29 06:50:28 +00:00
Robert Schulze
f24fab7747
Fix some #include atrocities
2022-09-28 13:49:28 +00:00
Robert Schulze
fd86829824
Consolidate config_core.h into config.h
...
Less duplication, less confusion ...
2022-09-28 13:31:57 +00:00
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Antonio Andelic
d0457addbd
Support filtered list
2022-09-27 15:02:26 +00:00
JackyWoo
d1c85f68e8
manually snapshot creation for keeper
2022-09-26 18:29:15 +08:00
Antonio Andelic
937d534cd4
Add support for simple list
2022-09-26 08:05:02 +00:00
Antonio Andelic
f833366555
Merge branch 'master' into keeper-multiread
2022-09-26 07:16:45 +00:00
Antonio Andelic
9da433a904
Fix unused
2022-09-22 14:33:35 +00:00
Antonio Andelic
44a3d6babe
Address PR comments
2022-09-22 13:03:27 +00:00
Antonio Andelic
6798b500e9
Wait on startup for Keeper
2022-09-21 15:12:16 +00:00
Antonio Andelic
eefbbf53e8
Extract S3 logic
2022-09-21 11:53:54 +00:00
Antonio Andelic
fc6c1faa57
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-21 06:57:58 +00:00
Antonio Andelic
1763af37b9
Merge pull request #41403 from ClickHouse/keeper-fix-shutdown-without-server
...
Fix Keeper segfault during shutdown when RAFT server failed to start
2022-09-19 12:03:58 +02:00
alesapin
b6d61318d0
One more validation for broken config in keeper
2022-09-18 00:04:16 +02:00
Alexander Tokmakov
42dab6dc02
Revert "Merge pull request #38200 from ClickHouse/keeper-linearizable-reads"
...
This reverts commit 222b964070da01982652
2022-09-17 19:22:58 +02:00
Antonio Andelic
81f7cf3ca6
Add support for MultiRead in Keeper
2022-09-16 12:00:35 +00:00
Antonio Andelic
acd9cb089e
Fix segfault during shutdown when RAFT server failed to start
2022-09-16 10:03:21 +00:00
Antonio Andelic
5662e0050c
Define client for multi read
2022-09-16 09:55:02 +00:00
Antonio Andelic
296fe01fb8
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-16 08:00:15 +00:00
Antonio Andelic
39307f9dba
Extract into function
2022-09-15 16:14:53 +00:00
Antonio Andelic
0dbd0c1f72
Merge branch 'master' into keeper-linearizable-reads
2022-09-15 16:03:18 +00:00
Antonio Andelic
ad7864cb5f
enable S3 if client is built
2022-09-15 14:06:22 +00:00
Antonio Andelic
d3d1676352
Fix config update detection
2022-09-15 13:48:39 +00:00
Antonio Andelic
d4e4ac3801
Lock file during S3 snapshot upload
2022-09-15 09:11:55 +00:00
Antonio Andelic
9fdd2f2e61
Basic implementation for S3 snapshot upload
2022-09-15 09:11:55 +00:00
Antonio Andelic
3a9996f04c
Add bg thread for snapshot upload to S3
2022-09-15 09:11:55 +00:00
Antonio Andelic
36f46891dd
Merge pull request #41215 from ClickHouse/keeper-close-connection-on-shutdown
...
Close sessions on Keeper shutdown
2022-09-13 19:09:34 +02:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage
...
KeeperMap storage engine
2022-09-13 14:15:09 +02:00
alesapin
69d08b1007
Merge pull request #41075 from ClickHouse/keeper-fix-possible-segfault
...
Fix possible segfault during Keeper shutdown
2022-09-13 12:13:24 +02:00
Antonio Andelic
a6b5ffec5d
Polishing
2022-09-13 09:51:46 +00:00
Antonio Andelic
b2cc6a8cc6
Use promise/future
2022-09-12 18:19:41 +00:00
Antonio Andelic
c2dfabe51c
Use shared_ptr for Event
2022-09-12 12:25:39 +00:00
Antonio Andelic
e985ee3354
Close sessions on Keeper shutdown
2022-09-12 12:22:48 +00:00
Antonio Andelic
32483aeec8
Merge branch 'master' into keeper-storage
2022-09-09 12:24:01 +00:00
Antonio Andelic
951ee44229
Process read requests with snapshot install
2022-09-07 11:51:56 +00:00
Antonio Andelic
a377a5f1e9
Change assert to LOGICAL_ERROR
2022-09-07 09:24:50 +00:00
Antonio Andelic
4cdf18cc81
Shutdown storage after RAFT
2022-09-07 09:20:47 +00:00
Antonio Andelic
c6b17bd7f9
Don't preprocess if storage finalized
2022-09-07 09:14:03 +00:00
Antonio Andelic
f0ba3ee927
Fix typo
2022-09-05 09:51:02 +02:00
Antonio Andelic
110ad53300
Merge branch 'master' into keeper-linearizable-reads
2022-09-05 07:12:45 +00:00
Antonio Andelic
b816d46520
Merge branch 'master' into fix-keeper-system-path-check
2022-09-02 13:42:16 +02:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage
2022-09-02 12:13:07 +02:00
Antonio Andelic
e64436fef3
Fix typos with new codespell
2022-09-02 08:54:48 +00:00
Antonio Andelic
ad2196155c
Add test for system node modification
2022-09-02 08:14:06 +00:00
Antonio Andelic
b7eebfc626
Correctly check if the node is using system path
2022-09-02 07:47:12 +00:00
Antonio Andelic
7164e59b54
Merge branch 'master' into keeper-linearizable-reads
2022-09-01 08:52:31 +00:00
Antonio Andelic
fa4c7259f9
Fix typos
2022-09-01 08:46:28 +00:00
Antonio Andelic
c7d1dd942c
Address PR comments
2022-09-01 07:56:06 +00:00
Antonio Andelic
21cdb9a500
Don't remove from cache if it wasn't added during rollback
2022-08-30 08:14:53 +00:00
Antonio Andelic
2416ddfae7
Add extra message about startup preprocessing
2022-08-30 08:14:31 +00:00
Antonio Andelic
57471d11db
Fix uncommitted auth
2022-08-29 12:55:30 +00:00
Antonio Andelic
50af0b1938
Add bunch of comments for request processing
2022-08-29 10:50:18 +00:00
Antonio Andelic
b4af21b6fb
Small fixes
2022-08-29 07:06:51 +00:00
Antonio Andelic
bff2b232ef
Cache deltas for path
2022-08-26 14:03:38 +00:00
Antonio Andelic
26d68dce23
Fix style
2022-08-26 10:55:07 +00:00
Antonio Andelic
dae65178b7
Improve preprocessing performance
2022-08-26 10:46:14 +00:00
Antonio Andelic
5d45f2faa9
Add comments
2022-08-23 08:22:03 +00:00
Antonio Andelic
46882791cd
Fix build
2022-08-23 07:29:47 +00:00
Antonio Andelic
444ac9af6c
Merge branch 'master' into keeper-linearizable-reads
2022-08-23 07:03:35 +00:00
Antonio Andelic
99d23cb67e
Merge branch 'master' into keeper-linearizable-reads
2022-08-23 07:03:20 +00:00
Antonio Andelic
a87a762b0c
Merge pull request #39973 from ClickHouse/keeper-listen-host
...
Support `interserver_listen_host` in Keeper
2022-08-23 09:03:08 +02:00
Alexey Milovidov
74e1f4dc61
Fix clang-tidy
2022-08-20 17:09:20 +02:00
Antonio Andelic
9396834762
Addressed comments
2022-08-17 09:29:08 +00:00
Antonio Andelic
24ee3e022d
Merge branch 'master' into keeper-listen-host
2022-08-09 09:39:27 +02:00
Alexey Milovidov
48bed932ec
Merge pull request #39931 from HarryLeeIBM/hlee-s390x-coordination
...
Fix Endian issue in KeeperSnapshotManager for s390x
2022-08-09 04:24:32 +03:00
HarryLeeIBM
42323cfb92
Use constexpr char array as magic number
2022-08-08 10:59:09 -07:00
Antonio Andelic
6b935a252f
Support multiple listen hosts
2022-08-08 09:08:19 +00:00
Alexey Milovidov
adc880f362
Merge pull request #39867 from ClickHouse/keeper-block-memory-tracker-commit
...
Block memory tracker in Keeper during commit
2022-08-06 02:02:29 +03:00
HarryLeeIBM
cbd01a3a73
Fix Endian issue in KeeperSnapshotManager
2022-08-05 11:07:42 -07:00
Antonio Andelic
07a85513e8
Merge branch 'master' into keeper-create-snapshot-on-exit
2022-08-04 13:12:26 +00:00
Antonio Andelic
10d7259c2b
Add log for snapshot on exit
2022-08-04 13:12:24 +00:00
Antonio Andelic
0a7d2e7b8a
Use LockMemoryExceptionThread
2022-08-04 13:03:05 +00:00
Antonio Andelic
278921be3b
Fix style
2022-08-04 07:25:28 +00:00
Antonio Andelic
fa98338ce1
Add comment for memory tracker blocking
2022-08-04 07:13:42 +00:00
Antonio Andelic
56ca93ca3f
Block memory tracker in Keeper during commit
2022-08-03 14:14:57 +00:00
Antonio Andelic
05467e315f
Merge pull request #39698 from ClickHouse/update-digest-version
...
Update Keeper version for digest
2022-08-03 13:13:00 +02:00
Antonio Andelic
b3b3c371f0
Update KeeperStorage.h
2022-08-02 09:20:02 +02:00
Antonio Andelic
7cdad88327
Create snapshot on shutdown
2022-07-30 11:34:17 +00:00
Antonio Andelic
21becb3030
Update version for digest
2022-07-29 07:36:19 +00:00
Antonio Andelic
e3cb65a99a
Rollback on failed preprocess
2022-07-29 06:51:53 +00:00
Antonio Andelic
c0a5d45258
Rollback on failed PreAppend
2022-07-28 14:06:45 +00:00
Antonio Andelic
1090d6bca7
Rollback request if storing log fails
2022-07-28 12:55:29 +00:00
Antonio Andelic
e62526720f
Address PR comments
2022-07-27 07:51:30 +00:00
Antonio Andelic
04b03b6a90
Don't allow to overwrite on startup
2022-07-25 12:38:48 +00:00
Antonio Andelic
afb6cb6824
Add KeeperContext
2022-07-23 14:45:31 +00:00
Antonio Andelic
6aff87d4b5
Ignore system paths from snapshots and logstore
2022-07-23 13:46:59 +00:00
Antonio Andelic
46a6fbd7aa
Small polish
2022-07-22 11:03:07 +00:00
Antonio Andelic
3040ff0959
Merge branch 'master' into keeper-version-check
2022-07-22 08:07:55 +00:00
Antonio Andelic
75476d5110
Store only root system node
2022-07-22 08:07:38 +00:00
Antonio Andelic
fdd79e62da
Merge pull request #39393 from ClickHouse/keeper-disable-digest-default
...
Disable real-time digest in Keeper by default
2022-07-21 15:15:22 +02:00
Antonio Andelic
7d30ab80c4
Fix compatibility
2022-07-21 09:31:06 +00:00
Alexey Milovidov
19c4c0bd6e
Update CoordinationSettings.cpp
2022-07-21 06:41:42 +03:00
Nikolai Kochetov
91043351aa
Fixing build.
2022-07-20 20:30:16 +00:00
Antonio Andelic
a4450225c7
Disable digest by default
2022-07-19 17:14:44 +00:00
Antonio Andelic
a3e00faaf9
Fix unit tests
2022-07-19 13:14:36 +00:00
Antonio Andelic
ce570b6ee3
Add logs and 4LW for api version
2022-07-19 09:02:57 +00:00
Antonio Andelic
e6ded88ea3
Small refactoring
2022-07-19 08:51:12 +00:00
Antonio Andelic
cfc741030f
Revert to the version with path
2022-07-18 14:30:15 +00:00
Antonio Andelic
48db20d5a1
Add 4LW for api version
2022-07-18 12:19:16 +00:00
Antonio Andelic
b8ffb151dc
Some small polishing
2022-07-18 08:52:52 +00:00
Antonio Andelic
7d7c4ce4cd
Add ApiVersion request
2022-07-18 08:44:38 +00:00
Antonio Andelic
73e0c35ab0
Use 4LW for api version
2022-07-15 10:13:39 +00:00
Antonio Andelic
d81758898a
Remove extra newline
2022-07-13 15:26:25 +00:00
Antonio Andelic
f05f22aa13
Merge branch 'master' into keeper-version-check
2022-07-13 15:25:48 +00:00
Antonio Andelic
3b5bdd1e2a
Add test for current API
2022-07-13 12:41:16 +00:00
Antonio Andelic
c296e84cab
fix unit tests in debug mode
2022-07-13 12:33:18 +00:00
Antonio Andelic
838a11a85a
Rename RaftAppendResult
2022-07-13 11:35:06 +00:00
Antonio Andelic
2634f80956
Remove trailing whitespaces
2022-07-13 11:06:58 +00:00
Antonio Andelic
b49ac48bfd
Merge branch 'master' into keeper-linearizable-reads
2022-07-13 09:30:21 +00:00
Nikita Mikhaylov
1f0280bcde
Fix path retrieval for Keeper's state ( #39148 )
2022-07-13 11:25:34 +02:00
Antonio Andelic
ee4828b084
Fix list watches
2022-07-13 09:23:05 +00:00
Antonio Andelic
8bd86c39f6
Add valid values for read_mode setting
2022-07-12 08:00:31 +00:00
Antonio Andelic
fd1e5d43a0
fix unit tests in debug mode
2022-07-12 07:52:50 +00:00
Antonio Andelic
5f661e3e2b
Merge branch 'master' into keeper-linearizable-reads
2022-07-12 07:33:34 +00:00
Antonio Andelic
0555477ec1
Remove debug logs
2022-07-12 07:05:29 +00:00
Antonio Andelic
6dcdf0fcb1
Throw CORRUPTED_DATA in debug mode for invalid checksum
2022-07-12 06:41:58 +00:00
Antonio Andelic
1f5b3b9922
Add restrictions for modification of internal paths
2022-07-11 13:06:24 +00:00
Antonio Andelic
568ba5d3cc
Address PR comments
2022-07-11 12:56:09 +00:00
Antonio Andelic
efc7f5c534
Merge branch 'master' into keeper-persist-state
2022-07-11 12:43:06 +00:00
Antonio Andelic
41dd4e13f4
Merge branch 'master' into keeper-version-check
2022-07-11 09:38:20 +00:00
Antonio Andelic
e4f7635df7
Format
2022-07-10 19:10:27 +00:00
Antonio Andelic
183f5cc940
Add checksum to the state file
2022-07-10 15:01:38 +00:00
Antonio Andelic
2b26865f90
Unit test fixes
2022-07-10 14:40:08 +00:00
Antonio Andelic
4622084d2a
Add test for persisted state
2022-07-10 09:47:04 +00:00
Antonio Andelic
f60074e9ec
Make state durable
2022-07-10 09:33:49 +00:00
Antonio Andelic
37f799550b
Merge pull request #38072 from lingpeng0314/master
...
Add Keeper related monitoring data
2022-07-10 09:39:57 +02:00
Alexander Tokmakov
7630fc76ae
avoid weird exception
2022-07-07 17:19:05 +02:00
Antonio Andelic
e96af48e28
Polish
2022-07-07 11:35:02 +00:00
alesapin
bd8a3ee841
Merge pull request #38338 from ClickHouse/zookeeper-add-extra-list-argument
...
Extend ZooKeeper list request with support for filtering persistent or ephemeral nodes only
2022-07-05 12:53:51 +02:00
Antonio Andelic
db94955319
Merge pull request #38556 from ClickHouse/keeper-dont-rollback-session-id
...
Don't rollback SessionID request in Keeper
2022-07-05 10:36:56 +02:00
alesapin
dde76824ca
Fix ub
2022-07-02 16:02:42 +02:00
Antonio Andelic
8c77632881
Fix style
2022-07-01 14:58:19 +00:00
Antonio Andelic
97e32e6ad9
Use commit_ext in unit tests
2022-07-01 14:55:26 +00:00
alesapin
919baf331e
Review fixes
2022-07-01 15:57:24 +02:00
Antonio Andelic
5a97c0fc8b
Finalize requests in separate thread
2022-07-01 13:19:28 +00:00
Antonio Andelic
ae0fe1ac85
Add setting for read mode
2022-07-01 09:30:59 +00:00
alesapin
66705eacb0
Add test for keeper mntr command
2022-07-01 00:07:22 +02:00
Antonio Andelic
7333e36639
Merge branch 'master' into keeper-linearizable-reads
2022-06-29 13:59:31 +00:00
Antonio Andelic
2de659715b
Don't rollback SessionID request
2022-06-29 10:21:37 +00:00
Antonio Andelic
7a8fbbf132
Merge branch 'master' into zookeeper-add-extra-list-argument
2022-06-29 08:31:03 +00:00
Antonio Andelic
1d26446f84
Define new list request type
2022-06-29 08:30:39 +00:00
Antonio Andelic
77144322d7
Add extra checks for read result
2022-06-29 07:39:39 +00:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to
...
Replace where possible by std::lock_guard which is more light-weight.
2022-06-28 19:19:06 +00:00
Antonio Andelic
73000a042d
Don't reconnect manually
2022-06-28 13:44:30 +00:00
Antonio Andelic
6d7050cb42
Additional refactoring for request processing
2022-06-28 09:54:16 +00:00
Antonio Andelic
23c705222f
Process read requests in bg thread
2022-06-28 08:48:19 +00:00
Antonio Andelic
a6c628cab4
Small fixes for queues
2022-06-27 14:34:27 +00:00
Antonio Andelic
dc5cc33769
Process outside of lock
2022-06-27 14:22:53 +00:00
Antonio Andelic
53e906fdd5
Improve batching by processing single request from each session
2022-06-27 12:05:27 +00:00
Antonio Andelic
8a3531c2db
Refactor batching requests logic
2022-06-27 07:30:59 +00:00
Antonio Andelic
0d906e311d
PR review fixes
2022-06-27 06:53:09 +00:00
Antonio Andelic
76c4fd9c8a
Some fixes for batch reads
2022-06-24 14:42:24 +00:00
Antonio Andelic
ecd545fe4f
Batch read requests
2022-06-24 11:36:12 +00:00
Antonio Andelic
91637bf79a
Merge branch 'master' into keeper-linearizable-reads
2022-06-23 14:07:00 +00:00
Antonio Andelic
2e71ae97a1
Add unit test for different list request types
2022-06-23 10:28:12 +00:00
Antonio Andelic
ae3d9fd962
Expose list request type in internal client
2022-06-23 10:28:12 +00:00
Antonio Andelic
568c957eb2
Fix TestKeeper
2022-06-23 10:28:12 +00:00
Antonio Andelic
3a71b63b5d
Add list request type
2022-06-23 10:28:12 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa
2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA)
...
- TSA is a static analyzer build by Google which finds race conditions
and deadlocks at compile time.
- It works by associating a shared member variable with a
synchronization primitive that protects it. The compiler can then
check at each access if proper locking happened before. A good
introduction are [0] and [1].
- TSA requires some help by the programmer via annotations. Luckily,
LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
std::shared_mutex and std::scoped_lock. This commit enables them
(--> contrib/libcxx-cmake/CMakeLists.txt).
- Further, this commit adds convenience macros for the low-level
annotations for use in ClickHouse (--> base/defines.h). For
demonstration, they are leveraged in a few places.
- As we compile with "-Wall -Wextra -Weverything", the required compiler
flag "-Wthread-safety-analysis" was already enabled. Negative checks
are an experimental feature of TSA and disabled
(--> cmake/warnings.cmake). Compile times did not increase noticeably.
- TSA is used in a few places with simple locking. I tried TSA also
where locking is more complex. The problem was usually that it is
unclear which data is protected by which lock :-(. But there was
definitely some weird code where locking looked broken. So there is
some potential to find bugs.
*** Limitations of TSA besides the ones listed in [1]:
- The programmer needs to know which lock protects which piece of shared
data. This is not always easy for large classes.
- Two synchronization primitives used in ClickHouse are not annotated in
libcxx:
(1) std::unique_lock: A releaseable lock handle often together with
std::condition_variable, e.g. in solve producer-consumer problems.
(2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
considered a design flaw + typically it is slower than a standard
mutex. In this commit, one std::recursive_mutex was converted to
std::mutex and annotated with TSA.
- For free-standing functions (e.g. helper functions) which are passed
shared data members, it can be tricky to specify the associated lock.
This is because the annotations use the normal C++ rules for symbol
resolution.
[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
2022-06-20 16:13:25 +02:00
lingpeng0314
f06b19bdbf
Add keeper monitoring data
2022-06-20 11:32:18 +08:00
Antonio Andelic
d56894e04e
Don't rollback session_id
2022-06-18 19:16:17 +00:00
Antonio Andelic
d2f3d581eb
Use fast lin reads in jepsen
2022-06-18 18:43:02 +00:00
Antonio Andelic
9fe06019c0
Some small fixes
2022-06-17 18:42:46 +00:00
Antonio Andelic
125e1bd182
Initial version for linearizable reads
2022-06-17 18:19:09 +00:00
Antonio Andelic
ac0b7ab20b
Fix backwards compatibility with older snapshots
2022-06-15 13:46:27 +00:00
Antonio Andelic
7e1f64002d
Address review comments
2022-06-15 12:48:30 +00:00
Antonio Andelic
6e55593398
Fix remove preprocess
2022-06-14 13:23:46 +00:00
Antonio Andelic
b7bd5a8eb1
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations
2022-06-14 12:51:35 +00:00
Maksim Kita
da8b1b1eba
Merge pull request #38025 from kitaisreal/use-base-sort-instead-of-standard
...
Use pdqsort instead of standard sort
2022-06-14 12:13:35 +02:00
Antonio Andelic
5d4b289550
Update stats in update delta
2022-06-14 09:14:53 +00:00
Antonio Andelic
bf27fe284c
Use std::list for deltas
2022-06-14 07:37:02 +00:00
Antonio Andelic
7e99e9fe43
Fix child num assert
2022-06-14 07:17:11 +00:00
Antonio Andelic
e5504f1b33
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations
2022-06-13 15:43:10 +00:00
Maksim Kita
98a89b50ff
Use pdqsort instead of standard sort
2022-06-13 15:31:08 +02:00
mergify[bot]
2bf9818af0
Merge branch 'master' into keeper-broken-to-detached
2022-06-13 10:04:54 +00:00
alesapin
aff6b7a8d6
Fix keeper storage
2022-06-12 00:01:41 +02:00
alesapin
aa1fb9b2e7
Remove trash
2022-06-10 15:27:27 +02:00
alesapin
4c574f30af
Add comment
2022-06-10 15:12:14 +02:00
alesapin
682eb6bcc9
Add checks for numChildren
2022-06-10 15:10:45 +02:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros
...
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.
__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
2022-06-10 10:22:31 +02:00
Antonio Andelic
2c37fe3d7b
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations
2022-06-06 12:39:55 +00:00
Antonio Andelic
3de1d10d93
move comment
2022-06-06 12:37:03 +00:00
Antonio Andelic
6bb2d897b4
Merge branch 'master' into keeper-broken-to-detached
2022-06-06 12:19:34 +00:00
Antonio Andelic
fbdc499533
Extract more logic into a function
2022-06-06 12:19:13 +00:00
Michael Lex
041b8f6a22
Fix bug in clickhouse-keeper: dead_session_check_period was passed as micros instead of millis. ( #37824 )
2022-06-03 15:35:45 +02:00
mergify[bot]
d115ddb40c
Merge branch 'master' into keeper-broken-to-detached
2022-06-02 07:39:13 +00:00
Alexey Milovidov
b5f48a7d3f
Merge branch 'master' of github.com:ClickHouse/ClickHouse into llvm-14
2022-06-01 22:09:58 +02:00
mergify[bot]
4aefcf29e3
Merge branch 'revert-37534-revert-37036-keeper-preprocess-operations' into keeper-real-time-digest
2022-06-01 10:17:49 +00:00
mergify[bot]
cbb26d1677
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations
2022-06-01 10:16:41 +00:00
