Antonio Andelic
47966c4e81
Finalize bufferg
2023-06-02 14:10:52 +00:00
Antonio Andelic
336c9d7136
Apply uncommitted state after snapshot deser
2023-06-02 13:31:14 +00:00
Antonio Andelic
ffd4f7f196
Add better support for state disks
2023-06-01 14:39:01 +00:00
Antonio Andelic
e8442b7a88
Add finalize calls
2023-06-01 12:54:00 +00:00
frinkr
c9d0d217f5
fix Keeper deadlock on exception when preprocessing requests
2023-06-01 09:57:42 +08:00
Konstantin Bogdanov
b61ffe3ff7
Merge checks
2023-06-01 03:22:40 +02:00
Konstantin Bogdanov
3a3539b996
Style fix
2023-06-01 03:22:40 +02:00
Konstantin Bogdanov
203276dfcc
Use storage.uncommited_state
2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
965f7850f8
Cleanup
2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
f84fdb7f10
Increment Keeper API version
2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
b53f36369e
Remove new request object
2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
0d6561ee77
Prototype
2023-06-01 03:22:23 +02:00
Antonio Andelic
21aba94909
Add support for latest snapshot disk
2023-05-30 13:22:40 +00:00
Antonio Andelic
67746bfff2
Merge branch 'master' into keeper-with-disks
2023-05-30 09:50:12 +00:00
Raúl Marín
1ae1f1273a
Avoid clang 15 crash
2023-05-29 11:40:45 +02:00
Antonio Andelic
0740bfbe4b
Better disks moving
2023-05-26 14:31:09 +00:00
Antonio Andelic
a9569478e6
fix typo
2023-05-26 13:09:09 +02:00
Antonio Andelic
8a2a63a7bd
Merge branch 'master' into keeper-with-disks
2023-05-26 10:48:41 +00:00
Antonio Andelic
3d98e591ba
Handle backwards compatibility
2023-05-26 10:46:18 +00:00
Antonio Andelic
767193caf5
Merge branch 'master' into keeper-log-improvements
2023-05-26 07:35:36 +00:00
Antonio Andelic
6a8a21e09a
Add test for s3 disk restore
2023-05-25 16:01:40 +00:00
Antonio Andelic
bb77441acb
Add support for old storage disks
2023-05-25 13:31:11 +00:00
Antonio Andelic
f519aa4613
Calculate size
2023-05-25 11:23:03 +00:00
Antonio Andelic
4483602c50
Remove double initialization
2023-05-25 08:52:14 +00:00
Antonio Andelic
7e68f61df1
Fix snapshot recovery
2023-05-25 08:43:11 +00:00
Antonio Andelic
092cf99147
Fix build
2023-05-24 13:45:14 +00:00
Antonio Andelic
fa3fb49d33
Merge branch 'master' into keeper-with-disks
2023-05-24 09:04:47 +00:00
Antonio Andelic
5db2160762
Fix build
2023-05-24 09:04:12 +00:00
Antonio Andelic
161afea266
Add support for changelog
2023-05-24 07:43:57 +00:00
kssenii
295fe3b228
Merge remote-tracking branch 'upstream/master' into abstract-async-prefetched-buffer
2023-05-22 20:01:38 +02:00
kssenii
c4d862a16f
Make async reader work with any impl
2023-05-22 19:54:04 +02:00
Antonio Andelic
bde2cf96b1
Better
2023-05-22 12:24:16 +00:00
alesapin
cc3897a84a
Merge pull request #50033 from kssenii/disk-object-storage-minor-changes
...
Get rid of indirect write buffer in object storages
2023-05-22 14:08:17 +02:00
Antonio Andelic
dc31396035
Merge branch 'master' into keeper-with-disks
2023-05-22 08:10:24 +00:00
kssenii
3e42ee7f2b
Get rid of finalize callback in object storages
2023-05-19 17:29:37 +02:00
Antonio Andelic
9c3b17fa18
Remove whitespace
2023-05-19 13:00:51 +02:00
Antonio Andelic
e46476dba2
Update src/Coordination/Changelog.cpp
...
Co-authored-by: alesapin <alesapin@clickhouse.com>
2023-05-19 12:44:20 +02:00
Antonio Andelic
6e468b29e8
Check return value of ftruncate
2023-05-19 10:15:06 +00:00
Antonio Andelic
bf6afd27a6
build with disks
2023-05-18 07:39:22 +00:00
Antonio Andelic
465123fa1b
Merge branch 'master' into keeper-with-disks
2023-05-16 08:07:06 +00:00
Azat Khuzhin
665545ec45
Fix "reference to local binding" after fixes for clang-17
...
Follow-up for: #49851 (cc @alexey-milovidov)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-15 12:45:20 +02:00
Alexey Milovidov
5a44dc26e7
Fixes for clang-17
2023-05-13 02:57:31 +02:00
Antonio Andelic
bb8b9dc9c3
Merge branch 'master' into keeper-log-improvements
2023-05-05 11:57:14 +00:00
alesapin
9456eb59fd
Merge pull request #48466 from ClickHouse/keeper-correctly-handle-snapshots
...
Correctly handle concurrent snapshots in Keeper
2023-05-04 19:16:00 +02:00
johanngan
731823b873
Add support for SSE-KMS configuration with S3
...
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html
Similar to the server_side_encryption_customer_key_base64 option for
configuring SSE-C with S3, add the following settings to configure
SSE-KMS on a per-endpoint/disk basis:
- server_side_encryption_kms_key_id
- server_side_encryption_kms_encryption_context
- server_side_encryption_kms_bucket_key_enabled
2023-05-03 21:35:38 -05:00
Antonio Andelic
b8b7a362b2
Merge branch 'master' into keeper-correctly-handle-snapshots
2023-05-03 13:51:09 +02:00
Antonio Andelic
78ecfac8d5
Merge branch 'master' into keeper-log-improvements
2023-05-03 08:31:21 +00:00
Antonio Andelic
dce61d7321
Merge branch 'master' into keeper-log-improvements
2023-05-02 11:14:48 +00:00
Antonio Andelic
f695040a19
Merge pull request #48897 from ClickHouse/keeper-check-not-exists
...
Add `CheckNotExists` request to Keeper
2023-05-02 13:14:22 +02:00
Raúl Marín
f0e045bb3d
Merge remote-tracking branch 'blessed/master' into arenita
2023-04-24 10:42:56 +02:00
Alexey Milovidov
530b764953
Fix IBM
2023-04-21 12:38:45 +02:00
Antonio Andelic
d500a5637d
Cache large requests
2023-04-21 09:43:04 +00:00
Antonio Andelic
b52646c555
Eliminate double log serializing
2023-04-21 08:20:35 +00:00
Antonio Andelic
58e9b56fcb
Fix CheckNotExists
2023-04-19 09:06:20 +00:00
Antonio Andelic
fb16623d48
Add CheckNotExists request to Keeper
2023-04-18 14:45:24 +00:00
Raúl Marín
2b70e08f23
Don't count unreserved bytes in Arenas as read_bytes
2023-04-13 12:43:24 +02:00
Antonio Andelic
b30aff06d8
Merge pull request #48595 from aalexfvk/alexfvk/fix_create_not_auth_problem
...
Fix bug in Keeper when a node is not created with scheme `auth` in ACL sometimes.
2023-04-11 14:41:37 +02:00
Aleksei Filatov
dd2364361c
Update src/Coordination/tests/gtest_coordination.cpp
...
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2023-04-11 10:45:30 +03:00
Azat Khuzhin
79b83c4fd2
Remove superfluous includes of logger_userful.h from headers
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-10 17:59:30 +02:00
Aleksei Filatov
e8bf96d126
Add unit tests
2023-04-10 13:02:43 +03:00
Aleksei Filatov
000c2b5b91
Correct FixupACL for auth scheme
2023-04-10 13:02:31 +03:00
Azat Khuzhin
011480924a
Use forward declaration of ThreadPool
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-07 11:25:35 +02:00
Antonio Andelic
88f77c686d
Correctly handle concurrent snapshots
2023-04-06 15:40:05 +00:00
Antonio Andelic
e982f2a67a
Merge branch 'master' into enable-env-credentials-default
2023-03-31 09:11:01 +00:00
Antonio Andelic
44e95aa65f
Merge branch 'master' into nosign-keyword-for-s3
2023-03-29 11:10:03 +00:00
Antonio Andelic
160aa186bb
Add support for NOSIGN keyword and no_sign_request config
2023-03-28 07:05:35 +00:00
Antonio Andelic
79e22d13fb
small fix
2023-03-27 07:41:31 +00:00
Antonio Andelic
f355bc5342
Merge branch 'master' into more-batching-keeper
2023-03-27 07:28:17 +00:00
Antonio Andelic
6c3765c8b6
Small fix
2023-03-24 20:40:45 +00:00
Antonio Andelic
f353561204
Avoid breaking batches with read requests
2023-03-24 14:37:40 +00:00
Alexander Gololobov
afe3af230f
Limit keeper request batching by size in bytes
2023-03-24 14:41:09 +01:00
Antonio Andelic
51416da818
Merge branch 'master' into enable-env-credentials-default
2023-03-13 10:20:21 +00:00
Antonio Andelic
a170a909a4
Add expiration window for S3 credentials
2023-03-10 10:06:32 +00:00
Antonio Andelic
83a3243758
Merge pull request #47065 from ClickHouse/debug-keeper
...
Fix Keeper with `force_sync = false`
2023-03-10 09:27:13 +01:00
Antonio Andelic
5bc21538e5
Enable use_environment_credentials by default
2023-03-09 10:31:55 +00:00
Antonio Andelic
0ad436aa2c
Revert debug changes
2023-03-09 08:07:59 +00:00
Antonio Andelic
b4061d8aa4
Add logs
2023-03-08 13:06:10 +00:00
Mike Kot
9920a52c51
use std::lerp, constexpr hex.h
2023-03-07 22:50:17 +00:00
Antonio Andelic
dd4db9132a
Merge branch 'master' into debug-keeper
2023-03-07 13:15:05 +01:00
Antonio Andelic
c38d820d0f
Add 4LW for cleaning resource
2023-03-05 16:57:18 +00:00
Antonio Andelic
cd4100cb19
Merge branch 'master' into fix-data-size
2023-03-03 15:33:43 +01:00
Antonio Andelic
fdcb5b784f
Merge branch 'master' into debug-keeper
2023-03-03 13:45:13 +01:00
Robert Schulze
740aeaba1f
Apply some CTAD
2023-03-02 13:36:47 +00:00
Antonio Andelic
6ebee202bd
Flush buffer when no sync
2023-03-01 14:47:15 +00:00
Antonio Andelic
c797122dbe
Better
2023-03-01 08:45:50 +00:00
Antonio Andelic
6e28406751
Merge branch 'master' into fix-data-size
2023-02-28 20:09:29 +00:00
Alexey Milovidov
d8cda3dbb8
Remove PVS-Studio
2023-02-19 23:30:05 +01:00
Alexander Tokmakov
e96340e183
Merge pull request #46410 from ClickHouse/fix_noisy_log_messages
...
Fix noisy log messages
2023-02-15 01:22:42 +03:00
Alexander Tokmakov
1bf3735ca9
fix noisy log messages
2023-02-14 15:33:27 +01:00
Sema Checherinda
9bef1bec28
Merge pull request #46274 from aiven-sal/aiven-sal/keeper
...
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
2023-02-14 12:44:04 +01:00
Sema Checherinda
5882f3878f
except zero values election_timeout
2023-02-13 12:22:08 +01:00
Salvatore Mesoraca
fff2c09b87
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
2023-02-13 10:09:45 +01:00
Antonio Andelic
f96d480563
Merge branch 'master' into add-support-for-settings-alias
2023-02-09 16:07:45 +01:00
Azat Khuzhin
8cc41b7f41
Check return value of ::close()
...
Note, that according close(2), EINTR should not be retriable for close()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-07 11:28:22 +01:00
Antonio Andelic
b11458f142
Fix erase
2023-02-07 07:51:22 +00:00
Antonio Andelic
9cde46acff
Correctly calculate size on update
2023-02-06 16:28:05 +00:00
Antonio Andelic
aca4f08bf5
Correctly calculate size after snapshot load
2023-02-06 14:59:23 +00:00
Antonio Andelic
8929838fcc
Merge branch 'master' into add-support-for-settings-alias
2023-02-03 17:47:00 +00:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
LiuYangkuan
f29700bd2f
use IDisk to do IO in Keeper's snapshots and logs
2023-02-02 19:47:30 +08:00
Antonio Andelic
f613bfd8d2
Merge branch 'master' into add-support-for-settings-alias
2023-02-02 11:15:39 +00:00
Antonio Andelic
85cfee4bb9
Better alias definition
2023-02-01 13:54:03 +00:00
Antonio Andelic
88c1744975
Better
2023-02-01 14:25:58 +01:00
Antonio Andelic
5de6f86c27
Fix clang-tidy error
2023-02-01 12:28:44 +00:00
Antonio Andelic
ec04cf6876
Merge branch 'master' into add-support-for-settings-alias
2023-02-01 09:01:01 +00:00
Antonio Andelic
ece92fa866
Update src/Coordination/Changelog.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2023-01-31 17:11:18 +01:00
Antonio Andelic
8457d3bf3d
Apply comments
2023-01-31 14:18:54 +00:00
Antonio Andelic
1786237be6
Merge branch 'master' into keeper-use-fallocate
2023-01-31 13:53:47 +00:00
Vitaly Baranov
f9282cfcac
Make version_id an optional parameter again.
2023-01-27 15:10:19 +01:00
Vitaly Baranov
aea9ccdb60
Pass request settings to S3::getObjectInfo().
2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed
Move getObjectInfo() to a separate header.
2023-01-27 15:09:38 +01:00
Antonio Andelic
714fad1529
Add support for settings alias
2023-01-26 14:06:46 +00:00
Antonio Andelic
ebfe7f9991
Fix style
2023-01-26 11:18:07 +00:00
Antonio Andelic
1186718658
Add overallocate size setting
2023-01-26 09:44:37 +00:00
Antonio Andelic
7bfdfa4bf7
Preallocate more space
2023-01-25 16:26:14 +00:00
Antonio Andelic
7421d8f872
Merge branch 'master' into keeper-use-fallocate
2023-01-25 11:36:47 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Alexander Tokmakov
870cfcc36a
less fmt::runtime usages
2023-01-17 00:11:59 +01:00
Antonio Andelic
6a0e481ed7
Merge branch 'master' into keeper-use-fallocate
2023-01-12 13:57:05 +00:00
Antonio Andelic
30510c5eb6
Merge pull request #44937 from ClickHouse/fix-changelog-deadlock
...
Fix deadlock in Keeper's changelog
2023-01-10 08:59:51 +01:00
Antonio Andelic
22679ddf11
Add comment
2023-01-09 14:26:20 +01:00
Antonio Andelic
a39f437306
Typo fix
2023-01-09 13:34:48 +01:00
Antonio Andelic
27b2bae640
make it better
2023-01-09 12:01:22 +00:00
Antonio Andelic
4fcbe14435
Small fixes for Coordination unit tests
2023-01-08 13:44:32 +00:00
Kseniia Sumarokova
573d3283b0
Merge pull request #44327 from kssenii/use-new-named-collections-code-2
...
Replace old named collections code with new (from #43147 ) part 2
2023-01-06 13:06:26 +01:00
Antonio Andelic
9b3314a931
Fix deadlock in Changelog flush
2023-01-05 14:51:05 +00:00
Antonio Andelic
136436958b
Merge pull request #44908 from ClickHouse/fix-keeper-snapshot-on-exit
...
Avoid Keeper crash on shutdown (fix `test_keeper_snapshot_on_exit`)
2023-01-05 08:59:58 +01:00
Antonio Andelic
7c2cdfa65a
Don't throw exception if queue is finished
2023-01-04 15:46:38 +00:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
Smita Kulkarni
8a5fd7f96d
Added comments and addressed review comments - Allow macro substitution in endpoint
2023-01-03 11:44:26 +01:00
Antonio Andelic
f2653c016a
Manually set deleted flag
2023-01-02 14:36:58 +00:00
Antonio Andelic
f32fab5025
Check if file exists before rename
2023-01-02 13:48:13 +00:00
Antonio Andelic
2e93640614
Fix ASAN error
2023-01-02 13:37:40 +00:00
Antonio Andelic
b845f880fb
Polish
2023-01-02 10:42:15 +00:00
Antonio Andelic
14355d8009
Merge branch 'master' into keeper-use-fallocate
2023-01-02 08:22:27 +00:00
Antonio Andelic
55548a6678
style fix
2023-01-02 08:22:00 +00:00
SmitaRKulkarni
f9a3aeb172
Merge branch 'master' into 40951_Allow_macro_substitution_in_endpoint
2022-12-30 10:43:14 +01:00
Alexey Milovidov
f0fa0977ed
Fix wrong log messages
2022-12-26 18:36:07 +01:00
Smita Kulkarni
3d62bc2635
Allow macro substitution in endpoint of disks
...
Implementation:
* Updated to substitute macros in endpoint for all disks (s3, hfs, web).
Testing:
* Added an integration test test/integration/test_endpoint_macro_substitution.
2022-12-23 15:09:24 +01:00
Antonio Andelic
64ade259cf
Fix tests
2022-12-22 10:20:55 +00:00
Antonio Andelic
88d41dea93
Merge branch 'master' into keeper-use-fallocate
2022-12-21 08:59:09 +00:00
Antonio Andelic
9711355cc9
Fix tests
2022-12-21 08:43:55 +00:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
Antonio Andelic
6d83221d4e
Remove unused error code
2022-12-19 12:20:26 +00:00
Antonio Andelic
c2c115d2ba
Add setting for max_log_file_size
2022-12-19 11:51:05 +00:00
Antonio Andelic
f241701c2a
Add checks when fallocate is not supported
2022-12-19 11:48:54 +00:00
Antonio Andelic
9a8b248735
Merge branch 'master' into keeper-use-fallocate
2022-12-19 09:32:05 +00:00
Nikolai Kochetov
62ff98344e
Validate s3 part upload settings.
2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Antonio Andelic
ee900ca7df
Add truncate
2022-12-15 13:09:52 +00:00
Antonio Andelic
bf6f163952
Merge branch 'master' into keeper-use-fallocate
2022-12-15 09:05:40 +00:00
Antonio Andelic
d454e00016
Changelog refactor
2022-12-15 08:12:02 +00:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
Antonio Andelic
662c1f956c
Changelog refactoring
2022-12-13 14:00:27 +00:00
Antonio Andelic
898cf867af
Try with fallocate
2022-12-13 11:08:59 +00:00
Vitaly Baranov
b91af1b650
Fix initialization of s3 request settings.
2022-12-10 05:43:51 +01:00
Antonio Andelic
e1a6a1182e
Throw when port already in use ( #43984 )
2022-12-07 13:27:49 +01:00
Antonio Andelic
200141da13
Use logger in tests
2022-12-02 08:16:18 +00:00
Antonio Andelic
7cd76df918
Merge branch 'master' into parallel-log-appending
2022-12-01 14:44:36 +00:00
Antonio Andelic
554ff1db6f
Small changes
2022-12-01 12:54:29 +00:00
Antonio Andelic
940fdd4abd
Remove comment
2022-12-01 12:24:27 +01:00
Antonio Andelic
25ae52111e
Protect writer with mutex
2022-12-01 11:18:23 +00:00
Antonio Andelic
c12c0cc8af
Merge branch 'master' into parallel-log-appending
2022-12-01 08:53:52 +00:00
Antonio Andelic
2f501d5327
Merge branch 'master' into keeper-prometheus
2022-11-30 13:24:37 +01:00
Antonio Andelic
fbda86ddc8
PR fixes
2022-11-30 13:24:08 +01:00
Antonio Andelic
7f897f1a20
Refactor
2022-11-29 13:00:13 +00:00
Antonio Andelic
c8b2cb5650
Update src/Coordination/CoordinationSettings.h
...
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
2022-11-29 10:08:25 +01:00
Antonio Andelic
678958603b
Merge branch 'master' into keeper-prometheus
2022-11-28 09:48:57 +01:00
Antonio Andelic
fe44ac0c86
Merge branch 'master' into parallel-log-appending
2022-11-28 08:45:20 +00:00
Antonio Andelic
e91314d57f
Use initialized flag
2022-11-28 08:44:46 +00:00
Antonio Andelic
cade21b76e
Fix 4LW test
2022-11-26 19:24:10 +00:00
Antonio Andelic
76eca8bdfb
Merge branch 'master' into more-aggresive-batching-keeper
2022-11-26 18:28:11 +00:00
Antonio Andelic
cd471e9c61
Add setting for quick batch size
2022-11-26 17:33:40 +00:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
Antonio Andelic
7437c719a7
Try batching requests more aggresively
2022-11-25 17:25:28 +00:00
Antonio Andelic
38d99c241f
Fix tests
2022-11-25 13:53:03 +00:00
Antonio Andelic
53f708366c
Fix data race in Keeper snapshot
2022-11-25 12:50:42 +00:00
Antonio Andelic
ce532cd12d
Fix circular deps
2022-11-24 13:43:05 +00:00
Antonio Andelic
a6f38cb1cc
Merge branch 'master' into keeper-prometheus
2022-11-24 12:12:08 +01:00
Antonio Andelic
0cc53995df
Merge branch 'master' into parallel-log-appending
2022-11-22 15:05:08 +01:00
Kseniia Sumarokova
5c90d5aa7e
Merge pull request #43253 from xiedeyantu/fix-s3-glob
...
fix s3 support question mark wildcard
2022-11-22 14:26:56 +01:00
Antonio Andelic
4827bcb6bb
Merge branch 'master' into parallel-log-appending
2022-11-22 14:14:17 +01:00
Antonio Andelic
55ef96dc11
Fix unit tests
2022-11-22 11:37:59 +00:00
Antonio Andelic
aeb0d39b50
Fix shutdown
2022-11-21 15:01:18 +00:00
Antonio Andelic
c9472d67b6
Refactor changelog
2022-11-21 12:22:36 +00:00
Antonio Andelic
ff681d5faa
Merge branch 'master' into parallel-log-appending
2022-11-21 11:59:16 +00:00
Antonio Andelic
3cb202a63b
Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader
...
Keeper support manually assigning leader
2022-11-21 09:30:00 +01:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus
2022-11-21 09:10:10 +01:00
xiedeyantu
c258d3ac8b
fix s3 support question mark wildcard
2022-11-18 12:11:22 +08:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
159743edd6
Merge pull request #43014 from ClickHouse/disk-s3-throttler
...
S3 request per second rate throttling
2022-11-16 18:51:06 +01:00
JackyWoo
f8d71f60a2
Merge branch 'master' into keeper_manually_assign_leader
2022-11-16 09:34:40 +08:00
Antonio Andelic
96838f5be7
Merge branch 'master' into keeper-prometheus
2022-11-15 12:00:02 +00:00
Antonio Andelic
d08dce5ec7
Test parallel append
2022-11-14 13:17:05 +00:00
Raúl Marín
54db7c6520
Enforce checking read output
2022-11-11 10:56:18 +01:00
Antonio Andelic
94aab4276c
Bring back heavy metrics
2022-11-10 12:56:27 +01:00
Antonio Andelic
8aeb406d82
Fix standalone build
2022-11-10 10:49:00 +01:00
Antonio Andelic
36d763011b
Compile standalone keeper
2022-11-10 09:23:52 +01:00
Antonio Andelic
1ed3930809
Add more metrics for Keeper
2022-11-09 15:51:41 +01:00
Antonio Andelic
c92ae5a385
Add suport for AsyncMetrics in standalone Keeper
2022-11-09 13:46:03 +01:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
JackyWoo
9060141f16
revert last change
2022-11-08 18:44:43 +08:00
JackyWoo
e865842098
Add rqld to white list
2022-11-08 16:03:13 +08:00
JackyWoo
eeedd7d9ba
Keeper support manually assigning leader
2022-11-08 12:01:53 +08:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Antonio Andelic
9ac829d4c4
Merge pull request #41766 from JackyWoo/manual_snapshot_for_keeper
...
Manually snapshot creation for keeper
2022-11-07 09:18:58 +01:00
JackyWoo
73dfe4acd7
Add to default white list
2022-11-05 17:56:55 +08:00
Antonio Andelic
25b19e6ba0
Merge branch 'master' into manual_snapshot_for_keeper
2022-11-03 08:55:01 +01:00
Antonio Andelic
664690f9a9
Add comments
2022-11-03 07:38:43 +00:00
Antonio Andelic
a6c4204b0c
Don't iterate all cached nodes on commit
2022-11-03 07:36:25 +00:00
JackyWoo
c7a0ebeb05
little fix
2022-10-25 17:46:24 +08:00
JackyWoo
9a36a509fe
fix test
2022-10-25 17:15:49 +08:00
JackyWoo
b5d1c4e657
replace snpd with lgif
2022-10-24 20:08:58 +08:00
JackyWoo
2f30c817bf
little fix
2022-10-24 17:23:47 +08:00
JackyWoo
39c88c74e8
check whether last manually created snapshot is done
2022-10-22 22:31:17 +08:00
JackyWoo
bf291790b7
Merge branch 'master' into manual_snapshot_for_keeper
2022-10-22 16:57:50 +08:00
Azat Khuzhin
905a95e166
Review fixes
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 22:40:13 +02:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Antonio Andelic
540728ab96
Simplify logic a bit
2022-10-18 10:58:25 +00:00
Antonio Andelic
1275c464aa
Fix unit test build
2022-10-18 10:44:22 +00:00
Antonio Andelic
8157fd0595
fix build
2022-10-17 14:16:56 +00:00
Antonio Andelic
238f86d892
Small refactoring
2022-10-17 11:06:33 +00:00
Antonio Andelic
7c7395575f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-10-17 11:02:36 +00:00
Robert Schulze
da5a2e2db0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup
...
Physical merge conflicts:
- src/Common/ZooKeeper/ZooKeeperImpl.cpp
- src/Core/config_core.h.in
- src/Functions/FunctionsAES.h
- src/Functions/config_functions.h.in
- src/configure_config.cmake
Logical merge conflicts:
- Functions/tryDecrypt.cpp
2022-10-06 08:43:25 +00:00
Antonio Andelic
94f1fe39bb
Add support for exists in multiread
2022-09-29 10:39:33 +00:00
Antonio Andelic
3109ce51c6
Merge branch 'master' into keeper-multiread
2022-09-29 06:50:49 +00:00
Antonio Andelic
349bd7fd9a
Dont fail MultiRead on first failed op
2022-09-29 06:50:28 +00:00
Robert Schulze
f24fab7747
Fix some #include atrocities
2022-09-28 13:49:28 +00:00
Robert Schulze
fd86829824
Consolidate config_core.h into config.h
...
Less duplication, less confusion ...
2022-09-28 13:31:57 +00:00
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Antonio Andelic
d0457addbd
Support filtered list
2022-09-27 15:02:26 +00:00
JackyWoo
d1c85f68e8
manually snapshot creation for keeper
2022-09-26 18:29:15 +08:00
Antonio Andelic
937d534cd4
Add support for simple list
2022-09-26 08:05:02 +00:00
Antonio Andelic
f833366555
Merge branch 'master' into keeper-multiread
2022-09-26 07:16:45 +00:00
Antonio Andelic
9da433a904
Fix unused
2022-09-22 14:33:35 +00:00
Antonio Andelic
44a3d6babe
Address PR comments
2022-09-22 13:03:27 +00:00
Antonio Andelic
6798b500e9
Wait on startup for Keeper
2022-09-21 15:12:16 +00:00
Antonio Andelic
eefbbf53e8
Extract S3 logic
2022-09-21 11:53:54 +00:00
Antonio Andelic
fc6c1faa57
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-21 06:57:58 +00:00
Antonio Andelic
1763af37b9
Merge pull request #41403 from ClickHouse/keeper-fix-shutdown-without-server
...
Fix Keeper segfault during shutdown when RAFT server failed to start
2022-09-19 12:03:58 +02:00
alesapin
b6d61318d0
One more validation for broken config in keeper
2022-09-18 00:04:16 +02:00
Alexander Tokmakov
42dab6dc02
Revert "Merge pull request #38200 from ClickHouse/keeper-linearizable-reads"
...
This reverts commit 222b964070
, reversing
changes made to da01982652
.
2022-09-17 19:22:58 +02:00
Antonio Andelic
81f7cf3ca6
Add support for MultiRead in Keeper
2022-09-16 12:00:35 +00:00
Antonio Andelic
acd9cb089e
Fix segfault during shutdown when RAFT server failed to start
2022-09-16 10:03:21 +00:00
Antonio Andelic
5662e0050c
Define client for multi read
2022-09-16 09:55:02 +00:00
Antonio Andelic
296fe01fb8
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-09-16 08:00:15 +00:00
Antonio Andelic
39307f9dba
Extract into function
2022-09-15 16:14:53 +00:00
Antonio Andelic
0dbd0c1f72
Merge branch 'master' into keeper-linearizable-reads
2022-09-15 16:03:18 +00:00
Antonio Andelic
ad7864cb5f
enable S3 if client is built
2022-09-15 14:06:22 +00:00
Antonio Andelic
d3d1676352
Fix config update detection
2022-09-15 13:48:39 +00:00
Antonio Andelic
d4e4ac3801
Lock file during S3 snapshot upload
2022-09-15 09:11:55 +00:00
Antonio Andelic
9fdd2f2e61
Basic implementation for S3 snapshot upload
2022-09-15 09:11:55 +00:00
Antonio Andelic
3a9996f04c
Add bg thread for snapshot upload to S3
2022-09-15 09:11:55 +00:00
Antonio Andelic
36f46891dd
Merge pull request #41215 from ClickHouse/keeper-close-connection-on-shutdown
...
Close sessions on Keeper shutdown
2022-09-13 19:09:34 +02:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage
...
KeeperMap storage engine
2022-09-13 14:15:09 +02:00
alesapin
69d08b1007
Merge pull request #41075 from ClickHouse/keeper-fix-possible-segfault
...
Fix possible segfault during Keeper shutdown
2022-09-13 12:13:24 +02:00
Antonio Andelic
a6b5ffec5d
Polishing
2022-09-13 09:51:46 +00:00
Antonio Andelic
b2cc6a8cc6
Use promise/future
2022-09-12 18:19:41 +00:00
Antonio Andelic
c2dfabe51c
Use shared_ptr for Event
2022-09-12 12:25:39 +00:00
Antonio Andelic
e985ee3354
Close sessions on Keeper shutdown
2022-09-12 12:22:48 +00:00
Antonio Andelic
32483aeec8
Merge branch 'master' into keeper-storage
2022-09-09 12:24:01 +00:00
Antonio Andelic
951ee44229
Process read requests with snapshot install
2022-09-07 11:51:56 +00:00
Antonio Andelic
a377a5f1e9
Change assert to LOGICAL_ERROR
2022-09-07 09:24:50 +00:00
Antonio Andelic
4cdf18cc81
Shutdown storage after RAFT
2022-09-07 09:20:47 +00:00
Antonio Andelic
c6b17bd7f9
Don't preprocess if storage finalized
2022-09-07 09:14:03 +00:00
Antonio Andelic
f0ba3ee927
Fix typo
2022-09-05 09:51:02 +02:00
Antonio Andelic
110ad53300
Merge branch 'master' into keeper-linearizable-reads
2022-09-05 07:12:45 +00:00
Antonio Andelic
b816d46520
Merge branch 'master' into fix-keeper-system-path-check
2022-09-02 13:42:16 +02:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage
2022-09-02 12:13:07 +02:00
Antonio Andelic
e64436fef3
Fix typos with new codespell
2022-09-02 08:54:48 +00:00
Antonio Andelic
ad2196155c
Add test for system node modification
2022-09-02 08:14:06 +00:00
Antonio Andelic
b7eebfc626
Correctly check if the node is using system path
2022-09-02 07:47:12 +00:00
Antonio Andelic
7164e59b54
Merge branch 'master' into keeper-linearizable-reads
2022-09-01 08:52:31 +00:00
Antonio Andelic
fa4c7259f9
Fix typos
2022-09-01 08:46:28 +00:00
Antonio Andelic
c7d1dd942c
Address PR comments
2022-09-01 07:56:06 +00:00
Antonio Andelic
21cdb9a500
Don't remove from cache if it wasn't added during rollback
2022-08-30 08:14:53 +00:00
Antonio Andelic
2416ddfae7
Add extra message about startup preprocessing
2022-08-30 08:14:31 +00:00
Antonio Andelic
57471d11db
Fix uncommitted auth
2022-08-29 12:55:30 +00:00
Antonio Andelic
50af0b1938
Add bunch of comments for request processing
2022-08-29 10:50:18 +00:00
Antonio Andelic
b4af21b6fb
Small fixes
2022-08-29 07:06:51 +00:00
Antonio Andelic
bff2b232ef
Cache deltas for path
2022-08-26 14:03:38 +00:00
Antonio Andelic
26d68dce23
Fix style
2022-08-26 10:55:07 +00:00
Antonio Andelic
dae65178b7
Improve preprocessing performance
2022-08-26 10:46:14 +00:00
Antonio Andelic
5d45f2faa9
Add comments
2022-08-23 08:22:03 +00:00
Antonio Andelic
46882791cd
Fix build
2022-08-23 07:29:47 +00:00
Antonio Andelic
444ac9af6c
Merge branch 'master' into keeper-linearizable-reads
2022-08-23 07:03:35 +00:00
Antonio Andelic
99d23cb67e
Merge branch 'master' into keeper-linearizable-reads
2022-08-23 07:03:20 +00:00
Antonio Andelic
a87a762b0c
Merge pull request #39973 from ClickHouse/keeper-listen-host
...
Support `interserver_listen_host` in Keeper
2022-08-23 09:03:08 +02:00
Alexey Milovidov
74e1f4dc61
Fix clang-tidy
2022-08-20 17:09:20 +02:00
Antonio Andelic
9396834762
Addressed comments
2022-08-17 09:29:08 +00:00
Antonio Andelic
24ee3e022d
Merge branch 'master' into keeper-listen-host
2022-08-09 09:39:27 +02:00
Alexey Milovidov
48bed932ec
Merge pull request #39931 from HarryLeeIBM/hlee-s390x-coordination
...
Fix Endian issue in KeeperSnapshotManager for s390x
2022-08-09 04:24:32 +03:00
HarryLeeIBM
42323cfb92
Use constexpr char array as magic number
2022-08-08 10:59:09 -07:00
Antonio Andelic
6b935a252f
Support multiple listen hosts
2022-08-08 09:08:19 +00:00
Alexey Milovidov
adc880f362
Merge pull request #39867 from ClickHouse/keeper-block-memory-tracker-commit
...
Block memory tracker in Keeper during commit
2022-08-06 02:02:29 +03:00
HarryLeeIBM
cbd01a3a73
Fix Endian issue in KeeperSnapshotManager
2022-08-05 11:07:42 -07:00
Antonio Andelic
07a85513e8
Merge branch 'master' into keeper-create-snapshot-on-exit
2022-08-04 13:12:26 +00:00
Antonio Andelic
10d7259c2b
Add log for snapshot on exit
2022-08-04 13:12:24 +00:00
Antonio Andelic
0a7d2e7b8a
Use LockMemoryExceptionThread
2022-08-04 13:03:05 +00:00
Antonio Andelic
278921be3b
Fix style
2022-08-04 07:25:28 +00:00
Antonio Andelic
fa98338ce1
Add comment for memory tracker blocking
2022-08-04 07:13:42 +00:00
Antonio Andelic
56ca93ca3f
Block memory tracker in Keeper during commit
2022-08-03 14:14:57 +00:00
Antonio Andelic
05467e315f
Merge pull request #39698 from ClickHouse/update-digest-version
...
Update Keeper version for digest
2022-08-03 13:13:00 +02:00
Antonio Andelic
b3b3c371f0
Update KeeperStorage.h
2022-08-02 09:20:02 +02:00
Antonio Andelic
7cdad88327
Create snapshot on shutdown
2022-07-30 11:34:17 +00:00
Antonio Andelic
21becb3030
Update version for digest
2022-07-29 07:36:19 +00:00
Antonio Andelic
e3cb65a99a
Rollback on failed preprocess
2022-07-29 06:51:53 +00:00
Antonio Andelic
c0a5d45258
Rollback on failed PreAppend
2022-07-28 14:06:45 +00:00
Antonio Andelic
1090d6bca7
Rollback request if storing log fails
2022-07-28 12:55:29 +00:00
Antonio Andelic
e62526720f
Address PR comments
2022-07-27 07:51:30 +00:00
Antonio Andelic
04b03b6a90
Don't allow to overwrite on startup
2022-07-25 12:38:48 +00:00
Antonio Andelic
afb6cb6824
Add KeeperContext
2022-07-23 14:45:31 +00:00
Antonio Andelic
6aff87d4b5
Ignore system paths from snapshots and logstore
2022-07-23 13:46:59 +00:00
Antonio Andelic
46a6fbd7aa
Small polish
2022-07-22 11:03:07 +00:00
Antonio Andelic
3040ff0959
Merge branch 'master' into keeper-version-check
2022-07-22 08:07:55 +00:00
Antonio Andelic
75476d5110
Store only root system node
2022-07-22 08:07:38 +00:00
Antonio Andelic
fdd79e62da
Merge pull request #39393 from ClickHouse/keeper-disable-digest-default
...
Disable real-time digest in Keeper by default
2022-07-21 15:15:22 +02:00
Antonio Andelic
7d30ab80c4
Fix compatibility
2022-07-21 09:31:06 +00:00
Alexey Milovidov
19c4c0bd6e
Update CoordinationSettings.cpp
2022-07-21 06:41:42 +03:00
Nikolai Kochetov
91043351aa
Fixing build.
2022-07-20 20:30:16 +00:00
Antonio Andelic
a4450225c7
Disable digest by default
2022-07-19 17:14:44 +00:00
Antonio Andelic
a3e00faaf9
Fix unit tests
2022-07-19 13:14:36 +00:00
Antonio Andelic
ce570b6ee3
Add logs and 4LW for api version
2022-07-19 09:02:57 +00:00
Antonio Andelic
e6ded88ea3
Small refactoring
2022-07-19 08:51:12 +00:00
Antonio Andelic
cfc741030f
Revert to the version with path
2022-07-18 14:30:15 +00:00
Antonio Andelic
48db20d5a1
Add 4LW for api version
2022-07-18 12:19:16 +00:00
Antonio Andelic
b8ffb151dc
Some small polishing
2022-07-18 08:52:52 +00:00
Antonio Andelic
7d7c4ce4cd
Add ApiVersion request
2022-07-18 08:44:38 +00:00
Antonio Andelic
73e0c35ab0
Use 4LW for api version
2022-07-15 10:13:39 +00:00
Antonio Andelic
d81758898a
Remove extra newline
2022-07-13 15:26:25 +00:00
Antonio Andelic
f05f22aa13
Merge branch 'master' into keeper-version-check
2022-07-13 15:25:48 +00:00
Antonio Andelic
3b5bdd1e2a
Add test for current API
2022-07-13 12:41:16 +00:00
Antonio Andelic
c296e84cab
fix unit tests in debug mode
2022-07-13 12:33:18 +00:00
Antonio Andelic
838a11a85a
Rename RaftAppendResult
2022-07-13 11:35:06 +00:00
Antonio Andelic
2634f80956
Remove trailing whitespaces
2022-07-13 11:06:58 +00:00
Antonio Andelic
b49ac48bfd
Merge branch 'master' into keeper-linearizable-reads
2022-07-13 09:30:21 +00:00
Nikita Mikhaylov
1f0280bcde
Fix path retrieval for Keeper's state ( #39148 )
2022-07-13 11:25:34 +02:00
Antonio Andelic
ee4828b084
Fix list watches
2022-07-13 09:23:05 +00:00
Antonio Andelic
8bd86c39f6
Add valid values for read_mode setting
2022-07-12 08:00:31 +00:00
Antonio Andelic
fd1e5d43a0
fix unit tests in debug mode
2022-07-12 07:52:50 +00:00
Antonio Andelic
5f661e3e2b
Merge branch 'master' into keeper-linearizable-reads
2022-07-12 07:33:34 +00:00
Antonio Andelic
0555477ec1
Remove debug logs
2022-07-12 07:05:29 +00:00
Antonio Andelic
6dcdf0fcb1
Throw CORRUPTED_DATA in debug mode for invalid checksum
2022-07-12 06:41:58 +00:00
Antonio Andelic
1f5b3b9922
Add restrictions for modification of internal paths
2022-07-11 13:06:24 +00:00
Antonio Andelic
568ba5d3cc
Address PR comments
2022-07-11 12:56:09 +00:00
Antonio Andelic
efc7f5c534
Merge branch 'master' into keeper-persist-state
2022-07-11 12:43:06 +00:00
Antonio Andelic
41dd4e13f4
Merge branch 'master' into keeper-version-check
2022-07-11 09:38:20 +00:00