thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
138,785 Commits 9,328 Branches 1,783 Tags 2.1 GiB
66cd879d2d
Commit Graph

1283 Commits

Author SHA1 Message Date
Antonio Andelic
47966c4e81 Finalize bufferg 2023-06-02 14:10:52 +00:00
Antonio Andelic
336c9d7136 Apply uncommitted state after snapshot deser 2023-06-02 13:31:14 +00:00
Antonio Andelic
ffd4f7f196 Add better support for state disks 2023-06-01 14:39:01 +00:00
Antonio Andelic
e8442b7a88 Add finalize calls 2023-06-01 12:54:00 +00:00
frinkr
c9d0d217f5 fix Keeper deadlock on exception when preprocessing requests 2023-06-01 09:57:42 +08:00
Konstantin Bogdanov
b61ffe3ff7
Merge checks 2023-06-01 03:22:40 +02:00
Konstantin Bogdanov
3a3539b996
Style fix 2023-06-01 03:22:40 +02:00
Konstantin Bogdanov
203276dfcc
Use storage.uncommited_state 2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
965f7850f8
Cleanup 2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
f84fdb7f10
Increment Keeper API version 2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
b53f36369e
Remove new request object 2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
0d6561ee77
Prototype 2023-06-01 03:22:23 +02:00
Antonio Andelic
21aba94909 Add support for latest snapshot disk 2023-05-30 13:22:40 +00:00
Antonio Andelic
67746bfff2 Merge branch 'master' into keeper-with-disks 2023-05-30 09:50:12 +00:00
Raúl Marín
1ae1f1273a Avoid clang 15 crash 2023-05-29 11:40:45 +02:00
Antonio Andelic
0740bfbe4b Better disks moving 2023-05-26 14:31:09 +00:00
Antonio Andelic
a9569478e6
fix typo 2023-05-26 13:09:09 +02:00
Antonio Andelic
8a2a63a7bd Merge branch 'master' into keeper-with-disks 2023-05-26 10:48:41 +00:00
Antonio Andelic
3d98e591ba Handle backwards compatibility 2023-05-26 10:46:18 +00:00
Antonio Andelic
767193caf5 Merge branch 'master' into keeper-log-improvements 2023-05-26 07:35:36 +00:00
Antonio Andelic
6a8a21e09a Add test for s3 disk restore 2023-05-25 16:01:40 +00:00
Antonio Andelic
bb77441acb Add support for old storage disks 2023-05-25 13:31:11 +00:00
Antonio Andelic
f519aa4613 Calculate size 2023-05-25 11:23:03 +00:00
Antonio Andelic
4483602c50 Remove double initialization 2023-05-25 08:52:14 +00:00
Antonio Andelic
7e68f61df1 Fix snapshot recovery 2023-05-25 08:43:11 +00:00
Antonio Andelic
092cf99147 Fix build 2023-05-24 13:45:14 +00:00
Antonio Andelic
fa3fb49d33 Merge branch 'master' into keeper-with-disks 2023-05-24 09:04:47 +00:00
Antonio Andelic
5db2160762 Fix build 2023-05-24 09:04:12 +00:00
Antonio Andelic
161afea266 Add support for changelog 2023-05-24 07:43:57 +00:00
kssenii
295fe3b228 Merge remote-tracking branch 'upstream/master' into abstract-async-prefetched-buffer 2023-05-22 20:01:38 +02:00
kssenii
c4d862a16f Make async reader work with any impl 2023-05-22 19:54:04 +02:00
Antonio Andelic
bde2cf96b1 Better 2023-05-22 12:24:16 +00:00
alesapin
cc3897a84a
Merge pull request #50033 from kssenii/disk-object-storage-minor-changes 
Get rid of indirect write buffer in object storages
 2023-05-22 14:08:17 +02:00
Antonio Andelic
dc31396035 Merge branch 'master' into keeper-with-disks 2023-05-22 08:10:24 +00:00
kssenii
3e42ee7f2b Get rid of finalize callback in object storages 2023-05-19 17:29:37 +02:00
Antonio Andelic
9c3b17fa18
Remove whitespace 2023-05-19 13:00:51 +02:00
Antonio Andelic
e46476dba2
Update src/Coordination/Changelog.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2023-05-19 12:44:20 +02:00
Antonio Andelic
6e468b29e8 Check return value of ftruncate 2023-05-19 10:15:06 +00:00
Antonio Andelic
bf6afd27a6 build with disks 2023-05-18 07:39:22 +00:00
Antonio Andelic
465123fa1b Merge branch 'master' into keeper-with-disks 2023-05-16 08:07:06 +00:00
Azat Khuzhin
665545ec45 Fix "reference to local binding" after fixes for clang-17 
Follow-up for: #49851 (cc @alexey-milovidov)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-15 12:45:20 +02:00
Alexey Milovidov
5a44dc26e7 Fixes for clang-17 2023-05-13 02:57:31 +02:00
Antonio Andelic
bb8b9dc9c3 Merge branch 'master' into keeper-log-improvements 2023-05-05 11:57:14 +00:00
alesapin
9456eb59fd
Merge pull request #48466 from ClickHouse/keeper-correctly-handle-snapshots 
Correctly handle concurrent snapshots in Keeper
 2023-05-04 19:16:00 +02:00
johanngan
731823b873 Add support for SSE-KMS configuration with S3 
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html

Similar to the server_side_encryption_customer_key_base64 option for
configuring SSE-C with S3, add the following settings to configure
SSE-KMS on a per-endpoint/disk basis:
  - server_side_encryption_kms_key_id
  - server_side_encryption_kms_encryption_context
  - server_side_encryption_kms_bucket_key_enabled
 2023-05-03 21:35:38 -05:00
Antonio Andelic
b8b7a362b2
Merge branch 'master' into keeper-correctly-handle-snapshots 2023-05-03 13:51:09 +02:00
Antonio Andelic
78ecfac8d5 Merge branch 'master' into keeper-log-improvements 2023-05-03 08:31:21 +00:00
Antonio Andelic
dce61d7321 Merge branch 'master' into keeper-log-improvements 2023-05-02 11:14:48 +00:00
Antonio Andelic
f695040a19
Merge pull request #48897 from ClickHouse/keeper-check-not-exists 
Add `CheckNotExists` request to Keeper
 2023-05-02 13:14:22 +02:00
Raúl Marín
f0e045bb3d Merge remote-tracking branch 'blessed/master' into arenita 2023-04-24 10:42:56 +02:00
Alexey Milovidov
530b764953 Fix IBM 2023-04-21 12:38:45 +02:00
Antonio Andelic
d500a5637d Cache large requests 2023-04-21 09:43:04 +00:00
Antonio Andelic
b52646c555 Eliminate double log serializing 2023-04-21 08:20:35 +00:00
Antonio Andelic
58e9b56fcb Fix CheckNotExists 2023-04-19 09:06:20 +00:00
Antonio Andelic
fb16623d48 Add CheckNotExists request to Keeper 2023-04-18 14:45:24 +00:00
Raúl Marín
2b70e08f23 Don't count unreserved bytes in Arenas as read_bytes 2023-04-13 12:43:24 +02:00
Antonio Andelic
b30aff06d8
Merge pull request #48595 from aalexfvk/alexfvk/fix_create_not_auth_problem 
Fix bug in Keeper when a node is not created with scheme `auth` in ACL sometimes.
 2023-04-11 14:41:37 +02:00
Aleksei Filatov
dd2364361c
Update src/Coordination/tests/gtest_coordination.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-04-11 10:45:30 +03:00
Azat Khuzhin
79b83c4fd2 Remove superfluous includes of logger_userful.h from headers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-10 17:59:30 +02:00
Aleksei Filatov
e8bf96d126 Add unit tests 2023-04-10 13:02:43 +03:00
Aleksei Filatov
000c2b5b91 Correct FixupACL for auth scheme 2023-04-10 13:02:31 +03:00
Azat Khuzhin
011480924a Use forward declaration of ThreadPool 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 11:25:35 +02:00
Antonio Andelic
88f77c686d Correctly handle concurrent snapshots 2023-04-06 15:40:05 +00:00
Antonio Andelic
e982f2a67a Merge branch 'master' into enable-env-credentials-default 2023-03-31 09:11:01 +00:00
Antonio Andelic
44e95aa65f Merge branch 'master' into nosign-keyword-for-s3 2023-03-29 11:10:03 +00:00
Antonio Andelic
160aa186bb Add support for NOSIGN keyword and no_sign_request config 2023-03-28 07:05:35 +00:00
Antonio Andelic
79e22d13fb small fix 2023-03-27 07:41:31 +00:00
Antonio Andelic
f355bc5342 Merge branch 'master' into more-batching-keeper 2023-03-27 07:28:17 +00:00
Antonio Andelic
6c3765c8b6 Small fix 2023-03-24 20:40:45 +00:00
Antonio Andelic
f353561204 Avoid breaking batches with read requests 2023-03-24 14:37:40 +00:00
Alexander Gololobov
afe3af230f Limit keeper request batching by size in bytes 2023-03-24 14:41:09 +01:00
Antonio Andelic
51416da818 Merge branch 'master' into enable-env-credentials-default 2023-03-13 10:20:21 +00:00
Antonio Andelic
a170a909a4 Add expiration window for S3 credentials 2023-03-10 10:06:32 +00:00
Antonio Andelic
83a3243758
Merge pull request #47065 from ClickHouse/debug-keeper 
Fix Keeper with `force_sync = false`
 2023-03-10 09:27:13 +01:00
Antonio Andelic
5bc21538e5 Enable use_environment_credentials by default 2023-03-09 10:31:55 +00:00
Antonio Andelic
0ad436aa2c Revert debug changes 2023-03-09 08:07:59 +00:00
Antonio Andelic
b4061d8aa4 Add logs 2023-03-08 13:06:10 +00:00
Mike Kot
9920a52c51 use std::lerp, constexpr hex.h 2023-03-07 22:50:17 +00:00
Antonio Andelic
dd4db9132a
Merge branch 'master' into debug-keeper 2023-03-07 13:15:05 +01:00
Antonio Andelic
c38d820d0f Add 4LW for cleaning resource 2023-03-05 16:57:18 +00:00
Antonio Andelic
cd4100cb19
Merge branch 'master' into fix-data-size 2023-03-03 15:33:43 +01:00
Antonio Andelic
fdcb5b784f
Merge branch 'master' into debug-keeper 2023-03-03 13:45:13 +01:00
Robert Schulze
740aeaba1f
Apply some CTAD 2023-03-02 13:36:47 +00:00
Antonio Andelic
6ebee202bd Flush buffer when no sync 2023-03-01 14:47:15 +00:00
Antonio Andelic
c797122dbe Better 2023-03-01 08:45:50 +00:00
Antonio Andelic
6e28406751 Merge branch 'master' into fix-data-size 2023-02-28 20:09:29 +00:00
Alexey Milovidov
d8cda3dbb8 Remove PVS-Studio 2023-02-19 23:30:05 +01:00
Alexander Tokmakov
e96340e183
Merge pull request #46410 from ClickHouse/fix_noisy_log_messages 
Fix noisy log messages
 2023-02-15 01:22:42 +03:00
Alexander Tokmakov
1bf3735ca9 fix noisy log messages 2023-02-14 15:33:27 +01:00
Sema Checherinda
9bef1bec28
Merge pull request #46274 from aiven-sal/aiven-sal/keeper 
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
 2023-02-14 12:44:04 +01:00
Sema Checherinda
5882f3878f
except zero values election_timeout 2023-02-13 12:22:08 +01:00
Salvatore Mesoraca
fff2c09b87
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms 2023-02-13 10:09:45 +01:00
Antonio Andelic
f96d480563
Merge branch 'master' into add-support-for-settings-alias 2023-02-09 16:07:45 +01:00
Azat Khuzhin
8cc41b7f41 Check return value of ::close() 
Note, that according close(2), EINTR should not be retriable for close()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-07 11:28:22 +01:00
Antonio Andelic
b11458f142 Fix erase 2023-02-07 07:51:22 +00:00
Antonio Andelic
9cde46acff Correctly calculate size on update 2023-02-06 16:28:05 +00:00
Antonio Andelic
aca4f08bf5 Correctly calculate size after snapshot load 2023-02-06 14:59:23 +00:00
Antonio Andelic
8929838fcc Merge branch 'master' into add-support-for-settings-alias 2023-02-03 17:47:00 +00:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution (#45783) 
* Update aws

* Define S3 client with bucket and endpoint resolution

* Add defines for ErrorCodes

* Use S3Client everywhere

* Remove unused errorcode

* Add DROP S3 CLIENT CACHE query

* Add a comment

* Fix style

* Update aws

* Update reference files

* Add missing include

* Fix unit test

* Remove unneeded declarations

* Correctly use RetryStrategy

* Rename S3Client to Client

* Fix retry count

* fix clang-tidy warnings
 2023-02-03 14:30:52 +01:00
LiuYangkuan
f29700bd2f use IDisk to do IO in Keeper's snapshots and logs 2023-02-02 19:47:30 +08:00
Antonio Andelic
f613bfd8d2 Merge branch 'master' into add-support-for-settings-alias 2023-02-02 11:15:39 +00:00
Antonio Andelic
85cfee4bb9 Better alias definition 2023-02-01 13:54:03 +00:00
Antonio Andelic
88c1744975
Better 2023-02-01 14:25:58 +01:00
Antonio Andelic
5de6f86c27 Fix clang-tidy error 2023-02-01 12:28:44 +00:00
Antonio Andelic
ec04cf6876 Merge branch 'master' into add-support-for-settings-alias 2023-02-01 09:01:01 +00:00
Antonio Andelic
ece92fa866
Update src/Coordination/Changelog.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2023-01-31 17:11:18 +01:00
Antonio Andelic
8457d3bf3d Apply comments 2023-01-31 14:18:54 +00:00
Antonio Andelic
1786237be6 Merge branch 'master' into keeper-use-fallocate 2023-01-31 13:53:47 +00:00
Vitaly Baranov
f9282cfcac Make version_id an optional parameter again. 2023-01-27 15:10:19 +01:00
Vitaly Baranov
aea9ccdb60 Pass request settings to S3::getObjectInfo(). 2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed Move getObjectInfo() to a separate header. 2023-01-27 15:09:38 +01:00
Antonio Andelic
714fad1529 Add support for settings alias 2023-01-26 14:06:46 +00:00
Antonio Andelic
ebfe7f9991 Fix style 2023-01-26 11:18:07 +00:00
Antonio Andelic
1186718658 Add overallocate size setting 2023-01-26 09:44:37 +00:00
Antonio Andelic
7bfdfa4bf7 Preallocate more space 2023-01-25 16:26:14 +00:00
Antonio Andelic
7421d8f872 Merge branch 'master' into keeper-use-fallocate 2023-01-25 11:36:47 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Alexander Tokmakov
870cfcc36a less fmt::runtime usages 2023-01-17 00:11:59 +01:00
Antonio Andelic
6a0e481ed7 Merge branch 'master' into keeper-use-fallocate 2023-01-12 13:57:05 +00:00
Antonio Andelic
30510c5eb6
Merge pull request #44937 from ClickHouse/fix-changelog-deadlock 
Fix deadlock in Keeper's changelog
 2023-01-10 08:59:51 +01:00
Antonio Andelic
22679ddf11
Add comment 2023-01-09 14:26:20 +01:00
Antonio Andelic
a39f437306
Typo fix 2023-01-09 13:34:48 +01:00
Antonio Andelic
27b2bae640 make it better 2023-01-09 12:01:22 +00:00
Antonio Andelic
4fcbe14435 Small fixes for Coordination unit tests 2023-01-08 13:44:32 +00:00
Kseniia Sumarokova
573d3283b0
Merge pull request #44327 from kssenii/use-new-named-collections-code-2 
Replace old named collections code with new (from #43147) part 2
 2023-01-06 13:06:26 +01:00
Antonio Andelic
9b3314a931 Fix deadlock in Changelog flush 2023-01-05 14:51:05 +00:00
Antonio Andelic
136436958b
Merge pull request #44908 from ClickHouse/fix-keeper-snapshot-on-exit 
Avoid Keeper crash on shutdown (fix `test_keeper_snapshot_on_exit`)
 2023-01-05 08:59:58 +01:00
Antonio Andelic
7c2cdfa65a Don't throw exception if queue is finished 2023-01-04 15:46:38 +00:00
kssenii
67509aa2d5 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2023-01-03 16:41:30 +01:00
Smita Kulkarni
8a5fd7f96d Added comments and addressed review comments - Allow macro substitution in endpoint 2023-01-03 11:44:26 +01:00
Antonio Andelic
f2653c016a Manually set deleted flag 2023-01-02 14:36:58 +00:00
Antonio Andelic
f32fab5025 Check if file exists before rename 2023-01-02 13:48:13 +00:00
Antonio Andelic
2e93640614 Fix ASAN error 2023-01-02 13:37:40 +00:00
Antonio Andelic
b845f880fb Polish 2023-01-02 10:42:15 +00:00
Antonio Andelic
14355d8009 Merge branch 'master' into keeper-use-fallocate 2023-01-02 08:22:27 +00:00
Antonio Andelic
55548a6678 style fix 2023-01-02 08:22:00 +00:00
SmitaRKulkarni
f9a3aeb172
Merge branch 'master' into 40951_Allow_macro_substitution_in_endpoint 2022-12-30 10:43:14 +01:00
Alexey Milovidov
f0fa0977ed Fix wrong log messages 2022-12-26 18:36:07 +01:00
Smita Kulkarni
3d62bc2635 Allow macro substitution in endpoint of disks 
Implementation:
* Updated to substitute macros in endpoint for all disks (s3, hfs, web).
Testing:
* Added an integration test test/integration/test_endpoint_macro_substitution.
 2022-12-23 15:09:24 +01:00
Antonio Andelic
64ade259cf Fix tests 2022-12-22 10:20:55 +00:00
Antonio Andelic
88d41dea93 Merge branch 'master' into keeper-use-fallocate 2022-12-21 08:59:09 +00:00
Antonio Andelic
9711355cc9 Fix tests 2022-12-21 08:43:55 +00:00
kssenii
6bd4f8c029 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2022-12-20 21:17:28 +01:00
Antonio Andelic
6d83221d4e Remove unused error code 2022-12-19 12:20:26 +00:00
Antonio Andelic
c2c115d2ba Add setting for max_log_file_size 2022-12-19 11:51:05 +00:00
Antonio Andelic
f241701c2a Add checks when fallocate is not supported 2022-12-19 11:48:54 +00:00
Antonio Andelic
9a8b248735 Merge branch 'master' into keeper-use-fallocate 2022-12-19 09:32:05 +00:00
Nikolai Kochetov
62ff98344e Validate s3 part upload settings. 2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd Replace old named collections code for url 2022-12-17 00:24:05 +01:00
Antonio Andelic
ee900ca7df Add truncate 2022-12-15 13:09:52 +00:00
Antonio Andelic
bf6f163952 Merge branch 'master' into keeper-use-fallocate 2022-12-15 09:05:40 +00:00
Antonio Andelic
d454e00016 Changelog refactor 2022-12-15 08:12:02 +00:00
Anton Popov
0c87031e80 Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-13 16:33:21 +00:00
Antonio Andelic
662c1f956c Changelog refactoring 2022-12-13 14:00:27 +00:00
Antonio Andelic
898cf867af Try with fallocate 2022-12-13 11:08:59 +00:00
Vitaly Baranov
b91af1b650 Fix initialization of s3 request settings. 2022-12-10 05:43:51 +01:00
Antonio Andelic
e1a6a1182e
Throw when port already in use (#43984) 2022-12-07 13:27:49 +01:00
Antonio Andelic
200141da13 Use logger in tests 2022-12-02 08:16:18 +00:00
Antonio Andelic
7cd76df918 Merge branch 'master' into parallel-log-appending 2022-12-01 14:44:36 +00:00
Antonio Andelic
554ff1db6f Small changes 2022-12-01 12:54:29 +00:00
Antonio Andelic
940fdd4abd
Remove comment 2022-12-01 12:24:27 +01:00
Antonio Andelic
25ae52111e Protect writer with mutex 2022-12-01 11:18:23 +00:00
Antonio Andelic
c12c0cc8af Merge branch 'master' into parallel-log-appending 2022-12-01 08:53:52 +00:00
Antonio Andelic
2f501d5327 Merge branch 'master' into keeper-prometheus 2022-11-30 13:24:37 +01:00
Antonio Andelic
fbda86ddc8 PR fixes 2022-11-30 13:24:08 +01:00
Antonio Andelic
7f897f1a20 Refactor 2022-11-29 13:00:13 +00:00
Antonio Andelic
c8b2cb5650
Update src/Coordination/CoordinationSettings.h 
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
 2022-11-29 10:08:25 +01:00
Antonio Andelic
678958603b
Merge branch 'master' into keeper-prometheus 2022-11-28 09:48:57 +01:00
Antonio Andelic
fe44ac0c86 Merge branch 'master' into parallel-log-appending 2022-11-28 08:45:20 +00:00
Antonio Andelic
e91314d57f Use initialized flag 2022-11-28 08:44:46 +00:00
Antonio Andelic
cade21b76e Fix 4LW test 2022-11-26 19:24:10 +00:00
Antonio Andelic
76eca8bdfb Merge branch 'master' into more-aggresive-batching-keeper 2022-11-26 18:28:11 +00:00
Antonio Andelic
cd471e9c61 Add setting for quick batch size 2022-11-26 17:33:40 +00:00
Anton Popov
65a78bcd91 improve performance of storage S3 2022-11-26 15:24:01 +00:00
Antonio Andelic
7437c719a7 Try batching requests more aggresively 2022-11-25 17:25:28 +00:00
Antonio Andelic
38d99c241f Fix tests 2022-11-25 13:53:03 +00:00
Antonio Andelic
53f708366c Fix data race in Keeper snapshot 2022-11-25 12:50:42 +00:00
Antonio Andelic
ce532cd12d Fix circular deps 2022-11-24 13:43:05 +00:00
Antonio Andelic
a6f38cb1cc
Merge branch 'master' into keeper-prometheus 2022-11-24 12:12:08 +01:00
Antonio Andelic
0cc53995df
Merge branch 'master' into parallel-log-appending 2022-11-22 15:05:08 +01:00
Kseniia Sumarokova
5c90d5aa7e
Merge pull request #43253 from xiedeyantu/fix-s3-glob 
fix s3 support question mark wildcard
 2022-11-22 14:26:56 +01:00
Antonio Andelic
4827bcb6bb
Merge branch 'master' into parallel-log-appending 2022-11-22 14:14:17 +01:00
Antonio Andelic
55ef96dc11 Fix unit tests 2022-11-22 11:37:59 +00:00
Antonio Andelic
aeb0d39b50 Fix shutdown 2022-11-21 15:01:18 +00:00
Antonio Andelic
c9472d67b6 Refactor changelog 2022-11-21 12:22:36 +00:00
Antonio Andelic
ff681d5faa Merge branch 'master' into parallel-log-appending 2022-11-21 11:59:16 +00:00
Antonio Andelic
3cb202a63b
Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader 
Keeper support manually assigning leader
 2022-11-21 09:30:00 +01:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus 2022-11-21 09:10:10 +01:00
xiedeyantu
c258d3ac8b fix s3 support question mark wildcard 2022-11-18 12:11:22 +08:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling"" 2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling" 2022-11-16 22:33:48 +03:00
Sergei Trifonov
159743edd6
Merge pull request #43014 from ClickHouse/disk-s3-throttler 
S3 request per second rate throttling
 2022-11-16 18:51:06 +01:00
JackyWoo
f8d71f60a2
Merge branch 'master' into keeper_manually_assign_leader 2022-11-16 09:34:40 +08:00
Antonio Andelic
96838f5be7 Merge branch 'master' into keeper-prometheus 2022-11-15 12:00:02 +00:00
Antonio Andelic
d08dce5ec7 Test parallel append 2022-11-14 13:17:05 +00:00
Raúl Marín
54db7c6520 Enforce checking read output 2022-11-11 10:56:18 +01:00
Antonio Andelic
94aab4276c Bring back heavy metrics 2022-11-10 12:56:27 +01:00
Antonio Andelic
8aeb406d82 Fix standalone build 2022-11-10 10:49:00 +01:00
Antonio Andelic
36d763011b Compile standalone keeper 2022-11-10 09:23:52 +01:00
Antonio Andelic
1ed3930809 Add more metrics for Keeper 2022-11-09 15:51:41 +01:00
Antonio Andelic
c92ae5a385 Add suport for AsyncMetrics in standalone Keeper 2022-11-09 13:46:03 +01:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler 2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421 rename ReadWriteSettings -> RequestSettings 2022-11-08 13:48:23 +00:00
JackyWoo
9060141f16 revert last change 2022-11-08 18:44:43 +08:00
JackyWoo
e865842098 Add rqld to white list 2022-11-08 16:03:13 +08:00
JackyWoo
eeedd7d9ba Keeper support manually assigning leader 2022-11-08 12:01:53 +08:00
serxa
2daec0b45e S3 request per second rate throttling + refactoring 2022-11-07 18:05:40 +00:00
Antonio Andelic
9ac829d4c4
Merge pull request #41766 from JackyWoo/manual_snapshot_for_keeper 
Manually snapshot creation for keeper
 2022-11-07 09:18:58 +01:00
JackyWoo
73dfe4acd7 Add to default white list 2022-11-05 17:56:55 +08:00
Antonio Andelic
25b19e6ba0
Merge branch 'master' into manual_snapshot_for_keeper 2022-11-03 08:55:01 +01:00
Antonio Andelic
664690f9a9 Add comments 2022-11-03 07:38:43 +00:00
Antonio Andelic
a6c4204b0c Don't iterate all cached nodes on commit 2022-11-03 07:36:25 +00:00
JackyWoo
c7a0ebeb05 little fix 2022-10-25 17:46:24 +08:00
JackyWoo
9a36a509fe fix test 2022-10-25 17:15:49 +08:00
JackyWoo
b5d1c4e657 replace snpd with lgif 2022-10-24 20:08:58 +08:00
JackyWoo
2f30c817bf little fix 2022-10-24 17:23:47 +08:00
JackyWoo
39c88c74e8 check whether last manually created snapshot is done 2022-10-22 22:31:17 +08:00
JackyWoo
bf291790b7
Merge branch 'master' into manual_snapshot_for_keeper 2022-10-22 16:57:50 +08:00
Azat Khuzhin
905a95e166 Review fixes 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 22:40:13 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Antonio Andelic
540728ab96 Simplify logic a bit 2022-10-18 10:58:25 +00:00
Antonio Andelic
1275c464aa Fix unit test build 2022-10-18 10:44:22 +00:00
Antonio Andelic
8157fd0595 fix build 2022-10-17 14:16:56 +00:00
Antonio Andelic
238f86d892 Small refactoring 2022-10-17 11:06:33 +00:00
Antonio Andelic
7c7395575f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-17 11:02:36 +00:00
Robert Schulze
da5a2e2db0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup 
Physical merge conflicts:
- src/Common/ZooKeeper/ZooKeeperImpl.cpp
- src/Core/config_core.h.in
- src/Functions/FunctionsAES.h
- src/Functions/config_functions.h.in
- src/configure_config.cmake

Logical merge conflicts:
- Functions/tryDecrypt.cpp
 2022-10-06 08:43:25 +00:00
Antonio Andelic
94f1fe39bb Add support for exists in multiread 2022-09-29 10:39:33 +00:00
Antonio Andelic
3109ce51c6 Merge branch 'master' into keeper-multiread 2022-09-29 06:50:49 +00:00
Antonio Andelic
349bd7fd9a Dont fail MultiRead on first failed op 2022-09-29 06:50:28 +00:00
Robert Schulze
f24fab7747
Fix some #include atrocities 2022-09-28 13:49:28 +00:00
Robert Schulze
fd86829824
Consolidate config_core.h into config.h 
Less duplication, less confusion ...
 2022-09-28 13:31:57 +00:00
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Antonio Andelic
d0457addbd Support filtered list 2022-09-27 15:02:26 +00:00
JackyWoo
d1c85f68e8 manually snapshot creation for keeper 2022-09-26 18:29:15 +08:00
Antonio Andelic
937d534cd4 Add support for simple list 2022-09-26 08:05:02 +00:00
Antonio Andelic
f833366555 Merge branch 'master' into keeper-multiread 2022-09-26 07:16:45 +00:00
Antonio Andelic
9da433a904 Fix unused 2022-09-22 14:33:35 +00:00
Antonio Andelic
44a3d6babe Address PR comments 2022-09-22 13:03:27 +00:00
Antonio Andelic
6798b500e9 Wait on startup for Keeper 2022-09-21 15:12:16 +00:00
Antonio Andelic
eefbbf53e8 Extract S3 logic 2022-09-21 11:53:54 +00:00
Antonio Andelic
fc6c1faa57 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-21 06:57:58 +00:00
Antonio Andelic
1763af37b9
Merge pull request #41403 from ClickHouse/keeper-fix-shutdown-without-server 
Fix Keeper segfault during shutdown when RAFT server failed to start
 2022-09-19 12:03:58 +02:00
alesapin
b6d61318d0 One more validation for broken config in keeper 2022-09-18 00:04:16 +02:00
Alexander Tokmakov
42dab6dc02 Revert "Merge pull request #38200 from ClickHouse/keeper-linearizable-reads" 
This reverts commit 222b964070, reversing
changes made to da01982652.
 2022-09-17 19:22:58 +02:00
Antonio Andelic
81f7cf3ca6 Add support for MultiRead in Keeper 2022-09-16 12:00:35 +00:00
Antonio Andelic
acd9cb089e Fix segfault during shutdown when RAFT server failed to start 2022-09-16 10:03:21 +00:00
Antonio Andelic
5662e0050c Define client for multi read 2022-09-16 09:55:02 +00:00
Antonio Andelic
296fe01fb8 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-16 08:00:15 +00:00
Antonio Andelic
39307f9dba Extract into function 2022-09-15 16:14:53 +00:00
Antonio Andelic
0dbd0c1f72 Merge branch 'master' into keeper-linearizable-reads 2022-09-15 16:03:18 +00:00
Antonio Andelic
ad7864cb5f enable S3 if client is built 2022-09-15 14:06:22 +00:00
Antonio Andelic
d3d1676352 Fix config update detection 2022-09-15 13:48:39 +00:00
Antonio Andelic
d4e4ac3801 Lock file during S3 snapshot upload 2022-09-15 09:11:55 +00:00
Antonio Andelic
9fdd2f2e61 Basic implementation for S3 snapshot upload 2022-09-15 09:11:55 +00:00
Antonio Andelic
3a9996f04c Add bg thread for snapshot upload to S3 2022-09-15 09:11:55 +00:00
Antonio Andelic
36f46891dd
Merge pull request #41215 from ClickHouse/keeper-close-connection-on-shutdown 
Close sessions on Keeper shutdown
 2022-09-13 19:09:34 +02:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage 
KeeperMap storage engine
 2022-09-13 14:15:09 +02:00
alesapin
69d08b1007
Merge pull request #41075 from ClickHouse/keeper-fix-possible-segfault 
Fix possible segfault during Keeper shutdown
 2022-09-13 12:13:24 +02:00
Antonio Andelic
a6b5ffec5d Polishing 2022-09-13 09:51:46 +00:00
Antonio Andelic
b2cc6a8cc6 Use promise/future 2022-09-12 18:19:41 +00:00
Antonio Andelic
c2dfabe51c Use shared_ptr for Event 2022-09-12 12:25:39 +00:00
Antonio Andelic
e985ee3354 Close sessions on Keeper shutdown 2022-09-12 12:22:48 +00:00
Antonio Andelic
32483aeec8 Merge branch 'master' into keeper-storage 2022-09-09 12:24:01 +00:00
Antonio Andelic
951ee44229 Process read requests with snapshot install 2022-09-07 11:51:56 +00:00
Antonio Andelic
a377a5f1e9 Change assert to LOGICAL_ERROR 2022-09-07 09:24:50 +00:00
Antonio Andelic
4cdf18cc81 Shutdown storage after RAFT 2022-09-07 09:20:47 +00:00
Antonio Andelic
c6b17bd7f9 Don't preprocess if storage finalized 2022-09-07 09:14:03 +00:00
Antonio Andelic
f0ba3ee927
Fix typo 2022-09-05 09:51:02 +02:00
Antonio Andelic
110ad53300 Merge branch 'master' into keeper-linearizable-reads 2022-09-05 07:12:45 +00:00
Antonio Andelic
b816d46520
Merge branch 'master' into fix-keeper-system-path-check 2022-09-02 13:42:16 +02:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage 2022-09-02 12:13:07 +02:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Antonio Andelic
ad2196155c Add test for system node modification 2022-09-02 08:14:06 +00:00
Antonio Andelic
b7eebfc626 Correctly check if the node is using system path 2022-09-02 07:47:12 +00:00
Antonio Andelic
7164e59b54 Merge branch 'master' into keeper-linearizable-reads 2022-09-01 08:52:31 +00:00
Antonio Andelic
fa4c7259f9 Fix typos 2022-09-01 08:46:28 +00:00
Antonio Andelic
c7d1dd942c Address PR comments 2022-09-01 07:56:06 +00:00
Antonio Andelic
21cdb9a500 Don't remove from cache if it wasn't added during rollback 2022-08-30 08:14:53 +00:00
Antonio Andelic
2416ddfae7 Add extra message about startup preprocessing 2022-08-30 08:14:31 +00:00
Antonio Andelic
57471d11db Fix uncommitted auth 2022-08-29 12:55:30 +00:00
Antonio Andelic
50af0b1938 Add bunch of comments for request processing 2022-08-29 10:50:18 +00:00
Antonio Andelic
b4af21b6fb Small fixes 2022-08-29 07:06:51 +00:00
Antonio Andelic
bff2b232ef Cache deltas for path 2022-08-26 14:03:38 +00:00
Antonio Andelic
26d68dce23 Fix style 2022-08-26 10:55:07 +00:00
Antonio Andelic
dae65178b7 Improve preprocessing performance 2022-08-26 10:46:14 +00:00
Antonio Andelic
5d45f2faa9 Add comments 2022-08-23 08:22:03 +00:00
Antonio Andelic
46882791cd Fix build 2022-08-23 07:29:47 +00:00
Antonio Andelic
444ac9af6c Merge branch 'master' into keeper-linearizable-reads 2022-08-23 07:03:35 +00:00
Antonio Andelic
99d23cb67e Merge branch 'master' into keeper-linearizable-reads 2022-08-23 07:03:20 +00:00
Antonio Andelic
a87a762b0c
Merge pull request #39973 from ClickHouse/keeper-listen-host 
Support `interserver_listen_host` in Keeper
 2022-08-23 09:03:08 +02:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
Antonio Andelic
9396834762 Addressed comments 2022-08-17 09:29:08 +00:00
Antonio Andelic
24ee3e022d
Merge branch 'master' into keeper-listen-host 2022-08-09 09:39:27 +02:00
Alexey Milovidov
48bed932ec
Merge pull request #39931 from HarryLeeIBM/hlee-s390x-coordination 
Fix Endian issue in KeeperSnapshotManager for s390x
 2022-08-09 04:24:32 +03:00
HarryLeeIBM
42323cfb92 Use constexpr char array as magic number 2022-08-08 10:59:09 -07:00
Antonio Andelic
6b935a252f Support multiple listen hosts 2022-08-08 09:08:19 +00:00
Alexey Milovidov
adc880f362
Merge pull request #39867 from ClickHouse/keeper-block-memory-tracker-commit 
Block memory tracker in Keeper during commit
 2022-08-06 02:02:29 +03:00
HarryLeeIBM
cbd01a3a73 Fix Endian issue in KeeperSnapshotManager 2022-08-05 11:07:42 -07:00
Antonio Andelic
07a85513e8 Merge branch 'master' into keeper-create-snapshot-on-exit 2022-08-04 13:12:26 +00:00
Antonio Andelic
10d7259c2b Add log for snapshot on exit 2022-08-04 13:12:24 +00:00
Antonio Andelic
0a7d2e7b8a Use LockMemoryExceptionThread 2022-08-04 13:03:05 +00:00
Antonio Andelic
278921be3b Fix style 2022-08-04 07:25:28 +00:00
Antonio Andelic
fa98338ce1 Add comment for memory tracker blocking 2022-08-04 07:13:42 +00:00
Antonio Andelic
56ca93ca3f Block memory tracker in Keeper during commit 2022-08-03 14:14:57 +00:00
Antonio Andelic
05467e315f
Merge pull request #39698 from ClickHouse/update-digest-version 
Update Keeper version for digest
 2022-08-03 13:13:00 +02:00
Antonio Andelic
b3b3c371f0
Update KeeperStorage.h 2022-08-02 09:20:02 +02:00
Antonio Andelic
7cdad88327 Create snapshot on shutdown 2022-07-30 11:34:17 +00:00
Antonio Andelic
21becb3030 Update version for digest 2022-07-29 07:36:19 +00:00
Antonio Andelic
e3cb65a99a Rollback on failed preprocess 2022-07-29 06:51:53 +00:00
Antonio Andelic
c0a5d45258 Rollback on failed PreAppend 2022-07-28 14:06:45 +00:00
Antonio Andelic
1090d6bca7 Rollback request if storing log fails 2022-07-28 12:55:29 +00:00
Antonio Andelic
e62526720f Address PR comments 2022-07-27 07:51:30 +00:00
Antonio Andelic
04b03b6a90 Don't allow to overwrite on startup 2022-07-25 12:38:48 +00:00
Antonio Andelic
afb6cb6824 Add KeeperContext 2022-07-23 14:45:31 +00:00
Antonio Andelic
6aff87d4b5 Ignore system paths from snapshots and logstore 2022-07-23 13:46:59 +00:00
Antonio Andelic
46a6fbd7aa Small polish 2022-07-22 11:03:07 +00:00
Antonio Andelic
3040ff0959 Merge branch 'master' into keeper-version-check 2022-07-22 08:07:55 +00:00
Antonio Andelic
75476d5110 Store only root system node 2022-07-22 08:07:38 +00:00
Antonio Andelic
fdd79e62da
Merge pull request #39393 from ClickHouse/keeper-disable-digest-default 
Disable real-time digest in Keeper by default
 2022-07-21 15:15:22 +02:00
Antonio Andelic
7d30ab80c4 Fix compatibility 2022-07-21 09:31:06 +00:00
Alexey Milovidov
19c4c0bd6e
Update CoordinationSettings.cpp 2022-07-21 06:41:42 +03:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Antonio Andelic
a4450225c7 Disable digest by default 2022-07-19 17:14:44 +00:00
Antonio Andelic
a3e00faaf9 Fix unit tests 2022-07-19 13:14:36 +00:00
Antonio Andelic
ce570b6ee3 Add logs and 4LW for api version 2022-07-19 09:02:57 +00:00
Antonio Andelic
e6ded88ea3 Small refactoring 2022-07-19 08:51:12 +00:00
Antonio Andelic
cfc741030f Revert to the version with path 2022-07-18 14:30:15 +00:00
Antonio Andelic
48db20d5a1 Add 4LW for api version 2022-07-18 12:19:16 +00:00
Antonio Andelic
b8ffb151dc Some small polishing 2022-07-18 08:52:52 +00:00
Antonio Andelic
7d7c4ce4cd Add ApiVersion request 2022-07-18 08:44:38 +00:00
Antonio Andelic
73e0c35ab0 Use 4LW for api version 2022-07-15 10:13:39 +00:00
Antonio Andelic
d81758898a Remove extra newline 2022-07-13 15:26:25 +00:00
Antonio Andelic
f05f22aa13 Merge branch 'master' into keeper-version-check 2022-07-13 15:25:48 +00:00
Antonio Andelic
3b5bdd1e2a Add test for current API 2022-07-13 12:41:16 +00:00
Antonio Andelic
c296e84cab fix unit tests in debug mode 2022-07-13 12:33:18 +00:00
Antonio Andelic
838a11a85a Rename RaftAppendResult 2022-07-13 11:35:06 +00:00
Antonio Andelic
2634f80956 Remove trailing whitespaces 2022-07-13 11:06:58 +00:00
Antonio Andelic
b49ac48bfd Merge branch 'master' into keeper-linearizable-reads 2022-07-13 09:30:21 +00:00
Nikita Mikhaylov
1f0280bcde
Fix path retrieval for Keeper's state (#39148) 2022-07-13 11:25:34 +02:00
Antonio Andelic
ee4828b084 Fix list watches 2022-07-13 09:23:05 +00:00
Antonio Andelic
8bd86c39f6 Add valid values for read_mode setting 2022-07-12 08:00:31 +00:00
Antonio Andelic
fd1e5d43a0 fix unit tests in debug mode 2022-07-12 07:52:50 +00:00
Antonio Andelic
5f661e3e2b Merge branch 'master' into keeper-linearizable-reads 2022-07-12 07:33:34 +00:00
Antonio Andelic
0555477ec1 Remove debug logs 2022-07-12 07:05:29 +00:00
Antonio Andelic
6dcdf0fcb1 Throw CORRUPTED_DATA in debug mode for invalid checksum 2022-07-12 06:41:58 +00:00
Antonio Andelic
1f5b3b9922 Add restrictions for modification of internal paths 2022-07-11 13:06:24 +00:00
Antonio Andelic
568ba5d3cc Address PR comments 2022-07-11 12:56:09 +00:00
Antonio Andelic
efc7f5c534 Merge branch 'master' into keeper-persist-state 2022-07-11 12:43:06 +00:00
Antonio Andelic
41dd4e13f4 Merge branch 'master' into keeper-version-check 2022-07-11 09:38:20 +00:00