936 Commits

Author	SHA1	Message	Date
Nikolai Kochetov	62ff98344e	Validate s3 part upload settings.	2022-12-17 14:09:53 +00:00
kssenii	30547d2dcd	Replace old named collections code for url	2022-12-17 00:24:05 +01:00
Antonio Andelic	ee900ca7df	Add truncate	2022-12-15 13:09:52 +00:00
Antonio Andelic	bf6f163952	Merge branch 'master' into keeper-use-fallocate	2022-12-15 09:05:40 +00:00
Antonio Andelic	d454e00016	Changelog refactor	2022-12-15 08:12:02 +00:00
Anton Popov	0c87031e80	Merge remote-tracking branch 'upstream/master' into HEAD	2022-12-13 16:33:21 +00:00
Antonio Andelic	662c1f956c	Changelog refactoring	2022-12-13 14:00:27 +00:00
Antonio Andelic	898cf867af	Try with fallocate	2022-12-13 11:08:59 +00:00
Vitaly Baranov	b91af1b650	Fix initialization of s3 request settings.	2022-12-10 05:43:51 +01:00
Antonio Andelic	e1a6a1182e	Throw when port already in use (#43984)	2022-12-07 13:27:49 +01:00
Antonio Andelic	200141da13	Use logger in tests	2022-12-02 08:16:18 +00:00
Antonio Andelic	7cd76df918	Merge branch 'master' into parallel-log-appending	2022-12-01 14:44:36 +00:00
Antonio Andelic	554ff1db6f	Small changes	2022-12-01 12:54:29 +00:00
Antonio Andelic	940fdd4abd	Remove comment	2022-12-01 12:24:27 +01:00
Antonio Andelic	25ae52111e	Protect writer with mutex	2022-12-01 11:18:23 +00:00
Antonio Andelic	c12c0cc8af	Merge branch 'master' into parallel-log-appending	2022-12-01 08:53:52 +00:00
Antonio Andelic	2f501d5327	Merge branch 'master' into keeper-prometheus	2022-11-30 13:24:37 +01:00
Antonio Andelic	fbda86ddc8	PR fixes	2022-11-30 13:24:08 +01:00
Antonio Andelic	7f897f1a20	Refactor	2022-11-29 13:00:13 +00:00
Antonio Andelic	c8b2cb5650	Update src/Coordination/CoordinationSettings.h ... Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>	2022-11-29 10:08:25 +01:00
Antonio Andelic	678958603b	Merge branch 'master' into keeper-prometheus	2022-11-28 09:48:57 +01:00
Antonio Andelic	fe44ac0c86	Merge branch 'master' into parallel-log-appending	2022-11-28 08:45:20 +00:00
Antonio Andelic	e91314d57f	Use initialized flag	2022-11-28 08:44:46 +00:00
Antonio Andelic	cade21b76e	Fix 4LW test	2022-11-26 19:24:10 +00:00
Antonio Andelic	76eca8bdfb	Merge branch 'master' into more-aggresive-batching-keeper	2022-11-26 18:28:11 +00:00
Antonio Andelic	cd471e9c61	Add setting for quick batch size	2022-11-26 17:33:40 +00:00
Anton Popov	65a78bcd91	improve performance of storage S3	2022-11-26 15:24:01 +00:00
Antonio Andelic	7437c719a7	Try batching requests more aggresively	2022-11-25 17:25:28 +00:00
Antonio Andelic	38d99c241f	Fix tests	2022-11-25 13:53:03 +00:00
Antonio Andelic	53f708366c	Fix data race in Keeper snapshot	2022-11-25 12:50:42 +00:00
Antonio Andelic	ce532cd12d	Fix circular deps	2022-11-24 13:43:05 +00:00
Antonio Andelic	a6f38cb1cc	Merge branch 'master' into keeper-prometheus	2022-11-24 12:12:08 +01:00
Antonio Andelic	0cc53995df	Merge branch 'master' into parallel-log-appending	2022-11-22 15:05:08 +01:00
Kseniia Sumarokova	5c90d5aa7e	Merge pull request #43253 from xiedeyantu/fix-s3-glob ... fix s3 support question mark wildcard	2022-11-22 14:26:56 +01:00
Antonio Andelic	4827bcb6bb	Merge branch 'master' into parallel-log-appending	2022-11-22 14:14:17 +01:00
Antonio Andelic	55ef96dc11	Fix unit tests	2022-11-22 11:37:59 +00:00
Antonio Andelic	aeb0d39b50	Fix shutdown	2022-11-21 15:01:18 +00:00
Antonio Andelic	c9472d67b6	Refactor changelog	2022-11-21 12:22:36 +00:00
Antonio Andelic	ff681d5faa	Merge branch 'master' into parallel-log-appending	2022-11-21 11:59:16 +00:00
Antonio Andelic	3cb202a63b	Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader ... Keeper support manually assigning leader	2022-11-21 09:30:00 +01:00
Antonio Andelic	e78761b66c	Merge branch 'master' into keeper-prometheus	2022-11-21 09:10:10 +01:00
xiedeyantu	c258d3ac8b	fix s3 support question mark wildcard	2022-11-18 12:11:22 +08:00
Sergei Trifonov	f2f0676bcc	Revert "Revert "S3 request per second rate throttling""	2022-11-17 17:35:04 +01:00
Alexander Tokmakov	9011a18234	Revert "S3 request per second rate throttling"	2022-11-16 22:33:48 +03:00
Sergei Trifonov	159743edd6	Merge pull request #43014 from ClickHouse/disk-s3-throttler ... S3 request per second rate throttling	2022-11-16 18:51:06 +01:00
JackyWoo	f8d71f60a2	Merge branch 'master' into keeper_manually_assign_leader	2022-11-16 09:34:40 +08:00
Antonio Andelic	96838f5be7	Merge branch 'master' into keeper-prometheus	2022-11-15 12:00:02 +00:00
Antonio Andelic	d08dce5ec7	Test parallel append	2022-11-14 13:17:05 +00:00
Raúl Marín	54db7c6520	Enforce checking read output	2022-11-11 10:56:18 +01:00
Antonio Andelic	94aab4276c	Bring back heavy metrics	2022-11-10 12:56:27 +01:00
Antonio Andelic	8aeb406d82	Fix standalone build	2022-11-10 10:49:00 +01:00
Antonio Andelic	36d763011b	Compile standalone keeper	2022-11-10 09:23:52 +01:00
Antonio Andelic	1ed3930809	Add more metrics for Keeper	2022-11-09 15:51:41 +01:00
Antonio Andelic	c92ae5a385	Add suport for AsyncMetrics in standalone Keeper	2022-11-09 13:46:03 +01:00
Sergei Trifonov	8eedd1e046	Merge branch 'master' into disk-s3-throttler	2022-11-08 15:00:56 +01:00
serxa	6d5d9ff421	rename ReadWriteSettings -> RequestSettings	2022-11-08 13:48:23 +00:00
JackyWoo	9060141f16	revert last change	2022-11-08 18:44:43 +08:00
JackyWoo	e865842098	Add rqld to white list	2022-11-08 16:03:13 +08:00
JackyWoo	eeedd7d9ba	Keeper support manually assigning leader	2022-11-08 12:01:53 +08:00
serxa	2daec0b45e	S3 request per second rate throttling + refactoring	2022-11-07 18:05:40 +00:00
Antonio Andelic	9ac829d4c4	Merge pull request #41766 from JackyWoo/manual_snapshot_for_keeper ... Manually snapshot creation for keeper	2022-11-07 09:18:58 +01:00
JackyWoo	73dfe4acd7	Add to default white list	2022-11-05 17:56:55 +08:00
Antonio Andelic	25b19e6ba0	Merge branch 'master' into manual_snapshot_for_keeper	2022-11-03 08:55:01 +01:00
Antonio Andelic	664690f9a9	Add comments	2022-11-03 07:38:43 +00:00
Antonio Andelic	a6c4204b0c	Don't iterate all cached nodes on commit	2022-11-03 07:36:25 +00:00
JackyWoo	c7a0ebeb05	little fix	2022-10-25 17:46:24 +08:00
JackyWoo	9a36a509fe	fix test	2022-10-25 17:15:49 +08:00
JackyWoo	b5d1c4e657	replace snpd with lgif	2022-10-24 20:08:58 +08:00
JackyWoo	2f30c817bf	little fix	2022-10-24 17:23:47 +08:00
JackyWoo	39c88c74e8	check whether last manually created snapshot is done	2022-10-22 22:31:17 +08:00
JackyWoo	bf291790b7	Merge branch 'master' into manual_snapshot_for_keeper	2022-10-22 16:57:50 +08:00
Azat Khuzhin	905a95e166	Review fixes ... Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>	2022-10-21 22:40:13 +02:00
Azat Khuzhin	4e76629aaf	Fixes for -Wshorten-64-to-32 ... - lots of static_cast - add safe_cast - types adjustments - config - IStorage::read/watch - ... - some TODO's (to convert types in future) P.S. That was quite a journey... v2: fixes after rebase v3: fix conflicts after #42308 merged Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>	2022-10-21 13:25:19 +02:00
Antonio Andelic	540728ab96	Simplify logic a bit	2022-10-18 10:58:25 +00:00
Antonio Andelic	1275c464aa	Fix unit test build	2022-10-18 10:44:22 +00:00
Antonio Andelic	8157fd0595	fix build	2022-10-17 14:16:56 +00:00
Antonio Andelic	238f86d892	Small refactoring	2022-10-17 11:06:33 +00:00
Antonio Andelic	7c7395575f	Merge branch 'master' into keeper-upload-snapshot-to-s3	2022-10-17 11:02:36 +00:00
Robert Schulze	da5a2e2db0	Merge remote-tracking branch 'origin/master' into generated-file-cleanup ... Physical merge conflicts: - src/Common/ZooKeeper/ZooKeeperImpl.cpp - src/Core/config_core.h.in - src/Functions/FunctionsAES.h - src/Functions/config_functions.h.in - src/configure_config.cmake Logical merge conflicts: - Functions/tryDecrypt.cpp	2022-10-06 08:43:25 +00:00
Antonio Andelic	94f1fe39bb	Add support for exists in multiread	2022-09-29 10:39:33 +00:00
Antonio Andelic	3109ce51c6	Merge branch 'master' into keeper-multiread	2022-09-29 06:50:49 +00:00
Antonio Andelic	349bd7fd9a	Dont fail MultiRead on first failed op	2022-09-29 06:50:28 +00:00
Robert Schulze	f24fab7747	Fix some #include atrocities	2022-09-28 13:49:28 +00:00
Robert Schulze	fd86829824	Consolidate config_core.h into config.h ... Less duplication, less confusion ...	2022-09-28 13:31:57 +00:00
Robert Schulze	6d70b4a1f6	Generate config_version.h into ${CONFIG_INCLUDE_PATH} ... This makes the target location consistent with other auto-generated files like config_formats.h, config_core.h, and config_functions.h and simplifies the build of clickhouse_common.	2022-09-28 12:48:26 +00:00
Robert Schulze	78fc36ca49	Generate config.h into ${CONFIG_INCLUDE_PATH} ... This makes the target location consistent with other auto-generated files like config_formats.h, config_core.h, and config_functions.h and simplifies the build of clickhouse_common.	2022-09-28 12:48:26 +00:00
Antonio Andelic	d0457addbd	Support filtered list	2022-09-27 15:02:26 +00:00
JackyWoo	d1c85f68e8	manually snapshot creation for keeper	2022-09-26 18:29:15 +08:00
Antonio Andelic	937d534cd4	Add support for simple list	2022-09-26 08:05:02 +00:00
Antonio Andelic	f833366555	Merge branch 'master' into keeper-multiread	2022-09-26 07:16:45 +00:00
Antonio Andelic	9da433a904	Fix unused	2022-09-22 14:33:35 +00:00
Antonio Andelic	44a3d6babe	Address PR comments	2022-09-22 13:03:27 +00:00
Antonio Andelic	6798b500e9	Wait on startup for Keeper	2022-09-21 15:12:16 +00:00
Antonio Andelic	eefbbf53e8	Extract S3 logic	2022-09-21 11:53:54 +00:00
Antonio Andelic	fc6c1faa57	Merge branch 'master' into keeper-upload-snapshot-to-s3	2022-09-21 06:57:58 +00:00
Antonio Andelic	1763af37b9	Merge pull request #41403 from ClickHouse/keeper-fix-shutdown-without-server ... Fix Keeper segfault during shutdown when RAFT server failed to start	2022-09-19 12:03:58 +02:00
alesapin	b6d61318d0	One more validation for broken config in keeper	2022-09-18 00:04:16 +02:00
Alexander Tokmakov	42dab6dc02	Revert "Merge pull request #38200 from ClickHouse/keeper-linearizable-reads" ... This reverts commit 222b964070, reversing changes made to da01982652.	2022-09-17 19:22:58 +02:00
Antonio Andelic	81f7cf3ca6	Add support for MultiRead in Keeper	2022-09-16 12:00:35 +00:00
Antonio Andelic	acd9cb089e	Fix segfault during shutdown when RAFT server failed to start	2022-09-16 10:03:21 +00:00
Antonio Andelic	5662e0050c	Define client for multi read	2022-09-16 09:55:02 +00:00
Antonio Andelic	296fe01fb8	Merge branch 'master' into keeper-upload-snapshot-to-s3	2022-09-16 08:00:15 +00:00
Antonio Andelic	39307f9dba	Extract into function	2022-09-15 16:14:53 +00:00
Antonio Andelic	0dbd0c1f72	Merge branch 'master' into keeper-linearizable-reads	2022-09-15 16:03:18 +00:00
Antonio Andelic	ad7864cb5f	enable S3 if client is built	2022-09-15 14:06:22 +00:00
Antonio Andelic	d3d1676352	Fix config update detection	2022-09-15 13:48:39 +00:00
Antonio Andelic	d4e4ac3801	Lock file during S3 snapshot upload	2022-09-15 09:11:55 +00:00
Antonio Andelic	9fdd2f2e61	Basic implementation for S3 snapshot upload	2022-09-15 09:11:55 +00:00
Antonio Andelic	3a9996f04c	Add bg thread for snapshot upload to S3	2022-09-15 09:11:55 +00:00
Antonio Andelic	36f46891dd	Merge pull request #41215 from ClickHouse/keeper-close-connection-on-shutdown ... Close sessions on Keeper shutdown	2022-09-13 19:09:34 +02:00
Antonio Andelic	baf7255cff	Merge pull request #39976 from ClickHouse/keeper-storage ... KeeperMap storage engine	2022-09-13 14:15:09 +02:00
alesapin	69d08b1007	Merge pull request #41075 from ClickHouse/keeper-fix-possible-segfault ... Fix possible segfault during Keeper shutdown	2022-09-13 12:13:24 +02:00
Antonio Andelic	a6b5ffec5d	Polishing	2022-09-13 09:51:46 +00:00
Antonio Andelic	b2cc6a8cc6	Use promise/future	2022-09-12 18:19:41 +00:00
Antonio Andelic	c2dfabe51c	Use shared_ptr for Event	2022-09-12 12:25:39 +00:00
Antonio Andelic	e985ee3354	Close sessions on Keeper shutdown	2022-09-12 12:22:48 +00:00
Antonio Andelic	32483aeec8	Merge branch 'master' into keeper-storage	2022-09-09 12:24:01 +00:00
Antonio Andelic	951ee44229	Process read requests with snapshot install	2022-09-07 11:51:56 +00:00
Antonio Andelic	a377a5f1e9	Change assert to LOGICAL_ERROR	2022-09-07 09:24:50 +00:00
Antonio Andelic	4cdf18cc81	Shutdown storage after RAFT	2022-09-07 09:20:47 +00:00
Antonio Andelic	c6b17bd7f9	Don't preprocess if storage finalized	2022-09-07 09:14:03 +00:00
Antonio Andelic	f0ba3ee927	Fix typo	2022-09-05 09:51:02 +02:00
Antonio Andelic	110ad53300	Merge branch 'master' into keeper-linearizable-reads	2022-09-05 07:12:45 +00:00
Antonio Andelic	b816d46520	Merge branch 'master' into fix-keeper-system-path-check	2022-09-02 13:42:16 +02:00
Antonio Andelic	9ca20b97c2	Merge branch 'master' into keeper-storage	2022-09-02 12:13:07 +02:00
Antonio Andelic	e64436fef3	Fix typos with new codespell	2022-09-02 08:54:48 +00:00
Antonio Andelic	ad2196155c	Add test for system node modification	2022-09-02 08:14:06 +00:00
Antonio Andelic	b7eebfc626	Correctly check if the node is using system path	2022-09-02 07:47:12 +00:00
Antonio Andelic	7164e59b54	Merge branch 'master' into keeper-linearizable-reads	2022-09-01 08:52:31 +00:00
Antonio Andelic	fa4c7259f9	Fix typos	2022-09-01 08:46:28 +00:00
Antonio Andelic	c7d1dd942c	Address PR comments	2022-09-01 07:56:06 +00:00
Antonio Andelic	21cdb9a500	Don't remove from cache if it wasn't added during rollback	2022-08-30 08:14:53 +00:00
Antonio Andelic	2416ddfae7	Add extra message about startup preprocessing	2022-08-30 08:14:31 +00:00
Antonio Andelic	57471d11db	Fix uncommitted auth	2022-08-29 12:55:30 +00:00
Antonio Andelic	50af0b1938	Add bunch of comments for request processing	2022-08-29 10:50:18 +00:00
Antonio Andelic	b4af21b6fb	Small fixes	2022-08-29 07:06:51 +00:00
Antonio Andelic	bff2b232ef	Cache deltas for path	2022-08-26 14:03:38 +00:00
Antonio Andelic	26d68dce23	Fix style	2022-08-26 10:55:07 +00:00
Antonio Andelic	dae65178b7	Improve preprocessing performance	2022-08-26 10:46:14 +00:00
Antonio Andelic	5d45f2faa9	Add comments	2022-08-23 08:22:03 +00:00
Antonio Andelic	46882791cd	Fix build	2022-08-23 07:29:47 +00:00
Antonio Andelic	444ac9af6c	Merge branch 'master' into keeper-linearizable-reads	2022-08-23 07:03:35 +00:00
Antonio Andelic	99d23cb67e	Merge branch 'master' into keeper-linearizable-reads	2022-08-23 07:03:20 +00:00
Antonio Andelic	a87a762b0c	Merge pull request #39973 from ClickHouse/keeper-listen-host ... Support `interserver_listen_host` in Keeper	2022-08-23 09:03:08 +02:00
Alexey Milovidov	74e1f4dc61	Fix clang-tidy	2022-08-20 17:09:20 +02:00
Antonio Andelic	9396834762	Addressed comments	2022-08-17 09:29:08 +00:00
Antonio Andelic	24ee3e022d	Merge branch 'master' into keeper-listen-host	2022-08-09 09:39:27 +02:00
Alexey Milovidov	48bed932ec	Merge pull request #39931 from HarryLeeIBM/hlee-s390x-coordination ... Fix Endian issue in KeeperSnapshotManager for s390x	2022-08-09 04:24:32 +03:00
HarryLeeIBM	42323cfb92	Use constexpr char array as magic number	2022-08-08 10:59:09 -07:00
Antonio Andelic	6b935a252f	Support multiple listen hosts	2022-08-08 09:08:19 +00:00