Antonio Andelic
6ebee202bd
Flush buffer when no sync
2023-03-01 14:47:15 +00:00
Antonio Andelic
c797122dbe
Better
2023-03-01 08:45:50 +00:00
Antonio Andelic
6e28406751
Merge branch 'master' into fix-data-size
2023-02-28 20:09:29 +00:00
Alexey Milovidov
d8cda3dbb8
Remove PVS-Studio
2023-02-19 23:30:05 +01:00
Alexander Tokmakov
e96340e183
Merge pull request #46410 from ClickHouse/fix_noisy_log_messages
...
Fix noisy log messages
2023-02-15 01:22:42 +03:00
Alexander Tokmakov
1bf3735ca9
fix noisy log messages
2023-02-14 15:33:27 +01:00
Sema Checherinda
9bef1bec28
Merge pull request #46274 from aiven-sal/aiven-sal/keeper
...
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
2023-02-14 12:44:04 +01:00
Sema Checherinda
5882f3878f
except zero values election_timeout
2023-02-13 12:22:08 +01:00
Salvatore Mesoraca
fff2c09b87
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
2023-02-13 10:09:45 +01:00
Antonio Andelic
f96d480563
Merge branch 'master' into add-support-for-settings-alias
2023-02-09 16:07:45 +01:00
Azat Khuzhin
8cc41b7f41
Check return value of ::close()
...
Note, that according close(2), EINTR should not be retriable for close()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-07 11:28:22 +01:00
Antonio Andelic
b11458f142
Fix erase
2023-02-07 07:51:22 +00:00
Antonio Andelic
9cde46acff
Correctly calculate size on update
2023-02-06 16:28:05 +00:00
Antonio Andelic
aca4f08bf5
Correctly calculate size after snapshot load
2023-02-06 14:59:23 +00:00
Antonio Andelic
8929838fcc
Merge branch 'master' into add-support-for-settings-alias
2023-02-03 17:47:00 +00:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution ( #45783 )
...
* Update aws
* Define S3 client with bucket and endpoint resolution
* Add defines for ErrorCodes
* Use S3Client everywhere
* Remove unused errorcode
* Add DROP S3 CLIENT CACHE query
* Add a comment
* Fix style
* Update aws
* Update reference files
* Add missing include
* Fix unit test
* Remove unneeded declarations
* Correctly use RetryStrategy
* Rename S3Client to Client
* Fix retry count
* fix clang-tidy warnings
2023-02-03 14:30:52 +01:00
LiuYangkuan
f29700bd2f
use IDisk to do IO in Keeper's snapshots and logs
2023-02-02 19:47:30 +08:00
Antonio Andelic
f613bfd8d2
Merge branch 'master' into add-support-for-settings-alias
2023-02-02 11:15:39 +00:00
Antonio Andelic
85cfee4bb9
Better alias definition
2023-02-01 13:54:03 +00:00
Antonio Andelic
88c1744975
Better
2023-02-01 14:25:58 +01:00
Antonio Andelic
5de6f86c27
Fix clang-tidy error
2023-02-01 12:28:44 +00:00
Antonio Andelic
ec04cf6876
Merge branch 'master' into add-support-for-settings-alias
2023-02-01 09:01:01 +00:00
Antonio Andelic
ece92fa866
Update src/Coordination/Changelog.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2023-01-31 17:11:18 +01:00
Antonio Andelic
8457d3bf3d
Apply comments
2023-01-31 14:18:54 +00:00
Antonio Andelic
1786237be6
Merge branch 'master' into keeper-use-fallocate
2023-01-31 13:53:47 +00:00
Vitaly Baranov
f9282cfcac
Make version_id an optional parameter again.
2023-01-27 15:10:19 +01:00
Vitaly Baranov
aea9ccdb60
Pass request settings to S3::getObjectInfo().
2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed
Move getObjectInfo() to a separate header.
2023-01-27 15:09:38 +01:00
Antonio Andelic
714fad1529
Add support for settings alias
2023-01-26 14:06:46 +00:00
Antonio Andelic
ebfe7f9991
Fix style
2023-01-26 11:18:07 +00:00
Antonio Andelic
1186718658
Add overallocate size setting
2023-01-26 09:44:37 +00:00
Antonio Andelic
7bfdfa4bf7
Preallocate more space
2023-01-25 16:26:14 +00:00
Antonio Andelic
7421d8f872
Merge branch 'master' into keeper-use-fallocate
2023-01-25 11:36:47 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Alexander Tokmakov
870cfcc36a
less fmt::runtime usages
2023-01-17 00:11:59 +01:00
Antonio Andelic
6a0e481ed7
Merge branch 'master' into keeper-use-fallocate
2023-01-12 13:57:05 +00:00
Antonio Andelic
30510c5eb6
Merge pull request #44937 from ClickHouse/fix-changelog-deadlock
...
Fix deadlock in Keeper's changelog
2023-01-10 08:59:51 +01:00
Antonio Andelic
22679ddf11
Add comment
2023-01-09 14:26:20 +01:00
Antonio Andelic
a39f437306
Typo fix
2023-01-09 13:34:48 +01:00
Antonio Andelic
27b2bae640
make it better
2023-01-09 12:01:22 +00:00
Antonio Andelic
4fcbe14435
Small fixes for Coordination unit tests
2023-01-08 13:44:32 +00:00
Kseniia Sumarokova
573d3283b0
Merge pull request #44327 from kssenii/use-new-named-collections-code-2
...
Replace old named collections code with new (from #43147 ) part 2
2023-01-06 13:06:26 +01:00
Antonio Andelic
9b3314a931
Fix deadlock in Changelog flush
2023-01-05 14:51:05 +00:00
Antonio Andelic
136436958b
Merge pull request #44908 from ClickHouse/fix-keeper-snapshot-on-exit
...
Avoid Keeper crash on shutdown (fix `test_keeper_snapshot_on_exit`)
2023-01-05 08:59:58 +01:00
Antonio Andelic
7c2cdfa65a
Don't throw exception if queue is finished
2023-01-04 15:46:38 +00:00
kssenii
67509aa2d5
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2023-01-03 16:41:30 +01:00
Smita Kulkarni
8a5fd7f96d
Added comments and addressed review comments - Allow macro substitution in endpoint
2023-01-03 11:44:26 +01:00
Antonio Andelic
f2653c016a
Manually set deleted flag
2023-01-02 14:36:58 +00:00
Antonio Andelic
f32fab5025
Check if file exists before rename
2023-01-02 13:48:13 +00:00
Antonio Andelic
2e93640614
Fix ASAN error
2023-01-02 13:37:40 +00:00
Antonio Andelic
b845f880fb
Polish
2023-01-02 10:42:15 +00:00
Antonio Andelic
14355d8009
Merge branch 'master' into keeper-use-fallocate
2023-01-02 08:22:27 +00:00
Antonio Andelic
55548a6678
style fix
2023-01-02 08:22:00 +00:00
SmitaRKulkarni
f9a3aeb172
Merge branch 'master' into 40951_Allow_macro_substitution_in_endpoint
2022-12-30 10:43:14 +01:00
Alexey Milovidov
f0fa0977ed
Fix wrong log messages
2022-12-26 18:36:07 +01:00
Smita Kulkarni
3d62bc2635
Allow macro substitution in endpoint of disks
...
Implementation:
* Updated to substitute macros in endpoint for all disks (s3, hfs, web).
Testing:
* Added an integration test test/integration/test_endpoint_macro_substitution.
2022-12-23 15:09:24 +01:00
Antonio Andelic
64ade259cf
Fix tests
2022-12-22 10:20:55 +00:00
Antonio Andelic
88d41dea93
Merge branch 'master' into keeper-use-fallocate
2022-12-21 08:59:09 +00:00
Antonio Andelic
9711355cc9
Fix tests
2022-12-21 08:43:55 +00:00
kssenii
6bd4f8c029
Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2
2022-12-20 21:17:28 +01:00
Antonio Andelic
6d83221d4e
Remove unused error code
2022-12-19 12:20:26 +00:00
Antonio Andelic
c2c115d2ba
Add setting for max_log_file_size
2022-12-19 11:51:05 +00:00
Antonio Andelic
f241701c2a
Add checks when fallocate is not supported
2022-12-19 11:48:54 +00:00
Antonio Andelic
9a8b248735
Merge branch 'master' into keeper-use-fallocate
2022-12-19 09:32:05 +00:00
Nikolai Kochetov
62ff98344e
Validate s3 part upload settings.
2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd
Replace old named collections code for url
2022-12-17 00:24:05 +01:00
Antonio Andelic
ee900ca7df
Add truncate
2022-12-15 13:09:52 +00:00
Antonio Andelic
bf6f163952
Merge branch 'master' into keeper-use-fallocate
2022-12-15 09:05:40 +00:00
Antonio Andelic
d454e00016
Changelog refactor
2022-12-15 08:12:02 +00:00
Anton Popov
0c87031e80
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-13 16:33:21 +00:00
Antonio Andelic
662c1f956c
Changelog refactoring
2022-12-13 14:00:27 +00:00
Antonio Andelic
898cf867af
Try with fallocate
2022-12-13 11:08:59 +00:00
Vitaly Baranov
b91af1b650
Fix initialization of s3 request settings.
2022-12-10 05:43:51 +01:00
Antonio Andelic
e1a6a1182e
Throw when port already in use ( #43984 )
2022-12-07 13:27:49 +01:00
Antonio Andelic
200141da13
Use logger in tests
2022-12-02 08:16:18 +00:00
Antonio Andelic
7cd76df918
Merge branch 'master' into parallel-log-appending
2022-12-01 14:44:36 +00:00
Antonio Andelic
554ff1db6f
Small changes
2022-12-01 12:54:29 +00:00
Antonio Andelic
940fdd4abd
Remove comment
2022-12-01 12:24:27 +01:00
Antonio Andelic
25ae52111e
Protect writer with mutex
2022-12-01 11:18:23 +00:00
Antonio Andelic
c12c0cc8af
Merge branch 'master' into parallel-log-appending
2022-12-01 08:53:52 +00:00
Antonio Andelic
2f501d5327
Merge branch 'master' into keeper-prometheus
2022-11-30 13:24:37 +01:00
Antonio Andelic
fbda86ddc8
PR fixes
2022-11-30 13:24:08 +01:00
Antonio Andelic
7f897f1a20
Refactor
2022-11-29 13:00:13 +00:00
Antonio Andelic
c8b2cb5650
Update src/Coordination/CoordinationSettings.h
...
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
2022-11-29 10:08:25 +01:00
Antonio Andelic
678958603b
Merge branch 'master' into keeper-prometheus
2022-11-28 09:48:57 +01:00
Antonio Andelic
fe44ac0c86
Merge branch 'master' into parallel-log-appending
2022-11-28 08:45:20 +00:00
Antonio Andelic
e91314d57f
Use initialized flag
2022-11-28 08:44:46 +00:00
Antonio Andelic
cade21b76e
Fix 4LW test
2022-11-26 19:24:10 +00:00
Antonio Andelic
76eca8bdfb
Merge branch 'master' into more-aggresive-batching-keeper
2022-11-26 18:28:11 +00:00
Antonio Andelic
cd471e9c61
Add setting for quick batch size
2022-11-26 17:33:40 +00:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
Antonio Andelic
7437c719a7
Try batching requests more aggresively
2022-11-25 17:25:28 +00:00
Antonio Andelic
38d99c241f
Fix tests
2022-11-25 13:53:03 +00:00
Antonio Andelic
53f708366c
Fix data race in Keeper snapshot
2022-11-25 12:50:42 +00:00
Antonio Andelic
ce532cd12d
Fix circular deps
2022-11-24 13:43:05 +00:00
Antonio Andelic
a6f38cb1cc
Merge branch 'master' into keeper-prometheus
2022-11-24 12:12:08 +01:00
Antonio Andelic
0cc53995df
Merge branch 'master' into parallel-log-appending
2022-11-22 15:05:08 +01:00
Kseniia Sumarokova
5c90d5aa7e
Merge pull request #43253 from xiedeyantu/fix-s3-glob
...
fix s3 support question mark wildcard
2022-11-22 14:26:56 +01:00
Antonio Andelic
4827bcb6bb
Merge branch 'master' into parallel-log-appending
2022-11-22 14:14:17 +01:00
Antonio Andelic
55ef96dc11
Fix unit tests
2022-11-22 11:37:59 +00:00
Antonio Andelic
aeb0d39b50
Fix shutdown
2022-11-21 15:01:18 +00:00
Antonio Andelic
c9472d67b6
Refactor changelog
2022-11-21 12:22:36 +00:00
Antonio Andelic
ff681d5faa
Merge branch 'master' into parallel-log-appending
2022-11-21 11:59:16 +00:00
Antonio Andelic
3cb202a63b
Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader
...
Keeper support manually assigning leader
2022-11-21 09:30:00 +01:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus
2022-11-21 09:10:10 +01:00
xiedeyantu
c258d3ac8b
fix s3 support question mark wildcard
2022-11-18 12:11:22 +08:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
159743edd6
Merge pull request #43014 from ClickHouse/disk-s3-throttler
...
S3 request per second rate throttling
2022-11-16 18:51:06 +01:00
JackyWoo
f8d71f60a2
Merge branch 'master' into keeper_manually_assign_leader
2022-11-16 09:34:40 +08:00
Antonio Andelic
96838f5be7
Merge branch 'master' into keeper-prometheus
2022-11-15 12:00:02 +00:00
Antonio Andelic
d08dce5ec7
Test parallel append
2022-11-14 13:17:05 +00:00
Raúl Marín
54db7c6520
Enforce checking read output
2022-11-11 10:56:18 +01:00
Antonio Andelic
94aab4276c
Bring back heavy metrics
2022-11-10 12:56:27 +01:00
Antonio Andelic
8aeb406d82
Fix standalone build
2022-11-10 10:49:00 +01:00
Antonio Andelic
36d763011b
Compile standalone keeper
2022-11-10 09:23:52 +01:00
Antonio Andelic
1ed3930809
Add more metrics for Keeper
2022-11-09 15:51:41 +01:00
Antonio Andelic
c92ae5a385
Add suport for AsyncMetrics in standalone Keeper
2022-11-09 13:46:03 +01:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
JackyWoo
9060141f16
revert last change
2022-11-08 18:44:43 +08:00
JackyWoo
e865842098
Add rqld to white list
2022-11-08 16:03:13 +08:00
JackyWoo
eeedd7d9ba
Keeper support manually assigning leader
2022-11-08 12:01:53 +08:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Antonio Andelic
9ac829d4c4
Merge pull request #41766 from JackyWoo/manual_snapshot_for_keeper
...
Manually snapshot creation for keeper
2022-11-07 09:18:58 +01:00
JackyWoo
73dfe4acd7
Add to default white list
2022-11-05 17:56:55 +08:00
Antonio Andelic
25b19e6ba0
Merge branch 'master' into manual_snapshot_for_keeper
2022-11-03 08:55:01 +01:00
Antonio Andelic
664690f9a9
Add comments
2022-11-03 07:38:43 +00:00
Antonio Andelic
a6c4204b0c
Don't iterate all cached nodes on commit
2022-11-03 07:36:25 +00:00
JackyWoo
c7a0ebeb05
little fix
2022-10-25 17:46:24 +08:00
JackyWoo
9a36a509fe
fix test
2022-10-25 17:15:49 +08:00
JackyWoo
b5d1c4e657
replace snpd with lgif
2022-10-24 20:08:58 +08:00
JackyWoo
2f30c817bf
little fix
2022-10-24 17:23:47 +08:00
JackyWoo
39c88c74e8
check whether last manually created snapshot is done
2022-10-22 22:31:17 +08:00
JackyWoo
bf291790b7
Merge branch 'master' into manual_snapshot_for_keeper
2022-10-22 16:57:50 +08:00
Azat Khuzhin
905a95e166
Review fixes
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 22:40:13 +02:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Antonio Andelic
540728ab96
Simplify logic a bit
2022-10-18 10:58:25 +00:00
Antonio Andelic
1275c464aa
Fix unit test build
2022-10-18 10:44:22 +00:00
Antonio Andelic
8157fd0595
fix build
2022-10-17 14:16:56 +00:00
Antonio Andelic
238f86d892
Small refactoring
2022-10-17 11:06:33 +00:00
Antonio Andelic
7c7395575f
Merge branch 'master' into keeper-upload-snapshot-to-s3
2022-10-17 11:02:36 +00:00
Robert Schulze
da5a2e2db0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup
...
Physical merge conflicts:
- src/Common/ZooKeeper/ZooKeeperImpl.cpp
- src/Core/config_core.h.in
- src/Functions/FunctionsAES.h
- src/Functions/config_functions.h.in
- src/configure_config.cmake
Logical merge conflicts:
- Functions/tryDecrypt.cpp
2022-10-06 08:43:25 +00:00
Antonio Andelic
94f1fe39bb
Add support for exists in multiread
2022-09-29 10:39:33 +00:00
Antonio Andelic
3109ce51c6
Merge branch 'master' into keeper-multiread
2022-09-29 06:50:49 +00:00
Antonio Andelic
349bd7fd9a
Dont fail MultiRead on first failed op
2022-09-29 06:50:28 +00:00
Robert Schulze
f24fab7747
Fix some #include atrocities
2022-09-28 13:49:28 +00:00
Robert Schulze
fd86829824
Consolidate config_core.h into config.h
...
Less duplication, less confusion ...
2022-09-28 13:31:57 +00:00
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
