thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-02 12:32:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
146,365 Commits 9,194 Branches 1,782 Tags 2.2 GiB
8a202d91ad
Commit Graph

1360 Commits

Author SHA1 Message Date
serxa
6a2534b555 fix keeper build #2 2023-09-10 17:43:04 +00:00
serxa
d326554caa fix keeper build 2023-09-10 14:55:40 +00:00
Antonio Andelic
4a1e2959fa Apply PR comments 2023-09-08 12:52:35 +00:00
Antonio Andelic
5f1a75ceae Fix tests 2023-09-01 12:13:33 +00:00
Antonio Andelic
7739dd7157 Fix integration tests 2023-08-31 14:52:42 +00:00
Antonio Andelic
dd824fe558 Merge branch 'master' into keeper-batch-flushes 2023-08-31 14:35:36 +00:00
Antonio Andelic
4137cb916d Merge branch 'master' into keeper-batch-flushes 2023-08-28 13:14:06 +00:00
alesapin
0e601b856f
Merge pull request #53751 from Algunenano/clang18 
Support clang-18 (Wmissing-field-initializers)
 2023-08-25 23:27:06 +02:00
Pengyuan Bian
c82cc148e9 Add configuration for reconnect limit in raft limit. 2023-08-25 00:38:49 +00:00
Raúl Marín
6490213207 Merge remote-tracking branch 'blessed/master' into clang18 2023-08-24 17:35:26 +02:00
Arthur Passos
2bade7db08
Add global proxy setting (#51749) 
* initial impl

* fix env ut

* move ut directory

* make sure no null proxy resolver is returned by ProxyConfigurationResolverProvider

* minor adjustment

* add a few tests, still incomplete

* add proxy support for url table function

* use proxy for select from url as well

* remove optional from return type, just returns empty config

* fix style

* style

* black

* ohg boy

* rm in progress file

* god pls don't let me kill anyone

* ...

* add use_aws guards

* remove hard coded s3 proxy resolver

* add concurrency-mt-unsafe

* aa

* black

* add logging back

* revert change

* imrpove code a bit

* helper functions and separate tests

* for some reason, this env test is not working..

* formatting

* :)

* clangtidy

* lint

* revert some stupid things

* small test adjusmtments

* simplify tests

* rename test

* remove extra line

* freaking style change

* simplify a bit

* fix segfault & remove an extra call

* tightly couple proxy provider with context..

* remove useless include

* rename config prefix parameter

* simplify provider a bit

* organize provider a bit

* add a few comments

* comment out proxy env tests

* fix nullptr in unit tests

* make sure old storage proxy config is properly covered without global context instance

* move a few functions from class to anonymous namespace

* fix no fallback for specific storage conf

* change API to accept http method instead of bool

* implement http/https distinction in listresolver, any still not implemented

* implement http/https distinction in remote resolver

* progress on code, improve tests and add url function working test

* use protcol instead of method for http and https

* small fix

* few more adjustments

* fix style

* black

* move enum to proxyconfiguration

* wip

* fix build

* fix ut

* delete atomicroundrobin class

* remove stale include

* add some tests.. need to spend some more time on the design..

* change design a bit

* progress

* use existing context for tests

* rename aux function and fix ut

* ..

* rename test

* try to simplify tests a bit

* simplify tests a bit more

* attempt to fix tests, accept more than one remote resolver

* use proper log id

* try waiting for resolver

* proper wait logic

* black

* empty

* address a few comments

* refactor tests

* remove old tests

* baclk

* use RAII to set/unset env

* black

* clang tidy

* fix env proxy not respecting any

* use log trace

* fix wrong logic in getRemoteREsolver

* fix wrong logic in getRemoteREsolver

* fix test

* remove unwanted code

* remove ClientConfigurationperRequest and auxilary classes

* remove unwanted code

* remove adapter test

* few adjustments and add test for s3 storage conf  with new proxy settings

* black

* use chassert for context

* Add getenv comment
 2023-08-24 16:07:26 +03:00
Raúl Marín
93dac0c880 Support clang-18 (Wmissing-field-initializers) 2023-08-23 15:53:45 +02:00
Antonio Andelic
625be86dc0 Fix tests 2023-08-23 09:00:57 +00:00
Antonio Andelic
59557ddd7c Merge branch 'master' into keeper-batch-flushes 2023-08-23 08:11:43 +00:00
Alexander Tokmakov
894ec8e0d0 less exceptions with runtime format string 2023-08-14 21:45:59 +02:00
Alexey Milovidov
aae35f3b1c
Merge pull request #53184 from ClickHouse/better-ccache 
Better usage of ccache
 2023-08-11 04:43:45 +03:00
Alexey Milovidov
ae6e59c4c5
Merge pull request #53099 from ClickHouse/remove-wikistat-loader 
Remove old util
 2023-08-10 17:01:57 +03:00
Alexey Milovidov
fa9abc5038 Better usage of ccache 2023-08-09 05:02:50 +02:00
Alexey Milovidov
aa757490bd Ditch tons of garbage 2023-08-09 02:19:02 +02:00
Alexey Milovidov
1ab99e5fdd Replace old docs with symlinks 2023-08-08 22:38:52 +02:00
Antonio Andelic
1dbe007ee5 Add coordination setting for async replication 2023-08-08 13:12:07 +00:00
Antonio Andelic
4b3e399f7d Wait for read requests 2023-08-07 13:04:38 +00:00
Antonio Andelic
c0ff486ada Merge branch 'master' into keeper-batch-flushes 2023-08-07 06:20:41 +00:00
Antonio Andelic
5cb856d167 Try batching multiple flush requests in Keeper 2023-08-04 11:53:22 +00:00
Vitaly Baranov
552331fd54 Use the same default paths for clickhouse_keeper (symlink) as for clickhouse_keeper (executable), 
i.e. "/var/lib/clickhouse-keeper/.."
 2023-08-04 13:38:21 +02:00
Anton Popov
92cb5e9ae3
Merge pull request #52687 from alkorgun/fix-named-collections-on-cluster-23.7 
Fix named collections on cluster 23.7
 2023-08-03 14:50:28 +02:00
Antonio Andelic
a6a495a767
Merge branch 'master' into fix-keeper-reductant-lock 2023-07-31 19:53:25 +02:00
Antonio Andelic
a473dc4b51 Wait for response 2023-07-31 09:24:06 +00:00
Илья Коргун
bcc41fbc2f Fix reference to a local binding declared in an enclosing function 2023-07-27 10:46:35 +03:00
Alexey Milovidov
21382afa2b Check for punctuation 2023-07-25 06:10:04 +02:00
Antonio Andelic
7b03f3277e
Try fix session on close test 2023-07-24 17:59:50 +02:00
Antonio Andelic
73155cf562
Merge branch 'master' into fix-keeper-reductant-lock 2023-07-24 10:17:35 +02:00
Antonio Andelic
10c2d5dade
Merge branch 'master' into feature/keeper-dyn-reconf 2023-07-21 13:24:18 +02:00
Antonio Andelic
4a47040763
Merge pull request #52358 from ClickHouse/fix-keeper-apply-snapshot 
Fix `apply_snapshot` in Keeper
 2023-07-21 12:52:12 +02:00
Antonio Andelic
f997adfe27 Retry if sessions not closed because missing leader 2023-07-20 14:02:55 +00:00
Antonio Andelic
a24bf14450 Use correct ZXID 2023-07-20 10:44:44 +00:00
Antonio Andelic
0c86df519f Fix unspported disks in Keeper 2023-07-19 12:41:25 +00:00
Antonio Andelic
61dc020b4d
Merge branch 'master' into feature/keeper-dyn-reconf 2023-07-17 12:50:17 +02:00
Antonio Andelic
9677cc5e2b
Apply suggestions from code review 2023-07-17 12:50:02 +02:00
frinkr
e1abdd0419
atomic exchange simplified 2023-07-17 10:51:24 +08:00
frinkr
6dceedd721
Merge branch 'ClickHouse:master' into fix-keeper-reductant-lock 2023-07-17 10:21:05 +08:00
Azat Khuzhin
6abdde12ca Print Zxid in keeper stat command in hex (so as ZooKeeper) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-14 21:17:09 +02:00
Alexey Milovidov
b1d1918607
Merge branch 'master' into retry 2023-07-08 03:08:48 +03:00
Alexey Milovidov
685f2949b7
Revert "Fix build" 2023-07-07 02:53:13 +03:00
Alexey Milovidov
c47b32b17a Fix build 2023-07-07 01:52:18 +02:00
Alexey Milovidov
48eb30de51 Fix build 2023-07-07 01:42:29 +02:00
Mike Kot
5302b478a4 proper reconfig batch handling 2023-07-07 00:20:54 +03:00
Mike Kot
bafcc3afdc remove reconfig in process flag as it's useless 2023-07-07 00:20:54 +03:00
Mike Kot
c46b125d0a review fixes 2023-07-07 00:20:54 +03:00
Mike Kot
2f0cd05497 handle leader removal corner cases 2023-07-07 00:20:54 +03:00
Mike Kot
1bef6fc76c process reconfig in keeper dispatcher 2023-07-07 00:20:54 +03:00
Mike Kot
297d566600 acl check 2023-07-07 00:20:54 +03:00
Mike Kot
b4f750ed66 review fixes 2023-07-07 00:20:54 +03:00
Mike Kot
8b6376005a "reconfig" support for CH Keeper 2023-07-07 00:20:54 +03:00
frinkr
36fbc22b16
Merge branch 'master' into fix-keeper-reductant-lock 2023-07-04 18:04:08 +08:00
Jiang Yuqing
f443a00d14
KeeperDispatcher: remove reductant lock as the ConcurrentBoundedQueue is thread-safe 2023-07-04 17:58:04 +08:00
Antonio Andelic
2058b1346e Merge branch 'master' into keeper-with-disks 2023-06-30 13:22:33 +00:00
Antonio Andelic
eff1cc0e5d
Remove unused errorcode 2023-06-28 10:20:33 +02:00
Antonio Andelic
be852d554b Use magic_enum for values 2023-06-28 07:45:20 +00:00
Antonio Andelic
efbe01b6bb Merge branch 'master' into keeper-add-api-version-config 2023-06-27 10:11:04 +00:00
Alexey Milovidov
7407330130
Merge branch 'master' into retry 2023-06-23 08:18:18 +03:00
Michael Kolupaev
4a570a05c9 Decrease default timeouts for S3 and HTTP requests 2023-06-21 18:08:50 +00:00
Sema Checherinda
1cb02e2710 do call finalize for all buffers 2023-06-16 16:38:18 +02:00
Antonio Andelic
4ac090b122 Fix tests 2023-06-13 11:01:44 +00:00
Antonio Andelic
6f1c486537
Fix build 2023-06-13 10:25:53 +02:00
Antonio Andelic
25f08f8d19
Remove duplicate include 2023-06-12 15:35:44 +02:00
Antonio Andelic
cb3ac81110
Merge branch 'master' into keeper-with-disks 2023-06-12 15:14:24 +02:00
Antonio Andelic
a7e6264d56 Add backward compatibility 2023-06-12 13:12:05 +00:00
Antonio Andelic
b634012feb Add feature flags for Keeper API 2023-06-12 10:57:03 +00:00
Antonio Andelic
91a4b6e331 Merge branch 'master' into keeper-add-api-version-config 2023-06-12 07:10:58 +00:00
Antonio Andelic
b8b70d78e1 Build fix 2023-06-09 13:34:27 +00:00
Antonio Andelic
07582d56f3 Add config for pinning api version 2023-06-09 13:18:13 +00:00
Antonio Andelic
ab8365630b Reject logs containing unknown operation 2023-06-09 10:10:40 +00:00
Antonio Andelic
f4d182ed67
Merge branch 'master' into keeper-operation-create-if-not-exists 2023-06-06 09:19:35 +02:00
Antonio Andelic
ece96f54e9 Fix tests 2023-06-05 09:47:58 +00:00
Antonio Andelic
c4873027c3 Merge branch 'master' into keeper-with-disks 2023-06-05 07:08:33 +00:00
Alexey Milovidov
bed19aa792
Merge pull request #50483 from ClickHouse/fix-keeper-snapshot-install 
Keeper fix: apply uncommitted state after snapshot install
 2023-06-05 04:39:33 +03:00
Antonio Andelic
47966c4e81 Finalize bufferg 2023-06-02 14:10:52 +00:00
Antonio Andelic
336c9d7136 Apply uncommitted state after snapshot deser 2023-06-02 13:31:14 +00:00
Antonio Andelic
ffd4f7f196 Add better support for state disks 2023-06-01 14:39:01 +00:00
Antonio Andelic
e8442b7a88 Add finalize calls 2023-06-01 12:54:00 +00:00
frinkr
c9d0d217f5 fix Keeper deadlock on exception when preprocessing requests 2023-06-01 09:57:42 +08:00
Konstantin Bogdanov
b61ffe3ff7
Merge checks 2023-06-01 03:22:40 +02:00
Konstantin Bogdanov
3a3539b996
Style fix 2023-06-01 03:22:40 +02:00
Konstantin Bogdanov
203276dfcc
Use storage.uncommited_state 2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
965f7850f8
Cleanup 2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
f84fdb7f10
Increment Keeper API version 2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
b53f36369e
Remove new request object 2023-06-01 03:22:39 +02:00
Konstantin Bogdanov
0d6561ee77
Prototype 2023-06-01 03:22:23 +02:00
Antonio Andelic
21aba94909 Add support for latest snapshot disk 2023-05-30 13:22:40 +00:00
Antonio Andelic
67746bfff2 Merge branch 'master' into keeper-with-disks 2023-05-30 09:50:12 +00:00
Raúl Marín
1ae1f1273a Avoid clang 15 crash 2023-05-29 11:40:45 +02:00
Antonio Andelic
0740bfbe4b Better disks moving 2023-05-26 14:31:09 +00:00
Antonio Andelic
a9569478e6
fix typo 2023-05-26 13:09:09 +02:00
Antonio Andelic
8a2a63a7bd Merge branch 'master' into keeper-with-disks 2023-05-26 10:48:41 +00:00
Antonio Andelic
3d98e591ba Handle backwards compatibility 2023-05-26 10:46:18 +00:00
Antonio Andelic
767193caf5 Merge branch 'master' into keeper-log-improvements 2023-05-26 07:35:36 +00:00
Antonio Andelic
6a8a21e09a Add test for s3 disk restore 2023-05-25 16:01:40 +00:00
Antonio Andelic
bb77441acb Add support for old storage disks 2023-05-25 13:31:11 +00:00
Antonio Andelic
f519aa4613 Calculate size 2023-05-25 11:23:03 +00:00
Antonio Andelic
4483602c50 Remove double initialization 2023-05-25 08:52:14 +00:00
Antonio Andelic
7e68f61df1 Fix snapshot recovery 2023-05-25 08:43:11 +00:00
Antonio Andelic
092cf99147 Fix build 2023-05-24 13:45:14 +00:00
Antonio Andelic
fa3fb49d33 Merge branch 'master' into keeper-with-disks 2023-05-24 09:04:47 +00:00
Antonio Andelic
5db2160762 Fix build 2023-05-24 09:04:12 +00:00
Antonio Andelic
161afea266 Add support for changelog 2023-05-24 07:43:57 +00:00
kssenii
295fe3b228 Merge remote-tracking branch 'upstream/master' into abstract-async-prefetched-buffer 2023-05-22 20:01:38 +02:00
kssenii
c4d862a16f Make async reader work with any impl 2023-05-22 19:54:04 +02:00
Antonio Andelic
bde2cf96b1 Better 2023-05-22 12:24:16 +00:00
alesapin
cc3897a84a
Merge pull request #50033 from kssenii/disk-object-storage-minor-changes 
Get rid of indirect write buffer in object storages
 2023-05-22 14:08:17 +02:00
Antonio Andelic
dc31396035 Merge branch 'master' into keeper-with-disks 2023-05-22 08:10:24 +00:00
kssenii
3e42ee7f2b Get rid of finalize callback in object storages 2023-05-19 17:29:37 +02:00
Antonio Andelic
9c3b17fa18
Remove whitespace 2023-05-19 13:00:51 +02:00
Antonio Andelic
e46476dba2
Update src/Coordination/Changelog.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2023-05-19 12:44:20 +02:00
Antonio Andelic
6e468b29e8 Check return value of ftruncate 2023-05-19 10:15:06 +00:00
Antonio Andelic
bf6afd27a6 build with disks 2023-05-18 07:39:22 +00:00
Antonio Andelic
465123fa1b Merge branch 'master' into keeper-with-disks 2023-05-16 08:07:06 +00:00
Azat Khuzhin
665545ec45 Fix "reference to local binding" after fixes for clang-17 
Follow-up for: #49851 (cc @alexey-milovidov)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-15 12:45:20 +02:00
Alexey Milovidov
5a44dc26e7 Fixes for clang-17 2023-05-13 02:57:31 +02:00
Antonio Andelic
bb8b9dc9c3 Merge branch 'master' into keeper-log-improvements 2023-05-05 11:57:14 +00:00
alesapin
9456eb59fd
Merge pull request #48466 from ClickHouse/keeper-correctly-handle-snapshots 
Correctly handle concurrent snapshots in Keeper
 2023-05-04 19:16:00 +02:00
johanngan
731823b873 Add support for SSE-KMS configuration with S3 
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html

Similar to the server_side_encryption_customer_key_base64 option for
configuring SSE-C with S3, add the following settings to configure
SSE-KMS on a per-endpoint/disk basis:
  - server_side_encryption_kms_key_id
  - server_side_encryption_kms_encryption_context
  - server_side_encryption_kms_bucket_key_enabled
 2023-05-03 21:35:38 -05:00
Antonio Andelic
b8b7a362b2
Merge branch 'master' into keeper-correctly-handle-snapshots 2023-05-03 13:51:09 +02:00
Antonio Andelic
78ecfac8d5 Merge branch 'master' into keeper-log-improvements 2023-05-03 08:31:21 +00:00
Antonio Andelic
dce61d7321 Merge branch 'master' into keeper-log-improvements 2023-05-02 11:14:48 +00:00
Antonio Andelic
f695040a19
Merge pull request #48897 from ClickHouse/keeper-check-not-exists 
Add `CheckNotExists` request to Keeper
 2023-05-02 13:14:22 +02:00
Raúl Marín
f0e045bb3d Merge remote-tracking branch 'blessed/master' into arenita 2023-04-24 10:42:56 +02:00
Alexey Milovidov
530b764953 Fix IBM 2023-04-21 12:38:45 +02:00
Antonio Andelic
d500a5637d Cache large requests 2023-04-21 09:43:04 +00:00
Antonio Andelic
b52646c555 Eliminate double log serializing 2023-04-21 08:20:35 +00:00
Antonio Andelic
58e9b56fcb Fix CheckNotExists 2023-04-19 09:06:20 +00:00
Antonio Andelic
fb16623d48 Add CheckNotExists request to Keeper 2023-04-18 14:45:24 +00:00
Raúl Marín
2b70e08f23 Don't count unreserved bytes in Arenas as read_bytes 2023-04-13 12:43:24 +02:00
Antonio Andelic
b30aff06d8
Merge pull request #48595 from aalexfvk/alexfvk/fix_create_not_auth_problem 
Fix bug in Keeper when a node is not created with scheme `auth` in ACL sometimes.
 2023-04-11 14:41:37 +02:00
Aleksei Filatov
dd2364361c
Update src/Coordination/tests/gtest_coordination.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-04-11 10:45:30 +03:00
Azat Khuzhin
79b83c4fd2 Remove superfluous includes of logger_userful.h from headers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-10 17:59:30 +02:00
Aleksei Filatov
e8bf96d126 Add unit tests 2023-04-10 13:02:43 +03:00
Aleksei Filatov
000c2b5b91 Correct FixupACL for auth scheme 2023-04-10 13:02:31 +03:00
Azat Khuzhin
011480924a Use forward declaration of ThreadPool 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 11:25:35 +02:00
Antonio Andelic
88f77c686d Correctly handle concurrent snapshots 2023-04-06 15:40:05 +00:00
Antonio Andelic
e982f2a67a Merge branch 'master' into enable-env-credentials-default 2023-03-31 09:11:01 +00:00
Antonio Andelic
44e95aa65f Merge branch 'master' into nosign-keyword-for-s3 2023-03-29 11:10:03 +00:00
Antonio Andelic
160aa186bb Add support for NOSIGN keyword and no_sign_request config 2023-03-28 07:05:35 +00:00
Antonio Andelic
79e22d13fb small fix 2023-03-27 07:41:31 +00:00
Antonio Andelic
f355bc5342 Merge branch 'master' into more-batching-keeper 2023-03-27 07:28:17 +00:00
Antonio Andelic
6c3765c8b6 Small fix 2023-03-24 20:40:45 +00:00
Antonio Andelic
f353561204 Avoid breaking batches with read requests 2023-03-24 14:37:40 +00:00
Alexander Gololobov
afe3af230f Limit keeper request batching by size in bytes 2023-03-24 14:41:09 +01:00
Antonio Andelic
51416da818 Merge branch 'master' into enable-env-credentials-default 2023-03-13 10:20:21 +00:00
Antonio Andelic
a170a909a4 Add expiration window for S3 credentials 2023-03-10 10:06:32 +00:00
Antonio Andelic
83a3243758
Merge pull request #47065 from ClickHouse/debug-keeper 
Fix Keeper with `force_sync = false`
 2023-03-10 09:27:13 +01:00
Antonio Andelic
5bc21538e5 Enable use_environment_credentials by default 2023-03-09 10:31:55 +00:00
Antonio Andelic
0ad436aa2c Revert debug changes 2023-03-09 08:07:59 +00:00
Antonio Andelic
b4061d8aa4 Add logs 2023-03-08 13:06:10 +00:00
Mike Kot
9920a52c51 use std::lerp, constexpr hex.h 2023-03-07 22:50:17 +00:00
Antonio Andelic
dd4db9132a
Merge branch 'master' into debug-keeper 2023-03-07 13:15:05 +01:00
Antonio Andelic
c38d820d0f Add 4LW for cleaning resource 2023-03-05 16:57:18 +00:00
Antonio Andelic
cd4100cb19
Merge branch 'master' into fix-data-size 2023-03-03 15:33:43 +01:00
Antonio Andelic
fdcb5b784f
Merge branch 'master' into debug-keeper 2023-03-03 13:45:13 +01:00
Robert Schulze
740aeaba1f
Apply some CTAD 2023-03-02 13:36:47 +00:00
Antonio Andelic
6ebee202bd Flush buffer when no sync 2023-03-01 14:47:15 +00:00
Antonio Andelic
c797122dbe Better 2023-03-01 08:45:50 +00:00
Antonio Andelic
6e28406751 Merge branch 'master' into fix-data-size 2023-02-28 20:09:29 +00:00
Alexey Milovidov
d8cda3dbb8 Remove PVS-Studio 2023-02-19 23:30:05 +01:00
Alexander Tokmakov
e96340e183
Merge pull request #46410 from ClickHouse/fix_noisy_log_messages 
Fix noisy log messages
 2023-02-15 01:22:42 +03:00
Alexander Tokmakov
1bf3735ca9 fix noisy log messages 2023-02-14 15:33:27 +01:00
Sema Checherinda
9bef1bec28
Merge pull request #46274 from aiven-sal/aiven-sal/keeper 
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms
 2023-02-14 12:44:04 +01:00
Sema Checherinda
5882f3878f
except zero values election_timeout 2023-02-13 12:22:08 +01:00
Salvatore Mesoraca
fff2c09b87
coordination: do not allow election_timeout_lower_bound_ms > election_timeout_upper_bound_ms 2023-02-13 10:09:45 +01:00
Antonio Andelic
f96d480563
Merge branch 'master' into add-support-for-settings-alias 2023-02-09 16:07:45 +01:00
Azat Khuzhin
8cc41b7f41 Check return value of ::close() 
Note, that according close(2), EINTR should not be retriable for close()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-07 11:28:22 +01:00
Antonio Andelic
b11458f142 Fix erase 2023-02-07 07:51:22 +00:00
Antonio Andelic
9cde46acff Correctly calculate size on update 2023-02-06 16:28:05 +00:00
Antonio Andelic
aca4f08bf5 Correctly calculate size after snapshot load 2023-02-06 14:59:23 +00:00
Antonio Andelic
8929838fcc Merge branch 'master' into add-support-for-settings-alias 2023-02-03 17:47:00 +00:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution (#45783) 
* Update aws

* Define S3 client with bucket and endpoint resolution

* Add defines for ErrorCodes

* Use S3Client everywhere

* Remove unused errorcode

* Add DROP S3 CLIENT CACHE query

* Add a comment

* Fix style

* Update aws

* Update reference files

* Add missing include

* Fix unit test

* Remove unneeded declarations

* Correctly use RetryStrategy

* Rename S3Client to Client

* Fix retry count

* fix clang-tidy warnings
 2023-02-03 14:30:52 +01:00
LiuYangkuan
f29700bd2f use IDisk to do IO in Keeper's snapshots and logs 2023-02-02 19:47:30 +08:00
Antonio Andelic
f613bfd8d2 Merge branch 'master' into add-support-for-settings-alias 2023-02-02 11:15:39 +00:00
Antonio Andelic
85cfee4bb9 Better alias definition 2023-02-01 13:54:03 +00:00
Antonio Andelic
88c1744975
Better 2023-02-01 14:25:58 +01:00
Antonio Andelic
5de6f86c27 Fix clang-tidy error 2023-02-01 12:28:44 +00:00
Antonio Andelic
ec04cf6876 Merge branch 'master' into add-support-for-settings-alias 2023-02-01 09:01:01 +00:00
Antonio Andelic
ece92fa866
Update src/Coordination/Changelog.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2023-01-31 17:11:18 +01:00
Antonio Andelic
8457d3bf3d Apply comments 2023-01-31 14:18:54 +00:00
Antonio Andelic
1786237be6 Merge branch 'master' into keeper-use-fallocate 2023-01-31 13:53:47 +00:00
Vitaly Baranov
f9282cfcac Make version_id an optional parameter again. 2023-01-27 15:10:19 +01:00
Vitaly Baranov
aea9ccdb60 Pass request settings to S3::getObjectInfo(). 2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed Move getObjectInfo() to a separate header. 2023-01-27 15:09:38 +01:00
Antonio Andelic
714fad1529 Add support for settings alias 2023-01-26 14:06:46 +00:00
Antonio Andelic
ebfe7f9991 Fix style 2023-01-26 11:18:07 +00:00
Antonio Andelic
1186718658 Add overallocate size setting 2023-01-26 09:44:37 +00:00
Antonio Andelic
7bfdfa4bf7 Preallocate more space 2023-01-25 16:26:14 +00:00
Antonio Andelic
7421d8f872 Merge branch 'master' into keeper-use-fallocate 2023-01-25 11:36:47 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Alexander Tokmakov
870cfcc36a less fmt::runtime usages 2023-01-17 00:11:59 +01:00
Antonio Andelic
6a0e481ed7 Merge branch 'master' into keeper-use-fallocate 2023-01-12 13:57:05 +00:00
Antonio Andelic
30510c5eb6
Merge pull request #44937 from ClickHouse/fix-changelog-deadlock 
Fix deadlock in Keeper's changelog
 2023-01-10 08:59:51 +01:00
Antonio Andelic
22679ddf11
Add comment 2023-01-09 14:26:20 +01:00
Antonio Andelic
a39f437306
Typo fix 2023-01-09 13:34:48 +01:00
Antonio Andelic
27b2bae640 make it better 2023-01-09 12:01:22 +00:00
Antonio Andelic
4fcbe14435 Small fixes for Coordination unit tests 2023-01-08 13:44:32 +00:00
Kseniia Sumarokova
573d3283b0
Merge pull request #44327 from kssenii/use-new-named-collections-code-2 
Replace old named collections code with new (from #43147) part 2
 2023-01-06 13:06:26 +01:00
Antonio Andelic
9b3314a931 Fix deadlock in Changelog flush 2023-01-05 14:51:05 +00:00
Antonio Andelic
136436958b
Merge pull request #44908 from ClickHouse/fix-keeper-snapshot-on-exit 
Avoid Keeper crash on shutdown (fix `test_keeper_snapshot_on_exit`)
 2023-01-05 08:59:58 +01:00
Antonio Andelic
7c2cdfa65a Don't throw exception if queue is finished 2023-01-04 15:46:38 +00:00
kssenii
67509aa2d5 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2023-01-03 16:41:30 +01:00
Smita Kulkarni
8a5fd7f96d Added comments and addressed review comments - Allow macro substitution in endpoint 2023-01-03 11:44:26 +01:00
Antonio Andelic
f2653c016a Manually set deleted flag 2023-01-02 14:36:58 +00:00
Antonio Andelic
f32fab5025 Check if file exists before rename 2023-01-02 13:48:13 +00:00
Antonio Andelic
2e93640614 Fix ASAN error 2023-01-02 13:37:40 +00:00
Antonio Andelic
b845f880fb Polish 2023-01-02 10:42:15 +00:00
Antonio Andelic
14355d8009 Merge branch 'master' into keeper-use-fallocate 2023-01-02 08:22:27 +00:00
Antonio Andelic
55548a6678 style fix 2023-01-02 08:22:00 +00:00
SmitaRKulkarni
f9a3aeb172
Merge branch 'master' into 40951_Allow_macro_substitution_in_endpoint 2022-12-30 10:43:14 +01:00
Alexey Milovidov
f0fa0977ed Fix wrong log messages 2022-12-26 18:36:07 +01:00
Smita Kulkarni
3d62bc2635 Allow macro substitution in endpoint of disks 
Implementation:
* Updated to substitute macros in endpoint for all disks (s3, hfs, web).
Testing:
* Added an integration test test/integration/test_endpoint_macro_substitution.
 2022-12-23 15:09:24 +01:00
Antonio Andelic
64ade259cf Fix tests 2022-12-22 10:20:55 +00:00
Antonio Andelic
88d41dea93 Merge branch 'master' into keeper-use-fallocate 2022-12-21 08:59:09 +00:00
Antonio Andelic
9711355cc9 Fix tests 2022-12-21 08:43:55 +00:00
kssenii
6bd4f8c029 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2022-12-20 21:17:28 +01:00
Antonio Andelic
6d83221d4e Remove unused error code 2022-12-19 12:20:26 +00:00
Antonio Andelic
c2c115d2ba Add setting for max_log_file_size 2022-12-19 11:51:05 +00:00
Antonio Andelic
f241701c2a Add checks when fallocate is not supported 2022-12-19 11:48:54 +00:00
Antonio Andelic
9a8b248735 Merge branch 'master' into keeper-use-fallocate 2022-12-19 09:32:05 +00:00
Nikolai Kochetov
62ff98344e Validate s3 part upload settings. 2022-12-17 14:09:53 +00:00
kssenii
30547d2dcd Replace old named collections code for url 2022-12-17 00:24:05 +01:00
Antonio Andelic
ee900ca7df Add truncate 2022-12-15 13:09:52 +00:00
Antonio Andelic
bf6f163952 Merge branch 'master' into keeper-use-fallocate 2022-12-15 09:05:40 +00:00
Antonio Andelic
d454e00016 Changelog refactor 2022-12-15 08:12:02 +00:00
Anton Popov
0c87031e80 Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-13 16:33:21 +00:00
Antonio Andelic
662c1f956c Changelog refactoring 2022-12-13 14:00:27 +00:00
Antonio Andelic
898cf867af Try with fallocate 2022-12-13 11:08:59 +00:00
Vitaly Baranov
b91af1b650 Fix initialization of s3 request settings. 2022-12-10 05:43:51 +01:00
Antonio Andelic
e1a6a1182e
Throw when port already in use (#43984) 2022-12-07 13:27:49 +01:00
Antonio Andelic
200141da13 Use logger in tests 2022-12-02 08:16:18 +00:00
Antonio Andelic
7cd76df918 Merge branch 'master' into parallel-log-appending 2022-12-01 14:44:36 +00:00
Antonio Andelic
554ff1db6f Small changes 2022-12-01 12:54:29 +00:00
Antonio Andelic
940fdd4abd
Remove comment 2022-12-01 12:24:27 +01:00
Antonio Andelic
25ae52111e Protect writer with mutex 2022-12-01 11:18:23 +00:00
Antonio Andelic
c12c0cc8af Merge branch 'master' into parallel-log-appending 2022-12-01 08:53:52 +00:00
Antonio Andelic
2f501d5327 Merge branch 'master' into keeper-prometheus 2022-11-30 13:24:37 +01:00
Antonio Andelic
fbda86ddc8 PR fixes 2022-11-30 13:24:08 +01:00
Antonio Andelic
7f897f1a20 Refactor 2022-11-29 13:00:13 +00:00
Antonio Andelic
c8b2cb5650
Update src/Coordination/CoordinationSettings.h 
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
 2022-11-29 10:08:25 +01:00
Antonio Andelic
678958603b
Merge branch 'master' into keeper-prometheus 2022-11-28 09:48:57 +01:00
Antonio Andelic
fe44ac0c86 Merge branch 'master' into parallel-log-appending 2022-11-28 08:45:20 +00:00
Antonio Andelic
e91314d57f Use initialized flag 2022-11-28 08:44:46 +00:00
Antonio Andelic
cade21b76e Fix 4LW test 2022-11-26 19:24:10 +00:00
Antonio Andelic
76eca8bdfb Merge branch 'master' into more-aggresive-batching-keeper 2022-11-26 18:28:11 +00:00
Antonio Andelic
cd471e9c61 Add setting for quick batch size 2022-11-26 17:33:40 +00:00
Anton Popov
65a78bcd91 improve performance of storage S3 2022-11-26 15:24:01 +00:00
Antonio Andelic
7437c719a7 Try batching requests more aggresively 2022-11-25 17:25:28 +00:00
Antonio Andelic
38d99c241f Fix tests 2022-11-25 13:53:03 +00:00
Antonio Andelic
53f708366c Fix data race in Keeper snapshot 2022-11-25 12:50:42 +00:00
Antonio Andelic
ce532cd12d Fix circular deps 2022-11-24 13:43:05 +00:00
Antonio Andelic
a6f38cb1cc
Merge branch 'master' into keeper-prometheus 2022-11-24 12:12:08 +01:00
Antonio Andelic
0cc53995df
Merge branch 'master' into parallel-log-appending 2022-11-22 15:05:08 +01:00
Kseniia Sumarokova
5c90d5aa7e
Merge pull request #43253 from xiedeyantu/fix-s3-glob 
fix s3 support question mark wildcard
 2022-11-22 14:26:56 +01:00
Antonio Andelic
4827bcb6bb
Merge branch 'master' into parallel-log-appending 2022-11-22 14:14:17 +01:00
Antonio Andelic
55ef96dc11 Fix unit tests 2022-11-22 11:37:59 +00:00
Antonio Andelic
aeb0d39b50 Fix shutdown 2022-11-21 15:01:18 +00:00
Antonio Andelic
c9472d67b6 Refactor changelog 2022-11-21 12:22:36 +00:00
Antonio Andelic
ff681d5faa Merge branch 'master' into parallel-log-appending 2022-11-21 11:59:16 +00:00
Antonio Andelic
3cb202a63b
Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader 
Keeper support manually assigning leader
 2022-11-21 09:30:00 +01:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus 2022-11-21 09:10:10 +01:00
xiedeyantu
c258d3ac8b fix s3 support question mark wildcard 2022-11-18 12:11:22 +08:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling"" 2022-11-17 17:35:04 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling" 2022-11-16 22:33:48 +03:00
Sergei Trifonov
159743edd6
Merge pull request #43014 from ClickHouse/disk-s3-throttler 
S3 request per second rate throttling
 2022-11-16 18:51:06 +01:00
JackyWoo
f8d71f60a2
Merge branch 'master' into keeper_manually_assign_leader 2022-11-16 09:34:40 +08:00
Antonio Andelic
96838f5be7 Merge branch 'master' into keeper-prometheus 2022-11-15 12:00:02 +00:00
Antonio Andelic
d08dce5ec7 Test parallel append 2022-11-14 13:17:05 +00:00
Raúl Marín
54db7c6520 Enforce checking read output 2022-11-11 10:56:18 +01:00
Antonio Andelic
94aab4276c Bring back heavy metrics 2022-11-10 12:56:27 +01:00
Antonio Andelic
8aeb406d82 Fix standalone build 2022-11-10 10:49:00 +01:00
Antonio Andelic
36d763011b Compile standalone keeper 2022-11-10 09:23:52 +01:00
Antonio Andelic
1ed3930809 Add more metrics for Keeper 2022-11-09 15:51:41 +01:00
Antonio Andelic
c92ae5a385 Add suport for AsyncMetrics in standalone Keeper 2022-11-09 13:46:03 +01:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler 2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421 rename ReadWriteSettings -> RequestSettings 2022-11-08 13:48:23 +00:00
JackyWoo
9060141f16 revert last change 2022-11-08 18:44:43 +08:00
JackyWoo
e865842098 Add rqld to white list 2022-11-08 16:03:13 +08:00
JackyWoo
eeedd7d9ba Keeper support manually assigning leader 2022-11-08 12:01:53 +08:00
serxa
2daec0b45e S3 request per second rate throttling + refactoring 2022-11-07 18:05:40 +00:00
Antonio Andelic
9ac829d4c4
Merge pull request #41766 from JackyWoo/manual_snapshot_for_keeper 
Manually snapshot creation for keeper
 2022-11-07 09:18:58 +01:00
JackyWoo
73dfe4acd7 Add to default white list 2022-11-05 17:56:55 +08:00
Antonio Andelic
25b19e6ba0
Merge branch 'master' into manual_snapshot_for_keeper 2022-11-03 08:55:01 +01:00
Antonio Andelic
664690f9a9 Add comments 2022-11-03 07:38:43 +00:00
Antonio Andelic
a6c4204b0c Don't iterate all cached nodes on commit 2022-11-03 07:36:25 +00:00
JackyWoo
c7a0ebeb05 little fix 2022-10-25 17:46:24 +08:00
JackyWoo
9a36a509fe fix test 2022-10-25 17:15:49 +08:00
JackyWoo
b5d1c4e657 replace snpd with lgif 2022-10-24 20:08:58 +08:00
JackyWoo
2f30c817bf little fix 2022-10-24 17:23:47 +08:00
JackyWoo
39c88c74e8 check whether last manually created snapshot is done 2022-10-22 22:31:17 +08:00
JackyWoo
bf291790b7
Merge branch 'master' into manual_snapshot_for_keeper 2022-10-22 16:57:50 +08:00
Azat Khuzhin
905a95e166 Review fixes 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 22:40:13 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Antonio Andelic
540728ab96 Simplify logic a bit 2022-10-18 10:58:25 +00:00
Antonio Andelic
1275c464aa Fix unit test build 2022-10-18 10:44:22 +00:00
Antonio Andelic
8157fd0595 fix build 2022-10-17 14:16:56 +00:00
Antonio Andelic
238f86d892 Small refactoring 2022-10-17 11:06:33 +00:00
Antonio Andelic
7c7395575f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-17 11:02:36 +00:00
Robert Schulze
da5a2e2db0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup 
Physical merge conflicts:
- src/Common/ZooKeeper/ZooKeeperImpl.cpp
- src/Core/config_core.h.in
- src/Functions/FunctionsAES.h
- src/Functions/config_functions.h.in
- src/configure_config.cmake

Logical merge conflicts:
- Functions/tryDecrypt.cpp
 2022-10-06 08:43:25 +00:00
Antonio Andelic
94f1fe39bb Add support for exists in multiread 2022-09-29 10:39:33 +00:00
Antonio Andelic
3109ce51c6 Merge branch 'master' into keeper-multiread 2022-09-29 06:50:49 +00:00
Antonio Andelic
349bd7fd9a Dont fail MultiRead on first failed op 2022-09-29 06:50:28 +00:00
Robert Schulze
f24fab7747
Fix some #include atrocities 2022-09-28 13:49:28 +00:00
Robert Schulze
fd86829824
Consolidate config_core.h into config.h 
Less duplication, less confusion ...
 2022-09-28 13:31:57 +00:00
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Antonio Andelic
d0457addbd Support filtered list 2022-09-27 15:02:26 +00:00
JackyWoo
d1c85f68e8 manually snapshot creation for keeper 2022-09-26 18:29:15 +08:00
Antonio Andelic
937d534cd4 Add support for simple list 2022-09-26 08:05:02 +00:00
Antonio Andelic
f833366555 Merge branch 'master' into keeper-multiread 2022-09-26 07:16:45 +00:00
Antonio Andelic
9da433a904 Fix unused 2022-09-22 14:33:35 +00:00
Antonio Andelic
44a3d6babe Address PR comments 2022-09-22 13:03:27 +00:00
Antonio Andelic
6798b500e9 Wait on startup for Keeper 2022-09-21 15:12:16 +00:00
Antonio Andelic
eefbbf53e8 Extract S3 logic 2022-09-21 11:53:54 +00:00
Antonio Andelic
fc6c1faa57 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-21 06:57:58 +00:00
Antonio Andelic
1763af37b9
Merge pull request #41403 from ClickHouse/keeper-fix-shutdown-without-server 
Fix Keeper segfault during shutdown when RAFT server failed to start
 2022-09-19 12:03:58 +02:00
alesapin
b6d61318d0 One more validation for broken config in keeper 2022-09-18 00:04:16 +02:00
Alexander Tokmakov
42dab6dc02 Revert "Merge pull request #38200 from ClickHouse/keeper-linearizable-reads" 
This reverts commit 222b964070, reversing
changes made to da01982652.
 2022-09-17 19:22:58 +02:00
Antonio Andelic
81f7cf3ca6 Add support for MultiRead in Keeper 2022-09-16 12:00:35 +00:00
Antonio Andelic
acd9cb089e Fix segfault during shutdown when RAFT server failed to start 2022-09-16 10:03:21 +00:00
Antonio Andelic
5662e0050c Define client for multi read 2022-09-16 09:55:02 +00:00
Antonio Andelic
296fe01fb8 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-16 08:00:15 +00:00
Antonio Andelic
39307f9dba Extract into function 2022-09-15 16:14:53 +00:00
Antonio Andelic
0dbd0c1f72 Merge branch 'master' into keeper-linearizable-reads 2022-09-15 16:03:18 +00:00
Antonio Andelic
ad7864cb5f enable S3 if client is built 2022-09-15 14:06:22 +00:00
Antonio Andelic
d3d1676352 Fix config update detection 2022-09-15 13:48:39 +00:00
Antonio Andelic
d4e4ac3801 Lock file during S3 snapshot upload 2022-09-15 09:11:55 +00:00
Antonio Andelic
9fdd2f2e61 Basic implementation for S3 snapshot upload 2022-09-15 09:11:55 +00:00
Antonio Andelic
3a9996f04c Add bg thread for snapshot upload to S3 2022-09-15 09:11:55 +00:00
Antonio Andelic
36f46891dd
Merge pull request #41215 from ClickHouse/keeper-close-connection-on-shutdown 
Close sessions on Keeper shutdown
 2022-09-13 19:09:34 +02:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage 
KeeperMap storage engine
 2022-09-13 14:15:09 +02:00
alesapin
69d08b1007
Merge pull request #41075 from ClickHouse/keeper-fix-possible-segfault 
Fix possible segfault during Keeper shutdown
 2022-09-13 12:13:24 +02:00
Antonio Andelic
a6b5ffec5d Polishing 2022-09-13 09:51:46 +00:00
Antonio Andelic
b2cc6a8cc6 Use promise/future 2022-09-12 18:19:41 +00:00
Antonio Andelic
c2dfabe51c Use shared_ptr for Event 2022-09-12 12:25:39 +00:00
Antonio Andelic
e985ee3354 Close sessions on Keeper shutdown 2022-09-12 12:22:48 +00:00
Antonio Andelic
32483aeec8 Merge branch 'master' into keeper-storage 2022-09-09 12:24:01 +00:00
Antonio Andelic
951ee44229 Process read requests with snapshot install 2022-09-07 11:51:56 +00:00
Antonio Andelic
a377a5f1e9 Change assert to LOGICAL_ERROR 2022-09-07 09:24:50 +00:00
Antonio Andelic
4cdf18cc81 Shutdown storage after RAFT 2022-09-07 09:20:47 +00:00
Antonio Andelic
c6b17bd7f9 Don't preprocess if storage finalized 2022-09-07 09:14:03 +00:00
Antonio Andelic
f0ba3ee927
Fix typo 2022-09-05 09:51:02 +02:00
Antonio Andelic
110ad53300 Merge branch 'master' into keeper-linearizable-reads 2022-09-05 07:12:45 +00:00
Antonio Andelic
b816d46520
Merge branch 'master' into fix-keeper-system-path-check 2022-09-02 13:42:16 +02:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage 2022-09-02 12:13:07 +02:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00