thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 11:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
78,792 Commits 9,326 Branches 1,784 Tags 2.1 GiB
60dd22dd4e
Commit Graph

610 Commits

Author SHA1 Message Date
Vitaly Baranov
a05184d83a Fix synchronization while updating from the config of an encrypted disk. 2021-08-01 22:55:36 +03:00
kssenii
f1e96dbc55 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into taiyang-li-disk-reload 2021-07-29 07:53:16 +00:00
Vitaly Baranov
51f882d8cc Support multiple keys for encrypted disk. Display error message if the key is probably wrong. 2021-07-24 23:45:12 +03:00
Vitaly Baranov
81ea8521b1 Set encryption algorithm explicitly. 2021-07-18 10:29:42 +03:00
Vitaly Baranov
958494b98e
Merge pull request #26377 from vitlibar/encrypted-disk-comments-and-improvements 
Add comments for encrypted disks; some improvements in code.
 2021-07-16 21:24:51 +03:00
Vitaly Baranov
9cc258327e Add comments for encrypted disks; some improvements in code. 2021-07-16 12:04:25 +03:00
Zhichang Yu
fe5d17680e Merge remote-tracking branch 'origin/master' into zero_copy_hdfs 2021-07-14 02:12:28 +00:00
Vitaly Baranov
bc5080d701 Rename ReadEncryptedBuffer -> ReadBufferFromEncryptedFile, 
WriteEncryptedBuffer -> WriteBufferFromEncryptedFile,
move file FileEncryption.h to the IO folder,
rename functions accroding to our code style,
add cpp files.
 2021-07-13 11:31:03 +03:00
Vitaly Baranov
042befb369 Fix compilation 2021-07-13 11:29:52 +03:00
Vitaly Baranov
e086e5e384
Merge branch 'master' into alexelex-encryption-1 2021-07-13 11:28:41 +03:00
Zhichang Yu
b4e6689bf9 fix test_hdfs_zero_copy_replication_move[tiered_copy-2] 2021-07-13 07:20:23 +00:00
Zhichang Yu
5047c758f4 fix per review 2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1 hdfs zero copy 2021-07-13 07:19:12 +00:00
Alexey Milovidov
81646f8389 Merge branch 'master' into ditch-aio 2021-07-11 03:06:45 +03:00
Anton Ivashkin
ac68d5ea71 Fix path concatenation 2021-07-08 18:39:41 +03:00
Anton Ivashkin
88bc3995e0 Fix throwing exception when iterate over non existing remote directory 2021-07-08 17:54:59 +03:00
Alexey Milovidov
21fb6ddea5 Remove AIO for sequential reads 2021-07-05 23:02:24 +03:00
Anton Ivashkin
e0199d8006 Fix DiskS3 restore 2021-06-22 20:05:20 +03:00
kssenii
babb11a887 Some fixes 2021-06-19 19:18:13 +00:00
kssenii
2031607642 Reimplement 2021-06-19 11:52:28 +00:00
kssenii
7f356a5fbc Fix readbuffer 2021-06-18 17:06:17 +00:00
kssenii
f33387837e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-06-18 14:27:48 +00:00
kssenii
0fa6dc4570 Reimplement 2021-06-18 11:17:37 +00:00
Vladimir
4de0d1a593
Merge pull request #24885 from Jokser/disk-s3-read-error-fix 2021-06-18 13:38:59 +03:00
Alexandra
5bc5a29c32 ContextConstPtr -> ContextPtr (resolve conflicts) 2021-06-18 05:36:50 +00:00
Latysheva Alexandra
48d3419160
Merge branch 'master' into alexelex-encryption-1 2021-06-18 11:28:08 +06:00
kssenii
cd885ee750 Fix build, ya make 2021-06-14 11:52:20 +00:00
kssenii
a1f11f5178 Fixes 2021-06-14 09:37:33 +00:00
kssenii
11faddb39f Better 2021-06-13 22:30:37 +00:00
kssenii
2cfcc9980c Better 2021-06-13 19:38:31 +00:00
kssenii
c80eac0dc3 First version 2021-06-13 19:36:56 +00:00
Alexey Milovidov
93b907931d Fix the annoying ya.make. 2021-06-13 21:21:44 +03:00
alexey-milovidov
05d1af153c
Merge branch 'master' into rename-const-context-ptr 2021-06-12 03:25:09 +03:00
alexey-milovidov
99f108d704
Merge pull request #25162 from kitaisreal/rewrite-function-to-subcolumn-visitor-add-into-ya-make 
RewriteFunctionToSubcolumnVisitor add into ya.make
 2021-06-12 02:58:53 +03:00
万康
3a404fcc1b fix build error 2021-06-11 23:21:18 +08:00
Maksim Kita
30c78c0a23 Fix arcadia 2021-06-11 14:26:33 +03:00
mergify[bot]
bfbc787b08
Merge branch 'master' into disk-s3-read-error-fix 2021-06-09 06:56:19 +00:00
Latysheva Alexandra
1a2cbe9275
Merge branch 'master' into alexelex-encryption-1 2021-06-07 21:18:54 +06:00
Ivan Lezhankin
0e9ce8c23a Fix Disks/ya.make 2021-06-07 17:22:24 +03:00
Pavel Kovalenko
6ba40c475a Code cleanup + comments. 2021-06-07 13:49:34 +03:00
mergify[bot]
b8dcce68b5
Merge branch 'master' into rename-const-context-ptr 2021-06-07 09:46:24 +00:00
Maksim Kita
16b3355ac1 Disks fix ya.make 2021-06-07 10:05:05 +03:00
mergify[bot]
d68d5e0ef1
Merge branch 'master' into rename-const-context-ptr 2021-06-04 14:41:41 +00:00
Pavel Kovalenko
6628b670df Fix reading from several buffers in ReadIndirectBufferFromRemoteFS 2021-06-03 14:34:23 +03:00
Pavel Kovalenko
2bd12a7612 Fix DiskS3 read error due to broken getPosition() method in ReadBufferFromS3.cpp 2021-06-02 18:03:25 +03:00
Nikita Mikhaylov
0055a924f6 better 2021-06-01 14:23:46 +00:00
Nikita Mikhaylov
3f18bde246 Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-s3-backoff 2021-06-01 13:39:09 +00:00
Nikolai Kochetov
dbaa6ffc62 Rename ContextConstPtr to ContextPtr. 2021-06-01 15:20:52 +03:00
Nikita Mikhaylov
e03c09c51b
Update DiskS3.cpp 2021-05-31 21:08:16 +03:00
Nikita Mikhaylov
0d8ede15a9
Merge branch 'master' into merging-s3-backoff 2021-05-31 21:07:00 +03:00
kssenii
69816e6eff Fix checks 2021-05-30 15:44:58 +03:00
kssenii
0a3589524e File movement 2021-05-29 00:57:53 +03:00
kssenii
2a631aaf08 Final fixes 2021-05-29 00:34:44 +03:00
kssenii
f66c67a979 Fixes 2021-05-27 15:42:46 +03:00
kssenii
f9088b98de Some disks update 2021-05-27 00:02:24 +03:00
kssenii
b28157a0df Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-27 00:01:25 +03:00
kssenii
73f16ee9ee Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-26 23:08:08 +03:00
kssenii
940eab6924 Review fixes 2021-05-26 09:26:19 +03:00
kssenii
379e26bcbc Fix checks 2021-05-25 17:37:35 +00:00
kssenii
e01abf807e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-05-25 17:32:43 +00:00
mergify[bot]
400b74d186
Merge branch 'master' into merging-s3-backoff 2021-05-25 14:02:38 +00:00
kssenii
75a6b21f19 Better 2021-05-25 11:29:46 +00:00
kssenii
31107816bb fs::absolute to fs::canonical 2021-05-24 20:37:54 +03:00
Anton Ivashkin
4b9488be40 Add log record for removed from AWS S3 keys 2021-05-24 12:34:32 +03:00
kssenii
ecd82213d1 Some clean up, more tests 2021-05-23 20:19:04 +00:00
kssenii
0f98731c1f More integration tests 2021-05-23 07:33:31 +00:00
Alexandra
2bb25f356d createDir->createDirs in DiskEncrypted initialization 2021-05-23 06:49:02 +00:00
kssenii
6931573f69 Fix checks 2021-05-21 09:22:20 +00:00
kssenii
06ff4ab9da Fix build 2021-05-20 13:43:08 +00:00
kssenii
2f6770c9a0 Unite writeBuffers 2021-05-20 10:46:26 +00:00
kssenii
550a4e33f9 Unite readBuffers 2021-05-20 09:57:59 +00:00
Alexandra
11e40ad000 style fixes 2021-05-19 22:06:25 +00:00
Nikita Mikhaylov
f33b70c54a better 2021-05-20 00:42:25 +03:00
kssenii
9a5a6f39b7 Simplify writeFile 2021-05-19 20:31:43 +00:00
kssenii
5196075fd9 Better 2021-05-19 12:34:01 +00:00
Alexandra
f71e97511b set use_ssl flag 2021-05-18 20:48:16 +00:00
Alexandra
b7a0f2db57 refactoring 2021-05-18 20:20:51 +00:00
Alexandra
e3a91f568e fix encryption (very small files encryption bug) and replace 64-iv to 128-iv 2021-05-18 20:02:19 +00:00
kssenii
5f083b779d A little better, fix checks 2021-05-18 14:25:19 +03:00
Alexandra
721ea4f251 remove unnecessary includes 2021-05-17 16:58:51 +00:00
Alexandra
af420569f8 remove debug information, fix seek issue and encryption issue 2021-05-17 16:15:21 +00:00
Vladimir Chebotarev
bdf03387ab Better S3 read retries. Renamed s3_max_single_read_retries -> s3_single_read_retry_attempts, added sleeps before next attempts. 2021-05-17 10:38:32 +03:00
Alexandra
10a5b734ee encryption initialization (without any flags like use_ssl for now) 2021-05-16 22:35:20 +00:00
kssenii
018d52d71f Directory path 2021-05-15 09:20:26 +03:00
Alexandra
24573794ce inherit DiskEncrypted from DiskDecorator; override copy (use IDisk::copy instead of LocalDisk::copy), applyNewSettings, onFreeze and removeShared* 2021-05-14 21:52:51 +00:00
Alexandra
730b9792bc add parameters DiskEncrypted IDisk::applyNewSettings usage 2021-05-14 21:48:34 +00:00
kssenii
db849eb3e5 Remaining poco 2021-05-14 11:02:17 +03:00
kssenii
b123ec1354 Poco::FileException, Poco::parentPath 2021-05-14 00:20:28 +03:00
Alexandra
5374ac5c11 resolve alter table issue and add relative path to DiskEncrypted that can be declared in config as a <path> 2021-05-13 16:21:45 +00:00
Alexandra
7c34083441 mutex loop resolved 2021-05-13 06:46:11 +00:00
Alexandra
628bf9b345 disk encryption init 2021-05-13 06:37:05 +00:00
kssenii
e466924898 fs::premissions 2021-05-13 02:01:35 +03:00
kssenii
17432b1a41 Poco::moveTo to fs version 2021-05-12 17:32:22 +03:00
kssenii
047f8a4e3c Correct merge 2021-05-12 13:10:06 +00:00
kssenii
850b6f02cb Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-05-11 14:22:30 +00:00
kssenii
393b026f2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-11 16:49:23 +03:00
Vladimir
d02d14fa00
Merge pull request #23846 from excitoon-favorites/s3regions 2021-05-10 12:14:08 +03:00
kssenii
ab1a05a1f4 Poco::Path to fs::path, less concatination 2021-05-09 14:59:49 +03:00
kssenii
2d4c8b394c Return Poco::moveTo 2021-05-08 14:34:50 +03:00
kssenii
dc38da5e32 Fix fs::copy to be like Poco::copyTo 2021-05-08 14:00:31 +03:00
kssenii
35f999bf04 Poco::createFile to fs::createFile 2021-05-08 02:41:47 +03:00
kssenii
a2cfbd74ec Fix fs::copy to work the same as Poco::copy 2021-05-07 22:20:16 +03:00
kssenii
140bf7e2ea Less Poco 2021-05-07 15:29:26 +03:00
kssenii
5068b163b8 Fix 2021-05-05 19:55:02 +03:00
kssenii
2dabdd0f73 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-05 18:42:40 +03:00
kssenii
9ec92ec514 Fix tests, less manual concatination of paths 2021-05-05 18:39:30 +03:00
kssenii
2b67341f28 Finish s3 2021-05-04 15:55:21 +03:00
vdimir
465341eafd
Add DiskRestartProxy.cpp to ya.make 2021-05-03 20:50:17 +03:00
kssenii
ad69fceea5 Restore copyFile 2021-05-02 19:59:18 +03:00
Vladimir Chebotarev
89242d4adc Fix. 2021-05-01 23:34:06 +03:00
Vladimir Chebotarev
a6ab040b07 Added region parameter for S3 storage and disk. 2021-05-01 20:45:35 +03:00
kssenii
32b2917141 Update s3/* 2021-05-01 18:10:04 +03:00
kssenii
ee06936596 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-01 17:24:31 +03:00
Vladimir
3b7c68196b
Merge pull request #23429 from Jokser/disk-s3-restart-possibility 2021-05-01 15:28:14 +03:00
kssenii
2887ce1f69 Temporary rollback 2021-05-01 14:23:13 +03:00
kssenii
6867ea2c73 Temporary fix 2021-04-30 19:14:04 +03:00
kssenii
c609119daf Update Disks/* 2021-04-30 18:13:19 +03:00
kssenii
7f2f138d56 Update DiskLocal and DiskMemory 2021-04-30 11:32:34 +03:00
Maksim Kita
318c4bb80d Add examples folder filter to ya.make.in 2021-04-30 11:25:52 +03:00
Pavel Kovalenko
0050dbb75b Apply disk settings on config reload. 2021-04-29 23:32:19 +03:00
kssenii
984d82c3f2 Update DiskLocal 2021-04-29 02:00:04 +03:00
Pavel Kovalenko
838ede6c62 Rename delegate -> impl. 2021-04-27 15:11:38 +03:00
Pavel Kovalenko
5cedd90bed WriteBufferFromFileDecorator destructor fixes. 2021-04-27 13:19:22 +03:00
Pavel Kovalenko
63e5f5d3ab DiskS3 and WriteBufferFromFileDecorator small refactoring. 2021-04-27 11:37:59 +03:00
Nikita Mikhaylov
9f55424250 move to examples everywhere 2021-04-27 01:51:42 +03:00
kssenii
814e727d0e Better 2021-04-22 08:21:37 +00:00
Pavel Kovalenko
37c637cc34 Minor fixes. 2021-04-21 13:34:07 +03:00
Pavel Kovalenko
2ab8013455 DiskRestartProxy improvements and tests. 2021-04-21 12:53:46 +03:00
kssenii
87b4843134 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-20 21:40:01 +00:00
kssenii
8689a24ca0 Slightly better, add integration test with hdfs disk as default 2021-04-20 20:51:00 +00:00
Pavel Kovalenko
9272237c7e Update DiskRestartProxy.cpp 2021-04-20 23:47:21 +03:00
Pavel Kovalenko
10cf881a08 Add SYSTEM RESTART DISK command. 2021-04-20 22:23:54 +03:00
Pavel Kovalenko
016a9c674b DiskRestartProxy refactoring. 2021-04-20 21:58:34 +03:00
Pavel Kovalenko
98f8cbbdb2 DiskRestartProxy refactoring. 2021-04-20 21:58:34 +03:00
Pavel Kovalenko
e8b4dd1e4b Introduce DiskRestartProxy. 2021-04-20 21:55:57 +03:00
kssenii
d5fb7ba094 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-20 13:38:29 +00:00
kssenii
22c6e37590 Better 2021-04-20 13:12:42 +00:00
Nikita Mikhaylov
f55eacee7d
Merge pull request #22988 from excitoon-favorites/s3connectionresetretries 
Retries on HTTP connection drops during reads from S3
 2021-04-20 14:59:21 +03:00
kssenii
a07d44e438 More common 2021-04-20 11:41:32 +00:00
kssenii
7d357baa40 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-20 08:08:50 +00:00
alexey-milovidov
4cef8c68f7
Merge pull request #23258 from azat/wundef 
Add -Wundef for gcc builds
 2021-04-19 22:45:22 +03:00
kssenii
e89594f376 Better 2021-04-19 15:50:46 +00:00
kssenii
9a91575990 One more test 2021-04-19 15:41:27 +00:00
kssenii
5df0fcc114 Fixes and tests 2021-04-19 15:33:33 +00:00
Nikita Mikhaylov
adc103e338
Merge pull request #21852 from excitoon-favorites/s3stscredentials 
Added insecure IMDS credentials provider for S3
 2021-04-19 14:47:44 +03:00
kssenii
ce16f4bb3f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-19 08:49:58 +00:00
kssenii
b37fb73468 Continued 2021-04-19 08:14:48 +00:00
Vladimir Chebotarev
9d4295f261 Removed infinite retries. 2021-04-19 11:02:36 +03:00
Vladimir
e4e2b04c53
Merge pull request #23112 from Jokser/disk-s3-restore-to-detached 2021-04-19 10:29:00 +03:00
Azat Khuzhin
2561a67fd8 Replace !__clang__ with !defined(__clang) to fix gcc builds 
$ gg 'if !__clang__' | cut -d: -f1 | sort -u | xargs sed -i 's/#if !__clang__/#if !defined(__clang__)/g'
 2021-04-18 23:37:50 +03:00
kssenii
d4db8671ed Better 2021-04-18 14:39:05 +00:00
kssenii
387494f0c8 Common interface 2021-04-18 13:37:23 +00:00
kssenii
82b1f1304f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-17 17:27:14 +00:00
Maksim Kita
d381e59791
Merge pull request #23114 from kitaisreal/fix-arcadia-build-s3 
Fix arcadia build S3
 2021-04-16 23:35:10 +03:00
Pavel Kovalenko
760a2ccedf DiskS3 restore file format change and minor improvements. 2021-04-16 17:43:16 +03:00
Nikita Mikhaylov
0ddb396338 add Y_IGNORE 2021-04-16 15:52:48 +03:00
Pavel Kovalenko
84a420b806 Replace dynamic log getting with private log field in Disk classes. 2021-04-15 23:38:44 +03:00
Pavel Kovalenko
6a067d705a Missed include. 2021-04-15 14:59:01 +03:00
Nikita Mikhaylov
a6b43f6765
Merge pull request #22594 from excitoon-favorites/betters3timeouts 
Consistent AWS timeouts
 2021-04-15 14:29:21 +03:00
Pavel Kovalenko
e53fc635f6 Disk S3 restore test improvements. 2021-04-15 11:26:44 +03:00
Vladimir Chebotarev
c0ea3ce08a Removed STS client. 2021-04-15 10:01:43 +03:00
Vladimir Chebotarev
3a467ffc50 Attempt to make credentials provider which uses internal credentials provider chain. 2021-04-15 10:01:13 +03:00
Vladimir Chebotarev
037a4de854 Added STS GetCallerIdentity credentials provider. 2021-04-15 10:00:18 +03:00
Vladimir Chebotarev
10bf56e540 Attempt to add retries to S3 reads. 2021-04-15 09:47:19 +03:00
Pavel Kovalenko
a2623a2d97 Rework Disk S3 revision number storing / searching. 2021-04-15 02:09:20 +03:00
Pavel Kovalenko
cad81e3742 Disk S3 possibility to restore parts to 'detached' directory. 2021-04-15 00:55:48 +03:00
Pavel Kovalenko
b1ba2cf2bc Disk S3 possibility to restore parts to 'detached' directory. 2021-04-14 19:21:26 +03:00
kssenii
ed8fec4cb1 Add missing methods 2021-04-12 10:13:53 +00:00
kssenii
ab06ca3144 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-12 07:07:48 +00:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
kssenii
eeb4dd4f4c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-05 11:30:28 +00:00
kssenii
908d872b76 Add seek method 2021-04-05 09:32:06 +00:00
Vladimir Chebotarev
0eb051961c Consistent AWS timeouts. 2021-04-04 14:06:10 +03:00
Alexey Milovidov
54caa61af5 Fix comments 2021-04-04 12:33:06 +03:00
kssenii
388a71e6b1 Fix build, ya check 2021-04-04 07:33:07 +00:00
kssenii
cbd678d031 Remove test 2021-04-03 12:10:49 +00:00
kssenii
8cbfbb5de7 Small refactoring, add hdfs to test_disk_types 2021-04-03 12:02:29 +00:00
kssenii
a2b6dbe18e Fix build, style, ya-make 2021-04-03 09:11:16 +00:00
kssenii
8d2709b212 Fix fast test 2021-04-02 22:09:12 +00:00
kssenii
e5ebdb0da5 Fix fast test 2021-04-02 21:47:29 +00:00
kssenii
a75aa5d101 Some refactoring 2021-04-02 21:02:48 +00:00
kssenii
a87234f67b Buildable version + small refactoring 2021-04-02 20:27:11 +00:00
alexey-milovidov
64922f3e6d
Merge branch 'master' into disk_reload 2021-04-02 21:38:13 +03:00
kssenii
fed821fe0e Merge branch 'master' of github.com:ClickHouse/ClickHouse into overshov-hdfs 2021-04-01 23:46:18 +03:00
Vladimir
57eb5f8772
Merge pull request #22070 from Jokser/disk-s3-migration 2021-03-31 11:06:40 +03:00
Pavel Kovalenko
0bb369d5e2 Code review fixes. 2021-03-30 13:21:46 +03:00
Alexey Milovidov
2a8ac01cdb Rename as suggested by Kita 2021-03-28 22:24:28 +03:00
Alexey Milovidov
50f712e198 Integrate mmap cache to the infrastructure 2021-03-28 04:10:30 +03:00
Pavel Kovalenko
0c0ddefc75 Disk S3 possibility to migrate to restorable schema 2021-03-24 13:12:47 +03:00
Pavel Kovalenko
9f9419cf14 Disk S3 possibility to migrate to restorable schema 2021-03-24 13:08:37 +03:00
Pavel Kovalenko
7e462a77a7 Disk S3 possibility to migrate to restorable schema 2021-03-24 13:07:01 +03:00
Pavel Kovalenko
e378c0bf8a Disk S3 possibility to migrate to restorable schema 2021-03-24 12:20:33 +03:00
Pavel Kovalenko
a92cf30b67 Code review fixes. 2021-03-23 13:33:07 +03:00
Pavel Kovalenko
7c0a1b9451 Cleanup destination directory during DiskCacheWrapper::moveFile() 2021-03-18 10:23:19 +03:00
Pavel Kovalenko
e6711e936e Cleanup destination directory during DiskCacheWrapper::moveFile() 2021-03-17 17:33:13 +03:00
Anton Ivashkin
5b267b7eec Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557 Fixes by review response 2021-02-26 12:48:57 +03:00
taiyang-li
1980e75a96 fix code style 2021-02-20 11:01:11 +08:00
taiyang-li
363b8904d5 merge master 2021-02-20 10:08:57 +08:00
taiyang-li
d370381980 fix code style 2021-02-20 09:47:53 +08:00
taiyang-li
e76b224f72 fix code style 2021-02-19 20:09:26 +08:00
taiyang-li
89d8c1264c add test of upload disk 2021-02-19 18:11:21 +08:00
alexey-milovidov
a299790522
Merge pull request #19748 from excitoon-favorites/s3ssec 
Added Server Side Encryption Customer Keys support in S3 client
 2021-02-15 15:23:14 +03:00
taiyang-li
3c61cecab5 add atomic for keep_free_space_bytes 2021-02-10 12:08:08 +08:00
Anton Ivashkin
4d44d75bc7 Fix build after merge one more time 2021-02-08 14:45:10 +03:00
Anton Ivashkin
1b6ce056a5 Merge master 2021-02-08 12:34:20 +03:00
Anton Ivashkin
cd94f708a1 Fix build after merge 2021-02-08 12:33:27 +03:00
alexey-milovidov
f2ab4f2ce9
Merge pull request #20037 from abyss7/better-read-buffers-2 
ReadBuffer: check for unread data on next()
 2021-02-07 01:33:06 +03:00
Anton Ivashkin
e64c63c611 Merge master 2021-02-05 20:10:06 +03:00
Anton Ivashkin
df6c882aab Fix build after merge 2021-02-05 18:52:40 +03:00
vdimir
1e2c302248
Merge pull request #18377 from Jokser/disk-s3-backup-restore-metadata 2021-02-05 10:44:59 +03:00
Ivan
d135a727e2
Merge branch 'master' into better-read-buffers-2 2021-02-05 02:17:05 +03:00
Ivan Lezhankin
84b88c6802 CC 2021-02-04 22:06:43 +03:00
kreuzerkrieg
1cc9e03d0c return DiskType instead of String IDisk::getType() as in the rest of storage interfaces 2021-02-04 10:01:21 +02:00
Pavel Kovalenko
62e96c138e Remove unused error code in DiskS3 2021-02-03 15:01:43 +03:00
Pavel Kovalenko
b6094859c7 Throw exception during restore DiskS3 if object doesn't have path in metadata. 2021-02-03 14:26:24 +03:00
Pavel Kovalenko
b7151aa754 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskDecorator.h
#	src/Disks/IDisk.h
#	src/Disks/S3/DiskS3.cpp
 2021-02-03 14:22:18 +03:00
taiyang-li
44200abfb4 add interface updateFromConfigIfChanged 2021-01-29 19:13:33 +08:00
Kruglov Pavel
caef103837
Merge branch 'master' into Add_IStoragePolicy_interface 2021-01-29 14:00:12 +03:00
Vladimir Chebotarev
579f8da573 Added SSE-C support in S3 client. 2021-01-28 09:32:41 +03:00
alexey-milovidov
7c240d0ca7
Merge pull request #17870 from excitoon-favorites/s3testiostream 
Update of AWS C++ SDK
 2021-01-28 07:12:52 +03:00
Vladimir Chebotarev
afdea5eff0 Changed handling of PocoHTTPClientConfiguration in order to ensure AWS SDK is initialized. 2021-01-27 09:38:35 +03:00
Anton Popov
09e5bdd1ef fix style check 2021-01-27 03:57:28 +03:00
Anton Popov
b15848e2f0
Update IDisk.h 2021-01-27 03:32:18 +03:00
Anton Popov
c7070da85a better abstractions in disk interface 2021-01-26 17:49:35 +03:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
taiyang-li
901de8968f update disk local if config changed 2021-01-26 15:07:17 +08:00
taiyang-li
ddb02282f1 enable local disk config reload 2021-01-24 11:53:07 +08:00
Alexander Gololobov
0a155ee2f1 Fix IDisk::open parameters to match posix open 2021-01-21 19:38:13 +03:00
Anton Ivashkin
357d98eb36 Merge master 2021-01-20 12:23:03 +03:00
Pavel Kovalenko
1e3a059f64 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskCacheWrapper.cpp
#	src/Disks/S3/DiskS3.cpp
 2021-01-18 13:39:49 +03:00
Alexander Kuzmenkov
d22c04568d
Merge pull request #13405 from excitoon-favorites/s3keepalive 
Connection pools for S3
 2021-01-18 12:52:37 +03:00
Alexey Milovidov
ee13dd4fef Fix error 2021-01-14 19:58:44 +03:00
Alexey Milovidov
8276a1c8d2 Faster parts removal, more safe and efficient interface of IDisk 2021-01-14 19:24:13 +03:00
Pavel Kovalenko
53389f79c0 Ability to backup-restore metadata files for DiskS3 (minor fixes) 2021-01-13 15:05:32 +03:00
Pavel Kovalenko
b09862b7b9 Ability to backup-restore metadata files for DiskS3 (fixes and tests) 2021-01-12 20:18:40 +03:00
Pavel Kovalenko
0856b2c514 Ability to backup-restore metadata files for DiskS3 (fixes and tests) 2021-01-11 20:37:08 +03:00
Alexey Milovidov
6eb5a5f4d9 Remove useless code 2021-01-10 03:28:59 +03:00
Vladimir Chebotarev
5cbc25c647 Added global setting s3_max_connections. 2021-01-05 13:34:55 +03:00
Pavel Kovalenko
2848b32af1 Ability to backup-restore metadata files for DiskS3 (WIP) 2020-12-23 18:11:37 +03:00
Pavel Kovalenko
cc3b5958b0 Ability to backup-restore metadata files for DiskS3 (WIP) 2020-12-23 15:35:52 +03:00
Pavel Kovalenko
18fe1c796b Ability to backup-restore metadata files for DiskS3 (WIP) 2020-12-22 21:47:47 +03:00
Anton Ivashkin
0f0500ca0c Merge master 2020-12-16 18:31:13 +03:00
Pavel Kovalenko
346c241c20 Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload 
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
 2020-12-14 11:00:30 +03:00
alexey-milovidov
04e222f6f3
Merge pull request #17986 from azat/fwd-decl 
More forward declaration for generic headers
 2020-12-13 01:12:05 +03:00
Azat Khuzhin
5b3ab48861 More forward declaration for generic headers 
The following headers are pretty generic, so use forward declaration as
much as possible:
- Context.h
- Settings.h
- ConnectionTimeouts.h
(Also this shows that some missing some includes -- this has been fixed)

And split ConnectionTimeouts.h into ConnectionTimeoutsContext.h (since
module part cannot be added for it, due to recursive build dependencies
that will be introduced)

Also remove Settings from the RemoteBlockInputStream/RemoteQueryExecutor
and just pass the context, since settings was passed only in speicifc
places, that can allow making a copy of Context (i.e. Copier).

Approx results (How much units will be recompiled after changing file X?):

- ConnectionTimeouts.h
  - mainline: 100

- Context.h:
  - mainline: ~800
  - patched:  415

- Settings.h:
  - mainline: 900-1K
  - patched:  440 (most of them because of the Context.h)
 2020-12-12 17:43:10 +03:00
Pavel Kovalenko
25df6bae78 Add nested exception when handle Disk S3 metadata file read error. 2020-12-12 17:11:47 +03:00
Pavel Kovalenko
59a5e63417 Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload 
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
#	src/Disks/S3/DiskS3.h
#	src/Disks/S3/registerDiskS3.cpp
#	src/IO/WriteBufferFromS3.cpp
#	src/IO/WriteBufferFromS3.h
 2020-12-11 18:28:41 +03:00
Pavel Kovalenko
72a55c192e Fix master branch merge artifact. 2020-12-11 18:23:59 +03:00
Pavel Kovalenko
75bdb813ea Merge remote-tracking branch 'origin/master' into disk-s3-remove-empty-metadata-files 
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
 2020-12-11 18:22:31 +03:00