thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 20:02:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
69,496 Commits 9,330 Branches 1,783 Tags 2.1 GiB
9869c8bb10
Commit Graph

409 Commits

Author SHA1 Message Date
kssenii
a2cfbd74ec Fix fs::copy to work the same as Poco::copy 2021-05-07 22:20:16 +03:00
kssenii
140bf7e2ea Less Poco 2021-05-07 15:29:26 +03:00
kssenii
5068b163b8 Fix 2021-05-05 19:55:02 +03:00
kssenii
2dabdd0f73 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-05 18:42:40 +03:00
kssenii
9ec92ec514 Fix tests, less manual concatination of paths 2021-05-05 18:39:30 +03:00
kssenii
2b67341f28 Finish s3 2021-05-04 15:55:21 +03:00
vdimir
465341eafd
Add DiskRestartProxy.cpp to ya.make 2021-05-03 20:50:17 +03:00
kssenii
ad69fceea5 Restore copyFile 2021-05-02 19:59:18 +03:00
Vladimir Chebotarev
89242d4adc Fix. 2021-05-01 23:34:06 +03:00
Vladimir Chebotarev
a6ab040b07 Added region parameter for S3 storage and disk. 2021-05-01 20:45:35 +03:00
kssenii
32b2917141 Update s3/* 2021-05-01 18:10:04 +03:00
kssenii
ee06936596 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-01 17:24:31 +03:00
Vladimir
3b7c68196b
Merge pull request #23429 from Jokser/disk-s3-restart-possibility 2021-05-01 15:28:14 +03:00
kssenii
2887ce1f69 Temporary rollback 2021-05-01 14:23:13 +03:00
kssenii
6867ea2c73 Temporary fix 2021-04-30 19:14:04 +03:00
kssenii
c609119daf Update Disks/* 2021-04-30 18:13:19 +03:00
kssenii
7f2f138d56 Update DiskLocal and DiskMemory 2021-04-30 11:32:34 +03:00
Maksim Kita
318c4bb80d Add examples folder filter to ya.make.in 2021-04-30 11:25:52 +03:00
Pavel Kovalenko
0050dbb75b Apply disk settings on config reload. 2021-04-29 23:32:19 +03:00
kssenii
984d82c3f2 Update DiskLocal 2021-04-29 02:00:04 +03:00
Pavel Kovalenko
838ede6c62 Rename delegate -> impl. 2021-04-27 15:11:38 +03:00
Pavel Kovalenko
5cedd90bed WriteBufferFromFileDecorator destructor fixes. 2021-04-27 13:19:22 +03:00
Pavel Kovalenko
63e5f5d3ab DiskS3 and WriteBufferFromFileDecorator small refactoring. 2021-04-27 11:37:59 +03:00
Nikita Mikhaylov
9f55424250 move to examples everywhere 2021-04-27 01:51:42 +03:00
kssenii
814e727d0e Better 2021-04-22 08:21:37 +00:00
Pavel Kovalenko
37c637cc34 Minor fixes. 2021-04-21 13:34:07 +03:00
Pavel Kovalenko
2ab8013455 DiskRestartProxy improvements and tests. 2021-04-21 12:53:46 +03:00
kssenii
87b4843134 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-20 21:40:01 +00:00
kssenii
8689a24ca0 Slightly better, add integration test with hdfs disk as default 2021-04-20 20:51:00 +00:00
Pavel Kovalenko
9272237c7e Update DiskRestartProxy.cpp 2021-04-20 23:47:21 +03:00
Pavel Kovalenko
10cf881a08 Add SYSTEM RESTART DISK command. 2021-04-20 22:23:54 +03:00
Pavel Kovalenko
016a9c674b DiskRestartProxy refactoring. 2021-04-20 21:58:34 +03:00
Pavel Kovalenko
98f8cbbdb2 DiskRestartProxy refactoring. 2021-04-20 21:58:34 +03:00
Pavel Kovalenko
e8b4dd1e4b Introduce DiskRestartProxy. 2021-04-20 21:55:57 +03:00
kssenii
d5fb7ba094 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-20 13:38:29 +00:00
kssenii
22c6e37590 Better 2021-04-20 13:12:42 +00:00
Nikita Mikhaylov
f55eacee7d
Merge pull request #22988 from excitoon-favorites/s3connectionresetretries 
Retries on HTTP connection drops during reads from S3
 2021-04-20 14:59:21 +03:00
kssenii
a07d44e438 More common 2021-04-20 11:41:32 +00:00
kssenii
7d357baa40 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-20 08:08:50 +00:00
alexey-milovidov
4cef8c68f7
Merge pull request #23258 from azat/wundef 
Add -Wundef for gcc builds
 2021-04-19 22:45:22 +03:00
kssenii
e89594f376 Better 2021-04-19 15:50:46 +00:00
kssenii
9a91575990 One more test 2021-04-19 15:41:27 +00:00
kssenii
5df0fcc114 Fixes and tests 2021-04-19 15:33:33 +00:00
Nikita Mikhaylov
adc103e338
Merge pull request #21852 from excitoon-favorites/s3stscredentials 
Added insecure IMDS credentials provider for S3
 2021-04-19 14:47:44 +03:00
kssenii
ce16f4bb3f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-19 08:49:58 +00:00
kssenii
b37fb73468 Continued 2021-04-19 08:14:48 +00:00
Vladimir Chebotarev
9d4295f261 Removed infinite retries. 2021-04-19 11:02:36 +03:00
Vladimir
e4e2b04c53
Merge pull request #23112 from Jokser/disk-s3-restore-to-detached 2021-04-19 10:29:00 +03:00
Azat Khuzhin
2561a67fd8 Replace !__clang__ with !defined(__clang) to fix gcc builds 
$ gg 'if !__clang__' | cut -d: -f1 | sort -u | xargs sed -i 's/#if !__clang__/#if !defined(__clang__)/g'
 2021-04-18 23:37:50 +03:00
kssenii
d4db8671ed Better 2021-04-18 14:39:05 +00:00
kssenii
387494f0c8 Common interface 2021-04-18 13:37:23 +00:00
kssenii
82b1f1304f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-17 17:27:14 +00:00
Maksim Kita
d381e59791
Merge pull request #23114 from kitaisreal/fix-arcadia-build-s3 
Fix arcadia build S3
 2021-04-16 23:35:10 +03:00
Pavel Kovalenko
760a2ccedf DiskS3 restore file format change and minor improvements. 2021-04-16 17:43:16 +03:00
Nikita Mikhaylov
0ddb396338 add Y_IGNORE 2021-04-16 15:52:48 +03:00
Pavel Kovalenko
84a420b806 Replace dynamic log getting with private log field in Disk classes. 2021-04-15 23:38:44 +03:00
Pavel Kovalenko
6a067d705a Missed include. 2021-04-15 14:59:01 +03:00
Nikita Mikhaylov
a6b43f6765
Merge pull request #22594 from excitoon-favorites/betters3timeouts 
Consistent AWS timeouts
 2021-04-15 14:29:21 +03:00
Pavel Kovalenko
e53fc635f6 Disk S3 restore test improvements. 2021-04-15 11:26:44 +03:00
Vladimir Chebotarev
c0ea3ce08a Removed STS client. 2021-04-15 10:01:43 +03:00
Vladimir Chebotarev
3a467ffc50 Attempt to make credentials provider which uses internal credentials provider chain. 2021-04-15 10:01:13 +03:00
Vladimir Chebotarev
037a4de854 Added STS GetCallerIdentity credentials provider. 2021-04-15 10:00:18 +03:00
Vladimir Chebotarev
10bf56e540 Attempt to add retries to S3 reads. 2021-04-15 09:47:19 +03:00
Pavel Kovalenko
a2623a2d97 Rework Disk S3 revision number storing / searching. 2021-04-15 02:09:20 +03:00
Pavel Kovalenko
cad81e3742 Disk S3 possibility to restore parts to 'detached' directory. 2021-04-15 00:55:48 +03:00
Pavel Kovalenko
b1ba2cf2bc Disk S3 possibility to restore parts to 'detached' directory. 2021-04-14 19:21:26 +03:00
kssenii
ed8fec4cb1 Add missing methods 2021-04-12 10:13:53 +00:00
kssenii
ab06ca3144 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-12 07:07:48 +00:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
kssenii
eeb4dd4f4c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk 2021-04-05 11:30:28 +00:00
kssenii
908d872b76 Add seek method 2021-04-05 09:32:06 +00:00
Vladimir Chebotarev
0eb051961c Consistent AWS timeouts. 2021-04-04 14:06:10 +03:00
Alexey Milovidov
54caa61af5 Fix comments 2021-04-04 12:33:06 +03:00
kssenii
388a71e6b1 Fix build, ya check 2021-04-04 07:33:07 +00:00
kssenii
cbd678d031 Remove test 2021-04-03 12:10:49 +00:00
kssenii
8cbfbb5de7 Small refactoring, add hdfs to test_disk_types 2021-04-03 12:02:29 +00:00
kssenii
a2b6dbe18e Fix build, style, ya-make 2021-04-03 09:11:16 +00:00
kssenii
8d2709b212 Fix fast test 2021-04-02 22:09:12 +00:00
kssenii
e5ebdb0da5 Fix fast test 2021-04-02 21:47:29 +00:00
kssenii
a75aa5d101 Some refactoring 2021-04-02 21:02:48 +00:00
kssenii
a87234f67b Buildable version + small refactoring 2021-04-02 20:27:11 +00:00
alexey-milovidov
64922f3e6d
Merge branch 'master' into disk_reload 2021-04-02 21:38:13 +03:00
kssenii
fed821fe0e Merge branch 'master' of github.com:ClickHouse/ClickHouse into overshov-hdfs 2021-04-01 23:46:18 +03:00
Vladimir
57eb5f8772
Merge pull request #22070 from Jokser/disk-s3-migration 2021-03-31 11:06:40 +03:00
Pavel Kovalenko
0bb369d5e2 Code review fixes. 2021-03-30 13:21:46 +03:00
Alexey Milovidov
2a8ac01cdb Rename as suggested by Kita 2021-03-28 22:24:28 +03:00
Alexey Milovidov
50f712e198 Integrate mmap cache to the infrastructure 2021-03-28 04:10:30 +03:00
Pavel Kovalenko
0c0ddefc75 Disk S3 possibility to migrate to restorable schema 2021-03-24 13:12:47 +03:00
Pavel Kovalenko
9f9419cf14 Disk S3 possibility to migrate to restorable schema 2021-03-24 13:08:37 +03:00
Pavel Kovalenko
7e462a77a7 Disk S3 possibility to migrate to restorable schema 2021-03-24 13:07:01 +03:00
Pavel Kovalenko
e378c0bf8a Disk S3 possibility to migrate to restorable schema 2021-03-24 12:20:33 +03:00
Pavel Kovalenko
a92cf30b67 Code review fixes. 2021-03-23 13:33:07 +03:00
Pavel Kovalenko
7c0a1b9451 Cleanup destination directory during DiskCacheWrapper::moveFile() 2021-03-18 10:23:19 +03:00
Pavel Kovalenko
e6711e936e Cleanup destination directory during DiskCacheWrapper::moveFile() 2021-03-17 17:33:13 +03:00
Anton Ivashkin
5b267b7eec Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557 Fixes by review response 2021-02-26 12:48:57 +03:00
taiyang-li
1980e75a96 fix code style 2021-02-20 11:01:11 +08:00
taiyang-li
363b8904d5 merge master 2021-02-20 10:08:57 +08:00
taiyang-li
d370381980 fix code style 2021-02-20 09:47:53 +08:00
taiyang-li
e76b224f72 fix code style 2021-02-19 20:09:26 +08:00
taiyang-li
89d8c1264c add test of upload disk 2021-02-19 18:11:21 +08:00
alexey-milovidov
a299790522
Merge pull request #19748 from excitoon-favorites/s3ssec 
Added Server Side Encryption Customer Keys support in S3 client
 2021-02-15 15:23:14 +03:00
taiyang-li
3c61cecab5 add atomic for keep_free_space_bytes 2021-02-10 12:08:08 +08:00
Anton Ivashkin
4d44d75bc7 Fix build after merge one more time 2021-02-08 14:45:10 +03:00
Anton Ivashkin
1b6ce056a5 Merge master 2021-02-08 12:34:20 +03:00
Anton Ivashkin
cd94f708a1 Fix build after merge 2021-02-08 12:33:27 +03:00
alexey-milovidov
f2ab4f2ce9
Merge pull request #20037 from abyss7/better-read-buffers-2 
ReadBuffer: check for unread data on next()
 2021-02-07 01:33:06 +03:00
Anton Ivashkin
e64c63c611 Merge master 2021-02-05 20:10:06 +03:00
Anton Ivashkin
df6c882aab Fix build after merge 2021-02-05 18:52:40 +03:00
vdimir
1e2c302248
Merge pull request #18377 from Jokser/disk-s3-backup-restore-metadata 2021-02-05 10:44:59 +03:00
Ivan
d135a727e2
Merge branch 'master' into better-read-buffers-2 2021-02-05 02:17:05 +03:00
Ivan Lezhankin
84b88c6802 CC 2021-02-04 22:06:43 +03:00
kreuzerkrieg
1cc9e03d0c return DiskType instead of String IDisk::getType() as in the rest of storage interfaces 2021-02-04 10:01:21 +02:00
Pavel Kovalenko
62e96c138e Remove unused error code in DiskS3 2021-02-03 15:01:43 +03:00
Pavel Kovalenko
b6094859c7 Throw exception during restore DiskS3 if object doesn't have path in metadata. 2021-02-03 14:26:24 +03:00
Pavel Kovalenko
b7151aa754 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskDecorator.h
#	src/Disks/IDisk.h
#	src/Disks/S3/DiskS3.cpp
 2021-02-03 14:22:18 +03:00
taiyang-li
44200abfb4 add interface updateFromConfigIfChanged 2021-01-29 19:13:33 +08:00
Kruglov Pavel
caef103837
Merge branch 'master' into Add_IStoragePolicy_interface 2021-01-29 14:00:12 +03:00
Vladimir Chebotarev
579f8da573 Added SSE-C support in S3 client. 2021-01-28 09:32:41 +03:00
alexey-milovidov
7c240d0ca7
Merge pull request #17870 from excitoon-favorites/s3testiostream 
Update of AWS C++ SDK
 2021-01-28 07:12:52 +03:00
Vladimir Chebotarev
afdea5eff0 Changed handling of PocoHTTPClientConfiguration in order to ensure AWS SDK is initialized. 2021-01-27 09:38:35 +03:00
Anton Popov
09e5bdd1ef fix style check 2021-01-27 03:57:28 +03:00
Anton Popov
b15848e2f0
Update IDisk.h 2021-01-27 03:32:18 +03:00
Anton Popov
c7070da85a better abstractions in disk interface 2021-01-26 17:49:35 +03:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
taiyang-li
901de8968f update disk local if config changed 2021-01-26 15:07:17 +08:00
taiyang-li
ddb02282f1 enable local disk config reload 2021-01-24 11:53:07 +08:00
Alexander Gololobov
0a155ee2f1 Fix IDisk::open parameters to match posix open 2021-01-21 19:38:13 +03:00
Anton Ivashkin
357d98eb36 Merge master 2021-01-20 12:23:03 +03:00
Pavel Kovalenko
1e3a059f64 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskCacheWrapper.cpp
#	src/Disks/S3/DiskS3.cpp
 2021-01-18 13:39:49 +03:00
Alexander Kuzmenkov
d22c04568d
Merge pull request #13405 from excitoon-favorites/s3keepalive 
Connection pools for S3
 2021-01-18 12:52:37 +03:00
Alexey Milovidov
ee13dd4fef Fix error 2021-01-14 19:58:44 +03:00
Alexey Milovidov
8276a1c8d2 Faster parts removal, more safe and efficient interface of IDisk 2021-01-14 19:24:13 +03:00
Pavel Kovalenko
53389f79c0 Ability to backup-restore metadata files for DiskS3 (minor fixes) 2021-01-13 15:05:32 +03:00
Pavel Kovalenko
b09862b7b9 Ability to backup-restore metadata files for DiskS3 (fixes and tests) 2021-01-12 20:18:40 +03:00
Pavel Kovalenko
0856b2c514 Ability to backup-restore metadata files for DiskS3 (fixes and tests) 2021-01-11 20:37:08 +03:00
Alexey Milovidov
6eb5a5f4d9 Remove useless code 2021-01-10 03:28:59 +03:00
Vladimir Chebotarev
5cbc25c647 Added global setting s3_max_connections. 2021-01-05 13:34:55 +03:00
Pavel Kovalenko
2848b32af1 Ability to backup-restore metadata files for DiskS3 (WIP) 2020-12-23 18:11:37 +03:00
Pavel Kovalenko
cc3b5958b0 Ability to backup-restore metadata files for DiskS3 (WIP) 2020-12-23 15:35:52 +03:00
Pavel Kovalenko
18fe1c796b Ability to backup-restore metadata files for DiskS3 (WIP) 2020-12-22 21:47:47 +03:00
Anton Ivashkin
0f0500ca0c Merge master 2020-12-16 18:31:13 +03:00
Pavel Kovalenko
346c241c20 Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload 
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
 2020-12-14 11:00:30 +03:00
alexey-milovidov
04e222f6f3
Merge pull request #17986 from azat/fwd-decl 
More forward declaration for generic headers
 2020-12-13 01:12:05 +03:00
Azat Khuzhin
5b3ab48861 More forward declaration for generic headers 
The following headers are pretty generic, so use forward declaration as
much as possible:
- Context.h
- Settings.h
- ConnectionTimeouts.h
(Also this shows that some missing some includes -- this has been fixed)

And split ConnectionTimeouts.h into ConnectionTimeoutsContext.h (since
module part cannot be added for it, due to recursive build dependencies
that will be introduced)

Also remove Settings from the RemoteBlockInputStream/RemoteQueryExecutor
and just pass the context, since settings was passed only in speicifc
places, that can allow making a copy of Context (i.e. Copier).

Approx results (How much units will be recompiled after changing file X?):

- ConnectionTimeouts.h
  - mainline: 100

- Context.h:
  - mainline: ~800
  - patched:  415

- Settings.h:
  - mainline: 900-1K
  - patched:  440 (most of them because of the Context.h)
 2020-12-12 17:43:10 +03:00
Pavel Kovalenko
25df6bae78 Add nested exception when handle Disk S3 metadata file read error. 2020-12-12 17:11:47 +03:00
Pavel Kovalenko
59a5e63417 Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload 
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
#	src/Disks/S3/DiskS3.h
#	src/Disks/S3/registerDiskS3.cpp
#	src/IO/WriteBufferFromS3.cpp
#	src/IO/WriteBufferFromS3.h
 2020-12-11 18:28:41 +03:00
Pavel Kovalenko
72a55c192e Fix master branch merge artifact. 2020-12-11 18:23:59 +03:00
Pavel Kovalenko
75bdb813ea Merge remote-tracking branch 'origin/master' into disk-s3-remove-empty-metadata-files 
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
 2020-12-11 18:22:31 +03:00
Pavel Kovalenko
de4a7a8865 Correct handling S3 disk metadata file read exceptions. 2020-12-11 14:27:58 +03:00