Anton Ivashkin
f833501c77
Merge branch 'master' into s3_zero_copy_replication
2020-10-22 11:17:21 +03:00
Vladimir Chebotarev
aa5f207fd4
Added disable_merges option for volumes in multi-disk configuration ( #13956 )
...
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-10-20 18:10:24 +03:00
Alexey Milovidov
94592819ad
Add comment
2020-10-20 04:29:13 +03:00
Pavel Kovalenko
a474bdcabf
Merge remote-tracking branch 'origin/master' into idempotent-part-move
2020-10-16 13:02:40 +03:00
Anton Ivashkin
b877459cf7
Zero copy replication over S3: check s3 storage
2020-10-14 18:05:59 +03:00
Alexey Milovidov
269b6383f5
Check for #pragma once in headers
2020-10-10 21:37:02 +03:00
Anton Ivashkin
766dbfd2be
Merge master
2020-10-09 15:00:14 +03:00
Anton Ivashkin
acf86568a7
S3 zero copy replication proof of concept
2020-10-08 18:45:10 +03:00
Pavel Kovalenko
69c126f1f1
Possibility to move part to another disk/volume if first attempt was failed.
2020-10-07 14:35:28 +03:00
Pavel Kovalenko
69b4bc6f33
Proper error handling during insert into MergeTree with S3.
2020-10-06 12:38:00 +03:00
alexey-milovidov
5e37ba4335
Update DiskS3.cpp
2020-10-03 03:47:45 +03:00
alexey-milovidov
3ca7fca48a
Update DiskS3.cpp
2020-10-03 03:42:33 +03:00
Anton Ivashkin
c18cdec77c
Use S3 multi object delete
...
Some strange with s3_with_proxy test
2020-09-28 11:47:04 +03:00
alexey-milovidov
1474b80ecc
Merge pull request #15008 from ClickHouse/clang-tidy-11-without-enabling
...
Fix terribly wrong code
2020-09-20 02:00:54 +03:00
Alexey Milovidov
ef5363b87f
Fix terribly wrong code
2020-09-19 02:05:13 +03:00
Pavel Kovalenko
da04a130ed
Add option to disable ttl move on data part insert - minor fixes.
2020-09-18 20:45:30 +03:00
Pavel Kovalenko
0da19ab46d
Add option to disable ttl move on data part insert
2020-09-18 18:30:00 +03:00
Artem Zuikov
51ba12c2c3
Try speedup build ( #14809 )
2020-09-15 12:55:57 +03:00
Anton Popov
af4089c606
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-11 00:39:21 +03:00
Pavel Kovalenko
3632b0bdb3
Add option to DiskS3 that allows it's usage if S3 unavailable ( #14497 )
2020-09-09 02:35:57 +03:00
alesapin
bea15175d6
Merge pull request #14496 from Jokser/disk-s3-shutdown
...
Disable S3 requests processing during context shutdown to speed up termination process
2020-09-08 15:21:54 +03:00
Pavel Kovalenko
5e39f8b32b
Add comment explaining DiskS3::shutdown
2020-09-08 15:20:35 +03:00
Alexey Milovidov
e3924b8057
Fix "Arcadia"
2020-09-08 01:14:13 +03:00
Pavel Kovalenko
c933f72adb
Disable S3 requests processing during context shutdown to speed up termination process.
2020-09-04 17:17:27 +03:00
Anton Popov
4331158d30
merge with master
2020-08-24 16:09:23 +03:00
Anton Popov
4df5c72878
Merge remote-tracking branch 'upstream/master' into HEAD
2020-08-24 12:57:27 +03:00
alexey-milovidov
7b53a0ef33
Revert "Added allow_merges option for volumes in multi-disk configuration ( #13402 )"
...
This reverts commit 1e2616542a
2020-08-21 18:44:29 +03:00
Vladimir Chebotarev
1e2616542a
Added allow_merges option for volumes in multi-disk configuration ( #13402 )
2020-08-21 12:04:13 +03:00
Alexander Kuzmenkov
ef5882639b
Merge pull request #13459 from Jokser/s3-multithreaded-disk-copy
...
Move parts from DiskLocal to DiskS3 in parallel
2020-08-12 07:36:03 +03:00
Pavel Kovalenko
ea46ee74ae
Don't allocate Executor instance on each disk->getExecutor() call.
2020-08-11 22:08:32 +03:00
Vladimir Chebotarev
7371cf6bd1
Proper remote host checking in S3 redirects ( #13404 )
...
* Proper remote host checking in S3 redirects.
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-08-11 09:09:08 +03:00
Pavel Kovalenko
bece61b1cc
Fix codestyle.
2020-08-10 14:40:31 +03:00
Pavel Kovalenko
f503273d41
Merge remote-tracking branch 'origin/master' into s3-multithreaded-disk-copy
...
# Conflicts:
# src/Disks/S3/DiskS3.cpp
2020-08-10 14:35:26 +03:00
Alexey Milovidov
b4ac900462
Fix "Arcadia" build
2020-08-09 02:38:53 +03:00
Alexander Kuzmenkov
6a596d5c40
Merge pull request #13076 from Jokser/s3-file-cache
...
Ability to cache mark and index files for S3 disk
2020-08-08 01:04:53 +03:00
Pavel Kovalenko
562102161b
Missed override.
2020-08-07 18:53:40 +03:00
Pavel Kovalenko
53ecd95302
Remove unused dependencies.
2020-08-07 14:44:35 +03:00
Pavel Kovalenko
a6ee09f696
Move parts to DiskS3 in parallel refactoring.
2020-08-07 14:40:19 +03:00
Pavel Kovalenko
7856852010
Move parts to DiskS3 in parallel.
2020-08-05 19:36:10 +03:00
Pavel Kovalenko
1703e5a1ca
Download through tmp file in DiskCacheWrapper.cpp
2020-08-03 15:37:53 +03:00
Alexey Milovidov
6f690b7c0d
Normalize ya.make files, fix "Arcadia" build
2020-08-02 16:57:38 +03:00
Pavel Kovalenko
a6d162961f
Fix directories creation in DiskCacheWrapper.cpp
2020-08-01 22:09:05 +03:00
Pavel Kovalenko
d7785f8c2d
Fix checkstyle.
2020-07-30 18:48:32 +03:00
Pavel Kovalenko
7a41fc1b9b
DiskS3 with cache tests.
2020-07-30 17:49:56 +03:00
alesapin
df0dcf4a31
Fix error codes
2020-07-30 17:27:11 +03:00
Pavel Kovalenko
00181d34fd
DiskS3 caching improvements.
2020-07-30 16:42:05 +03:00
alesapin
1c8aa4b6c1
Fix get disk interface
2020-07-30 14:08:56 +03:00
alesapin
d3ea1d914b
Tiny fixes and avoid dynamic casts
2020-07-30 13:04:49 +03:00
Pavel Kovalenko
ba8d770a76
Change comment in DiskCacheWrapper.cpp
2020-07-29 22:54:00 +03:00
Pavel Kovalenko
644d4c7aaa
Cache mark and index files for DiskS3.
2020-07-29 21:15:20 +03:00
Pavel Kovalenko
2b962e699b
Cache mark and index files for DiskS3.
2020-07-29 18:05:02 +03:00
Vladimir Chebotarev
faedb04722
Minor fixes.
2020-07-28 19:45:46 +03:00
Vladimir Chebotarev
6d22069f55
Review fixes.
2020-07-27 18:59:09 +03:00
Vladimir Chebotarev
ee21fde3fe
Review fixes.
2020-07-26 21:27:36 +03:00
Gleb Novikov
aac97957ce
Fixed integration test_tmp_policy
2020-07-26 21:27:36 +03:00
Gleb Novikov
99d52552e1
Small fixes for tests
2020-07-26 21:27:36 +03:00
Gleb Novikov
f4080ed6cc
removed unused error code
2020-07-26 21:27:36 +03:00
Gleb Novikov
e4699e297e
removed default clause in switch for VolumeType enum
2020-07-26 21:27:36 +03:00
Gleb Novikov
2c39584a44
Braces style fix
2020-07-26 21:27:36 +03:00
Gleb Novikov
adef249c34
Small fix of VolumeRAID1Ptr
2020-07-26 21:27:36 +03:00
Gleb Novikov
7f5b6fba78
Generic volume is coming...
...
1. SingleDiskVolume for temporary volumes
2. Generic VolumePtr in StoragePolicies
3. Removed max_data_part_size in system.storage_policies, added volume_type
2020-07-26 21:27:36 +03:00
Gleb Novikov
96c7cc8724
Small clang-10 warning fixes + ya.make
2020-07-26 21:27:36 +03:00
Gleb Novikov
db56820fab
Style fixes and added UNKNOWN_RAID_TYPE exception to createVolumeFromConfig
2020-07-26 21:27:36 +03:00
Gleb Novikov
3585b24f16
MultiDiskReservation, VolumeRAID1 and createVolumeFromConfig
...
1. Implemented MultiDiskReservation to maintain multiple reservations under one
2. Implemented VolumeRAID1 as child of VolumeJBOD (since RAID1 is quite similar)
3. Implemented createVolumeFromConfig
2020-07-26 21:27:36 +03:00
Alexey Milovidov
6df282e813
Fixup
2020-07-16 11:33:51 +03:00
Alexey Milovidov
a6016b95ce
Fix clang-tidy
2020-07-16 05:34:43 +03:00
Alexey Milovidov
3408b7e259
Merge branch 'master' into log-engine-rollback-on-insert-error
2020-07-16 05:34:02 +03:00
Vladimir Chebotarev
a479a20ba4
Review fixes.
2020-07-13 03:24:49 +03:00
Vladimir Chebotarev
d3ade3492e
Update DiskS3.h
2020-07-13 03:09:37 +03:00
alexey-milovidov
c6a5b77843
Update registerDiskS3.cpp
2020-07-13 03:04:46 +03:00
Vladimir Chebotarev
4802a785b5
Added read_seek_threshold setting to DiskS3.
2020-07-13 02:51:25 +03:00
Vladimir Chebotarev
718523b0ee
Working wrapper.
2020-07-13 02:47:22 +03:00
Alexey Milovidov
204a4af394
Rollback insertion error in Log engines #12402
2020-07-12 05:32:18 +03:00
alexey-milovidov
26d32f2ce5
Merge pull request #12412 from Jokser/s3-https
...
S3 HTTPS integration tests
2020-07-12 03:36:37 +03:00
Pavel Kovalenko
f43428a4d5
S3 HTTPS integration test.
2020-07-10 22:42:18 +03:00
Vladimir Chebotarev
faf6be6576
Implemented single part uploads for DiskS3 ( #12026 )
...
* Implemented single part uploads for DiskS3.
* Added `min_multi_part_upload_size` to disk configuration.
2020-07-09 17:09:17 +03:00
Alexander Kazakov
1c0c8a7bf6
Merge pull request #11892 from excitoon-favorites/relatives3paths
...
Switched paths in S3 metadata to relative
2020-07-07 00:13:07 +03:00
alexey-milovidov
ec563e54dc
Merge pull request #12138 from ClickHouse/codeql
...
Fix warnings from CodeQL
2020-07-05 18:22:49 +03:00
alexey-milovidov
c257255759
Merge pull request #12134 from matwey/fix/missed_include
...
Add missed <atomic> include in ProxyListConfiguration
2020-07-05 04:22:42 +03:00
Alexey Milovidov
96e8e93c64
Fix warnings from CodeQL
2020-07-05 02:16:16 +03:00
Matwey V. Kornilov
caafbe19a4
Add missed <atomic> include in ProxyListConfiguration
2020-07-04 15:32:02 +03:00
Anton Ivashkin
23b44ca6fe
Add type column in system.disks
2020-07-03 15:45:54 +03:00
Vladimir Chebotarev
8128ca10f4
Switched paths in S3 metadata to relative.
2020-06-30 07:59:22 +03:00
Alexey Milovidov
1462a66d1e
Fix typos
2020-06-27 22:05:00 +03:00
Anton Popov
b2aa565a37
allow to turn on fsync on inserts, merges and fetches
2020-06-27 17:52:16 +03:00
Anton Popov
f8f67a788e
allow to turn on fsync on inserts and merges
2020-06-25 19:55:45 +03:00
Alexey Milovidov
25f941020b
Remove namespace pollution
2020-05-31 00:57:37 +03:00
Alexey Milovidov
5aff138956
Preparation for structured logging
2020-05-31 00:35:52 +03:00
Alexey Milovidov
7e1813825b
Return old names of macros
2020-05-24 01:24:01 +03:00
Alexey Milovidov
18febd7b97
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_[^\_(]+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+"\);' | while read file; do perl -pne 's/(LOG_[^\_(]+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)"\);/${1}_FORMATTED(${2}, "${3}{}${5}{}${7}{}${9}{}${11}", ${4}, ${6}, ${8}, ${10});/' $file > ${file}.tmp; mv ${file}.tmp $file; done
2020-05-23 22:56:05 +03:00
Alexey Milovidov
d5b7883451
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_[^\_(]+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_[^\_(]+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5{}\7{}\9", \4, \6, \8);/'
2020-05-23 22:41:08 +03:00
Alexey Milovidov
0ab3ca8534
Apply all transformations again
2020-05-23 22:32:44 +03:00
Alexey Milovidov
29762240de
Remove duplicate whitespaces (preparation)
2020-05-23 22:31:54 +03:00
Alexey Milovidov
7fed65cbe2
Remove duplicate whitespaces (preparation)
2020-05-23 22:14:58 +03:00
Alexey Milovidov
86b4d5a86f
Remove duplicate whitespaces (preparation)
2020-05-23 22:09:06 +03:00
Alexey Milovidov
ab0562a574
Make all LOG in single line (preparation)
2020-05-23 22:05:41 +03:00
Alexey Milovidov
9d24908e53
Progress on task
2020-05-23 20:52:11 +03:00
Alexey Milovidov
241f8c5431
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" \+ [^+]+ \+ "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" \+ ([^+]+) \+ "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/'
2020-05-23 20:10:21 +03:00
Alexey Milovidov
f68d1ceb4f
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}\5{}\7{}", \4, \6, \8);/'
2020-05-23 20:02:09 +03:00
Alexey Milovidov
8042e5febe
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}\5{}", \4, \6);/'
2020-05-23 19:58:15 +03:00
Alexey Milovidov
e391b77d81
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/'
2020-05-23 19:56:05 +03:00
Alexey Milovidov
ee4ffbc332
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}", \4);/'
2020-05-23 19:47:56 +03:00
alexey-milovidov
ac6deaf8ec
Merge pull request #11036 from excitoon-favorites/s3fixes
...
Fixed parsing of S3 URL
2020-05-23 17:06:59 +03:00
alesapin
70fe119426
Add clear names
2020-05-22 13:33:57 +03:00
Vladimir Chebotarev
98f31374d5
Added improved S3 URL parsing to DiskS3.
2020-05-22 11:18:34 +03:00
Alexander Kuzmenkov
f98ffdbc4c
Merge pull request #11087 from azat/context-fwd-decl
...
[RFC] Forward declaration for Context as much as possible.
2020-05-21 19:43:29 +03:00
Azat Khuzhin
d93b9a57f6
Forward declaration for Context as much as possible.
...
Now after changing Context.h 488 modules will be recompiled instead of 582.
2020-05-21 01:53:18 +03:00
Pavel Kovalenko
d98c1589ed
Better error handling in Proxy Resolver.
2020-05-20 19:01:05 +03:00
Pavel Kovalenko
138154ba6a
Fix memory-leak in registerDiskS3 during config->createView invocations.
2020-05-20 18:42:45 +03:00
Oleg Ershov
8858319281
global update
2020-05-20 06:14:58 +03:00
Gleb Novikov
665c6b7f59
Added new files to ya make
2020-05-19 12:38:43 +03:00
alesapin
bec7d1ce93
Fix style check
2020-05-18 16:49:19 +03:00
alesapin
ab491c0dd9
boop CI
2020-05-18 11:42:39 +03:00
Gleb Novikov
1a25ac6e1f
Merge branch 'master' into refactor-reservations
2020-05-16 23:34:45 +03:00
Gleb Novikov
45b84d491e
small fixes of review remarks
2020-05-16 23:31:17 +03:00
Oleg Ershov
dd210cfeaf
Fix disk issues
2020-05-15 20:55:18 +03:00
Oleg Ershov
ae715a8d1b
Add base
2020-05-14 18:45:30 +03:00
Pavel Kovalenko
506634786e
Handle exceptions in ProxyResolverConfiguration properly.
2020-05-12 17:34:37 +03:00
Pavel Kovalenko
8bea42e250
Fix includes in ProxyResolverConfiguration.h
2020-05-12 12:23:18 +03:00
Gleb Novikov
fbd00d6af3
removed default value from overriden functions in IReservation children
2020-05-10 12:33:52 +03:00
Gleb Novikov
a961e049a2
VolumeType as class with enum member class
2020-05-10 09:26:33 +03:00
Gleb Novikov
390b39b272
VolumePtr instead of DiskPtr in MergeTreeData*
2020-05-10 00:24:15 +03:00
Gleb Novikov
b54209e43e
Modified reservation to work with multiple disks
2020-05-09 19:03:22 +03:00
Pavel Kovalenko
8d8dc3035c
Fix naming.
2020-05-08 16:10:21 +03:00
Pavel Kovalenko
8602f7b47c
Minor updates in proxy resolver parameters naming.
2020-05-08 13:58:56 +03:00
Pavel Kovalenko
6165c2aa99
Add possibility to use proxy resolver in DiskS3.
2020-05-08 13:53:12 +03:00
Gleb Novikov
6e6a195c13
Style fixes, removed redundant includes
2020-05-04 23:20:51 +03:00
Gleb Novikov
c637d99e07
Volumes and storages refactoring:
...
1. Moved Volume to separate file
2. Created IVolume interface and implemented current behaviour in implementation of new interface — VolumeJBOD
3. Replaced all old volume usages with new VolumeJBOD. Where it is unnecessary to have JBOD — left just IVolume.
4. Removed old Volume completely
5. Moved StoragePolicy to separated files
6. Moved DiskSelector to separated files
7. Removed DiskSpaceMonitor file
2020-05-04 23:15:38 +03:00
Pavel Kovalenko
0b70c2ca08
Fixed ya.make for S3.
2020-04-29 21:38:35 +03:00
Pavel Kovalenko
f17e022fa4
Code style fixes.
2020-04-29 15:05:46 +03:00
Pavel Kovalenko
a1acbcb019
Code style fixes.
2020-04-29 11:45:13 +03:00
Pavel Kovalenko
e5e84960ea
S3 with proxy integration test and fixes.
2020-04-29 11:40:29 +03:00
Pavel Kovalenko
2ecf4b5a23
Dynamic proxy configuration for DiskS3.
2020-04-29 11:40:29 +03:00
Alexey Milovidov
be22a4b94e
Checkpoint
2020-04-22 08:39:31 +03:00
Ivan Lezhankin
e230632645
Changes required for auto-sync with Arcadia
2020-04-16 15:31:57 +03:00
Ivan Lezhankin
06446b4f08
dbms/ → src/
2020-04-03 18:14:31 +03:00
