Kseniia Sumarokova
9ae047df1d
Update CachedReadBufferFromRemoteFS.cpp
2022-04-13 14:30:10 +02:00
kssenii
a6dc437911
Merge master
2022-04-13 14:23:48 +02:00
Kseniia Sumarokova
1b267462d8
Update CachedReadBufferFromRemoteFS.cpp
2022-04-13 11:27:47 +02:00
kssenii
37242a0103
Allow non-evictable file segments based on predicate
2022-04-12 16:55:25 +02:00
kssenii
9804c39de7
Composable cache
2022-04-12 14:33:13 +02:00
kssenii
d2ccf876c6
Fix
2022-04-11 17:51:49 +02:00
Raúl Marín
6e346a7fc3
Support atomic replace under OSX
2022-04-11 14:32:55 +02:00
alesapin
324020a46f
Restore class
2022-04-10 15:59:43 +02:00
alesapin
f61cad144c
Get rid of strange abstraction
2022-04-10 15:55:05 +02:00
Kseniia Sumarokova
867b85dfad
Merge branch 'master' into change-hashed-path
2022-04-09 16:57:57 +02:00
kssenii
138588be29
Change hashed path
2022-04-08 17:14:13 +02:00
alesapin
4a03402da5
Update ReadBufferFromRemoteFSGather.h
2022-04-08 16:41:37 +02:00
alesapin
60e36abfe9
Get rid of metadata in buffers
2022-04-08 16:03:00 +02:00
Sergei Trifonov
fd001b5c80
fix param order
2022-04-08 15:51:53 +02:00
alesapin
ad9fdc1d79
Don't use metadata path in write buffers
2022-04-08 15:30:46 +02:00
alesapin
0db8808f2e
Remove unused field from buffers
2022-04-08 15:06:43 +02:00
kssenii
87855c4ae7
Merge master
2022-04-08 12:32:48 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements
...
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
2022-04-08 12:06:26 +02:00
kssenii
0b7af1f26c
Fix checks
2022-04-08 01:58:55 +02:00
kssenii
5dce2f18b5
Better
2022-04-07 18:48:35 +02:00
Sergei Trifonov
9c2804b924
merge from master + resolve conflicts
2022-04-07 11:22:01 +02:00
vdimir
4cb7b7e49b
Create parent directories in DiskLocal::replaceFile
2022-04-06 11:15:37 +00:00
kssenii
6c6fb5c3e8
Fix race in cached buffer
2022-04-04 16:32:37 +02:00
kssenii
4e50da722b
Better s3 settings
2022-04-04 16:14:56 +02:00
kssenii
6c8e073a61
Merge master
2022-04-01 16:48:29 +02:00
alesapin
f2c6387a8d
Merge pull request #35381 from kssenii/remote-filesystem-cache-fixes
...
Cache fixes for high concurrency on corner cases
2022-04-01 10:48:31 +02:00
alesapin
e12860f646
Update src/Disks/IO/CachedReadBufferFromRemoteFS.cpp
2022-03-31 11:42:42 +02:00
alesapin
4c66c2e10f
Update src/Disks/IO/CachedReadBufferFromRemoteFS.cpp
2022-03-31 11:42:38 +02:00
kssenii
0fc92fe2aa
Get rid of all "remote" mentions
2022-03-30 17:15:29 +02:00
kssenii
98ad3f4a91
Fix tests, rename some remote mentions
2022-03-30 12:21:59 +02:00
Antonio Andelic
03a09dc914
Merge pull request #35571 from ClickHouse/parallel-s3-downloading
...
Parallel S3 downloading
2022-03-30 08:14:49 +02:00
kssenii
f0e0d97752
Fix lask of query id in merges
2022-03-29 19:49:42 +02:00
kssenii
421b1e5a81
Update
2022-03-29 17:33:02 +02:00
Antonio Andelic
9990abb76a
Use compile-time check for Exception messages, fix wrong messages
2022-03-29 13:16:11 +00:00
Antonio Andelic
f0b4a9fd15
Fix construction of ReadBufferFromS3
2022-03-29 10:56:49 +00:00
kssenii
eab04674fa
Fix review comments
2022-03-29 12:38:17 +02:00
Azat Khuzhin
6da2c48b7a
Add missing __builtin_unreachable for switch/case for gcc
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-26 23:42:21 +03:00
kssenii
141c290344
Fix build
2022-03-25 18:31:15 +01:00
kssenii
e59dbe72c7
Revert accidental commit
2022-03-25 17:53:02 +01:00
kssenii
9aaba43a0a
Remove incorrect assertion
2022-03-25 16:29:30 +01:00
kssenii
0c6577eee0
Merge master
2022-03-24 20:49:02 +01:00
kssenii
6769b84638
Merge master
2022-03-24 20:47:43 +01:00
kssenii
f9b538fb0c
Fix
2022-03-24 15:38:29 +01:00
kssenii
d6ab6de30a
Minor changes
2022-03-24 15:32:08 +01:00
kssenii
d4161b5925
Add optin read_from_cache_if_exists_otherwise_bypass_cache (for merges)
2022-03-23 20:24:00 +01:00
kssenii
d2a3cfe5dc
Cache on all write operations
2022-03-23 19:14:33 +01:00
kssenii
e39aba37a2
Minor changes
2022-03-23 18:11:52 +01:00
Sergei Trifonov
bf9c6be6c6
move threadPoolCallbackRunner from Common into Interpreters folder to fix split build issue
2022-03-23 17:57:19 +01:00
kssenii
34c0690a69
Better
2022-03-23 15:43:14 +01:00
ianton-ru
32c6390400
Merge branch 'master' into MDB-16962
2022-03-23 16:53:30 +03:00
Anton Ivashkin
66c026d0ac
Fix collision of S3 operation log revision
2022-03-23 16:48:05 +03:00
kssenii
f3e1ca44a9
Add system.remote_data_paths table
2022-03-23 13:01:18 +01:00
kssenii
ccdf7d0231
Fix typos check
2022-03-21 21:51:17 +01:00
kssenii
afd0c64a1a
Better
2022-03-21 15:04:16 +01:00
kssenii
eabbce69a7
Add system drop remote filesystem cache query, allow to write cache on insert
2022-03-21 12:32:44 +01:00
kssenii
59b7394caf
WriteSettings
2022-03-21 09:52:48 +01:00
kssenii
1d80262a2a
Add write settings, file cache settings
2022-03-21 09:50:50 +01:00
mergify[bot]
7ac606fa65
Merge branch 'master' into parallel-multipart-upload-for-s3storage
2022-03-20 18:25:28 +00:00
kssenii
43ed2beb3c
Merge master
2022-03-18 12:33:08 +01:00
kssenii
af3bb3b7af
Add WriteSettings
2022-03-18 12:27:38 +01:00
kssenii
44fb6fb86f
Revert some changes
2022-03-18 10:16:06 +01:00
kssenii
a7c9fe5c78
Finally-finally fixed
2022-03-17 20:29:07 +01:00
kssenii
c698fb9539
Debug
2022-03-17 18:24:00 +01:00
kssenii
1b6e7fea2f
Finally fixed
2022-03-17 16:40:10 +01:00
Nikolai Kochetov
ee9c2ec735
Merge pull request #34780 from azat/mt-delayed-part-flush
...
Do not delay final part writing by default (fixes possible Memory limit exceeded during INSERT)
2022-03-17 12:30:51 +01:00
kssenii
f354b704e0
Add more checks, different counting for download size
2022-03-17 11:52:59 +01:00
Sergei Trifonov
bbe657ba24
allow parallelization of multipart S3 storage upload fixes #34350
2022-03-16 19:41:41 +01:00
Nikolai Kochetov
a380aa6b8a
Merge pull request #35294 from ClickHouse/reload-remote_url_allow_hosts
...
Reload remote_url_allow_hosts after config update.
2022-03-15 22:07:16 +01:00
Nikolai Kochetov
cf260b8508
Merge pull request #35305 from ClickHouse/try-to-fix-using-deleted-memory-tracker-inside-s3-disk
...
Maybe fix use-after-free inside S3 upload thread
2022-03-15 22:06:17 +01:00
Nikolai Kochetov
b4aed421de
Reset THreadPool's thread memory tracker parent to global after S3 upload task is finished.
2022-03-15 15:33:43 +00:00
Nikolai Kochetov
97aa6c82ce
Reload remote_url_allow_hosts after config update.
2022-03-15 13:00:31 +00:00
Saad Ur Rahman
7bdbaf28d8
[Disks] bugfix 🤦♂️
...
Correcting order of parameters for <ReadBufferFromS3> in <readSchemaVersion>.
2022-03-14 22:42:09 -04:00
Saad Ur Rahman
40fbea49f7
[Disks] bugfix
...
Fixed order of parameters to <DiskS3> constructor:
- name
- bucket
- key
- versionId
- ...
2022-03-14 22:37:47 -04:00
Saad Ur Rahman
fea8824489
[Disks] Added Version ID parameter to registerDisksS3.
2022-03-14 22:06:09 -04:00
Saad Ur Rahman
e181207f2d
[Disks] Version ID parameter added to readFile
2022-03-14 21:30:23 -04:00
Saad Ur Rahman
98b2994352
[Disks] ReadBufferFromS3 requires a version id.
2022-03-14 21:30:23 -04:00
Saad Ur Rahman
e3a96393a1
[Disks] Version ID added.
...
Version ID added to constructor.
2022-03-14 21:30:23 -04:00
Saad Ur Rahman
f299bf46e2
[Disks] readSchemaVersion supplies empty Version ID
...
Empty version ID supplied to retrieve version ID of object.
2022-03-14 20:11:20 -04:00
mergify[bot]
ee3f4fb0c3
Merge branch 'master' into add-cache-optimization
2022-03-14 20:58:52 +00:00
Maksim Kita
538f8cbaad
Fix clang-tidy warnings in Disks, Formats, Functions folders
2022-03-14 18:17:35 +00:00
kssenii
fc9c6eab9d
Add optimization
2022-03-14 17:58:19 +01:00
alesapin
b149f28cd3
Merge pull request #35209 from kssenii/cache-fix
...
Cache fix
2022-03-14 11:07:40 +01:00
Kseniia Sumarokova
35e5b4e8a5
Update CachedReadBufferFromRemoteFS.cpp
2022-03-13 12:37:00 +01:00
kssenii
a18bb606a2
Update
2022-03-11 14:24:02 +01:00
kssenii
2b478e2501
Fix
2022-03-11 12:17:17 +01:00
kssenii
c86955deb2
Fix typo
2022-03-10 20:22:39 +01:00
kssenii
ddcb020d26
Fix test
2022-03-10 19:16:26 +01:00
kssenii
6571c7faa0
Clean up code
2022-03-09 10:36:52 +01:00
kssenii
96789fa5ce
Fix test, disable cache for DiskCacheWrapper
2022-03-09 09:46:04 +01:00
kssenii
16e84c265b
Fix build, update test
2022-03-08 21:41:58 +01:00
Azat Khuzhin
4200b56f3a
Introduce IDisk::supportParallelWrite()
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-08 22:17:36 +03:00
kssenii
5260822964
Merge master
2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0
Fix split build
2022-03-08 18:05:55 +01:00
kssenii
f3cba6f270
Remove already fixed crutch
2022-03-08 14:30:21 +01:00
kssenii
28a3aece60
Fix uncaught exception from destructor
2022-03-08 10:58:37 +01:00
Kseniia Sumarokova
517e878c6e
Merge pull request #35099 from ClickHouse/tavplubix-patch-1
...
Fix inconsistency in DiskLocal
2022-03-08 10:18:07 +01:00
kssenii
0737aef472
Fix split build
2022-03-07 22:02:07 +01:00
kssenii
eeb0ae9816
Fix detaching query
2022-03-07 17:05:36 +01:00
tavplubix
84e22fb32b
Update DiskLocal.cpp
2022-03-07 18:59:00 +03:00
kssenii
059d239815
Simplify code
2022-03-07 15:17:38 +01:00
kssenii
692b247379
Fix some corner cases
2022-03-07 14:31:21 +01:00
mergify[bot]
88052e2d7c
Merge branch 'master' into finally-enable-s3-async-writes-again
2022-03-07 12:55:52 +00:00
kssenii
2dcfe66236
Fix race
2022-03-06 20:33:07 +01:00
Kseniia Sumarokova
299e11bb2b
Update DiskCacheWrapper.cpp
2022-03-03 11:11:57 +01:00
kssenii
192b839cab
Fix
2022-03-02 19:11:45 +01:00
kssenii
b2cab429a7
Move initialization out of constructor
2022-03-01 18:12:34 +01:00
kssenii
438c63f3d8
Style
2022-03-01 17:52:38 +01:00
Nikolai Kochetov
eaf6d8c5f6
Update DiskS3.cpp
2022-02-24 17:35:29 +01:00
kssenii
8d6f1982df
Tiny refactoring
2022-02-23 14:43:40 +01:00
kssenii
36a76d6a75
Fix test
2022-02-22 15:18:28 +01:00
kssenii
c637385dd0
Merge master
2022-02-22 13:17:51 +01:00
Kseniia Sumarokova
eeea322556
Merge pull request #34629 from amosbird/remotefsimprove
...
Some refactoring and improvement over async and remote buffer related stuff
2022-02-22 11:36:40 +01:00
kssenii
fad7c042c1
Fix cache with non-async reads
2022-02-21 19:50:27 +01:00
kssenii
4543513a5d
Add comments
2022-02-21 13:54:03 +01:00
kssenii
e8b2e47e93
Style
2022-02-19 20:38:23 +01:00
kssenii
6b60f7c57f
Add limit for file segment size and split it if exceeds the limit
2022-02-19 18:59:22 +01:00
Amos Bird
84d111cde9
Better ReadBuffer asynchronicity
2022-02-20 01:08:21 +08:00
Nikolai Kochetov
40a48c5670
Enable async writes to S3.
2022-02-18 18:32:04 +00:00
kssenii
acf9a092a9
Review fixes
2022-02-18 19:09:54 +01:00
kssenii
353ffacca3
Add test
2022-02-17 19:57:04 +01:00
Amos Bird
dc0b4a8f73
Remove useless template
2022-02-17 14:31:21 +08:00
kssenii
47f94120da
Merge master
2022-02-16 14:43:28 +01:00
kssenii
3bd3e51aa0
Fix tests
2022-02-16 14:08:41 +01:00
Vladimir C
724340418e
Merge pull request #34617 from Algunenano/freebsd_build
...
Attempt to fix freeBSD build
2022-02-15 20:12:44 +01:00
Raúl Marín
72efc00830
Attempt to fix freeBSD build
2022-02-15 17:23:01 +01:00
alesapin
bc2d0ee7c7
Merge pull request #34215 from ClickHouse/revert-34211-revert-34153-add_func_tests_over_s3
...
Add func tests run with s3 and fix several bugs
2022-02-15 19:07:11 +03:00
kssenii
886b300b8d
Less seeks
2022-02-15 16:12:25 +01:00
mergify[bot]
34973f1d5c
Merge branch 'master' into first-test
2022-02-15 12:15:53 +00:00
alesapin
447cd56cb9
Fix comments
2022-02-15 12:11:50 +03:00
alesapin
e15396d90c
Fix race condition:
2022-02-14 22:19:49 +03:00
kssenii
bdbbe85f95
Fixes
2022-02-14 19:28:48 +01:00
zxealous
3d29920005
Fix compilation issues with DiskHDFS
2022-02-14 13:14:20 +08:00
Danila Kutenin
1c589681a0
Have a correct include which was inlined
2022-02-12 12:49:23 +00:00
alesapin
088f8cacf3
Small improvements
2022-02-10 19:50:21 +03:00
Anton Popov
70986a70a1
support TTL TO [DISK|VOLUME] [IF EXISTS]
2022-02-10 19:26:23 +03:00
alesapin
3387f9e597
Possible fix DiskHDFS
2022-02-10 16:55:12 +03:00
alesapin
10c3e6e546
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-09 14:14:58 +03:00
alesapin
36909a986f
Fix bug with files remove
2022-02-08 22:21:16 +03:00
alesapin
02a93cb852
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-08 19:42:27 +03:00
alesapin
f6e7cea1f8
Add settings for S3 multipart upload parts size
2022-02-08 19:38:04 +03:00
alesapin
07d5227568
Disable thread pool
2022-02-08 14:08:00 +03:00
alesapin
773d9cede3
Fix cache wrapper
2022-02-08 14:05:38 +03:00
alesapin
3af6012cb4
Revert "Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer""""
...
This reverts commit 2bc2ea485e
2022-02-08 11:01:26 +03:00
alesapin
2bc2ea485e
Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"""
...
This reverts commit fb77d7a7d5
2022-02-08 10:56:29 +03:00
kssenii
2e58733750
Merge master
2022-02-07 15:23:10 +01:00
Anton Popov
1b16db72c3
fix consecutive backward seeks in seekable read buffers
2022-02-07 17:20:26 +03:00
Nikolai Kochetov
40b86e3dae
Add blind change.
2022-02-07 12:02:21 +00:00
alesapin
fb77d7a7d5
Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer""
...
This reverts commit 875e5413ad
2022-02-07 12:36:54 +03:00
alesapin
25c0b07e68
Disable some tests
2022-02-04 14:31:58 +03:00
alesapin
875e5413ad
Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"
...
This reverts commit b92efed350ecce006cb2
2022-02-04 14:30:33 +03:00
alesapin
5e049fdd47
Merge with master
2022-02-04 12:28:19 +03:00
alesapin
2ed45b2a98
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-04 11:23:46 +03:00
alesapin
85fa727e32
Fix tidy
2022-02-04 11:21:03 +03:00
alesapin
a61b231ddd
Better debug
2022-02-03 11:23:09 +03:00
alesapin
ab546f61e3
Disable cache by default
2022-02-02 22:06:08 +03:00
Nikolai Kochetov
2003a96b58
Probably fix something
2022-02-02 17:48:41 +00:00
alesapin
80800e051e
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-02 19:42:04 +03:00
Nikolai Kochetov
ea044fc6b2
Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer
2022-02-02 19:40:41 +03:00
alesapin
b9c118524f
Fix race condition on hardlink/erase/read metadata
2022-02-02 19:40:21 +03:00
kssenii
aeec749173
Fix predownload case part 1
2022-02-02 17:25:25 +03:00
kssenii
5cba11428a
Fixes
2022-02-02 00:25:16 +03:00
Azat Khuzhin
bedf208cbd
Use fmt::runtime() for LOG_* for non constexpr
...
Here is oneliner:
$ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'
Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:
$ cat fmt.cocci
@@
expression log;
expression var;
@@
-LOG_DEBUG(log, var)
+LOG_DEBUG(log, fmt::runtime(var))
I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-01 14:30:03 +03:00
Amos Bird
e29c77f793
Fix link order issue of WriteBufferFromS3
...
(cherry picked from commit f22b09f4fc
2022-02-01 10:42:06 +00:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3""
2022-02-01 13:36:51 +03:00
alesapin
18f08ed932
Revert "Revert "Add func tests run with s3""
2022-02-01 10:20:06 +03:00
alexey-milovidov
99392b5ca7
Merge pull request #13544 from amosbird/mdha
...
Multi-Disk auto-recovery.
2022-02-01 06:13:26 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3"
2022-02-01 05:49:40 +03:00
alexey-milovidov
cbfcd45be3
Revert "Add func tests run with s3"
2022-02-01 05:46:13 +03:00
alexey-milovidov
a2aa147ce0
Merge pull request #34153 from ClickHouse/add_func_tests_over_s3
...
Add func tests run with s3
2022-02-01 05:46:09 +03:00
Amos Bird
ec7d367814
DiskLocal checker
...
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
2022-02-01 05:55:27 +08:00
alesapin
cbf4790ea4
More clear error
2022-01-31 13:59:16 +03:00
kssenii
218a633fec
Some fixes, more assertions and logging
2022-01-31 10:51:34 +03:00
kssenii
6b03c577d6
Make downloader's term resticted to one nextImpl
2022-01-29 23:58:53 +03:00
Nikolai Kochetov
1c9f026178
Merge branch 'master' into add-pool-to-s3-write-buffer
2022-01-28 16:01:42 +03:00
kssenii
c91b86e220
Better
2022-01-26 21:56:08 +03:00
Nikolai Kochetov
a8171269a1
Review fixes.
2022-01-26 17:55:24 +00:00
Nikolai Kochetov
efaed154d2
Update src/Disks/IDisk.h
...
Co-authored-by: alesapin <alesapin@clickhouse.com>
2022-01-26 14:58:38 +03:00
Nikolai Kochetov
2e88319b72
Fix integration tests.
2022-01-25 18:48:28 +00:00
Nikolai Kochetov
fcc29dbd15
Try to fix integration tests.
2022-01-25 15:26:36 +00:00
kssenii
aef99dedba
Fixes
2022-01-25 01:46:44 +03:00
Nikolai Kochetov
4afa75e80b
Fix deduplication.
2022-01-24 14:43:36 +00:00
kssenii
898139acae
Fix tests
2022-01-24 12:32:39 +03:00
kssenii
710bba895e
Fix partially downloaded no continuation case
2022-01-23 23:49:13 +03:00
kssenii
cf49f41f75
Better
2022-01-23 19:51:18 +03:00
zhongyuankai
040b71617c
Fix disk using the same path
2022-01-23 16:08:19 +08:00
alexey-milovidov
d1431fffd2
Merge pull request #33912 from amosbird/fixdiskcacherace
...
Fix race in DiskCacheWrapper
2022-01-23 08:47:44 +03:00
kssenii
690db142b9
Better
2022-01-23 02:12:33 +03:00
Kseniia Sumarokova
2c43e4eb57
Merge branch 'master' into local-cache-for-remote-fs
2022-01-22 22:53:55 +03:00
kssenii
36a41ac146
Better
2022-01-22 20:42:39 +03:00
Amos Bird
da66239864
Fix race in DiskCacheWrapper
2022-01-22 23:56:30 +08:00
kssenii
9f295ebf92
Fix threadpool reads
2022-01-22 01:07:20 +03:00
Maksim Kita
a3adf12a50
Merge pull request #33862 from ClickHouse/fix-s3-proxy-config-resolver-ipv6-only
...
Try every resolved ip address while getting S3 proxy.
2022-01-21 20:29:19 +01:00
Nikolai Kochetov
a024ddd1d4
Merge branch 'add-pool-to-s3-write-buffer' of github.com:ClickHouse/ClickHouse into add-pool-to-s3-write-buffer
2022-01-21 18:42:24 +00:00
Nikolai Kochetov
b3cbb63487
Merge branch 'master' into add-pool-to-s3-write-buffer
2022-01-21 21:41:54 +03:00
Nikolai Kochetov
ae1b669fca
Even more async writes.
2022-01-21 15:55:19 +00:00
kssenii
1858f69f5a
Make space reservation incremental
2022-01-21 18:39:34 +03:00
Nikolai Kochetov
a178e54d10
Try every resolved ip address while getting S3 proxy.
2022-01-21 13:19:25 +03:00
kssenii
a566099759
Local cache for remote filesystem
2022-01-19 00:42:11 +03:00
Anton Ivashkin
7c26a832f8
Fix restore S3 disk without detached directories
2022-01-18 12:31:31 +03:00
Nikolai Kochetov
6d49a62666
Some more async writes.
2022-01-14 19:53:55 +00:00
Nikolai Kochetov
843983ea06
Add batch removing to IDiskl.
2022-01-12 15:58:04 +00:00
alesapin
733ed7cb54
Merge pull request #32061 from ianton-ru/MDB-15474
...
Change ZooKeeper path for zero-copy locks for shared data
2022-01-08 17:17:11 +03:00
xiedeyantu
1edfc03825
Update AsynchronousReadIndirectBufferFromRemoteFS.h
...
update 1024 * 1024 to DBMS_DEFAULT_BUFFER_SIZE
2022-01-07 21:06:53 +08:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474
2022-01-07 10:12:25 +00:00
Azat Khuzhin
71a99ab27c
Accept real file size in createReadBufferFromFileBase()
...
Right now streams relies on correct file size not the number of bytes
that will be read from the stream, to overcome one bug in the linux
kernel that may return EIINVAL for pread() with offset pass the EOF.
v2: Swap read_hint and file_size (since it is easy to miss something)
Before the first argument to readFile()/createReadBufferFromFileBase()
was read_hint, not the file_size, and let's preserve the order, since
it is easy to miss something
This will also fix 02051_read_settings test automatically because now
MergeTreeReaderStream will pass estimated_sum_mark_range_bytes to
read_hint not file_size, previously it cause on of the following errors:
- Attempt to read after EOF w/ O_DIRECT
- and LOGICAL_ERROR while adjusting granulas w/o O_DIRECT
This will also improve zero-length reads guard (via
ReadBufferFromEmptyFile), that had been added in #30190
v3: fix for other storages that wasn't enabled in fast-test
v4: ignore ENOENT/ENOTSUP in readFile
2022-01-04 10:53:17 +03:00
alesapin
7a3c874651
Merge branch 'master' into ianton-ru-MDB-15474
2021-12-29 19:01:43 +03:00
Nikolai Kochetov
68bf38133a
Add pool to WriteBufferFromS3
2021-12-29 14:04:21 +00:00
Anton Ivashkin
2d87f0a0e3
Fix debug build
2021-12-28 11:45:38 +03:00
Alexey Milovidov
f03cb4c762
Cleanup trash from Kafka and HDFS
2021-12-25 06:10:59 +03:00
Maksim Kita
60d16afde7
Merge pull request #33027 from ClickHouse/merge-33023
...
Merge #33023
2021-12-22 13:44:14 +03:00
Kseniia Sumarokova
484a4d62dd
Merge pull request #32835 from kssenii/fising-s3
...
Fix optimization with lazy seek for async reads from remote fs
2021-12-22 09:38:23 +03:00
Alexey Milovidov
7b48cb813e
Merge branch 'Issue129' of github.com:DevTeamBK/ClickHouse into merge-33023
2021-12-22 02:17:00 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474
2021-12-21 17:38:36 +03:00
Anton Ivashkin
0c0bf66334
Merge master
2021-12-21 17:27:54 +03:00
kssenii
ebcb66a361
Fix
2021-12-21 10:58:02 +03:00
Nikolai Kochetov
35883e0dae
Merge pull request #32979 from ClickHouse/decrease-some-s3-log-level
...
Decrease log level for some s3 messages.
2021-12-20 20:56:22 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment
...
Loops remove postfix increment
2021-12-20 19:51:07 +03:00
Nikolai Kochetov
7dea7b7f76
Decrease log level for some s3 messages.
2021-12-20 18:18:54 +03:00
Anton Ivashkin
f0b9a4327a
Use table UUID in zero-copy shared label in ZooKeeper
2021-12-20 16:19:00 +03:00
Maksim Kita
51477adf1b
Updated additional cases
2021-12-20 15:55:07 +03:00
Maksim Kita
3feab5a975
Containers iteration fix erase
2021-12-20 13:42:31 +03:00
root
6b3366dc70
[Issue #129 ] Fix Weak Regular Expression [v21.3-insecure]
...
Problem Descripton:
==================
The “key_regexp” constant in the “DiskS3” class contains a regular expression (RegEx) for checking a key path.
However, the RegEx starts and ends with “.*”.
This is a wildcard that an attacker could use to exploit this file path check by padding the beginning and end with whatever path they’d like.
Recommendation:
==============
Tighten up the RegEx by replacing the “.*” at the beginning and end with more specific items.
Fix:
===
Replace the .* at the end with $ as there is no need to continue after the key file.
Leave the beginning .* as is because the path for the key can have any characters or in any path of the system.
2021-12-17 10:30:16 -05:00
kssenii
40c266840c
Fixes
2021-12-17 18:16:29 +03:00
kssenii
ed42620530
Rename BlobStorage to AzureBlobStorage
2021-12-17 12:24:05 +03:00
kssenii
ca283a4bb8
Fix optimization with lazy seek
2021-12-17 00:18:28 +03:00
alesapin
c338bee718
Revert "Fix build issue related to azure blob storage"
2021-12-16 14:37:12 +03:00
alesapin
c0cebaea79
Merge pull request #32788 from amosbird/buildfix
...
Fix build issue related to azure blob storage
2021-12-16 13:45:22 +03:00
Amos Bird
9c9f4a3e6c
build fix
2021-12-15 16:55:18 +08:00
alesapin
9974f10631
Fix clang tidy
2021-12-14 15:34:20 +03:00
Kseniia Sumarokova
78ed383f15
Merge pull request #31505 from ContentSquare/azure_blob_storage_dependency
...
Azure Blob Storage Disks
2021-12-13 18:28:48 +03:00
Azat Khuzhin
57c027be0e
Remove arcadia build support
2021-12-11 21:25:23 +03:00
Jakub Kuklis
578aec97e9
Removing TODO and NOTE marks used mainly as hooks for review in the PR
2021-12-10 22:04:38 +01:00
Jakub Kuklis
a3bc6d2fb7
Style corrections
2021-12-10 22:04:38 +01:00
Jakub Kuklis
74e26354cc
Committing missing files
2021-12-10 22:04:38 +01:00
Jakub Kuklis
41a88e49c9
Abstracting metadata creation for remote disks registrations, putting Blob Storage auth in a separate file
2021-12-10 22:04:38 +01:00
Jakub Kuklis
e58f74d8dc
Connecting to Azurite and invoking simple INSERTs works in the integration test
2021-12-10 22:04:38 +01:00
Jakub Kuklis
8afe69ff93
Comming back to StorageSharedKeyCredential for the integration test, get 400 Bad requests from Azurite when trying to create a container now
2021-12-10 22:04:38 +01:00
Jakub Kuklis
f180a7ba5f
Solved Invalid storage account error with connection strings
2021-12-10 22:04:38 +01:00
Jakub Kuklis
008141de36
Enable authentication via StorageSharedKeyCredential
2021-12-10 22:04:38 +01:00
Jakub Kuklis
d51f979c7a
Adding endpoint validation
2021-12-10 22:04:38 +01:00
Jakub Kuklis
2b7725bfc7
Enable connecting directly to storage accounts and creating new containers
2021-12-10 22:04:38 +01:00
Jakub Kuklis
910db6ca57
Code cleanup addressing PR remarks
2021-12-10 22:04:38 +01:00
Jakub Kuklis
f6c5a2c63c
Removing file checks after changing blob filenames
2021-12-10 22:04:38 +01:00
Jakub Kuklis
3aa6a0ead3
Adding random postfix to blob names, needs code cleanup
2021-12-10 22:04:38 +01:00
Jakub Kuklis
5ed511f9da
Working skeleton for Blob Storage integration test
2021-12-10 22:04:37 +01:00
Jakub Kuklis
0c65cfed1f
Addressing first PR remarks
2021-12-10 22:04:37 +01:00
Jakub Kuklis
7c2e116214
Adding max_retries config
2021-12-10 22:04:37 +01:00
Jakub Kuklis
a33cea0525
Refactors after proper rebase on master
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b3728dbbaf
Rebase on master
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b8ef47006c
Minor refactors before PR
2021-12-10 22:04:37 +01:00
