kssenii
421b1e5a81
Update
2022-03-29 17:33:02 +02:00
Antonio Andelic
9990abb76a
Use compile-time check for Exception messages, fix wrong messages
2022-03-29 13:16:11 +00:00
Antonio Andelic
f0b4a9fd15
Fix construction of ReadBufferFromS3
2022-03-29 10:56:49 +00:00
kssenii
eab04674fa
Fix review comments
2022-03-29 12:38:17 +02:00
Azat Khuzhin
6da2c48b7a
Add missing __builtin_unreachable for switch/case for gcc
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-26 23:42:21 +03:00
kssenii
141c290344
Fix build
2022-03-25 18:31:15 +01:00
kssenii
e59dbe72c7
Revert accidental commit
2022-03-25 17:53:02 +01:00
kssenii
9aaba43a0a
Remove incorrect assertion
2022-03-25 16:29:30 +01:00
kssenii
0c6577eee0
Merge master
2022-03-24 20:49:02 +01:00
kssenii
6769b84638
Merge master
2022-03-24 20:47:43 +01:00
kssenii
f9b538fb0c
Fix
2022-03-24 15:38:29 +01:00
kssenii
d6ab6de30a
Minor changes
2022-03-24 15:32:08 +01:00
kssenii
d4161b5925
Add optin read_from_cache_if_exists_otherwise_bypass_cache (for merges)
2022-03-23 20:24:00 +01:00
kssenii
d2a3cfe5dc
Cache on all write operations
2022-03-23 19:14:33 +01:00
kssenii
e39aba37a2
Minor changes
2022-03-23 18:11:52 +01:00
Sergei Trifonov
bf9c6be6c6
move threadPoolCallbackRunner from Common into Interpreters folder to fix split build issue
2022-03-23 17:57:19 +01:00
kssenii
34c0690a69
Better
2022-03-23 15:43:14 +01:00
ianton-ru
32c6390400
Merge branch 'master' into MDB-16962
2022-03-23 16:53:30 +03:00
Anton Ivashkin
66c026d0ac
Fix collision of S3 operation log revision
2022-03-23 16:48:05 +03:00
kssenii
f3e1ca44a9
Add system.remote_data_paths table
2022-03-23 13:01:18 +01:00
kssenii
ccdf7d0231
Fix typos check
2022-03-21 21:51:17 +01:00
kssenii
afd0c64a1a
Better
2022-03-21 15:04:16 +01:00
kssenii
eabbce69a7
Add system drop remote filesystem cache query, allow to write cache on insert
2022-03-21 12:32:44 +01:00
kssenii
59b7394caf
WriteSettings
2022-03-21 09:52:48 +01:00
kssenii
1d80262a2a
Add write settings, file cache settings
2022-03-21 09:50:50 +01:00
mergify[bot]
7ac606fa65
Merge branch 'master' into parallel-multipart-upload-for-s3storage
2022-03-20 18:25:28 +00:00
kssenii
43ed2beb3c
Merge master
2022-03-18 12:33:08 +01:00
kssenii
af3bb3b7af
Add WriteSettings
2022-03-18 12:27:38 +01:00
kssenii
44fb6fb86f
Revert some changes
2022-03-18 10:16:06 +01:00
kssenii
a7c9fe5c78
Finally-finally fixed
2022-03-17 20:29:07 +01:00
kssenii
c698fb9539
Debug
2022-03-17 18:24:00 +01:00
kssenii
1b6e7fea2f
Finally fixed
2022-03-17 16:40:10 +01:00
Nikolai Kochetov
ee9c2ec735
Merge pull request #34780 from azat/mt-delayed-part-flush
...
Do not delay final part writing by default (fixes possible Memory limit exceeded during INSERT)
2022-03-17 12:30:51 +01:00
kssenii
f354b704e0
Add more checks, different counting for download size
2022-03-17 11:52:59 +01:00
Sergei Trifonov
bbe657ba24
allow parallelization of multipart S3 storage upload fixes #34350
2022-03-16 19:41:41 +01:00
Nikolai Kochetov
a380aa6b8a
Merge pull request #35294 from ClickHouse/reload-remote_url_allow_hosts
...
Reload remote_url_allow_hosts after config update.
2022-03-15 22:07:16 +01:00
Nikolai Kochetov
cf260b8508
Merge pull request #35305 from ClickHouse/try-to-fix-using-deleted-memory-tracker-inside-s3-disk
...
Maybe fix use-after-free inside S3 upload thread
2022-03-15 22:06:17 +01:00
Nikolai Kochetov
b4aed421de
Reset THreadPool's thread memory tracker parent to global after S3 upload task is finished.
2022-03-15 15:33:43 +00:00
Nikolai Kochetov
97aa6c82ce
Reload remote_url_allow_hosts after config update.
2022-03-15 13:00:31 +00:00
Saad Ur Rahman
7bdbaf28d8
[Disks] bugfix 🤦♂️
...
Correcting order of parameters for <ReadBufferFromS3> in <readSchemaVersion>.
2022-03-14 22:42:09 -04:00
Saad Ur Rahman
40fbea49f7
[Disks] bugfix
...
Fixed order of parameters to <DiskS3> constructor:
- name
- bucket
- key
- versionId
- ...
2022-03-14 22:37:47 -04:00
Saad Ur Rahman
fea8824489
[Disks] Added Version ID parameter to registerDisksS3.
2022-03-14 22:06:09 -04:00
Saad Ur Rahman
e181207f2d
[Disks] Version ID parameter added to readFile
2022-03-14 21:30:23 -04:00
Saad Ur Rahman
98b2994352
[Disks] ReadBufferFromS3 requires a version id.
2022-03-14 21:30:23 -04:00
Saad Ur Rahman
e3a96393a1
[Disks] Version ID added.
...
Version ID added to constructor.
2022-03-14 21:30:23 -04:00
Saad Ur Rahman
f299bf46e2
[Disks] readSchemaVersion supplies empty Version ID
...
Empty version ID supplied to retrieve version ID of object.
2022-03-14 20:11:20 -04:00
mergify[bot]
ee3f4fb0c3
Merge branch 'master' into add-cache-optimization
2022-03-14 20:58:52 +00:00
Maksim Kita
538f8cbaad
Fix clang-tidy warnings in Disks, Formats, Functions folders
2022-03-14 18:17:35 +00:00
kssenii
fc9c6eab9d
Add optimization
2022-03-14 17:58:19 +01:00
alesapin
b149f28cd3
Merge pull request #35209 from kssenii/cache-fix
...
Cache fix
2022-03-14 11:07:40 +01:00
Kseniia Sumarokova
35e5b4e8a5
Update CachedReadBufferFromRemoteFS.cpp
2022-03-13 12:37:00 +01:00
kssenii
a18bb606a2
Update
2022-03-11 14:24:02 +01:00
kssenii
2b478e2501
Fix
2022-03-11 12:17:17 +01:00
kssenii
c86955deb2
Fix typo
2022-03-10 20:22:39 +01:00
kssenii
ddcb020d26
Fix test
2022-03-10 19:16:26 +01:00
kssenii
6571c7faa0
Clean up code
2022-03-09 10:36:52 +01:00
kssenii
96789fa5ce
Fix test, disable cache for DiskCacheWrapper
2022-03-09 09:46:04 +01:00
kssenii
16e84c265b
Fix build, update test
2022-03-08 21:41:58 +01:00
Azat Khuzhin
4200b56f3a
Introduce IDisk::supportParallelWrite()
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-08 22:17:36 +03:00
kssenii
5260822964
Merge master
2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0
Fix split build
2022-03-08 18:05:55 +01:00
kssenii
f3cba6f270
Remove already fixed crutch
2022-03-08 14:30:21 +01:00
kssenii
28a3aece60
Fix uncaught exception from destructor
2022-03-08 10:58:37 +01:00
Kseniia Sumarokova
517e878c6e
Merge pull request #35099 from ClickHouse/tavplubix-patch-1
...
Fix inconsistency in DiskLocal
2022-03-08 10:18:07 +01:00
kssenii
0737aef472
Fix split build
2022-03-07 22:02:07 +01:00
kssenii
eeb0ae9816
Fix detaching query
2022-03-07 17:05:36 +01:00
tavplubix
84e22fb32b
Update DiskLocal.cpp
2022-03-07 18:59:00 +03:00
kssenii
059d239815
Simplify code
2022-03-07 15:17:38 +01:00
kssenii
692b247379
Fix some corner cases
2022-03-07 14:31:21 +01:00
mergify[bot]
88052e2d7c
Merge branch 'master' into finally-enable-s3-async-writes-again
2022-03-07 12:55:52 +00:00
kssenii
2dcfe66236
Fix race
2022-03-06 20:33:07 +01:00
Kseniia Sumarokova
299e11bb2b
Update DiskCacheWrapper.cpp
2022-03-03 11:11:57 +01:00
kssenii
192b839cab
Fix
2022-03-02 19:11:45 +01:00
kssenii
b2cab429a7
Move initialization out of constructor
2022-03-01 18:12:34 +01:00
kssenii
438c63f3d8
Style
2022-03-01 17:52:38 +01:00
Nikolai Kochetov
eaf6d8c5f6
Update DiskS3.cpp
2022-02-24 17:35:29 +01:00
kssenii
8d6f1982df
Tiny refactoring
2022-02-23 14:43:40 +01:00
kssenii
36a76d6a75
Fix test
2022-02-22 15:18:28 +01:00
kssenii
c637385dd0
Merge master
2022-02-22 13:17:51 +01:00
Kseniia Sumarokova
eeea322556
Merge pull request #34629 from amosbird/remotefsimprove
...
Some refactoring and improvement over async and remote buffer related stuff
2022-02-22 11:36:40 +01:00
kssenii
fad7c042c1
Fix cache with non-async reads
2022-02-21 19:50:27 +01:00
kssenii
4543513a5d
Add comments
2022-02-21 13:54:03 +01:00
kssenii
e8b2e47e93
Style
2022-02-19 20:38:23 +01:00
kssenii
6b60f7c57f
Add limit for file segment size and split it if exceeds the limit
2022-02-19 18:59:22 +01:00
Amos Bird
84d111cde9
Better ReadBuffer asynchronicity
2022-02-20 01:08:21 +08:00
Nikolai Kochetov
40a48c5670
Enable async writes to S3.
2022-02-18 18:32:04 +00:00
kssenii
acf9a092a9
Review fixes
2022-02-18 19:09:54 +01:00
kssenii
353ffacca3
Add test
2022-02-17 19:57:04 +01:00
Amos Bird
dc0b4a8f73
Remove useless template
2022-02-17 14:31:21 +08:00
kssenii
47f94120da
Merge master
2022-02-16 14:43:28 +01:00
kssenii
3bd3e51aa0
Fix tests
2022-02-16 14:08:41 +01:00
Vladimir C
724340418e
Merge pull request #34617 from Algunenano/freebsd_build
...
Attempt to fix freeBSD build
2022-02-15 20:12:44 +01:00
Raúl Marín
72efc00830
Attempt to fix freeBSD build
2022-02-15 17:23:01 +01:00
alesapin
bc2d0ee7c7
Merge pull request #34215 from ClickHouse/revert-34211-revert-34153-add_func_tests_over_s3
...
Add func tests run with s3 and fix several bugs
2022-02-15 19:07:11 +03:00
kssenii
886b300b8d
Less seeks
2022-02-15 16:12:25 +01:00
mergify[bot]
34973f1d5c
Merge branch 'master' into first-test
2022-02-15 12:15:53 +00:00
alesapin
447cd56cb9
Fix comments
2022-02-15 12:11:50 +03:00
alesapin
e15396d90c
Fix race condition:
2022-02-14 22:19:49 +03:00
kssenii
bdbbe85f95
Fixes
2022-02-14 19:28:48 +01:00
zxealous
3d29920005
Fix compilation issues with DiskHDFS
2022-02-14 13:14:20 +08:00
Danila Kutenin
1c589681a0
Have a correct include which was inlined
2022-02-12 12:49:23 +00:00
alesapin
088f8cacf3
Small improvements
2022-02-10 19:50:21 +03:00
Anton Popov
70986a70a1
support TTL TO [DISK|VOLUME] [IF EXISTS]
2022-02-10 19:26:23 +03:00
alesapin
3387f9e597
Possible fix DiskHDFS
2022-02-10 16:55:12 +03:00
alesapin
10c3e6e546
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-09 14:14:58 +03:00
alesapin
36909a986f
Fix bug with files remove
2022-02-08 22:21:16 +03:00
alesapin
02a93cb852
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-08 19:42:27 +03:00
alesapin
f6e7cea1f8
Add settings for S3 multipart upload parts size
2022-02-08 19:38:04 +03:00
alesapin
07d5227568
Disable thread pool
2022-02-08 14:08:00 +03:00
alesapin
773d9cede3
Fix cache wrapper
2022-02-08 14:05:38 +03:00
alesapin
3af6012cb4
Revert "Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer""""
...
This reverts commit 2bc2ea485e
2022-02-08 11:01:26 +03:00
alesapin
2bc2ea485e
Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"""
...
This reverts commit fb77d7a7d5
2022-02-08 10:56:29 +03:00
kssenii
2e58733750
Merge master
2022-02-07 15:23:10 +01:00
Anton Popov
1b16db72c3
fix consecutive backward seeks in seekable read buffers
2022-02-07 17:20:26 +03:00
Nikolai Kochetov
40b86e3dae
Add blind change.
2022-02-07 12:02:21 +00:00
alesapin
fb77d7a7d5
Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer""
...
This reverts commit 875e5413ad
2022-02-07 12:36:54 +03:00
alesapin
25c0b07e68
Disable some tests
2022-02-04 14:31:58 +03:00
alesapin
875e5413ad
Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"
...
This reverts commit b92efed350ecce006cb2
2022-02-04 14:30:33 +03:00
alesapin
5e049fdd47
Merge with master
2022-02-04 12:28:19 +03:00
alesapin
2ed45b2a98
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-04 11:23:46 +03:00
alesapin
85fa727e32
Fix tidy
2022-02-04 11:21:03 +03:00
alesapin
a61b231ddd
Better debug
2022-02-03 11:23:09 +03:00
alesapin
ab546f61e3
Disable cache by default
2022-02-02 22:06:08 +03:00
Nikolai Kochetov
2003a96b58
Probably fix something
2022-02-02 17:48:41 +00:00
alesapin
80800e051e
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-02 19:42:04 +03:00
Nikolai Kochetov
ea044fc6b2
Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer
2022-02-02 19:40:41 +03:00
alesapin
b9c118524f
Fix race condition on hardlink/erase/read metadata
2022-02-02 19:40:21 +03:00
kssenii
aeec749173
Fix predownload case part 1
2022-02-02 17:25:25 +03:00
kssenii
5cba11428a
Fixes
2022-02-02 00:25:16 +03:00
Azat Khuzhin
bedf208cbd
Use fmt::runtime() for LOG_* for non constexpr
...
Here is oneliner:
$ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'
Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:
$ cat fmt.cocci
@@
expression log;
expression var;
@@
-LOG_DEBUG(log, var)
+LOG_DEBUG(log, fmt::runtime(var))
I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-01 14:30:03 +03:00
Amos Bird
e29c77f793
Fix link order issue of WriteBufferFromS3
...
(cherry picked from commit f22b09f4fc
2022-02-01 10:42:06 +00:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3""
2022-02-01 13:36:51 +03:00
alesapin
18f08ed932
Revert "Revert "Add func tests run with s3""
2022-02-01 10:20:06 +03:00
alexey-milovidov
99392b5ca7
Merge pull request #13544 from amosbird/mdha
...
Multi-Disk auto-recovery.
2022-02-01 06:13:26 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3"
2022-02-01 05:49:40 +03:00
alexey-milovidov
cbfcd45be3
Revert "Add func tests run with s3"
2022-02-01 05:46:13 +03:00
alexey-milovidov
a2aa147ce0
Merge pull request #34153 from ClickHouse/add_func_tests_over_s3
...
Add func tests run with s3
2022-02-01 05:46:09 +03:00
Amos Bird
ec7d367814
DiskLocal checker
...
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
2022-02-01 05:55:27 +08:00
alesapin
cbf4790ea4
More clear error
2022-01-31 13:59:16 +03:00
kssenii
218a633fec
Some fixes, more assertions and logging
2022-01-31 10:51:34 +03:00
kssenii
6b03c577d6
Make downloader's term resticted to one nextImpl
2022-01-29 23:58:53 +03:00
Nikolai Kochetov
1c9f026178
Merge branch 'master' into add-pool-to-s3-write-buffer
2022-01-28 16:01:42 +03:00
kssenii
c91b86e220
Better
2022-01-26 21:56:08 +03:00
Nikolai Kochetov
a8171269a1
Review fixes.
2022-01-26 17:55:24 +00:00
Nikolai Kochetov
efaed154d2
Update src/Disks/IDisk.h
...
Co-authored-by: alesapin <alesapin@clickhouse.com>
2022-01-26 14:58:38 +03:00
Nikolai Kochetov
2e88319b72
Fix integration tests.
2022-01-25 18:48:28 +00:00
Nikolai Kochetov
fcc29dbd15
Try to fix integration tests.
2022-01-25 15:26:36 +00:00
kssenii
aef99dedba
Fixes
2022-01-25 01:46:44 +03:00
Nikolai Kochetov
4afa75e80b
Fix deduplication.
2022-01-24 14:43:36 +00:00
kssenii
898139acae
Fix tests
2022-01-24 12:32:39 +03:00
kssenii
710bba895e
Fix partially downloaded no continuation case
2022-01-23 23:49:13 +03:00
kssenii
cf49f41f75
Better
2022-01-23 19:51:18 +03:00
zhongyuankai
040b71617c
Fix disk using the same path
2022-01-23 16:08:19 +08:00
alexey-milovidov
d1431fffd2
Merge pull request #33912 from amosbird/fixdiskcacherace
...
Fix race in DiskCacheWrapper
2022-01-23 08:47:44 +03:00
kssenii
690db142b9
Better
2022-01-23 02:12:33 +03:00
Kseniia Sumarokova
2c43e4eb57
Merge branch 'master' into local-cache-for-remote-fs
2022-01-22 22:53:55 +03:00
kssenii
36a41ac146
Better
2022-01-22 20:42:39 +03:00
Amos Bird
da66239864
Fix race in DiskCacheWrapper
2022-01-22 23:56:30 +08:00
kssenii
9f295ebf92
Fix threadpool reads
2022-01-22 01:07:20 +03:00
Maksim Kita
a3adf12a50
Merge pull request #33862 from ClickHouse/fix-s3-proxy-config-resolver-ipv6-only
...
Try every resolved ip address while getting S3 proxy.
2022-01-21 20:29:19 +01:00
Nikolai Kochetov
a024ddd1d4
Merge branch 'add-pool-to-s3-write-buffer' of github.com:ClickHouse/ClickHouse into add-pool-to-s3-write-buffer
2022-01-21 18:42:24 +00:00
Nikolai Kochetov
b3cbb63487
Merge branch 'master' into add-pool-to-s3-write-buffer
2022-01-21 21:41:54 +03:00
Nikolai Kochetov
ae1b669fca
Even more async writes.
2022-01-21 15:55:19 +00:00
kssenii
1858f69f5a
Make space reservation incremental
2022-01-21 18:39:34 +03:00
Nikolai Kochetov
a178e54d10
Try every resolved ip address while getting S3 proxy.
2022-01-21 13:19:25 +03:00
kssenii
a566099759
Local cache for remote filesystem
2022-01-19 00:42:11 +03:00
Anton Ivashkin
7c26a832f8
Fix restore S3 disk without detached directories
2022-01-18 12:31:31 +03:00
Nikolai Kochetov
6d49a62666
Some more async writes.
2022-01-14 19:53:55 +00:00
Nikolai Kochetov
843983ea06
Add batch removing to IDiskl.
2022-01-12 15:58:04 +00:00
alesapin
733ed7cb54
Merge pull request #32061 from ianton-ru/MDB-15474
...
Change ZooKeeper path for zero-copy locks for shared data
2022-01-08 17:17:11 +03:00
xiedeyantu
1edfc03825
Update AsynchronousReadIndirectBufferFromRemoteFS.h
...
update 1024 * 1024 to DBMS_DEFAULT_BUFFER_SIZE
2022-01-07 21:06:53 +08:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474
2022-01-07 10:12:25 +00:00
Azat Khuzhin
71a99ab27c
Accept real file size in createReadBufferFromFileBase()
...
Right now streams relies on correct file size not the number of bytes
that will be read from the stream, to overcome one bug in the linux
kernel that may return EIINVAL for pread() with offset pass the EOF.
v2: Swap read_hint and file_size (since it is easy to miss something)
Before the first argument to readFile()/createReadBufferFromFileBase()
was read_hint, not the file_size, and let's preserve the order, since
it is easy to miss something
This will also fix 02051_read_settings test automatically because now
MergeTreeReaderStream will pass estimated_sum_mark_range_bytes to
read_hint not file_size, previously it cause on of the following errors:
- Attempt to read after EOF w/ O_DIRECT
- and LOGICAL_ERROR while adjusting granulas w/o O_DIRECT
This will also improve zero-length reads guard (via
ReadBufferFromEmptyFile), that had been added in #30190
v3: fix for other storages that wasn't enabled in fast-test
v4: ignore ENOENT/ENOTSUP in readFile
2022-01-04 10:53:17 +03:00
alesapin
7a3c874651
Merge branch 'master' into ianton-ru-MDB-15474
2021-12-29 19:01:43 +03:00
Nikolai Kochetov
68bf38133a
Add pool to WriteBufferFromS3
2021-12-29 14:04:21 +00:00
Anton Ivashkin
2d87f0a0e3
Fix debug build
2021-12-28 11:45:38 +03:00
Alexey Milovidov
f03cb4c762
Cleanup trash from Kafka and HDFS
2021-12-25 06:10:59 +03:00
Maksim Kita
60d16afde7
Merge pull request #33027 from ClickHouse/merge-33023
...
Merge #33023
2021-12-22 13:44:14 +03:00
Kseniia Sumarokova
484a4d62dd
Merge pull request #32835 from kssenii/fising-s3
...
Fix optimization with lazy seek for async reads from remote fs
2021-12-22 09:38:23 +03:00
Alexey Milovidov
7b48cb813e
Merge branch 'Issue129' of github.com:DevTeamBK/ClickHouse into merge-33023
2021-12-22 02:17:00 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474
2021-12-21 17:38:36 +03:00
Anton Ivashkin
0c0bf66334
Merge master
2021-12-21 17:27:54 +03:00
kssenii
ebcb66a361
Fix
2021-12-21 10:58:02 +03:00
Nikolai Kochetov
35883e0dae
Merge pull request #32979 from ClickHouse/decrease-some-s3-log-level
...
Decrease log level for some s3 messages.
2021-12-20 20:56:22 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment
...
Loops remove postfix increment
2021-12-20 19:51:07 +03:00
Nikolai Kochetov
7dea7b7f76
Decrease log level for some s3 messages.
2021-12-20 18:18:54 +03:00
Anton Ivashkin
f0b9a4327a
Use table UUID in zero-copy shared label in ZooKeeper
2021-12-20 16:19:00 +03:00
Maksim Kita
51477adf1b
Updated additional cases
2021-12-20 15:55:07 +03:00
Maksim Kita
3feab5a975
Containers iteration fix erase
2021-12-20 13:42:31 +03:00
root
6b3366dc70
[Issue #129 ] Fix Weak Regular Expression [v21.3-insecure]
...
Problem Descripton:
==================
The “key_regexp” constant in the “DiskS3” class contains a regular expression (RegEx) for checking a key path.
However, the RegEx starts and ends with “.*”.
This is a wildcard that an attacker could use to exploit this file path check by padding the beginning and end with whatever path they’d like.
Recommendation:
==============
Tighten up the RegEx by replacing the “.*” at the beginning and end with more specific items.
Fix:
===
Replace the .* at the end with $ as there is no need to continue after the key file.
Leave the beginning .* as is because the path for the key can have any characters or in any path of the system.
2021-12-17 10:30:16 -05:00
kssenii
40c266840c
Fixes
2021-12-17 18:16:29 +03:00
kssenii
ed42620530
Rename BlobStorage to AzureBlobStorage
2021-12-17 12:24:05 +03:00
kssenii
ca283a4bb8
Fix optimization with lazy seek
2021-12-17 00:18:28 +03:00
alesapin
c338bee718
Revert "Fix build issue related to azure blob storage"
2021-12-16 14:37:12 +03:00
alesapin
c0cebaea79
Merge pull request #32788 from amosbird/buildfix
...
Fix build issue related to azure blob storage
2021-12-16 13:45:22 +03:00
Amos Bird
9c9f4a3e6c
build fix
2021-12-15 16:55:18 +08:00
alesapin
9974f10631
Fix clang tidy
2021-12-14 15:34:20 +03:00
Kseniia Sumarokova
78ed383f15
Merge pull request #31505 from ContentSquare/azure_blob_storage_dependency
...
Azure Blob Storage Disks
2021-12-13 18:28:48 +03:00
Azat Khuzhin
57c027be0e
Remove arcadia build support
2021-12-11 21:25:23 +03:00
Jakub Kuklis
578aec97e9
Removing TODO and NOTE marks used mainly as hooks for review in the PR
2021-12-10 22:04:38 +01:00
Jakub Kuklis
a3bc6d2fb7
Style corrections
2021-12-10 22:04:38 +01:00
Jakub Kuklis
74e26354cc
Committing missing files
2021-12-10 22:04:38 +01:00
Jakub Kuklis
41a88e49c9
Abstracting metadata creation for remote disks registrations, putting Blob Storage auth in a separate file
2021-12-10 22:04:38 +01:00
Jakub Kuklis
e58f74d8dc
Connecting to Azurite and invoking simple INSERTs works in the integration test
2021-12-10 22:04:38 +01:00
Jakub Kuklis
8afe69ff93
Comming back to StorageSharedKeyCredential for the integration test, get 400 Bad requests from Azurite when trying to create a container now
2021-12-10 22:04:38 +01:00
Jakub Kuklis
f180a7ba5f
Solved Invalid storage account error with connection strings
2021-12-10 22:04:38 +01:00
Jakub Kuklis
008141de36
Enable authentication via StorageSharedKeyCredential
2021-12-10 22:04:38 +01:00
Jakub Kuklis
d51f979c7a
Adding endpoint validation
2021-12-10 22:04:38 +01:00
Jakub Kuklis
2b7725bfc7
Enable connecting directly to storage accounts and creating new containers
2021-12-10 22:04:38 +01:00
Jakub Kuklis
910db6ca57
Code cleanup addressing PR remarks
2021-12-10 22:04:38 +01:00
Jakub Kuklis
f6c5a2c63c
Removing file checks after changing blob filenames
2021-12-10 22:04:38 +01:00
Jakub Kuklis
3aa6a0ead3
Adding random postfix to blob names, needs code cleanup
2021-12-10 22:04:38 +01:00
Jakub Kuklis
5ed511f9da
Working skeleton for Blob Storage integration test
2021-12-10 22:04:37 +01:00
Jakub Kuklis
0c65cfed1f
Addressing first PR remarks
2021-12-10 22:04:37 +01:00
Jakub Kuklis
7c2e116214
Adding max_retries config
2021-12-10 22:04:37 +01:00
Jakub Kuklis
a33cea0525
Refactors after proper rebase on master
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b3728dbbaf
Rebase on master
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b8ef47006c
Minor refactors before PR
2021-12-10 22:04:37 +01:00
Ryad ZENINE
c4421d6866
Small changes following review
2021-12-10 22:04:37 +01:00
Jakub Kuklis
7f595eedbc
Catch Blob Storage exceptions
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b3c4a92cc0
Code cleanup
2021-12-10 22:04:37 +01:00
Jakub Kuklis
c4691affb2
Adding a new test for read with offset
2021-12-10 22:04:37 +01:00
Jakub Kuklis
9a00acfb72
Fixing read size error plus minor improvements
2021-12-10 22:04:37 +01:00
Jakub Kuklis
c9f3164cba
Bringing back old single part upload limit
2021-12-10 22:04:37 +01:00
Jakub Kuklis
0d7a4b10c3
Using unique_ptr for Blob Client in Read Buffer
2021-12-10 22:04:37 +01:00
Jakub Kuklis
5e2b2b1eee
Better search for blobs with a given name
2021-12-10 22:04:37 +01:00
Jakub Kuklis
33247db3f4
Adding finalize() to WriteBuffer, always using block blob uploads, code cleanup
2021-12-10 22:04:37 +01:00
Jakub Kuklis
bf2476a076
Shared pointer for Azure client and fixing path reference in ReadBuffer
2021-12-10 22:04:37 +01:00
Jakub Kuklis
60dd22dd4e
Checking for success of file removal operation
2021-12-10 22:04:37 +01:00
Jakub Kuklis
486f267b3b
Further code cleanup and error codes improvement
2021-12-10 22:04:37 +01:00
Jakub Kuklis
6755c5e897
File deletion prototype
2021-12-10 22:04:37 +01:00
Jakub Kuklis
a0cfafa387
checkUniqueId implementation using ListBlobs
2021-12-10 22:04:37 +01:00
Jakub Kuklis
95816875f4
Change false to true in supportZeroCopyReplication
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b41ab197e8
Blob Storage endpoint url verification using re2
2021-12-10 22:04:37 +01:00
Jakub Kuklis
63ec8d4244
Adding multipart upload to WriteBuffer, retries in ReadBuffer, additional settings for buffers
2021-12-10 22:04:37 +01:00
Jakub Kuklis
02060791e3
Adding settings getter and applyNewSettings
2021-12-10 22:04:37 +01:00
Jakub Kuklis
3f28f2c62b
Enabling local cache for DiskBlobStorage
2021-12-10 22:04:37 +01:00
Jakub Kuklis
e3daf8aa49
Increase default thread pool size
2021-12-10 22:04:37 +01:00
Jakub Kuklis
dbb98e3636
Code cleanup, leaving some debug information printing under #ifdef VERBOSE_DEBUG_MODE
2021-12-10 22:04:37 +01:00
Jakub Kuklis
8230f8039c
Two column MergeTree table with Blob Storage disk work! Fixed buffer in ReadBufferFromBlobStorage to be permanent (class attribute)
2021-12-10 22:04:37 +01:00
Jakub Kuklis
2b4d3665ab
Basic inserts and selects work for a MergeTree table with a Blob Disk, based on a single column table
2021-12-10 22:04:37 +01:00
Jakub Kuklis
4b51ddee1b
Initial check with write and read works for the Blob Storage Disk
2021-12-10 22:04:37 +01:00
Jakub Kuklis
03f85dee41
Managed to read markers file in SELECT, getting error for CODEC version now
2021-12-10 22:04:37 +01:00
Jakub Kuklis
c43b90c58c
Progressing with WriteBufferFromBlobStorage
2021-12-10 22:04:37 +01:00
Jakub Kuklis
fb4c1d6686
Starting insertion pipeline
2021-12-10 22:04:37 +01:00
Jakub Kuklis
6436f5b04d
Adding an authenticated client to DiskBlobStorage
2021-12-10 22:04:37 +01:00
Jakub Kuklis
ee1a568919
The following query passes without erors: CREATE TABLE ... ENGINE = MergeTree ... storage_policy = 'blob_storage', debug constants present in the code
2021-12-10 22:04:37 +01:00
Jakub Kuklis
cacdac7ba6
Moving implementation to the .cpp file
2021-12-10 22:04:37 +01:00
Jakub Kuklis
88b5feb59d
Registering the Blob Storage Disk
2021-12-10 22:04:36 +01:00
Jakub Kuklis
20dda7f666
Giving Disk Blob Storage files proper names
2021-12-10 22:04:36 +01:00
Jakub Kuklis
0cf338e165
Skeleton for DiskBlobStorage
2021-12-10 22:04:36 +01:00
Jakub Kuklis
43f420cba8
Adding a flag to use Azure Blob Storage
2021-12-10 22:04:36 +01:00
Jakub Kuklis
162e2b334b
Initial working example of Azure Blob Storage manipulation from within ClickHouse
2021-12-10 22:04:36 +01:00
leosunli
3946b156a1
Fix bug
...
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
2021-12-06 14:32:32 +08:00
leosunli
6f611eec91
Fix bug
...
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
2021-12-06 14:32:32 +08:00
leosunli
e81a1dbe6a
Make HDFS replication configurable in WriteBufferFromHDFSImpl#WriteBufferFromHDFSImpl
...
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
2021-12-06 14:32:31 +08:00
leosunli
5be58a8839
Make HDFS replication configurable in WriteBufferFromHDFSImpl#WriteBufferFromHDFSImpl
...
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
2021-12-06 14:32:31 +08:00
sunlisheng
0f045fa20a
rename replication
...
Signed-off-by: sunlisheng <lisheng.sls@alibaba-inc.com>
2021-12-06 14:32:31 +08:00
sunlisheng
ef77f45f59
Make HDFS replication configurable in WriteBufferFromHDFSImpl#WriteBufferFromHDFSImpl
...
Signed-off-by: sunlisheng <lisheng.sls@alibaba-inc.com>
2021-12-06 14:32:31 +08:00
Anton Ivashkin
0e685c1aba
Fix types
2021-12-01 19:11:17 +03:00
Anton Ivashkin
e0a16a483e
Merge master
2021-12-01 19:03:31 +03:00
Anton Ivashkin
0f9038ebed
Zero-copy: move shared mark outside table node in ZooKeeper
2021-11-29 19:05:31 +03:00
Kseniia Sumarokova
93cf66df12
Merge pull request #30936 from kssenii/seekable-read-buffers
...
Reduce memory usage for some formats when reading with s3/url/hdfs
2021-11-25 11:19:24 +03:00
Kruglov Pavel
d9c1a0c8ec
Merge branch 'master' into fix-write-buffers
2021-11-20 17:48:24 +03:00
kssenii
ff969b4605
Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers
2021-11-20 15:03:13 +03:00
Vitaly Baranov
33ea7a7262
Rename RowPolicy::ConditionType -> RowPolicyFilterType and move it to Access/Common.
2021-11-19 00:14:23 +03:00
alexey-milovidov
cb6342025d
Merge pull request #31229 from zhanglistar/fsync
...
Change all fsync to fdatasync
2021-11-14 05:16:01 +03:00
Kseniia Sumarokova
59d4eb99c2
Merge pull request #31291 from kssenii/update-async-reads-setting
...
Make remote_filesystem_read_method=threadpool by default
2021-11-13 19:19:35 +03:00
kssenii
f18dcd2287
Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers
2021-11-13 14:38:57 +03:00
Kseniia Sumarokova
c7be79b4e7
Merge pull request #31136 from LiuYangkuan/IDiskRemote_metadata_disk
...
use DiskPtr instead of OS's file system API in class IDiskRemote
2021-11-13 12:16:05 +03:00
Kseniia Sumarokova
c6fd56ede9
Update DiskLocal.cpp
2021-11-13 00:00:32 +03:00
LiuYangkuan
4366fcbc83
fix build check and make rename robust in DiskS3::restoreFileOperations
2021-11-12 22:54:10 +08:00
kssenii
6f895036e6
Disable for hdfs until tested, adjust tests with profile events
2021-11-12 14:09:19 +03:00
zhanglistar
4e7823ade7
1. add macosx fix
2021-11-12 15:49:21 +08:00
LiuYangkuan
f7078a17ce
add DiskLocalDirectoryIterator() when path not directory or nonexist, fix remove from_path after moveDirectory in DiskS3::restoreFileOperations
2021-11-12 11:36:48 +08:00
kssenii
031754b465
Fix
2021-11-12 01:37:18 +03:00
Kseniia Sumarokova
e709c3ba90
Update AsynchronousReadIndirectBufferFromRemoteFS.h
2021-11-12 01:13:44 +03:00
Kseniia Sumarokova
2313981fd7
Merge pull request #31260 from azat/external-cleanup
...
Cleanup extern ProfileEvents/CurrentMetrics and add a style check
2021-11-12 00:02:57 +03:00
Kseniia Sumarokova
e20cb0c10a
Merge branch 'master' into retriable-http
2021-11-11 11:52:06 +03:00
avogar
c521a9131a
Small refactoring of WriteBiffer-s
2021-11-11 02:11:18 +03:00
kssenii
62c70ce5cf
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-11-10 21:16:11 +00:00
Azat Khuzhin
baf14444e6
Cleanup ProfileEvents and CurrentMetrics
2021-11-10 21:15:27 +03:00
LiuYangkuan
006dca1362
use DiskPtr instead of OS's file system API in class IDiskRemote
2021-11-10 22:50:15 +08:00
zhanglistar
aaad3f84da
Merge branch 'ClickHouse:master' into fsync
2021-11-10 17:44:08 +08:00
zhanglistar
96b8df5899
1. chagne all fsync to fdatasync
2021-11-10 17:40:16 +08:00
kssenii
618c332f9c
Rename read_threadpool to threadpool
2021-11-10 07:46:18 +00:00
alexey-milovidov
7006147268
Merge pull request #31159 from snar/freebsd-disk-space
...
correct disk space calculations
2021-11-10 02:48:29 +03:00
Kruglov Pavel
17085183bf
Merge pull request #31042 from Avogar/hdfs-bad-url
...
Add additional hdfs url check
2021-11-09 16:23:59 +03:00
kssenii
36f213e51c
Fix build
2021-11-09 07:11:44 +00:00
Kseniia Sumarokova
c458f73138
Update AsynchronousReadIndirectBufferFromRemoteFS.cpp
2021-11-08 22:12:04 +03:00
kssenii
83d64f1c2d
Better
2021-11-08 13:32:32 +03:00
kssenii
b4bebc5977
Add comments
2021-11-08 12:01:24 +03:00
Alexandre Snarskii
248709ea16
correct disk space calculations
2021-11-07 19:51:48 +03:00
kssenii
9189966106
Fix
2021-11-05 20:01:21 +03:00
kssenii
4bc2a29345
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-11-04 14:41:24 +00:00
kssenii
b95f0738fb
Fix
2021-11-04 14:39:53 +00:00
kssenii
ec11179f91
Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers
2021-11-03 14:33:31 +03:00
kssenii
45ea820297
Reduce memory usage for some formats
2021-11-03 14:30:03 +03:00
avogar
3839d521a0
Add additional hdfs url check
2021-11-03 14:01:50 +03:00
Vladimir C
77edd081a5
Fix whitespace in src/Disks/S3/DiskS3.cpp
2021-11-01 13:00:21 +03:00
Vladimir C
2e58a3bf62
Merge branch 'master' into MDB-15168
2021-11-01 11:59:13 +03:00
kssenii
2940d9fd19
Squashed commit of the following:
...
commit a1acc7ed3485bd158cf42f26b2d2a1bca84b7269
Author: kssenii <sumarokovakseniia@mail.ru>
Date: Sun Oct 31 14:47:25 2021 +0000
Fix
commit 12a27d445bf17e321a7355f1ba857d11b70a285chttps://github.com/ClickHouse/ClickHouse into retriable-http
commit 55f1ba857daeba8104d182f3754b8a2647b88a667024f51b552bb586bed3https://github.com/ClickHouse/ClickHouse into retriable-http
commit 7024f51b557de5fca0751ecb6ad05d855b10261chttps://github.com/ClickHouse/ClickHouse into retriable-http
commit 1ecb6ad05d710ee9687828d2485716158b4c26b7https://github.com/ClickHouse/ClickHouse into retriable-http
commit 28d24857167c20ca07b7ccb02cac569e4fe0f3c8d1138a8a25https://github.com/ClickHouse/ClickHouse into retriable-http
commit 9e4fe0f3c89814cb1b45daed77038bhttps://github.com/ClickHouse/ClickHouse into retriable-http
commit 9814cb1b451c8b1b1133969999ff10https://github.com/ClickHouse/ClickHouse into retriable-http
commit 1c8b1b1133e7dbfff5d23329b668d6198adc7ecdhttps://github.com/ClickHouse/ClickHouse into retriable-http
2021-10-31 17:53:08 +03:00
kssenii
12a27d445b
Adjutments after merge with master
2021-10-29 09:29:35 +00:00
kssenii
f17e321a73
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-10-29 08:03:25 +00:00
kssenii
55f1ba857d
Small fixes
2021-10-29 07:38:42 +00:00
kssenii
4f3433b4e1
Some clean up before merging
2021-10-29 01:04:52 +03:00
kssenii
7e2ea97e3c
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-28 18:46:21 +03:00
Anton Ivashkin
6cda420b56
Cleanup after fail in localBackup
2021-10-28 17:50:59 +03:00
kssenii
aeba8104d1
Some more fixes
2021-10-28 10:28:05 +00:00
kssenii
3d01fbf603
Review fixes
2021-10-28 11:32:37 +03:00
Alexey Milovidov
8b4a6a2416
Remove cruft
2021-10-28 02:10:39 +03:00
kssenii
9461e225bb
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-27 22:47:47 +03:00
kssenii
2647b88a66
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-10-27 19:38:19 +00:00
kssenii
7024f51b55
Review fixes and add test
2021-10-27 19:37:36 +00:00
Azat Khuzhin
9936ee5181
Update gtest_disk_encrypted for new readFile() interface
2021-10-27 09:37:26 +03:00
Azat Khuzhin
7ce01b328e
Update gtest_disk_encrypted for new createReadBufferFromFileBase() interface
2021-10-27 09:37:26 +03:00
kssenii
52296e8b04
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-25 00:22:32 +03:00
kssenii
658b33717f
A little nicer
2021-10-24 20:43:34 +03:00
kssenii
7de5fca075
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-10-24 17:04:24 +00:00
Azat Khuzhin
34d8f78c33
Explictly pass size in DiskCacheWrapper::writeFile()
...
v0: Add explicit default value for DiskLocal::readFile()
v2: Explictly pass size in DiskCacheWrapper::writeFile()
(since v0 not google-default-arguments compatible)
2021-10-24 18:58:22 +03:00
Azat Khuzhin
4599e9191b
Fix explicit 0 size for readFile()
2021-10-24 18:58:22 +03:00
Azat Khuzhin
2b96433daf
Replace estimated_size with optional<size> for IDisk::readFile()
...
v2: rebase to fix conflicts in DiskEncrypted.cpp
2021-10-24 18:58:11 +03:00
Vitaly Baranov
c0b01e7b0b
Merge pull request #30494 from vitlibar/fix-reading-from-empty-file-on-encrypted-disk
...
Fix reading from empty file on encrypted disk
2021-10-24 16:24:47 +03:00
kssenii
1ecb6ad05d
Better
2021-10-23 19:00:14 +00:00
Vitaly Baranov
26cb62de32
Add more tests for the "encrypted" disk type.
2021-10-23 20:58:37 +03:00
kssenii
7f2c5dcaae
Fix clang-tidy
2021-10-22 01:08:58 +03:00
kssenii
56f0560c46
Asserts and read till end option
2021-10-21 20:54:03 +03:00
Vitaly Baranov
172a6f2e0d
Fix reading from empty file on encrypted disk.
2021-10-21 19:31:29 +03:00
kssenii
120b01e089
Better
2021-10-21 18:22:17 +03:00
kssenii
e4e157688d
Review fixes
2021-10-21 01:57:43 +03:00
kssenii
31cd71c849
Clean up
2021-10-20 09:37:18 +03:00
kssenii
143e912e4e
Fix split build
2021-10-19 22:45:46 +03:00
kssenii
49106f407f
Fix some checks
2021-10-19 09:34:27 +03:00
kssenii
d5d4817350
Add disk proxies back, add setReadUntilPosition to ReadBuffer
2021-10-18 18:35:11 +03:00
kssenii
28d2485716
Some fixes
2021-10-15 17:27:26 +00:00
kssenii
7c20ca07b7
Fix style check
2021-10-15 12:59:24 +00:00
kssenii
c1c574e9ca
Better
2021-10-15 13:14:57 +03:00
kssenii
3995506d37
Adjust range reader for remote fs reads
2021-10-15 11:36:26 +03:00
kssenii
1c8b1b1133
Better
2021-10-12 09:33:07 +00:00
kssenii
31d98b0992
Remove redundant
2021-10-11 08:17:22 +03:00
kssenii
448d63a0fc
Range read with upper bound offset
2021-10-11 01:53:51 +03:00
kssenii
1f8b449bc5
Lazy seek avoiding
2021-10-11 00:51:43 +03:00
kssenii
e3d2942aa6
Correct merge
2021-10-10 20:24:36 +03:00
kssenii
fde6f0507d
Merge remote-tracking branch 'origin/retriable-http' into disk-async-read
2021-10-10 20:23:44 +03:00
kssenii
3329b668d6
Fix style check
2021-10-10 15:51:37 +00:00
kssenii
40434519e8
Fix disk web
2021-10-08 16:02:11 +03:00
kssenii
f3f820c6e0
HTTP read buffer retries
2021-10-08 15:41:55 +03:00
kssenii
d6e1d0503d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-07 16:21:00 +03:00
Filatenkov Artur
dc657a2c5e
Add error for multiple keys without current_key_id ( #29546 )
...
* add error for multiple keys without current_key_id
* add exception if one key with non zero id is used without current_key_id
* refactoring
2021-10-05 14:37:54 +03:00
Azat Khuzhin
a7793101bc
Fix data-race in WriteIndirectBuffer (used in StripeLog)
...
CI founds [1]:
WARNING: ThreadSanitizer: data race (pid=497)
Write of size 8 at 0x7b30000260b8 by thread T381 (mutexes: write M546759850860896464):
...
3 DB::DiskMemory::writeFile() obj-x86_64-linux-gnu/../src/Disks/DiskMemory.cpp:339:15 (clickhouse+0x14a1849f)
4 DB::FileChecker::save() const obj-x86_64-linux-gnu/../src/Common/FileChecker.cpp:111:50 (clickhouse+0x15bbe19f)
5 DB::TinyLogSink::TinyLogSink() obj-x86_64-linux-gnu/../src/Storages/StorageTinyLog.cpp:214:34 (clickhouse+0x15bca215)
Previous read of size 8 at 0x7b30000260b8 by thread T460:
...
2 DB::WriteIndirectBuffer::finalize() obj-x86_64-linux-gnu/../src/Disks/DiskMemory.cpp:106:33 (clickhouse+0x14a1e044)
9 DB::StripeLogSink::onFinish() obj-x86_64-linux-gnu/../src/Storages/StorageStripeLog.cpp:227:30 (clickhouse+0x15bd241a)
[1]: https://clickhouse-test-reports.s3.yandex.net/29680/d0fc26f91a0141b56a0550741219c3dc43630e03/stress_test_(thread)/stderr.log
2021-10-04 01:58:35 +03:00
Mike Kot
38232c286d
fixing paths
2021-10-02 21:45:05 +02:00
Mike Kot
65e6e211b4
Merge remote-tracking branch 'upstream/master' into improvement/fn-traits
2021-10-02 21:37:50 +02:00
Alexey Milovidov
fe6b7c77c7
Rename "common" to "base"
2021-10-02 10:13:14 +03:00
kssenii
7ff092712b
Better
2021-10-01 18:13:16 +03:00
Mike Kot
38b02b121a
Merge remote-tracking branch 'upstream/master' into improvement/fn-traits
2021-10-01 16:55:28 +02:00
kssenii
55972db866
Small fix and more comments
2021-10-01 11:38:58 +03:00
kssenii
e4bc81ac78
Better
2021-09-30 21:45:57 +03:00
kssenii
cc9ce13bac
Finally all fixed
2021-09-30 15:35:59 +03:00
kssenii
85aae03a26
Fix
2021-09-29 22:54:49 +03:00
Mike Kot
7670ea50c1
Replacing std::function to Fn in some places
...
to avoid dynamic allocating while keeping desired function signature
clear.
Simplifying SimpleCache (CachedFn) using C++20
2021-09-29 13:09:39 +02:00
kssenii
20192352ad
Better
2021-09-29 11:03:36 +03:00
kssenii
ebca30694d
Fix
2021-09-27 18:25:30 +03:00
kssenii
efddc03246
Async read for disk web
2021-09-26 22:49:28 +03:00
alexey-milovidov
85fc160eb4
Merge pull request #29346 from ClickHouse/remove-ya-make
...
Remove ya.make
2021-09-25 19:43:40 +03:00
Alexey Milovidov
cd7f9d981c
Remove ya.make
2021-09-25 04:22:54 +03:00
alexey-milovidov
62d606bb76
Update DiskWebServer.h
2021-09-25 02:58:12 +03:00
Alexey Milovidov
4a44bcaced
Fix trivial mistake in DiskWebServer
2021-09-25 01:33:40 +03:00
kssenii
6219d541a5
Finally fixed
2021-09-24 15:29:04 +03:00
kssenii
c1ea44b9c2
Fixes
2021-09-24 13:38:08 +03:00
kssenii
1b9565dfa8
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-09-22 22:08:44 +03:00
kssenii
e301457e91
Fix prefetch mistake
2021-09-21 15:40:48 +03:00
kssenii
975033e6e3
Fix
2021-09-21 09:44:54 +00:00
kssenii
40ee75fe97
Common thread pool for disks
2021-09-21 10:00:56 +03:00
kssenii
cacc81ca61
Fix
2021-09-21 02:31:02 +03:00
kssenii
0104cb1c8e
Better
2021-09-21 01:51:17 +03:00
kssenii
f312760dca
Better
2021-09-20 19:57:15 +03:00
kssenii
11ae94b8d3
Async read for remote fs
2021-09-20 16:16:46 +03:00
Kseniia Sumarokova
2a8f355326
Merge pull request #29179 from kssenii/better-retries
...
Better retries for disk web
2021-09-20 08:25:49 +03:00
kssenii
12b90da5de
Better retries
2021-09-19 20:50:50 +00:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask ( #25165 )
...
Introduced MergeTask and MutateTask
2021-09-17 00:19:58 +03:00
Kseniia Sumarokova
ef6ba90be1
Update DiskWebServer.h
2021-09-13 20:00:18 +03:00
kssenii
9ec6a9b4c9
Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-web-disk-2
2021-09-13 08:32:34 +03:00
kssenii
147e413346
Add retries with backoff, better exceptions
2021-09-13 00:04:05 +03:00
Mike Kot
c4385a2c40
Simplifying exception messages
2021-09-11 00:40:18 +02:00
kssenii
939032b0eb
Slightly better
2021-09-10 13:41:41 +03:00
kssenii
c23e17d369
Fix
2021-09-09 15:25:44 +03:00
kssenii
72505db3bd
Fix checks
2021-09-09 10:10:34 +03:00
kssenii
1101ee7808
Update example
2021-09-08 22:43:46 +03:00
kssenii
7345ad8e09
Add no throw
2021-09-08 22:40:26 +03:00
kssenii
7141b5d041
Get rid of regexp
2021-09-08 20:22:24 +03:00
Mike Kot
8e9aacadd1
Initial: replacing hardcoded toString for enums with magic_enum
2021-09-06 16:24:03 +02:00
kssenii
97e0a0310a
Slightly better
2021-09-04 12:02:07 +03:00
kssenii
32686e11cb
Fix style checl
2021-09-04 11:30:30 +03:00
kssenii
6ee94c923f
Fix
2021-09-04 00:51:11 +03:00
Kseniia Sumarokova
63a0eb3922
Update DiskWebServer.cpp
2021-08-31 21:51:00 +03:00
kssenii
248fe1fd09
Fix disk
2021-08-31 16:54:43 +03:00
Nikita Mikhaylov
00394014ff
Merge pull request #28382 from DimasKovas/fix_arcadia_build_31_08_2021
...
Try to fix arcadia build (generate ya.make)
2021-08-31 13:49:32 +03:00
Dmitrii Kovalkov
b1eb48804b
Run generate-ya-make.sh
2021-08-31 09:05:33 +03:00
Alexey Milovidov
ca6f6306f3
Merge branch 'master' into async-reads
2021-08-31 02:52:58 +03:00
Kseniia Sumarokova
00bea71610
Merge pull request #25251 from kssenii/disk-over-web-server
...
ClickHouse over web server with static files
2021-08-30 16:18:59 +03:00
Dmitrii Kovalkov
9871ad70ff
Exclude fuzzers
2021-08-30 11:12:25 +03:00
Dmitrii Kovalkov
8acf0fa16a
run generate-ya-make
2021-08-30 10:57:40 +03:00
kssenii
3e7f022d66
Extend usage comment explanation a little
2021-08-28 20:51:20 +00:00
kssenii
f27f519aa2
Fix build and add example
2021-08-28 20:35:51 +00:00
Alexey Milovidov
da4865a5d8
Remove excessive logging from DiskS3
2021-08-25 03:22:26 +03:00
Alexey Milovidov
183891eb0a
Add IDisk::isRemote method
2021-08-25 02:27:20 +03:00
Alexey Milovidov
fd27a2faef
Remove imperfect code
2021-08-25 02:05:55 +03:00
Alexey Milovidov
7c1d0a3baf
Progress on development
2021-08-25 01:24:47 +03:00
Alexey Milovidov
8f57216180
Progress on development
2021-08-25 00:45:58 +03:00
Alexey Milovidov
4d9ad3725d
Merge branch 'master' into async-reads
2021-08-25 00:45:40 +03:00
Kseniia Sumarokova
d78e551de9
Update src/Disks/DiskWebServer.cpp
...
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
2021-08-24 12:24:54 +03:00
kssenii
b536653851
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-24 07:29:50 +00:00
kssenii
5307d31924
Throw for alter and silence for drop
2021-08-23 23:48:53 +00:00
Vitaly Baranov
63e8bc1f20
Added new commands BACKUP and RESTORE.
2021-08-23 23:41:45 +03:00
Anton Ivashkin
0e0e07ef46
Allow Multi Part Upload for small objects
2021-08-20 19:13:48 +03:00
Anton Ivashkin
51b543fb5c
Call AbortMultipartUpload when Multipart Upload Copy failed
2021-08-20 18:57:03 +03:00
vdimir
e77ff2740a
Add common/unit.h
2021-08-20 13:57:05 +03:00
Anton Ivashkin
b3737afe04
Use Multipart copy upload for large S3 objects
2021-08-19 13:09:58 +03:00
kssenii
709bc1a013
Continued
2021-08-16 14:50:26 +00:00
kssenii
9a02c6ad05
Catch any error if it happens on disk load when server is starting up
2021-08-16 13:18:33 +00:00
Alexey Milovidov
d184b79bba
Progress on async reads.
2021-08-16 03:00:32 +03:00
kssenii
073d7fdd5e
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-06 19:42:29 +00:00
Vladimir C
9de16c5017
Merge pull request #27216 from ianton-ru/MDB-13990
2021-08-06 18:12:00 +03:00
Anton Ivashkin
0e040ca40c
Add 10 seconds cache for S3 proxy resolver
2021-08-05 18:52:04 +03:00
Maksim Kita
4dc4854fad
Merge pull request #27168 from excitoon-favorites/safers3readbuffer
...
Safer `ReadBufferFromS3` for merges and backports
2021-08-04 15:01:21 +03:00
Vladimir Chebotarev
cc32a61488
Fix.
2021-08-04 10:40:02 +03:00
kssenii
087ddeb0dc
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into taiyang-li-disk-reload
2021-08-02 15:16:09 +00:00
kssenii
b25dfdd6af
Use method which was added for s3 instead
2021-08-02 08:15:21 +00:00
Kseniia Sumarokova
322870acd2
Update IDisk.h
2021-08-02 09:36:44 +03:00
Kseniia Sumarokova
9007ddfe22
Update DiskLocal.h
2021-08-02 09:34:39 +03:00
Kseniia Sumarokova
583c2d67cb
Update DiskSelector.cpp
2021-08-02 09:32:23 +03:00
Vitaly Baranov
a05184d83a
Fix synchronization while updating from the config of an encrypted disk.
2021-08-01 22:55:36 +03:00
kssenii
f1e96dbc55
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into taiyang-li-disk-reload
2021-07-29 07:53:16 +00:00
Vitaly Baranov
51f882d8cc
Support multiple keys for encrypted disk. Display error message if the key is probably wrong.
2021-07-24 23:45:12 +03:00
Vitaly Baranov
81ea8521b1
Set encryption algorithm explicitly.
2021-07-18 10:29:42 +03:00
Vitaly Baranov
958494b98e
Merge pull request #26377 from vitlibar/encrypted-disk-comments-and-improvements
...
Add comments for encrypted disks; some improvements in code.
2021-07-16 21:24:51 +03:00
Vitaly Baranov
9cc258327e
Add comments for encrypted disks; some improvements in code.
2021-07-16 12:04:25 +03:00
Zhichang Yu
fe5d17680e
Merge remote-tracking branch 'origin/master' into zero_copy_hdfs
2021-07-14 02:12:28 +00:00
Vitaly Baranov
bc5080d701
Rename ReadEncryptedBuffer -> ReadBufferFromEncryptedFile,
...
WriteEncryptedBuffer -> WriteBufferFromEncryptedFile,
move file FileEncryption.h to the IO folder,
rename functions accroding to our code style,
add cpp files.
2021-07-13 11:31:03 +03:00
Vitaly Baranov
042befb369
Fix compilation
2021-07-13 11:29:52 +03:00
Vitaly Baranov
e086e5e384
Merge branch 'master' into alexelex-encryption-1
2021-07-13 11:28:41 +03:00
Zhichang Yu
b4e6689bf9
fix test_hdfs_zero_copy_replication_move[tiered_copy-2]
2021-07-13 07:20:23 +00:00
Zhichang Yu
5047c758f4
fix per review
2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1
hdfs zero copy
2021-07-13 07:19:12 +00:00
Alexey Milovidov
81646f8389
Merge branch 'master' into ditch-aio
2021-07-11 03:06:45 +03:00
Anton Ivashkin
ac68d5ea71
Fix path concatenation
2021-07-08 18:39:41 +03:00
Anton Ivashkin
88bc3995e0
Fix throwing exception when iterate over non existing remote directory
2021-07-08 17:54:59 +03:00
Alexey Milovidov
21fb6ddea5
Remove AIO for sequential reads
2021-07-05 23:02:24 +03:00
Anton Ivashkin
e0199d8006
Fix DiskS3 restore
2021-06-22 20:05:20 +03:00
kssenii
babb11a887
Some fixes
2021-06-19 19:18:13 +00:00
