Kseniia Sumarokova
35e5b4e8a5
Update CachedReadBufferFromRemoteFS.cpp
2022-03-13 12:37:00 +01:00
kssenii
a18bb606a2
Update
2022-03-11 14:24:02 +01:00
kssenii
2b478e2501
Fix
2022-03-11 12:17:17 +01:00
kssenii
c86955deb2
Fix typo
2022-03-10 20:22:39 +01:00
kssenii
ddcb020d26
Fix test
2022-03-10 19:16:26 +01:00
kssenii
6571c7faa0
Clean up code
2022-03-09 10:36:52 +01:00
kssenii
96789fa5ce
Fix test, disable cache for DiskCacheWrapper
2022-03-09 09:46:04 +01:00
kssenii
16e84c265b
Fix build, update test
2022-03-08 21:41:58 +01:00
Azat Khuzhin
4200b56f3a
Introduce IDisk::supportParallelWrite()
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-08 22:17:36 +03:00
kssenii
5260822964
Merge master
2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0
Fix split build
2022-03-08 18:05:55 +01:00
kssenii
f3cba6f270
Remove already fixed crutch
2022-03-08 14:30:21 +01:00
kssenii
28a3aece60
Fix uncaught exception from destructor
2022-03-08 10:58:37 +01:00
Kseniia Sumarokova
517e878c6e
Merge pull request #35099 from ClickHouse/tavplubix-patch-1
...
Fix inconsistency in DiskLocal
2022-03-08 10:18:07 +01:00
kssenii
0737aef472
Fix split build
2022-03-07 22:02:07 +01:00
kssenii
eeb0ae9816
Fix detaching query
2022-03-07 17:05:36 +01:00
tavplubix
84e22fb32b
Update DiskLocal.cpp
2022-03-07 18:59:00 +03:00
kssenii
059d239815
Simplify code
2022-03-07 15:17:38 +01:00
kssenii
692b247379
Fix some corner cases
2022-03-07 14:31:21 +01:00
mergify[bot]
88052e2d7c
Merge branch 'master' into finally-enable-s3-async-writes-again
2022-03-07 12:55:52 +00:00
kssenii
2dcfe66236
Fix race
2022-03-06 20:33:07 +01:00
Kseniia Sumarokova
299e11bb2b
Update DiskCacheWrapper.cpp
2022-03-03 11:11:57 +01:00
kssenii
192b839cab
Fix
2022-03-02 19:11:45 +01:00
kssenii
b2cab429a7
Move initialization out of constructor
2022-03-01 18:12:34 +01:00
kssenii
438c63f3d8
Style
2022-03-01 17:52:38 +01:00
Nikolai Kochetov
eaf6d8c5f6
Update DiskS3.cpp
2022-02-24 17:35:29 +01:00
kssenii
8d6f1982df
Tiny refactoring
2022-02-23 14:43:40 +01:00
kssenii
36a76d6a75
Fix test
2022-02-22 15:18:28 +01:00
kssenii
c637385dd0
Merge master
2022-02-22 13:17:51 +01:00
Kseniia Sumarokova
eeea322556
Merge pull request #34629 from amosbird/remotefsimprove
...
Some refactoring and improvement over async and remote buffer related stuff
2022-02-22 11:36:40 +01:00
kssenii
fad7c042c1
Fix cache with non-async reads
2022-02-21 19:50:27 +01:00
kssenii
4543513a5d
Add comments
2022-02-21 13:54:03 +01:00
kssenii
e8b2e47e93
Style
2022-02-19 20:38:23 +01:00
kssenii
6b60f7c57f
Add limit for file segment size and split it if exceeds the limit
2022-02-19 18:59:22 +01:00
Amos Bird
84d111cde9
Better ReadBuffer asynchronicity
2022-02-20 01:08:21 +08:00
Nikolai Kochetov
40a48c5670
Enable async writes to S3.
2022-02-18 18:32:04 +00:00
kssenii
acf9a092a9
Review fixes
2022-02-18 19:09:54 +01:00
kssenii
353ffacca3
Add test
2022-02-17 19:57:04 +01:00
Amos Bird
dc0b4a8f73
Remove useless template
2022-02-17 14:31:21 +08:00
kssenii
47f94120da
Merge master
2022-02-16 14:43:28 +01:00
kssenii
3bd3e51aa0
Fix tests
2022-02-16 14:08:41 +01:00
Vladimir C
724340418e
Merge pull request #34617 from Algunenano/freebsd_build
...
Attempt to fix freeBSD build
2022-02-15 20:12:44 +01:00
Raúl Marín
72efc00830
Attempt to fix freeBSD build
2022-02-15 17:23:01 +01:00
alesapin
bc2d0ee7c7
Merge pull request #34215 from ClickHouse/revert-34211-revert-34153-add_func_tests_over_s3
...
Add func tests run with s3 and fix several bugs
2022-02-15 19:07:11 +03:00
kssenii
886b300b8d
Less seeks
2022-02-15 16:12:25 +01:00
mergify[bot]
34973f1d5c
Merge branch 'master' into first-test
2022-02-15 12:15:53 +00:00
alesapin
447cd56cb9
Fix comments
2022-02-15 12:11:50 +03:00
alesapin
e15396d90c
Fix race condition:
2022-02-14 22:19:49 +03:00
kssenii
bdbbe85f95
Fixes
2022-02-14 19:28:48 +01:00
zxealous
3d29920005
Fix compilation issues with DiskHDFS
2022-02-14 13:14:20 +08:00
Danila Kutenin
1c589681a0
Have a correct include which was inlined
2022-02-12 12:49:23 +00:00
alesapin
088f8cacf3
Small improvements
2022-02-10 19:50:21 +03:00
Anton Popov
70986a70a1
support TTL TO [DISK|VOLUME] [IF EXISTS]
2022-02-10 19:26:23 +03:00
alesapin
3387f9e597
Possible fix DiskHDFS
2022-02-10 16:55:12 +03:00
alesapin
10c3e6e546
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-09 14:14:58 +03:00
alesapin
36909a986f
Fix bug with files remove
2022-02-08 22:21:16 +03:00
alesapin
02a93cb852
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-08 19:42:27 +03:00
alesapin
f6e7cea1f8
Add settings for S3 multipart upload parts size
2022-02-08 19:38:04 +03:00
alesapin
07d5227568
Disable thread pool
2022-02-08 14:08:00 +03:00
alesapin
773d9cede3
Fix cache wrapper
2022-02-08 14:05:38 +03:00
alesapin
3af6012cb4
Revert "Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer""""
...
This reverts commit 2bc2ea485e
2022-02-08 11:01:26 +03:00
alesapin
2bc2ea485e
Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"""
...
This reverts commit fb77d7a7d5
2022-02-08 10:56:29 +03:00
kssenii
2e58733750
Merge master
2022-02-07 15:23:10 +01:00
Anton Popov
1b16db72c3
fix consecutive backward seeks in seekable read buffers
2022-02-07 17:20:26 +03:00
Nikolai Kochetov
40b86e3dae
Add blind change.
2022-02-07 12:02:21 +00:00
alesapin
fb77d7a7d5
Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer""
...
This reverts commit 875e5413ad
2022-02-07 12:36:54 +03:00
alesapin
25c0b07e68
Disable some tests
2022-02-04 14:31:58 +03:00
alesapin
875e5413ad
Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"
...
This reverts commit b92efed350ecce006cb2
2022-02-04 14:30:33 +03:00
alesapin
5e049fdd47
Merge with master
2022-02-04 12:28:19 +03:00
alesapin
2ed45b2a98
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-04 11:23:46 +03:00
alesapin
85fa727e32
Fix tidy
2022-02-04 11:21:03 +03:00
alesapin
a61b231ddd
Better debug
2022-02-03 11:23:09 +03:00
alesapin
ab546f61e3
Disable cache by default
2022-02-02 22:06:08 +03:00
Nikolai Kochetov
2003a96b58
Probably fix something
2022-02-02 17:48:41 +00:00
alesapin
80800e051e
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-02 19:42:04 +03:00
Nikolai Kochetov
ea044fc6b2
Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer
2022-02-02 19:40:41 +03:00
alesapin
b9c118524f
Fix race condition on hardlink/erase/read metadata
2022-02-02 19:40:21 +03:00
kssenii
aeec749173
Fix predownload case part 1
2022-02-02 17:25:25 +03:00
kssenii
5cba11428a
Fixes
2022-02-02 00:25:16 +03:00
Azat Khuzhin
bedf208cbd
Use fmt::runtime() for LOG_* for non constexpr
...
Here is oneliner:
$ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'
Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:
$ cat fmt.cocci
@@
expression log;
expression var;
@@
-LOG_DEBUG(log, var)
+LOG_DEBUG(log, fmt::runtime(var))
I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-01 14:30:03 +03:00
Amos Bird
e29c77f793
Fix link order issue of WriteBufferFromS3
...
(cherry picked from commit f22b09f4fc
2022-02-01 10:42:06 +00:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3""
2022-02-01 13:36:51 +03:00
alesapin
18f08ed932
Revert "Revert "Add func tests run with s3""
2022-02-01 10:20:06 +03:00
alexey-milovidov
99392b5ca7
Merge pull request #13544 from amosbird/mdha
...
Multi-Disk auto-recovery.
2022-02-01 06:13:26 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3"
2022-02-01 05:49:40 +03:00
alexey-milovidov
cbfcd45be3
Revert "Add func tests run with s3"
2022-02-01 05:46:13 +03:00
alexey-milovidov
a2aa147ce0
Merge pull request #34153 from ClickHouse/add_func_tests_over_s3
...
Add func tests run with s3
2022-02-01 05:46:09 +03:00
Amos Bird
ec7d367814
DiskLocal checker
...
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
2022-02-01 05:55:27 +08:00
alesapin
cbf4790ea4
More clear error
2022-01-31 13:59:16 +03:00
kssenii
218a633fec
Some fixes, more assertions and logging
2022-01-31 10:51:34 +03:00
kssenii
6b03c577d6
Make downloader's term resticted to one nextImpl
2022-01-29 23:58:53 +03:00
Nikolai Kochetov
1c9f026178
Merge branch 'master' into add-pool-to-s3-write-buffer
2022-01-28 16:01:42 +03:00
kssenii
c91b86e220
Better
2022-01-26 21:56:08 +03:00
Nikolai Kochetov
a8171269a1
Review fixes.
2022-01-26 17:55:24 +00:00
Nikolai Kochetov
efaed154d2
Update src/Disks/IDisk.h
...
Co-authored-by: alesapin <alesapin@clickhouse.com>
2022-01-26 14:58:38 +03:00
Nikolai Kochetov
2e88319b72
Fix integration tests.
2022-01-25 18:48:28 +00:00
Nikolai Kochetov
fcc29dbd15
Try to fix integration tests.
2022-01-25 15:26:36 +00:00
kssenii
aef99dedba
Fixes
2022-01-25 01:46:44 +03:00
Nikolai Kochetov
4afa75e80b
Fix deduplication.
2022-01-24 14:43:36 +00:00
kssenii
898139acae
Fix tests
2022-01-24 12:32:39 +03:00
kssenii
710bba895e
Fix partially downloaded no continuation case
2022-01-23 23:49:13 +03:00
kssenii
cf49f41f75
Better
2022-01-23 19:51:18 +03:00
zhongyuankai
040b71617c
Fix disk using the same path
2022-01-23 16:08:19 +08:00
alexey-milovidov
d1431fffd2
Merge pull request #33912 from amosbird/fixdiskcacherace
...
Fix race in DiskCacheWrapper
2022-01-23 08:47:44 +03:00
kssenii
690db142b9
Better
2022-01-23 02:12:33 +03:00
Kseniia Sumarokova
2c43e4eb57
Merge branch 'master' into local-cache-for-remote-fs
2022-01-22 22:53:55 +03:00
kssenii
36a41ac146
Better
2022-01-22 20:42:39 +03:00
Amos Bird
da66239864
Fix race in DiskCacheWrapper
2022-01-22 23:56:30 +08:00
kssenii
9f295ebf92
Fix threadpool reads
2022-01-22 01:07:20 +03:00
Maksim Kita
a3adf12a50
Merge pull request #33862 from ClickHouse/fix-s3-proxy-config-resolver-ipv6-only
...
Try every resolved ip address while getting S3 proxy.
2022-01-21 20:29:19 +01:00
Nikolai Kochetov
a024ddd1d4
Merge branch 'add-pool-to-s3-write-buffer' of github.com:ClickHouse/ClickHouse into add-pool-to-s3-write-buffer
2022-01-21 18:42:24 +00:00
Nikolai Kochetov
b3cbb63487
Merge branch 'master' into add-pool-to-s3-write-buffer
2022-01-21 21:41:54 +03:00
Nikolai Kochetov
ae1b669fca
Even more async writes.
2022-01-21 15:55:19 +00:00
kssenii
1858f69f5a
Make space reservation incremental
2022-01-21 18:39:34 +03:00
Nikolai Kochetov
a178e54d10
Try every resolved ip address while getting S3 proxy.
2022-01-21 13:19:25 +03:00
kssenii
a566099759
Local cache for remote filesystem
2022-01-19 00:42:11 +03:00
Anton Ivashkin
7c26a832f8
Fix restore S3 disk without detached directories
2022-01-18 12:31:31 +03:00
Nikolai Kochetov
6d49a62666
Some more async writes.
2022-01-14 19:53:55 +00:00
Nikolai Kochetov
843983ea06
Add batch removing to IDiskl.
2022-01-12 15:58:04 +00:00
alesapin
733ed7cb54
Merge pull request #32061 from ianton-ru/MDB-15474
...
Change ZooKeeper path for zero-copy locks for shared data
2022-01-08 17:17:11 +03:00
xiedeyantu
1edfc03825
Update AsynchronousReadIndirectBufferFromRemoteFS.h
...
update 1024 * 1024 to DBMS_DEFAULT_BUFFER_SIZE
2022-01-07 21:06:53 +08:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474
2022-01-07 10:12:25 +00:00
Azat Khuzhin
71a99ab27c
Accept real file size in createReadBufferFromFileBase()
...
Right now streams relies on correct file size not the number of bytes
that will be read from the stream, to overcome one bug in the linux
kernel that may return EIINVAL for pread() with offset pass the EOF.
v2: Swap read_hint and file_size (since it is easy to miss something)
Before the first argument to readFile()/createReadBufferFromFileBase()
was read_hint, not the file_size, and let's preserve the order, since
it is easy to miss something
This will also fix 02051_read_settings test automatically because now
MergeTreeReaderStream will pass estimated_sum_mark_range_bytes to
read_hint not file_size, previously it cause on of the following errors:
- Attempt to read after EOF w/ O_DIRECT
- and LOGICAL_ERROR while adjusting granulas w/o O_DIRECT
This will also improve zero-length reads guard (via
ReadBufferFromEmptyFile), that had been added in #30190
v3: fix for other storages that wasn't enabled in fast-test
v4: ignore ENOENT/ENOTSUP in readFile
2022-01-04 10:53:17 +03:00
alesapin
7a3c874651
Merge branch 'master' into ianton-ru-MDB-15474
2021-12-29 19:01:43 +03:00
Nikolai Kochetov
68bf38133a
Add pool to WriteBufferFromS3
2021-12-29 14:04:21 +00:00
Anton Ivashkin
2d87f0a0e3
Fix debug build
2021-12-28 11:45:38 +03:00
Alexey Milovidov
f03cb4c762
Cleanup trash from Kafka and HDFS
2021-12-25 06:10:59 +03:00
Maksim Kita
60d16afde7
Merge pull request #33027 from ClickHouse/merge-33023
...
Merge #33023
2021-12-22 13:44:14 +03:00
Kseniia Sumarokova
484a4d62dd
Merge pull request #32835 from kssenii/fising-s3
...
Fix optimization with lazy seek for async reads from remote fs
2021-12-22 09:38:23 +03:00
Alexey Milovidov
7b48cb813e
Merge branch 'Issue129' of github.com:DevTeamBK/ClickHouse into merge-33023
2021-12-22 02:17:00 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474
2021-12-21 17:38:36 +03:00
Anton Ivashkin
0c0bf66334
Merge master
2021-12-21 17:27:54 +03:00
kssenii
ebcb66a361
Fix
2021-12-21 10:58:02 +03:00
Nikolai Kochetov
35883e0dae
Merge pull request #32979 from ClickHouse/decrease-some-s3-log-level
...
Decrease log level for some s3 messages.
2021-12-20 20:56:22 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment
...
Loops remove postfix increment
2021-12-20 19:51:07 +03:00
Nikolai Kochetov
7dea7b7f76
Decrease log level for some s3 messages.
2021-12-20 18:18:54 +03:00
Anton Ivashkin
f0b9a4327a
Use table UUID in zero-copy shared label in ZooKeeper
2021-12-20 16:19:00 +03:00
Maksim Kita
51477adf1b
Updated additional cases
2021-12-20 15:55:07 +03:00
Maksim Kita
3feab5a975
Containers iteration fix erase
2021-12-20 13:42:31 +03:00
root
6b3366dc70
[Issue #129 ] Fix Weak Regular Expression [v21.3-insecure]
...
Problem Descripton:
==================
The “key_regexp” constant in the “DiskS3” class contains a regular expression (RegEx) for checking a key path.
However, the RegEx starts and ends with “.*”.
This is a wildcard that an attacker could use to exploit this file path check by padding the beginning and end with whatever path they’d like.
Recommendation:
==============
Tighten up the RegEx by replacing the “.*” at the beginning and end with more specific items.
Fix:
===
Replace the .* at the end with $ as there is no need to continue after the key file.
Leave the beginning .* as is because the path for the key can have any characters or in any path of the system.
2021-12-17 10:30:16 -05:00
kssenii
40c266840c
Fixes
2021-12-17 18:16:29 +03:00
kssenii
ed42620530
Rename BlobStorage to AzureBlobStorage
2021-12-17 12:24:05 +03:00
kssenii
ca283a4bb8
Fix optimization with lazy seek
2021-12-17 00:18:28 +03:00
alesapin
c338bee718
Revert "Fix build issue related to azure blob storage"
2021-12-16 14:37:12 +03:00
alesapin
c0cebaea79
Merge pull request #32788 from amosbird/buildfix
...
Fix build issue related to azure blob storage
2021-12-16 13:45:22 +03:00
Amos Bird
9c9f4a3e6c
build fix
2021-12-15 16:55:18 +08:00
alesapin
9974f10631
Fix clang tidy
2021-12-14 15:34:20 +03:00
Kseniia Sumarokova
78ed383f15
Merge pull request #31505 from ContentSquare/azure_blob_storage_dependency
...
Azure Blob Storage Disks
2021-12-13 18:28:48 +03:00
Azat Khuzhin
57c027be0e
Remove arcadia build support
2021-12-11 21:25:23 +03:00
Jakub Kuklis
578aec97e9
Removing TODO and NOTE marks used mainly as hooks for review in the PR
2021-12-10 22:04:38 +01:00
Jakub Kuklis
a3bc6d2fb7
Style corrections
2021-12-10 22:04:38 +01:00
Jakub Kuklis
74e26354cc
Committing missing files
2021-12-10 22:04:38 +01:00
Jakub Kuklis
41a88e49c9
Abstracting metadata creation for remote disks registrations, putting Blob Storage auth in a separate file
2021-12-10 22:04:38 +01:00
Jakub Kuklis
e58f74d8dc
Connecting to Azurite and invoking simple INSERTs works in the integration test
2021-12-10 22:04:38 +01:00
Jakub Kuklis
8afe69ff93
Comming back to StorageSharedKeyCredential for the integration test, get 400 Bad requests from Azurite when trying to create a container now
2021-12-10 22:04:38 +01:00
Jakub Kuklis
f180a7ba5f
Solved Invalid storage account error with connection strings
2021-12-10 22:04:38 +01:00
Jakub Kuklis
008141de36
Enable authentication via StorageSharedKeyCredential
2021-12-10 22:04:38 +01:00
Jakub Kuklis
d51f979c7a
Adding endpoint validation
2021-12-10 22:04:38 +01:00
Jakub Kuklis
2b7725bfc7
Enable connecting directly to storage accounts and creating new containers
2021-12-10 22:04:38 +01:00
Jakub Kuklis
910db6ca57
Code cleanup addressing PR remarks
2021-12-10 22:04:38 +01:00
Jakub Kuklis
f6c5a2c63c
Removing file checks after changing blob filenames
2021-12-10 22:04:38 +01:00
Jakub Kuklis
3aa6a0ead3
Adding random postfix to blob names, needs code cleanup
2021-12-10 22:04:38 +01:00
Jakub Kuklis
5ed511f9da
Working skeleton for Blob Storage integration test
2021-12-10 22:04:37 +01:00
Jakub Kuklis
0c65cfed1f
Addressing first PR remarks
2021-12-10 22:04:37 +01:00
Jakub Kuklis
7c2e116214
Adding max_retries config
2021-12-10 22:04:37 +01:00
Jakub Kuklis
a33cea0525
Refactors after proper rebase on master
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b3728dbbaf
Rebase on master
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b8ef47006c
Minor refactors before PR
2021-12-10 22:04:37 +01:00
Ryad ZENINE
c4421d6866
Small changes following review
2021-12-10 22:04:37 +01:00
Jakub Kuklis
7f595eedbc
Catch Blob Storage exceptions
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b3c4a92cc0
Code cleanup
2021-12-10 22:04:37 +01:00
Jakub Kuklis
c4691affb2
Adding a new test for read with offset
2021-12-10 22:04:37 +01:00
Jakub Kuklis
9a00acfb72
Fixing read size error plus minor improvements
2021-12-10 22:04:37 +01:00
Jakub Kuklis
c9f3164cba
Bringing back old single part upload limit
2021-12-10 22:04:37 +01:00
Jakub Kuklis
0d7a4b10c3
Using unique_ptr for Blob Client in Read Buffer
2021-12-10 22:04:37 +01:00
Jakub Kuklis
5e2b2b1eee
Better search for blobs with a given name
2021-12-10 22:04:37 +01:00
Jakub Kuklis
33247db3f4
Adding finalize() to WriteBuffer, always using block blob uploads, code cleanup
2021-12-10 22:04:37 +01:00
Jakub Kuklis
bf2476a076
Shared pointer for Azure client and fixing path reference in ReadBuffer
2021-12-10 22:04:37 +01:00
Jakub Kuklis
60dd22dd4e
Checking for success of file removal operation
2021-12-10 22:04:37 +01:00
Jakub Kuklis
486f267b3b
Further code cleanup and error codes improvement
2021-12-10 22:04:37 +01:00
Jakub Kuklis
6755c5e897
File deletion prototype
2021-12-10 22:04:37 +01:00
Jakub Kuklis
a0cfafa387
checkUniqueId implementation using ListBlobs
2021-12-10 22:04:37 +01:00
Jakub Kuklis
95816875f4
Change false to true in supportZeroCopyReplication
2021-12-10 22:04:37 +01:00
Jakub Kuklis
b41ab197e8
Blob Storage endpoint url verification using re2
2021-12-10 22:04:37 +01:00
Jakub Kuklis
63ec8d4244
Adding multipart upload to WriteBuffer, retries in ReadBuffer, additional settings for buffers
2021-12-10 22:04:37 +01:00
Jakub Kuklis
02060791e3
Adding settings getter and applyNewSettings
2021-12-10 22:04:37 +01:00
Jakub Kuklis
3f28f2c62b
Enabling local cache for DiskBlobStorage
2021-12-10 22:04:37 +01:00
Jakub Kuklis
e3daf8aa49
Increase default thread pool size
2021-12-10 22:04:37 +01:00
Jakub Kuklis
dbb98e3636
Code cleanup, leaving some debug information printing under #ifdef VERBOSE_DEBUG_MODE
2021-12-10 22:04:37 +01:00
Jakub Kuklis
8230f8039c
Two column MergeTree table with Blob Storage disk work! Fixed buffer in ReadBufferFromBlobStorage to be permanent (class attribute)
2021-12-10 22:04:37 +01:00
Jakub Kuklis
2b4d3665ab
Basic inserts and selects work for a MergeTree table with a Blob Disk, based on a single column table
2021-12-10 22:04:37 +01:00
Jakub Kuklis
4b51ddee1b
Initial check with write and read works for the Blob Storage Disk
2021-12-10 22:04:37 +01:00
Jakub Kuklis
03f85dee41
Managed to read markers file in SELECT, getting error for CODEC version now
2021-12-10 22:04:37 +01:00
Jakub Kuklis
c43b90c58c
Progressing with WriteBufferFromBlobStorage
2021-12-10 22:04:37 +01:00
Jakub Kuklis
fb4c1d6686
Starting insertion pipeline
2021-12-10 22:04:37 +01:00
Jakub Kuklis
6436f5b04d
Adding an authenticated client to DiskBlobStorage
2021-12-10 22:04:37 +01:00
Jakub Kuklis
ee1a568919
The following query passes without erors: CREATE TABLE ... ENGINE = MergeTree ... storage_policy = 'blob_storage', debug constants present in the code
2021-12-10 22:04:37 +01:00
Jakub Kuklis
cacdac7ba6
Moving implementation to the .cpp file
2021-12-10 22:04:37 +01:00
Jakub Kuklis
88b5feb59d
Registering the Blob Storage Disk
2021-12-10 22:04:36 +01:00
Jakub Kuklis
20dda7f666
Giving Disk Blob Storage files proper names
2021-12-10 22:04:36 +01:00
Jakub Kuklis
0cf338e165
Skeleton for DiskBlobStorage
2021-12-10 22:04:36 +01:00
Jakub Kuklis
43f420cba8
Adding a flag to use Azure Blob Storage
2021-12-10 22:04:36 +01:00
Jakub Kuklis
162e2b334b
Initial working example of Azure Blob Storage manipulation from within ClickHouse
2021-12-10 22:04:36 +01:00
leosunli
3946b156a1
Fix bug
...
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
2021-12-06 14:32:32 +08:00
leosunli
6f611eec91
Fix bug
...
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
2021-12-06 14:32:32 +08:00
leosunli
e81a1dbe6a
Make HDFS replication configurable in WriteBufferFromHDFSImpl#WriteBufferFromHDFSImpl
...
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
2021-12-06 14:32:31 +08:00
leosunli
5be58a8839
Make HDFS replication configurable in WriteBufferFromHDFSImpl#WriteBufferFromHDFSImpl
...
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
2021-12-06 14:32:31 +08:00
sunlisheng
0f045fa20a
rename replication
...
Signed-off-by: sunlisheng <lisheng.sls@alibaba-inc.com>
2021-12-06 14:32:31 +08:00
sunlisheng
ef77f45f59
Make HDFS replication configurable in WriteBufferFromHDFSImpl#WriteBufferFromHDFSImpl
...
Signed-off-by: sunlisheng <lisheng.sls@alibaba-inc.com>
2021-12-06 14:32:31 +08:00
Anton Ivashkin
0e685c1aba
Fix types
2021-12-01 19:11:17 +03:00
Anton Ivashkin
e0a16a483e
Merge master
2021-12-01 19:03:31 +03:00
Anton Ivashkin
0f9038ebed
Zero-copy: move shared mark outside table node in ZooKeeper
2021-11-29 19:05:31 +03:00
Kseniia Sumarokova
93cf66df12
Merge pull request #30936 from kssenii/seekable-read-buffers
...
Reduce memory usage for some formats when reading with s3/url/hdfs
2021-11-25 11:19:24 +03:00
Kruglov Pavel
d9c1a0c8ec
Merge branch 'master' into fix-write-buffers
2021-11-20 17:48:24 +03:00
kssenii
ff969b4605
Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers
2021-11-20 15:03:13 +03:00
Vitaly Baranov
33ea7a7262
Rename RowPolicy::ConditionType -> RowPolicyFilterType and move it to Access/Common.
2021-11-19 00:14:23 +03:00
alexey-milovidov
cb6342025d
Merge pull request #31229 from zhanglistar/fsync
...
Change all fsync to fdatasync
2021-11-14 05:16:01 +03:00
Kseniia Sumarokova
59d4eb99c2
Merge pull request #31291 from kssenii/update-async-reads-setting
...
Make remote_filesystem_read_method=threadpool by default
2021-11-13 19:19:35 +03:00
kssenii
f18dcd2287
Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers
2021-11-13 14:38:57 +03:00
Kseniia Sumarokova
c7be79b4e7
Merge pull request #31136 from LiuYangkuan/IDiskRemote_metadata_disk
...
use DiskPtr instead of OS's file system API in class IDiskRemote
2021-11-13 12:16:05 +03:00
Kseniia Sumarokova
c6fd56ede9
Update DiskLocal.cpp
2021-11-13 00:00:32 +03:00
LiuYangkuan
4366fcbc83
fix build check and make rename robust in DiskS3::restoreFileOperations
2021-11-12 22:54:10 +08:00
kssenii
6f895036e6
Disable for hdfs until tested, adjust tests with profile events
2021-11-12 14:09:19 +03:00
zhanglistar
4e7823ade7
1. add macosx fix
2021-11-12 15:49:21 +08:00
LiuYangkuan
f7078a17ce
add DiskLocalDirectoryIterator() when path not directory or nonexist, fix remove from_path after moveDirectory in DiskS3::restoreFileOperations
2021-11-12 11:36:48 +08:00
kssenii
031754b465
Fix
2021-11-12 01:37:18 +03:00
Kseniia Sumarokova
e709c3ba90
Update AsynchronousReadIndirectBufferFromRemoteFS.h
2021-11-12 01:13:44 +03:00
Kseniia Sumarokova
2313981fd7
Merge pull request #31260 from azat/external-cleanup
...
Cleanup extern ProfileEvents/CurrentMetrics and add a style check
2021-11-12 00:02:57 +03:00
Kseniia Sumarokova
e20cb0c10a
Merge branch 'master' into retriable-http
2021-11-11 11:52:06 +03:00
avogar
c521a9131a
Small refactoring of WriteBiffer-s
2021-11-11 02:11:18 +03:00
kssenii
62c70ce5cf
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-11-10 21:16:11 +00:00
Azat Khuzhin
baf14444e6
Cleanup ProfileEvents and CurrentMetrics
2021-11-10 21:15:27 +03:00
LiuYangkuan
006dca1362
use DiskPtr instead of OS's file system API in class IDiskRemote
2021-11-10 22:50:15 +08:00
zhanglistar
aaad3f84da
Merge branch 'ClickHouse:master' into fsync
2021-11-10 17:44:08 +08:00
zhanglistar
96b8df5899
1. chagne all fsync to fdatasync
2021-11-10 17:40:16 +08:00
kssenii
618c332f9c
Rename read_threadpool to threadpool
2021-11-10 07:46:18 +00:00
alexey-milovidov
7006147268
Merge pull request #31159 from snar/freebsd-disk-space
...
correct disk space calculations
2021-11-10 02:48:29 +03:00
Kruglov Pavel
17085183bf
Merge pull request #31042 from Avogar/hdfs-bad-url
...
Add additional hdfs url check
2021-11-09 16:23:59 +03:00
kssenii
36f213e51c
Fix build
2021-11-09 07:11:44 +00:00
Kseniia Sumarokova
c458f73138
Update AsynchronousReadIndirectBufferFromRemoteFS.cpp
2021-11-08 22:12:04 +03:00
kssenii
83d64f1c2d
Better
2021-11-08 13:32:32 +03:00
kssenii
b4bebc5977
Add comments
2021-11-08 12:01:24 +03:00
Alexandre Snarskii
248709ea16
correct disk space calculations
2021-11-07 19:51:48 +03:00
kssenii
9189966106
Fix
2021-11-05 20:01:21 +03:00
kssenii
4bc2a29345
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-11-04 14:41:24 +00:00
kssenii
b95f0738fb
Fix
2021-11-04 14:39:53 +00:00
kssenii
ec11179f91
Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers
2021-11-03 14:33:31 +03:00
kssenii
45ea820297
Reduce memory usage for some formats
2021-11-03 14:30:03 +03:00
avogar
3839d521a0
Add additional hdfs url check
2021-11-03 14:01:50 +03:00
Vladimir C
77edd081a5
Fix whitespace in src/Disks/S3/DiskS3.cpp
2021-11-01 13:00:21 +03:00
Vladimir C
2e58a3bf62
Merge branch 'master' into MDB-15168
2021-11-01 11:59:13 +03:00
kssenii
2940d9fd19
Squashed commit of the following:
...
commit a1acc7ed3485bd158cf42f26b2d2a1bca84b7269
Author: kssenii <sumarokovakseniia@mail.ru>
Date: Sun Oct 31 14:47:25 2021 +0000
Fix
commit 12a27d445bf17e321a7355f1ba857d11b70a285chttps://github.com/ClickHouse/ClickHouse into retriable-http
commit 55f1ba857daeba8104d182f3754b8a2647b88a667024f51b552bb586bed3https://github.com/ClickHouse/ClickHouse into retriable-http
commit 7024f51b557de5fca0751ecb6ad05d855b10261chttps://github.com/ClickHouse/ClickHouse into retriable-http
commit 1ecb6ad05d710ee9687828d2485716158b4c26b7https://github.com/ClickHouse/ClickHouse into retriable-http
commit 28d24857167c20ca07b7ccb02cac569e4fe0f3c8d1138a8a25https://github.com/ClickHouse/ClickHouse into retriable-http
commit 9e4fe0f3c89814cb1b45daed77038bhttps://github.com/ClickHouse/ClickHouse into retriable-http
commit 9814cb1b451c8b1b1133969999ff10https://github.com/ClickHouse/ClickHouse into retriable-http
commit 1c8b1b1133e7dbfff5d23329b668d6198adc7ecdhttps://github.com/ClickHouse/ClickHouse into retriable-http
2021-10-31 17:53:08 +03:00
kssenii
12a27d445b
Adjutments after merge with master
2021-10-29 09:29:35 +00:00
kssenii
f17e321a73
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-10-29 08:03:25 +00:00
kssenii
55f1ba857d
Small fixes
2021-10-29 07:38:42 +00:00
kssenii
4f3433b4e1
Some clean up before merging
2021-10-29 01:04:52 +03:00
kssenii
7e2ea97e3c
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-28 18:46:21 +03:00
Anton Ivashkin
6cda420b56
Cleanup after fail in localBackup
2021-10-28 17:50:59 +03:00
kssenii
aeba8104d1
Some more fixes
2021-10-28 10:28:05 +00:00
kssenii
3d01fbf603
Review fixes
2021-10-28 11:32:37 +03:00
Alexey Milovidov
8b4a6a2416
Remove cruft
2021-10-28 02:10:39 +03:00
kssenii
9461e225bb
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-27 22:47:47 +03:00
kssenii
2647b88a66
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-10-27 19:38:19 +00:00
kssenii
7024f51b55
Review fixes and add test
2021-10-27 19:37:36 +00:00
Azat Khuzhin
9936ee5181
Update gtest_disk_encrypted for new readFile() interface
2021-10-27 09:37:26 +03:00
Azat Khuzhin
7ce01b328e
Update gtest_disk_encrypted for new createReadBufferFromFileBase() interface
2021-10-27 09:37:26 +03:00
kssenii
52296e8b04
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-25 00:22:32 +03:00
kssenii
658b33717f
A little nicer
2021-10-24 20:43:34 +03:00
kssenii
7de5fca075
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
2021-10-24 17:04:24 +00:00
Azat Khuzhin
34d8f78c33
Explictly pass size in DiskCacheWrapper::writeFile()
...
v0: Add explicit default value for DiskLocal::readFile()
v2: Explictly pass size in DiskCacheWrapper::writeFile()
(since v0 not google-default-arguments compatible)
2021-10-24 18:58:22 +03:00
Azat Khuzhin
4599e9191b
Fix explicit 0 size for readFile()
2021-10-24 18:58:22 +03:00
Azat Khuzhin
2b96433daf
Replace estimated_size with optional<size> for IDisk::readFile()
...
v2: rebase to fix conflicts in DiskEncrypted.cpp
2021-10-24 18:58:11 +03:00
Vitaly Baranov
c0b01e7b0b
Merge pull request #30494 from vitlibar/fix-reading-from-empty-file-on-encrypted-disk
...
Fix reading from empty file on encrypted disk
2021-10-24 16:24:47 +03:00
kssenii
1ecb6ad05d
Better
2021-10-23 19:00:14 +00:00
Vitaly Baranov
26cb62de32
Add more tests for the "encrypted" disk type.
2021-10-23 20:58:37 +03:00
kssenii
7f2c5dcaae
Fix clang-tidy
2021-10-22 01:08:58 +03:00
kssenii
56f0560c46
Asserts and read till end option
2021-10-21 20:54:03 +03:00
Vitaly Baranov
172a6f2e0d
Fix reading from empty file on encrypted disk.
2021-10-21 19:31:29 +03:00
kssenii
120b01e089
Better
2021-10-21 18:22:17 +03:00
kssenii
e4e157688d
Review fixes
2021-10-21 01:57:43 +03:00
kssenii
31cd71c849
Clean up
2021-10-20 09:37:18 +03:00
kssenii
143e912e4e
Fix split build
2021-10-19 22:45:46 +03:00
kssenii
49106f407f
Fix some checks
2021-10-19 09:34:27 +03:00
kssenii
d5d4817350
Add disk proxies back, add setReadUntilPosition to ReadBuffer
2021-10-18 18:35:11 +03:00
kssenii
28d2485716
Some fixes
2021-10-15 17:27:26 +00:00
kssenii
7c20ca07b7
Fix style check
2021-10-15 12:59:24 +00:00
kssenii
c1c574e9ca
Better
2021-10-15 13:14:57 +03:00
kssenii
3995506d37
Adjust range reader for remote fs reads
2021-10-15 11:36:26 +03:00
kssenii
1c8b1b1133
Better
2021-10-12 09:33:07 +00:00
kssenii
31d98b0992
Remove redundant
2021-10-11 08:17:22 +03:00
kssenii
448d63a0fc
Range read with upper bound offset
2021-10-11 01:53:51 +03:00
kssenii
1f8b449bc5
Lazy seek avoiding
2021-10-11 00:51:43 +03:00
kssenii
e3d2942aa6
Correct merge
2021-10-10 20:24:36 +03:00
kssenii
fde6f0507d
Merge remote-tracking branch 'origin/retriable-http' into disk-async-read
2021-10-10 20:23:44 +03:00
kssenii
3329b668d6
Fix style check
2021-10-10 15:51:37 +00:00
kssenii
40434519e8
Fix disk web
2021-10-08 16:02:11 +03:00
kssenii
f3f820c6e0
HTTP read buffer retries
2021-10-08 15:41:55 +03:00
kssenii
d6e1d0503d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-07 16:21:00 +03:00
Filatenkov Artur
dc657a2c5e
Add error for multiple keys without current_key_id ( #29546 )
...
* add error for multiple keys without current_key_id
* add exception if one key with non zero id is used without current_key_id
* refactoring
2021-10-05 14:37:54 +03:00
Azat Khuzhin
a7793101bc
Fix data-race in WriteIndirectBuffer (used in StripeLog)
...
CI founds [1]:
WARNING: ThreadSanitizer: data race (pid=497)
Write of size 8 at 0x7b30000260b8 by thread T381 (mutexes: write M546759850860896464):
...
3 DB::DiskMemory::writeFile() obj-x86_64-linux-gnu/../src/Disks/DiskMemory.cpp:339:15 (clickhouse+0x14a1849f)
4 DB::FileChecker::save() const obj-x86_64-linux-gnu/../src/Common/FileChecker.cpp:111:50 (clickhouse+0x15bbe19f)
5 DB::TinyLogSink::TinyLogSink() obj-x86_64-linux-gnu/../src/Storages/StorageTinyLog.cpp:214:34 (clickhouse+0x15bca215)
Previous read of size 8 at 0x7b30000260b8 by thread T460:
...
2 DB::WriteIndirectBuffer::finalize() obj-x86_64-linux-gnu/../src/Disks/DiskMemory.cpp:106:33 (clickhouse+0x14a1e044)
9 DB::StripeLogSink::onFinish() obj-x86_64-linux-gnu/../src/Storages/StorageStripeLog.cpp:227:30 (clickhouse+0x15bd241a)
[1]: https://clickhouse-test-reports.s3.yandex.net/29680/d0fc26f91a0141b56a0550741219c3dc43630e03/stress_test_(thread)/stderr.log
2021-10-04 01:58:35 +03:00
Mike Kot
38232c286d
fixing paths
2021-10-02 21:45:05 +02:00
Mike Kot
65e6e211b4
Merge remote-tracking branch 'upstream/master' into improvement/fn-traits
2021-10-02 21:37:50 +02:00
Alexey Milovidov
fe6b7c77c7
Rename "common" to "base"
2021-10-02 10:13:14 +03:00
kssenii
7ff092712b
Better
2021-10-01 18:13:16 +03:00
Mike Kot
38b02b121a
Merge remote-tracking branch 'upstream/master' into improvement/fn-traits
2021-10-01 16:55:28 +02:00
kssenii
55972db866
Small fix and more comments
2021-10-01 11:38:58 +03:00
kssenii
e4bc81ac78
Better
2021-09-30 21:45:57 +03:00
kssenii
cc9ce13bac
Finally all fixed
2021-09-30 15:35:59 +03:00
kssenii
85aae03a26
Fix
2021-09-29 22:54:49 +03:00
Mike Kot
7670ea50c1
Replacing std::function to Fn in some places
...
to avoid dynamic allocating while keeping desired function signature
clear.
Simplifying SimpleCache (CachedFn) using C++20
2021-09-29 13:09:39 +02:00
kssenii
20192352ad
Better
2021-09-29 11:03:36 +03:00
kssenii
ebca30694d
Fix
2021-09-27 18:25:30 +03:00
kssenii
efddc03246
Async read for disk web
2021-09-26 22:49:28 +03:00
alexey-milovidov
85fc160eb4
Merge pull request #29346 from ClickHouse/remove-ya-make
...
Remove ya.make
2021-09-25 19:43:40 +03:00
Alexey Milovidov
cd7f9d981c
Remove ya.make
2021-09-25 04:22:54 +03:00
alexey-milovidov
62d606bb76
Update DiskWebServer.h
2021-09-25 02:58:12 +03:00
Alexey Milovidov
4a44bcaced
Fix trivial mistake in DiskWebServer
2021-09-25 01:33:40 +03:00
kssenii
6219d541a5
Finally fixed
2021-09-24 15:29:04 +03:00
kssenii
c1ea44b9c2
Fixes
2021-09-24 13:38:08 +03:00
kssenii
1b9565dfa8
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-09-22 22:08:44 +03:00
kssenii
e301457e91
Fix prefetch mistake
2021-09-21 15:40:48 +03:00
kssenii
975033e6e3
Fix
2021-09-21 09:44:54 +00:00
kssenii
40ee75fe97
Common thread pool for disks
2021-09-21 10:00:56 +03:00
kssenii
cacc81ca61
Fix
2021-09-21 02:31:02 +03:00
kssenii
0104cb1c8e
Better
2021-09-21 01:51:17 +03:00
kssenii
f312760dca
Better
2021-09-20 19:57:15 +03:00
kssenii
11ae94b8d3
Async read for remote fs
2021-09-20 16:16:46 +03:00
Kseniia Sumarokova
2a8f355326
Merge pull request #29179 from kssenii/better-retries
...
Better retries for disk web
2021-09-20 08:25:49 +03:00
kssenii
12b90da5de
Better retries
2021-09-19 20:50:50 +00:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask ( #25165 )
...
Introduced MergeTask and MutateTask
2021-09-17 00:19:58 +03:00
Kseniia Sumarokova
ef6ba90be1
Update DiskWebServer.h
2021-09-13 20:00:18 +03:00
kssenii
9ec6a9b4c9
Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-web-disk-2
2021-09-13 08:32:34 +03:00
kssenii
147e413346
Add retries with backoff, better exceptions
2021-09-13 00:04:05 +03:00
Mike Kot
c4385a2c40
Simplifying exception messages
2021-09-11 00:40:18 +02:00
kssenii
939032b0eb
Slightly better
2021-09-10 13:41:41 +03:00
kssenii
c23e17d369
Fix
2021-09-09 15:25:44 +03:00
kssenii
72505db3bd
Fix checks
2021-09-09 10:10:34 +03:00
kssenii
1101ee7808
Update example
2021-09-08 22:43:46 +03:00
kssenii
7345ad8e09
Add no throw
2021-09-08 22:40:26 +03:00
kssenii
7141b5d041
Get rid of regexp
2021-09-08 20:22:24 +03:00
Mike Kot
8e9aacadd1
Initial: replacing hardcoded toString for enums with magic_enum
2021-09-06 16:24:03 +02:00
kssenii
97e0a0310a
Slightly better
2021-09-04 12:02:07 +03:00
kssenii
32686e11cb
Fix style checl
2021-09-04 11:30:30 +03:00
kssenii
6ee94c923f
Fix
2021-09-04 00:51:11 +03:00
Kseniia Sumarokova
63a0eb3922
Update DiskWebServer.cpp
2021-08-31 21:51:00 +03:00
kssenii
248fe1fd09
Fix disk
2021-08-31 16:54:43 +03:00
Nikita Mikhaylov
00394014ff
Merge pull request #28382 from DimasKovas/fix_arcadia_build_31_08_2021
...
Try to fix arcadia build (generate ya.make)
2021-08-31 13:49:32 +03:00
Dmitrii Kovalkov
b1eb48804b
Run generate-ya-make.sh
2021-08-31 09:05:33 +03:00
Alexey Milovidov
ca6f6306f3
Merge branch 'master' into async-reads
2021-08-31 02:52:58 +03:00
Kseniia Sumarokova
00bea71610
Merge pull request #25251 from kssenii/disk-over-web-server
...
ClickHouse over web server with static files
2021-08-30 16:18:59 +03:00
Dmitrii Kovalkov
9871ad70ff
Exclude fuzzers
2021-08-30 11:12:25 +03:00
Dmitrii Kovalkov
8acf0fa16a
run generate-ya-make
2021-08-30 10:57:40 +03:00
kssenii
3e7f022d66
Extend usage comment explanation a little
2021-08-28 20:51:20 +00:00
kssenii
f27f519aa2
Fix build and add example
2021-08-28 20:35:51 +00:00
Alexey Milovidov
da4865a5d8
Remove excessive logging from DiskS3
2021-08-25 03:22:26 +03:00
Alexey Milovidov
183891eb0a
Add IDisk::isRemote method
2021-08-25 02:27:20 +03:00
Alexey Milovidov
fd27a2faef
Remove imperfect code
2021-08-25 02:05:55 +03:00
Alexey Milovidov
7c1d0a3baf
Progress on development
2021-08-25 01:24:47 +03:00
Alexey Milovidov
8f57216180
Progress on development
2021-08-25 00:45:58 +03:00
Alexey Milovidov
4d9ad3725d
Merge branch 'master' into async-reads
2021-08-25 00:45:40 +03:00
Kseniia Sumarokova
d78e551de9
Update src/Disks/DiskWebServer.cpp
...
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
2021-08-24 12:24:54 +03:00
kssenii
b536653851
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-24 07:29:50 +00:00
kssenii
5307d31924
Throw for alter and silence for drop
2021-08-23 23:48:53 +00:00
Vitaly Baranov
63e8bc1f20
Added new commands BACKUP and RESTORE.
2021-08-23 23:41:45 +03:00
Anton Ivashkin
0e0e07ef46
Allow Multi Part Upload for small objects
2021-08-20 19:13:48 +03:00
Anton Ivashkin
51b543fb5c
Call AbortMultipartUpload when Multipart Upload Copy failed
2021-08-20 18:57:03 +03:00
vdimir
e77ff2740a
Add common/unit.h
2021-08-20 13:57:05 +03:00
Anton Ivashkin
b3737afe04
Use Multipart copy upload for large S3 objects
2021-08-19 13:09:58 +03:00
kssenii
709bc1a013
Continued
2021-08-16 14:50:26 +00:00
kssenii
9a02c6ad05
Catch any error if it happens on disk load when server is starting up
2021-08-16 13:18:33 +00:00
Alexey Milovidov
d184b79bba
Progress on async reads.
2021-08-16 03:00:32 +03:00
kssenii
073d7fdd5e
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-06 19:42:29 +00:00
Vladimir C
9de16c5017
Merge pull request #27216 from ianton-ru/MDB-13990
2021-08-06 18:12:00 +03:00
Anton Ivashkin
0e040ca40c
Add 10 seconds cache for S3 proxy resolver
2021-08-05 18:52:04 +03:00
Maksim Kita
4dc4854fad
Merge pull request #27168 from excitoon-favorites/safers3readbuffer
...
Safer `ReadBufferFromS3` for merges and backports
2021-08-04 15:01:21 +03:00
Vladimir Chebotarev
cc32a61488
Fix.
2021-08-04 10:40:02 +03:00
kssenii
087ddeb0dc
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into taiyang-li-disk-reload
2021-08-02 15:16:09 +00:00
kssenii
b25dfdd6af
Use method which was added for s3 instead
2021-08-02 08:15:21 +00:00
Kseniia Sumarokova
322870acd2
Update IDisk.h
2021-08-02 09:36:44 +03:00
Kseniia Sumarokova
9007ddfe22
Update DiskLocal.h
2021-08-02 09:34:39 +03:00
Kseniia Sumarokova
583c2d67cb
Update DiskSelector.cpp
2021-08-02 09:32:23 +03:00
Vitaly Baranov
a05184d83a
Fix synchronization while updating from the config of an encrypted disk.
2021-08-01 22:55:36 +03:00
kssenii
f1e96dbc55
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into taiyang-li-disk-reload
2021-07-29 07:53:16 +00:00
Vitaly Baranov
51f882d8cc
Support multiple keys for encrypted disk. Display error message if the key is probably wrong.
2021-07-24 23:45:12 +03:00
Vitaly Baranov
81ea8521b1
Set encryption algorithm explicitly.
2021-07-18 10:29:42 +03:00
Vitaly Baranov
958494b98e
Merge pull request #26377 from vitlibar/encrypted-disk-comments-and-improvements
...
Add comments for encrypted disks; some improvements in code.
2021-07-16 21:24:51 +03:00
Vitaly Baranov
9cc258327e
Add comments for encrypted disks; some improvements in code.
2021-07-16 12:04:25 +03:00
Zhichang Yu
fe5d17680e
Merge remote-tracking branch 'origin/master' into zero_copy_hdfs
2021-07-14 02:12:28 +00:00
Vitaly Baranov
bc5080d701
Rename ReadEncryptedBuffer -> ReadBufferFromEncryptedFile,
...
WriteEncryptedBuffer -> WriteBufferFromEncryptedFile,
move file FileEncryption.h to the IO folder,
rename functions accroding to our code style,
add cpp files.
2021-07-13 11:31:03 +03:00
Vitaly Baranov
042befb369
Fix compilation
2021-07-13 11:29:52 +03:00
Vitaly Baranov
e086e5e384
Merge branch 'master' into alexelex-encryption-1
2021-07-13 11:28:41 +03:00
Zhichang Yu
b4e6689bf9
fix test_hdfs_zero_copy_replication_move[tiered_copy-2]
2021-07-13 07:20:23 +00:00
Zhichang Yu
5047c758f4
fix per review
2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1
hdfs zero copy
2021-07-13 07:19:12 +00:00
Alexey Milovidov
81646f8389
Merge branch 'master' into ditch-aio
2021-07-11 03:06:45 +03:00
Anton Ivashkin
ac68d5ea71
Fix path concatenation
2021-07-08 18:39:41 +03:00
Anton Ivashkin
88bc3995e0
Fix throwing exception when iterate over non existing remote directory
2021-07-08 17:54:59 +03:00
Alexey Milovidov
21fb6ddea5
Remove AIO for sequential reads
2021-07-05 23:02:24 +03:00
Anton Ivashkin
e0199d8006
Fix DiskS3 restore
2021-06-22 20:05:20 +03:00
kssenii
babb11a887
Some fixes
2021-06-19 19:18:13 +00:00
kssenii
2031607642
Reimplement
2021-06-19 11:52:28 +00:00
kssenii
7f356a5fbc
Fix readbuffer
2021-06-18 17:06:17 +00:00
kssenii
f33387837e
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-06-18 14:27:48 +00:00
kssenii
0fa6dc4570
Reimplement
2021-06-18 11:17:37 +00:00
Vladimir
4de0d1a593
Merge pull request #24885 from Jokser/disk-s3-read-error-fix
2021-06-18 13:38:59 +03:00
Alexandra
5bc5a29c32
ContextConstPtr -> ContextPtr (resolve conflicts)
2021-06-18 05:36:50 +00:00
Latysheva Alexandra
48d3419160
Merge branch 'master' into alexelex-encryption-1
2021-06-18 11:28:08 +06:00
kssenii
cd885ee750
Fix build, ya make
2021-06-14 11:52:20 +00:00
kssenii
a1f11f5178
Fixes
2021-06-14 09:37:33 +00:00
kssenii
11faddb39f
Better
2021-06-13 22:30:37 +00:00
kssenii
2cfcc9980c
Better
2021-06-13 19:38:31 +00:00
kssenii
c80eac0dc3
First version
2021-06-13 19:36:56 +00:00
Alexey Milovidov
93b907931d
Fix the annoying ya.make.
2021-06-13 21:21:44 +03:00
alexey-milovidov
05d1af153c
Merge branch 'master' into rename-const-context-ptr
2021-06-12 03:25:09 +03:00
alexey-milovidov
99f108d704
Merge pull request #25162 from kitaisreal/rewrite-function-to-subcolumn-visitor-add-into-ya-make
...
RewriteFunctionToSubcolumnVisitor add into ya.make
2021-06-12 02:58:53 +03:00
万康
3a404fcc1b
fix build error
2021-06-11 23:21:18 +08:00
Maksim Kita
30c78c0a23
Fix arcadia
2021-06-11 14:26:33 +03:00
mergify[bot]
bfbc787b08
Merge branch 'master' into disk-s3-read-error-fix
2021-06-09 06:56:19 +00:00
Latysheva Alexandra
1a2cbe9275
Merge branch 'master' into alexelex-encryption-1
2021-06-07 21:18:54 +06:00
Ivan Lezhankin
0e9ce8c23a
Fix Disks/ya.make
2021-06-07 17:22:24 +03:00
Pavel Kovalenko
6ba40c475a
Code cleanup + comments.
2021-06-07 13:49:34 +03:00
mergify[bot]
b8dcce68b5
Merge branch 'master' into rename-const-context-ptr
2021-06-07 09:46:24 +00:00
Maksim Kita
16b3355ac1
Disks fix ya.make
2021-06-07 10:05:05 +03:00
mergify[bot]
d68d5e0ef1
Merge branch 'master' into rename-const-context-ptr
2021-06-04 14:41:41 +00:00
Pavel Kovalenko
6628b670df
Fix reading from several buffers in ReadIndirectBufferFromRemoteFS
2021-06-03 14:34:23 +03:00
Pavel Kovalenko
2bd12a7612
Fix DiskS3 read error due to broken getPosition() method in ReadBufferFromS3.cpp
2021-06-02 18:03:25 +03:00
Nikita Mikhaylov
0055a924f6
better
2021-06-01 14:23:46 +00:00
Nikita Mikhaylov
3f18bde246
Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-s3-backoff
2021-06-01 13:39:09 +00:00
Nikolai Kochetov
dbaa6ffc62
Rename ContextConstPtr to ContextPtr.
2021-06-01 15:20:52 +03:00
Nikita Mikhaylov
e03c09c51b
Update DiskS3.cpp
2021-05-31 21:08:16 +03:00
Nikita Mikhaylov
0d8ede15a9
Merge branch 'master' into merging-s3-backoff
2021-05-31 21:07:00 +03:00
kssenii
69816e6eff
Fix checks
2021-05-30 15:44:58 +03:00
kssenii
0a3589524e
File movement
2021-05-29 00:57:53 +03:00
kssenii
2a631aaf08
Final fixes
2021-05-29 00:34:44 +03:00
kssenii
f66c67a979
Fixes
2021-05-27 15:42:46 +03:00
kssenii
f9088b98de
Some disks update
2021-05-27 00:02:24 +03:00
kssenii
b28157a0df
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-27 00:01:25 +03:00
kssenii
73f16ee9ee
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-26 23:08:08 +03:00
kssenii
940eab6924
Review fixes
2021-05-26 09:26:19 +03:00
kssenii
379e26bcbc
Fix checks
2021-05-25 17:37:35 +00:00
kssenii
e01abf807e
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk
2021-05-25 17:32:43 +00:00
mergify[bot]
400b74d186
Merge branch 'master' into merging-s3-backoff
2021-05-25 14:02:38 +00:00
kssenii
75a6b21f19
Better
2021-05-25 11:29:46 +00:00
kssenii
31107816bb
fs::absolute to fs::canonical
2021-05-24 20:37:54 +03:00
Anton Ivashkin
4b9488be40
Add log record for removed from AWS S3 keys
2021-05-24 12:34:32 +03:00
kssenii
ecd82213d1
Some clean up, more tests
2021-05-23 20:19:04 +00:00
kssenii
0f98731c1f
More integration tests
2021-05-23 07:33:31 +00:00
Alexandra
2bb25f356d
createDir->createDirs in DiskEncrypted initialization
2021-05-23 06:49:02 +00:00
kssenii
6931573f69
Fix checks
2021-05-21 09:22:20 +00:00
kssenii
06ff4ab9da
Fix build
2021-05-20 13:43:08 +00:00
kssenii
2f6770c9a0
Unite writeBuffers
2021-05-20 10:46:26 +00:00
kssenii
550a4e33f9
Unite readBuffers
2021-05-20 09:57:59 +00:00
Alexandra
11e40ad000
style fixes
2021-05-19 22:06:25 +00:00
Nikita Mikhaylov
f33b70c54a
better
2021-05-20 00:42:25 +03:00
kssenii
9a5a6f39b7
Simplify writeFile
2021-05-19 20:31:43 +00:00
kssenii
5196075fd9
Better
2021-05-19 12:34:01 +00:00
Alexandra
f71e97511b
set use_ssl flag
2021-05-18 20:48:16 +00:00
Alexandra
b7a0f2db57
refactoring
2021-05-18 20:20:51 +00:00
Alexandra
e3a91f568e
fix encryption (very small files encryption bug) and replace 64-iv to 128-iv
2021-05-18 20:02:19 +00:00
kssenii
5f083b779d
A little better, fix checks
2021-05-18 14:25:19 +03:00
Alexandra
721ea4f251
remove unnecessary includes
2021-05-17 16:58:51 +00:00
Alexandra
af420569f8
remove debug information, fix seek issue and encryption issue
2021-05-17 16:15:21 +00:00
Vladimir Chebotarev
bdf03387ab
Better S3 read retries. Renamed s3_max_single_read_retries -> s3_single_read_retry_attempts, added sleeps before next attempts.
2021-05-17 10:38:32 +03:00
Alexandra
10a5b734ee
encryption initialization (without any flags like use_ssl for now)
2021-05-16 22:35:20 +00:00
kssenii
018d52d71f
Directory path
2021-05-15 09:20:26 +03:00
Alexandra
24573794ce
inherit DiskEncrypted from DiskDecorator; override copy (use IDisk::copy instead of LocalDisk::copy), applyNewSettings, onFreeze and removeShared*
2021-05-14 21:52:51 +00:00
Alexandra
730b9792bc
add parameters DiskEncrypted IDisk::applyNewSettings usage
2021-05-14 21:48:34 +00:00
kssenii
db849eb3e5
Remaining poco
2021-05-14 11:02:17 +03:00
kssenii
b123ec1354
Poco::FileException, Poco::parentPath
2021-05-14 00:20:28 +03:00
Alexandra
5374ac5c11
resolve alter table issue and add relative path to DiskEncrypted that can be declared in config as a <path>
2021-05-13 16:21:45 +00:00
Alexandra
7c34083441
mutex loop resolved
2021-05-13 06:46:11 +00:00
Alexandra
628bf9b345
disk encryption init
2021-05-13 06:37:05 +00:00
kssenii
e466924898
fs::premissions
2021-05-13 02:01:35 +03:00
kssenii
17432b1a41
Poco::moveTo to fs version
2021-05-12 17:32:22 +03:00
kssenii
047f8a4e3c
Correct merge
2021-05-12 13:10:06 +00:00
kssenii
850b6f02cb
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk
2021-05-11 14:22:30 +00:00
kssenii
393b026f2e
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-11 16:49:23 +03:00
Vladimir
d02d14fa00
Merge pull request #23846 from excitoon-favorites/s3regions
2021-05-10 12:14:08 +03:00
kssenii
ab1a05a1f4
Poco::Path to fs::path, less concatination
2021-05-09 14:59:49 +03:00
kssenii
2d4c8b394c
Return Poco::moveTo
2021-05-08 14:34:50 +03:00
kssenii
dc38da5e32
Fix fs::copy to be like Poco::copyTo
2021-05-08 14:00:31 +03:00
kssenii
35f999bf04
Poco::createFile to fs::createFile
2021-05-08 02:41:47 +03:00
kssenii
a2cfbd74ec
Fix fs::copy to work the same as Poco::copy
2021-05-07 22:20:16 +03:00
kssenii
140bf7e2ea
Less Poco
2021-05-07 15:29:26 +03:00
kssenii
5068b163b8
Fix
2021-05-05 19:55:02 +03:00
kssenii
2dabdd0f73
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-05 18:42:40 +03:00
kssenii
9ec92ec514
Fix tests, less manual concatination of paths
2021-05-05 18:39:30 +03:00
kssenii
2b67341f28
Finish s3
2021-05-04 15:55:21 +03:00
vdimir
465341eafd
Add DiskRestartProxy.cpp to ya.make
2021-05-03 20:50:17 +03:00
kssenii
ad69fceea5
Restore copyFile
2021-05-02 19:59:18 +03:00
Vladimir Chebotarev
89242d4adc
Fix.
2021-05-01 23:34:06 +03:00
Vladimir Chebotarev
a6ab040b07
Added region parameter for S3 storage and disk.
2021-05-01 20:45:35 +03:00
kssenii
32b2917141
Update s3/*
2021-05-01 18:10:04 +03:00
kssenii
ee06936596
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-01 17:24:31 +03:00
Vladimir
3b7c68196b
Merge pull request #23429 from Jokser/disk-s3-restart-possibility
2021-05-01 15:28:14 +03:00
kssenii
2887ce1f69
Temporary rollback
2021-05-01 14:23:13 +03:00
kssenii
6867ea2c73
Temporary fix
2021-04-30 19:14:04 +03:00
kssenii
c609119daf
Update Disks/*
2021-04-30 18:13:19 +03:00
kssenii
7f2f138d56
Update DiskLocal and DiskMemory
2021-04-30 11:32:34 +03:00
Maksim Kita
318c4bb80d
Add examples folder filter to ya.make.in
2021-04-30 11:25:52 +03:00
