thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-19 21:03:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
100,834 Commits 9,338 Branches 1,784 Tags 2.2 GiB
761274d857
Commit Graph

1527 Commits

Author SHA1 Message Date
alesapin
c862f89b8d Fix tidy 2022-05-26 15:43:21 +02:00
KinderRiven
824628c0da fix style 2022-05-26 16:51:16 +08:00
KinderRiven
822ecd982f better & support clean stash 2022-05-26 16:36:05 +08:00
alesapin
c7b16065e1 Merge with master 2022-05-25 21:47:05 +02:00
kssenii
b89a4edc04 Merge master 2022-05-25 20:55:04 +02:00
alesapin
6f5c86e55e Merge branch 'master' into i_object_storage 2022-05-25 20:49:01 +02:00
kssenii
0dbc09ef17 Continuation 2022-05-25 20:28:46 +02:00
alesapin
0a3597da72
Merge pull request #34915 from ianton-ru/MDB-16962 
Fix collision of S3 operation log revision
 2022-05-25 18:15:31 +02:00
kssenii
0556237b68 Continuation 2022-05-25 16:49:40 +02:00
kssenii
35d2dec8d5 Fix 2022-05-24 16:15:55 +02:00
alesapin
164f8227f9 Fix memory sanitizer 2022-05-24 12:38:11 +02:00
alesapin
1e1e6d4fa0 Review fixes 2022-05-23 16:24:06 +02:00
alesapin
2064934e59
Update src/Disks/ObjectStorages/DiskObjectStorage.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-05-23 16:02:29 +02:00
alesapin
1a78ea75bb
Update src/Disks/ObjectStorages/DiskObjectStorageMetadataHelper.h 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-05-23 16:01:51 +02:00
alesapin
1b17086266
Update src/Disks/ObjectStorages/DiskObjectStorageMetadataHelper.h 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-05-23 16:01:44 +02:00
alesapin
cef3d96cfe
Update src/Disks/ObjectStorages/S3/S3ObjectStorage.h 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-05-23 14:46:25 +02:00
alesapin
216184dfd3 Rename one more file 2022-05-23 13:20:32 +02:00
taiyang-li
14f84f02d5 Merge branch 'master' into async_hdfs_read_buffer 2022-05-23 18:36:21 +08:00
alesapin
0c61579859 More comments 2022-05-22 23:15:42 +02:00
alesapin
06c3dd69c0 Move directories 2022-05-22 13:51:48 +02:00
alesapin
eb69d963e2 Missed change 2022-05-22 12:16:50 +02:00
Robert Schulze
19d53c14fa
Merge pull request #37382 from ClickHouse/wc++98-compat-extra-semi 
Enable -Wc++98-compat-extra-semi
 2022-05-22 09:40:45 +02:00
kssenii
9aa20193d5 Merge master 2022-05-22 00:56:20 +02:00
kssenii
ffd8d9723a Continuation 2022-05-22 00:21:40 +02:00
alesapin
b3bc0a18a0 fix test 2022-05-22 00:13:01 +02:00
alesapin
5b08edefd1 Fix I don't understand 2022-05-20 20:07:15 +02:00
alesapin
d414d85654 Fix hdfs bug 2022-05-20 17:43:48 +02:00
alesapin
dc4bc2908e Fix azure 2022-05-20 15:14:01 +02:00
Robert Schulze
0f6715bd91
Follow-up to PR #37300: semicolon warnings 
In PR #37300, Alexej asked why we the compiler does not warn about
unnecessary semicolons, e.g.

  f()
  {
  }; // <-- here

The answer is surprising: In C++98, above syntax was disallowed but by
most compilers accepted it regardless. C++>11 introduced "empty
declarations" which made the syntax legal.

The previous behavior can be restored using flag
-Wc++98-compat-extra-semi. This finds many useless semicolons which were
removed in this change. Unfortunately, there are also false positives
which would require #pragma-s and HAS_* logic (--> check_flags.cmake) to
suppress. In the end, -Wc++98-compat-extra-semi comes with extra effort
for little benefit. Therefore, this change only fixes some semicolons
but does not enable the flag.
 2022-05-20 15:06:34 +02:00
alesapin
6ff221a822 Merge with master 2022-05-20 12:07:34 +02:00
alesapin
654b27e307 Merge branch 'master' into i_object_storage 2022-05-20 11:56:13 +02:00
Kseniia Sumarokova
5b179bda3d
Merge pull request #37338 from xiedeyantu/optimize-cache 
optimize file segment PARTIALLY_DOWNLOADED get read buffer
 2022-05-20 11:11:56 +02:00
Sergei Trifonov
2450ab9079
Merge pull request #36686 from azat/disk-least-used 
Implement least_used load balancing algorithm for disks inside volume
 2022-05-20 09:16:07 +02:00
Alexander Tokmakov
c2762fd8b3
Merge pull request #37319 from lingo-xp/bug-fix 
[bug-fix] root_dir is not set in copyThroughBuffers
 2022-05-19 22:17:10 +03:00
alesapin
e278bfa81e Fix fast test build 2022-05-19 21:03:49 +02:00
alesapin
92c15ec97c Get rid of IDiskRemote 2022-05-19 20:07:15 +02:00
alesapin
c87c3fcfd9 Move Azure blob storage 2022-05-19 19:25:45 +02:00
xiedeyantu
6338368a1c optimize file segment getCacheReadBuffer 2022-05-19 20:14:52 +08:00
Azat Khuzhin
a350991284 Disks: More accurate free space calculation for least_used 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-19 14:10:11 +03:00
Azat Khuzhin
757894b10f Disks: Introduce IReservation::getUnreservedSpace() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-19 14:10:11 +03:00
Azat Khuzhin
1d98913f90 Disks: Reduce number of statfs() calls for least_used disk load balancing policy 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-19 14:10:11 +03:00
Azat Khuzhin
ba26b3cf4c Implement lead_used load balancing algorithm for disks inside volume 
v2: rebase on top removed raid1
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-19 14:10:11 +03:00
Azat Khuzhin
4bc849b9f0 Disks: Remove unused src/Disks/SingleDiskVolume.cpp 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-19 14:10:11 +03:00
Azat Khuzhin
9d734bd677 Disks: Remove mentions about RAID1 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-19 14:10:11 +03:00
alesapin
41f7424ba6 Improve deletion logic 2022-05-18 17:13:17 +02:00
Alexander Tokmakov
7ab0aa7882
Update src/Disks/DiskLocal.cpp 2022-05-18 17:46:28 +03:00
Alexander Tokmakov
6098cf7cfe
Update src/Disks/DiskLocal.cpp 2022-05-18 17:46:23 +03:00
lingo-xp
46fa56c25c
[bug-fix] wrong default value in copyThroughBuffers 2022-05-18 18:02:15 +08:00
Robert Schulze
0c55ac76d2
A few clangtidy updates 
Enable:

- bugprone-lambda-function-name: "Checks for attempts to get the name of
  a function from within a lambda expression. The name of a lambda is
  always something like operator(), which is almost never what was
  intended."

- bugprone-unhandled-self-assignment: "Finds user-defined copy
  assignment operators which do not protect the code against
  self-assignment either by checking self-assignment explicitly or using
  the copy-and-swap or the copy-and-move method.""

- hicpp-invalid-access-moved: "Warns if an object is used after it has
  been moved."

- hicpp-use-noexcept: "This check replaces deprecated dynamic exception
  specifications with the appropriate noexcept specification (introduced
  in C++11)"

- hicpp-use-override: "Adds override (introduced in C++11) to overridden
  virtual functions and removes virtual from those functions as it is
  not required."

- performance-type-promotion-in-math-fn: "Finds calls to C math library
  functions (from math.h or, in C++, cmath) with implicit float to
  double promotions."

Split up:

- cppcoreguidelines-*. Some of them may be useful (haven't checked in
  detail), therefore allow to toggle them individually.

Disable:

- linuxkernel-*. Obvious.
 2022-05-17 20:56:57 +02:00
alesapin
824176f61d Merge branch 'master' into i_object_storage 2022-05-17 15:49:15 +02:00
kssenii
e4e6e44e86 Merge master 2022-05-17 15:47:33 +02:00
alesapin
f9cd820854 Fix style 2022-05-17 15:39:46 +02:00
alesapin
1c90f326c2 Remove DiskHDFS 2022-05-17 15:25:05 +02:00
alesapin
4d4634b12d
Merge pull request #37164 from kssenii/ficx-async-reads-assertions 
fix async reads from remote fs internal setting not being always turned on
 2022-05-17 11:29:14 +02:00
taiyang-li
14ab7eb5a3 merge master and solve conflict 2022-05-17 16:28:08 +08:00
kssenii
6a51dea791 Merge master 2022-05-16 22:13:22 +02:00
kssenii
5e3410f60a Better 2022-05-16 22:09:11 +02:00
alesapin
466a02ba8f Follow HDFS 2022-05-16 21:31:40 +02:00
alesapin
ce6d5f7dcc Merge branch 'master' into i_object_storage 2022-05-16 17:42:22 +02:00
kssenii
f3c1f3796b Some fixes 2022-05-16 16:16:38 +02:00
Robert Schulze
d66dcdad79
Fix new occurrences of new clang-tidy warnings 2022-05-16 11:31:36 +02:00
kssenii
f7493965b7 Remove additional check for merge tree 2022-05-16 11:19:57 +02:00
kssenii
49201f217f Revert 2022-05-16 11:17:56 +02:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies 2022-05-16 10:12:50 +02:00
alesapin
4f1d4dde22 Disable s3 logging by default 2022-05-16 00:16:00 +02:00
Sergei Trifonov
811896446f
Update IVolume.h 2022-05-15 08:57:35 +02:00
alesapin
61014dbde6 Simlify strange test 2022-05-14 18:41:54 +02:00
alesapin
0e0b535b25 Fix failing test 2022-05-14 16:54:39 +02:00
kssenii
646cf4cc95 Better 2022-05-14 14:26:04 +02:00
alesapin
47834cbabe Satisfy clang-tidy and style check 2022-05-14 14:06:09 +02:00
alesapin
4540cf8925 Style check fixes 2022-05-14 13:46:47 +02:00
alesapin
cc304a3178 Merge branch 'master' into i_object_storage 2022-05-13 23:44:00 +02:00
kssenii
47ff7b6132 Fixes after merge 2022-05-13 19:22:22 +02:00
kssenii
58faaf0ea3 Merge master 2022-05-13 18:51:07 +02:00
alesapin
b6f4a1b9f9 Missed change 2022-05-13 17:21:04 +02:00
alesapin
eba60ff38f Fix restorable schema 2022-05-13 17:00:47 +02:00
kssenii
5b9b053c61 Fix encrypted 2022-05-13 13:22:29 +02:00
Kseniia Sumarokova
6d5137879b
Merge pull request #37054 from kssenii/better-read-from-cache 
Better read from cache buffer
 2022-05-13 12:20:06 +02:00
Antonio Andelic
e4cd605edb
Merge pull request #37100 from ucasfl/fix-s3 
fix wrong argument in proxy resolver of DiskS3
 2022-05-13 10:00:13 +02:00
Kseniia Sumarokova
320efb2221
Merge branch 'master' into better-read-from-cache 2022-05-12 23:35:39 +02:00
Kseniia Sumarokova
3fd9517ff8
Merge pull request #37130 from ClickHouse/kssenii-patch-3 
Update default remote fs read method in ReadSettings
 2022-05-12 23:34:28 +02:00
alesapin
d8580c8cb8 Digging 2022-05-12 19:51:04 +02:00
Kseniia Sumarokova
6e0a910fcd
Merge branch 'master' into better-read-from-cache 2022-05-12 19:44:18 +02:00
alesapin
e7296a2b28
Merge pull request #36825 from kssenii/cache-fix-1 
Removed forceful drop cache command, fix detached status state
 2022-05-12 17:18:21 +02:00
kssenii
483e056bcc Merge master 2022-05-12 14:58:24 +02:00
kssenii
3c241582db Fix assertions 2022-05-12 14:54:18 +02:00
alesapin
ea389eeaca Fix fast test build 2022-05-12 12:44:32 +02:00
Kseniia Sumarokova
59bfcd917b
Merge pull request #36876 from kssenii/extend-cache-log 
Log into filesystem_cache_log when cache not even attempted
 2022-05-12 12:38:36 +02:00
mergify[bot]
21b4effce8
Merge branch 'master' into fix-s3 2022-05-12 06:50:23 +00:00
alesapin
5dcd25be23 Initial implementation 2022-05-12 00:04:54 +02:00
kssenii
810967fae3 Remove the check back 2022-05-11 23:01:25 +02:00
kssenii
dc1edbe22f Fix 2022-05-11 15:48:32 +02:00
kssenii
0adfee3eca Move assertion one layer higher 2022-05-11 12:33:40 +02:00
kssenii
18dd74be0c Remove unneede changes 2022-05-11 10:26:30 +02:00
fenglv
175175a16e fix wrong argument in proxy resolver of DiskS3 2022-05-11 04:16:05 +00:00
kssenii
ccab49e3cf Merge master 2022-05-10 19:33:34 +02:00
kssenii
f4b45a5805 Better profile events 2022-05-10 16:00:53 +02:00
kssenii
2f0f76369a Better 2022-05-09 21:25:26 +02:00
Anton Ivashkin
6a6505c224 Merge master 2022-05-09 17:21:34 +03:00
Anton Ivashkin
5c72da83d3 Fix typos 2022-05-09 17:19:53 +03:00
Robert Schulze
61cbcbf073
Enable clang-tidy readability-misleading-indentation 
Official docs:

  Correct indentation helps to understand code. Mismatch of the
  syntactical structure and the indentation of the code may hide serious
  problems.
 2022-05-08 19:12:01 +02:00
kssenii
0e9e1dd000 Fix assert in case cache forcefully dropped 2022-05-07 23:19:44 +02:00
Alexey Milovidov
4eac661588
Merge branch 'master' into master 2022-05-07 13:21:59 +03:00
kssenii
7e9913509f Fix build 2022-05-06 14:22:01 +02:00
kssenii
c60a3be0ce Better 2022-05-06 13:42:52 +02:00
kssenii
fd1ab053f8 Merge master 2022-05-06 12:50:33 +02:00
kssenii
23d9230acd Do not throw logical error for detached file segment 2022-05-06 12:48:16 +02:00
Kseniia Sumarokova
6d302c59e8
Merge pull request #36951 from ClickHouse/remove-strange-code 
Remove strange code
 2022-05-06 11:12:12 +02:00
Alexey Milovidov
996d838ca3
Merge pull request #36817 from DevTeamBK/Clang-tidy-Fixes 
Clang -Tidy Fixes
 2022-05-06 02:01:00 +03:00
Alexey Milovidov
490bdf5249 Fix style 2022-05-05 23:46:57 +02:00
Alexey Milovidov
e73dddb8c5 Remove strange code 2022-05-05 22:59:41 +02:00
kssenii
808ab75d74 Fix 2022-05-04 23:17:31 +02:00
Kruglov Pavel
d91b9b3049
Merge pull request #36882 from kssenii/add-current-metrics-for-filesystem-cache 
Add some CurrentMetrics for fs cache
 2022-05-04 14:44:12 +02:00
kssenii
ab65e163fa Add current metrics 2022-05-03 19:35:04 +02:00
kssenii
0266cdf125 Add entries to cache log when cache was not event attempted 2022-05-03 18:29:41 +02:00
kssenii
000f2c9f7a Better 2022-05-03 15:55:36 +02:00
Azat Khuzhin
da974585f1 Remove unimplemented RAID1 support 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-02 13:59:13 +03:00
KinderRiven
9c44ef1188 fix 2022-05-01 19:45:16 +08:00
KinderRiven
9413aa423b fix 2022-05-01 19:45:16 +08:00
KinderRiven
e0fc7870a8 fix style 2022-05-01 19:45:16 +08:00
KinderRiven
1145570c08 fix and add test for system.filesystem_cache_log 2022-05-01 19:45:16 +08:00
KinderRiven
e1acacf831 fix 2022-05-01 19:45:16 +08:00
KinderRiven
01210c8d38 fix header 2022-05-01 19:45:16 +08:00
KinderRiven
d80aa0fd17 impl system.filesystem_cache_log 2022-05-01 19:45:16 +08:00
KinderRiven
d595df1604 fix 2022-05-01 19:45:16 +08:00
KinderRiven
6a2e4559c8 impl system.cache_log 2022-05-01 19:45:16 +08:00
KinderRiven
ce9a6965d0 impl system.log_table 2022-05-01 19:45:16 +08:00
KinderRiven
9c9870f021 build cache_log table 2022-05-01 19:45:16 +08:00
KinderRiven
5a361a0db5 impl cache_log in filecache 2022-05-01 19:45:16 +08:00
Kseniia Sumarokova
66f62b2ba6
Merge pull request #36632 from DevTeamBK/Fix-Coverity-Issues 
Minor Coverity defects fixes
 2022-04-30 15:11:40 +02:00
Kseniia Sumarokova
56f37d54ab
Merge pull request #36809 from ClickHouse/kssenii-patch-3 
Update CachedReadBufferFromRemoteFS.cpp
 2022-04-30 12:57:51 +02:00
Alexey Milovidov
1ddb04b992
Merge pull request #36715 from amosbird/refactorbase 
Reorganize source files so that base won't depend on Common
 2022-04-30 09:40:58 +03:00
HeenaBansal2009
042157efca Clang -Tidy Fixes 2022-04-29 20:17:45 -07:00
Kseniia Sumarokova
7191d3d75b
Update CachedReadBufferFromRemoteFS.cpp 2022-04-29 19:13:35 +02:00
HeenaBansal2009
631b0154c0 Reverted change from ReadBufferFromRemoteFSGather.cpp 2022-04-29 07:21:35 -07:00
Kseniia Sumarokova
ca994e0861
Merge pull request #36660 from kssenii/fix-stress-test 
Fix stress test after 36639
 2022-04-29 12:56:25 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
mergify[bot]
41aa85fd4c
Merge branch 'master' into fix-stress-test 2022-04-29 00:54:09 +00:00
kssenii
84490d5198 Fix 2022-04-28 18:33:25 +02:00
HeenaBansal2009
3ac9434c57 Merge branch 'master' into Fix-Coverity-Issues 2022-04-28 08:14:50 -07:00
Sergei Trifonov
11f40376ae
Merge pull request #34571 from surahman/CH-31221-AWS-S3-object-version-id 
[CH-31221] AWS S3 Object Version Specification
 2022-04-28 15:45:33 +02:00
Kseniia Sumarokova
4c371f710e
Merge pull request #36676 from kssenii/refactor-with-size-buffer 
Better version of SeekableReadBufferWithSize
 2022-04-28 13:44:25 +02:00
kssenii
da1392877a Better 2022-04-27 20:01:59 +02:00
HeenaBansal2009
a34fca27d5 Incorporated review comments 2022-04-26 11:12:14 -07:00
HeenaBansal2009
13a9a8fd5e Style check fix 2022-04-26 07:13:09 -07:00
HeenaBansal2009
267f426605 Minor Coverity defetcs fixes 2022-04-26 07:13:09 -07:00
kssenii
0c43b7b08a Merge master 2022-04-26 16:10:47 +02:00
kssenii
9d364cdce2 Refactor 2022-04-26 15:33:53 +02:00
kssenii
a395532d54 Add support for exact right bounded reads for some other buffers 2022-04-26 12:56:40 +02:00
Sergei Trifonov
c1a0165f7d merge from master and fix conflicts 2022-04-26 12:39:16 +02:00
Sergei Trifonov
2aa17630be fix build 2022-04-26 12:30:04 +02:00
Vitaly Baranov
88b0cf3ca0 Fix TemporaryFileOnDisk 2022-04-25 16:33:38 +02:00
alesapin
2559d62d74 Better logging and bugfixes 2022-04-22 15:35:25 +02:00
mergify[bot]
cd8fa75546
Merge branch 'master' into fix_trash 2022-04-21 21:45:31 +00:00
alesapin
cbced6567d Fix build 2022-04-21 17:32:23 +02:00
kssenii
1b008b0014 Fix test 2022-04-21 14:48:07 +02:00
alesapin
5465415751 Fix replace/move partition with zero copy replication 2022-04-21 14:39:12 +02:00
taiyang-li
957f3f13f7 fix build error 2022-04-21 20:19:29 +08:00
kssenii
24e376337d Merge with master 2022-04-21 12:33:56 +02:00
taiyang-li
fb34a4ec01 ready for review 2022-04-21 17:27:41 +08:00
taiyang-li
1ea96a8732 merge master and solve conflict 2022-04-21 17:16:07 +08:00
alesapin
c14e2e0b96 Fix more 2022-04-20 21:08:26 +02:00
Sergei Trifonov
34842f209c merge master and fix conflicts 2022-04-20 18:35:56 +02:00
Kseniia Sumarokova
df8312b357
Merge pull request #36388 from kssenii/fix-s3-tests-3 
Fix test 01161_all_system_tables under s3 storage
 2022-04-20 15:46:12 +02:00
alesapin
829854113b Add a little thread safety 2022-04-20 15:10:36 +02:00
taiyang-li
d533b569ad debugging 2022-04-20 19:58:31 +08:00
alesapin
612a81ba24
Update src/Disks/IDisk.h 
Co-authored-by: tavplubix <tavplubix@clickhouse.com>
 2022-04-20 13:58:31 +02:00
alesapin
a9131de657 Merge branch 'fix_trash' of github.com:ClickHouse/ClickHouse into fix_trash 2022-04-20 13:27:03 +02:00
alesapin
2eee79dc2a Review fixes 2022-04-20 13:26:20 +02:00
alesapin
40c15222f8 Merge branch 'master' into fix_trash 2022-04-20 12:45:49 +02:00
kssenii
5fc4aaaa2d Better 2022-04-20 10:42:07 +02:00
Kseniia Sumarokova
f37812f89a
Merge pull request #36079 from kssenii/change-hashed-path 
Change hashed path for cache
 2022-04-20 10:27:44 +02:00
taiyang-li
fb6a56d4b0 finish debug 2022-04-20 16:24:18 +08:00
Kseniia Sumarokova
766a84e64f
Merge pull request #35915 from kssenii/better-s3-settings 
Better settings configuration for s3
 2022-04-20 10:22:06 +02:00
alesapin
033a3475a5
Update src/Disks/IDiskRemote.cpp 2022-04-19 17:57:12 +02:00
kssenii
2fb331bcec Better 2022-04-19 17:32:46 +02:00
kssenii
7cfc0ac433 Merge with master 2022-04-19 15:13:45 +02:00
alesapin
7c2fcee351 Simplify interface 2022-04-19 14:13:11 +02:00
alesapin
cc06bc3d99 Add some clarifications 2022-04-19 14:01:30 +02:00
Sergei Trifonov
75c1053ad7 Merge remote-tracking branch 'origin/master' into surahman-CH-31221-AWS-S3-object-version-id 2022-04-19 13:17:25 +02:00
mergify[bot]
64d5e0c8a2
Merge branch 'master' into change-hashed-path 2022-04-19 09:33:48 +00:00
mergify[bot]
d22f538b41
Merge branch 'master' into fix-s3-tests-3 2022-04-19 09:32:54 +00:00
kssenii
1f9a597b77 Merge master 2022-04-19 11:31:13 +02:00
Kseniia Sumarokova
f7a2fa487d
Fix style check 2022-04-19 10:45:48 +02:00
taiyang-li
99ad2f5fa6 Merge remote-tracking branch 'origin/master' into async_hdfs_read_buffer 2022-04-19 15:49:38 +08:00
alesapin
bd7b3847c1 Some code 2022-04-19 01:09:09 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
kssenii
40f7ddb6cc Fix 2022-04-18 15:07:07 +02:00
alesapin
c4b70f3ed3 Merge branch 'master' into fix_trash 2022-04-18 12:20:09 +02:00
Alexey Milovidov
7de4f2b6c1 Merge branch 'master' into remove-arcadia 2022-04-17 16:30:09 +02:00
taiyang-li
0ad2a76fae Merge remote-tracking branch 'origin/master' into async_hdfs_read_buffer 2022-04-16 18:45:39 +08:00
taiyang-li
cd83fd5f8a tobe debug 2022-04-16 18:41:18 +08:00
taiyang-li
f20e80c318 wip: deving 2022-04-16 16:19:48 +08:00
Alexey Milovidov
e128d89957 Remove Arcadia 2022-04-16 00:24:58 +02:00
Alexey Milovidov
416fa95b87 Remove "Arcadia" build system 2022-04-15 23:57:09 +02:00
alesapin
2fcc00380d Get rid of iheritance 2022-04-15 14:00:52 +02:00
kssenii
3e42a1711d Fix 2022-04-14 13:50:13 +02:00
Kseniia Sumarokova
04926d882a
Merge branch 'master' into change-hashed-path 2022-04-14 11:18:42 +02:00
mergify[bot]
2c5d6d132e
Merge branch 'master' into cache-fix 2022-04-13 13:32:56 +00:00
kssenii
0912b038fa Merge master 2022-04-13 15:30:01 +02:00
kssenii
7f78196b68 Update config 2022-04-13 15:11:40 +02:00
kssenii
05ceae428b Fix tests 2022-04-13 14:54:43 +02:00
Sergei Trifonov
9d1a2da685 merge master and fix conflicts 2022-04-13 14:53:36 +02:00
Kseniia Sumarokova
9ae047df1d
Update CachedReadBufferFromRemoteFS.cpp 2022-04-13 14:30:10 +02:00
kssenii
a6dc437911 Merge master 2022-04-13 14:23:48 +02:00
Kseniia Sumarokova
1b267462d8
Update CachedReadBufferFromRemoteFS.cpp 2022-04-13 11:27:47 +02:00
taiyang-li
b4cd8561a5 wip 2022-04-13 16:06:19 +08:00
kssenii
37242a0103 Allow non-evictable file segments based on predicate 2022-04-12 16:55:25 +02:00
kssenii
9804c39de7 Composable cache 2022-04-12 14:33:13 +02:00
kssenii
d2ccf876c6 Fix 2022-04-11 17:51:49 +02:00
Raúl Marín
6e346a7fc3 Support atomic replace under OSX 2022-04-11 14:32:55 +02:00
alesapin
324020a46f Restore class 2022-04-10 15:59:43 +02:00
alesapin
f61cad144c Get rid of strange abstraction 2022-04-10 15:55:05 +02:00
Kseniia Sumarokova
867b85dfad
Merge branch 'master' into change-hashed-path 2022-04-09 16:57:57 +02:00
kssenii
138588be29 Change hashed path 2022-04-08 17:14:13 +02:00
alesapin
4a03402da5
Update ReadBufferFromRemoteFSGather.h 2022-04-08 16:41:37 +02:00
alesapin
60e36abfe9 Get rid of metadata in buffers 2022-04-08 16:03:00 +02:00
Sergei Trifonov
fd001b5c80 fix param order 2022-04-08 15:51:53 +02:00
alesapin
ad9fdc1d79 Don't use metadata path in write buffers 2022-04-08 15:30:46 +02:00
alesapin
0db8808f2e Remove unused field from buffers 2022-04-08 15:06:43 +02:00
kssenii
87855c4ae7 Merge master 2022-04-08 12:32:48 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements 
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
 2022-04-08 12:06:26 +02:00
kssenii
0b7af1f26c Fix checks 2022-04-08 01:58:55 +02:00
kssenii
5dce2f18b5 Better 2022-04-07 18:48:35 +02:00
Sergei Trifonov
9c2804b924 merge from master + resolve conflicts 2022-04-07 11:22:01 +02:00
vdimir
4cb7b7e49b
Create parent directories in DiskLocal::replaceFile 2022-04-06 11:15:37 +00:00
kssenii
6c6fb5c3e8 Fix race in cached buffer 2022-04-04 16:32:37 +02:00
kssenii
4e50da722b Better s3 settings 2022-04-04 16:14:56 +02:00
kssenii
6c8e073a61 Merge master 2022-04-01 16:48:29 +02:00
alesapin
f2c6387a8d
Merge pull request #35381 from kssenii/remote-filesystem-cache-fixes 
Cache fixes for high concurrency on corner cases
 2022-04-01 10:48:31 +02:00
alesapin
e12860f646
Update src/Disks/IO/CachedReadBufferFromRemoteFS.cpp 2022-03-31 11:42:42 +02:00
alesapin
4c66c2e10f
Update src/Disks/IO/CachedReadBufferFromRemoteFS.cpp 2022-03-31 11:42:38 +02:00
kssenii
0fc92fe2aa Get rid of all "remote" mentions 2022-03-30 17:15:29 +02:00
kssenii
98ad3f4a91 Fix tests, rename some remote mentions 2022-03-30 12:21:59 +02:00
Antonio Andelic
03a09dc914
Merge pull request #35571 from ClickHouse/parallel-s3-downloading 
Parallel S3 downloading
 2022-03-30 08:14:49 +02:00
kssenii
f0e0d97752 Fix lask of query id in merges 2022-03-29 19:49:42 +02:00
kssenii
421b1e5a81 Update 2022-03-29 17:33:02 +02:00
Antonio Andelic
9990abb76a Use compile-time check for Exception messages, fix wrong messages 2022-03-29 13:16:11 +00:00
Antonio Andelic
f0b4a9fd15 Fix construction of ReadBufferFromS3 2022-03-29 10:56:49 +00:00
kssenii
eab04674fa Fix review comments 2022-03-29 12:38:17 +02:00
Azat Khuzhin
6da2c48b7a Add missing __builtin_unreachable for switch/case for gcc 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-26 23:42:21 +03:00
kssenii
141c290344 Fix build 2022-03-25 18:31:15 +01:00
kssenii
e59dbe72c7 Revert accidental commit 2022-03-25 17:53:02 +01:00
kssenii
9aaba43a0a Remove incorrect assertion 2022-03-25 16:29:30 +01:00
kssenii
0c6577eee0 Merge master 2022-03-24 20:49:02 +01:00
kssenii
6769b84638 Merge master 2022-03-24 20:47:43 +01:00
kssenii
f9b538fb0c Fix 2022-03-24 15:38:29 +01:00
kssenii
d6ab6de30a Minor changes 2022-03-24 15:32:08 +01:00
kssenii
d4161b5925 Add optin read_from_cache_if_exists_otherwise_bypass_cache (for merges) 2022-03-23 20:24:00 +01:00
kssenii
d2a3cfe5dc Cache on all write operations 2022-03-23 19:14:33 +01:00
kssenii
e39aba37a2 Minor changes 2022-03-23 18:11:52 +01:00
Sergei Trifonov
bf9c6be6c6 move threadPoolCallbackRunner from Common into Interpreters folder to fix split build issue 2022-03-23 17:57:19 +01:00
kssenii
34c0690a69 Better 2022-03-23 15:43:14 +01:00
ianton-ru
32c6390400
Merge branch 'master' into MDB-16962 2022-03-23 16:53:30 +03:00
Anton Ivashkin
66c026d0ac Fix collision of S3 operation log revision 2022-03-23 16:48:05 +03:00
kssenii
f3e1ca44a9 Add system.remote_data_paths table 2022-03-23 13:01:18 +01:00
kssenii
ccdf7d0231 Fix typos check 2022-03-21 21:51:17 +01:00
kssenii
afd0c64a1a Better 2022-03-21 15:04:16 +01:00
kssenii
eabbce69a7 Add system drop remote filesystem cache query, allow to write cache on insert 2022-03-21 12:32:44 +01:00
kssenii
59b7394caf WriteSettings 2022-03-21 09:52:48 +01:00
kssenii
1d80262a2a Add write settings, file cache settings 2022-03-21 09:50:50 +01:00
mergify[bot]
7ac606fa65
Merge branch 'master' into parallel-multipart-upload-for-s3storage 2022-03-20 18:25:28 +00:00
kssenii
43ed2beb3c Merge master 2022-03-18 12:33:08 +01:00
kssenii
af3bb3b7af Add WriteSettings 2022-03-18 12:27:38 +01:00
kssenii
44fb6fb86f Revert some changes 2022-03-18 10:16:06 +01:00
kssenii
a7c9fe5c78 Finally-finally fixed 2022-03-17 20:29:07 +01:00
kssenii
c698fb9539 Debug 2022-03-17 18:24:00 +01:00
kssenii
1b6e7fea2f Finally fixed 2022-03-17 16:40:10 +01:00
Nikolai Kochetov
ee9c2ec735
Merge pull request #34780 from azat/mt-delayed-part-flush 
Do not delay final part writing by default (fixes possible Memory limit exceeded during INSERT)
 2022-03-17 12:30:51 +01:00
kssenii
f354b704e0 Add more checks, different counting for download size 2022-03-17 11:52:59 +01:00
Sergei Trifonov
bbe657ba24 allow parallelization of multipart S3 storage upload fixes #34350 2022-03-16 19:41:41 +01:00
Nikolai Kochetov
a380aa6b8a
Merge pull request #35294 from ClickHouse/reload-remote_url_allow_hosts 
Reload remote_url_allow_hosts after config update.
 2022-03-15 22:07:16 +01:00
Nikolai Kochetov
cf260b8508
Merge pull request #35305 from ClickHouse/try-to-fix-using-deleted-memory-tracker-inside-s3-disk 
Maybe fix use-after-free inside S3 upload thread
 2022-03-15 22:06:17 +01:00
Nikolai Kochetov
b4aed421de Reset THreadPool's thread memory tracker parent to global after S3 upload task is finished. 2022-03-15 15:33:43 +00:00
Nikolai Kochetov
97aa6c82ce Reload remote_url_allow_hosts after config update. 2022-03-15 13:00:31 +00:00
Saad Ur Rahman
7bdbaf28d8
[Disks] bugfix 🤦‍♂️ 
Correcting order of parameters for <ReadBufferFromS3> in <readSchemaVersion>.
 2022-03-14 22:42:09 -04:00
Saad Ur Rahman
40fbea49f7
[Disks] bugfix 
Fixed order of parameters to <DiskS3> constructor:
 - name
 - bucket
 - key
 - versionId
 - ...
 2022-03-14 22:37:47 -04:00
Saad Ur Rahman
fea8824489
[Disks] Added Version ID parameter to registerDisksS3. 2022-03-14 22:06:09 -04:00
Saad Ur Rahman
e181207f2d
[Disks] Version ID parameter added to readFile 2022-03-14 21:30:23 -04:00
Saad Ur Rahman
98b2994352
[Disks] ReadBufferFromS3 requires a version id. 2022-03-14 21:30:23 -04:00
Saad Ur Rahman
e3a96393a1
[Disks] Version ID added. 
Version ID added to constructor.
 2022-03-14 21:30:23 -04:00
Saad Ur Rahman
f299bf46e2
[Disks] readSchemaVersion supplies empty Version ID 
Empty version ID supplied to retrieve version ID of object.
 2022-03-14 20:11:20 -04:00
mergify[bot]
ee3f4fb0c3
Merge branch 'master' into add-cache-optimization 2022-03-14 20:58:52 +00:00
Maksim Kita
538f8cbaad Fix clang-tidy warnings in Disks, Formats, Functions folders 2022-03-14 18:17:35 +00:00
kssenii
fc9c6eab9d Add optimization 2022-03-14 17:58:19 +01:00
alesapin
b149f28cd3
Merge pull request #35209 from kssenii/cache-fix 
Cache fix
 2022-03-14 11:07:40 +01:00
Kseniia Sumarokova
35e5b4e8a5
Update CachedReadBufferFromRemoteFS.cpp 2022-03-13 12:37:00 +01:00
kssenii
a18bb606a2 Update 2022-03-11 14:24:02 +01:00
kssenii
2b478e2501 Fix 2022-03-11 12:17:17 +01:00
kssenii
c86955deb2 Fix typo 2022-03-10 20:22:39 +01:00
kssenii
ddcb020d26 Fix test 2022-03-10 19:16:26 +01:00
kssenii
6571c7faa0 Clean up code 2022-03-09 10:36:52 +01:00
kssenii
96789fa5ce Fix test, disable cache for DiskCacheWrapper 2022-03-09 09:46:04 +01:00
kssenii
16e84c265b Fix build, update test 2022-03-08 21:41:58 +01:00
Azat Khuzhin
4200b56f3a Introduce IDisk::supportParallelWrite() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:17:36 +03:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0 Fix split build 2022-03-08 18:05:55 +01:00
kssenii
f3cba6f270 Remove already fixed crutch 2022-03-08 14:30:21 +01:00
kssenii
28a3aece60 Fix uncaught exception from destructor 2022-03-08 10:58:37 +01:00
Kseniia Sumarokova
517e878c6e
Merge pull request #35099 from ClickHouse/tavplubix-patch-1 
Fix inconsistency in DiskLocal
 2022-03-08 10:18:07 +01:00
kssenii
0737aef472 Fix split build 2022-03-07 22:02:07 +01:00
kssenii
eeb0ae9816 Fix detaching query 2022-03-07 17:05:36 +01:00
tavplubix
84e22fb32b
Update DiskLocal.cpp 2022-03-07 18:59:00 +03:00
kssenii
059d239815 Simplify code 2022-03-07 15:17:38 +01:00
kssenii
692b247379 Fix some corner cases 2022-03-07 14:31:21 +01:00
mergify[bot]
88052e2d7c
Merge branch 'master' into finally-enable-s3-async-writes-again 2022-03-07 12:55:52 +00:00
kssenii
2dcfe66236 Fix race 2022-03-06 20:33:07 +01:00
Kseniia Sumarokova
299e11bb2b
Update DiskCacheWrapper.cpp 2022-03-03 11:11:57 +01:00
kssenii
192b839cab Fix 2022-03-02 19:11:45 +01:00
kssenii
b2cab429a7 Move initialization out of constructor 2022-03-01 18:12:34 +01:00
kssenii
438c63f3d8 Style 2022-03-01 17:52:38 +01:00
Nikolai Kochetov
eaf6d8c5f6
Update DiskS3.cpp 2022-02-24 17:35:29 +01:00
kssenii
8d6f1982df Tiny refactoring 2022-02-23 14:43:40 +01:00
kssenii
36a76d6a75 Fix test 2022-02-22 15:18:28 +01:00
kssenii
c637385dd0 Merge master 2022-02-22 13:17:51 +01:00
Kseniia Sumarokova
eeea322556
Merge pull request #34629 from amosbird/remotefsimprove 
Some refactoring and improvement over async and remote buffer related stuff
 2022-02-22 11:36:40 +01:00
kssenii
fad7c042c1 Fix cache with non-async reads 2022-02-21 19:50:27 +01:00
kssenii
4543513a5d Add comments 2022-02-21 13:54:03 +01:00
kssenii
e8b2e47e93 Style 2022-02-19 20:38:23 +01:00
kssenii
6b60f7c57f Add limit for file segment size and split it if exceeds the limit 2022-02-19 18:59:22 +01:00
Amos Bird
84d111cde9
Better ReadBuffer asynchronicity 2022-02-20 01:08:21 +08:00
Nikolai Kochetov
40a48c5670 Enable async writes to S3. 2022-02-18 18:32:04 +00:00
kssenii
acf9a092a9 Review fixes 2022-02-18 19:09:54 +01:00
kssenii
353ffacca3 Add test 2022-02-17 19:57:04 +01:00
Amos Bird
dc0b4a8f73
Remove useless template 2022-02-17 14:31:21 +08:00
kssenii
47f94120da Merge master 2022-02-16 14:43:28 +01:00
kssenii
3bd3e51aa0 Fix tests 2022-02-16 14:08:41 +01:00
Vladimir C
724340418e
Merge pull request #34617 from Algunenano/freebsd_build 
Attempt to fix freeBSD build
 2022-02-15 20:12:44 +01:00
Raúl Marín
72efc00830 Attempt to fix freeBSD build 2022-02-15 17:23:01 +01:00
alesapin
bc2d0ee7c7
Merge pull request #34215 from ClickHouse/revert-34211-revert-34153-add_func_tests_over_s3 
Add func tests run with s3 and fix several bugs
 2022-02-15 19:07:11 +03:00
kssenii
886b300b8d Less seeks 2022-02-15 16:12:25 +01:00
mergify[bot]
34973f1d5c
Merge branch 'master' into first-test 2022-02-15 12:15:53 +00:00
alesapin
447cd56cb9 Fix comments 2022-02-15 12:11:50 +03:00
alesapin
e15396d90c Fix race condition: 2022-02-14 22:19:49 +03:00
kssenii
bdbbe85f95 Fixes 2022-02-14 19:28:48 +01:00
zxealous
3d29920005 Fix compilation issues with DiskHDFS 2022-02-14 13:14:20 +08:00
Danila Kutenin
1c589681a0
Have a correct include which was inlined 2022-02-12 12:49:23 +00:00
alesapin
088f8cacf3 Small improvements 2022-02-10 19:50:21 +03:00
Anton Popov
70986a70a1 support TTL TO [DISK|VOLUME] [IF EXISTS] 2022-02-10 19:26:23 +03:00
alesapin
3387f9e597 Possible fix DiskHDFS 2022-02-10 16:55:12 +03:00
alesapin
10c3e6e546 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-09 14:14:58 +03:00
alesapin
36909a986f Fix bug with files remove 2022-02-08 22:21:16 +03:00
alesapin
02a93cb852 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-08 19:42:27 +03:00
alesapin
f6e7cea1f8 Add settings for S3 multipart upload parts size 2022-02-08 19:38:04 +03:00
alesapin
07d5227568 Disable thread pool 2022-02-08 14:08:00 +03:00
alesapin
773d9cede3 Fix cache wrapper 2022-02-08 14:05:38 +03:00
alesapin
3af6012cb4 Revert "Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"""" 
This reverts commit 2bc2ea485e.
 2022-02-08 11:01:26 +03:00
alesapin
2bc2ea485e Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer""" 
This reverts commit fb77d7a7d5.
 2022-02-08 10:56:29 +03:00
kssenii
2e58733750 Merge master 2022-02-07 15:23:10 +01:00
Anton Popov
1b16db72c3 fix consecutive backward seeks in seekable read buffers 2022-02-07 17:20:26 +03:00
Nikolai Kochetov
40b86e3dae Add blind change. 2022-02-07 12:02:21 +00:00
alesapin
fb77d7a7d5 Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"" 
This reverts commit 875e5413ad.
 2022-02-07 12:36:54 +03:00
alesapin
25c0b07e68 Disable some tests 2022-02-04 14:31:58 +03:00
alesapin
875e5413ad Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer" 
This reverts commit b92efed350, reversing
changes made to ecce006cb2.
 2022-02-04 14:30:33 +03:00
alesapin
5e049fdd47 Merge with master 2022-02-04 12:28:19 +03:00
alesapin
2ed45b2a98 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-04 11:23:46 +03:00
alesapin
85fa727e32 Fix tidy 2022-02-04 11:21:03 +03:00
alesapin
a61b231ddd Better debug 2022-02-03 11:23:09 +03:00
alesapin
ab546f61e3 Disable cache by default 2022-02-02 22:06:08 +03:00
Nikolai Kochetov
2003a96b58 Probably fix something 2022-02-02 17:48:41 +00:00
alesapin
80800e051e Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-02 19:42:04 +03:00
Nikolai Kochetov
ea044fc6b2 Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer 2022-02-02 19:40:41 +03:00
alesapin
b9c118524f Fix race condition on hardlink/erase/read metadata 2022-02-02 19:40:21 +03:00
kssenii
aeec749173 Fix predownload case part 1 2022-02-02 17:25:25 +03:00
kssenii
5cba11428a Fixes 2022-02-02 00:25:16 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Amos Bird
e29c77f793 Fix link order issue of WriteBufferFromS3 
(cherry picked from commit f22b09f4fc)
 2022-02-01 10:42:06 +00:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3"" 2022-02-01 13:36:51 +03:00
alesapin
18f08ed932
Revert "Revert "Add func tests run with s3"" 2022-02-01 10:20:06 +03:00
alexey-milovidov
99392b5ca7
Merge pull request #13544 from amosbird/mdha 
Multi-Disk auto-recovery.
 2022-02-01 06:13:26 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3" 2022-02-01 05:49:40 +03:00
alexey-milovidov
cbfcd45be3
Revert "Add func tests run with s3" 2022-02-01 05:46:13 +03:00
alexey-milovidov
a2aa147ce0
Merge pull request #34153 from ClickHouse/add_func_tests_over_s3 
Add func tests run with s3
 2022-02-01 05:46:09 +03:00
Amos Bird
ec7d367814
DiskLocal checker 
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
 2022-02-01 05:55:27 +08:00
alesapin
cbf4790ea4 More clear error 2022-01-31 13:59:16 +03:00
kssenii
218a633fec Some fixes, more assertions and logging 2022-01-31 10:51:34 +03:00
kssenii
6b03c577d6 Make downloader's term resticted to one nextImpl 2022-01-29 23:58:53 +03:00
Nikolai Kochetov
1c9f026178 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-28 16:01:42 +03:00
kssenii
c91b86e220 Better 2022-01-26 21:56:08 +03:00
Nikolai Kochetov
a8171269a1 Review fixes. 2022-01-26 17:55:24 +00:00
Nikolai Kochetov
efaed154d2
Update src/Disks/IDisk.h 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-01-26 14:58:38 +03:00
Nikolai Kochetov
2e88319b72 Fix integration tests. 2022-01-25 18:48:28 +00:00
Nikolai Kochetov
fcc29dbd15 Try to fix integration tests. 2022-01-25 15:26:36 +00:00
kssenii
aef99dedba Fixes 2022-01-25 01:46:44 +03:00
Nikolai Kochetov
4afa75e80b Fix deduplication. 2022-01-24 14:43:36 +00:00
kssenii
898139acae Fix tests 2022-01-24 12:32:39 +03:00
kssenii
710bba895e Fix partially downloaded no continuation case 2022-01-23 23:49:13 +03:00
kssenii
cf49f41f75 Better 2022-01-23 19:51:18 +03:00
zhongyuankai
040b71617c Fix disk using the same path 2022-01-23 16:08:19 +08:00
alexey-milovidov
d1431fffd2
Merge pull request #33912 from amosbird/fixdiskcacherace 
Fix race in DiskCacheWrapper
 2022-01-23 08:47:44 +03:00
kssenii
690db142b9 Better 2022-01-23 02:12:33 +03:00
Kseniia Sumarokova
2c43e4eb57
Merge branch 'master' into local-cache-for-remote-fs 2022-01-22 22:53:55 +03:00
kssenii
36a41ac146 Better 2022-01-22 20:42:39 +03:00
Amos Bird
da66239864
Fix race in DiskCacheWrapper 2022-01-22 23:56:30 +08:00
kssenii
9f295ebf92 Fix threadpool reads 2022-01-22 01:07:20 +03:00
Maksim Kita
a3adf12a50
Merge pull request #33862 from ClickHouse/fix-s3-proxy-config-resolver-ipv6-only 
Try every resolved ip address while getting S3 proxy.
 2022-01-21 20:29:19 +01:00
Nikolai Kochetov
a024ddd1d4 Merge branch 'add-pool-to-s3-write-buffer' of github.com:ClickHouse/ClickHouse into add-pool-to-s3-write-buffer 2022-01-21 18:42:24 +00:00
Nikolai Kochetov
b3cbb63487 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:41:54 +03:00
Nikolai Kochetov
ae1b669fca Even more async writes. 2022-01-21 15:55:19 +00:00
kssenii
1858f69f5a Make space reservation incremental 2022-01-21 18:39:34 +03:00
Nikolai Kochetov
a178e54d10 Try every resolved ip address while getting S3 proxy. 2022-01-21 13:19:25 +03:00
kssenii
a566099759 Local cache for remote filesystem 2022-01-19 00:42:11 +03:00
Anton Ivashkin
7c26a832f8 Fix restore S3 disk without detached directories 2022-01-18 12:31:31 +03:00
Nikolai Kochetov
6d49a62666 Some more async writes. 2022-01-14 19:53:55 +00:00
Nikolai Kochetov
843983ea06 Add batch removing to IDiskl. 2022-01-12 15:58:04 +00:00
alesapin
733ed7cb54
Merge pull request #32061 from ianton-ru/MDB-15474 
Change ZooKeeper path for zero-copy locks for shared data
 2022-01-08 17:17:11 +03:00
xiedeyantu
1edfc03825
Update AsynchronousReadIndirectBufferFromRemoteFS.h 
update 1024 * 1024 to DBMS_DEFAULT_BUFFER_SIZE
 2022-01-07 21:06:53 +08:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474 2022-01-07 10:12:25 +00:00
Azat Khuzhin
71a99ab27c Accept real file size in createReadBufferFromFileBase() 
Right now streams relies on correct file size not the number of bytes
that will be read from the stream, to overcome one bug in the linux
kernel that may return EIINVAL for pread() with offset pass the EOF.

v2: Swap read_hint and file_size (since it is easy to miss something)

Before the first argument to readFile()/createReadBufferFromFileBase()
was read_hint, not the file_size, and let's preserve the order, since
it is easy to miss something

This will also fix 02051_read_settings test automatically because now
MergeTreeReaderStream will pass estimated_sum_mark_range_bytes to
read_hint not file_size, previously it cause on of the following errors:
- Attempt to read after EOF w/ O_DIRECT
- and LOGICAL_ERROR while adjusting granulas w/o O_DIRECT

This will also improve zero-length reads guard (via
ReadBufferFromEmptyFile), that had been added in #30190

v3: fix for other storages that wasn't enabled in fast-test
v4: ignore ENOENT/ENOTSUP in readFile
 2022-01-04 10:53:17 +03:00
alesapin
7a3c874651 Merge branch 'master' into ianton-ru-MDB-15474 2021-12-29 19:01:43 +03:00
Nikolai Kochetov
68bf38133a Add pool to WriteBufferFromS3 2021-12-29 14:04:21 +00:00
Anton Ivashkin
2d87f0a0e3 Fix debug build 2021-12-28 11:45:38 +03:00
Alexey Milovidov
f03cb4c762 Cleanup trash from Kafka and HDFS 2021-12-25 06:10:59 +03:00
Maksim Kita
60d16afde7
Merge pull request #33027 from ClickHouse/merge-33023 
Merge #33023
 2021-12-22 13:44:14 +03:00
Kseniia Sumarokova
484a4d62dd
Merge pull request #32835 from kssenii/fising-s3 
Fix optimization with lazy seek for async reads from remote fs
 2021-12-22 09:38:23 +03:00
Alexey Milovidov
7b48cb813e Merge branch 'Issue129' of github.com:DevTeamBK/ClickHouse into merge-33023 2021-12-22 02:17:00 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474 2021-12-21 17:38:36 +03:00
Anton Ivashkin
0c0bf66334 Merge master 2021-12-21 17:27:54 +03:00
kssenii
ebcb66a361 Fix 2021-12-21 10:58:02 +03:00
Nikolai Kochetov
35883e0dae
Merge pull request #32979 from ClickHouse/decrease-some-s3-log-level 
Decrease log level for some s3 messages.
 2021-12-20 20:56:22 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment 
Loops remove postfix increment
 2021-12-20 19:51:07 +03:00
Nikolai Kochetov
7dea7b7f76 Decrease log level for some s3 messages. 2021-12-20 18:18:54 +03:00
Anton Ivashkin
f0b9a4327a Use table UUID in zero-copy shared label in ZooKeeper 2021-12-20 16:19:00 +03:00
Maksim Kita
51477adf1b Updated additional cases 2021-12-20 15:55:07 +03:00
Maksim Kita
3feab5a975 Containers iteration fix erase 2021-12-20 13:42:31 +03:00
root
6b3366dc70 [Issue #129] Fix Weak Regular Expression [v21.3-insecure] 
Problem Descripton:
==================
The “key_regexp” constant in the “DiskS3” class contains a regular expression (RegEx) for checking a key path.
However, the RegEx starts and ends with “.*”.
This is a wildcard that an attacker could use to exploit this file path check by padding the beginning and end with whatever path they’d like.

Recommendation:
==============
Tighten up the RegEx by replacing the “.*” at the beginning and end with more specific items.

Fix:
===
Replace the .* at the end with $ as there is no need to continue after the key file.
Leave the beginning .* as is because the path for the key can have any characters or in any path of the system.
 2021-12-17 10:30:16 -05:00
kssenii
40c266840c Fixes 2021-12-17 18:16:29 +03:00
kssenii
ed42620530 Rename BlobStorage to AzureBlobStorage 2021-12-17 12:24:05 +03:00
kssenii
ca283a4bb8 Fix optimization with lazy seek 2021-12-17 00:18:28 +03:00
alesapin
c338bee718
Revert "Fix build issue related to azure blob storage" 2021-12-16 14:37:12 +03:00
alesapin
c0cebaea79
Merge pull request #32788 from amosbird/buildfix 
Fix build issue related to azure blob storage
 2021-12-16 13:45:22 +03:00
Amos Bird
9c9f4a3e6c
build fix 2021-12-15 16:55:18 +08:00
alesapin
9974f10631 Fix clang tidy 2021-12-14 15:34:20 +03:00
Kseniia Sumarokova
78ed383f15
Merge pull request #31505 from ContentSquare/azure_blob_storage_dependency 
Azure Blob Storage Disks
 2021-12-13 18:28:48 +03:00
Azat Khuzhin
57c027be0e Remove arcadia build support 2021-12-11 21:25:23 +03:00
Jakub Kuklis
578aec97e9 Removing TODO and NOTE marks used mainly as hooks for review in the PR 2021-12-10 22:04:38 +01:00
Jakub Kuklis
a3bc6d2fb7 Style corrections 2021-12-10 22:04:38 +01:00
Jakub Kuklis
74e26354cc Committing missing files 2021-12-10 22:04:38 +01:00
Jakub Kuklis
41a88e49c9 Abstracting metadata creation for remote disks registrations, putting Blob Storage auth in a separate file 2021-12-10 22:04:38 +01:00
Jakub Kuklis
e58f74d8dc Connecting to Azurite and invoking simple INSERTs works in the integration test 2021-12-10 22:04:38 +01:00
Jakub Kuklis
8afe69ff93 Comming back to StorageSharedKeyCredential for the integration test, get 400 Bad requests from Azurite when trying to create a container now 2021-12-10 22:04:38 +01:00
Jakub Kuklis
f180a7ba5f Solved Invalid storage account error with connection strings 2021-12-10 22:04:38 +01:00
Jakub Kuklis
008141de36 Enable authentication via StorageSharedKeyCredential 2021-12-10 22:04:38 +01:00
Jakub Kuklis
d51f979c7a Adding endpoint validation 2021-12-10 22:04:38 +01:00
Jakub Kuklis
2b7725bfc7 Enable connecting directly to storage accounts and creating new containers 2021-12-10 22:04:38 +01:00
Jakub Kuklis
910db6ca57 Code cleanup addressing PR remarks 2021-12-10 22:04:38 +01:00
Jakub Kuklis
f6c5a2c63c Removing file checks after changing blob filenames 2021-12-10 22:04:38 +01:00
Jakub Kuklis
3aa6a0ead3 Adding random postfix to blob names, needs code cleanup 2021-12-10 22:04:38 +01:00
Jakub Kuklis
5ed511f9da Working skeleton for Blob Storage integration test 2021-12-10 22:04:37 +01:00
Jakub Kuklis
0c65cfed1f Addressing first PR remarks 2021-12-10 22:04:37 +01:00
Jakub Kuklis
7c2e116214 Adding max_retries config 2021-12-10 22:04:37 +01:00
Jakub Kuklis
a33cea0525 Refactors after proper rebase on master 2021-12-10 22:04:37 +01:00
Jakub Kuklis
b3728dbbaf Rebase on master 2021-12-10 22:04:37 +01:00
Jakub Kuklis
b8ef47006c Minor refactors before PR 2021-12-10 22:04:37 +01:00
Ryad ZENINE
c4421d6866 Small changes following review 2021-12-10 22:04:37 +01:00
Jakub Kuklis
7f595eedbc Catch Blob Storage exceptions 2021-12-10 22:04:37 +01:00
Jakub Kuklis
b3c4a92cc0 Code cleanup 2021-12-10 22:04:37 +01:00
Jakub Kuklis
c4691affb2 Adding a new test for read with offset 2021-12-10 22:04:37 +01:00
Jakub Kuklis
9a00acfb72 Fixing read size error plus minor improvements 2021-12-10 22:04:37 +01:00
Jakub Kuklis
c9f3164cba Bringing back old single part upload limit 2021-12-10 22:04:37 +01:00
Jakub Kuklis
0d7a4b10c3 Using unique_ptr for Blob Client in Read Buffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
5e2b2b1eee Better search for blobs with a given name 2021-12-10 22:04:37 +01:00
Jakub Kuklis
33247db3f4 Adding finalize() to WriteBuffer, always using block blob uploads, code cleanup 2021-12-10 22:04:37 +01:00
Jakub Kuklis
bf2476a076 Shared pointer for Azure client and fixing path reference in ReadBuffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
60dd22dd4e Checking for success of file removal operation 2021-12-10 22:04:37 +01:00
Jakub Kuklis
486f267b3b Further code cleanup and error codes improvement 2021-12-10 22:04:37 +01:00
Jakub Kuklis
6755c5e897 File deletion prototype 2021-12-10 22:04:37 +01:00
Jakub Kuklis
a0cfafa387 checkUniqueId implementation using ListBlobs 2021-12-10 22:04:37 +01:00
Jakub Kuklis
95816875f4 Change false to true in supportZeroCopyReplication 2021-12-10 22:04:37 +01:00
Jakub Kuklis
b41ab197e8 Blob Storage endpoint url verification using re2 2021-12-10 22:04:37 +01:00
Jakub Kuklis
63ec8d4244 Adding multipart upload to WriteBuffer, retries in ReadBuffer, additional settings for buffers 2021-12-10 22:04:37 +01:00
Jakub Kuklis
02060791e3 Adding settings getter and applyNewSettings 2021-12-10 22:04:37 +01:00
Jakub Kuklis
3f28f2c62b Enabling local cache for DiskBlobStorage 2021-12-10 22:04:37 +01:00
Jakub Kuklis
e3daf8aa49 Increase default thread pool size 2021-12-10 22:04:37 +01:00
Jakub Kuklis
dbb98e3636 Code cleanup, leaving some debug information printing under #ifdef VERBOSE_DEBUG_MODE 2021-12-10 22:04:37 +01:00
Jakub Kuklis
8230f8039c Two column MergeTree table with Blob Storage disk work! Fixed buffer in ReadBufferFromBlobStorage to be permanent (class attribute) 2021-12-10 22:04:37 +01:00
Jakub Kuklis
2b4d3665ab Basic inserts and selects work for a MergeTree table with a Blob Disk, based on a single column table 2021-12-10 22:04:37 +01:00
Jakub Kuklis
4b51ddee1b Initial check with write and read works for the Blob Storage Disk 2021-12-10 22:04:37 +01:00
Jakub Kuklis
03f85dee41 Managed to read markers file in SELECT, getting error for CODEC version now 2021-12-10 22:04:37 +01:00
Jakub Kuklis
c43b90c58c Progressing with WriteBufferFromBlobStorage 2021-12-10 22:04:37 +01:00
Jakub Kuklis
fb4c1d6686 Starting insertion pipeline 2021-12-10 22:04:37 +01:00
Jakub Kuklis
6436f5b04d Adding an authenticated client to DiskBlobStorage 2021-12-10 22:04:37 +01:00
Jakub Kuklis
ee1a568919 The following query passes without erors: CREATE TABLE ... ENGINE = MergeTree ... storage_policy = 'blob_storage', debug constants present in the code 2021-12-10 22:04:37 +01:00
Jakub Kuklis
cacdac7ba6 Moving implementation to the .cpp file 2021-12-10 22:04:37 +01:00
Jakub Kuklis
88b5feb59d Registering the Blob Storage Disk 2021-12-10 22:04:36 +01:00
Jakub Kuklis
20dda7f666 Giving Disk Blob Storage files proper names 2021-12-10 22:04:36 +01:00
Jakub Kuklis
0cf338e165 Skeleton for DiskBlobStorage 2021-12-10 22:04:36 +01:00
Jakub Kuklis
43f420cba8 Adding a flag to use Azure Blob Storage 2021-12-10 22:04:36 +01:00
Jakub Kuklis
162e2b334b Initial working example of Azure Blob Storage manipulation from within ClickHouse 2021-12-10 22:04:36 +01:00
leosunli
3946b156a1 Fix bug 
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
 2021-12-06 14:32:32 +08:00
leosunli
6f611eec91 Fix bug 
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
 2021-12-06 14:32:32 +08:00
leosunli
e81a1dbe6a Make HDFS replication configurable in WriteBufferFromHDFSImpl#WriteBufferFromHDFSImpl 
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
 2021-12-06 14:32:31 +08:00
leosunli
5be58a8839 Make HDFS replication configurable in WriteBufferFromHDFSImpl#WriteBufferFromHDFSImpl 
Signed-off-by: leosunli <lisheng.sun08@gmail.com>
 2021-12-06 14:32:31 +08:00
sunlisheng
0f045fa20a rename replication 
Signed-off-by: sunlisheng <lisheng.sls@alibaba-inc.com>
 2021-12-06 14:32:31 +08:00
sunlisheng
ef77f45f59 Make HDFS replication configurable in WriteBufferFromHDFSImpl#WriteBufferFromHDFSImpl 
Signed-off-by: sunlisheng <lisheng.sls@alibaba-inc.com>
 2021-12-06 14:32:31 +08:00
Anton Ivashkin
0e685c1aba Fix types 2021-12-01 19:11:17 +03:00
Anton Ivashkin
e0a16a483e Merge master 2021-12-01 19:03:31 +03:00
Anton Ivashkin
0f9038ebed Zero-copy: move shared mark outside table node in ZooKeeper 2021-11-29 19:05:31 +03:00
Kseniia Sumarokova
93cf66df12
Merge pull request #30936 from kssenii/seekable-read-buffers 
Reduce memory usage for some formats when reading with s3/url/hdfs
 2021-11-25 11:19:24 +03:00
Kruglov Pavel
d9c1a0c8ec
Merge branch 'master' into fix-write-buffers 2021-11-20 17:48:24 +03:00