thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 03:42:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
100,647 Commits 9,315 Branches 1,783 Tags 2.1 GiB
891e167509
Commit Graph

2156 Commits

Author SHA1 Message Date
kssenii
0fd2c4b2c0 Continuation 2022-06-10 12:33:29 +02:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros 
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.

__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
 2022-06-10 10:22:31 +02:00
Vladimir Chebotarev
512f16f5fa Better parsing of versionId in S3::URI::URI. 2022-06-10 10:06:31 +03:00
Vladimir Chebotarev
3b344a3d26
Got rid of S3AuthSigner (#37769) 2022-06-09 14:44:50 +02:00
KinderRiven
6883ecda96 fix 2022-06-08 14:19:50 +08:00
Azat Khuzhin
71285edfbd Fix reading of empty S3 files 
It is possible for ReadBufferFromS3::nextImpl() called even after eof(),
at least once, and in this case, if the file was empty, then local
working_buffer will be null, while impl.working_buffer will be empty,
but not null, and so local position() after impl->position() =
position() will be incorrect.

I found this with test_storage_s3/test.py::test_empty_file in debug
build, assertion catched this, so maybe it worth get back debug
integration build...

v2: fix test_log_family_s3 failures
    https://s3.amazonaws.com/clickhouse-test-reports/37801/b5e6e2ddae94d6a7eac551309cb67003dff97df1/integration_tests__asan__actions__[2/3].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-07 20:41:26 +03:00
taiyang-li
73a484256e Merge branch 'master' into async_hdfs_read_buffer 2022-06-07 12:16:46 +08:00
KinderRiven
3b2a427002 fix 2022-06-07 11:50:56 +08:00
taiyang-li
b36d9f8143 refactor readinto 2022-06-06 12:58:22 +08:00
KinderRiven
13e74bef5e support skip_download_if_exceeds_query_cache 2022-06-06 02:36:23 +08:00
KinderRiven
3ffb0b3549 support_max_query_cache_size 2022-06-05 16:21:36 +08:00
kssenii
f41621736d Fixes 2022-06-03 17:03:05 +02:00
Alexander Tokmakov
084dd6f3c2
Merge pull request #37767 from excitoon-favorites/removeperrequestconfigurationfromawssdkcpp 
Moved `ClientConfigurationPerRequest` to ClickHouse
 2022-06-03 17:42:50 +03:00
zvonand
e6498a3235 update comment 2022-06-03 14:52:34 +03:00
taiyang-li
f202c35311 Merge branch 'master' into async_hdfs_read_buffer 2022-06-03 17:52:09 +08:00
kssenii
3d85391195 Better 2022-06-02 18:00:13 +02:00
kssenii
92a799a0c4 Add ObjectStorageProxy, CachedObjectStorage 2022-06-02 16:45:21 +02:00
Kruglov Pavel
37a4d11c64
Merge pull request #37645 from azat/fix-write 
Fix possible "Cannot write to finalized buffer"
 2022-06-02 14:01:30 +02:00
Antonio Andelic
46abbac352
Merge pull request #37726 from ClickHouse/fix-parallel-download-exception 
Initialize `ParallelReadBuffer` after construction
 2022-06-02 12:35:10 +02:00
Vladimir Chebotarev
a857bc2ccf
Update S3Common.cpp 2022-06-02 08:46:41 +03:00
Vladimir Chebotarev
5fcf840156
Typo. 2022-06-02 08:43:44 +03:00
Vladimir Chebotarev
d5022a0c01 Moved ClientConfigurationPerRequest from ClickHouse/aws-sdk-cpp#1 and ClickHouse/aws-sdk-cpp#2 to ClickHouse. 2022-06-02 06:07:01 +03:00
Azat Khuzhin
62d78d8f20 Fix WriteBufferFromS3 is_finalized check in case of exception 
WriteBufferFromS3::is_finalized is not set if finalizeImpl() throws,
while WriteBuffer::finalized correctly set even in case of exception, so
it should be used instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-01 21:50:30 +03:00
Antonio Andelic
08c20be4d0 Cleaner exception handling in ParallelReadBuffer 2022-06-01 11:51:01 +00:00
Antonio Andelic
f49dd19e7a Revert "Initialize ParallelReadBuffer after construction" 
This reverts commit 31e1e67836.
 2022-06-01 11:43:58 +00:00
Antonio Andelic
31e1e67836 Initialize ParallelReadBuffer after construction 2022-06-01 06:25:32 +00:00
zvonand
8e28ba2583 fix DT64 reader 2022-05-31 18:15:51 +03:00
kssenii
372709db53 Add test 2022-05-30 21:32:49 +02:00
kssenii
bfbc0e3184 Better, important fixes 2022-05-30 20:35:31 +02:00
taiyang-li
dbb8a09825 merge master and solve conflict 2022-05-30 10:47:04 +08:00
kssenii
25b1ed526d Merge master 2022-05-27 16:50:45 +02:00
kssenii
b78455168e Continuation 2022-05-27 16:25:33 +02:00
alesapin
6d6779f17a
Merge pull request #37139 from ClickHouse/i_object_storage 
Separate object storage operations from disks
 2022-05-27 13:59:50 +02:00
Alexey Milovidov
8ba865bb60
Merge pull request #37344 from excitoon-favorites/fixs3colonandequalssign 
Fixed error with symbols in key name in S3
 2022-05-27 00:58:35 +03:00
kssenii
cbb93fa0ce Continuation 2022-05-26 18:54:59 +02:00
kssenii
b89a4edc04 Merge master 2022-05-25 20:55:04 +02:00
alesapin
6f5c86e55e Merge branch 'master' into i_object_storage 2022-05-25 20:49:01 +02:00
kssenii
0dbc09ef17 Continuation 2022-05-25 20:28:46 +02:00
kssenii
0556237b68 Continuation 2022-05-25 16:49:40 +02:00
taiyang-li
1d9f65a7d4 Merge branch 'master' into async_hdfs_read_buffer 2022-05-25 17:10:22 +08:00
Alexey Milovidov
516fba27dc Merge branch 'master' into allow-setuid-inside-clickhouse 2022-05-24 23:31:14 +02:00
kssenii
35d2dec8d5 Fix 2022-05-24 16:15:55 +02:00
Kruglov Pavel
6c9a524f6b
Merge pull request #37192 from Avogar/formats-with-names 
Improve performance and memory usage for select of subset of columns for some formats
 2022-05-24 13:28:14 +02:00
李扬
e33cfc889c
Fix bug of datetime64 parsed from string '1969-12-31 23:59:59.123' (#37039) 2022-05-24 10:47:17 +02:00
mergify[bot]
51ff49a0ee
Merge branch 'master' into i_object_storage 2022-05-23 20:29:49 +00:00
alesapin
1e1e6d4fa0 Review fixes 2022-05-23 16:24:06 +02:00
alesapin
216184dfd3 Rename one more file 2022-05-23 13:20:32 +02:00
taiyang-li
14f84f02d5 Merge branch 'master' into async_hdfs_read_buffer 2022-05-23 18:36:21 +08:00
Kruglov Pavel
f539fb835d
Merge branch 'master' into formats-with-names 2022-05-23 12:14:20 +02:00
Alexey Milovidov
c541d30a38 Fix error 2022-05-23 02:48:05 +02:00
Alexey Milovidov
cc985d9b92 Fix error 2022-05-23 02:30:33 +02:00
kssenii
9aa20193d5 Merge master 2022-05-22 00:56:20 +02:00
kssenii
ffd8d9723a Continuation 2022-05-22 00:21:40 +02:00
avogar
a4cf07708c Fix comments 2022-05-20 14:57:27 +00:00
avogar
566d1b15fd Merge branch 'master' of github.com:ClickHouse/ClickHouse into formats-with-names 2022-05-20 13:54:52 +00:00
avogar
a6a430c5ee Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-columns-formats 2022-05-20 11:08:30 +00:00
alesapin
654b27e307 Merge branch 'master' into i_object_storage 2022-05-20 11:56:13 +02:00
Vladimir Chebotarev
cdbe6672cb Added one more symbol and proper comment. 2022-05-20 11:35:18 +03:00
alesapin
c87c3fcfd9 Move Azure blob storage 2022-05-19 19:25:45 +02:00
Vladimir Chebotarev
1ad19051e5 Fixed error with symbols in key name in S3. 2022-05-19 11:01:05 +03:00
Nikolai Kochetov
64ecb3941c
Merge pull request #37259 from ClickHouse/clangtidies2 
Activate more clangtidies
 2022-05-18 13:01:40 +02:00
alesapin
e99c90118e Merge branch 'master' into i_object_storage 2022-05-18 12:36:22 +02:00
Kseniia Sumarokova
38c87ab9cd
Merge pull request #36493 from kssenii/fix-encrypted-buffer 
Fix offset update ReadBufferFromEncryptedFile
 2022-05-18 09:46:17 +02:00
Robert Schulze
0c55ac76d2
A few clangtidy updates 
Enable:

- bugprone-lambda-function-name: "Checks for attempts to get the name of
  a function from within a lambda expression. The name of a lambda is
  always something like operator(), which is almost never what was
  intended."

- bugprone-unhandled-self-assignment: "Finds user-defined copy
  assignment operators which do not protect the code against
  self-assignment either by checking self-assignment explicitly or using
  the copy-and-swap or the copy-and-move method.""

- hicpp-invalid-access-moved: "Warns if an object is used after it has
  been moved."

- hicpp-use-noexcept: "This check replaces deprecated dynamic exception
  specifications with the appropriate noexcept specification (introduced
  in C++11)"

- hicpp-use-override: "Adds override (introduced in C++11) to overridden
  virtual functions and removes virtual from those functions as it is
  not required."

- performance-type-promotion-in-math-fn: "Finds calls to C math library
  functions (from math.h or, in C++, cmath) with implicit float to
  double promotions."

Split up:

- cppcoreguidelines-*. Some of them may be useful (haven't checked in
  detail), therefore allow to toggle them individually.

Disable:

- linuxkernel-*. Obvious.
 2022-05-17 20:56:57 +02:00
kssenii
e4e6e44e86 Merge master 2022-05-17 15:47:33 +02:00
kssenii
b53ca506e6 Add unit test 2022-05-17 14:55:42 +02:00
alesapin
4d4634b12d
Merge pull request #37164 from kssenii/ficx-async-reads-assertions 
fix async reads from remote fs internal setting not being always turned on
 2022-05-17 11:29:14 +02:00
taiyang-li
14ab7eb5a3 merge master and solve conflict 2022-05-17 16:28:08 +08:00
kssenii
6a51dea791 Merge master 2022-05-16 22:13:22 +02:00
alesapin
ceb7249916 Fix stupid buge 2022-05-16 19:57:48 +02:00
alesapin
ce6d5f7dcc Merge branch 'master' into i_object_storage 2022-05-16 17:42:22 +02:00
kssenii
f7493965b7 Remove additional check for merge tree 2022-05-16 11:19:57 +02:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies 2022-05-16 10:12:50 +02:00
alesapin
4f1d4dde22 Disable s3 logging by default 2022-05-16 00:16:00 +02:00
kssenii
646cf4cc95 Better 2022-05-14 14:26:04 +02:00
Kseniia Sumarokova
f4a8a0cefe
Merge pull request #37200 from kssenii/add-s3-requests-metric 
Add S3Requests metric
 2022-05-14 10:08:21 +02:00
alesapin
cc304a3178 Merge branch 'master' into i_object_storage 2022-05-13 23:44:00 +02:00
avogar
68bb07d166 Better naming 2022-05-13 18:39:19 +00:00
kssenii
58faaf0ea3 Merge master 2022-05-13 18:51:07 +02:00
kssenii
5dd3ad08a1 Add S3Requests metric 2022-05-13 18:12:46 +02:00
Kruglov Pavel
4549d2f40f
Fix build 2022-05-13 17:41:26 +02:00
kssenii
d8ec49d9b3 Fix 2022-05-13 17:17:13 +02:00
avogar
febabb2787 Fix build 2022-05-13 15:08:02 +00:00
kssenii
0870b9376c Fix 2022-05-13 16:30:59 +02:00
avogar
cef13c2c02 Allow to skip unknown columns in Native format 2022-05-13 14:27:15 +00:00
avogar
b17fec659a Improve performance and memory usage for select of subset of columns for some formats 2022-05-13 13:51:28 +00:00
Nikolai Kochetov
b2c363a38a
Merge pull request #37157 from ClickHouse/try-enable-cache-for-https-session 
Enable DNS cache for HTTPSClientSession.
 2022-05-13 15:42:30 +02:00
wuxiaobai24
f3e573bfde Merge remote-tracking branch 'upstream/master' into zstd_window_log_max 2022-05-13 20:42:54 +08:00
mergify[bot]
4a661b6e78
Merge branch 'master' into json-columns-formats 2022-05-13 11:32:03 +00:00
wuxiaobai24
ab5636a46a fix 2022-05-13 17:52:39 +08:00
Kseniia Sumarokova
320efb2221
Merge branch 'master' into better-read-from-cache 2022-05-12 23:35:39 +02:00
Kseniia Sumarokova
3fd9517ff8
Merge pull request #37130 from ClickHouse/kssenii-patch-3 
Update default remote fs read method in ReadSettings
 2022-05-12 23:34:28 +02:00
Nikolai Kochetov
f8bd4fa0f0 Remove old comment. 2022-05-12 15:24:36 +00:00
Nikolai Kochetov
77fb1b84ac Enable DNS cache for HTTPSClientSession. 2022-05-12 15:21:29 +00:00
kssenii
483e056bcc Merge master 2022-05-12 14:58:24 +02:00
Kseniia Sumarokova
59bfcd917b
Merge pull request #36876 from kssenii/extend-cache-log 
Log into filesystem_cache_log when cache not even attempted
 2022-05-12 12:38:36 +02:00
Kseniia Sumarokova
2a4f763a5e
Merge pull request #37075 from kssenii/fix-progress-for-insert-select 
Fix progress indication for `INSERT SELECT` in clickhouse-local and for file progress in client
 2022-05-12 12:05:53 +02:00
alesapin
5dcd25be23 Initial implementation 2022-05-12 00:04:54 +02:00
Kseniia Sumarokova
cf96f3ce3f
Update Progress.cpp 2022-05-11 22:52:30 +02:00
Kseniia Sumarokova
ec4955f157
Update ReadSettings.h 2022-05-11 18:32:31 +02:00
kssenii
c84e0d13fa Fix test 2022-05-11 14:36:34 +02:00
kssenii
61f2737e17 Merge master 2022-05-10 19:31:22 +02:00
wuxiaobai24
85356bbf64 fix 2022-05-11 00:53:09 +08:00
kssenii
f4b45a5805 Better profile events 2022-05-10 16:00:53 +02:00
kssenii
0feda67ec4 Fix progress for insert select 2022-05-10 15:04:08 +02:00
avogar
f6b16880bd Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-columns-formats 2022-05-10 12:57:18 +00:00
mergify[bot]
2c911da020
Merge branch 'master' into dont_finalize_s3_buffer 2022-05-10 11:26:51 +00:00
kssenii
2f0f76369a Better 2022-05-09 21:25:26 +02:00
Robert Schulze
1b81bb49b4
Enable clang-tidy modernize-deprecated-headers & hicpp-deprecated-headers 
Official docs:

  Some headers from C library were deprecated in C++ and are no longer
  welcome in C++ codebases. Some have no effect in C++. For more details
  refer to the C++ 14 Standard [depr.c.headers] section. This check
  replaces C standard library headers with their C++ alternatives and
  removes redundant ones.
 2022-05-09 08:23:33 +02:00
wuxiaobai24
58755cb156 add zstd_window_log_max setting 2022-05-08 14:02:40 +08:00
Alexey Milovidov
4eac661588
Merge branch 'master' into master 2022-05-07 13:21:59 +03:00
avogar
62a7ba3f26 Add columnar JSON formats 2022-05-06 16:48:48 +00:00
kssenii
fd1ab053f8 Merge master 2022-05-06 12:50:33 +02:00
mergify[bot]
9ce436733f
Merge branch 'master' into fix-current-size-count-in-cache 2022-05-05 17:33:56 +00:00
kssenii
0117fd40a6 Review fixes 2022-05-05 17:39:36 +02:00
Kseniia Sumarokova
224ab38b65
Merge pull request #36801 from ClickHouse/parallel-url-threads 
Use mutex per worker in ParallelReadBuffer
 2022-05-05 16:36:55 +02:00
Alexander Gololobov
7e3a805ae3
Merge pull request #36572 from ClickHouse/fix-trash-s3-metrics 
Everything related to S3 metrics was not good
 2022-05-05 11:44:02 +02:00
alesapin
b76406fc8f
Merge pull request #36910 from ClickHouse/add_test 
Fix bug in keeper which could lead to corrupted compressed logs
 2022-05-04 23:44:45 +02:00
Alexander Tokmakov
e30365a531
Merge pull request #36872 from ClickHouse/fix_exception_message 
Add extra info when sending exception
 2022-05-04 23:24:47 +03:00
Kruglov Pavel
77e55c344c
Merge pull request #36667 from Avogar/mysqldump-format 
Add MySQLDump input format
 2022-05-04 19:49:48 +02:00
alesapin
63f15368ff Remove unused header 2022-05-04 18:30:23 +02:00
alesapin
c09825919b Better comments 2022-05-04 18:24:28 +02:00
alesapin
c6556da5fc Fix bug which can lead to corrupted logs in compressed format 2022-05-04 18:08:00 +02:00
mergify[bot]
e0fa563092
Merge branch 'master' into fix_exception_message 2022-05-04 15:56:03 +00:00
mergify[bot]
6ecddf4077
Merge branch 'master' into fix-trash-s3-metrics 2022-05-04 10:14:29 +00:00
Antonio Andelic
29e0edd49f Unlock mutex before destroying worker 2022-05-04 06:30:54 +00:00
Antonio Andelic
056fac9aef Merge branch 'master' into parallel-url-threads 2022-05-04 06:26:56 +00:00
mergify[bot]
64084b5e32
Merge branch 'master' into shared_ptr_helper3 2022-05-03 20:46:16 +00:00
kssenii
0266cdf125 Add entries to cache log when cache was not event attempted 2022-05-03 18:29:41 +02:00
Dmitry Novik
5ba7a55c18
Merge pull request #36650 from bigo-sg/hive_text_parallel_parsing 
Parallel parsing of hive text format
 2022-05-03 15:56:28 +02:00
Kruglov Pavel
a902b9d1e7
Try fix builds 2022-05-03 15:56:25 +02:00
Alexander Tokmakov
94af11dd74 add extra info when sending exception 2022-05-03 15:13:47 +02:00
alesapin
991f78b485 Get rid of atomic 2022-05-03 12:17:50 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable 
... IStorage has deleted copy ctor / assignment already
 2022-05-03 09:07:08 +02:00
Antonio Andelic
fd0ba2d10b Protect background exception 2022-05-03 06:34:03 +00:00
Antonio Andelic
bfdb4566c6 Use mutex per worker 2022-05-03 06:18:39 +00:00
alesapin
98e3b16560 Terminate if S3 buffer is not finalized 2022-05-02 22:14:18 +02:00
avogar
533e4adf6c Try to fix build 2022-05-02 12:07:31 +00:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Alexey Milovidov
ef9633499c Merge branch 'master' into fix-trash-s3-metrics 2022-05-02 03:55:10 +02:00
Alexey Milovidov
1ddb04b992
Merge pull request #36715 from amosbird/refactorbase 
Reorganize source files so that base won't depend on Common
 2022-04-30 09:40:58 +03:00
Kseniia Sumarokova
ca994e0861
Merge pull request #36660 from kssenii/fix-stress-test 
Fix stress test after 36639
 2022-04-29 12:56:25 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
Kseniia Sumarokova
4102502717
Merge pull request #36619 from ClickHouse/parallel-url-performance-issue 
ParallelReadBuffer small improvements
 2022-04-29 02:55:30 +02:00
mergify[bot]
41aa85fd4c
Merge branch 'master' into fix-stress-test 2022-04-29 00:54:09 +00:00
kssenii
1cbf672237 Another fix 2022-04-28 22:51:57 +02:00
Sergei Trifonov
11f40376ae
Merge pull request #34571 from surahman/CH-31221-AWS-S3-object-version-id 
[CH-31221] AWS S3 Object Version Specification
 2022-04-28 15:45:33 +02:00
Alexander Tokmakov
21a021c210
Merge pull request #36745 from vdimir/fix-build-seekable-read-buff 
Fix build SeekableReadBufferWithSize -> SeekableReadBuffer
 2022-04-28 16:34:24 +03:00
vdimir
7c3cf30163
Fix build SeekableReadBufferWithSize -> SeekableReadBuffer 2022-04-28 15:05:04 +02:00
Anton Kozlov
ca519ca75e
Improve performance of file descriptor cache by narrowing mutex scopes (#36682) 2022-04-28 14:56:53 +02:00
Kseniia Sumarokova
4c371f710e
Merge pull request #36676 from kssenii/refactor-with-size-buffer 
Better version of SeekableReadBufferWithSize
 2022-04-28 13:44:25 +02:00
kssenii
9b38c9f7eb Fix 2022-04-28 13:06:43 +02:00
Antonio Andelic
a693898038 Merge branch 'master' into parallel-url-performance-issue 2022-04-28 07:29:07 +00:00
Vitaly Baranov
641a5f5e35
Merge pull request #36198 from vitlibar/backup-improvements-3 
Backup for replicated tables and other improvements
 2022-04-28 09:23:53 +02:00
kssenii
37d11cfb1c Merge master 2022-04-28 00:44:15 +02:00
kssenii
67e0cf9ab6 One more fix 2022-04-28 00:37:57 +02:00
Kseniia Sumarokova
39b3c1c6af
Merge pull request #36697 from KinderRiven/fix_bug_for_local_cache 
Fix deadlock for local cache
 2022-04-27 18:11:43 +02:00
KinderRiven
cd540c3c51 fix 2022-04-27 19:36:25 +08:00
KinderRiven
813f1bd001 fix 2022-04-27 19:31:50 +08:00
KinderRiven
ad4fb637f0 fix set better state 2022-04-27 19:17:46 +08:00
KinderRiven
504e2c26db fix bug for deadlock in local cache 2022-04-27 17:34:14 +08:00
Alexey Milovidov
e85c5679e9
Update S3Common.cpp 2022-04-27 03:33:01 +03:00
Alexey Milovidov
3c53cd42bb Remove excessive logging from S3 2022-04-27 02:30:13 +02:00
kssenii
0c43b7b08a Merge master 2022-04-26 16:10:47 +02:00
kssenii
9d364cdce2 Refactor 2022-04-26 15:33:53 +02:00
kssenii
a395532d54 Add support for exact right bounded reads for some other buffers 2022-04-26 12:56:40 +02:00
Sergei Trifonov
c1a0165f7d merge from master and fix conflicts 2022-04-26 12:39:16 +02:00
taiyang-li
99dee35b6e parallel parsing of hive text format 2022-04-26 14:33:10 +08:00
Vitaly Baranov
3966ee1e30 Use SeekableReadBuffer instead of ReadBuffer in IBackupEntry. 2022-04-25 16:34:33 +02:00
Antonio Andelic
4e3b79ec0c remove arena 2022-04-25 12:23:24 +00:00
Antonio Andelic
3dd38f4917 Use vector for segments 2022-04-25 12:10:41 +00:00
Antonio Andelic
ef00ac12fa Use Memory 2022-04-25 11:39:07 +00:00
taiyang-li
f040c8ae65 merge master and solve conflict 2022-04-24 11:42:36 +08:00
Alexey Milovidov
6de4f2bbff Everything related to S3 metrics was a trash 2022-04-24 03:42:18 +02:00
Kseniia Sumarokova
33bb48106f
Merge pull request #36314 from CurtizJ/print-bad-filenames 
Show names of erroneous files in case of parsing errors while executing table functions
 2022-04-22 13:24:55 +02:00
Sergei Trifonov
b371251e80 Merge branch 'master' into surahman-CH-31221-AWS-S3-object-version-id 2022-04-22 08:39:02 +02:00
Maksim Kita
57444fc7d3
Merge pull request #36444 from rschu1ze/clang-tidy-fixes 
Clang tidy fixes
 2022-04-21 16:11:27 +02:00
kssenii
24e376337d Merge with master 2022-04-21 12:33:56 +02:00
taiyang-li
1ea96a8732 merge master and solve conflict 2022-04-21 17:16:07 +08:00
Sergei Trifonov
34842f209c merge master and fix conflicts 2022-04-20 18:35:56 +02:00
taiyang-li
d533b569ad debugging 2022-04-20 19:58:31 +08:00
李扬
a1e54c3918
Fix wrong result of datetime64 when negative (#35440) 2022-04-20 13:11:31 +02:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings 
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
 2022-04-20 10:29:05 +02:00
Kseniia Sumarokova
f37812f89a
Merge pull request #36079 from kssenii/change-hashed-path 
Change hashed path for cache
 2022-04-20 10:27:44 +02:00
taiyang-li
fb6a56d4b0 finish debug 2022-04-20 16:24:18 +08:00
Kseniia Sumarokova
766a84e64f
Merge pull request #35915 from kssenii/better-s3-settings 
Better settings configuration for s3
 2022-04-20 10:22:06 +02:00
Anton Popov
bee4ca9b62 add more tests for error diagnostics in files 2022-04-19 15:56:34 +00:00
Anton Popov
3e361c9759 Merge remote-tracking branch 'upstream/master' into HEAD 2022-04-19 14:18:04 +00:00
kssenii
7cfc0ac433 Merge with master 2022-04-19 15:13:45 +02:00
Sergei Trifonov
75c1053ad7 Merge remote-tracking branch 'origin/master' into surahman-CH-31221-AWS-S3-object-version-id 2022-04-19 13:17:25 +02:00
mergify[bot]
64d5e0c8a2
Merge branch 'master' into change-hashed-path 2022-04-19 09:33:48 +00:00
kssenii
1f9a597b77 Merge master 2022-04-19 11:31:13 +02:00
taiyang-li
99ad2f5fa6 Merge remote-tracking branch 'origin/master' into async_hdfs_read_buffer 2022-04-19 15:49:38 +08:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
Alexey Milovidov
f6ab2bd523
Merge pull request #36312 from ClickHouse/remove-arcadia 
Remove remaining parts of Arcadia
 2022-04-18 07:02:54 +03:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
taiyang-li
0ad2a76fae Merge remote-tracking branch 'origin/master' into async_hdfs_read_buffer 2022-04-16 18:45:39 +08:00
taiyang-li
cd83fd5f8a tobe debug 2022-04-16 18:41:18 +08:00
Alexey Milovidov
33ffdcaa4f Return two changes 2022-04-16 02:18:22 +02:00
Anton Popov
2de6668b3f show names of erroneous files 2022-04-16 00:10:47 +00:00
Alexey Milovidov
40357637ec Remove Arcadia 2022-04-16 00:33:32 +02:00
Alexey Milovidov
cbeeb7ec4f Remove Arcadia 2022-04-16 00:20:47 +02:00
Alexey Milovidov
416fa95b87 Remove "Arcadia" build system 2022-04-15 23:57:09 +02:00
mergify[bot]
7bad91e0f4
Merge branch 'master' into fix-parallel-read-buffer 2022-04-14 09:49:50 +00:00
Kruglov Pavel
4f9ee879d4
Merge pull request #36212 from Avogar/revert-fix 
Revert reverting "Fix crash in ParallelReadBuffer"
 2022-04-14 11:26:31 +02:00
Kseniia Sumarokova
04926d882a
Merge branch 'master' into change-hashed-path 2022-04-14 11:18:42 +02:00
Kseniia Sumarokova
4d667301cc
Merge pull request #36135 from kssenii/cache-fix 
Fix after #35475
 2022-04-14 11:11:13 +02:00
avogar
ca14a29dbe Revert reverting "Fix crash in ParallelReadBuffer" 2022-04-13 20:39:12 +00:00
tavplubix
b2cd23658a
Revert "Fix crash in ParallelReadBuffer" 2022-04-13 22:45:34 +03:00
mergify[bot]
2c5d6d132e
Merge branch 'master' into cache-fix 2022-04-13 13:32:56 +00:00
kssenii
0912b038fa Merge master 2022-04-13 15:30:01 +02:00
Sergei Trifonov
9d1a2da685 merge master and fix conflicts 2022-04-13 14:53:36 +02:00
kssenii
a6dc437911 Merge master 2022-04-13 14:23:48 +02:00
Kruglov Pavel
b2905c5aed
Update src/IO/ParallelReadBuffer.cpp 
Co-authored-by: tavplubix <tavplubix@gmail.com>
 2022-04-13 13:50:11 +02:00
avogar
a4d74f975e Use atomic instead of mutex + condvar in ParallelReadBuffer 2022-04-13 10:20:29 +00:00
kssenii
37242a0103 Allow non-evictable file segments based on predicate 2022-04-12 16:55:25 +02:00
avogar
de9c9462e5 Fix crash in ParallelReadBuffer 2022-04-12 13:54:53 +00:00
kssenii
9804c39de7 Composable cache 2022-04-12 14:33:13 +02:00
kssenii
d2ccf876c6 Fix 2022-04-11 17:51:49 +02:00
Kseniia Sumarokova
867b85dfad
Merge branch 'master' into change-hashed-path 2022-04-09 16:57:57 +02:00
tavplubix
faae540a85
Merge pull request #24258 from ClickHouse/mvcc_prototype 
MVCC prototype for MergeTree
 2022-04-08 22:02:13 +03:00
Kseniia Sumarokova
10d609fd0d
Update WriteBufferFromS3.cpp 2022-04-08 17:42:05 +02:00
kssenii
138588be29 Change hashed path 2022-04-08 17:14:13 +02:00
Kruglov Pavel
d7b88d7683
Merge pull request #35582 from Avogar/improve-schema-inference 
Improve schema inference and add some fixes
 2022-04-08 13:44:52 +02:00
Alexander Tokmakov
49c35f3261 Merge branch 'master' into mvcc_prototype 2022-04-08 13:34:40 +02:00
kssenii
87855c4ae7 Merge master 2022-04-08 12:32:48 +02:00
kssenii
ef2b238444 Fix 2022-04-08 12:30:24 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements 
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
 2022-04-08 12:06:26 +02:00
kssenii
5dce2f18b5 Better 2022-04-07 18:48:35 +02:00
Sergei Trifonov
01425df758 fix build 2022-04-07 16:07:12 +02:00
Alexander Tokmakov
8290ffa88d Merge branch 'master' into mvcc_prototype 2022-04-07 13:50:42 +02:00
avogar
d2017a63b1 Merge branch 'master' of github.com:ClickHouse/ClickHouse into improve-schema-inference 2022-04-07 11:36:40 +00:00
Sergei Trifonov
03f76dda1d better 2022-04-07 12:03:54 +02:00
Sergei Trifonov
9c2804b924 merge from master + resolve conflicts 2022-04-07 11:22:01 +02:00
Yakov Olkhovskiy
c73115ffb4
Merge pull request #35840 from zzsmdfj/issue/#34966_fix_dateTime_deserialize 
to #34966_fix_dateTime_deserialize
 2022-04-06 08:10:23 -04:00
Kruglov Pavel
9141066de3
Merge branch 'master' into improve-schema-inference 2022-04-06 13:51:07 +02:00
Yakov Olkhovskiy
ea9ce3ea18 'T' is added as delimiter, tests added 2022-04-05 19:50:16 -04:00
Alexander Tokmakov
1fe50ad201 Merge branch 'master' into mvcc_prototype 2022-04-05 14:38:02 +02:00
Alexander Tokmakov
da00beaf7f Merge branch 'master' into mvcc_prototype 2022-04-05 11:14:42 +02:00
Kseniia Sumarokova
77c7545395
Merge pull request #35922 from kssenii/fix-race-in-cached-buffer 
Fix race in cached buffer
 2022-04-05 10:55:58 +02:00
Alexey Milovidov
4d6c030d23
Revert "clang-tidy report issues with Medium priority" 2022-04-04 23:41:42 +03:00
kssenii
79627798c4 Fix race 2022-04-04 20:44:39 +02:00
Kseniia Sumarokova
d697576967
Update ReadBufferFromS3.h 2022-04-04 17:53:01 +02:00
kssenii
6c6fb5c3e8 Fix race in cached buffer 2022-04-04 16:32:37 +02:00
kssenii
4e50da722b Better s3 settings 2022-04-04 16:14:56 +02:00
kssenii
36c583d0de Better version of cache on insert 2022-04-04 10:58:59 +02:00
kssenii
6c8e073a61 Merge master 2022-04-01 16:48:29 +02:00
zzsmdfj
3cae0c74d9 to #34966_fix_dateTime_deserialize 2022-04-01 20:16:07 +08:00
zzsmdfj
ececee3817 to #34966_fix_dateTime_deserialize 2022-04-01 20:13:34 +08:00
kssenii
603dcbb1b7 Extend test 2022-03-31 18:23:15 +02:00
mergify[bot]
24ade25d61
Merge branch 'master' into improve-schema-inference 2022-03-31 13:42:47 +00:00
Alexander Tokmakov
5a50ad9de3 Merge branch 'master' into mvcc_prototype 2022-03-31 11:35:04 +02:00
kssenii
0fc92fe2aa Get rid of all "remote" mentions 2022-03-30 17:15:29 +02:00
Antonio Andelic
a3ad99be03 Create CallbackRunner in Storage 2022-03-30 11:24:50 +00:00
Antonio Andelic
de2c6684a3 Remove setup and cleanup 2022-03-30 09:00:06 +00:00
Antonio Andelic
1173eab15e Use same IOThreadPool for URL and S3 2022-03-30 08:15:20 +00:00
Antonio Andelic
03a09dc914
Merge pull request #35571 from ClickHouse/parallel-s3-downloading 
Parallel S3 downloading
 2022-03-30 08:14:49 +02:00
kssenii
f0e0d97752 Fix lask of query id in merges 2022-03-29 19:49:42 +02:00
Alexander Tokmakov
287d858fda Merge branch 'master' into mvcc_prototype 2022-03-29 16:24:12 +02:00
mergify[bot]
343588de2c
Merge branch 'master' into improve-schema-inference 2022-03-29 13:06:00 +00:00
Anton Popov
d42632dc13
Merge pull request #32928 from bigo-sg/rocksdb_metacache 
Speed up parts loading process of MergeTree by metadata cache
 2022-03-29 14:05:22 +02:00
Maksim Kita
a1a4552740
Merge pull request #35184 from DevTeamBK/clang-tidy-issues 
clang-tidy report issues with Medium priority
 2022-03-29 13:19:54 +02:00
taiyang-li
834ac3a1b9 Merge remote-tracking branch 'ck/master' into rocksdb_metacache 2022-03-29 18:32:06 +08:00
taiyang-li
38702be40c Merge remote-tracking branch 'ck/master' into rocksdb_metacache 2022-03-28 21:03:22 +08:00
Antonio Andelic
d9d826c813 Address PR review 2022-03-28 08:19:23 +00:00
Azat Khuzhin
6a3986f41b Fix redundant-move in ReadWriteBufferFromHTTP.h 2022-03-26 23:42:46 +03:00
kssenii
d6ab6de30a Minor changes 2022-03-24 15:32:08 +01:00
avogar
557edbd172 Add some improvements and fixes in schema inference 2022-03-24 12:54:12 +00:00
Antonio Andelic
62a495a5fc Small refactor 2022-03-24 09:45:30 +00:00
kssenii
d4161b5925 Add optin read_from_cache_if_exists_otherwise_bypass_cache (for merges) 2022-03-23 20:24:00 +01:00
kssenii
d2a3cfe5dc Cache on all write operations 2022-03-23 19:14:33 +01:00
kssenii
c2c7f365ce Merge master 2022-03-23 16:58:35 +01:00
kssenii
34c0690a69 Better 2022-03-23 15:43:14 +01:00
Alexander Tokmakov
bcec53aadc Merge branch 'master' into mvcc_prototype 2022-03-23 11:28:43 +01:00
Antonio Andelic
131b3a091c Refactor StorageS3 2022-03-23 08:57:01 +00:00
Antonio Andelic
f32ef2a556 Small polishing for S3 reader 2022-03-23 08:15:18 +00:00
taiyang-li
8dbf1c60e7 merge master and fix conflict 2022-03-23 11:36:50 +08:00
Antonio Andelic
6785ad165a Fix issue for mutliple download threads 2022-03-22 14:06:30 +00:00
Antonio Andelic
df99a9cc3c Merge branch 'parallel-downloading-url-engine' into parallel-s3-downloading 2022-03-22 12:49:35 +00:00
Antonio Andelic
5da3058368 Merge branch 'master' into parallel-downloading-url-engine 2022-03-22 12:20:36 +00:00
Alexander Tokmakov
3c762f566d Merge branch 'master' into mvcc_prototype 2022-03-21 20:16:29 +01:00
Antonio Andelic
04e6e6ab5e Add ParallelReadBuffer for S3 2022-03-21 14:52:26 +00:00
kssenii
afd0c64a1a Better 2022-03-21 15:04:16 +01:00
kssenii
eabbce69a7 Add system drop remote filesystem cache query, allow to write cache on insert 2022-03-21 12:32:44 +01:00
Vitaly Baranov
7a46e7a0d1
Merge branch 'master' into improve-backup-and-restore 2022-03-21 11:49:26 +01:00
Vitaly Baranov
51fc556fcd Implement BACKUP and RESTORE for MatViews. 2022-03-21 11:45:21 +01:00
Maksim Kita
695b1300f9
Merge pull request #35383 from rschu1ze/enable-if-to-concepts-pt2 
Replace more uses of std::enable_if for SFINAE by concepts
 2022-03-21 10:47:28 +01:00
Antonio Andelic
b8c43ff2f2 Merge branch 'master' into parallel-downloading-url-engine 2022-03-21 09:11:01 +00:00
kssenii
59b7394caf WriteSettings 2022-03-21 09:52:48 +01:00
kssenii
1d80262a2a Add write settings, file cache settings 2022-03-21 09:50:50 +01:00
Alexander Tokmakov
9e05b12d2c Merge branch 'master' into mvcc_prototype 2022-03-20 22:42:26 +01:00
Vitaly Baranov
c884cf901f Use BackupEntryFromAppendOnlyFile for Log engine, 
and use a BackupEntriesBatch for Memort engine to improve performance.
A lot of minor corrections.
 2022-03-20 20:02:15 +01:00
Vitaly Baranov
a1b364626a Improve zip archive utilities. 2022-03-20 20:02:15 +01:00
alesapin
5902304fdb Remove some useless code from S3Common 2022-03-19 23:28:20 +01:00
kssenii
af3bb3b7af Add WriteSettings 2022-03-18 12:27:38 +01:00
Antonio Andelic
f2a7c6c6fe Add support for canceling a single worker 2022-03-18 09:28:56 +00:00
Robert Schulze
df73c75456
Replace more uses of std::enable_if for SFINAE by concepts 
- enable_if is usually regarded as fragile and unreadable
- C++20 concepts are much easier to read and produce more expressive error messages

- this is follow-up to PR #35347 but this time most of the remaining and
  more complex usages of enable_if in the codebase were replaced.
 2022-03-17 20:44:55 +01:00
Antonio Andelic
b789a41662 Fix ASAN problems 2022-03-17 15:18:20 +00:00
Alexander Tokmakov
d04dc03fa4 Merge branch 'master' into mvcc_prototype 2022-03-17 15:24:32 +01:00
Antonio Andelic
103a3fa140 Merge branch 'master' into parallel-downloading-url-engine 2022-03-17 10:11:58 +00:00
Antonio Andelic
44de5a581c Fix split build 2022-03-17 10:09:16 +00:00
alesapin
457fa0d1ba
Merge pull request #23932 from CurtizJ/dynamic-columns 
Support of dynamic subcolumns (JSON data type)
 2022-03-17 10:45:46 +01:00
Maksim Kita
4d8c2b2009
Merge pull request #35347 from rschu1ze/enable-if-to-concepts 
Replace a few uses of enable_if for SFINAE by concepts
 2022-03-17 10:34:08 +01:00
Alexander Tokmakov
4b3e13a4fe Merge branch 'master' into mvcc_prototype 2022-03-16 21:06:19 +01:00
Alexander Tokmakov
c2ac8d4a5c review fixes 2022-03-16 21:05:34 +01:00
Robert Schulze
b56c28d841
Replace a few uses of enable_if for SFINAE by concepts 
- enable_if is usually regarded as fragile and unreadable

- C++20 concepts are much easier to read and produce more expressive
  error messages
 2022-03-16 19:51:38 +01:00
Antonio Andelic
0979155f2f Address PR comments 2022-03-16 15:45:17 +00:00
Anton Popov
0ba78c3c3a Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-16 15:28:09 +00:00
Antonio Andelic
84f86bf3da Limit number of bytes read per worker 2022-03-16 12:34:14 +00:00
Antonio Andelic
2291b7c1e1 Merge branch 'master' into parallel-downloading-url-engine 2022-03-16 09:59:52 +00:00
Antonio Andelic
3465664871 Attach thread pool threads to counters 2022-03-16 09:26:01 +00:00
Kruglov Pavel
2b7e89e171
Merge pull request #35300 from Avogar/fix-bzip2 
Fix possible assertion while using bzip2 compression
 2022-03-15 22:17:37 +01:00
Kruglov Pavel
20e17a6d3e
Merge pull request #35296 from Avogar/fix-lz4 
Fix possible segfault while using lz4 compression
 2022-03-15 22:17:14 +01:00
Kruglov Pavel
6fd1a8281f
Merge pull request #35295 from Avogar/fix-lzma 
Fix possible Assertion failed while using lzma compression
 2022-03-15 22:14:35 +01:00
Nikolai Kochetov
a380aa6b8a
Merge pull request #35294 from ClickHouse/reload-remote_url_allow_hosts 
Reload remote_url_allow_hosts after config update.
 2022-03-15 22:07:16 +01:00
Saad Ur Rahman
275bf0ef81
[IO] fixed bug in versionId extraction introduced during rebase. 2022-03-15 12:13:44 -04:00
avogar
ec3c5f86ad Fix possible assertion while using bzip2 compression 2022-03-15 14:23:52 +00:00
avogar
0cfaf48e1d Fix possible segfault while using lz4 compression 2022-03-15 13:46:14 +00:00
Alexander Tokmakov
1f571b7734 Merge branch 'master' into mvcc_prototype 2022-03-15 14:45:06 +01:00
Kruglov Pavel
24ac217be3
Remove redundant include 2022-03-15 14:07:31 +01:00
avogar
90ce1709d7 Fix possible Assertion failed while using lzma compression 2022-03-15 13:00:53 +00:00
Nikolai Kochetov
97aa6c82ce Reload remote_url_allow_hosts after config update. 2022-03-15 13:00:31 +00:00
mergify[bot]
d613ad9445
Merge branch 'master' into fix-brotli 2022-03-15 08:25:12 +00:00
Antonio Andelic
067b79b00b Merge branch 'master' into parallel-downloading-url-engine 2022-03-15 07:55:41 +00:00
Saad Ur Rahman
7b35c79b04
[IO] versionId extraction simplified. 
Reducing cognitive load.
 2022-03-14 22:27:58 -04:00
Saad Ur Rahman
d9b370cf1d
[IO] S3 URI versionId extraction performance tweak. 2022-03-14 22:17:50 -04:00
Saad Ur Rahman
1a85e9b60d
[IO] Extracting Version ID in S3::URI 
Extracting Version ID form query string in URI.
 2022-03-14 22:16:24 -04:00
Saad Ur Rahman
ec28af055f
[IO] S3 URI data struct contains Version ID 
Added Version ID field to S3::URI struct.
TODO: Extract version id if present.
 2022-03-14 22:11:18 -04:00
Saad Ur Rahman
c7402c2a39
[IO] Version ID provided during buffer read 
The Version ID, if provided, is to be set in the request to read the buffer from S3. If not provided the latest version of the object must be retrieved.

Version ID set to last parameter to allow defaulting and backward compatibility with calls.
 2022-03-14 20:10:14 -04:00
Alexander Tokmakov
9702b5177d Merge branch 'master' into mvcc_prototype 2022-03-14 21:45:38 +01:00
avogar
4b12c6e6b3 Fix possible Abort while using Brotli compression 2022-03-14 18:58:05 +00:00
Maksim Kita
e30117a3d6 Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Antonio Andelic
5f8d9885b3 Ignore read bytes 2022-03-14 15:50:30 +00:00
Azat Khuzhin
9f7fae56b0 WriteBufferFromS3: remove unused headers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-14 12:07:47 +03:00
Antonio Andelic
f5d3a8a31d Polishing 2022-03-11 13:38:19 +00:00
Antonio Andelic
28e9508c4e Improve shared pool and add settings 2022-03-11 08:33:34 +00:00
Antonio Andelic
cce318273b Use a shared IO thread pool 2022-03-11 08:33:34 +00:00
Antonio Andelic
adaaaaf061 Unblock readers before stoping 2022-03-11 08:33:34 +00:00
Antonio Andelic
df0f5e20d0 Refactor 2022-03-11 08:33:34 +00:00
Antonio Andelic
7b526b25a8 Initial implementation with Arena 2022-03-11 08:33:34 +00:00
Antonio Andelic
16c20332cf Fix deadlocks 2022-03-11 08:33:34 +00:00
Antonio Andelic
c6664bbbdf Add seekable ParallelReadBuffer 2022-03-11 08:33:34 +00:00
Antonio Andelic
29c32ed831 Refactor code 2022-03-11 08:33:34 +00:00
Antonio Andelic
0a1a3a230e Add support for parallel http range requests 2022-03-11 08:33:34 +00:00
Antonio Andelic
aad87879cf Small refactor 2022-03-11 08:33:34 +00:00
Kseniia Sumarokova
818459b9f0
Merge pull request #33717 from kssenii/local-cache-for-remote-fs 
Local cache for remote filesystem
 2022-03-11 07:23:10 +01:00
Rajkumar
137e4b79c1 override needed 2022-03-10 12:46:32 -08:00
Kseniia Sumarokova
5149bd2d29
Merge pull request #35172 from kssenii/add-non-retriable-errors-for-url 
Add non-retriable errors check for HTTP read buffer
 2022-03-10 20:28:37 +01:00
Rajkumar
0de950c3f8 style correction 2022-03-10 10:29:57 -08:00