thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 11:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
100,647 Commits 9,326 Branches 1,784 Tags 2.1 GiB
891e167509
Commit Graph

2156 Commits

Author SHA1 Message Date
kssenii
c84e0d13fa Fix test 2022-05-11 14:36:34 +02:00
kssenii
61f2737e17 Merge master 2022-05-10 19:31:22 +02:00
wuxiaobai24
85356bbf64 fix 2022-05-11 00:53:09 +08:00
kssenii
f4b45a5805 Better profile events 2022-05-10 16:00:53 +02:00
kssenii
0feda67ec4 Fix progress for insert select 2022-05-10 15:04:08 +02:00
avogar
f6b16880bd Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-columns-formats 2022-05-10 12:57:18 +00:00
mergify[bot]
2c911da020
Merge branch 'master' into dont_finalize_s3_buffer 2022-05-10 11:26:51 +00:00
kssenii
2f0f76369a Better 2022-05-09 21:25:26 +02:00
Robert Schulze
1b81bb49b4
Enable clang-tidy modernize-deprecated-headers & hicpp-deprecated-headers 
Official docs:

  Some headers from C library were deprecated in C++ and are no longer
  welcome in C++ codebases. Some have no effect in C++. For more details
  refer to the C++ 14 Standard [depr.c.headers] section. This check
  replaces C standard library headers with their C++ alternatives and
  removes redundant ones.
 2022-05-09 08:23:33 +02:00
wuxiaobai24
58755cb156 add zstd_window_log_max setting 2022-05-08 14:02:40 +08:00
Alexey Milovidov
4eac661588
Merge branch 'master' into master 2022-05-07 13:21:59 +03:00
avogar
62a7ba3f26 Add columnar JSON formats 2022-05-06 16:48:48 +00:00
kssenii
fd1ab053f8 Merge master 2022-05-06 12:50:33 +02:00
mergify[bot]
9ce436733f
Merge branch 'master' into fix-current-size-count-in-cache 2022-05-05 17:33:56 +00:00
kssenii
0117fd40a6 Review fixes 2022-05-05 17:39:36 +02:00
Kseniia Sumarokova
224ab38b65
Merge pull request #36801 from ClickHouse/parallel-url-threads 
Use mutex per worker in ParallelReadBuffer
 2022-05-05 16:36:55 +02:00
Alexander Gololobov
7e3a805ae3
Merge pull request #36572 from ClickHouse/fix-trash-s3-metrics 
Everything related to S3 metrics was not good
 2022-05-05 11:44:02 +02:00
alesapin
b76406fc8f
Merge pull request #36910 from ClickHouse/add_test 
Fix bug in keeper which could lead to corrupted compressed logs
 2022-05-04 23:44:45 +02:00
Alexander Tokmakov
e30365a531
Merge pull request #36872 from ClickHouse/fix_exception_message 
Add extra info when sending exception
 2022-05-04 23:24:47 +03:00
Kruglov Pavel
77e55c344c
Merge pull request #36667 from Avogar/mysqldump-format 
Add MySQLDump input format
 2022-05-04 19:49:48 +02:00
alesapin
63f15368ff Remove unused header 2022-05-04 18:30:23 +02:00
alesapin
c09825919b Better comments 2022-05-04 18:24:28 +02:00
alesapin
c6556da5fc Fix bug which can lead to corrupted logs in compressed format 2022-05-04 18:08:00 +02:00
mergify[bot]
e0fa563092
Merge branch 'master' into fix_exception_message 2022-05-04 15:56:03 +00:00
mergify[bot]
6ecddf4077
Merge branch 'master' into fix-trash-s3-metrics 2022-05-04 10:14:29 +00:00
Antonio Andelic
29e0edd49f Unlock mutex before destroying worker 2022-05-04 06:30:54 +00:00
Antonio Andelic
056fac9aef Merge branch 'master' into parallel-url-threads 2022-05-04 06:26:56 +00:00
mergify[bot]
64084b5e32
Merge branch 'master' into shared_ptr_helper3 2022-05-03 20:46:16 +00:00
kssenii
0266cdf125 Add entries to cache log when cache was not event attempted 2022-05-03 18:29:41 +02:00
Dmitry Novik
5ba7a55c18
Merge pull request #36650 from bigo-sg/hive_text_parallel_parsing 
Parallel parsing of hive text format
 2022-05-03 15:56:28 +02:00
Kruglov Pavel
a902b9d1e7
Try fix builds 2022-05-03 15:56:25 +02:00
Alexander Tokmakov
94af11dd74 add extra info when sending exception 2022-05-03 15:13:47 +02:00
alesapin
991f78b485 Get rid of atomic 2022-05-03 12:17:50 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable 
... IStorage has deleted copy ctor / assignment already
 2022-05-03 09:07:08 +02:00
Antonio Andelic
fd0ba2d10b Protect background exception 2022-05-03 06:34:03 +00:00
Antonio Andelic
bfdb4566c6 Use mutex per worker 2022-05-03 06:18:39 +00:00
alesapin
98e3b16560 Terminate if S3 buffer is not finalized 2022-05-02 22:14:18 +02:00
avogar
533e4adf6c Try to fix build 2022-05-02 12:07:31 +00:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Alexey Milovidov
ef9633499c Merge branch 'master' into fix-trash-s3-metrics 2022-05-02 03:55:10 +02:00
Alexey Milovidov
1ddb04b992
Merge pull request #36715 from amosbird/refactorbase 
Reorganize source files so that base won't depend on Common
 2022-04-30 09:40:58 +03:00
Kseniia Sumarokova
ca994e0861
Merge pull request #36660 from kssenii/fix-stress-test 
Fix stress test after 36639
 2022-04-29 12:56:25 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
Kseniia Sumarokova
4102502717
Merge pull request #36619 from ClickHouse/parallel-url-performance-issue 
ParallelReadBuffer small improvements
 2022-04-29 02:55:30 +02:00
mergify[bot]
41aa85fd4c
Merge branch 'master' into fix-stress-test 2022-04-29 00:54:09 +00:00
kssenii
1cbf672237 Another fix 2022-04-28 22:51:57 +02:00
Sergei Trifonov
11f40376ae
Merge pull request #34571 from surahman/CH-31221-AWS-S3-object-version-id 
[CH-31221] AWS S3 Object Version Specification
 2022-04-28 15:45:33 +02:00
Alexander Tokmakov
21a021c210
Merge pull request #36745 from vdimir/fix-build-seekable-read-buff 
Fix build SeekableReadBufferWithSize -> SeekableReadBuffer
 2022-04-28 16:34:24 +03:00
vdimir
7c3cf30163
Fix build SeekableReadBufferWithSize -> SeekableReadBuffer 2022-04-28 15:05:04 +02:00
Anton Kozlov
ca519ca75e
Improve performance of file descriptor cache by narrowing mutex scopes (#36682) 2022-04-28 14:56:53 +02:00
Kseniia Sumarokova
4c371f710e
Merge pull request #36676 from kssenii/refactor-with-size-buffer 
Better version of SeekableReadBufferWithSize
 2022-04-28 13:44:25 +02:00
kssenii
9b38c9f7eb Fix 2022-04-28 13:06:43 +02:00
Antonio Andelic
a693898038 Merge branch 'master' into parallel-url-performance-issue 2022-04-28 07:29:07 +00:00
Vitaly Baranov
641a5f5e35
Merge pull request #36198 from vitlibar/backup-improvements-3 
Backup for replicated tables and other improvements
 2022-04-28 09:23:53 +02:00
kssenii
37d11cfb1c Merge master 2022-04-28 00:44:15 +02:00
kssenii
67e0cf9ab6 One more fix 2022-04-28 00:37:57 +02:00
Kseniia Sumarokova
39b3c1c6af
Merge pull request #36697 from KinderRiven/fix_bug_for_local_cache 
Fix deadlock for local cache
 2022-04-27 18:11:43 +02:00
KinderRiven
cd540c3c51 fix 2022-04-27 19:36:25 +08:00
KinderRiven
813f1bd001 fix 2022-04-27 19:31:50 +08:00
KinderRiven
ad4fb637f0 fix set better state 2022-04-27 19:17:46 +08:00
KinderRiven
504e2c26db fix bug for deadlock in local cache 2022-04-27 17:34:14 +08:00
Alexey Milovidov
e85c5679e9
Update S3Common.cpp 2022-04-27 03:33:01 +03:00
Alexey Milovidov
3c53cd42bb Remove excessive logging from S3 2022-04-27 02:30:13 +02:00
kssenii
0c43b7b08a Merge master 2022-04-26 16:10:47 +02:00
kssenii
9d364cdce2 Refactor 2022-04-26 15:33:53 +02:00
kssenii
a395532d54 Add support for exact right bounded reads for some other buffers 2022-04-26 12:56:40 +02:00
Sergei Trifonov
c1a0165f7d merge from master and fix conflicts 2022-04-26 12:39:16 +02:00
taiyang-li
99dee35b6e parallel parsing of hive text format 2022-04-26 14:33:10 +08:00
Vitaly Baranov
3966ee1e30 Use SeekableReadBuffer instead of ReadBuffer in IBackupEntry. 2022-04-25 16:34:33 +02:00
Antonio Andelic
4e3b79ec0c remove arena 2022-04-25 12:23:24 +00:00
Antonio Andelic
3dd38f4917 Use vector for segments 2022-04-25 12:10:41 +00:00
Antonio Andelic
ef00ac12fa Use Memory 2022-04-25 11:39:07 +00:00
taiyang-li
f040c8ae65 merge master and solve conflict 2022-04-24 11:42:36 +08:00
Alexey Milovidov
6de4f2bbff Everything related to S3 metrics was a trash 2022-04-24 03:42:18 +02:00
Kseniia Sumarokova
33bb48106f
Merge pull request #36314 from CurtizJ/print-bad-filenames 
Show names of erroneous files in case of parsing errors while executing table functions
 2022-04-22 13:24:55 +02:00
Sergei Trifonov
b371251e80 Merge branch 'master' into surahman-CH-31221-AWS-S3-object-version-id 2022-04-22 08:39:02 +02:00
Maksim Kita
57444fc7d3
Merge pull request #36444 from rschu1ze/clang-tidy-fixes 
Clang tidy fixes
 2022-04-21 16:11:27 +02:00
kssenii
24e376337d Merge with master 2022-04-21 12:33:56 +02:00
taiyang-li
1ea96a8732 merge master and solve conflict 2022-04-21 17:16:07 +08:00
Sergei Trifonov
34842f209c merge master and fix conflicts 2022-04-20 18:35:56 +02:00
taiyang-li
d533b569ad debugging 2022-04-20 19:58:31 +08:00
李扬
a1e54c3918
Fix wrong result of datetime64 when negative (#35440) 2022-04-20 13:11:31 +02:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings 
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
 2022-04-20 10:29:05 +02:00
Kseniia Sumarokova
f37812f89a
Merge pull request #36079 from kssenii/change-hashed-path 
Change hashed path for cache
 2022-04-20 10:27:44 +02:00
taiyang-li
fb6a56d4b0 finish debug 2022-04-20 16:24:18 +08:00
Kseniia Sumarokova
766a84e64f
Merge pull request #35915 from kssenii/better-s3-settings 
Better settings configuration for s3
 2022-04-20 10:22:06 +02:00
Anton Popov
bee4ca9b62 add more tests for error diagnostics in files 2022-04-19 15:56:34 +00:00
Anton Popov
3e361c9759 Merge remote-tracking branch 'upstream/master' into HEAD 2022-04-19 14:18:04 +00:00
kssenii
7cfc0ac433 Merge with master 2022-04-19 15:13:45 +02:00
Sergei Trifonov
75c1053ad7 Merge remote-tracking branch 'origin/master' into surahman-CH-31221-AWS-S3-object-version-id 2022-04-19 13:17:25 +02:00
mergify[bot]
64d5e0c8a2
Merge branch 'master' into change-hashed-path 2022-04-19 09:33:48 +00:00
kssenii
1f9a597b77 Merge master 2022-04-19 11:31:13 +02:00
taiyang-li
99ad2f5fa6 Merge remote-tracking branch 'origin/master' into async_hdfs_read_buffer 2022-04-19 15:49:38 +08:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
Alexey Milovidov
f6ab2bd523
Merge pull request #36312 from ClickHouse/remove-arcadia 
Remove remaining parts of Arcadia
 2022-04-18 07:02:54 +03:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
taiyang-li
0ad2a76fae Merge remote-tracking branch 'origin/master' into async_hdfs_read_buffer 2022-04-16 18:45:39 +08:00
taiyang-li
cd83fd5f8a tobe debug 2022-04-16 18:41:18 +08:00
Alexey Milovidov
33ffdcaa4f Return two changes 2022-04-16 02:18:22 +02:00
Anton Popov
2de6668b3f show names of erroneous files 2022-04-16 00:10:47 +00:00
Alexey Milovidov
40357637ec Remove Arcadia 2022-04-16 00:33:32 +02:00
Alexey Milovidov
cbeeb7ec4f Remove Arcadia 2022-04-16 00:20:47 +02:00
Alexey Milovidov
416fa95b87 Remove "Arcadia" build system 2022-04-15 23:57:09 +02:00
mergify[bot]
7bad91e0f4
Merge branch 'master' into fix-parallel-read-buffer 2022-04-14 09:49:50 +00:00
Kruglov Pavel
4f9ee879d4
Merge pull request #36212 from Avogar/revert-fix 
Revert reverting "Fix crash in ParallelReadBuffer"
 2022-04-14 11:26:31 +02:00
Kseniia Sumarokova
04926d882a
Merge branch 'master' into change-hashed-path 2022-04-14 11:18:42 +02:00
Kseniia Sumarokova
4d667301cc
Merge pull request #36135 from kssenii/cache-fix 
Fix after #35475
 2022-04-14 11:11:13 +02:00
avogar
ca14a29dbe Revert reverting "Fix crash in ParallelReadBuffer" 2022-04-13 20:39:12 +00:00
tavplubix
b2cd23658a
Revert "Fix crash in ParallelReadBuffer" 2022-04-13 22:45:34 +03:00
mergify[bot]
2c5d6d132e
Merge branch 'master' into cache-fix 2022-04-13 13:32:56 +00:00
kssenii
0912b038fa Merge master 2022-04-13 15:30:01 +02:00
Sergei Trifonov
9d1a2da685 merge master and fix conflicts 2022-04-13 14:53:36 +02:00
kssenii
a6dc437911 Merge master 2022-04-13 14:23:48 +02:00
Kruglov Pavel
b2905c5aed
Update src/IO/ParallelReadBuffer.cpp 
Co-authored-by: tavplubix <tavplubix@gmail.com>
 2022-04-13 13:50:11 +02:00
avogar
a4d74f975e Use atomic instead of mutex + condvar in ParallelReadBuffer 2022-04-13 10:20:29 +00:00
kssenii
37242a0103 Allow non-evictable file segments based on predicate 2022-04-12 16:55:25 +02:00
avogar
de9c9462e5 Fix crash in ParallelReadBuffer 2022-04-12 13:54:53 +00:00
kssenii
9804c39de7 Composable cache 2022-04-12 14:33:13 +02:00
kssenii
d2ccf876c6 Fix 2022-04-11 17:51:49 +02:00
Kseniia Sumarokova
867b85dfad
Merge branch 'master' into change-hashed-path 2022-04-09 16:57:57 +02:00
tavplubix
faae540a85
Merge pull request #24258 from ClickHouse/mvcc_prototype 
MVCC prototype for MergeTree
 2022-04-08 22:02:13 +03:00
Kseniia Sumarokova
10d609fd0d
Update WriteBufferFromS3.cpp 2022-04-08 17:42:05 +02:00
kssenii
138588be29 Change hashed path 2022-04-08 17:14:13 +02:00
Kruglov Pavel
d7b88d7683
Merge pull request #35582 from Avogar/improve-schema-inference 
Improve schema inference and add some fixes
 2022-04-08 13:44:52 +02:00
Alexander Tokmakov
49c35f3261 Merge branch 'master' into mvcc_prototype 2022-04-08 13:34:40 +02:00
kssenii
87855c4ae7 Merge master 2022-04-08 12:32:48 +02:00
kssenii
ef2b238444 Fix 2022-04-08 12:30:24 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements 
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
 2022-04-08 12:06:26 +02:00
kssenii
5dce2f18b5 Better 2022-04-07 18:48:35 +02:00
Sergei Trifonov
01425df758 fix build 2022-04-07 16:07:12 +02:00
Alexander Tokmakov
8290ffa88d Merge branch 'master' into mvcc_prototype 2022-04-07 13:50:42 +02:00
avogar
d2017a63b1 Merge branch 'master' of github.com:ClickHouse/ClickHouse into improve-schema-inference 2022-04-07 11:36:40 +00:00
Sergei Trifonov
03f76dda1d better 2022-04-07 12:03:54 +02:00
Sergei Trifonov
9c2804b924 merge from master + resolve conflicts 2022-04-07 11:22:01 +02:00
Yakov Olkhovskiy
c73115ffb4
Merge pull request #35840 from zzsmdfj/issue/#34966_fix_dateTime_deserialize 
to #34966_fix_dateTime_deserialize
 2022-04-06 08:10:23 -04:00
Kruglov Pavel
9141066de3
Merge branch 'master' into improve-schema-inference 2022-04-06 13:51:07 +02:00
Yakov Olkhovskiy
ea9ce3ea18 'T' is added as delimiter, tests added 2022-04-05 19:50:16 -04:00
Alexander Tokmakov
1fe50ad201 Merge branch 'master' into mvcc_prototype 2022-04-05 14:38:02 +02:00
Alexander Tokmakov
da00beaf7f Merge branch 'master' into mvcc_prototype 2022-04-05 11:14:42 +02:00
Kseniia Sumarokova
77c7545395
Merge pull request #35922 from kssenii/fix-race-in-cached-buffer 
Fix race in cached buffer
 2022-04-05 10:55:58 +02:00
Alexey Milovidov
4d6c030d23
Revert "clang-tidy report issues with Medium priority" 2022-04-04 23:41:42 +03:00
kssenii
79627798c4 Fix race 2022-04-04 20:44:39 +02:00
Kseniia Sumarokova
d697576967
Update ReadBufferFromS3.h 2022-04-04 17:53:01 +02:00
kssenii
6c6fb5c3e8 Fix race in cached buffer 2022-04-04 16:32:37 +02:00
kssenii
4e50da722b Better s3 settings 2022-04-04 16:14:56 +02:00
kssenii
36c583d0de Better version of cache on insert 2022-04-04 10:58:59 +02:00
kssenii
6c8e073a61 Merge master 2022-04-01 16:48:29 +02:00
zzsmdfj
3cae0c74d9 to #34966_fix_dateTime_deserialize 2022-04-01 20:16:07 +08:00
zzsmdfj
ececee3817 to #34966_fix_dateTime_deserialize 2022-04-01 20:13:34 +08:00
kssenii
603dcbb1b7 Extend test 2022-03-31 18:23:15 +02:00
mergify[bot]
24ade25d61
Merge branch 'master' into improve-schema-inference 2022-03-31 13:42:47 +00:00
Alexander Tokmakov
5a50ad9de3 Merge branch 'master' into mvcc_prototype 2022-03-31 11:35:04 +02:00
kssenii
0fc92fe2aa Get rid of all "remote" mentions 2022-03-30 17:15:29 +02:00
Antonio Andelic
a3ad99be03 Create CallbackRunner in Storage 2022-03-30 11:24:50 +00:00
Antonio Andelic
de2c6684a3 Remove setup and cleanup 2022-03-30 09:00:06 +00:00
Antonio Andelic
1173eab15e Use same IOThreadPool for URL and S3 2022-03-30 08:15:20 +00:00
Antonio Andelic
03a09dc914
Merge pull request #35571 from ClickHouse/parallel-s3-downloading 
Parallel S3 downloading
 2022-03-30 08:14:49 +02:00
kssenii
f0e0d97752 Fix lask of query id in merges 2022-03-29 19:49:42 +02:00
Alexander Tokmakov
287d858fda Merge branch 'master' into mvcc_prototype 2022-03-29 16:24:12 +02:00
mergify[bot]
343588de2c
Merge branch 'master' into improve-schema-inference 2022-03-29 13:06:00 +00:00
Anton Popov
d42632dc13
Merge pull request #32928 from bigo-sg/rocksdb_metacache 
Speed up parts loading process of MergeTree by metadata cache
 2022-03-29 14:05:22 +02:00
Maksim Kita
a1a4552740
Merge pull request #35184 from DevTeamBK/clang-tidy-issues 
clang-tidy report issues with Medium priority
 2022-03-29 13:19:54 +02:00
taiyang-li
834ac3a1b9 Merge remote-tracking branch 'ck/master' into rocksdb_metacache 2022-03-29 18:32:06 +08:00
taiyang-li
38702be40c Merge remote-tracking branch 'ck/master' into rocksdb_metacache 2022-03-28 21:03:22 +08:00
Antonio Andelic
d9d826c813 Address PR review 2022-03-28 08:19:23 +00:00
Azat Khuzhin
6a3986f41b Fix redundant-move in ReadWriteBufferFromHTTP.h 2022-03-26 23:42:46 +03:00
kssenii
d6ab6de30a Minor changes 2022-03-24 15:32:08 +01:00
avogar
557edbd172 Add some improvements and fixes in schema inference 2022-03-24 12:54:12 +00:00
Antonio Andelic
62a495a5fc Small refactor 2022-03-24 09:45:30 +00:00
kssenii
d4161b5925 Add optin read_from_cache_if_exists_otherwise_bypass_cache (for merges) 2022-03-23 20:24:00 +01:00
kssenii
d2a3cfe5dc Cache on all write operations 2022-03-23 19:14:33 +01:00
kssenii
c2c7f365ce Merge master 2022-03-23 16:58:35 +01:00
kssenii
34c0690a69 Better 2022-03-23 15:43:14 +01:00
Alexander Tokmakov
bcec53aadc Merge branch 'master' into mvcc_prototype 2022-03-23 11:28:43 +01:00
Antonio Andelic
131b3a091c Refactor StorageS3 2022-03-23 08:57:01 +00:00
Antonio Andelic
f32ef2a556 Small polishing for S3 reader 2022-03-23 08:15:18 +00:00
taiyang-li
8dbf1c60e7 merge master and fix conflict 2022-03-23 11:36:50 +08:00
Antonio Andelic
6785ad165a Fix issue for mutliple download threads 2022-03-22 14:06:30 +00:00
Antonio Andelic
df99a9cc3c Merge branch 'parallel-downloading-url-engine' into parallel-s3-downloading 2022-03-22 12:49:35 +00:00
Antonio Andelic
5da3058368 Merge branch 'master' into parallel-downloading-url-engine 2022-03-22 12:20:36 +00:00
Alexander Tokmakov
3c762f566d Merge branch 'master' into mvcc_prototype 2022-03-21 20:16:29 +01:00
Antonio Andelic
04e6e6ab5e Add ParallelReadBuffer for S3 2022-03-21 14:52:26 +00:00
kssenii
afd0c64a1a Better 2022-03-21 15:04:16 +01:00
kssenii
eabbce69a7 Add system drop remote filesystem cache query, allow to write cache on insert 2022-03-21 12:32:44 +01:00
Vitaly Baranov
7a46e7a0d1
Merge branch 'master' into improve-backup-and-restore 2022-03-21 11:49:26 +01:00
Vitaly Baranov
51fc556fcd Implement BACKUP and RESTORE for MatViews. 2022-03-21 11:45:21 +01:00
Maksim Kita
695b1300f9
Merge pull request #35383 from rschu1ze/enable-if-to-concepts-pt2 
Replace more uses of std::enable_if for SFINAE by concepts
 2022-03-21 10:47:28 +01:00
Antonio Andelic
b8c43ff2f2 Merge branch 'master' into parallel-downloading-url-engine 2022-03-21 09:11:01 +00:00
kssenii
59b7394caf WriteSettings 2022-03-21 09:52:48 +01:00
kssenii
1d80262a2a Add write settings, file cache settings 2022-03-21 09:50:50 +01:00
Alexander Tokmakov
9e05b12d2c Merge branch 'master' into mvcc_prototype 2022-03-20 22:42:26 +01:00
Vitaly Baranov
c884cf901f Use BackupEntryFromAppendOnlyFile for Log engine, 
and use a BackupEntriesBatch for Memort engine to improve performance.
A lot of minor corrections.
 2022-03-20 20:02:15 +01:00
Vitaly Baranov
a1b364626a Improve zip archive utilities. 2022-03-20 20:02:15 +01:00
alesapin
5902304fdb Remove some useless code from S3Common 2022-03-19 23:28:20 +01:00
kssenii
af3bb3b7af Add WriteSettings 2022-03-18 12:27:38 +01:00
Antonio Andelic
f2a7c6c6fe Add support for canceling a single worker 2022-03-18 09:28:56 +00:00
Robert Schulze
df73c75456
Replace more uses of std::enable_if for SFINAE by concepts 
- enable_if is usually regarded as fragile and unreadable
- C++20 concepts are much easier to read and produce more expressive error messages

- this is follow-up to PR #35347 but this time most of the remaining and
  more complex usages of enable_if in the codebase were replaced.
 2022-03-17 20:44:55 +01:00
Antonio Andelic
b789a41662 Fix ASAN problems 2022-03-17 15:18:20 +00:00
Alexander Tokmakov
d04dc03fa4 Merge branch 'master' into mvcc_prototype 2022-03-17 15:24:32 +01:00
Antonio Andelic
103a3fa140 Merge branch 'master' into parallel-downloading-url-engine 2022-03-17 10:11:58 +00:00
Antonio Andelic
44de5a581c Fix split build 2022-03-17 10:09:16 +00:00
alesapin
457fa0d1ba
Merge pull request #23932 from CurtizJ/dynamic-columns 
Support of dynamic subcolumns (JSON data type)
 2022-03-17 10:45:46 +01:00
Maksim Kita
4d8c2b2009
Merge pull request #35347 from rschu1ze/enable-if-to-concepts 
Replace a few uses of enable_if for SFINAE by concepts
 2022-03-17 10:34:08 +01:00
Alexander Tokmakov
4b3e13a4fe Merge branch 'master' into mvcc_prototype 2022-03-16 21:06:19 +01:00
Alexander Tokmakov
c2ac8d4a5c review fixes 2022-03-16 21:05:34 +01:00
Robert Schulze
b56c28d841
Replace a few uses of enable_if for SFINAE by concepts 
- enable_if is usually regarded as fragile and unreadable

- C++20 concepts are much easier to read and produce more expressive
  error messages
 2022-03-16 19:51:38 +01:00
Antonio Andelic
0979155f2f Address PR comments 2022-03-16 15:45:17 +00:00
Anton Popov
0ba78c3c3a Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-16 15:28:09 +00:00
Antonio Andelic
84f86bf3da Limit number of bytes read per worker 2022-03-16 12:34:14 +00:00
Antonio Andelic
2291b7c1e1 Merge branch 'master' into parallel-downloading-url-engine 2022-03-16 09:59:52 +00:00
Antonio Andelic
3465664871 Attach thread pool threads to counters 2022-03-16 09:26:01 +00:00
Kruglov Pavel
2b7e89e171
Merge pull request #35300 from Avogar/fix-bzip2 
Fix possible assertion while using bzip2 compression
 2022-03-15 22:17:37 +01:00
Kruglov Pavel
20e17a6d3e
Merge pull request #35296 from Avogar/fix-lz4 
Fix possible segfault while using lz4 compression
 2022-03-15 22:17:14 +01:00
Kruglov Pavel
6fd1a8281f
Merge pull request #35295 from Avogar/fix-lzma 
Fix possible Assertion failed while using lzma compression
 2022-03-15 22:14:35 +01:00
Nikolai Kochetov
a380aa6b8a
Merge pull request #35294 from ClickHouse/reload-remote_url_allow_hosts 
Reload remote_url_allow_hosts after config update.
 2022-03-15 22:07:16 +01:00
Saad Ur Rahman
275bf0ef81
[IO] fixed bug in versionId extraction introduced during rebase. 2022-03-15 12:13:44 -04:00
avogar
ec3c5f86ad Fix possible assertion while using bzip2 compression 2022-03-15 14:23:52 +00:00
avogar
0cfaf48e1d Fix possible segfault while using lz4 compression 2022-03-15 13:46:14 +00:00
Alexander Tokmakov
1f571b7734 Merge branch 'master' into mvcc_prototype 2022-03-15 14:45:06 +01:00
Kruglov Pavel
24ac217be3
Remove redundant include 2022-03-15 14:07:31 +01:00
avogar
90ce1709d7 Fix possible Assertion failed while using lzma compression 2022-03-15 13:00:53 +00:00
Nikolai Kochetov
97aa6c82ce Reload remote_url_allow_hosts after config update. 2022-03-15 13:00:31 +00:00
mergify[bot]
d613ad9445
Merge branch 'master' into fix-brotli 2022-03-15 08:25:12 +00:00
Antonio Andelic
067b79b00b Merge branch 'master' into parallel-downloading-url-engine 2022-03-15 07:55:41 +00:00
Saad Ur Rahman
7b35c79b04
[IO] versionId extraction simplified. 
Reducing cognitive load.
 2022-03-14 22:27:58 -04:00
Saad Ur Rahman
d9b370cf1d
[IO] S3 URI versionId extraction performance tweak. 2022-03-14 22:17:50 -04:00
Saad Ur Rahman
1a85e9b60d
[IO] Extracting Version ID in S3::URI 
Extracting Version ID form query string in URI.
 2022-03-14 22:16:24 -04:00
Saad Ur Rahman
ec28af055f
[IO] S3 URI data struct contains Version ID 
Added Version ID field to S3::URI struct.
TODO: Extract version id if present.
 2022-03-14 22:11:18 -04:00
Saad Ur Rahman
c7402c2a39
[IO] Version ID provided during buffer read 
The Version ID, if provided, is to be set in the request to read the buffer from S3. If not provided the latest version of the object must be retrieved.

Version ID set to last parameter to allow defaulting and backward compatibility with calls.
 2022-03-14 20:10:14 -04:00
Alexander Tokmakov
9702b5177d Merge branch 'master' into mvcc_prototype 2022-03-14 21:45:38 +01:00
avogar
4b12c6e6b3 Fix possible Abort while using Brotli compression 2022-03-14 18:58:05 +00:00
Maksim Kita
e30117a3d6 Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Antonio Andelic
5f8d9885b3 Ignore read bytes 2022-03-14 15:50:30 +00:00
Azat Khuzhin
9f7fae56b0 WriteBufferFromS3: remove unused headers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-14 12:07:47 +03:00
Antonio Andelic
f5d3a8a31d Polishing 2022-03-11 13:38:19 +00:00
Antonio Andelic
28e9508c4e Improve shared pool and add settings 2022-03-11 08:33:34 +00:00
Antonio Andelic
cce318273b Use a shared IO thread pool 2022-03-11 08:33:34 +00:00
Antonio Andelic
adaaaaf061 Unblock readers before stoping 2022-03-11 08:33:34 +00:00
Antonio Andelic
df0f5e20d0 Refactor 2022-03-11 08:33:34 +00:00
Antonio Andelic
7b526b25a8 Initial implementation with Arena 2022-03-11 08:33:34 +00:00
Antonio Andelic
16c20332cf Fix deadlocks 2022-03-11 08:33:34 +00:00
Antonio Andelic
c6664bbbdf Add seekable ParallelReadBuffer 2022-03-11 08:33:34 +00:00
Antonio Andelic
29c32ed831 Refactor code 2022-03-11 08:33:34 +00:00
Antonio Andelic
0a1a3a230e Add support for parallel http range requests 2022-03-11 08:33:34 +00:00
Antonio Andelic
aad87879cf Small refactor 2022-03-11 08:33:34 +00:00
Kseniia Sumarokova
818459b9f0
Merge pull request #33717 from kssenii/local-cache-for-remote-fs 
Local cache for remote filesystem
 2022-03-11 07:23:10 +01:00
Rajkumar
137e4b79c1 override needed 2022-03-10 12:46:32 -08:00
Kseniia Sumarokova
5149bd2d29
Merge pull request #35172 from kssenii/add-non-retriable-errors-for-url 
Add non-retriable errors check for HTTP read buffer
 2022-03-10 20:28:37 +01:00
Rajkumar
0de950c3f8 style correction 2022-03-10 10:29:57 -08:00
Rajkumar
81ad3ca220 syle check 2022-03-10 10:19:03 -08:00
Rajkumar
a17e5efea2 adding destructors 2022-03-10 07:31:06 -08:00
Alexander Tokmakov
061fa6a6f2 Merge branch 'master' into mvcc_prototype 2022-03-10 13:13:04 +01:00
kssenii
050802b99b Add non-retriable errors 2022-03-10 11:38:46 +01:00
lgbo-ustc
fdd423a3da fixed code style 2022-03-10 12:13:19 +08:00
lgbo-ustc
e4883f31b7 update tests 
1. fixed code style in src/IO/tests/gtest_hadoop_snappy_decoder.cpp
2. enable tests 01060_avro.sh
 2022-03-10 09:46:43 +08:00
lgbo-ustc
8dc8c87fd1 add a test case 2022-03-09 10:03:04 +08:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0 Fix split build 2022-03-08 18:05:55 +01:00
kssenii
f3cba6f270 Remove already fixed crutch 2022-03-08 14:30:21 +01:00
lgbo-ustc
7f89a1bcf3 add some usage test 2022-03-08 20:00:39 +08:00
lgbo-ustc
225b0bd914 fixed bug: call need_more_input repeatly, overwrite the buffer 2022-03-08 17:17:06 +08:00
taiyang-li
b4174b0bef merge master and fix conflicts 2022-03-08 11:39:25 +08:00
kssenii
0737aef472 Fix split build 2022-03-07 22:02:07 +01:00
Alexander Tokmakov
8acfb8d27f Merge branch 'master' into mvcc_prototype 2022-03-07 17:40:15 +01:00
kssenii
eeb0ae9816 Fix detaching query 2022-03-07 17:05:36 +01:00
Anton Popov
0bc57da238 Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-07 14:46:08 +00:00
kssenii
692b247379 Fix some corner cases 2022-03-07 14:31:21 +01:00
kssenii
2dcfe66236 Fix race 2022-03-06 20:33:07 +01:00
Anton Popov
df3b07fe7c Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-03 22:25:28 +00:00
vdimir
d3ee3dee85
Catch exception in ~WriteBufferFromS3 2022-03-03 12:54:24 +00:00
Maksim Kita
b1a956c5f1 clang-tidy check performance-move-const-arg fix 2022-03-02 18:15:27 +00:00
Maksim Kita
1f5837359e clang-tidy check performance-noexcept-move-constructor fix 2022-03-02 18:15:27 +00:00
kssenii
d5952109fb Merge master 2022-03-02 18:15:25 +01:00
HaiBo Li
2ea18b3085
Fix the bug that the toDatetime function overflows (#32898) 
* Fixed overflow bug of the function toDatetime/toDate/toDate32/parseDateTimeBestEffort
 2022-03-02 10:06:38 +03:00
Anton Popov
2758db5341 add more comments 2022-03-01 19:32:55 +03:00
Danila Kutenin
3ce71f211a First portion of force inlines 2022-02-27 10:43:17 +00:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
Alexander Tokmakov
aa6b9a2abc Merge branch 'master' into mvcc_prototype 2022-02-23 23:22:03 +03:00
kssenii
c637385dd0 Merge master 2022-02-22 13:17:51 +01:00
Kseniia Sumarokova
eeea322556
Merge pull request #34629 from amosbird/remotefsimprove 
Some refactoring and improvement over async and remote buffer related stuff
 2022-02-22 11:36:40 +01:00
kssenii
fad7c042c1 Fix cache with non-async reads 2022-02-21 19:50:27 +01:00
Amos Bird
01652d9892
Add more assertion 2022-02-21 22:42:43 +08:00
kssenii
4543513a5d Add comments 2022-02-21 13:54:03 +01:00
Maksim Kita
883c7c21d1
Merge pull request #34690 from nvartolomei/nv/always-increment 
Always update ProfileEvents (even on exceptions)
 2022-02-21 00:03:44 +01:00
Maksim Kita
e049f6ada9
Update WriteBufferFromPocoSocket.cpp 2022-02-20 16:59:39 +01:00
Maksim Kita
d44f48fe1f
Update ReadBufferFromPocoSocket.cpp 2022-02-20 16:59:17 +01:00
kssenii
e8b2e47e93 Style 2022-02-19 20:38:23 +01:00
Amos Bird
84d111cde9
Better ReadBuffer asynchronicity 2022-02-20 01:08:21 +08:00
kssenii
acf9a092a9 Review fixes 2022-02-18 19:09:54 +01:00
Nicolae Vartolomei
1778be516b Always update ProfileEvents (even on exceptions) 2022-02-17 16:29:47 +00:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
alesapin
6df2c9c2d8
Merge pull request #34433 from ClickHouse/s3-fix-content-type 
Workaround for bug when AWS mistakenly set Content-Type to XML
 2022-02-17 13:11:14 +03:00
kssenii
47f94120da Merge master 2022-02-16 14:43:28 +01:00
kssenii
3bd3e51aa0 Fix tests 2022-02-16 14:08:41 +01:00
alesapin
36b1944cb0 Merge branch 'master' into s3-fix-content-type 2022-02-15 12:39:49 +03:00
李扬
f52b67b939
Merge branch 'master' into rocksdb_metacache 2022-02-15 02:16:29 -06:00
Alexander Tokmakov
1e4e569151 Merge branch 'master' into mvcc_prototype 2022-02-15 02:26:47 +03:00
kssenii
bdbbe85f95 Fixes 2022-02-14 19:28:48 +01:00
alesapin
89373155fc Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-13 21:07:54 +03:00
Maksim Kita
cb09aadc72
Merge pull request #34524 from FrankChen021/urlengine 
Fix compression support in URL engine
 2022-02-13 14:39:36 +01:00
Saad Ur Rahman
03f81c8685
[CH-34239] HTTP User-Agent header (#34330) 
* [IO] HTTP User-Agent header

Added HTTP User-Agent Header to HTTP requests.

User-Agent: ClickHouse/VERSION_STRING
Input header vectors could potentially contain User-Agent. If so, do not set another.

* [Tests] HTTP User Agent

Added stateless functional test 02205_HTTP_user_agent.

Co-authored-by: Filatenkov Artur <58165623+FArthur-cmd@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-02-13 00:40:11 +03:00
李扬
daa27d0bda
Merge branch 'master' into rocksdb_metacache 2022-02-12 07:50:12 -06:00
Azat Khuzhin
cc55c8b982 Fix gtest_archive_reader_and_writer in case of !USE_MINIZIP 
Empty arrays/vectors are not supported by ValuesIn:

    /src/ch/clickhouse/src/IO/tests/gtest_archive_reader_and_writer.cpp:341:59: error: no matching function for call to 'ValuesIn'
    INSTANTIATE_TEST_SUITE_P(All, ArchiveReaderAndWriterTest, ::testing::ValuesIn(supported_archive_file_exts));
                                                              ^~~~~~~~~~~~~~~~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:459:43: note: expanded from macro 'INSTANTIATE_TEST_SUITE_P'
        return GTEST_EXPAND_(GTEST_GET_FIRST_(__VA_ARGS__, DUMMY_PARAM_));        \
                                              ^~~~~~~~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:453:38: note: expanded from macro 'GTEST_GET_FIRST_'
    #define GTEST_GET_FIRST_(first, ...) first
                                         ^~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:452:28: note: expanded from macro 'GTEST_EXPAND_'
    #define GTEST_EXPAND_(arg) arg
                               ^~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:301:29: note: candidate template ignored: substitution failure [with T = std::vector<std::string>, N = 0]: zero-length arrays are not permitted in C++
    internal::ParamGenerator<T> ValuesIn(const T (&array)[N]) {
                                ^                         ~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:306:58: note: candidate template ignored: substitution failure [with Container = std::vector<std::string> [0]]: type 'std::vector<std::string> [0]' cannot be used prior to '::' because it has no members
    internal::ParamGenerator<typename Container::value_type> ValuesIn(
                                      ~~~~~~~~~              ^
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:294:1: note: candidate function template not viable: requires 2 arguments, but 1 was provided
    ValuesIn(ForwardIterator begin, ForwardIterator end) {
    ^
    /src/ch/clickhouse/src/IO/tests/gtest_archive_reader_and_writer.cpp:341:59: error: no matching function for call to 'ValuesIn'
    INSTANTIATE_TEST_SUITE_P(All, ArchiveReaderAndWriterTest, ::testing::ValuesIn(supported_archive_file_exts));
                                                              ^~~~~~~~~~~~~~~~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:468:32: note: expanded from macro 'INSTANTIATE_TEST_SUITE_P'
          auto t = std::make_tuple(__VA_ARGS__);                                  \
                                   ^~~~~~~~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:301:29: note: candidate template ignored: substitution failure [with T = std::vector<std::string>, N = 0]
    internal::ParamGenerator<T> ValuesIn(const T (&array)[N]) {
                                ^
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:306:58: note: candidate template ignored: substitution failure [with Container = std::vector<std::string> [0]]: type 'std::vector<std::string> [0]' cannot be used prior to '::' because it has no members
    internal::ParamGenerator<typename Container::value_type> ValuesIn(
                                      ~~~~~~~~~              ^
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:294:1: note: candidate function template not viable: requires 2 arguments, but 1 was provided
    ValuesIn(ForwardIterator begin, ForwardIterator end) {
    ^
    2 errors generated.
    ninja: build stopped: subcommand failed.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-12 15:01:20 +03:00
Frank Chen
7b7bc8f6c3 Address review comment 2022-02-12 10:46:01 +08:00
Alexander Tokmakov
07e66e690d Merge branch 'master' into mvcc_prototype 2022-02-11 15:53:32 +03:00
mergify[bot]
f099048dd1
Merge branch 'master' into urlengine 2022-02-11 11:03:21 +00:00
Frank Chen
c8cddd7077 Fix compression for URL engine 2022-02-11 17:45:49 +08:00
alesapin
84b5fbf43c Fix buffer HDFS crash 2022-02-10 16:35:50 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
alesapin
b84a171aa7 Fix style 2022-02-09 14:13:53 +03:00
alesapin
437940b29d
Merge pull request #34422 from ClickHouse/add_settings_to_part_size_in_s3 
Add settings for multipart upload part size in s3
 2022-02-09 12:51:20 +03:00
Kseniia Sumarokova
86956ca08b
Merge pull request #34392 from kssenii/http-buffer-skip-not-found-url-for-globs 
Allow to skip not found urls for globs
 2022-02-09 09:32:09 +01:00
Alexey Milovidov
734b08177e Fix excessive logging while writing to s3 2022-02-09 07:19:33 +03:00
taiyang-li
d04ccc0489 Merge branch 'master' into rocksdb_metacache 2022-02-09 11:54:10 +08:00
Alexey Milovidov
92fb6bf735 Workaround for bug in AWS 2022-02-09 05:25:22 +03:00
kssenii
e8a8f46864 Fix style check 2022-02-08 21:58:03 +01:00
alesapin
f6e7cea1f8 Add settings for S3 multipart upload parts size 2022-02-08 19:38:04 +03:00
Vitaly Baranov
9a8e45805a
Merge pull request #34226 from vitlibar/add-submodule-minizip 
Add submodule minizip
 2022-02-08 22:07:34 +07:00
Kruglov Pavel
720310fcf1
Merge branch 'master' into http-buffer-skip-not-found-url-for-globs 2022-02-08 16:56:22 +03:00
kssenii
dc5f035265 Fix 2022-02-08 13:28:10 +01:00
kssenii
3834bdbae0 Fixes 2022-02-08 11:11:18 +01:00
taiyang-li
b6132d490f merge master and solve conflict 2022-02-08 15:24:59 +08:00
kssenii
eba3011ada Fix 2022-02-07 20:40:47 +01:00
Anton Popov
d32d6453d3
Update BufferBase.h 2022-02-07 17:49:53 +03:00
kssenii
2e58733750 Merge master 2022-02-07 15:23:10 +01:00
Anton Popov
1b16db72c3 fix consecutive backward seeks in seekable read buffers 2022-02-07 17:20:26 +03:00
Anton Popov
ae1fc94fb5 add unit test 2022-02-06 20:45:59 +00:00
Anton Popov
10b8684003 fix rare bug in reading of empty arrays 2022-02-06 20:45:59 +00:00
Alexander Tokmakov
45be75b4db Merge branch 'master' into mvcc_prototype 2022-02-06 23:36:08 +03:00
Vitaly Baranov
d9bdbf47b7 ReadBufferFromMemory now can seek backwards after been read up to EOF. 2022-02-05 23:45:20 +07:00
Vitaly Baranov
23fac284ea Add utility classes ZipArchiveReader and ZipArchiveWriter. 2022-02-05 23:45:20 +07:00
Nikolai Kochetov
daeeb6f3a2
Merge pull request #34316 from ClickHouse/probably-fix-data-race-in-WriteBufferFromS3 
Probably fix data race in WriteBufferFromS3 destructor.
 2022-02-04 21:04:46 +03:00
Nikolai Kochetov
a968c869fd Probably fix data race in WriteBufferFromS3 destructor. 2022-02-04 12:28:14 +00:00
Maksim Kita
074b827cf3
Merge pull request #34306 from ClickHouse/line-as-string-low-performance 
Fix terribly low performance of `LineAsString` format
 2022-02-04 13:11:27 +01:00
Alexey Milovidov
e20aec6866 Fix linkage 2022-02-04 01:44:43 +03:00
Alexey Milovidov
4a83dbc514 Fix linkage 2022-02-04 00:26:44 +03:00
Alexey Milovidov
7c12f5f37a Fix terribly low performance of LineAsString format 2022-02-04 00:07:31 +03:00
Alexander Tokmakov
ca5f951558 Merge branch 'master' into mvcc_prototype 2022-02-03 18:56:44 +03:00
Nikolai Kochetov
2003a96b58 Probably fix something 2022-02-02 17:48:41 +00:00
Nikolai Kochetov
ea044fc6b2 Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer 2022-02-02 19:40:41 +03:00
kssenii
aeec749173 Fix predownload case part 1 2022-02-02 17:25:25 +03:00
kssenii
5cba11428a Fixes 2022-02-02 00:25:16 +03:00
Sergei Trifonov
68bc456830
Merge pull request #34223 from azat/bump-fmt 
Bump fmtlib from 7.0.0 to 8.1.1
 2022-02-02 00:03:25 +03:00
alexey-milovidov
2b1d1a9a6f
Merge pull request #34208 from ClickHouse/parse-date-time-best-effort-more-cases 
More cases for `parseDateTimeBestEffort`
 2022-02-01 16:22:52 +03:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Azat Khuzhin
3b3635c6d5 Fix formatting error in logging messages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Amos Bird
e29c77f793 Fix link order issue of WriteBufferFromS3 
(cherry picked from commit f22b09f4fc)
 2022-02-01 10:42:06 +00:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3"" 2022-02-01 13:36:51 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3" 2022-02-01 05:49:40 +03:00
Alexey Milovidov
989a5ad35e More cases for parseDateTimeBestEffort 2022-02-01 03:29:27 +03:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines 
Fix benign race condition for storage HDFS, S3, URL
 2022-01-31 22:41:54 +03:00
alesapin
31753afb7e Fix cancel logic in parallel parsing 2022-01-31 16:38:15 +03:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
kssenii
6b03c577d6 Make downloader's term resticted to one nextImpl 2022-01-29 23:58:53 +03:00
Alexander Tokmakov
fb9b2d5326 Merge branch 'master' into mvcc_prototype 2022-01-28 21:18:36 +03:00
Nikolai Kochetov
1c9f026178 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-28 16:01:42 +03:00
taiyang-li
3de8bde7ce Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-28 09:58:52 +08:00
kssenii
c91b86e220 Better 2022-01-26 21:56:08 +03:00
Nikolai Kochetov
a8171269a1 Review fixes. 2022-01-26 17:55:24 +00:00
Nikolai Kochetov
2e88319b72 Fix integration tests. 2022-01-25 18:48:28 +00:00
Nikolai Kochetov
fcc29dbd15 Try to fix integration tests. 2022-01-25 15:26:36 +00:00
Kruglov Pavel
1b19139f74
Merge pull request #31862 from Avogar/fix-lz4 
Try to fix lz4 compression
 2022-01-25 14:34:45 +03:00
Nikolai Kochetov
6c65915241 Fix style 2022-01-24 16:12:23 +00:00
Nikolai Kochetov
506ee8c024 Refactor some code. 2022-01-24 15:55:29 +00:00
Nikolai Kochetov
4afa75e80b Fix deduplication. 2022-01-24 14:43:36 +00:00
taiyang-li
73def8b483 merge master and solve conflict 2022-01-24 11:01:43 +08:00
Kseniia Sumarokova
2c43e4eb57
Merge branch 'master' into local-cache-for-remote-fs 2022-01-22 22:53:55 +03:00
alexey-milovidov
eb6849f7c7
Merge pull request #33842 from azat/cmake-contrib-fixes 
More cmake external modules cleanups
 2022-01-22 10:34:54 +03:00
Nikolai Kochetov
b3cbb63487 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:41:54 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001 Merge branch 'master' into mvcc_prototype 2022-01-21 18:59:01 +03:00
Azat Khuzhin
2aa7fb28a0 contrib/dragonbox: add ALIAS library 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-21 10:11:22 +03:00
Azat Khuzhin
c07a2c856a Fix bypass virtual dispatch in ZstdDeflatingAppendableWriteBuffer 
clang-tidy [1]:

    "Call to virtual method 'ZstdDeflatingAppendableWriteBuffer::finalizeAfter' during destruction bypasses virtual dispatch"

  [1]: https://s3.amazonaws.com/clickhouse-builds/33840/d021190b8c7ae0f5dc2decb953c471ea9d770981/binary_tidy/build_log.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-20 20:40:04 +03:00
Azat Khuzhin
d021190b8c keeper: fix memory leak in case of compression is used (default) 
In case of compression is used, ZstdDeflatingAppendableWriteBuffer is
used, but it has a leak, since it frees ZSTD_CCtx only if there was
write while this is not correct, since it is created anyway.

This was found with jemalloc profile, keeper-bench and the following
keeper settings:

- force_sync=false
- snapshot_distance=100
- reserved_log_items=0
- rotate_log_storage_interval=100

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-20 19:03:00 +03:00
Alexander Tokmakov
0ea0a8ccc7 Merge branch 'master' into mvcc_prototype 2022-01-20 17:05:55 +03:00
mergify[bot]
9326f99d78
Merge branch 'master' into fix-lz4 2022-01-20 11:53:51 +00:00
Azat Khuzhin
ca8525a733 Remove unbundled zlib support 2022-01-20 08:47:16 +03:00
Alexander Tokmakov
e9a5a64a71 Merge branch 'master' into mvcc_prototype 2022-01-19 21:41:23 +03:00
kssenii
a566099759 Local cache for remote filesystem 2022-01-19 00:42:11 +03:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl 
RFC: Split headers, move SystemLog into module, more forward declarations
 2022-01-18 17:22:49 +03:00
Alexander Tokmakov
4dd216ce8f Merge branch 'master' into mvcc_prototype 2022-01-14 17:11:32 +03:00
taiyang-li
66813a3aa9 merge master 2022-01-12 16:56:29 +08:00
Maksim Kita
c1ec85d87a Build with ENABLE_TESTS fix 2022-01-11 12:30:40 +03:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Alexander Tokmakov
158fbaaa29 fix 2022-01-10 20:07:01 +03:00
Alexander Tokmakov
bf2b6c2c37 Merge branch 'master' into mvcc_prototype 2022-01-10 19:53:50 +03:00
taiyang-li
83d064c24a fix unit test and build error 2022-01-06 11:41:24 +08:00
taiyang-li
2bb7ec8f72 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-06 10:44:17 +08:00
taiyang-li
c0a9c2b916 refactor metadatacache to reduce using of USE_ROCKSDB 2022-01-05 19:51:50 +08:00
Azat Khuzhin
2cdc5b37e3 Do not try to read pass EOF (to workaround a bug in a kernel) 
For unaligned offset pread() may return EINVAL even if the offset pass
EOF, although it should not, since otherwise there is no abiliity to
rely on read() == 0 is EOF (with pread() loop).

Here is a reproducer for the problem on 4.9.0-12-amd64:

    $ head -c27 /dev/urandom > /tmp/pread.issue
    $ xfs_io
    xfs_io> open -d /tmp/pread.issue
    xfs_io> pread 1000 4096
    pread: Invalid argument

And this is how it should work:

    xfs_io> pread 29 4096
    read 0/4096 bytes at offset 29

Note, here I use interactive mode since we had old xfs_io that does not
allow to execute multiple commands at once, and to avoid EMFILE issue

Here is some history of a patches that affects this behaviour in the
linux kernel:

- the issue had been introduced in
  torvalds/linux@9fe55eea7e v3.14
  ("Fix race when checking i_size on direct i/o read")
- an attempt to fix it had been made in
  torvalds/linux@74cedf9b6c v4.4
  ("direct-io: Fix negative return from dio read beyond eof")
- but this wasn't enough, since alignment check was earlier, so
  eventually fixed in
  torvalds/linux@41b21af388 v5.10
  ("direct-io: defer alignment check until after the EOF check")

Someone may ask why CI does not shows the issue, since:
- it had 4.19 kernel when CI was in yandex
- now it has 5.4 when CI is in AWS
Since both of those kernels does not have the last patch.

But, this bug requires the following conditions to met:
- index_granularity_bytes=0
- min_merge_bytes_to_use_direct_io=1
Which was not covered by CI yet.
 2022-01-04 10:53:17 +03:00
Azat Khuzhin
71a99ab27c Accept real file size in createReadBufferFromFileBase() 
Right now streams relies on correct file size not the number of bytes
that will be read from the stream, to overcome one bug in the linux
kernel that may return EIINVAL for pread() with offset pass the EOF.

v2: Swap read_hint and file_size (since it is easy to miss something)

Before the first argument to readFile()/createReadBufferFromFileBase()
was read_hint, not the file_size, and let's preserve the order, since
it is easy to miss something

This will also fix 02051_read_settings test automatically because now
MergeTreeReaderStream will pass estimated_sum_mark_range_bytes to
read_hint not file_size, previously it cause on of the following errors:
- Attempt to read after EOF w/ O_DIRECT
- and LOGICAL_ERROR while adjusting granulas w/o O_DIRECT

This will also improve zero-length reads guard (via
ReadBufferFromEmptyFile), that had been added in #30190

v3: fix for other storages that wasn't enabled in fast-test
v4: ignore ENOENT/ENOTSUP in readFile
 2022-01-04 10:53:17 +03:00
Azat Khuzhin
e548dae8d9 Pass size of the file to the readers 
v2: s/size/file_size/g
 2022-01-04 10:53:17 +03:00
taiyang-li
1e102bc1b2 merge master 2022-01-01 09:01:06 +08:00
Alexander Tokmakov
0d91887cdc save versions of parts 2021-12-30 16:15:28 +03:00
Nikolay Degterinsky
79e15e84d5 Refactor 2021-12-30 04:47:34 +00:00
Kruglov Pavel
489a30859f
Merge pull request #32455 from Avogar/schema-inference 
Automatic schema inference for input formats
 2021-12-29 21:03:48 +03:00
Nikolai Kochetov
68bf38133a Add pool to WriteBufferFromS3 2021-12-29 14:04:21 +00:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
benbiti
9004e98992 parseDateTimeBestEffort support Unix Timestamp with Milliseconds 2021-12-29 13:43:54 +08:00
Alexander Tokmakov
8aec3ae94f Merge branch 'master' into mvcc_prototype 2021-12-28 14:24:36 +03:00
taiyang-li
9036b18c2f merge master 2021-12-27 15:12:48 +08:00
alexey-milovidov
47d50c3bd1
Merge pull request #33140 from kssenii/azure-tests-fix 
Fix azure blob storage tests failures
 2021-12-25 06:25:26 +03:00
alexey-milovidov
d15da636f6
Update ReadBufferFromAzureBlobStorage.cpp 2021-12-25 05:57:15 +03:00
alexey-milovidov
bcd4df789d
Update WriteBufferFromAzureBlobStorage.cpp 2021-12-25 05:56:15 +03:00
alexey-milovidov
f2075fac64
Update ReadBufferFromAzureBlobStorage.cpp 2021-12-25 05:55:49 +03:00
kssenii
b4420cfa3e Fix 2021-12-24 22:51:36 +03:00
Alexey Milovidov
29d28c531f Move code around to avoid dlsym on Musl 2021-12-24 12:25:27 +03:00
Kruglov Pavel
f38bc9fc38
Merge branch 'master' into fix-lz4 2021-12-23 16:13:26 +03:00
Nikolay Degterinsky
826f7cb0f5
Merge pull request #32240 from benbiti/master 
Fix unix timestamp Millisecond convert to DateTime64, fractional part calc reversed
 2021-12-23 12:11:53 +03:00
Kseniia Sumarokova
f0434f21ce
Merge pull request #33039 from kssenii/fix-azure-exception 
Fix exception in azure write buffer desctructor
 2021-12-23 10:26:40 +03:00
Vitaly Baranov
7660530fcc
Merge pull request #32955 from azat/read-fix 
Fix UB in case of unexpected EOF during filling a set from HTTP query
 2021-12-23 06:41:14 +03:00
Alexander Tokmakov
b7184732d5 Merge branch 'master' into mvcc_prototype 2021-12-22 19:46:09 +03:00
Dmitry Novik
d256c3a9ca
Merge pull request #29224 from ClickHouse/fix_IP_conversion 
Merging #27914
 2021-12-22 18:36:13 +03:00
mergify[bot]
04f727e84c
Merge branch 'master' into read-fix 2021-12-22 13:49:04 +00:00
kssenii
65c5334bb0 Safer 2021-12-22 14:22:14 +03:00
kssenii
060d3228e9 Fix 2021-12-22 13:43:21 +03:00
kssenii
d8ed4db6d4 Fix 2021-12-22 11:42:23 +03:00
Kseniia Sumarokova
484a4d62dd
Merge pull request #32835 from kssenii/fising-s3 
Fix optimization with lazy seek for async reads from remote fs
 2021-12-22 09:38:23 +03:00
benbiti
81a7a393c8 fix unix timestamp convert, fractional reversed 2021-12-22 01:32:16 +08:00
mergify[bot]
7da6d8b310
Merge branch 'master' into fix_IP_conversion 2021-12-21 12:16:05 +00:00
kssenii
ebcb66a361 Fix 2021-12-21 10:58:02 +03:00
taiyang-li
2597925724 merge master 2021-12-21 15:55:39 +08:00
Azat Khuzhin
4230f04f72 Fix UB in LimitReadBuffer in case of unexpected EOF 
Fixes the following possible assertions:
- ./src/IO/ReadBuffer.h:58: bool DB::ReadBuffer::next(): Assertion `!hasPendingData()' failed.
- ./src/IO/LimitReadBuffer.cpp:17: virtual bool DB::LimitReadBuffer::nextImpl(): Assertion `position() >= in->position()' failed.

Fixes: 02151_http_s_structure_set_eof
 2021-12-21 08:38:41 +03:00
Alexander Tokmakov
9cd49bc0ec Merge branch 'master' into mvcc_prototype 2021-12-20 22:06:22 +03:00
Nikolai Kochetov
35883e0dae
Merge pull request #32979 from ClickHouse/decrease-some-s3-log-level 
Decrease log level for some s3 messages.
 2021-12-20 20:56:22 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment 
Loops remove postfix increment
 2021-12-20 19:51:07 +03:00
Vitaly Baranov
e61d3eef0c
Merge pull request #32747 from vitlibar/improve-grpc-compression 
Improve gRPC compression support
 2021-12-20 19:44:58 +03:00
Nikolai Kochetov
7dea7b7f76 Decrease log level for some s3 messages. 2021-12-20 18:18:54 +03:00
Maksim Kita
51477adf1b Updated additional cases 2021-12-20 15:55:07 +03:00
Vitaly Baranov
02d6cea857 Lz4DeflatingWriteBuffer now supports changes in the nested buffer between calls. 2021-12-19 16:29:57 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
kssenii
40c266840c Fixes 2021-12-17 18:16:29 +03:00
kssenii
ed42620530 Rename BlobStorage to AzureBlobStorage 2021-12-17 12:24:05 +03:00
Alexander Tokmakov
32e62ed5c2 Merge branch 'master' into mvcc_prototype 2021-12-17 10:43:46 +03:00
taiyang-li
d033fc4c24 merge master and fix conflict 2021-12-17 15:11:21 +08:00
kssenii
ca283a4bb8 Fix optimization with lazy seek 2021-12-17 00:18:28 +03:00
alesapin
c338bee718
Revert "Fix build issue related to azure blob storage" 2021-12-16 14:37:12 +03:00
alesapin
c0cebaea79
Merge pull request #32788 from amosbird/buildfix 
Fix build issue related to azure blob storage
 2021-12-16 13:45:22 +03:00
Kseniia Sumarokova
947ef81f53
Update ReadWriteBufferFromHTTP.h 2021-12-16 09:14:09 +03:00
mergify[bot]
41ae7c8eb1
Merge branch 'master' into fix_IP_conversion 2021-12-15 19:19:47 +00:00
Amos Bird
9c9f4a3e6c
build fix 2021-12-15 16:55:18 +08:00
mergify[bot]
4300f77532
Merge branch 'master' into fix_IP_conversion 2021-12-14 20:12:34 +00:00
Alexander Tokmakov
d7ad72838c Merge branch 'master' into mvcc_prototype 2021-12-14 23:07:52 +03:00
alesapin
9974f10631 Fix clang tidy 2021-12-14 15:34:20 +03:00
taiyang-li
ca3f7425a4 fix code 2021-12-14 17:37:31 +08:00
taiyang-li
8234d1176f merge master 2021-12-14 10:39:21 +08:00
alexey-milovidov
730c16bd0c
Merge pull request #32649 from vdimir/issue_32504 
Reset pipeline in BlockIO::onFinish
 2021-12-13 21:05:33 +03:00
alexey-milovidov
fd9d40925a
Merge pull request #32640 from ClickHouse/fix-function-file 
Fix surprisingly bad code in function `file`
 2021-12-13 19:34:36 +03:00