Commit Graph

1622 Commits

Author SHA1 Message Date
Antonio Andelic
5f8d9885b3 Ignore read bytes 2022-03-14 15:50:30 +00:00
Azat Khuzhin
9f7fae56b0 WriteBufferFromS3: remove unused headers
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-14 12:07:47 +03:00
Antonio Andelic
f5d3a8a31d Polishing 2022-03-11 13:38:19 +00:00
Antonio Andelic
28e9508c4e Improve shared pool and add settings 2022-03-11 08:33:34 +00:00
Antonio Andelic
cce318273b Use a shared IO thread pool 2022-03-11 08:33:34 +00:00
Antonio Andelic
adaaaaf061 Unblock readers before stoping 2022-03-11 08:33:34 +00:00
Antonio Andelic
df0f5e20d0 Refactor 2022-03-11 08:33:34 +00:00
Antonio Andelic
7b526b25a8 Initial implementation with Arena 2022-03-11 08:33:34 +00:00
Antonio Andelic
16c20332cf Fix deadlocks 2022-03-11 08:33:34 +00:00
Antonio Andelic
c6664bbbdf Add seekable ParallelReadBuffer 2022-03-11 08:33:34 +00:00
Antonio Andelic
29c32ed831 Refactor code 2022-03-11 08:33:34 +00:00
Antonio Andelic
0a1a3a230e Add support for parallel http range requests 2022-03-11 08:33:34 +00:00
Antonio Andelic
aad87879cf Small refactor 2022-03-11 08:33:34 +00:00
Kseniia Sumarokova
818459b9f0
Merge pull request #33717 from kssenii/local-cache-for-remote-fs
Local cache for remote filesystem
2022-03-11 07:23:10 +01:00
Rajkumar
137e4b79c1 override needed 2022-03-10 12:46:32 -08:00
Kseniia Sumarokova
5149bd2d29
Merge pull request #35172 from kssenii/add-non-retriable-errors-for-url
Add non-retriable errors check for HTTP read buffer
2022-03-10 20:28:37 +01:00
Rajkumar
0de950c3f8 style correction 2022-03-10 10:29:57 -08:00
Rajkumar
81ad3ca220 syle check 2022-03-10 10:19:03 -08:00
Rajkumar
a17e5efea2 adding destructors 2022-03-10 07:31:06 -08:00
Alexander Tokmakov
061fa6a6f2 Merge branch 'master' into mvcc_prototype 2022-03-10 13:13:04 +01:00
kssenii
050802b99b Add non-retriable errors 2022-03-10 11:38:46 +01:00
lgbo-ustc
fdd423a3da fixed code style 2022-03-10 12:13:19 +08:00
lgbo-ustc
e4883f31b7 update tests
1. fixed code style in src/IO/tests/gtest_hadoop_snappy_decoder.cpp
2. enable tests 01060_avro.sh
2022-03-10 09:46:43 +08:00
lgbo-ustc
8dc8c87fd1 add a test case 2022-03-09 10:03:04 +08:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0 Fix split build 2022-03-08 18:05:55 +01:00
kssenii
f3cba6f270 Remove already fixed crutch 2022-03-08 14:30:21 +01:00
lgbo-ustc
7f89a1bcf3 add some usage test 2022-03-08 20:00:39 +08:00
lgbo-ustc
225b0bd914 fixed bug: call need_more_input repeatly, overwrite the buffer 2022-03-08 17:17:06 +08:00
taiyang-li
b4174b0bef merge master and fix conflicts 2022-03-08 11:39:25 +08:00
kssenii
0737aef472 Fix split build 2022-03-07 22:02:07 +01:00
Alexander Tokmakov
8acfb8d27f Merge branch 'master' into mvcc_prototype 2022-03-07 17:40:15 +01:00
kssenii
eeb0ae9816 Fix detaching query 2022-03-07 17:05:36 +01:00
Anton Popov
0bc57da238 Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-07 14:46:08 +00:00
kssenii
692b247379 Fix some corner cases 2022-03-07 14:31:21 +01:00
kssenii
2dcfe66236 Fix race 2022-03-06 20:33:07 +01:00
Anton Popov
df3b07fe7c Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-03 22:25:28 +00:00
vdimir
d3ee3dee85
Catch exception in ~WriteBufferFromS3 2022-03-03 12:54:24 +00:00
Maksim Kita
b1a956c5f1 clang-tidy check performance-move-const-arg fix 2022-03-02 18:15:27 +00:00
Maksim Kita
1f5837359e clang-tidy check performance-noexcept-move-constructor fix 2022-03-02 18:15:27 +00:00
kssenii
d5952109fb Merge master 2022-03-02 18:15:25 +01:00
HaiBo Li
2ea18b3085
Fix the bug that the toDatetime function overflows (#32898)
* Fixed overflow bug of the function toDatetime/toDate/toDate32/parseDateTimeBestEffort
2022-03-02 10:06:38 +03:00
Anton Popov
2758db5341 add more comments 2022-03-01 19:32:55 +03:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
Alexander Tokmakov
aa6b9a2abc Merge branch 'master' into mvcc_prototype 2022-02-23 23:22:03 +03:00
kssenii
c637385dd0 Merge master 2022-02-22 13:17:51 +01:00
Kseniia Sumarokova
eeea322556
Merge pull request #34629 from amosbird/remotefsimprove
Some refactoring and improvement over async and remote buffer related stuff
2022-02-22 11:36:40 +01:00
kssenii
fad7c042c1 Fix cache with non-async reads 2022-02-21 19:50:27 +01:00
Amos Bird
01652d9892
Add more assertion 2022-02-21 22:42:43 +08:00
kssenii
4543513a5d Add comments 2022-02-21 13:54:03 +01:00
Maksim Kita
883c7c21d1
Merge pull request #34690 from nvartolomei/nv/always-increment
Always update ProfileEvents (even on exceptions)
2022-02-21 00:03:44 +01:00
Maksim Kita
e049f6ada9
Update WriteBufferFromPocoSocket.cpp 2022-02-20 16:59:39 +01:00
Maksim Kita
d44f48fe1f
Update ReadBufferFromPocoSocket.cpp 2022-02-20 16:59:17 +01:00
kssenii
e8b2e47e93 Style 2022-02-19 20:38:23 +01:00
Amos Bird
84d111cde9
Better ReadBuffer asynchronicity 2022-02-20 01:08:21 +08:00
kssenii
acf9a092a9 Review fixes 2022-02-18 19:09:54 +01:00
Nicolae Vartolomei
1778be516b Always update ProfileEvents (even on exceptions) 2022-02-17 16:29:47 +00:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
alesapin
6df2c9c2d8
Merge pull request #34433 from ClickHouse/s3-fix-content-type
Workaround for bug when AWS mistakenly set Content-Type to XML
2022-02-17 13:11:14 +03:00
kssenii
47f94120da Merge master 2022-02-16 14:43:28 +01:00
kssenii
3bd3e51aa0 Fix tests 2022-02-16 14:08:41 +01:00
alesapin
36b1944cb0 Merge branch 'master' into s3-fix-content-type 2022-02-15 12:39:49 +03:00
李扬
f52b67b939
Merge branch 'master' into rocksdb_metacache 2022-02-15 02:16:29 -06:00
Alexander Tokmakov
1e4e569151 Merge branch 'master' into mvcc_prototype 2022-02-15 02:26:47 +03:00
kssenii
bdbbe85f95 Fixes 2022-02-14 19:28:48 +01:00
alesapin
89373155fc Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-13 21:07:54 +03:00
Maksim Kita
cb09aadc72
Merge pull request #34524 from FrankChen021/urlengine
Fix compression support in URL engine
2022-02-13 14:39:36 +01:00
Saad Ur Rahman
03f81c8685
[CH-34239] HTTP User-Agent header (#34330)
* [IO] HTTP User-Agent header

Added HTTP User-Agent Header to HTTP requests.

User-Agent: ClickHouse/VERSION_STRING
Input header vectors could potentially contain User-Agent. If so, do not set another.

* [Tests] HTTP User Agent

Added stateless functional test 02205_HTTP_user_agent.

Co-authored-by: Filatenkov Artur <58165623+FArthur-cmd@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-13 00:40:11 +03:00
李扬
daa27d0bda
Merge branch 'master' into rocksdb_metacache 2022-02-12 07:50:12 -06:00
Azat Khuzhin
cc55c8b982 Fix gtest_archive_reader_and_writer in case of !USE_MINIZIP
Empty arrays/vectors are not supported by ValuesIn:

    /src/ch/clickhouse/src/IO/tests/gtest_archive_reader_and_writer.cpp:341:59: error: no matching function for call to 'ValuesIn'
    INSTANTIATE_TEST_SUITE_P(All, ArchiveReaderAndWriterTest, ::testing::ValuesIn(supported_archive_file_exts));
                                                              ^~~~~~~~~~~~~~~~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:459:43: note: expanded from macro 'INSTANTIATE_TEST_SUITE_P'
        return GTEST_EXPAND_(GTEST_GET_FIRST_(__VA_ARGS__, DUMMY_PARAM_));        \
                                              ^~~~~~~~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:453:38: note: expanded from macro 'GTEST_GET_FIRST_'
    #define GTEST_GET_FIRST_(first, ...) first
                                         ^~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:452:28: note: expanded from macro 'GTEST_EXPAND_'
    #define GTEST_EXPAND_(arg) arg
                               ^~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:301:29: note: candidate template ignored: substitution failure [with T = std::vector<std::string>, N = 0]: zero-length arrays are not permitted in C++
    internal::ParamGenerator<T> ValuesIn(const T (&array)[N]) {
                                ^                         ~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:306:58: note: candidate template ignored: substitution failure [with Container = std::vector<std::string> [0]]: type 'std::vector<std::string> [0]' cannot be used prior to '::' because it has no members
    internal::ParamGenerator<typename Container::value_type> ValuesIn(
                                      ~~~~~~~~~              ^
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:294:1: note: candidate function template not viable: requires 2 arguments, but 1 was provided
    ValuesIn(ForwardIterator begin, ForwardIterator end) {
    ^
    /src/ch/clickhouse/src/IO/tests/gtest_archive_reader_and_writer.cpp:341:59: error: no matching function for call to 'ValuesIn'
    INSTANTIATE_TEST_SUITE_P(All, ArchiveReaderAndWriterTest, ::testing::ValuesIn(supported_archive_file_exts));
                                                              ^~~~~~~~~~~~~~~~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:468:32: note: expanded from macro 'INSTANTIATE_TEST_SUITE_P'
          auto t = std::make_tuple(__VA_ARGS__);                                  \
                                   ^~~~~~~~~~~
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:301:29: note: candidate template ignored: substitution failure [with T = std::vector<std::string>, N = 0]
    internal::ParamGenerator<T> ValuesIn(const T (&array)[N]) {
                                ^
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:306:58: note: candidate template ignored: substitution failure [with Container = std::vector<std::string> [0]]: type 'std::vector<std::string> [0]' cannot be used prior to '::' because it has no members
    internal::ParamGenerator<typename Container::value_type> ValuesIn(
                                      ~~~~~~~~~              ^
    /src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:294:1: note: candidate function template not viable: requires 2 arguments, but 1 was provided
    ValuesIn(ForwardIterator begin, ForwardIterator end) {
    ^
    2 errors generated.
    ninja: build stopped: subcommand failed.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-12 15:01:20 +03:00
Frank Chen
7b7bc8f6c3 Address review comment 2022-02-12 10:46:01 +08:00
Alexander Tokmakov
07e66e690d Merge branch 'master' into mvcc_prototype 2022-02-11 15:53:32 +03:00
mergify[bot]
f099048dd1
Merge branch 'master' into urlengine 2022-02-11 11:03:21 +00:00
Frank Chen
c8cddd7077 Fix compression for URL engine 2022-02-11 17:45:49 +08:00
alesapin
84b5fbf43c Fix buffer HDFS crash 2022-02-10 16:35:50 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
alesapin
b84a171aa7 Fix style 2022-02-09 14:13:53 +03:00
alesapin
437940b29d
Merge pull request #34422 from ClickHouse/add_settings_to_part_size_in_s3
Add settings for multipart upload part size in s3
2022-02-09 12:51:20 +03:00
Kseniia Sumarokova
86956ca08b
Merge pull request #34392 from kssenii/http-buffer-skip-not-found-url-for-globs
Allow to skip not found urls for globs
2022-02-09 09:32:09 +01:00
Alexey Milovidov
734b08177e Fix excessive logging while writing to s3 2022-02-09 07:19:33 +03:00
taiyang-li
d04ccc0489 Merge branch 'master' into rocksdb_metacache 2022-02-09 11:54:10 +08:00
Alexey Milovidov
92fb6bf735 Workaround for bug in AWS 2022-02-09 05:25:22 +03:00
kssenii
e8a8f46864 Fix style check 2022-02-08 21:58:03 +01:00
alesapin
f6e7cea1f8 Add settings for S3 multipart upload parts size 2022-02-08 19:38:04 +03:00
Vitaly Baranov
9a8e45805a
Merge pull request #34226 from vitlibar/add-submodule-minizip
Add submodule minizip
2022-02-08 22:07:34 +07:00
Kruglov Pavel
720310fcf1
Merge branch 'master' into http-buffer-skip-not-found-url-for-globs 2022-02-08 16:56:22 +03:00
kssenii
dc5f035265 Fix 2022-02-08 13:28:10 +01:00
kssenii
3834bdbae0 Fixes 2022-02-08 11:11:18 +01:00
taiyang-li
b6132d490f merge master and solve conflict 2022-02-08 15:24:59 +08:00
kssenii
eba3011ada Fix 2022-02-07 20:40:47 +01:00
Anton Popov
d32d6453d3
Update BufferBase.h 2022-02-07 17:49:53 +03:00
kssenii
2e58733750 Merge master 2022-02-07 15:23:10 +01:00
Anton Popov
1b16db72c3 fix consecutive backward seeks in seekable read buffers 2022-02-07 17:20:26 +03:00
Anton Popov
ae1fc94fb5 add unit test 2022-02-06 20:45:59 +00:00
Anton Popov
10b8684003 fix rare bug in reading of empty arrays 2022-02-06 20:45:59 +00:00
Alexander Tokmakov
45be75b4db Merge branch 'master' into mvcc_prototype 2022-02-06 23:36:08 +03:00
Vitaly Baranov
d9bdbf47b7 ReadBufferFromMemory now can seek backwards after been read up to EOF. 2022-02-05 23:45:20 +07:00
Vitaly Baranov
23fac284ea Add utility classes ZipArchiveReader and ZipArchiveWriter. 2022-02-05 23:45:20 +07:00
Nikolai Kochetov
daeeb6f3a2
Merge pull request #34316 from ClickHouse/probably-fix-data-race-in-WriteBufferFromS3
Probably fix data race in WriteBufferFromS3 destructor.
2022-02-04 21:04:46 +03:00
Nikolai Kochetov
a968c869fd Probably fix data race in WriteBufferFromS3 destructor. 2022-02-04 12:28:14 +00:00
Maksim Kita
074b827cf3
Merge pull request #34306 from ClickHouse/line-as-string-low-performance
Fix terribly low performance of `LineAsString` format
2022-02-04 13:11:27 +01:00
Alexey Milovidov
e20aec6866 Fix linkage 2022-02-04 01:44:43 +03:00
Alexey Milovidov
4a83dbc514 Fix linkage 2022-02-04 00:26:44 +03:00
Alexey Milovidov
7c12f5f37a Fix terribly low performance of LineAsString format 2022-02-04 00:07:31 +03:00
Alexander Tokmakov
ca5f951558 Merge branch 'master' into mvcc_prototype 2022-02-03 18:56:44 +03:00
Nikolai Kochetov
2003a96b58 Probably fix something 2022-02-02 17:48:41 +00:00
Nikolai Kochetov
ea044fc6b2 Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer 2022-02-02 19:40:41 +03:00
kssenii
aeec749173 Fix predownload case part 1 2022-02-02 17:25:25 +03:00
kssenii
5cba11428a Fixes 2022-02-02 00:25:16 +03:00
Sergei Trifonov
68bc456830
Merge pull request #34223 from azat/bump-fmt
Bump fmtlib from 7.0.0 to 8.1.1
2022-02-02 00:03:25 +03:00
alexey-milovidov
2b1d1a9a6f
Merge pull request #34208 from ClickHouse/parse-date-time-best-effort-more-cases
More cases for `parseDateTimeBestEffort`
2022-02-01 16:22:52 +03:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Azat Khuzhin
3b3635c6d5 Fix formatting error in logging messages
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-01 14:30:04 +03:00
Amos Bird
e29c77f793 Fix link order issue of WriteBufferFromS3
(cherry picked from commit f22b09f4fc)
2022-02-01 10:42:06 +00:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3"" 2022-02-01 13:36:51 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3" 2022-02-01 05:49:40 +03:00
Alexey Milovidov
989a5ad35e More cases for parseDateTimeBestEffort 2022-02-01 03:29:27 +03:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines
Fix benign race condition for storage HDFS, S3, URL
2022-01-31 22:41:54 +03:00
alesapin
31753afb7e Fix cancel logic in parallel parsing 2022-01-31 16:38:15 +03:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
kssenii
6b03c577d6 Make downloader's term resticted to one nextImpl 2022-01-29 23:58:53 +03:00
Alexander Tokmakov
fb9b2d5326 Merge branch 'master' into mvcc_prototype 2022-01-28 21:18:36 +03:00
Nikolai Kochetov
1c9f026178 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-28 16:01:42 +03:00
taiyang-li
3de8bde7ce Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-28 09:58:52 +08:00
kssenii
c91b86e220 Better 2022-01-26 21:56:08 +03:00
Nikolai Kochetov
a8171269a1 Review fixes. 2022-01-26 17:55:24 +00:00
Nikolai Kochetov
2e88319b72 Fix integration tests. 2022-01-25 18:48:28 +00:00
Nikolai Kochetov
fcc29dbd15 Try to fix integration tests. 2022-01-25 15:26:36 +00:00
Kruglov Pavel
1b19139f74
Merge pull request #31862 from Avogar/fix-lz4
Try to fix lz4 compression
2022-01-25 14:34:45 +03:00
Nikolai Kochetov
6c65915241 Fix style 2022-01-24 16:12:23 +00:00
Nikolai Kochetov
506ee8c024 Refactor some code. 2022-01-24 15:55:29 +00:00
Nikolai Kochetov
4afa75e80b Fix deduplication. 2022-01-24 14:43:36 +00:00
taiyang-li
73def8b483 merge master and solve conflict 2022-01-24 11:01:43 +08:00
Kseniia Sumarokova
2c43e4eb57
Merge branch 'master' into local-cache-for-remote-fs 2022-01-22 22:53:55 +03:00
alexey-milovidov
eb6849f7c7
Merge pull request #33842 from azat/cmake-contrib-fixes
More cmake external modules cleanups
2022-01-22 10:34:54 +03:00
Nikolai Kochetov
b3cbb63487 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:41:54 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001 Merge branch 'master' into mvcc_prototype 2022-01-21 18:59:01 +03:00
Azat Khuzhin
2aa7fb28a0 contrib/dragonbox: add ALIAS library
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-21 10:11:22 +03:00
Azat Khuzhin
c07a2c856a Fix bypass virtual dispatch in ZstdDeflatingAppendableWriteBuffer
clang-tidy [1]:

    "Call to virtual method 'ZstdDeflatingAppendableWriteBuffer::finalizeAfter' during destruction bypasses virtual dispatch"

  [1]: https://s3.amazonaws.com/clickhouse-builds/33840/d021190b8c7ae0f5dc2decb953c471ea9d770981/binary_tidy/build_log.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-20 20:40:04 +03:00
Azat Khuzhin
d021190b8c keeper: fix memory leak in case of compression is used (default)
In case of compression is used, ZstdDeflatingAppendableWriteBuffer is
used, but it has a leak, since it frees ZSTD_CCtx only if there was
write while this is not correct, since it is created anyway.

This was found with jemalloc profile, keeper-bench and the following
keeper settings:

- force_sync=false
- snapshot_distance=100
- reserved_log_items=0
- rotate_log_storage_interval=100

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-20 19:03:00 +03:00
Alexander Tokmakov
0ea0a8ccc7 Merge branch 'master' into mvcc_prototype 2022-01-20 17:05:55 +03:00
mergify[bot]
9326f99d78
Merge branch 'master' into fix-lz4 2022-01-20 11:53:51 +00:00
Azat Khuzhin
ca8525a733 Remove unbundled zlib support 2022-01-20 08:47:16 +03:00
Alexander Tokmakov
e9a5a64a71 Merge branch 'master' into mvcc_prototype 2022-01-19 21:41:23 +03:00
kssenii
a566099759 Local cache for remote filesystem 2022-01-19 00:42:11 +03:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl
RFC: Split headers, move SystemLog into module, more forward declarations
2022-01-18 17:22:49 +03:00
Alexander Tokmakov
4dd216ce8f Merge branch 'master' into mvcc_prototype 2022-01-14 17:11:32 +03:00
taiyang-li
66813a3aa9 merge master 2022-01-12 16:56:29 +08:00