Antonio Andelic
5f8d9885b3
Ignore read bytes
2022-03-14 15:50:30 +00:00
Azat Khuzhin
9f7fae56b0
WriteBufferFromS3: remove unused headers
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-14 12:07:47 +03:00
Antonio Andelic
f5d3a8a31d
Polishing
2022-03-11 13:38:19 +00:00
Antonio Andelic
28e9508c4e
Improve shared pool and add settings
2022-03-11 08:33:34 +00:00
Antonio Andelic
cce318273b
Use a shared IO thread pool
2022-03-11 08:33:34 +00:00
Antonio Andelic
adaaaaf061
Unblock readers before stoping
2022-03-11 08:33:34 +00:00
Antonio Andelic
df0f5e20d0
Refactor
2022-03-11 08:33:34 +00:00
Antonio Andelic
7b526b25a8
Initial implementation with Arena
2022-03-11 08:33:34 +00:00
Antonio Andelic
16c20332cf
Fix deadlocks
2022-03-11 08:33:34 +00:00
Antonio Andelic
c6664bbbdf
Add seekable ParallelReadBuffer
2022-03-11 08:33:34 +00:00
Antonio Andelic
29c32ed831
Refactor code
2022-03-11 08:33:34 +00:00
Antonio Andelic
0a1a3a230e
Add support for parallel http range requests
2022-03-11 08:33:34 +00:00
Antonio Andelic
aad87879cf
Small refactor
2022-03-11 08:33:34 +00:00
Kseniia Sumarokova
818459b9f0
Merge pull request #33717 from kssenii/local-cache-for-remote-fs
...
Local cache for remote filesystem
2022-03-11 07:23:10 +01:00
Rajkumar
137e4b79c1
override needed
2022-03-10 12:46:32 -08:00
Kseniia Sumarokova
5149bd2d29
Merge pull request #35172 from kssenii/add-non-retriable-errors-for-url
...
Add non-retriable errors check for HTTP read buffer
2022-03-10 20:28:37 +01:00
Rajkumar
0de950c3f8
style correction
2022-03-10 10:29:57 -08:00
Rajkumar
81ad3ca220
syle check
2022-03-10 10:19:03 -08:00
Rajkumar
a17e5efea2
adding destructors
2022-03-10 07:31:06 -08:00
Alexander Tokmakov
061fa6a6f2
Merge branch 'master' into mvcc_prototype
2022-03-10 13:13:04 +01:00
kssenii
050802b99b
Add non-retriable errors
2022-03-10 11:38:46 +01:00
lgbo-ustc
fdd423a3da
fixed code style
2022-03-10 12:13:19 +08:00
lgbo-ustc
e4883f31b7
update tests
...
1. fixed code style in src/IO/tests/gtest_hadoop_snappy_decoder.cpp
2. enable tests 01060_avro.sh
2022-03-10 09:46:43 +08:00
lgbo-ustc
8dc8c87fd1
add a test case
2022-03-09 10:03:04 +08:00
kssenii
5260822964
Merge master
2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0
Fix split build
2022-03-08 18:05:55 +01:00
kssenii
f3cba6f270
Remove already fixed crutch
2022-03-08 14:30:21 +01:00
lgbo-ustc
7f89a1bcf3
add some usage test
2022-03-08 20:00:39 +08:00
lgbo-ustc
225b0bd914
fixed bug: call need_more_input repeatly, overwrite the buffer
2022-03-08 17:17:06 +08:00
taiyang-li
b4174b0bef
merge master and fix conflicts
2022-03-08 11:39:25 +08:00
kssenii
0737aef472
Fix split build
2022-03-07 22:02:07 +01:00
Alexander Tokmakov
8acfb8d27f
Merge branch 'master' into mvcc_prototype
2022-03-07 17:40:15 +01:00
kssenii
eeb0ae9816
Fix detaching query
2022-03-07 17:05:36 +01:00
Anton Popov
0bc57da238
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-07 14:46:08 +00:00
kssenii
692b247379
Fix some corner cases
2022-03-07 14:31:21 +01:00
kssenii
2dcfe66236
Fix race
2022-03-06 20:33:07 +01:00
Anton Popov
df3b07fe7c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-03 22:25:28 +00:00
vdimir
d3ee3dee85
Catch exception in ~WriteBufferFromS3
2022-03-03 12:54:24 +00:00
Maksim Kita
b1a956c5f1
clang-tidy check performance-move-const-arg fix
2022-03-02 18:15:27 +00:00
Maksim Kita
1f5837359e
clang-tidy check performance-noexcept-move-constructor fix
2022-03-02 18:15:27 +00:00
kssenii
d5952109fb
Merge master
2022-03-02 18:15:25 +01:00
HaiBo Li
2ea18b3085
Fix the bug that the toDatetime function overflows ( #32898 )
...
* Fixed overflow bug of the function toDatetime/toDate/toDate32/parseDateTimeBestEffort
2022-03-02 10:06:38 +03:00
Anton Popov
2758db5341
add more comments
2022-03-01 19:32:55 +03:00
Anton Popov
fcdebea925
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-25 13:41:30 +03:00
Alexander Tokmakov
aa6b9a2abc
Merge branch 'master' into mvcc_prototype
2022-02-23 23:22:03 +03:00
kssenii
c637385dd0
Merge master
2022-02-22 13:17:51 +01:00
Kseniia Sumarokova
eeea322556
Merge pull request #34629 from amosbird/remotefsimprove
...
Some refactoring and improvement over async and remote buffer related stuff
2022-02-22 11:36:40 +01:00
kssenii
fad7c042c1
Fix cache with non-async reads
2022-02-21 19:50:27 +01:00
Amos Bird
01652d9892
Add more assertion
2022-02-21 22:42:43 +08:00
kssenii
4543513a5d
Add comments
2022-02-21 13:54:03 +01:00
Maksim Kita
883c7c21d1
Merge pull request #34690 from nvartolomei/nv/always-increment
...
Always update ProfileEvents (even on exceptions)
2022-02-21 00:03:44 +01:00
Maksim Kita
e049f6ada9
Update WriteBufferFromPocoSocket.cpp
2022-02-20 16:59:39 +01:00
Maksim Kita
d44f48fe1f
Update ReadBufferFromPocoSocket.cpp
2022-02-20 16:59:17 +01:00
kssenii
e8b2e47e93
Style
2022-02-19 20:38:23 +01:00
Amos Bird
84d111cde9
Better ReadBuffer asynchronicity
2022-02-20 01:08:21 +08:00
kssenii
acf9a092a9
Review fixes
2022-02-18 19:09:54 +01:00
Nicolae Vartolomei
1778be516b
Always update ProfileEvents (even on exceptions)
2022-02-17 16:29:47 +00:00
Alexander Tokmakov
dae044f86b
Merge branch 'master' into mvcc_prototype
2022-02-17 13:49:37 +03:00
alesapin
6df2c9c2d8
Merge pull request #34433 from ClickHouse/s3-fix-content-type
...
Workaround for bug when AWS mistakenly set Content-Type to XML
2022-02-17 13:11:14 +03:00
kssenii
47f94120da
Merge master
2022-02-16 14:43:28 +01:00
kssenii
3bd3e51aa0
Fix tests
2022-02-16 14:08:41 +01:00
alesapin
36b1944cb0
Merge branch 'master' into s3-fix-content-type
2022-02-15 12:39:49 +03:00
李扬
f52b67b939
Merge branch 'master' into rocksdb_metacache
2022-02-15 02:16:29 -06:00
Alexander Tokmakov
1e4e569151
Merge branch 'master' into mvcc_prototype
2022-02-15 02:26:47 +03:00
kssenii
bdbbe85f95
Fixes
2022-02-14 19:28:48 +01:00
alesapin
89373155fc
Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3
2022-02-13 21:07:54 +03:00
Maksim Kita
cb09aadc72
Merge pull request #34524 from FrankChen021/urlengine
...
Fix compression support in URL engine
2022-02-13 14:39:36 +01:00
Saad Ur Rahman
03f81c8685
[CH-34239] HTTP User-Agent header ( #34330 )
...
* [IO] HTTP User-Agent header
Added HTTP User-Agent Header to HTTP requests.
User-Agent: ClickHouse/VERSION_STRING
Input header vectors could potentially contain User-Agent. If so, do not set another.
* [Tests] HTTP User Agent
Added stateless functional test 02205_HTTP_user_agent.
Co-authored-by: Filatenkov Artur <58165623+FArthur-cmd@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-02-13 00:40:11 +03:00
李扬
daa27d0bda
Merge branch 'master' into rocksdb_metacache
2022-02-12 07:50:12 -06:00
Azat Khuzhin
cc55c8b982
Fix gtest_archive_reader_and_writer in case of !USE_MINIZIP
...
Empty arrays/vectors are not supported by ValuesIn:
/src/ch/clickhouse/src/IO/tests/gtest_archive_reader_and_writer.cpp:341:59: error: no matching function for call to 'ValuesIn'
INSTANTIATE_TEST_SUITE_P(All, ArchiveReaderAndWriterTest, ::testing::ValuesIn(supported_archive_file_exts));
^~~~~~~~~~~~~~~~~~~
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:459:43: note: expanded from macro 'INSTANTIATE_TEST_SUITE_P'
return GTEST_EXPAND_(GTEST_GET_FIRST_(__VA_ARGS__, DUMMY_PARAM_)); \
^~~~~~~~~~~
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:453:38: note: expanded from macro 'GTEST_GET_FIRST_'
#define GTEST_GET_FIRST_(first, ...) first
^~~~~
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:452:28: note: expanded from macro 'GTEST_EXPAND_'
#define GTEST_EXPAND_(arg) arg
^~~
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:301:29: note: candidate template ignored: substitution failure [with T = std::vector<std::string>, N = 0]: zero-length arrays are not permitted in C++
internal::ParamGenerator<T> ValuesIn(const T (&array)[N]) {
^ ~
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:306:58: note: candidate template ignored: substitution failure [with Container = std::vector<std::string> [0]]: type 'std::vector<std::string> [0]' cannot be used prior to '::' because it has no members
internal::ParamGenerator<typename Container::value_type> ValuesIn(
~~~~~~~~~ ^
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:294:1: note: candidate function template not viable: requires 2 arguments, but 1 was provided
ValuesIn(ForwardIterator begin, ForwardIterator end) {
^
/src/ch/clickhouse/src/IO/tests/gtest_archive_reader_and_writer.cpp:341:59: error: no matching function for call to 'ValuesIn'
INSTANTIATE_TEST_SUITE_P(All, ArchiveReaderAndWriterTest, ::testing::ValuesIn(supported_archive_file_exts));
^~~~~~~~~~~~~~~~~~~
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:468:32: note: expanded from macro 'INSTANTIATE_TEST_SUITE_P'
auto t = std::make_tuple(__VA_ARGS__); \
^~~~~~~~~~~
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:301:29: note: candidate template ignored: substitution failure [with T = std::vector<std::string>, N = 0]
internal::ParamGenerator<T> ValuesIn(const T (&array)[N]) {
^
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:306:58: note: candidate template ignored: substitution failure [with Container = std::vector<std::string> [0]]: type 'std::vector<std::string> [0]' cannot be used prior to '::' because it has no members
internal::ParamGenerator<typename Container::value_type> ValuesIn(
~~~~~~~~~ ^
/src/ch/clickhouse/contrib/googletest/googletest/include/gtest/gtest-param-test.h:294:1: note: candidate function template not viable: requires 2 arguments, but 1 was provided
ValuesIn(ForwardIterator begin, ForwardIterator end) {
^
2 errors generated.
ninja: build stopped: subcommand failed.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-12 15:01:20 +03:00
Frank Chen
7b7bc8f6c3
Address review comment
2022-02-12 10:46:01 +08:00
Alexander Tokmakov
07e66e690d
Merge branch 'master' into mvcc_prototype
2022-02-11 15:53:32 +03:00
mergify[bot]
f099048dd1
Merge branch 'master' into urlengine
2022-02-11 11:03:21 +00:00
Frank Chen
c8cddd7077
Fix compression for URL engine
2022-02-11 17:45:49 +08:00
alesapin
84b5fbf43c
Fix buffer HDFS crash
2022-02-10 16:35:50 +03:00
Anton Popov
18940b8637
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-09 23:38:38 +03:00
alesapin
b84a171aa7
Fix style
2022-02-09 14:13:53 +03:00
alesapin
437940b29d
Merge pull request #34422 from ClickHouse/add_settings_to_part_size_in_s3
...
Add settings for multipart upload part size in s3
2022-02-09 12:51:20 +03:00
Kseniia Sumarokova
86956ca08b
Merge pull request #34392 from kssenii/http-buffer-skip-not-found-url-for-globs
...
Allow to skip not found urls for globs
2022-02-09 09:32:09 +01:00
Alexey Milovidov
734b08177e
Fix excessive logging while writing to s3
2022-02-09 07:19:33 +03:00
taiyang-li
d04ccc0489
Merge branch 'master' into rocksdb_metacache
2022-02-09 11:54:10 +08:00
Alexey Milovidov
92fb6bf735
Workaround for bug in AWS
2022-02-09 05:25:22 +03:00
kssenii
e8a8f46864
Fix style check
2022-02-08 21:58:03 +01:00
alesapin
f6e7cea1f8
Add settings for S3 multipart upload parts size
2022-02-08 19:38:04 +03:00
Vitaly Baranov
9a8e45805a
Merge pull request #34226 from vitlibar/add-submodule-minizip
...
Add submodule minizip
2022-02-08 22:07:34 +07:00
Kruglov Pavel
720310fcf1
Merge branch 'master' into http-buffer-skip-not-found-url-for-globs
2022-02-08 16:56:22 +03:00
kssenii
dc5f035265
Fix
2022-02-08 13:28:10 +01:00
kssenii
3834bdbae0
Fixes
2022-02-08 11:11:18 +01:00
taiyang-li
b6132d490f
merge master and solve conflict
2022-02-08 15:24:59 +08:00
kssenii
eba3011ada
Fix
2022-02-07 20:40:47 +01:00
Anton Popov
d32d6453d3
Update BufferBase.h
2022-02-07 17:49:53 +03:00
kssenii
2e58733750
Merge master
2022-02-07 15:23:10 +01:00
Anton Popov
1b16db72c3
fix consecutive backward seeks in seekable read buffers
2022-02-07 17:20:26 +03:00
Anton Popov
ae1fc94fb5
add unit test
2022-02-06 20:45:59 +00:00
Anton Popov
10b8684003
fix rare bug in reading of empty arrays
2022-02-06 20:45:59 +00:00
Alexander Tokmakov
45be75b4db
Merge branch 'master' into mvcc_prototype
2022-02-06 23:36:08 +03:00
Vitaly Baranov
d9bdbf47b7
ReadBufferFromMemory now can seek backwards after been read up to EOF.
2022-02-05 23:45:20 +07:00
Vitaly Baranov
23fac284ea
Add utility classes ZipArchiveReader and ZipArchiveWriter.
2022-02-05 23:45:20 +07:00
Nikolai Kochetov
daeeb6f3a2
Merge pull request #34316 from ClickHouse/probably-fix-data-race-in-WriteBufferFromS3
...
Probably fix data race in WriteBufferFromS3 destructor.
2022-02-04 21:04:46 +03:00
Nikolai Kochetov
a968c869fd
Probably fix data race in WriteBufferFromS3 destructor.
2022-02-04 12:28:14 +00:00
Maksim Kita
074b827cf3
Merge pull request #34306 from ClickHouse/line-as-string-low-performance
...
Fix terribly low performance of `LineAsString` format
2022-02-04 13:11:27 +01:00
Alexey Milovidov
e20aec6866
Fix linkage
2022-02-04 01:44:43 +03:00
Alexey Milovidov
4a83dbc514
Fix linkage
2022-02-04 00:26:44 +03:00
Alexey Milovidov
7c12f5f37a
Fix terribly low performance of LineAsString
format
2022-02-04 00:07:31 +03:00
Alexander Tokmakov
ca5f951558
Merge branch 'master' into mvcc_prototype
2022-02-03 18:56:44 +03:00
Nikolai Kochetov
2003a96b58
Probably fix something
2022-02-02 17:48:41 +00:00
Nikolai Kochetov
ea044fc6b2
Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer
2022-02-02 19:40:41 +03:00
kssenii
aeec749173
Fix predownload case part 1
2022-02-02 17:25:25 +03:00
kssenii
5cba11428a
Fixes
2022-02-02 00:25:16 +03:00
Sergei Trifonov
68bc456830
Merge pull request #34223 from azat/bump-fmt
...
Bump fmtlib from 7.0.0 to 8.1.1
2022-02-02 00:03:25 +03:00
alexey-milovidov
2b1d1a9a6f
Merge pull request #34208 from ClickHouse/parse-date-time-best-effort-more-cases
...
More cases for `parseDateTimeBestEffort`
2022-02-01 16:22:52 +03:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
Azat Khuzhin
3b3635c6d5
Fix formatting error in logging messages
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-01 14:30:04 +03:00
Amos Bird
e29c77f793
Fix link order issue of WriteBufferFromS3
...
(cherry picked from commit f22b09f4fc
)
2022-02-01 10:42:06 +00:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3""
2022-02-01 13:36:51 +03:00
Alexander Tokmakov
2e4ae37d98
Merge branch 'master' into mvcc_prototype
2022-02-01 13:20:03 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3"
2022-02-01 05:49:40 +03:00
Alexey Milovidov
989a5ad35e
More cases for parseDateTimeBestEffort
2022-02-01 03:29:27 +03:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines
...
Fix benign race condition for storage HDFS, S3, URL
2022-01-31 22:41:54 +03:00
alesapin
31753afb7e
Fix cancel logic in parallel parsing
2022-01-31 16:38:15 +03:00
Anton Popov
78b9f15abb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-30 03:24:37 +03:00
kssenii
6b03c577d6
Make downloader's term resticted to one nextImpl
2022-01-29 23:58:53 +03:00
Alexander Tokmakov
fb9b2d5326
Merge branch 'master' into mvcc_prototype
2022-01-28 21:18:36 +03:00
Nikolai Kochetov
1c9f026178
Merge branch 'master' into add-pool-to-s3-write-buffer
2022-01-28 16:01:42 +03:00
taiyang-li
3de8bde7ce
Merge remote-tracking branch 'origin/master' into rocksdb_metacache
2022-01-28 09:58:52 +08:00
kssenii
c91b86e220
Better
2022-01-26 21:56:08 +03:00
Nikolai Kochetov
a8171269a1
Review fixes.
2022-01-26 17:55:24 +00:00
Nikolai Kochetov
2e88319b72
Fix integration tests.
2022-01-25 18:48:28 +00:00
Nikolai Kochetov
fcc29dbd15
Try to fix integration tests.
2022-01-25 15:26:36 +00:00
Kruglov Pavel
1b19139f74
Merge pull request #31862 from Avogar/fix-lz4
...
Try to fix lz4 compression
2022-01-25 14:34:45 +03:00
Nikolai Kochetov
6c65915241
Fix style
2022-01-24 16:12:23 +00:00
Nikolai Kochetov
506ee8c024
Refactor some code.
2022-01-24 15:55:29 +00:00
Nikolai Kochetov
4afa75e80b
Fix deduplication.
2022-01-24 14:43:36 +00:00
taiyang-li
73def8b483
merge master and solve conflict
2022-01-24 11:01:43 +08:00
Kseniia Sumarokova
2c43e4eb57
Merge branch 'master' into local-cache-for-remote-fs
2022-01-22 22:53:55 +03:00
alexey-milovidov
eb6849f7c7
Merge pull request #33842 from azat/cmake-contrib-fixes
...
More cmake external modules cleanups
2022-01-22 10:34:54 +03:00
Nikolai Kochetov
b3cbb63487
Merge branch 'master' into add-pool-to-s3-write-buffer
2022-01-21 21:41:54 +03:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001
Merge branch 'master' into mvcc_prototype
2022-01-21 18:59:01 +03:00
Azat Khuzhin
2aa7fb28a0
contrib/dragonbox: add ALIAS library
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-21 10:11:22 +03:00
Azat Khuzhin
c07a2c856a
Fix bypass virtual dispatch in ZstdDeflatingAppendableWriteBuffer
...
clang-tidy [1]:
"Call to virtual method 'ZstdDeflatingAppendableWriteBuffer::finalizeAfter' during destruction bypasses virtual dispatch"
[1]: https://s3.amazonaws.com/clickhouse-builds/33840/d021190b8c7ae0f5dc2decb953c471ea9d770981/binary_tidy/build_log.log
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-20 20:40:04 +03:00
Azat Khuzhin
d021190b8c
keeper: fix memory leak in case of compression is used (default)
...
In case of compression is used, ZstdDeflatingAppendableWriteBuffer is
used, but it has a leak, since it frees ZSTD_CCtx only if there was
write while this is not correct, since it is created anyway.
This was found with jemalloc profile, keeper-bench and the following
keeper settings:
- force_sync=false
- snapshot_distance=100
- reserved_log_items=0
- rotate_log_storage_interval=100
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-20 19:03:00 +03:00
Alexander Tokmakov
0ea0a8ccc7
Merge branch 'master' into mvcc_prototype
2022-01-20 17:05:55 +03:00
mergify[bot]
9326f99d78
Merge branch 'master' into fix-lz4
2022-01-20 11:53:51 +00:00
Azat Khuzhin
ca8525a733
Remove unbundled zlib support
2022-01-20 08:47:16 +03:00
Alexander Tokmakov
e9a5a64a71
Merge branch 'master' into mvcc_prototype
2022-01-19 21:41:23 +03:00
kssenii
a566099759
Local cache for remote filesystem
2022-01-19 00:42:11 +03:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl
...
RFC: Split headers, move SystemLog into module, more forward declarations
2022-01-18 17:22:49 +03:00
Alexander Tokmakov
4dd216ce8f
Merge branch 'master' into mvcc_prototype
2022-01-14 17:11:32 +03:00
taiyang-li
66813a3aa9
merge master
2022-01-12 16:56:29 +08:00