thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 11:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
84,585 Commits 9,326 Branches 1,784 Tags 2.1 GiB
7d50bd1eb3
Commit Graph

1477 Commits

Author SHA1 Message Date
Azat Khuzhin
3b3635c6d5 Fix formatting error in logging messages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Amos Bird
e29c77f793 Fix link order issue of WriteBufferFromS3 
(cherry picked from commit f22b09f4fc)
 2022-02-01 10:42:06 +00:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3"" 2022-02-01 13:36:51 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3" 2022-02-01 05:49:40 +03:00
Alexey Milovidov
989a5ad35e More cases for parseDateTimeBestEffort 2022-02-01 03:29:27 +03:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines 
Fix benign race condition for storage HDFS, S3, URL
 2022-01-31 22:41:54 +03:00
alesapin
31753afb7e Fix cancel logic in parallel parsing 2022-01-31 16:38:15 +03:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
kssenii
6b03c577d6 Make downloader's term resticted to one nextImpl 2022-01-29 23:58:53 +03:00
Nikolai Kochetov
1c9f026178 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-28 16:01:42 +03:00
kssenii
c91b86e220 Better 2022-01-26 21:56:08 +03:00
Nikolai Kochetov
a8171269a1 Review fixes. 2022-01-26 17:55:24 +00:00
Nikolai Kochetov
2e88319b72 Fix integration tests. 2022-01-25 18:48:28 +00:00
Nikolai Kochetov
fcc29dbd15 Try to fix integration tests. 2022-01-25 15:26:36 +00:00
Kruglov Pavel
1b19139f74
Merge pull request #31862 from Avogar/fix-lz4 
Try to fix lz4 compression
 2022-01-25 14:34:45 +03:00
Nikolai Kochetov
6c65915241 Fix style 2022-01-24 16:12:23 +00:00
Nikolai Kochetov
506ee8c024 Refactor some code. 2022-01-24 15:55:29 +00:00
Nikolai Kochetov
4afa75e80b Fix deduplication. 2022-01-24 14:43:36 +00:00
Kseniia Sumarokova
2c43e4eb57
Merge branch 'master' into local-cache-for-remote-fs 2022-01-22 22:53:55 +03:00
alexey-milovidov
eb6849f7c7
Merge pull request #33842 from azat/cmake-contrib-fixes 
More cmake external modules cleanups
 2022-01-22 10:34:54 +03:00
Nikolai Kochetov
b3cbb63487 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:41:54 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Azat Khuzhin
2aa7fb28a0 contrib/dragonbox: add ALIAS library 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-21 10:11:22 +03:00
Azat Khuzhin
c07a2c856a Fix bypass virtual dispatch in ZstdDeflatingAppendableWriteBuffer 
clang-tidy [1]:

    "Call to virtual method 'ZstdDeflatingAppendableWriteBuffer::finalizeAfter' during destruction bypasses virtual dispatch"

  [1]: https://s3.amazonaws.com/clickhouse-builds/33840/d021190b8c7ae0f5dc2decb953c471ea9d770981/binary_tidy/build_log.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-20 20:40:04 +03:00
Azat Khuzhin
d021190b8c keeper: fix memory leak in case of compression is used (default) 
In case of compression is used, ZstdDeflatingAppendableWriteBuffer is
used, but it has a leak, since it frees ZSTD_CCtx only if there was
write while this is not correct, since it is created anyway.

This was found with jemalloc profile, keeper-bench and the following
keeper settings:

- force_sync=false
- snapshot_distance=100
- reserved_log_items=0
- rotate_log_storage_interval=100

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-20 19:03:00 +03:00
mergify[bot]
9326f99d78
Merge branch 'master' into fix-lz4 2022-01-20 11:53:51 +00:00
Azat Khuzhin
ca8525a733 Remove unbundled zlib support 2022-01-20 08:47:16 +03:00
kssenii
a566099759 Local cache for remote filesystem 2022-01-19 00:42:11 +03:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl 
RFC: Split headers, move SystemLog into module, more forward declarations
 2022-01-18 17:22:49 +03:00
taiyang-li
66813a3aa9 merge master 2022-01-12 16:56:29 +08:00
Maksim Kita
c1ec85d87a Build with ENABLE_TESTS fix 2022-01-11 12:30:40 +03:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
2cdc5b37e3 Do not try to read pass EOF (to workaround a bug in a kernel) 
For unaligned offset pread() may return EINVAL even if the offset pass
EOF, although it should not, since otherwise there is no abiliity to
rely on read() == 0 is EOF (with pread() loop).

Here is a reproducer for the problem on 4.9.0-12-amd64:

    $ head -c27 /dev/urandom > /tmp/pread.issue
    $ xfs_io
    xfs_io> open -d /tmp/pread.issue
    xfs_io> pread 1000 4096
    pread: Invalid argument

And this is how it should work:

    xfs_io> pread 29 4096
    read 0/4096 bytes at offset 29

Note, here I use interactive mode since we had old xfs_io that does not
allow to execute multiple commands at once, and to avoid EMFILE issue

Here is some history of a patches that affects this behaviour in the
linux kernel:

- the issue had been introduced in
  torvalds/linux@9fe55eea7e v3.14
  ("Fix race when checking i_size on direct i/o read")
- an attempt to fix it had been made in
  torvalds/linux@74cedf9b6c v4.4
  ("direct-io: Fix negative return from dio read beyond eof")
- but this wasn't enough, since alignment check was earlier, so
  eventually fixed in
  torvalds/linux@41b21af388 v5.10
  ("direct-io: defer alignment check until after the EOF check")

Someone may ask why CI does not shows the issue, since:
- it had 4.19 kernel when CI was in yandex
- now it has 5.4 when CI is in AWS
Since both of those kernels does not have the last patch.

But, this bug requires the following conditions to met:
- index_granularity_bytes=0
- min_merge_bytes_to_use_direct_io=1
Which was not covered by CI yet.
 2022-01-04 10:53:17 +03:00
Azat Khuzhin
71a99ab27c Accept real file size in createReadBufferFromFileBase() 
Right now streams relies on correct file size not the number of bytes
that will be read from the stream, to overcome one bug in the linux
kernel that may return EIINVAL for pread() with offset pass the EOF.

v2: Swap read_hint and file_size (since it is easy to miss something)

Before the first argument to readFile()/createReadBufferFromFileBase()
was read_hint, not the file_size, and let's preserve the order, since
it is easy to miss something

This will also fix 02051_read_settings test automatically because now
MergeTreeReaderStream will pass estimated_sum_mark_range_bytes to
read_hint not file_size, previously it cause on of the following errors:
- Attempt to read after EOF w/ O_DIRECT
- and LOGICAL_ERROR while adjusting granulas w/o O_DIRECT

This will also improve zero-length reads guard (via
ReadBufferFromEmptyFile), that had been added in #30190

v3: fix for other storages that wasn't enabled in fast-test
v4: ignore ENOENT/ENOTSUP in readFile
 2022-01-04 10:53:17 +03:00
Azat Khuzhin
e548dae8d9 Pass size of the file to the readers 
v2: s/size/file_size/g
 2022-01-04 10:53:17 +03:00
taiyang-li
1e102bc1b2 merge master 2022-01-01 09:01:06 +08:00
Nikolay Degterinsky
79e15e84d5 Refactor 2021-12-30 04:47:34 +00:00
Kruglov Pavel
489a30859f
Merge pull request #32455 from Avogar/schema-inference 
Automatic schema inference for input formats
 2021-12-29 21:03:48 +03:00
Nikolai Kochetov
68bf38133a Add pool to WriteBufferFromS3 2021-12-29 14:04:21 +00:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
benbiti
9004e98992 parseDateTimeBestEffort support Unix Timestamp with Milliseconds 2021-12-29 13:43:54 +08:00
taiyang-li
9036b18c2f merge master 2021-12-27 15:12:48 +08:00
alexey-milovidov
47d50c3bd1
Merge pull request #33140 from kssenii/azure-tests-fix 
Fix azure blob storage tests failures
 2021-12-25 06:25:26 +03:00
alexey-milovidov
d15da636f6
Update ReadBufferFromAzureBlobStorage.cpp 2021-12-25 05:57:15 +03:00
alexey-milovidov
bcd4df789d
Update WriteBufferFromAzureBlobStorage.cpp 2021-12-25 05:56:15 +03:00
alexey-milovidov
f2075fac64
Update ReadBufferFromAzureBlobStorage.cpp 2021-12-25 05:55:49 +03:00
kssenii
b4420cfa3e Fix 2021-12-24 22:51:36 +03:00
Alexey Milovidov
29d28c531f Move code around to avoid dlsym on Musl 2021-12-24 12:25:27 +03:00
Kruglov Pavel
f38bc9fc38
Merge branch 'master' into fix-lz4 2021-12-23 16:13:26 +03:00
Nikolay Degterinsky
826f7cb0f5
Merge pull request #32240 from benbiti/master 
Fix unix timestamp Millisecond convert to DateTime64, fractional part calc reversed
 2021-12-23 12:11:53 +03:00
Kseniia Sumarokova
f0434f21ce
Merge pull request #33039 from kssenii/fix-azure-exception 
Fix exception in azure write buffer desctructor
 2021-12-23 10:26:40 +03:00
Vitaly Baranov
7660530fcc
Merge pull request #32955 from azat/read-fix 
Fix UB in case of unexpected EOF during filling a set from HTTP query
 2021-12-23 06:41:14 +03:00
Dmitry Novik
d256c3a9ca
Merge pull request #29224 from ClickHouse/fix_IP_conversion 
Merging #27914
 2021-12-22 18:36:13 +03:00
mergify[bot]
04f727e84c
Merge branch 'master' into read-fix 2021-12-22 13:49:04 +00:00
kssenii
65c5334bb0 Safer 2021-12-22 14:22:14 +03:00
kssenii
060d3228e9 Fix 2021-12-22 13:43:21 +03:00
kssenii
d8ed4db6d4 Fix 2021-12-22 11:42:23 +03:00
Kseniia Sumarokova
484a4d62dd
Merge pull request #32835 from kssenii/fising-s3 
Fix optimization with lazy seek for async reads from remote fs
 2021-12-22 09:38:23 +03:00
benbiti
81a7a393c8 fix unix timestamp convert, fractional reversed 2021-12-22 01:32:16 +08:00
mergify[bot]
7da6d8b310
Merge branch 'master' into fix_IP_conversion 2021-12-21 12:16:05 +00:00
kssenii
ebcb66a361 Fix 2021-12-21 10:58:02 +03:00
taiyang-li
2597925724 merge master 2021-12-21 15:55:39 +08:00
Azat Khuzhin
4230f04f72 Fix UB in LimitReadBuffer in case of unexpected EOF 
Fixes the following possible assertions:
- ./src/IO/ReadBuffer.h:58: bool DB::ReadBuffer::next(): Assertion `!hasPendingData()' failed.
- ./src/IO/LimitReadBuffer.cpp:17: virtual bool DB::LimitReadBuffer::nextImpl(): Assertion `position() >= in->position()' failed.

Fixes: 02151_http_s_structure_set_eof
 2021-12-21 08:38:41 +03:00
Nikolai Kochetov
35883e0dae
Merge pull request #32979 from ClickHouse/decrease-some-s3-log-level 
Decrease log level for some s3 messages.
 2021-12-20 20:56:22 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment 
Loops remove postfix increment
 2021-12-20 19:51:07 +03:00
Vitaly Baranov
e61d3eef0c
Merge pull request #32747 from vitlibar/improve-grpc-compression 
Improve gRPC compression support
 2021-12-20 19:44:58 +03:00
Nikolai Kochetov
7dea7b7f76 Decrease log level for some s3 messages. 2021-12-20 18:18:54 +03:00
Maksim Kita
51477adf1b Updated additional cases 2021-12-20 15:55:07 +03:00
Vitaly Baranov
02d6cea857 Lz4DeflatingWriteBuffer now supports changes in the nested buffer between calls. 2021-12-19 16:29:57 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
kssenii
40c266840c Fixes 2021-12-17 18:16:29 +03:00
kssenii
ed42620530 Rename BlobStorage to AzureBlobStorage 2021-12-17 12:24:05 +03:00
taiyang-li
d033fc4c24 merge master and fix conflict 2021-12-17 15:11:21 +08:00
kssenii
ca283a4bb8 Fix optimization with lazy seek 2021-12-17 00:18:28 +03:00
alesapin
c338bee718
Revert "Fix build issue related to azure blob storage" 2021-12-16 14:37:12 +03:00
alesapin
c0cebaea79
Merge pull request #32788 from amosbird/buildfix 
Fix build issue related to azure blob storage
 2021-12-16 13:45:22 +03:00
Kseniia Sumarokova
947ef81f53
Update ReadWriteBufferFromHTTP.h 2021-12-16 09:14:09 +03:00
mergify[bot]
41ae7c8eb1
Merge branch 'master' into fix_IP_conversion 2021-12-15 19:19:47 +00:00
Amos Bird
9c9f4a3e6c
build fix 2021-12-15 16:55:18 +08:00
mergify[bot]
4300f77532
Merge branch 'master' into fix_IP_conversion 2021-12-14 20:12:34 +00:00
alesapin
9974f10631 Fix clang tidy 2021-12-14 15:34:20 +03:00
taiyang-li
ca3f7425a4 fix code 2021-12-14 17:37:31 +08:00
taiyang-li
8234d1176f merge master 2021-12-14 10:39:21 +08:00
alexey-milovidov
730c16bd0c
Merge pull request #32649 from vdimir/issue_32504 
Reset pipeline in BlockIO::onFinish
 2021-12-13 21:05:33 +03:00
alexey-milovidov
fd9d40925a
Merge pull request #32640 from ClickHouse/fix-function-file 
Fix surprisingly bad code in function `file`
 2021-12-13 19:34:36 +03:00
Kseniia Sumarokova
78ed383f15
Merge pull request #31505 from ContentSquare/azure_blob_storage_dependency 
Azure Blob Storage Disks
 2021-12-13 18:28:48 +03:00
vdimir
bda4ecbe90
Catch exceprion in ~ZlibDeflatingWriteBuffer 2021-12-13 12:29:35 +03:00
Alexey Milovidov
71926a3a97 Fix surprisingly bad code in function "file" 2021-12-13 07:57:54 +03:00
李扬
8675086104
Merge branch 'master' into hive_table 2021-12-12 09:01:46 -06:00
Jakub Kuklis
578aec97e9 Removing TODO and NOTE marks used mainly as hooks for review in the PR 2021-12-10 22:04:38 +01:00
Jakub Kuklis
85807adec4 Fixing variable shadow error 2021-12-10 22:04:38 +01:00
Jakub Kuklis
910db6ca57 Code cleanup addressing PR remarks 2021-12-10 22:04:38 +01:00
Jakub Kuklis
6acbec4455 Rebased on master 2021-12-10 22:04:38 +01:00
Jakub Kuklis
0c65cfed1f Addressing first PR remarks 2021-12-10 22:04:37 +01:00
Jakub Kuklis
7c2e116214 Adding max_retries config 2021-12-10 22:04:37 +01:00
Jakub Kuklis
a33cea0525 Refactors after proper rebase on master 2021-12-10 22:04:37 +01:00
Jakub Kuklis
b3728dbbaf Rebase on master 2021-12-10 22:04:37 +01:00
Jakub Kuklis
b8ef47006c Minor refactors before PR 2021-12-10 22:04:37 +01:00
Ryad ZENINE
c4421d6866 Small changes following review 2021-12-10 22:04:37 +01:00
Jakub Kuklis
7f595eedbc Catch Blob Storage exceptions 2021-12-10 22:04:37 +01:00
Jakub Kuklis
9a00acfb72 Fixing read size error plus minor improvements 2021-12-10 22:04:37 +01:00
Jakub Kuklis
fd8ebe0d9f Maximum read 2021-12-10 22:04:37 +01:00
Jakub Kuklis
0d7a4b10c3 Using unique_ptr for Blob Client in Read Buffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
12924b3659 Minor improvements 2021-12-10 22:04:37 +01:00
Jakub Kuklis
fa2261dee6 Changing offset in ReadBuffer to size_t 2021-12-10 22:04:37 +01:00
Jakub Kuklis
33247db3f4 Adding finalize() to WriteBuffer, always using block blob uploads, code cleanup 2021-12-10 22:04:37 +01:00
Jakub Kuklis
bf2476a076 Shared pointer for Azure client and fixing path reference in ReadBuffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
486f267b3b Further code cleanup and error codes improvement 2021-12-10 22:04:37 +01:00
Jakub Kuklis
956348073b ReadBufferFromBlobStorage operating like a proper stream, rather than a single download-read buffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
63ec8d4244 Adding multipart upload to WriteBuffer, retries in ReadBuffer, additional settings for buffers 2021-12-10 22:04:37 +01:00
Jakub Kuklis
dbb98e3636 Code cleanup, leaving some debug information printing under #ifdef VERBOSE_DEBUG_MODE 2021-12-10 22:04:37 +01:00
Jakub Kuklis
8230f8039c Two column MergeTree table with Blob Storage disk work! Fixed buffer in ReadBufferFromBlobStorage to be permanent (class attribute) 2021-12-10 22:04:37 +01:00
Jakub Kuklis
2b4d3665ab Basic inserts and selects work for a MergeTree table with a Blob Disk, based on a single column table 2021-12-10 22:04:37 +01:00
Jakub Kuklis
4b51ddee1b Initial check with write and read works for the Blob Storage Disk 2021-12-10 22:04:37 +01:00
Jakub Kuklis
03f85dee41 Managed to read markers file in SELECT, getting error for CODEC version now 2021-12-10 22:04:37 +01:00
Jakub Kuklis
482a10e62e Uploading to Blob Storage on INSERT seems to do something reasonable 2021-12-10 22:04:37 +01:00
Jakub Kuklis
c43b90c58c Progressing with WriteBufferFromBlobStorage 2021-12-10 22:04:37 +01:00
Jakub Kuklis
ee1a568919 The following query passes without erors: CREATE TABLE ... ENGINE = MergeTree ... storage_policy = 'blob_storage', debug constants present in the code 2021-12-10 22:04:37 +01:00
Jakub Kuklis
0cf338e165 Skeleton for DiskBlobStorage 2021-12-10 22:04:36 +01:00
taiyang-li
02b877c978 move remote buffer from src/IO to src/Storages 2021-12-10 10:27:53 +08:00
Alexey Milovidov
dc3565b3fe Quote more identifiers 2021-12-09 23:51:52 +03:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
taiyang-li
0eef092ded fix building 2021-12-09 16:57:33 +08:00
lgbo
38020a33e4 some refinement 2021-12-08 10:07:04 +08:00
lgbo-ustc
54cc65ede8 fixed code style 2021-12-07 17:12:26 +08:00
lgbo-ustc
5a8c56bd55 fixed erros on creating dir on startup 2021-12-07 16:35:08 +08:00
lgbo-ustc
b6c230606e fixed config error in server startup 2021-12-07 14:46:17 +08:00
Kseniia Sumarokova
926fd568c7
Merge pull request #32113 from FrankChen021/url_http_header 
Set Content-Type in HTTP packets issued from URL engine
 2021-12-07 08:52:36 +03:00
lgbo-ustc
15504a4bfa fixed code style 2021-12-07 12:32:50 +08:00
lgbo-ustc
562a0499a1 fixed code style 2021-12-07 12:29:20 +08:00
lgbo-ustc
fcea033a07 fixed code style 2021-12-07 12:17:47 +08:00
lgbo-ustc
07045bc451 fixed a exception message 2021-12-07 11:18:44 +08:00
lgbo-ustc
0c1733817b update LRUCache set() 2021-12-07 10:57:05 +08:00
lgbo-ustc
d4dcbd59fb fixed code style 2021-12-06 19:59:15 +08:00
lgbo-ustc
dd70209623 fixed code style 2021-12-06 18:19:36 +08:00
lgbo-ustc
5913d67553 fixed code style 2021-12-06 16:51:49 +08:00
lgbo-ustc
ddac2a33bd fixed code style 2021-12-06 16:46:13 +08:00
Frank Chen
59a701ee6e
Fix style 2021-12-06 16:31:47 +08:00
Frank Chen
bf3d3ea25c
Fix style 2021-12-06 16:20:10 +08:00
lgbo-ustc
3c56a5deac refector 
1. Make the lru cache pilicy in RemoteReadBufferCache into Common/UnreleasableLRUCache.h
2. If try to use local cached file fail, the RemoteReadBuffer::create() return the original ReadBuffer
3. Remove LocalFileReader
4. RemoteCacheController does not hold a context now, and the download process starts in RemoteReadBufferCache
   once a new RemoteCacheController is create successly
5. Make an abstract class RemoteFileMetaDataBase for descripting a remote file meta data.
6. Remote file meta data need to  a version, can cache multi-version for a file now
 2021-12-06 16:17:32 +08:00
lgbo-ustc
01940c3f01 coding refinement 
1. make remote file meta data be a abstract class
2. make cache controller download process be started by RemoteReadBufferCache
 2021-12-06 10:30:59 +08:00
avogar
7549619b25 Improve skiping unknown fields with Quoted escaping rule in Template/CustomSeparated formats 2021-12-03 16:25:35 +03:00
taiyang-li
041e3074df Merge branch 'hive_table' of https://github.com/bigo-sg/ClickHouse into bigo_hive_table 2021-12-02 20:23:39 +08:00
mergify[bot]
a1b31018d9
Merge branch 'master' into fix-lz4 2021-12-02 12:00:43 +00:00
Kruglov Pavel
049b2c0c14
Merge pull request #31072 from MaxWk/feature-support-bool-type 
Feature support bool type
 2021-12-02 14:50:29 +03:00
taiyang-li
2f4e7e1d4e merge master 2021-12-02 19:48:21 +08:00
Anton Popov
a79dbb6b6a
Merge pull request #32079 from Avogar/parse-plus-in-float 
Allow plus before Float32/Float64
 2021-12-02 14:03:48 +03:00
lgbo-ustc
1c9d0b86cc fixed code style 2021-12-02 18:19:15 +08:00
lgbo-ustc
c8f536a0a4 Use read/write buffer to read/write files 2021-12-02 18:11:35 +08:00
mergify[bot]
cba9f3f14c
Merge branch 'master' into url_http_header 2021-12-02 09:13:41 +00:00