thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 03:42:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
86,455 Commits 9,315 Branches 1,783 Tags 2.1 GiB
cd83fd5f8a
Commit Graph

1622 Commits

Author SHA1 Message Date
Maksim Kita
074b827cf3
Merge pull request #34306 from ClickHouse/line-as-string-low-performance 
Fix terribly low performance of `LineAsString` format
 2022-02-04 13:11:27 +01:00
Alexey Milovidov
e20aec6866 Fix linkage 2022-02-04 01:44:43 +03:00
Alexey Milovidov
4a83dbc514 Fix linkage 2022-02-04 00:26:44 +03:00
Alexey Milovidov
7c12f5f37a Fix terribly low performance of LineAsString format 2022-02-04 00:07:31 +03:00
Alexander Tokmakov
ca5f951558 Merge branch 'master' into mvcc_prototype 2022-02-03 18:56:44 +03:00
Nikolai Kochetov
2003a96b58 Probably fix something 2022-02-02 17:48:41 +00:00
Nikolai Kochetov
ea044fc6b2 Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer 2022-02-02 19:40:41 +03:00
kssenii
aeec749173 Fix predownload case part 1 2022-02-02 17:25:25 +03:00
kssenii
5cba11428a Fixes 2022-02-02 00:25:16 +03:00
Sergei Trifonov
68bc456830
Merge pull request #34223 from azat/bump-fmt 
Bump fmtlib from 7.0.0 to 8.1.1
 2022-02-02 00:03:25 +03:00
alexey-milovidov
2b1d1a9a6f
Merge pull request #34208 from ClickHouse/parse-date-time-best-effort-more-cases 
More cases for `parseDateTimeBestEffort`
 2022-02-01 16:22:52 +03:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Azat Khuzhin
3b3635c6d5 Fix formatting error in logging messages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Amos Bird
e29c77f793 Fix link order issue of WriteBufferFromS3 
(cherry picked from commit f22b09f4fc)
 2022-02-01 10:42:06 +00:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3"" 2022-02-01 13:36:51 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3" 2022-02-01 05:49:40 +03:00
Alexey Milovidov
989a5ad35e More cases for parseDateTimeBestEffort 2022-02-01 03:29:27 +03:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines 
Fix benign race condition for storage HDFS, S3, URL
 2022-01-31 22:41:54 +03:00
alesapin
31753afb7e Fix cancel logic in parallel parsing 2022-01-31 16:38:15 +03:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
kssenii
6b03c577d6 Make downloader's term resticted to one nextImpl 2022-01-29 23:58:53 +03:00
Alexander Tokmakov
fb9b2d5326 Merge branch 'master' into mvcc_prototype 2022-01-28 21:18:36 +03:00
Nikolai Kochetov
1c9f026178 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-28 16:01:42 +03:00
taiyang-li
3de8bde7ce Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-28 09:58:52 +08:00
kssenii
c91b86e220 Better 2022-01-26 21:56:08 +03:00
Nikolai Kochetov
a8171269a1 Review fixes. 2022-01-26 17:55:24 +00:00
Nikolai Kochetov
2e88319b72 Fix integration tests. 2022-01-25 18:48:28 +00:00
Nikolai Kochetov
fcc29dbd15 Try to fix integration tests. 2022-01-25 15:26:36 +00:00
Kruglov Pavel
1b19139f74
Merge pull request #31862 from Avogar/fix-lz4 
Try to fix lz4 compression
 2022-01-25 14:34:45 +03:00
Nikolai Kochetov
6c65915241 Fix style 2022-01-24 16:12:23 +00:00
Nikolai Kochetov
506ee8c024 Refactor some code. 2022-01-24 15:55:29 +00:00
Nikolai Kochetov
4afa75e80b Fix deduplication. 2022-01-24 14:43:36 +00:00
taiyang-li
73def8b483 merge master and solve conflict 2022-01-24 11:01:43 +08:00
Kseniia Sumarokova
2c43e4eb57
Merge branch 'master' into local-cache-for-remote-fs 2022-01-22 22:53:55 +03:00
alexey-milovidov
eb6849f7c7
Merge pull request #33842 from azat/cmake-contrib-fixes 
More cmake external modules cleanups
 2022-01-22 10:34:54 +03:00
Nikolai Kochetov
b3cbb63487 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:41:54 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001 Merge branch 'master' into mvcc_prototype 2022-01-21 18:59:01 +03:00
Azat Khuzhin
2aa7fb28a0 contrib/dragonbox: add ALIAS library 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-21 10:11:22 +03:00
Azat Khuzhin
c07a2c856a Fix bypass virtual dispatch in ZstdDeflatingAppendableWriteBuffer 
clang-tidy [1]:

    "Call to virtual method 'ZstdDeflatingAppendableWriteBuffer::finalizeAfter' during destruction bypasses virtual dispatch"

  [1]: https://s3.amazonaws.com/clickhouse-builds/33840/d021190b8c7ae0f5dc2decb953c471ea9d770981/binary_tidy/build_log.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-20 20:40:04 +03:00
Azat Khuzhin
d021190b8c keeper: fix memory leak in case of compression is used (default) 
In case of compression is used, ZstdDeflatingAppendableWriteBuffer is
used, but it has a leak, since it frees ZSTD_CCtx only if there was
write while this is not correct, since it is created anyway.

This was found with jemalloc profile, keeper-bench and the following
keeper settings:

- force_sync=false
- snapshot_distance=100
- reserved_log_items=0
- rotate_log_storage_interval=100

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-20 19:03:00 +03:00
Alexander Tokmakov
0ea0a8ccc7 Merge branch 'master' into mvcc_prototype 2022-01-20 17:05:55 +03:00
mergify[bot]
9326f99d78
Merge branch 'master' into fix-lz4 2022-01-20 11:53:51 +00:00
Azat Khuzhin
ca8525a733 Remove unbundled zlib support 2022-01-20 08:47:16 +03:00
Alexander Tokmakov
e9a5a64a71 Merge branch 'master' into mvcc_prototype 2022-01-19 21:41:23 +03:00
kssenii
a566099759 Local cache for remote filesystem 2022-01-19 00:42:11 +03:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl 
RFC: Split headers, move SystemLog into module, more forward declarations
 2022-01-18 17:22:49 +03:00
Alexander Tokmakov
4dd216ce8f Merge branch 'master' into mvcc_prototype 2022-01-14 17:11:32 +03:00
taiyang-li
66813a3aa9 merge master 2022-01-12 16:56:29 +08:00
Maksim Kita
c1ec85d87a Build with ENABLE_TESTS fix 2022-01-11 12:30:40 +03:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Alexander Tokmakov
158fbaaa29 fix 2022-01-10 20:07:01 +03:00
Alexander Tokmakov
bf2b6c2c37 Merge branch 'master' into mvcc_prototype 2022-01-10 19:53:50 +03:00
taiyang-li
83d064c24a fix unit test and build error 2022-01-06 11:41:24 +08:00
taiyang-li
2bb7ec8f72 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-06 10:44:17 +08:00
taiyang-li
c0a9c2b916 refactor metadatacache to reduce using of USE_ROCKSDB 2022-01-05 19:51:50 +08:00
Azat Khuzhin
2cdc5b37e3 Do not try to read pass EOF (to workaround a bug in a kernel) 
For unaligned offset pread() may return EINVAL even if the offset pass
EOF, although it should not, since otherwise there is no abiliity to
rely on read() == 0 is EOF (with pread() loop).

Here is a reproducer for the problem on 4.9.0-12-amd64:

    $ head -c27 /dev/urandom > /tmp/pread.issue
    $ xfs_io
    xfs_io> open -d /tmp/pread.issue
    xfs_io> pread 1000 4096
    pread: Invalid argument

And this is how it should work:

    xfs_io> pread 29 4096
    read 0/4096 bytes at offset 29

Note, here I use interactive mode since we had old xfs_io that does not
allow to execute multiple commands at once, and to avoid EMFILE issue

Here is some history of a patches that affects this behaviour in the
linux kernel:

- the issue had been introduced in
  torvalds/linux@9fe55eea7e v3.14
  ("Fix race when checking i_size on direct i/o read")
- an attempt to fix it had been made in
  torvalds/linux@74cedf9b6c v4.4
  ("direct-io: Fix negative return from dio read beyond eof")
- but this wasn't enough, since alignment check was earlier, so
  eventually fixed in
  torvalds/linux@41b21af388 v5.10
  ("direct-io: defer alignment check until after the EOF check")

Someone may ask why CI does not shows the issue, since:
- it had 4.19 kernel when CI was in yandex
- now it has 5.4 when CI is in AWS
Since both of those kernels does not have the last patch.

But, this bug requires the following conditions to met:
- index_granularity_bytes=0
- min_merge_bytes_to_use_direct_io=1
Which was not covered by CI yet.
 2022-01-04 10:53:17 +03:00
Azat Khuzhin
71a99ab27c Accept real file size in createReadBufferFromFileBase() 
Right now streams relies on correct file size not the number of bytes
that will be read from the stream, to overcome one bug in the linux
kernel that may return EIINVAL for pread() with offset pass the EOF.

v2: Swap read_hint and file_size (since it is easy to miss something)

Before the first argument to readFile()/createReadBufferFromFileBase()
was read_hint, not the file_size, and let's preserve the order, since
it is easy to miss something

This will also fix 02051_read_settings test automatically because now
MergeTreeReaderStream will pass estimated_sum_mark_range_bytes to
read_hint not file_size, previously it cause on of the following errors:
- Attempt to read after EOF w/ O_DIRECT
- and LOGICAL_ERROR while adjusting granulas w/o O_DIRECT

This will also improve zero-length reads guard (via
ReadBufferFromEmptyFile), that had been added in #30190

v3: fix for other storages that wasn't enabled in fast-test
v4: ignore ENOENT/ENOTSUP in readFile
 2022-01-04 10:53:17 +03:00
Azat Khuzhin
e548dae8d9 Pass size of the file to the readers 
v2: s/size/file_size/g
 2022-01-04 10:53:17 +03:00
taiyang-li
1e102bc1b2 merge master 2022-01-01 09:01:06 +08:00
Alexander Tokmakov
0d91887cdc save versions of parts 2021-12-30 16:15:28 +03:00
Nikolay Degterinsky
79e15e84d5 Refactor 2021-12-30 04:47:34 +00:00
Kruglov Pavel
489a30859f
Merge pull request #32455 from Avogar/schema-inference 
Automatic schema inference for input formats
 2021-12-29 21:03:48 +03:00
Nikolai Kochetov
68bf38133a Add pool to WriteBufferFromS3 2021-12-29 14:04:21 +00:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
benbiti
9004e98992 parseDateTimeBestEffort support Unix Timestamp with Milliseconds 2021-12-29 13:43:54 +08:00
Alexander Tokmakov
8aec3ae94f Merge branch 'master' into mvcc_prototype 2021-12-28 14:24:36 +03:00
taiyang-li
9036b18c2f merge master 2021-12-27 15:12:48 +08:00
alexey-milovidov
47d50c3bd1
Merge pull request #33140 from kssenii/azure-tests-fix 
Fix azure blob storage tests failures
 2021-12-25 06:25:26 +03:00
alexey-milovidov
d15da636f6
Update ReadBufferFromAzureBlobStorage.cpp 2021-12-25 05:57:15 +03:00
alexey-milovidov
bcd4df789d
Update WriteBufferFromAzureBlobStorage.cpp 2021-12-25 05:56:15 +03:00
alexey-milovidov
f2075fac64
Update ReadBufferFromAzureBlobStorage.cpp 2021-12-25 05:55:49 +03:00
kssenii
b4420cfa3e Fix 2021-12-24 22:51:36 +03:00
Alexey Milovidov
29d28c531f Move code around to avoid dlsym on Musl 2021-12-24 12:25:27 +03:00
Kruglov Pavel
f38bc9fc38
Merge branch 'master' into fix-lz4 2021-12-23 16:13:26 +03:00
Nikolay Degterinsky
826f7cb0f5
Merge pull request #32240 from benbiti/master 
Fix unix timestamp Millisecond convert to DateTime64, fractional part calc reversed
 2021-12-23 12:11:53 +03:00
Kseniia Sumarokova
f0434f21ce
Merge pull request #33039 from kssenii/fix-azure-exception 
Fix exception in azure write buffer desctructor
 2021-12-23 10:26:40 +03:00
Vitaly Baranov
7660530fcc
Merge pull request #32955 from azat/read-fix 
Fix UB in case of unexpected EOF during filling a set from HTTP query
 2021-12-23 06:41:14 +03:00
Alexander Tokmakov
b7184732d5 Merge branch 'master' into mvcc_prototype 2021-12-22 19:46:09 +03:00
Dmitry Novik
d256c3a9ca
Merge pull request #29224 from ClickHouse/fix_IP_conversion 
Merging #27914
 2021-12-22 18:36:13 +03:00
mergify[bot]
04f727e84c
Merge branch 'master' into read-fix 2021-12-22 13:49:04 +00:00
kssenii
65c5334bb0 Safer 2021-12-22 14:22:14 +03:00
kssenii
060d3228e9 Fix 2021-12-22 13:43:21 +03:00
kssenii
d8ed4db6d4 Fix 2021-12-22 11:42:23 +03:00
Kseniia Sumarokova
484a4d62dd
Merge pull request #32835 from kssenii/fising-s3 
Fix optimization with lazy seek for async reads from remote fs
 2021-12-22 09:38:23 +03:00
benbiti
81a7a393c8 fix unix timestamp convert, fractional reversed 2021-12-22 01:32:16 +08:00
mergify[bot]
7da6d8b310
Merge branch 'master' into fix_IP_conversion 2021-12-21 12:16:05 +00:00
kssenii
ebcb66a361 Fix 2021-12-21 10:58:02 +03:00
taiyang-li
2597925724 merge master 2021-12-21 15:55:39 +08:00
Azat Khuzhin
4230f04f72 Fix UB in LimitReadBuffer in case of unexpected EOF 
Fixes the following possible assertions:
- ./src/IO/ReadBuffer.h:58: bool DB::ReadBuffer::next(): Assertion `!hasPendingData()' failed.
- ./src/IO/LimitReadBuffer.cpp:17: virtual bool DB::LimitReadBuffer::nextImpl(): Assertion `position() >= in->position()' failed.

Fixes: 02151_http_s_structure_set_eof
 2021-12-21 08:38:41 +03:00
Alexander Tokmakov
9cd49bc0ec Merge branch 'master' into mvcc_prototype 2021-12-20 22:06:22 +03:00
Nikolai Kochetov
35883e0dae
Merge pull request #32979 from ClickHouse/decrease-some-s3-log-level 
Decrease log level for some s3 messages.
 2021-12-20 20:56:22 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment 
Loops remove postfix increment
 2021-12-20 19:51:07 +03:00
Vitaly Baranov
e61d3eef0c
Merge pull request #32747 from vitlibar/improve-grpc-compression 
Improve gRPC compression support
 2021-12-20 19:44:58 +03:00
Nikolai Kochetov
7dea7b7f76 Decrease log level for some s3 messages. 2021-12-20 18:18:54 +03:00
Maksim Kita
51477adf1b Updated additional cases 2021-12-20 15:55:07 +03:00
Vitaly Baranov
02d6cea857 Lz4DeflatingWriteBuffer now supports changes in the nested buffer between calls. 2021-12-19 16:29:57 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
kssenii
40c266840c Fixes 2021-12-17 18:16:29 +03:00
kssenii
ed42620530 Rename BlobStorage to AzureBlobStorage 2021-12-17 12:24:05 +03:00
Alexander Tokmakov
32e62ed5c2 Merge branch 'master' into mvcc_prototype 2021-12-17 10:43:46 +03:00
taiyang-li
d033fc4c24 merge master and fix conflict 2021-12-17 15:11:21 +08:00
kssenii
ca283a4bb8 Fix optimization with lazy seek 2021-12-17 00:18:28 +03:00
alesapin
c338bee718
Revert "Fix build issue related to azure blob storage" 2021-12-16 14:37:12 +03:00
alesapin
c0cebaea79
Merge pull request #32788 from amosbird/buildfix 
Fix build issue related to azure blob storage
 2021-12-16 13:45:22 +03:00
Kseniia Sumarokova
947ef81f53
Update ReadWriteBufferFromHTTP.h 2021-12-16 09:14:09 +03:00
mergify[bot]
41ae7c8eb1
Merge branch 'master' into fix_IP_conversion 2021-12-15 19:19:47 +00:00
Amos Bird
9c9f4a3e6c
build fix 2021-12-15 16:55:18 +08:00
mergify[bot]
4300f77532
Merge branch 'master' into fix_IP_conversion 2021-12-14 20:12:34 +00:00
Alexander Tokmakov
d7ad72838c Merge branch 'master' into mvcc_prototype 2021-12-14 23:07:52 +03:00
alesapin
9974f10631 Fix clang tidy 2021-12-14 15:34:20 +03:00
taiyang-li
ca3f7425a4 fix code 2021-12-14 17:37:31 +08:00
taiyang-li
8234d1176f merge master 2021-12-14 10:39:21 +08:00
alexey-milovidov
730c16bd0c
Merge pull request #32649 from vdimir/issue_32504 
Reset pipeline in BlockIO::onFinish
 2021-12-13 21:05:33 +03:00
alexey-milovidov
fd9d40925a
Merge pull request #32640 from ClickHouse/fix-function-file 
Fix surprisingly bad code in function `file`
 2021-12-13 19:34:36 +03:00
Kseniia Sumarokova
78ed383f15
Merge pull request #31505 from ContentSquare/azure_blob_storage_dependency 
Azure Blob Storage Disks
 2021-12-13 18:28:48 +03:00
vdimir
bda4ecbe90
Catch exceprion in ~ZlibDeflatingWriteBuffer 2021-12-13 12:29:35 +03:00
Alexey Milovidov
71926a3a97 Fix surprisingly bad code in function "file" 2021-12-13 07:57:54 +03:00
李扬
8675086104
Merge branch 'master' into hive_table 2021-12-12 09:01:46 -06:00
Jakub Kuklis
578aec97e9 Removing TODO and NOTE marks used mainly as hooks for review in the PR 2021-12-10 22:04:38 +01:00
Jakub Kuklis
85807adec4 Fixing variable shadow error 2021-12-10 22:04:38 +01:00
Jakub Kuklis
910db6ca57 Code cleanup addressing PR remarks 2021-12-10 22:04:38 +01:00
Jakub Kuklis
6acbec4455 Rebased on master 2021-12-10 22:04:38 +01:00
Jakub Kuklis
0c65cfed1f Addressing first PR remarks 2021-12-10 22:04:37 +01:00
Jakub Kuklis
7c2e116214 Adding max_retries config 2021-12-10 22:04:37 +01:00
Jakub Kuklis
a33cea0525 Refactors after proper rebase on master 2021-12-10 22:04:37 +01:00
Jakub Kuklis
b3728dbbaf Rebase on master 2021-12-10 22:04:37 +01:00
Jakub Kuklis
b8ef47006c Minor refactors before PR 2021-12-10 22:04:37 +01:00
Ryad ZENINE
c4421d6866 Small changes following review 2021-12-10 22:04:37 +01:00
Jakub Kuklis
7f595eedbc Catch Blob Storage exceptions 2021-12-10 22:04:37 +01:00
Jakub Kuklis
9a00acfb72 Fixing read size error plus minor improvements 2021-12-10 22:04:37 +01:00
Jakub Kuklis
fd8ebe0d9f Maximum read 2021-12-10 22:04:37 +01:00
Jakub Kuklis
0d7a4b10c3 Using unique_ptr for Blob Client in Read Buffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
12924b3659 Minor improvements 2021-12-10 22:04:37 +01:00
Jakub Kuklis
fa2261dee6 Changing offset in ReadBuffer to size_t 2021-12-10 22:04:37 +01:00
Jakub Kuklis
33247db3f4 Adding finalize() to WriteBuffer, always using block blob uploads, code cleanup 2021-12-10 22:04:37 +01:00
Jakub Kuklis
bf2476a076 Shared pointer for Azure client and fixing path reference in ReadBuffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
486f267b3b Further code cleanup and error codes improvement 2021-12-10 22:04:37 +01:00
Jakub Kuklis
956348073b ReadBufferFromBlobStorage operating like a proper stream, rather than a single download-read buffer 2021-12-10 22:04:37 +01:00
Jakub Kuklis
63ec8d4244 Adding multipart upload to WriteBuffer, retries in ReadBuffer, additional settings for buffers 2021-12-10 22:04:37 +01:00
Jakub Kuklis
dbb98e3636 Code cleanup, leaving some debug information printing under #ifdef VERBOSE_DEBUG_MODE 2021-12-10 22:04:37 +01:00
Jakub Kuklis
8230f8039c Two column MergeTree table with Blob Storage disk work! Fixed buffer in ReadBufferFromBlobStorage to be permanent (class attribute) 2021-12-10 22:04:37 +01:00
Jakub Kuklis
2b4d3665ab Basic inserts and selects work for a MergeTree table with a Blob Disk, based on a single column table 2021-12-10 22:04:37 +01:00
Jakub Kuklis
4b51ddee1b Initial check with write and read works for the Blob Storage Disk 2021-12-10 22:04:37 +01:00
Jakub Kuklis
03f85dee41 Managed to read markers file in SELECT, getting error for CODEC version now 2021-12-10 22:04:37 +01:00
Jakub Kuklis
482a10e62e Uploading to Blob Storage on INSERT seems to do something reasonable 2021-12-10 22:04:37 +01:00
Jakub Kuklis
c43b90c58c Progressing with WriteBufferFromBlobStorage 2021-12-10 22:04:37 +01:00
Jakub Kuklis
ee1a568919 The following query passes without erors: CREATE TABLE ... ENGINE = MergeTree ... storage_policy = 'blob_storage', debug constants present in the code 2021-12-10 22:04:37 +01:00
Jakub Kuklis
0cf338e165 Skeleton for DiskBlobStorage 2021-12-10 22:04:36 +01:00
taiyang-li
02b877c978 move remote buffer from src/IO to src/Storages 2021-12-10 10:27:53 +08:00
Alexey Milovidov
dc3565b3fe Quote more identifiers 2021-12-09 23:51:52 +03:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
taiyang-li
0eef092ded fix building 2021-12-09 16:57:33 +08:00
lgbo
38020a33e4 some refinement 2021-12-08 10:07:04 +08:00
Alexander Tokmakov
7fcb79ae72 Merge branch 'master' into mvcc_prototype 2021-12-07 14:39:29 +03:00
lgbo-ustc
54cc65ede8 fixed code style 2021-12-07 17:12:26 +08:00
lgbo-ustc
5a8c56bd55 fixed erros on creating dir on startup 2021-12-07 16:35:08 +08:00
lgbo-ustc
b6c230606e fixed config error in server startup 2021-12-07 14:46:17 +08:00
Kseniia Sumarokova
926fd568c7
Merge pull request #32113 from FrankChen021/url_http_header 
Set Content-Type in HTTP packets issued from URL engine
 2021-12-07 08:52:36 +03:00
lgbo-ustc
15504a4bfa fixed code style 2021-12-07 12:32:50 +08:00
lgbo-ustc
562a0499a1 fixed code style 2021-12-07 12:29:20 +08:00
lgbo-ustc
fcea033a07 fixed code style 2021-12-07 12:17:47 +08:00
lgbo-ustc
07045bc451 fixed a exception message 2021-12-07 11:18:44 +08:00
lgbo-ustc
0c1733817b update LRUCache set() 2021-12-07 10:57:05 +08:00
lgbo-ustc
d4dcbd59fb fixed code style 2021-12-06 19:59:15 +08:00
lgbo-ustc
dd70209623 fixed code style 2021-12-06 18:19:36 +08:00
lgbo-ustc
5913d67553 fixed code style 2021-12-06 16:51:49 +08:00
lgbo-ustc
ddac2a33bd fixed code style 2021-12-06 16:46:13 +08:00
Frank Chen
59a701ee6e
Fix style 2021-12-06 16:31:47 +08:00
Frank Chen
bf3d3ea25c
Fix style 2021-12-06 16:20:10 +08:00
lgbo-ustc
3c56a5deac refector 
1. Make the lru cache pilicy in RemoteReadBufferCache into Common/UnreleasableLRUCache.h
2. If try to use local cached file fail, the RemoteReadBuffer::create() return the original ReadBuffer
3. Remove LocalFileReader
4. RemoteCacheController does not hold a context now, and the download process starts in RemoteReadBufferCache
   once a new RemoteCacheController is create successly
5. Make an abstract class RemoteFileMetaDataBase for descripting a remote file meta data.
6. Remote file meta data need to  a version, can cache multi-version for a file now
 2021-12-06 16:17:32 +08:00
lgbo-ustc
01940c3f01 coding refinement 
1. make remote file meta data be a abstract class
2. make cache controller download process be started by RemoteReadBufferCache
 2021-12-06 10:30:59 +08:00
avogar
7549619b25 Improve skiping unknown fields with Quoted escaping rule in Template/CustomSeparated formats 2021-12-03 16:25:35 +03:00
taiyang-li
041e3074df Merge branch 'hive_table' of https://github.com/bigo-sg/ClickHouse into bigo_hive_table 2021-12-02 20:23:39 +08:00
mergify[bot]
a1b31018d9
Merge branch 'master' into fix-lz4 2021-12-02 12:00:43 +00:00
Kruglov Pavel
049b2c0c14
Merge pull request #31072 from MaxWk/feature-support-bool-type 
Feature support bool type
 2021-12-02 14:50:29 +03:00
taiyang-li
2f4e7e1d4e merge master 2021-12-02 19:48:21 +08:00
Anton Popov
a79dbb6b6a
Merge pull request #32079 from Avogar/parse-plus-in-float 
Allow plus before Float32/Float64
 2021-12-02 14:03:48 +03:00
lgbo-ustc
1c9d0b86cc fixed code style 2021-12-02 18:19:15 +08:00
lgbo-ustc
c8f536a0a4 Use read/write buffer to read/write files 2021-12-02 18:11:35 +08:00
mergify[bot]
cba9f3f14c
Merge branch 'master' into url_http_header 2021-12-02 09:13:41 +00:00
frank chen
8e9f4949e4 Set Content-Type in HTTP packets issued from URL engine 
Signed-off-by: frank chen <frank.chen021@outlook.com>
 2021-12-02 15:49:34 +08:00
Kseniia Sumarokova
563b615904
Merge pull request #31766 from kssenii/odbc-fix 
Fix crash with empty result on odbc query
 2021-12-01 21:51:08 +03:00
avogar
e2eea4f361 Allow plus before Float32/Float64 2021-12-01 21:20:00 +03:00
Anton Popov
6f4d9a53b2 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-12-01 15:54:33 +03:00
taiyang-li
701ad45aaa Merge branch 'hive_table' of https://github.com/bigo-sg/ClickHouse into bigo_hive_table 2021-12-01 14:14:06 +08:00
taiyang-li
4aeadf3967 fix build error 2021-12-01 14:13:48 +08:00
lgbo-ustc
cce42ee311 fix logs 2021-12-01 11:32:58 +08:00
lgbo-ustc
0264afd7da remove some comments 2021-12-01 11:03:33 +08:00
lgbo-ustc
aaa5d8f002 rewrite thread pool in remote file cache 2021-12-01 10:45:25 +08:00
MaxWk
02ce70f738 fix ci check 2021-12-01 00:13:11 +08:00
lgbo-ustc
81eafdb708 fixed 2021-11-30 19:09:09 +08:00
taiyang-li
d213500a3e remove blank at end of line 2021-11-30 18:23:24 +08:00
lgbo-ustc
1014ca8c58 make two loops in clearing cached files into one 2021-11-30 17:57:58 +08:00
Kseniia Sumarokova
7e324d5894
Ping CI 2021-11-30 12:21:26 +03:00
lgbo-ustc
b023dd34c0 fixed code style 2021-11-30 17:11:58 +08:00
taiyang-li
cacf516e3e calculate column value by default expression & apply defaults_for_omitted_fields_ in ArrowColumnToCHColumn 2021-11-30 14:52:26 +08:00
taiyang-li
ad6ba24efd fix ArrowColumnToCHColumn 2021-11-30 10:49:57 +08:00
taiyang-li
6922f09ea3 reuse seekable read buffer with size 2021-11-29 20:19:36 +08:00
mergify[bot]
8d5460b469
Merge branch 'master' into feature-support-bool-type 2021-11-29 11:50:18 +00:00
taiyang-li
03e7f0f6f7 extract hivemetastore client from context 2021-11-29 18:23:57 +08:00
MaxWk
beaa5b5596 support TRUE/FALSE for vaules 2021-11-29 18:05:15 +08:00
avogar
bcf53c8876 Try to fix lz4 compression 2021-11-29 12:41:13 +03:00
taiyang-li
1c4efe5b13 fix code style of logger and exceptin 2021-11-29 17:01:34 +08:00
taiyang-li
7e0069aacb fix code style 2021-11-29 16:24:45 +08:00
taiyang-li
c7a0fe467f fix code stylke 2021-11-29 16:22:43 +08:00
lgbo-ustc
b28f3a2ad4 fixed code style 2021-11-29 15:00:41 +08:00
taiyang-li
83be8d28e6 fix code stylke 2021-11-29 14:50:33 +08:00
taiyang-li
fd4462db64 fix code style 2021-11-29 13:01:03 +08:00
taiyang-li
01f29fcf21 fix code style 2021-11-29 12:17:22 +08:00
taiyang-li
e8addd3243 fix code style 2021-11-29 12:01:52 +08:00
taiyang-li
85cb7fc23e fix code style 2021-11-29 11:47:31 +08:00
taiyang-li
919b439b7c remove useless code 2021-11-29 11:37:07 +08:00
taiyang-li
e67407b5c5 fix code style 2021-11-29 11:30:11 +08:00
taiyang-li
72f60cceb9
Merge branch 'master' into hive_table 2021-11-25 17:33:26 +08:00
kssenii
39588579dc Fix 2021-11-25 12:26:26 +03:00
Kseniia Sumarokova
87b920594e
Merge pull request #31678 from ucasfl/log 
fix typo
 2021-11-25 11:29:36 +03:00
Kseniia Sumarokova
93cf66df12
Merge pull request #30936 from kssenii/seekable-read-buffers 
Reduce memory usage for some formats when reading with s3/url/hdfs
 2021-11-25 11:19:24 +03:00
alexey-milovidov
ac66433650
Merge pull request #31672 from ClickHouse/better-crosscompiling-freebsd 
More easy cross-compiling for FreeBSD
 2021-11-25 01:31:41 +03:00
taiyang-li
4d750c9dc5 fix code typo 2021-11-24 17:52:02 +08:00
taiyang-li
89dcef69d5 merge master 2021-11-24 14:38:04 +08:00
feng lv
89e9569ad6 fix typo 2021-11-24 04:34:25 +00:00
lgbo
ab0508dfc8
Update src/IO/RemoteReadBufferCache.h 
fixed code style

Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2021-11-24 09:39:29 +08:00
Alexey Milovidov
e59e833cc6 Fix build 2021-11-24 03:11:22 +03:00
Alexey Milovidov
1acbafdffe Remove outstandingly wrong code 2021-11-24 00:48:57 +03:00
lgbo
996d7125c0
Merge branch 'master' into hive_table 2021-11-23 10:19:02 +08:00
Anton Popov
ccd78e3838 Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-22 17:19:35 +03:00
mergify[bot]
a7ba3e23a0
Merge branch 'master' into fix-write-buffers 2021-11-22 11:24:27 +00:00
avogar
b8695e8236 Fix comments 2021-11-22 14:19:26 +03:00
Kruglov Pavel
814a36ba69
Merge pull request #31434 from Avogar/custom-with-names-and-types 
Add formats CustomSeparatedWithNames/WithNamesAndTypes
 2021-11-22 13:24:00 +03:00
Kruglov Pavel
d9c1a0c8ec
Merge branch 'master' into fix-write-buffers 2021-11-20 17:48:24 +03:00
kssenii
ff969b4605 Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers 2021-11-20 15:03:13 +03:00
kssenii
f29ef7d18b Add one more test 2021-11-20 15:01:45 +03:00
Kruglov Pavel
4856b50c23
Merge pull request #31409 from excitoon-favorites/s3envsegfaults 
Resolve `nullptr` in STS credentials provider for S3
 2021-11-19 18:45:38 +03:00
Kruglov Pavel
eb1cc834dc
Merge branch 'master' into custom-with-names-and-types 2021-11-19 16:51:25 +03:00
taiyang-li
e8644807fe merge master and solve conflict 2021-11-19 15:01:58 +08:00
Vladimir Chebotarev
8b622bd4ff Fixed a typo. 2021-11-18 10:54:59 +03:00
Alexander Tokmakov
06bed68f6e Merge branch 'master' into mvcc_prototype 2021-11-17 23:03:19 +03:00
alexey-milovidov
1e058fb207
Merge pull request #30190 from azat/ReadBuffer-throw-on-empty 
Do not allow zero-length reads
 2021-11-17 10:21:08 +03:00
avogar
73d1918410 tmp 2021-11-16 17:10:30 +03:00
Vladimir Chebotarev
1e79177ce2 Build fix. 2021-11-16 12:31:51 +03:00
lgbo-ustc
92b8ec5758 fixed code style 2021-11-16 11:11:58 +08:00
lgbo-ustc
970d90ab15 fixed code style 2021-11-16 11:11:10 +08:00
Ilya Yatsishin
e5301ab712
Merge pull request #31286 from cfcz48/add_oss_support 
Aliyun OSS Storage support
 2021-11-15 17:00:27 +03:00
Alexander Tokmakov
ce2f692bb4 Merge branch 'master' into mvcc_prototype 2021-11-15 14:56:50 +03:00
lgbo-ustc
41d3ec2481 fixed typos 2021-11-15 19:08:58 +08:00
lgbo-ustc
78eefe2cac fixed code style 2021-11-15 17:51:09 +08:00
avogar
8c87f8861c Fix style 2021-11-15 12:48:28 +03:00
avogar
ba06953ee7 Fix tests 2021-11-15 12:47:00 +03:00