thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-12 09:22:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
130,829 Commits 9,297 Branches 1,782 Tags 2.1 GiB
febdd542f8
Commit Graph

3210 Commits

Author SHA1 Message Date
serxa
2ab922777a fix typos 2023-09-14 09:21:23 +00:00
serxa
f3e6ba9bd1 fix style 2023-09-14 09:17:55 +00:00
serxa
09d4bf6930 add IO scheduling node for bandwidth throttling 2023-09-14 08:51:47 +00:00
Michael Kolupaev
df71dcd94d Fix data race in copyFromIStreamWithProgressCallback 2023-09-11 21:04:37 +00:00
Sergei Trifonov
08bad4d440
Merge pull request #47009 from ClickHouse/disks-and-io-sheduler 2023-09-11 07:22:14 +02:00
robot-clickhouse-ci-2
47c0f8c99f
Merge pull request #54179 from joelynch/http-query-timing 
Add elapsed_ns to HTTP headers X-ClickHouse-Progress and X-ClickHouse-Summary
 2023-09-11 06:18:47 +02:00
serxa
1b3da7fc1e fix build 2023-09-09 09:26:18 +00:00
Duc Canh Le
52dfaa54bf pass http retry timeout as ms 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-09-08 08:02:30 +00:00
Nikolay Degterinsky
3b43667397
Merge pull request #54330 from HarryLeeIBM/hlee-s390x-read-float 
Fix precise float parsing issue on s390x
 2023-09-07 17:51:40 +02:00
serxa
a94ae9a7ab review fixes 2023-09-07 14:02:58 +00:00
Sergei Trifonov
958f102daa
Update src/IO/IResourceManager.h 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-09-07 15:58:31 +02:00
Sergei Trifonov
c9635ddacd
Update src/IO/ISchedulerNode.h 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-09-07 15:48:49 +02:00
Joe Lynch
a5b508d6c5
Fix json in header 2023-09-07 14:59:33 +02:00
serxa
77697211cd add docs for system.scheduler + small fix 2023-09-06 19:30:39 +00:00
Antonio Andelic
52e4f3c571 Fix style 2023-09-06 09:27:35 +00:00
Antonio Andelic
af43463bfb Better 2023-09-06 08:54:39 +00:00
Antonio Andelic
d0c70c0f8a Merge branch 'master' into add-sso-for-s3 2023-09-06 07:29:05 +00:00
HarryLeeIBM
d695417869 Fix precise float parsing issue on s390x 2023-09-05 13:38:02 -07:00
Joe Lynch
767f2472c2
Add elapsed_ns to HTTP headers X-ClickHouse-Progress and X-ClickHouse-Summary 
The TCP interface progress has this field. This is not a super accurate
measure of server side query time, but trying to measure from the client
is even worse.
 2023-09-05 21:03:13 +02:00
Antonio Andelic
e8034a2a52 Add SSO provider 2023-09-05 07:46:41 +00:00
irenjj
c9261bbf18 Merge remote-tracking branch 'upstream/master' into feat_markdown 2023-09-04 22:35:53 +08:00
serxa
aa8e9c3cbc fix style 2023-09-04 14:18:54 +00:00
Sergei Trifonov
258da91b5f
Merge branch 'master' into disks-and-io-sheduler 2023-09-04 15:14:28 +02:00
serxa
b5233efc20 fix system table and scheduler config reloading 2023-09-04 12:35:17 +00:00
serxa
8fdd118e0c fix build 2023-09-03 22:40:00 +00:00
serxa
adcab639a4 add system.scheduler table for introspection 2023-09-03 21:48:33 +00:00
Kseniia Sumarokova
b3319f7908
Minor changes (#54171) 2023-09-03 15:47:52 +02:00
kssenii
9eb1dfcd12 Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
Robert Schulze
aefb543734
Merge remote-tracking branch 'rschu1ze/master' into feat_markdown 2023-08-31 11:32:44 +00:00
serxa
9c92d413b3 Merge branch 'master' into disks-and-io-sheduler 2023-08-30 11:18:49 +00:00
Antonio Andelic
ddb58217d4 Merge branch 'master' into archive-improvements-2 2023-08-30 07:43:25 +00:00
irenjj
473ba91c21 Refactor writeAnyMarkdownEscapedString and Add more info into doc. 2023-08-29 23:54:40 +08:00
Antonio Andelic
5a0c2ca108 Merge branch 'master' into archive-improvements-2 2023-08-28 08:34:42 +00:00
irenjj
51aa89eed8 Add a setting to automatically escape special characters in Markdown. 2023-08-28 00:10:33 +08:00
alesapin
0e601b856f
Merge pull request #53751 from Algunenano/clang18 
Support clang-18 (Wmissing-field-initializers)
 2023-08-25 23:27:06 +02:00
Raúl Marín
6490213207 Merge remote-tracking branch 'blessed/master' into clang18 2023-08-24 17:35:26 +02:00
Kruglov Pavel
44db5fa992
Merge branch 'master' into cache-count 2023-08-24 17:21:18 +02:00
Arthur Passos
2bade7db08
Add global proxy setting (#51749) 
* initial impl

* fix env ut

* move ut directory

* make sure no null proxy resolver is returned by ProxyConfigurationResolverProvider

* minor adjustment

* add a few tests, still incomplete

* add proxy support for url table function

* use proxy for select from url as well

* remove optional from return type, just returns empty config

* fix style

* style

* black

* ohg boy

* rm in progress file

* god pls don't let me kill anyone

* ...

* add use_aws guards

* remove hard coded s3 proxy resolver

* add concurrency-mt-unsafe

* aa

* black

* add logging back

* revert change

* imrpove code a bit

* helper functions and separate tests

* for some reason, this env test is not working..

* formatting

* :)

* clangtidy

* lint

* revert some stupid things

* small test adjusmtments

* simplify tests

* rename test

* remove extra line

* freaking style change

* simplify a bit

* fix segfault & remove an extra call

* tightly couple proxy provider with context..

* remove useless include

* rename config prefix parameter

* simplify provider a bit

* organize provider a bit

* add a few comments

* comment out proxy env tests

* fix nullptr in unit tests

* make sure old storage proxy config is properly covered without global context instance

* move a few functions from class to anonymous namespace

* fix no fallback for specific storage conf

* change API to accept http method instead of bool

* implement http/https distinction in listresolver, any still not implemented

* implement http/https distinction in remote resolver

* progress on code, improve tests and add url function working test

* use protcol instead of method for http and https

* small fix

* few more adjustments

* fix style

* black

* move enum to proxyconfiguration

* wip

* fix build

* fix ut

* delete atomicroundrobin class

* remove stale include

* add some tests.. need to spend some more time on the design..

* change design a bit

* progress

* use existing context for tests

* rename aux function and fix ut

* ..

* rename test

* try to simplify tests a bit

* simplify tests a bit more

* attempt to fix tests, accept more than one remote resolver

* use proper log id

* try waiting for resolver

* proper wait logic

* black

* empty

* address a few comments

* refactor tests

* remove old tests

* baclk

* use RAII to set/unset env

* black

* clang tidy

* fix env proxy not respecting any

* use log trace

* fix wrong logic in getRemoteREsolver

* fix wrong logic in getRemoteREsolver

* fix test

* remove unwanted code

* remove ClientConfigurationperRequest and auxilary classes

* remove unwanted code

* remove adapter test

* few adjustments and add test for s3 storage conf  with new proxy settings

* black

* use chassert for context

* Add getenv comment
 2023-08-24 16:07:26 +03:00
Antonio Andelic
8e1d38d377 Merge branch 'master' into archive-improvements-2 2023-08-24 13:03:36 +00:00
avogar
894513f6cd Fix tests 2023-08-23 18:43:08 +00:00
Raúl Marín
93dac0c880 Support clang-18 (Wmissing-field-initializers) 2023-08-23 15:53:45 +02:00
Kruglov Pavel
592fa77987
Merge branch 'master' into cache-count 2023-08-23 15:18:02 +02:00
Robert Schulze
bea651e828
Merge pull request #53657 from rschu1ze/cache-improvements 
Expose more settings from caches
 2023-08-23 14:58:45 +02:00
Sema Checherinda
60103c577d
Merge pull request #53651 from CheSema/read_beyon_last_offset 
fix Logical Error in AsynchronousBoundedReadBuffer
 2023-08-23 11:21:47 +02:00
Antonio Andelic
83d4b819f3 Better support for reading from archives 2023-08-23 08:10:30 +00:00
Robert Schulze
d5a7277a59
Better parameter name 2023-08-22 15:43:13 +00:00
Kruglov Pavel
c0bdd0e00b
Merge branch 'master' into cache-count 2023-08-22 14:42:22 +02:00
avogar
b4145aeddc Cache number of rows in files for count in file/s3/url/hdfs/azure functions 2023-08-22 11:59:59 +00:00
Sema Checherinda
7e4e6e31dc
NOLINT on sscanf in gtest 2023-08-22 13:32:21 +02:00
Robert Schulze
f2fb7365e6
Add settings for 1. size_ratio (all caches) and 2. policy (index caches) 
1. is an expert-level setting, default is 0.5, applies only to SLRU.

Also, I noticed that we expose cache policy settings for the mark and
the uncompresed cache but not for the index mark and the index
uncompressed cache. Changed that as well, it simplifies the code a bit.
 2023-08-22 11:04:29 +00:00
Robert Schulze
acaa2eee58
Merge pull request #53556 from ClibMouse/feature/big-endian-UUID-correction 
Change Big Endian-UUID to work the same as Little Endian-UUID
 2023-08-22 11:57:44 +02:00
Sema Checherinda
ae5f66da1e fix special build 2023-08-22 11:34:08 +02:00
Michael Kolupaev
6009e1b293
Merge pull request #53324 from bigo-sg/ch_gluten_2583 
Implement native orc input format without arrow to improve performance
 2023-08-21 13:44:57 -07:00
Alexey Milovidov
316664456f
Merge pull request #53588 from kitaisreal/aarch64-neon-memequal-wide 
AARCH64 Neon memequal wide
 2023-08-21 21:11:01 +03:00
Sema Checherinda
81577e041b fix Logical Error in AsynchronousBoundedReadBuffer 2023-08-21 19:30:04 +02:00
kothiga
52d47785cb
Fix deserialization for quantile deterministic. 2023-08-21 09:52:14 -07:00
Antonio Andelic
ed15babddd
Merge pull request #53198 from ClickHouse/improve-archive-reading 
Improve reading from archives
 2023-08-21 14:11:25 +02:00
Maksim Kita
f32effb627 AARCH64 Neon memequal wide 2023-08-20 12:45:18 +03:00
gyfis
efa53f873c
💄 [S3::URI] Fix comment typos around versionId 2023-08-20 09:08:57 +02:00
Alexey Milovidov
d133cafd9c Better documentation about redirects 2023-08-20 00:38:18 +02:00
Yakov Olkhovskiy
257380df03
Merge pull request #50267 from HarryLeeIBM/hlee-s390x-proto-endian 
Fix endian issues in native protocol
 2023-08-19 11:16:24 -04:00
kothiga
f33c585bc5
Addressing feedback. 2023-08-18 13:50:31 -07:00
Austin Kothig
6b42975d33
Change BE-UUID to work the same as LE-UUID. Included high and low getters to provide cleaner code when accessing undertype. 2023-08-18 08:19:46 -07:00
Harry Lee
1645439000
Merge branch 'master' into hlee-s390x-proto-endian 2023-08-17 18:09:04 -04:00
Michael Kolupaev
557bfea4d8 Use long timeout for S3 copy requests 2023-08-17 19:36:45 +00:00
Robert Schulze
ec7daed52f
Merge pull request #50276 from ClibMouse/feature/mergetree-checksum-big-endian-support 
MergeTree & SipHash checksum big-endian support
 2023-08-17 13:27:18 +02:00
taiyang-li
bfa9d361cc fix tests 2023-08-15 12:16:22 +08:00
ltrk2
2846ea49b4
Merge branch 'master' into feature/mergetree-checksum-big-endian-support 2023-08-14 13:02:30 -04:00
Alexey Milovidov
7911945a74 Make one exception message longer 2023-08-13 22:40:50 +02:00
robot-ch-test-poll
967067bbb8
Merge pull request #53250 from evillique/postgresql-single-quotes 
Fix character escaping in the PostgreSQL engine
 2023-08-12 13:15:49 +02:00
Alexey Milovidov
a3208f0a9a Merge branch 'master' into ditch-tons-of-garbage-2 2023-08-11 00:20:43 +02:00
Antonio Andelic
478a5006b1 Merge branch 'master' into improve-archive-reading 2023-08-10 15:52:30 +00:00
ltrk2
a2054c04dd
Merge branch 'master' into feature/mergetree-checksum-big-endian-support 2023-08-10 10:21:34 -04:00
Alexey Milovidov
bb57caa83e
Merge pull request #53180 from ClickHouse/ditch-tons-of-garbage 
Ditch tons of garbage
 2023-08-10 16:59:51 +03:00
Kruglov Pavel
6600f87f86
Merge branch 'master' into http-valid-json-on-exception 2023-08-10 13:53:32 +02:00
Antonio Andelic
aa258a0e61 Merge branch 'master' into improve-archive-reading 2023-08-10 06:59:04 +00:00
Nikolay Degterinsky
7321f5e543 Better 2023-08-10 06:34:10 +00:00
Alexey Milovidov
52c3704dc7
Merge pull request #52683 from rschu1ze/readBinary-specializations 
Remove unneeded readBinary() specializations + update docs
 2023-08-10 04:04:57 +03:00
ltrk2
139e9433a8
Merge branch 'master' into feature/mergetree-checksum-big-endian-support 2023-08-09 09:48:00 -04:00
Antonio Andelic
0e17d26b88 More formats supported, read single archive from 1 thread 2023-08-09 11:58:37 +00:00
Alexey Milovidov
1c9cff551e Continue on garbage removal 2023-08-09 02:58:12 +02:00
Alexey Milovidov
aa757490bd Ditch tons of garbage 2023-08-09 02:19:02 +02:00
Michael Kolupaev
e4b2b3f533
Merge pull request #52828 from ClickHouse/illogical 
Don't report LOGICAL_ERROR if a file got truncated during read
 2023-08-08 13:13:20 -07:00
Alexander Tokmakov
9d29b7cdbf
Merge pull request #53109 from ClickHouse/tavplubix-patch-8 
Update PocoHTTPClient.cpp
 2023-08-07 19:33:39 +03:00
Antonio Andelic
f5597b1f8e Revert changes from https://github.com/ClickHouse/ClickHouse/pull/53064 2023-08-07 12:21:50 +00:00
Alexander Tokmakov
d35dc46061
Update PocoHTTPClient.cpp 2023-08-07 14:39:58 +03:00
Kseniia Sumarokova
b1f0cb83ff
Merge pull request #52621 from valbok/readbuffer 
MaterializedMySQL: Fix the infinite loop in ReadBuffer::read
 2023-08-07 11:56:23 +02:00
Antonio Andelic
cafa5ef2f3 Merge branch 'master' into add-reading-from-archives 2023-08-06 10:21:47 +00:00
Antonio Andelic
1d793e69f2 Fix 2023-08-05 17:06:44 +00:00
Alexey Milovidov
70630a273a
Merge branch 'master' into readBinary-specializations 2023-08-05 03:16:29 +03:00
Michael Kolupaev
9a128fa554 Fix ZstdDeflatingWriteBuffer truncating the output sometimes 2023-08-04 20:47:45 +00:00
Antonio Andelic
49ae853ae6 Fix sanitizer warnings 2023-08-04 13:57:18 +00:00
Val Doroshchuk
3cea474ccf MaterializedMySQL: Fix the infinite loop in ReadBuffer::read 
This fixes https://github.com/ClickHouse/ClickHouse/issues/33549

The infinite loop caused because sometimes the pos of ReadBuffer overflows the end.

It happens f.e. when `MySQLPacketPayloadReadBuffer::nextImpl` might return empty buffer,
because `in.nextIfAtEnd();` could not read more bytes and thus no bytes available (pos == end).
It might happen when a network error or the connection to MySQL was closed or killed.
This leads to empty `working_buffer` but successful returning from the func.
And check `if (in.eof())` from `MySQLBinlogEventReadBuffer::nextImpl()` fails and also causes empty its `working_buffer` and successul return.

At the end `payload.ignore(1)` and `payload.read(c)` produces the infinite loop since it is not eof() and ++pos overflows the end of the buffer.

Should be tested by `test_mysql_kill*` from test.py
 2023-08-04 11:34:31 +02:00
Antonio Andelic
8e7eb7f3fa Fix tests 2023-08-04 08:59:55 +00:00
Antonio Andelic
27701ed24e Small fixes 2023-08-03 11:44:24 +00:00
Antonio Andelic
d9f136ce0c Add tests for globs 2023-08-03 10:42:47 +00:00
Antonio Andelic
96bad0537a Merge branch 'master' into add-reading-from-archives 2023-08-03 08:36:42 +00:00
ltrk2
27a2d4d1c7
Merge branch 'master' into feature/mergetree-checksum-big-endian-support 2023-08-02 11:36:43 -04:00
Antonio Andelic
9423976b7a Add support for file globs 2023-08-02 07:42:56 +00:00
Yakov Olkhovskiy
357fee99ff
Merge pull request #52791 from zvonand/zvonand-float-parsing 
Add option to switch float parsing methods
 2023-08-02 00:23:23 -04:00
avogar
d12e96177a Fix tests 2023-08-01 16:17:03 +00:00
Antonio Andelic
e366ca61f5 Merge branch 'master' into add-reading-from-archives 2023-08-01 15:33:20 +00:00
Kruglov Pavel
23aab71d7c
Merge branch 'master' into http-valid-json-on-exception 2023-08-01 16:47:31 +02:00
Antonio Andelic
c24dc8e64e Multithreaded read of files from archives: 2023-08-01 13:48:49 +00:00
avogar
a71cd56a90 Output valid JSON/XML on excetpion during HTTP query execution 2023-08-01 10:06:56 +00:00
Andrey Zvonov
99cd8f9ba9
Merge branch 'master' into zvonand-float-parsing 2023-07-31 23:43:24 +03:00
Nikolai Kochetov
c4a00b8c68 Throw S3Exception whenever possible. 2023-07-31 15:39:20 +00:00
Michael Kolupaev
8b0f52f411 Don't report LOGICAL_ERROR if a file got truncated during read 2023-07-31 15:04:30 +00:00
Antonio Andelic
86a251931b Fix style 2023-07-31 14:53:40 +00:00
zvonand
e50ffa1181 fix style 2023-07-31 16:44:46 +02:00
zvonand
2088f846ad update as per reviews 2023-07-31 16:26:18 +02:00
Antonio Andelic
6a1cf5c683 Extend tests 2023-07-31 13:27:06 +00:00
Antonio Andelic
7df5c83046 Merge branch 'master' into add-reading-from-archives 2023-07-31 09:27:01 +00:00
Antonio Andelic
df45602165 Refactoring 2023-07-31 08:50:09 +00:00
Alexey Milovidov
c825f15b74
Merge pull request #52441 from CheSema/reset-connection-s3 
add tests with connection reset by peer error, and retry it inside client
 2023-07-31 09:58:41 +03:00
zvonand
0ff0a4e41f diverge parsing methods 2023-07-30 19:42:10 +02:00
ltrk2
6c9a1b14ef
Merge branch 'master' into feature/mergetree-checksum-big-endian-support 2023-07-28 16:18:46 -04:00
Antonio Andelic
1fc1b6aae4 More fixes 2023-07-28 13:00:35 +00:00
Antonio Andelic
e83e0ec2cd Fix build 2023-07-28 12:26:56 +00:00
Sema Checherinda
115a085d0f
Merge branch 'master' into reset-connection-s3 2023-07-28 14:00:14 +04:00
Antonio Andelic
720d587e85 Merge branch 'master' into add-reading-from-archives 2023-07-28 08:49:00 +00:00
Robert Schulze
b98cb7fa14
Fix build 2023-07-27 17:50:39 +00:00
Robert Schulze
8c14c4b5e8
Remove unneeded readBinary() specializations + update docs 
The default instantiation

    template <typename T>
    requires is_arithmetic_v<T>
    inline void readBinary(T & x, ReadBuffer & buf) { readPODBinary(x, buf); }

covers Int32, Int128, Int256, UInt32, UInt128 and UInt256 cases already.
 2023-07-27 15:13:54 +00:00
Alexey Milovidov
6aab4cc835 Check for unexpected cyrillic 2023-07-27 05:25:40 +02:00
robot-ch-test-poll2
7bcef0a6c0
Merge pull request #52520 from zvonand/revert-52450-remove-to-decimal-string 
Fix `toDecimalString` function
 2023-07-27 00:18:36 +02:00
Sema Checherinda
7d430b8037 retry ConnectionResetException 2023-07-26 17:31:14 +04:00
Kruglov Pavel
15cc046883
Merge branch 'master' into better-progress-bar-2 2023-07-26 13:12:24 +02:00
Sema Checherinda
83d869860e
Merge pull request #51946 from arenadata/ADQM-940 
Expose Peak memory usage in query statistics.
 2023-07-25 22:17:58 +03:00
Sema Checherinda
c341df1949 draft with retry ConnectionResetException 2023-07-25 19:59:39 +04:00
Andrey Zvonov
15b3a637e9
Merge branch 'master' into revert-52450-remove-to-decimal-string 2023-07-25 13:21:00 +03:00
Alexey Milovidov
21382afa2b Check for punctuation 2023-07-25 06:10:04 +02:00
Alexey Milovidov
168b84a592
Merge pull request #52337 from Avogar/no-decode-url 
Allow to disable decoding/encoding path in uri in URL engine
 2023-07-25 05:43:06 +03:00
Dmitry Kardymon
d4d381de7e Merge remote-tracking branch 'origin/master' into ADQM-940 2023-07-24 20:04:10 +00:00
Kruglov Pavel
fec5675cd4
Merge branch 'master' into better-progress-bar-2 2023-07-24 19:59:38 +02:00
Nikita Taranov
c6e6fd7613
Shard OpenedFileCache to avoid lock contention (#51341) 
* shard OpenedFileCache to avoid lock contention

* Update OpenedFileCache.h

* fix build

---------

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2023-07-24 15:58:21 +02:00
Andrey Zvonov
21097209d2
Revert "Remove toDecimalString" 2023-07-24 15:41:21 +03:00
Dmitry Kardymon
62f9a953fe Merge remote-tracking branch 'origin/master' into ADQM-940 2023-07-24 04:52:12 +00:00
Alexey Milovidov
d7cdfb47d3 Fix merge 2023-07-24 01:56:04 +02:00
Alexey Milovidov
5c15e6021b Merge branch 'master' into remove-to-decimal-string 2023-07-24 01:51:43 +02:00
Alexey Milovidov
2467eaf32d
Merge pull request #52392 from azat/netlink-provider 
Rename TaskStatsInfoGetter into NetlinkMetricsProvider
 2023-07-22 19:14:15 +03:00
Kseniia Sumarokova
d34f2bed07
Merge pull request #51386 from bigo-sg/fix_hdfs_read_buffer_heap_overflow 
Fix heap overflow in read buffer from hdfs when read_until_position is not zero
 2023-07-22 15:00:58 +02:00
Alexey Milovidov
5f4756fb33 Remove toDecimalString 2023-07-22 04:54:58 +02:00
ltrk2
90a2c460c6
Merge branch 'master' into feature/mergetree-checksum-big-endian-support 2023-07-21 08:07:18 -04:00
Azat Khuzhin
f0e277f94a Rename TaskStatsInfoGetter into NetlinkMetricsProvider 
There is ProcfsMetricsProvider, so by analogy to it.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-21 06:48:18 +02:00
Alexander Tokmakov
a39ba00ec3
Merge pull request #52116 from aalexfvk/http_failover 
Handle multiple resolved IPs for HTTP/HTTPS session connection
 2023-07-20 18:25:45 +03:00
Alexander Tokmakov
e467264588
Update src/IO/HTTPCommon.cpp 2023-07-20 17:56:30 +03:00
avogar
2b8e4ebd4c Allow to disable decoding/encoding path in uri in URL engine 2023-07-19 19:48:39 +00:00
ltrk2
ba4072f049 Adapt changes around SipHash 2023-07-19 10:01:58 -07:00
ltrk2
51e2c58a53 Implement endianness-independent SipHash and MergeTree checksum serialization 2023-07-19 10:01:55 -07:00
Dmitry Kardymon
68a501a14e Merge remote-tracking branch 'origin/master' into ADQM-940 2023-07-18 18:16:09 +00:00
Kruglov Pavel
64e88cde21
Merge branch 'master' into better-progress-bar-2 2023-07-18 13:37:53 +02:00
Andrey Zvonov
f2e68d1d7e
Merge branch 'master' into zvonand-issue-42138 2023-07-16 01:06:41 +03:00
Vitaly Baranov
2f23b6a593
Merge branch 'master' into backups-native-copy 2023-07-15 08:40:17 +02:00
Aleksei Filatov
08defa36b2 Add code 2023-07-15 08:18:35 +03:00
zvonand
cbe0a0e748 upd threshold due to dt64 range change 2023-07-15 03:04:56 +02:00
李扬
6aec25e681
Merge branch 'master' into fix_hdfs_read_buffer_heap_overflow 2023-07-11 10:03:09 +08:00
Dmitry Kardymon
09ba975ffc Merge remote-tracking branch 'origin/master' into ADQM-940 2023-07-10 13:32:29 +00:00
Azat Khuzhin
1844ac37d7 Remove create_read_buffer argument for copyS3File() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-09 08:20:03 +02:00
Azat Khuzhin
1590ffa3b1 Remove copyS3FileNative() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-09 08:20:03 +02:00
Azat Khuzhin
5835e72fd6 More generic approach to disable native copy 
Previous patch implements this only for BACKUP/RESTORE, but it can be
useful for regular disks as well, so add allow_native_copy for disks.

Note, that there is s3_allow_native_copy query setting, since it looks
redundant, since it make sense only for S3 disks, and not on a per query
basis.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-09 08:20:02 +02:00
Alexey Milovidov
59eadca95c
Merge branch 'master' into less-logs-2 2023-07-09 08:49:44 +03:00
Alexey Milovidov
3d4800995f
Merge pull request #49732 from nickitat/impr_prefetch 
Improve reading with prefetch
 2023-07-09 06:10:58 +03:00
Alexey Milovidov
b1d1918607
Merge branch 'master' into retry 2023-07-08 03:08:48 +03:00
Dmitry Kardymon
3fcfbb1038 Merge remote-tracking branch 'clickhouse/master' into ADQM-940 2023-07-07 15:09:41 +03:00
Robert Schulze
271297823a
Allow var-int encoded 64-bit integers with MSB=1 
Resolves: #51486

Until now, it was illegal to encode 64-bit (unsigned) integers with
MSB=1, i.e. values > (1ULL<<63) - 1, as var-int. In more detail, the
var-int code used by ClickHouse server and client spent at most 9 bytes
per value such that 9 * 7 = 63 bits could be encoded. Some 3rd party
clients (e.g. Rust clickhouse-rs) had the same limitation, whereas other
clients understand the full range (Python clickhouse-driver).

PRs #47608 and #48628 added sanity checks as asserts or exceptions
during var-int encoding on the server side. This was considered okay as
such huge integers so far occurred only during testing (usually fuzzing)
but not in practice.

Issue #51486 is a new fuzzing issue where the exception thrown from the
sanity check led to a half-baked progress packet and as a result, a
logical error / server crash.

The only fix which is not another bandaid is to allow the full range in
var-int coding. Clients will have to allow the full range too, a note
will be added to the changelog. (the alternative was to create another
protocol version but as var-int is used all over the place this was
considered infeasible)

Review note: this is the relevant commit.
 2023-07-06 20:23:23 +00:00
Nikita Taranov
63b9c1ac06 add test 2023-07-06 18:58:13 +02:00
Robert Schulze
3f744c1e14
Cosmetics: rename template parameter 2023-07-06 14:47:40 +00:00
Robert Schulze
7644f0b37c
Cosmetics: move code around 2023-07-06 14:44:06 +00:00
Nikita Taranov
aec7205636 rework pool usage 2023-07-06 14:41:09 +02:00
Nikita Taranov
c23e29d6aa don't account session's memory in thread/user mem tracker 2023-07-06 14:41:03 +02:00
Sema Checherinda
8f7381f0a2
Merge branch 'master' into less-logs-2 2023-07-06 12:07:58 +02:00
Kruglov Pavel
60cf8a4c94
Merge branch 'master' into better-progress-bar-2 2023-07-05 19:58:55 +02:00
李扬
3b860cd685
Merge branch 'master' into fix_hdfs_read_buffer_heap_overflow 2023-07-04 10:04:57 +08:00
Antonio Andelic
eba60dd81e
Merge branch 'master' into keeper-with-disks 2023-07-01 20:59:16 +02:00
Sema Checherinda
f6ad94d5bc
Merge branch 'master' into no-finalize-WriteBufferFromOStream 2023-06-30 22:12:31 +02:00
Yakov Olkhovskiy
023e40535a
Merge pull request #51349 from ClickHouse/ft-thread-scheduling-stacktrace 
Collect stack traces from job's scheduling and print along with exception's stack trace.
 2023-06-30 11:01:58 -04:00
Sema Checherinda
c45fed58fc
Merge branch 'master' into less-logs-2 2023-06-30 15:55:17 +02:00
Antonio Andelic
2058b1346e Merge branch 'master' into keeper-with-disks 2023-06-30 13:22:33 +00:00
Kseniia Sumarokova
f6742f61cb
Merge branch 'master' into fix-logical-error-read-unexpected-size 2023-06-30 13:02:49 +02:00
Sema Checherinda
e724397d13
Merge branch 'master' into less-logs-2 2023-06-30 08:02:22 +02:00
Sema Checherinda
2a1f34e3f9
Merge branch 'master' into no-finalize-WriteBufferFromOStream 2023-06-30 08:01:05 +02:00
Alexey Milovidov
9a9f992489
Merge branch 'master' into no-finalize-WriteBufferFromOStream 2023-06-29 23:13:32 +03:00
Dmitry Kardymon
54838e92d4 Remove whitespace 2023-06-29 13:20:38 +00:00
kssenii
b56d1602e8 Invalidate opened file cache when removing file 2023-06-29 14:25:13 +02:00
Dmitry Kardymon
bdfaffb6e2 Add/fix tests 2023-06-29 12:08:08 +00:00
Sema Checherinda
919bf54294 fix race condition 2023-06-29 12:38:46 +02:00
Alexey Milovidov
efb901245b
Merge branch 'master' into ft-thread-scheduling-stacktrace 2023-06-29 11:38:54 +03:00
Igor Nikonov
def0adf8f4
Merge branch 'master' into many-fixes-2 2023-06-29 10:33:59 +02:00
Dmitry Kardymon
ffa4f37c9a Try to fix style 2023-06-29 06:59:42 +00:00
avogar
4eeb431003 Merge branch 'master' of github.com:ClickHouse/ClickHouse into better-progress-bar-2 2023-06-28 18:53:08 +00:00
Nikita Mikhaylov
48e8475511
Merge pull request #51484 from kssenii/try-data-race-in-bounded-read-buffer 
Fix race in BoundedReadBuffer
 2023-06-27 19:51:09 +02:00
kssenii
648b647f5f Fix race 2023-06-27 15:54:06 +02:00
Nikita Mikhaylov
d24c5ab01f
Merge branch 'master' into fix-ip-aggregate-state 2023-06-27 14:11:26 +02:00
Harry Lee
cb13022583
Merge branch 'master' into hlee-s390x-proto-endian 2023-06-26 15:13:04 -04:00
Sema Checherinda
ae08fb2019 no finalize in d-tor WriteBufferFromOStream 2023-06-26 12:02:09 +02:00
taiyang-li
d208b0de3d add example for test 2023-06-26 16:50:18 +08:00
taiyang-li
1d2600f706 add example for test 2023-06-26 16:47:13 +08:00
Sema Checherinda
91c374fcd2
Merge branch 'master' into less-logs-2 2023-06-26 07:55:23 +02:00
Sema Checherinda
79a03432bf add test, add comment 2023-06-25 13:27:07 +02:00
Alexey Milovidov
cca3f9937c
Merge branch 'master' into retry 2023-06-24 22:29:17 +03:00
Yakov Olkhovskiy
f11e1c25ee added to fatal errors handler, added comments 2023-06-24 17:24:54 +00:00
robot-clickhouse-ci-1
c66ff09627
Merge pull request #51253 from kssenii/delete-non-evictable-feature 
Delete feature do_not_evict_index_and_mark_files in fs cache
 2023-06-24 02:37:52 +02:00
Sema Checherinda
5b6dabdc34 introduce LogSeriesLimiterPtrt 2023-06-23 23:06:32 +02:00
avogar
938a9ee3c1 Better 2023-06-23 14:03:59 +00:00
avogar
c679dd400e Make better 2023-06-23 13:43:40 +00:00
Sema Checherinda
977cd03cf2
Merge branch 'master' into memory-leak 2023-06-23 15:35:53 +02:00
Robert Schulze
43c97d76bf
Merge pull request #44149 from zvonand/zvonand-implicit-tz 
Session time zones
 2023-06-23 14:28:49 +02:00
Sema Checherinda
9680596d36 decrease log level, make logs shorter 2023-06-23 13:31:54 +02:00
Alexey Milovidov
7407330130
Merge branch 'master' into retry 2023-06-23 08:18:18 +03:00
Sema Checherinda
2b01711565 fix assert in test, revert debug message 2023-06-22 21:06:53 +02:00
avogar
cf082f2f9a Use read_bytes/total_bytes_to_read for progress bar in s3/file/url/... table functions 2023-06-22 17:24:43 +00:00
Andrey Zvonov
d4316c7189
Merge branch 'master' into zvonand-implicit-tz 2023-06-22 17:23:20 +02:00
Alexander Tokmakov
35c5483ad2
Update WriteBufferFromOStream.cpp 2023-06-22 14:56:20 +03:00
Alexander Tokmakov
6929a9b852
Update WriteBufferFromOStream.cpp 2023-06-22 14:01:28 +03:00
Michael Kolupaev
c96a3a8b15 Unforget to pass callback to readBigAt() in ParallelReadBuffer 2023-06-21 17:14:19 -07:00
Sema Checherinda
23120c4467 fix build 2023-06-21 22:32:33 +02:00
Sema Checherinda
c2fa7a1030
Merge branch 'master' into memory-leak 2023-06-21 21:15:01 +02:00
Sema Checherinda
01de36f1fa
Merge pull request #50395 from CheSema/better-log 
require `finalize()` call before d-tor for all writes buffers
 2023-06-21 21:12:02 +02:00
kssenii
9aa4647f4f Delete feature 2023-06-21 20:48:12 +02:00
Michael Kolupaev
4a570a05c9 Decrease default timeouts for S3 and HTTP requests 2023-06-21 18:08:50 +00:00
Andrey Zvonov
b4f2e4020d
Merge branch 'master' into zvonand-implicit-tz 2023-06-21 13:41:35 +02:00
Kruglov Pavel
bbd70932e2
Merge pull request #51158 from Avogar/extend-progress-protocol 
Add total_bytes_to_read to Progress packet in TCP protocol for better progress bar
 2023-06-21 12:43:45 +02:00
Kruglov Pavel
8f8cd97fd8
Merge pull request #51088 from Avogar/better-progress-bar 
Improve progress bar for file/s3/hdfs/url table functions. Step 1
 2023-06-21 12:42:25 +02:00
Sema Checherinda
4d08df8d29 fix finalization of cascade write buffer 2023-06-21 12:27:39 +02:00
Sema Checherinda
9b0c3359cf
Merge branch 'master' into better-log 2023-06-20 20:37:36 +02:00
Nikita Taranov
876d5ae0a7 fix ReadBufferFromS3 2023-06-20 20:27:43 +02:00
Sema Checherinda
fd292dc730 work with comment on the PR 2023-06-20 20:02:04 +02:00
Andrey Zvonov
07191ce10d
Merge branch 'master' into zvonand-implicit-tz 2023-06-20 00:24:53 +02:00
Sergei Trifonov
5867f39fc7
Merge pull request #50675 from ClickHouse/unify-priorities-pools 
Unify priorities for connection pools
 2023-06-19 17:16:15 +02:00
Yakov Olkhovskiy
4004be74dc
Merge branch 'master' into fix-ip-aggregate-state 2023-06-19 10:58:06 -04:00
avogar
cf8f277c32 Add total_bytes_to_read to TCP protocol for better Progress bar 2023-06-19 13:58:08 +00:00
Sema Checherinda
ce7cd67981 finalize CascadeWriteBuffer 2023-06-18 17:35:12 +02:00
Michael Kolupaev
03baa19e88 Fix ParallelReadBuffer seek 2023-06-16 17:50:51 -07:00
Sema Checherinda
e8d4dedbcc fix build and tests 2023-06-16 19:52:52 +02:00
Yakov Olkhovskiy
cf301324fb
Merge branch 'master' into fix-ip-aggregate-state 2023-06-16 12:49:17 -04:00
avogar
3209ebe34b Improve progress bar for file/s3/hdfs/url table functions. Step 1 2023-06-16 15:51:18 +00:00
Salvatore Mesoraca
6b49968bc4
Generate safe IVs 
IVs should never be re-used when using CTR.
The previous implementation had a 50% probability to generate
an IV that was already used after 65536 generations.
This happens because std::mt19937 is seeded using a 32bit integer
and returns a 32 bits integer.
Because of the birthday problem, collisions have a 50% chance after
only 2^16 IV's generations.
CH uses one IV per file and, for most use cases, 65k files are
not as many as they seem.
Also, the initial entropy is gathered using std::random_device
which is "best-effort" and it's allowed by the standard to
return a fixed sequence of numbers.
 2023-06-16 17:09:08 +02:00
Sema Checherinda
47484faace do abort/cancel/deletion merge/mutate task with exception context 2023-06-16 16:48:21 +02:00
Sema Checherinda
cf31fdb4d5 no finalize at d-tor in compression 2023-06-16 16:38:18 +02:00
Sema Checherinda
d21fc22607 fix 02428_combinators_with_over_statement 2023-06-16 16:38:18 +02:00
Sema Checherinda
1cb02e2710 do call finalize for all buffers 2023-06-16 16:38:18 +02:00
Sema Checherinda
9a4962bd68 add error log message when no finalize called before d-tor 2023-06-16 16:38:18 +02:00
Nikita Taranov
b546d8e665 review fixes + test 2023-06-16 15:30:56 +02:00
zvonand
4155d13d69 merge master and resolve conflict 2023-06-16 02:20:32 +02:00
Robert Schulze
74cb79769b
Merge pull request #50925 from arenadata/ADQM-812 
Implement support of syslog format in the parseDateTimeBestEffort() function
 2023-06-15 21:04:50 +02:00
Kseniia Sumarokova
31e08635bf
Merge branch 'master' into unify-priorities-pools 2023-06-15 12:51:45 +02:00
Sema Checherinda
7d59af1f77 test compressed write to S3 2023-06-14 14:28:57 +02:00
Nikita Taranov
1dddcc9472 use connection pool 2023-06-14 12:51:02 +02:00
Kseniia Sumarokova
e2d8299b23
Merge pull request #50952 from nickitat/fix_remote_read_perf_degr 
Fix logic in `AsynchronousBoundedReadBuffer::seek`
 2023-06-14 11:25:26 +02:00
Antonio Andelic
4825286b0c
Merge branch 'master' into keeper-with-disks 2023-06-14 10:48:53 +02:00