3210 Commits

Author	SHA1	Message	Date
Robert Schulze	acaa2eee58	Merge pull request #53556 from ClibMouse/feature/big-endian-UUID-correction ... Change Big Endian-UUID to work the same as Little Endian-UUID	2023-08-22 11:57:44 +02:00
Sema Checherinda	ae5f66da1e	fix special build	2023-08-22 11:34:08 +02:00
Michael Kolupaev	6009e1b293	Merge pull request #53324 from bigo-sg/ch_gluten_2583 ... Implement native orc input format without arrow to improve performance	2023-08-21 13:44:57 -07:00
Alexey Milovidov	316664456f	Merge pull request #53588 from kitaisreal/aarch64-neon-memequal-wide ... AARCH64 Neon memequal wide	2023-08-21 21:11:01 +03:00
Sema Checherinda	81577e041b	fix Logical Error in AsynchronousBoundedReadBuffer	2023-08-21 19:30:04 +02:00
kothiga	52d47785cb	Fix deserialization for quantile deterministic.	2023-08-21 09:52:14 -07:00
Antonio Andelic	ed15babddd	Merge pull request #53198 from ClickHouse/improve-archive-reading ... Improve reading from archives	2023-08-21 14:11:25 +02:00
Maksim Kita	f32effb627	AARCH64 Neon memequal wide	2023-08-20 12:45:18 +03:00
gyfis	efa53f873c	💄 [S3::URI] Fix comment typos around versionId	2023-08-20 09:08:57 +02:00
Alexey Milovidov	d133cafd9c	Better documentation about redirects	2023-08-20 00:38:18 +02:00
Yakov Olkhovskiy	257380df03	Merge pull request #50267 from HarryLeeIBM/hlee-s390x-proto-endian ... Fix endian issues in native protocol	2023-08-19 11:16:24 -04:00
kothiga	f33c585bc5	Addressing feedback.	2023-08-18 13:50:31 -07:00
Austin Kothig	6b42975d33	Change BE-UUID to work the same as LE-UUID. Included high and low getters to provide cleaner code when accessing undertype.	2023-08-18 08:19:46 -07:00
Harry Lee	1645439000	Merge branch 'master' into hlee-s390x-proto-endian	2023-08-17 18:09:04 -04:00
Michael Kolupaev	557bfea4d8	Use long timeout for S3 copy requests	2023-08-17 19:36:45 +00:00
Robert Schulze	ec7daed52f	Merge pull request #50276 from ClibMouse/feature/mergetree-checksum-big-endian-support ... MergeTree & SipHash checksum big-endian support	2023-08-17 13:27:18 +02:00
taiyang-li	bfa9d361cc	fix tests	2023-08-15 12:16:22 +08:00
ltrk2	2846ea49b4	Merge branch 'master' into feature/mergetree-checksum-big-endian-support	2023-08-14 13:02:30 -04:00
Alexey Milovidov	7911945a74	Make one exception message longer	2023-08-13 22:40:50 +02:00
robot-ch-test-poll	967067bbb8	Merge pull request #53250 from evillique/postgresql-single-quotes ... Fix character escaping in the PostgreSQL engine	2023-08-12 13:15:49 +02:00
Alexey Milovidov	a3208f0a9a	Merge branch 'master' into ditch-tons-of-garbage-2	2023-08-11 00:20:43 +02:00
Antonio Andelic	478a5006b1	Merge branch 'master' into improve-archive-reading	2023-08-10 15:52:30 +00:00
ltrk2	a2054c04dd	Merge branch 'master' into feature/mergetree-checksum-big-endian-support	2023-08-10 10:21:34 -04:00
Alexey Milovidov	bb57caa83e	Merge pull request #53180 from ClickHouse/ditch-tons-of-garbage ... Ditch tons of garbage	2023-08-10 16:59:51 +03:00
Kruglov Pavel	6600f87f86	Merge branch 'master' into http-valid-json-on-exception	2023-08-10 13:53:32 +02:00
Antonio Andelic	aa258a0e61	Merge branch 'master' into improve-archive-reading	2023-08-10 06:59:04 +00:00
Nikolay Degterinsky	7321f5e543	Better	2023-08-10 06:34:10 +00:00
Alexey Milovidov	52c3704dc7	Merge pull request #52683 from rschu1ze/readBinary-specializations ... Remove unneeded readBinary() specializations + update docs	2023-08-10 04:04:57 +03:00
ltrk2	139e9433a8	Merge branch 'master' into feature/mergetree-checksum-big-endian-support	2023-08-09 09:48:00 -04:00
Antonio Andelic	0e17d26b88	More formats supported, read single archive from 1 thread	2023-08-09 11:58:37 +00:00
Alexey Milovidov	1c9cff551e	Continue on garbage removal	2023-08-09 02:58:12 +02:00
Alexey Milovidov	aa757490bd	Ditch tons of garbage	2023-08-09 02:19:02 +02:00
Michael Kolupaev	e4b2b3f533	Merge pull request #52828 from ClickHouse/illogical ... Don't report LOGICAL_ERROR if a file got truncated during read	2023-08-08 13:13:20 -07:00
Alexander Tokmakov	9d29b7cdbf	Merge pull request #53109 from ClickHouse/tavplubix-patch-8 ... Update PocoHTTPClient.cpp	2023-08-07 19:33:39 +03:00
Antonio Andelic	f5597b1f8e	Revert changes from https://github.com/ClickHouse/ClickHouse/pull/53064	2023-08-07 12:21:50 +00:00
Alexander Tokmakov	d35dc46061	Update PocoHTTPClient.cpp	2023-08-07 14:39:58 +03:00
Kseniia Sumarokova	b1f0cb83ff	Merge pull request #52621 from valbok/readbuffer ... MaterializedMySQL: Fix the infinite loop in ReadBuffer::read	2023-08-07 11:56:23 +02:00
Antonio Andelic	cafa5ef2f3	Merge branch 'master' into add-reading-from-archives	2023-08-06 10:21:47 +00:00
Antonio Andelic	1d793e69f2	Fix	2023-08-05 17:06:44 +00:00
Alexey Milovidov	70630a273a	Merge branch 'master' into readBinary-specializations	2023-08-05 03:16:29 +03:00
Michael Kolupaev	9a128fa554	Fix ZstdDeflatingWriteBuffer truncating the output sometimes	2023-08-04 20:47:45 +00:00
Antonio Andelic	49ae853ae6	Fix sanitizer warnings	2023-08-04 13:57:18 +00:00
Val Doroshchuk	3cea474ccf	MaterializedMySQL: Fix the infinite loop in ReadBuffer::read ... This fixes https://github.com/ClickHouse/ClickHouse/issues/33549 The infinite loop caused because sometimes the pos of ReadBuffer overflows the end. It happens f.e. when `MySQLPacketPayloadReadBuffer::nextImpl` might return empty buffer, because `in.nextIfAtEnd();` could not read more bytes and thus no bytes available (pos == end). It might happen when a network error or the connection to MySQL was closed or killed. This leads to empty `working_buffer` but successful returning from the func. And check `if (in.eof())` from `MySQLBinlogEventReadBuffer::nextImpl()` fails and also causes empty its `working_buffer` and successul return. At the end `payload.ignore(1)` and `payload.read(c)` produces the infinite loop since it is not eof() and ++pos overflows the end of the buffer. Should be tested by `test_mysql_kill*` from test.py	2023-08-04 11:34:31 +02:00
Antonio Andelic	8e7eb7f3fa	Fix tests	2023-08-04 08:59:55 +00:00
Antonio Andelic	27701ed24e	Small fixes	2023-08-03 11:44:24 +00:00
Antonio Andelic	d9f136ce0c	Add tests for globs	2023-08-03 10:42:47 +00:00
Antonio Andelic	96bad0537a	Merge branch 'master' into add-reading-from-archives	2023-08-03 08:36:42 +00:00
ltrk2	27a2d4d1c7	Merge branch 'master' into feature/mergetree-checksum-big-endian-support	2023-08-02 11:36:43 -04:00
Antonio Andelic	9423976b7a	Add support for file globs	2023-08-02 07:42:56 +00:00
Yakov Olkhovskiy	357fee99ff	Merge pull request #52791 from zvonand/zvonand-float-parsing ... Add option to switch float parsing methods	2023-08-02 00:23:23 -04:00
avogar	d12e96177a	Fix tests	2023-08-01 16:17:03 +00:00
Antonio Andelic	e366ca61f5	Merge branch 'master' into add-reading-from-archives	2023-08-01 15:33:20 +00:00
Kruglov Pavel	23aab71d7c	Merge branch 'master' into http-valid-json-on-exception	2023-08-01 16:47:31 +02:00
Antonio Andelic	c24dc8e64e	Multithreaded read of files from archives:	2023-08-01 13:48:49 +00:00
avogar	a71cd56a90	Output valid JSON/XML on excetpion during HTTP query execution	2023-08-01 10:06:56 +00:00
Andrey Zvonov	99cd8f9ba9	Merge branch 'master' into zvonand-float-parsing	2023-07-31 23:43:24 +03:00
Nikolai Kochetov	c4a00b8c68	Throw S3Exception whenever possible.	2023-07-31 15:39:20 +00:00
Michael Kolupaev	8b0f52f411	Don't report LOGICAL_ERROR if a file got truncated during read	2023-07-31 15:04:30 +00:00
Antonio Andelic	86a251931b	Fix style	2023-07-31 14:53:40 +00:00
zvonand	e50ffa1181	fix style	2023-07-31 16:44:46 +02:00
zvonand	2088f846ad	update as per reviews	2023-07-31 16:26:18 +02:00
Antonio Andelic	6a1cf5c683	Extend tests	2023-07-31 13:27:06 +00:00
Antonio Andelic	7df5c83046	Merge branch 'master' into add-reading-from-archives	2023-07-31 09:27:01 +00:00
Antonio Andelic	df45602165	Refactoring	2023-07-31 08:50:09 +00:00
Alexey Milovidov	c825f15b74	Merge pull request #52441 from CheSema/reset-connection-s3 ... add tests with connection reset by peer error, and retry it inside client	2023-07-31 09:58:41 +03:00
zvonand	0ff0a4e41f	diverge parsing methods	2023-07-30 19:42:10 +02:00
ltrk2	6c9a1b14ef	Merge branch 'master' into feature/mergetree-checksum-big-endian-support	2023-07-28 16:18:46 -04:00
Antonio Andelic	1fc1b6aae4	More fixes	2023-07-28 13:00:35 +00:00
Antonio Andelic	e83e0ec2cd	Fix build	2023-07-28 12:26:56 +00:00
Sema Checherinda	115a085d0f	Merge branch 'master' into reset-connection-s3	2023-07-28 14:00:14 +04:00
Antonio Andelic	720d587e85	Merge branch 'master' into add-reading-from-archives	2023-07-28 08:49:00 +00:00
Robert Schulze	b98cb7fa14	Fix build	2023-07-27 17:50:39 +00:00
Robert Schulze	8c14c4b5e8	Remove unneeded readBinary() specializations + update docs ... The default instantiation template <typename T> requires is_arithmetic_v<T> inline void readBinary(T & x, ReadBuffer & buf) { readPODBinary(x, buf); } covers Int32, Int128, Int256, UInt32, UInt128 and UInt256 cases already.	2023-07-27 15:13:54 +00:00
Alexey Milovidov	6aab4cc835	Check for unexpected cyrillic	2023-07-27 05:25:40 +02:00
robot-ch-test-poll2	7bcef0a6c0	Merge pull request #52520 from zvonand/revert-52450-remove-to-decimal-string ... Fix `toDecimalString` function	2023-07-27 00:18:36 +02:00
Sema Checherinda	7d430b8037	retry ConnectionResetException	2023-07-26 17:31:14 +04:00
Kruglov Pavel	15cc046883	Merge branch 'master' into better-progress-bar-2	2023-07-26 13:12:24 +02:00
Sema Checherinda	83d869860e	Merge pull request #51946 from arenadata/ADQM-940 ... Expose Peak memory usage in query statistics.	2023-07-25 22:17:58 +03:00
Sema Checherinda	c341df1949	draft with retry ConnectionResetException	2023-07-25 19:59:39 +04:00
Andrey Zvonov	15b3a637e9	Merge branch 'master' into revert-52450-remove-to-decimal-string	2023-07-25 13:21:00 +03:00
Alexey Milovidov	21382afa2b	Check for punctuation	2023-07-25 06:10:04 +02:00
Alexey Milovidov	168b84a592	Merge pull request #52337 from Avogar/no-decode-url ... Allow to disable decoding/encoding path in uri in URL engine	2023-07-25 05:43:06 +03:00
Dmitry Kardymon	d4d381de7e	Merge remote-tracking branch 'origin/master' into ADQM-940	2023-07-24 20:04:10 +00:00
Kruglov Pavel	fec5675cd4	Merge branch 'master' into better-progress-bar-2	2023-07-24 19:59:38 +02:00
Nikita Taranov	c6e6fd7613	Shard OpenedFileCache to avoid lock contention (#51341) ... * shard OpenedFileCache to avoid lock contention * Update OpenedFileCache.h * fix build --------- Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>	2023-07-24 15:58:21 +02:00
Andrey Zvonov	21097209d2	Revert "Remove toDecimalString"	2023-07-24 15:41:21 +03:00
Dmitry Kardymon	62f9a953fe	Merge remote-tracking branch 'origin/master' into ADQM-940	2023-07-24 04:52:12 +00:00
Alexey Milovidov	d7cdfb47d3	Fix merge	2023-07-24 01:56:04 +02:00
Alexey Milovidov	5c15e6021b	Merge branch 'master' into remove-to-decimal-string	2023-07-24 01:51:43 +02:00
Alexey Milovidov	2467eaf32d	Merge pull request #52392 from azat/netlink-provider ... Rename TaskStatsInfoGetter into NetlinkMetricsProvider	2023-07-22 19:14:15 +03:00
Kseniia Sumarokova	d34f2bed07	Merge pull request #51386 from bigo-sg/fix_hdfs_read_buffer_heap_overflow ... Fix heap overflow in read buffer from hdfs when read_until_position is not zero	2023-07-22 15:00:58 +02:00
Alexey Milovidov	5f4756fb33	Remove toDecimalString	2023-07-22 04:54:58 +02:00
ltrk2	90a2c460c6	Merge branch 'master' into feature/mergetree-checksum-big-endian-support	2023-07-21 08:07:18 -04:00
Azat Khuzhin	f0e277f94a	Rename TaskStatsInfoGetter into NetlinkMetricsProvider ... There is ProcfsMetricsProvider, so by analogy to it. Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>	2023-07-21 06:48:18 +02:00
Alexander Tokmakov	a39ba00ec3	Merge pull request #52116 from aalexfvk/http_failover ... Handle multiple resolved IPs for HTTP/HTTPS session connection	2023-07-20 18:25:45 +03:00
Alexander Tokmakov	e467264588	Update src/IO/HTTPCommon.cpp	2023-07-20 17:56:30 +03:00
avogar	2b8e4ebd4c	Allow to disable decoding/encoding path in uri in URL engine	2023-07-19 19:48:39 +00:00
ltrk2	ba4072f049	Adapt changes around SipHash	2023-07-19 10:01:58 -07:00
ltrk2	51e2c58a53	Implement endianness-independent SipHash and MergeTree checksum serialization	2023-07-19 10:01:55 -07:00
Dmitry Kardymon	68a501a14e	Merge remote-tracking branch 'origin/master' into ADQM-940	2023-07-18 18:16:09 +00:00
Kruglov Pavel	64e88cde21	Merge branch 'master' into better-progress-bar-2	2023-07-18 13:37:53 +02:00
Andrey Zvonov	f2e68d1d7e	Merge branch 'master' into zvonand-issue-42138	2023-07-16 01:06:41 +03:00
Vitaly Baranov	2f23b6a593	Merge branch 'master' into backups-native-copy	2023-07-15 08:40:17 +02:00
Aleksei Filatov	08defa36b2	Add code	2023-07-15 08:18:35 +03:00
zvonand	cbe0a0e748	upd threshold due to dt64 range change	2023-07-15 03:04:56 +02:00
李扬	6aec25e681	Merge branch 'master' into fix_hdfs_read_buffer_heap_overflow	2023-07-11 10:03:09 +08:00
Dmitry Kardymon	09ba975ffc	Merge remote-tracking branch 'origin/master' into ADQM-940	2023-07-10 13:32:29 +00:00
Azat Khuzhin	1844ac37d7	Remove create_read_buffer argument for copyS3File() ... Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>	2023-07-09 08:20:03 +02:00
Azat Khuzhin	1590ffa3b1	Remove copyS3FileNative() ... Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>	2023-07-09 08:20:03 +02:00
Azat Khuzhin	5835e72fd6	More generic approach to disable native copy ... Previous patch implements this only for BACKUP/RESTORE, but it can be useful for regular disks as well, so add allow_native_copy for disks. Note, that there is s3_allow_native_copy query setting, since it looks redundant, since it make sense only for S3 disks, and not on a per query basis. Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>	2023-07-09 08:20:02 +02:00
Alexey Milovidov	59eadca95c	Merge branch 'master' into less-logs-2	2023-07-09 08:49:44 +03:00
Alexey Milovidov	3d4800995f	Merge pull request #49732 from nickitat/impr_prefetch ... Improve reading with prefetch	2023-07-09 06:10:58 +03:00
Alexey Milovidov	b1d1918607	Merge branch 'master' into retry	2023-07-08 03:08:48 +03:00
Dmitry Kardymon	3fcfbb1038	Merge remote-tracking branch 'clickhouse/master' into ADQM-940	2023-07-07 15:09:41 +03:00
Robert Schulze	271297823a	Allow var-int encoded 64-bit integers with MSB=1 ... Resolves: #51486 Until now, it was illegal to encode 64-bit (unsigned) integers with MSB=1, i.e. values > (1ULL<<63) - 1, as var-int. In more detail, the var-int code used by ClickHouse server and client spent at most 9 bytes per value such that 9 * 7 = 63 bits could be encoded. Some 3rd party clients (e.g. Rust clickhouse-rs) had the same limitation, whereas other clients understand the full range (Python clickhouse-driver). PRs #47608 and #48628 added sanity checks as asserts or exceptions during var-int encoding on the server side. This was considered okay as such huge integers so far occurred only during testing (usually fuzzing) but not in practice. Issue #51486 is a new fuzzing issue where the exception thrown from the sanity check led to a half-baked progress packet and as a result, a logical error / server crash. The only fix which is not another bandaid is to allow the full range in var-int coding. Clients will have to allow the full range too, a note will be added to the changelog. (the alternative was to create another protocol version but as var-int is used all over the place this was considered infeasible) Review note: this is the relevant commit.	2023-07-06 20:23:23 +00:00
Nikita Taranov	63b9c1ac06	add test	2023-07-06 18:58:13 +02:00
Robert Schulze	3f744c1e14	Cosmetics: rename template parameter	2023-07-06 14:47:40 +00:00
Robert Schulze	7644f0b37c	Cosmetics: move code around	2023-07-06 14:44:06 +00:00
Nikita Taranov	aec7205636	rework pool usage	2023-07-06 14:41:09 +02:00
Nikita Taranov	c23e29d6aa	don't account session's memory in thread/user mem tracker	2023-07-06 14:41:03 +02:00
Sema Checherinda	8f7381f0a2	Merge branch 'master' into less-logs-2	2023-07-06 12:07:58 +02:00
Kruglov Pavel	60cf8a4c94	Merge branch 'master' into better-progress-bar-2	2023-07-05 19:58:55 +02:00
李扬	3b860cd685	Merge branch 'master' into fix_hdfs_read_buffer_heap_overflow	2023-07-04 10:04:57 +08:00
Antonio Andelic	eba60dd81e	Merge branch 'master' into keeper-with-disks	2023-07-01 20:59:16 +02:00
Sema Checherinda	f6ad94d5bc	Merge branch 'master' into no-finalize-WriteBufferFromOStream	2023-06-30 22:12:31 +02:00
Yakov Olkhovskiy	023e40535a	Merge pull request #51349 from ClickHouse/ft-thread-scheduling-stacktrace ... Collect stack traces from job's scheduling and print along with exception's stack trace.	2023-06-30 11:01:58 -04:00
Sema Checherinda	c45fed58fc	Merge branch 'master' into less-logs-2	2023-06-30 15:55:17 +02:00
Antonio Andelic	2058b1346e	Merge branch 'master' into keeper-with-disks	2023-06-30 13:22:33 +00:00
Kseniia Sumarokova	f6742f61cb	Merge branch 'master' into fix-logical-error-read-unexpected-size	2023-06-30 13:02:49 +02:00
Sema Checherinda	e724397d13	Merge branch 'master' into less-logs-2	2023-06-30 08:02:22 +02:00
Sema Checherinda	2a1f34e3f9	Merge branch 'master' into no-finalize-WriteBufferFromOStream	2023-06-30 08:01:05 +02:00
Alexey Milovidov	9a9f992489	Merge branch 'master' into no-finalize-WriteBufferFromOStream	2023-06-29 23:13:32 +03:00
Dmitry Kardymon	54838e92d4	Remove whitespace	2023-06-29 13:20:38 +00:00
kssenii	b56d1602e8	Invalidate opened file cache when removing file	2023-06-29 14:25:13 +02:00
Dmitry Kardymon	bdfaffb6e2	Add/fix tests	2023-06-29 12:08:08 +00:00
Sema Checherinda	919bf54294	fix race condition	2023-06-29 12:38:46 +02:00
Alexey Milovidov	efb901245b	Merge branch 'master' into ft-thread-scheduling-stacktrace	2023-06-29 11:38:54 +03:00
Igor Nikonov	def0adf8f4	Merge branch 'master' into many-fixes-2	2023-06-29 10:33:59 +02:00
Dmitry Kardymon	ffa4f37c9a	Try to fix style	2023-06-29 06:59:42 +00:00
avogar	4eeb431003	Merge branch 'master' of github.com:ClickHouse/ClickHouse into better-progress-bar-2	2023-06-28 18:53:08 +00:00
Nikita Mikhaylov	48e8475511	Merge pull request #51484 from kssenii/try-data-race-in-bounded-read-buffer ... Fix race in BoundedReadBuffer	2023-06-27 19:51:09 +02:00
kssenii	648b647f5f	Fix race	2023-06-27 15:54:06 +02:00
Nikita Mikhaylov	d24c5ab01f	Merge branch 'master' into fix-ip-aggregate-state	2023-06-27 14:11:26 +02:00
Harry Lee	cb13022583	Merge branch 'master' into hlee-s390x-proto-endian	2023-06-26 15:13:04 -04:00
Sema Checherinda	ae08fb2019	no finalize in d-tor WriteBufferFromOStream	2023-06-26 12:02:09 +02:00
taiyang-li	d208b0de3d	add example for test	2023-06-26 16:50:18 +08:00
taiyang-li	1d2600f706	add example for test	2023-06-26 16:47:13 +08:00
Sema Checherinda	91c374fcd2	Merge branch 'master' into less-logs-2	2023-06-26 07:55:23 +02:00
Sema Checherinda	79a03432bf	add test, add comment	2023-06-25 13:27:07 +02:00
Alexey Milovidov	cca3f9937c	Merge branch 'master' into retry	2023-06-24 22:29:17 +03:00
Yakov Olkhovskiy	f11e1c25ee	added to fatal errors handler, added comments	2023-06-24 17:24:54 +00:00
robot-clickhouse-ci-1	c66ff09627	Merge pull request #51253 from kssenii/delete-non-evictable-feature ... Delete feature do_not_evict_index_and_mark_files in fs cache	2023-06-24 02:37:52 +02:00
Sema Checherinda	5b6dabdc34	introduce LogSeriesLimiterPtrt	2023-06-23 23:06:32 +02:00
avogar	938a9ee3c1	Better	2023-06-23 14:03:59 +00:00
avogar	c679dd400e	Make better	2023-06-23 13:43:40 +00:00
Sema Checherinda	977cd03cf2	Merge branch 'master' into memory-leak	2023-06-23 15:35:53 +02:00
Robert Schulze	43c97d76bf	Merge pull request #44149 from zvonand/zvonand-implicit-tz ... Session time zones	2023-06-23 14:28:49 +02:00
Sema Checherinda	9680596d36	decrease log level, make logs shorter	2023-06-23 13:31:54 +02:00
Alexey Milovidov	7407330130	Merge branch 'master' into retry	2023-06-23 08:18:18 +03:00
Sema Checherinda	2b01711565	fix assert in test, revert debug message	2023-06-22 21:06:53 +02:00
avogar	cf082f2f9a	Use read_bytes/total_bytes_to_read for progress bar in s3/file/url/... table functions	2023-06-22 17:24:43 +00:00
Andrey Zvonov	d4316c7189	Merge branch 'master' into zvonand-implicit-tz	2023-06-22 17:23:20 +02:00
Alexander Tokmakov	35c5483ad2	Update WriteBufferFromOStream.cpp	2023-06-22 14:56:20 +03:00
Alexander Tokmakov	6929a9b852	Update WriteBufferFromOStream.cpp	2023-06-22 14:01:28 +03:00
Michael Kolupaev	c96a3a8b15	Unforget to pass callback to readBigAt() in ParallelReadBuffer	2023-06-21 17:14:19 -07:00
Sema Checherinda	23120c4467	fix build	2023-06-21 22:32:33 +02:00
Sema Checherinda	c2fa7a1030	Merge branch 'master' into memory-leak	2023-06-21 21:15:01 +02:00
Sema Checherinda	01de36f1fa	Merge pull request #50395 from CheSema/better-log ... require `finalize()` call before d-tor for all writes buffers	2023-06-21 21:12:02 +02:00
kssenii	9aa4647f4f	Delete feature	2023-06-21 20:48:12 +02:00
Michael Kolupaev	4a570a05c9	Decrease default timeouts for S3 and HTTP requests	2023-06-21 18:08:50 +00:00
Andrey Zvonov	b4f2e4020d	Merge branch 'master' into zvonand-implicit-tz	2023-06-21 13:41:35 +02:00
Kruglov Pavel	bbd70932e2	Merge pull request #51158 from Avogar/extend-progress-protocol ... Add total_bytes_to_read to Progress packet in TCP protocol for better progress bar	2023-06-21 12:43:45 +02:00
Kruglov Pavel	8f8cd97fd8	Merge pull request #51088 from Avogar/better-progress-bar ... Improve progress bar for file/s3/hdfs/url table functions. Step 1	2023-06-21 12:42:25 +02:00
Sema Checherinda	4d08df8d29	fix finalization of cascade write buffer	2023-06-21 12:27:39 +02:00
Sema Checherinda	9b0c3359cf	Merge branch 'master' into better-log	2023-06-20 20:37:36 +02:00
Nikita Taranov	876d5ae0a7	fix ReadBufferFromS3	2023-06-20 20:27:43 +02:00
Sema Checherinda	fd292dc730	work with comment on the PR	2023-06-20 20:02:04 +02:00
Andrey Zvonov	07191ce10d	Merge branch 'master' into zvonand-implicit-tz	2023-06-20 00:24:53 +02:00
Sergei Trifonov	5867f39fc7	Merge pull request #50675 from ClickHouse/unify-priorities-pools ... Unify priorities for connection pools	2023-06-19 17:16:15 +02:00
Yakov Olkhovskiy	4004be74dc	Merge branch 'master' into fix-ip-aggregate-state	2023-06-19 10:58:06 -04:00
avogar	cf8f277c32	Add total_bytes_to_read to TCP protocol for better Progress bar	2023-06-19 13:58:08 +00:00
Sema Checherinda	ce7cd67981	finalize CascadeWriteBuffer	2023-06-18 17:35:12 +02:00
Michael Kolupaev	03baa19e88	Fix ParallelReadBuffer seek	2023-06-16 17:50:51 -07:00
Sema Checherinda	e8d4dedbcc	fix build and tests	2023-06-16 19:52:52 +02:00
Yakov Olkhovskiy	cf301324fb	Merge branch 'master' into fix-ip-aggregate-state	2023-06-16 12:49:17 -04:00
avogar	3209ebe34b	Improve progress bar for file/s3/hdfs/url table functions. Step 1	2023-06-16 15:51:18 +00:00
Salvatore Mesoraca	6b49968bc4	Generate safe IVs ... IVs should never be re-used when using CTR. The previous implementation had a 50% probability to generate an IV that was already used after 65536 generations. This happens because std::mt19937 is seeded using a 32bit integer and returns a 32 bits integer. Because of the birthday problem, collisions have a 50% chance after only 2^16 IV's generations. CH uses one IV per file and, for most use cases, 65k files are not as many as they seem. Also, the initial entropy is gathered using std::random_device which is "best-effort" and it's allowed by the standard to return a fixed sequence of numbers.	2023-06-16 17:09:08 +02:00
Sema Checherinda	47484faace	do abort/cancel/deletion merge/mutate task with exception context	2023-06-16 16:48:21 +02:00
Sema Checherinda	cf31fdb4d5	no finalize at d-tor in compression	2023-06-16 16:38:18 +02:00
Sema Checherinda	d21fc22607	fix 02428_combinators_with_over_statement	2023-06-16 16:38:18 +02:00
Sema Checherinda	1cb02e2710	do call finalize for all buffers	2023-06-16 16:38:18 +02:00
Sema Checherinda	9a4962bd68	add error log message when no finalize called before d-tor	2023-06-16 16:38:18 +02:00
Nikita Taranov	b546d8e665	review fixes + test	2023-06-16 15:30:56 +02:00
zvonand	4155d13d69	merge master and resolve conflict	2023-06-16 02:20:32 +02:00
Robert Schulze	74cb79769b	Merge pull request #50925 from arenadata/ADQM-812 ... Implement support of syslog format in the parseDateTimeBestEffort() function	2023-06-15 21:04:50 +02:00
Kseniia Sumarokova	31e08635bf	Merge branch 'master' into unify-priorities-pools	2023-06-15 12:51:45 +02:00
Sema Checherinda	7d59af1f77	test compressed write to S3	2023-06-14 14:28:57 +02:00
Nikita Taranov	1dddcc9472	use connection pool	2023-06-14 12:51:02 +02:00
Kseniia Sumarokova	e2d8299b23	Merge pull request #50952 from nickitat/fix_remote_read_perf_degr ... Fix logic in `AsynchronousBoundedReadBuffer::seek`	2023-06-14 11:25:26 +02:00
Antonio Andelic	4825286b0c	Merge branch 'master' into keeper-with-disks	2023-06-14 10:48:53 +02:00
Nikita Taranov	1bc5598aa7	impl	2023-06-13 20:02:50 +02:00
Victor Krasnov	f01b96f9f9	Merge branch 'master' into ADQM-812	2023-06-13 19:52:10 +03:00
Robert Schulze	8358d29ac7	Merge pull request #50405 from ClibMouse/feature/reservoir-sampler-big-endian-support ... Implement big-endian support for the deterministic reservoir sampler	2023-06-13 09:55:23 +02:00
Victor Krasnov	b3ef2a4860	Merge branch 'master' of github.com:ClickHouse/ClickHouse into ADQM-812	2023-06-13 03:55:38 +00:00
Antonio Andelic	cb3ac81110	Merge branch 'master' into keeper-with-disks	2023-06-12 15:14:24 +02:00
Alexey Milovidov	ed9c7c9841	Merge pull request #50711 from kssenii/get-rid-of-uneeded-code ... Refactor reading from object storages	2023-06-10 19:10:25 +03:00
Yakov Olkhovskiy	b9df4304d4	Merge branch 'master' into fix-ip-aggregate-state	2023-06-09 10:06:43 -04:00
Andrey Zvonov	d95be4162f	Merge branch 'master' into zvonand-implicit-tz	2023-06-08 18:34:45 +03:00
Yakov Olkhovskiy	393c0c3a4f	Merge pull request #50240 from aalexfvk/fix_https_replication_changed_ip_bug ... Fix reconnecting of HTTPS session when target host IP was changed	2023-06-08 08:58:48 -04:00
kssenii	1cbcd2f2ef	Refactor reading from object storages	2023-06-08 14:33:59 +02:00
serxa	2df0e0c669	Unify priorities for connections	2023-06-07 16:25:52 +00:00
Antonio Andelic	42798e446b	Merge branch 'master' into keeper-with-disks	2023-06-06 14:55:47 +00:00
Nikita Mikhaylov	e87348010d	Rework loading and removing of data parts for MergeTree tables. (#49474) ... Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>	2023-06-06 14:42:56 +02:00
Victor Krasnov	3f77b778e3	Fix runtime bug	2023-06-06 11:42:45 +03:00
Robert Schulze	2e16b497f5	Merge pull request #50519 from ClibMouse/feature/uuid-serialization ... Implement endianness-independent serialization for UUID	2023-06-06 09:18:19 +02:00
Victor Krasnov	6d25e5a0d7	Substitute missing year in parseDateTimeBestEffortImpl()	2023-06-06 07:37:14 +03:00
Yakov Olkhovskiy	760483d8d3	multiple fixes	2023-06-06 04:18:51 +00:00
Michael Kolupaev	9f80900d6f	Changes related to an internal feature	2023-06-05 18:20:38 -07:00
ltrk2	35439a8b06	Use reverse iterator for little-endian version	2023-06-05 10:47:52 -07:00
ltrk2	3938309374	Implement review comments	2023-06-05 08:18:03 -07:00
Alexey Milovidov	fb4e950f9f	Merge pull request #50546 from auxten/patch-1 ... Resize BufferFromVector underlying vector only pos_offset == vector.size()	2023-06-05 11:12:57 +03:00
Antonio Andelic	c4873027c3	Merge branch 'master' into keeper-with-disks	2023-06-05 07:08:33 +00:00
Alexey Milovidov	d66962b864	Merge pull request #50252 from Avogar/fix-secure-async-read-write ... Fix bugs in Poco sockets in non-blocking mode, use true non-blocking sockets	2023-06-05 06:44:22 +03:00
auxten	e5c95add52	use old_size ... Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>	2023-06-05 08:43:55 +08:00
Alexey Gerasimchuk	9958731c27	Merge branch 'master' into ADQM-830	2023-06-05 07:46:47 +10:00
auxten	db806bd394	Resize underlying vector only pos_offset == vector.size()	2023-06-04 17:44:29 +08:00
Alexey Milovidov	8c1091dc58	Merge branch 'master' into fix-secure-async-read-write	2023-06-04 05:06:59 +03:00
Sema Checherinda	2177e562b8	Merge pull request #50518 from CheSema/lesser-traces ... less traces in logs	2023-06-03 10:57:45 +02:00
ltrk2	50654435dc	Implement endianness-independent serialization for UUID	2023-06-02 19:36:37 +00:00
Sema Checherinda	991d1b97fc	less traces in logs	2023-06-02 20:48:31 +02:00
Kseniia Sumarokova	7b0d1531c2	Merge pull request #50444 from CheSema/less-logs-second-try ... Revert "Revert "less logs in WriteBufferFromS3" (#50390)"	2023-06-02 18:43:44 +02:00
Antonio Andelic	5f625dcdd9	Merge branch 'master' into keeper-with-disks	2023-06-02 13:35:18 +00:00
Kruglov Pavel	9922053419	Merge pull request #49948 from Avogar/handshake-timeout ... Add separate handshake_timeout for receiving Hello packet from replica	2023-06-02 14:07:27 +02:00
Michael Kolupaev	b51064a508	Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead	2023-06-01 18:48:30 -07:00
Andrey Zvonov	2e427ee028	Merge branch 'master' into zvonand-implicit-tz	2023-06-02 01:15:56 +03:00
Sema Checherinda	02e986a9e7	Revert "Revert "less logs in WriteBufferFromS3" (#50390)" ... This reverts commit 4d4112ff53.	2023-06-01 19:06:11 +02:00
Kruglov Pavel	e924aafdf6	Merge branch 'master' into fix-secure-async-read-write	2023-06-01 16:24:54 +02:00
Antonio Andelic	514beb783b	Merge branch 'master' into keeper-with-disks	2023-06-01 12:19:43 +00:00
Sema Checherinda	de62c9e558	Merge pull request #50410 from ClickHouse/clang-17-2 ... Support clang-17	2023-06-01 10:48:58 +02:00
Alexey Milovidov	c54abdb4f8	Support clang-17	2023-06-01 01:32:03 +02:00
ltrk2	f2e076a443	Implement big-endian support for the deterministic reservoir sampler	2023-05-31 19:33:32 +00:00
Sema Checherinda	aedd3afb8a	fix hung in unit tests (#50391) ... * fix hung in unit tests * Update gtest_writebuffer_s3.cpp * Update gtest_writebuffer_s3.cpp --------- Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>	2023-05-31 19:20:58 +03:00
Kruglov Pavel	0beca0336d	Merge pull request #49112 from ClickHouse/Avogar-patch-3 ... Fix possible terminate called for uncaught exception in some places	2023-05-31 16:55:43 +02:00
Alexander Tokmakov	4d4112ff53	Revert "less logs in WriteBufferFromS3" (#50390)	2023-05-31 15:26:56 +03:00
Kruglov Pavel	ec10415328	Merge branch 'master' into fix-secure-async-read-write	2023-05-31 13:16:52 +02:00
Vitaly Baranov	6d45d0c374	Use fingerprints instead of key IDs in encrypted disks (#49882) ... * Use fingerprints instead of key IDs to find keys in encrypted disks. Always use little endian in the headers of encryption files. * Add tests. * Fix copying binary files to test containers. * Fix ownership for copied files in test containers. * Add comments after review. --------- Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>	2023-05-31 13:11:10 +02:00
Sema Checherinda	ab027ca1ce	Merge pull request #50275 from CheSema/d-tor-free ... buffers d-tor finalize free	2023-05-31 12:26:19 +02:00
Alexey Gerasimchuk	44ba35d2c1	Merge branch 'master' into ADQM-830	2023-05-31 15:07:37 +10:00
Sema Checherinda	91a3c88126	less logs in WriteBufferFromS3	2023-05-30 15:49:47 +02:00
Sema Checherinda	0a128cec61	remove tricky debug trap	2023-05-30 12:28:58 +02:00
Sema Checherinda	0b4ea3e2e1	remove reminder comments	2023-05-30 12:26:20 +02:00
Sema Checherinda	b13990efcc	fix build	2023-05-30 12:26:20 +02:00
Sema Checherinda	13dcb62ffb	fix logs engins	2023-05-30 12:26:20 +02:00
Sema Checherinda	a5dcd8dabb	do not call finalize in d-tor MergedBlockOutputStream::Finalizer	2023-05-30 12:26:20 +02:00
Sema Checherinda	d95e5b51af	mark all finalize calls in buffers d-tors with issue id	2023-05-30 12:26:20 +02:00
Sema Checherinda	fe3939287b	add test, fix .gin_sid files	2023-05-30 12:26:20 +02:00
Sema Checherinda	0e019c8e83	turn off some d-tor finalize	2023-05-30 12:26:20 +02:00
Sema Checherinda	100b4d0969	cope with finalize in d-tors	2023-05-30 12:26:20 +02:00
Andrey Zvonov	87fc780c14	Merge branch 'master' into zvonand-implicit-tz	2023-05-30 13:13:46 +03:00
Antonio Andelic	67746bfff2	Merge branch 'master' into keeper-with-disks	2023-05-30 09:50:12 +00:00
nikitakeba	f604fb82b2	Merge branch 'master' into add-reading-from-archives-support	2023-05-29 23:34:19 +03:00
Nikita Keba	04450a2042	add CheckFileInfo Unit Tests	2023-05-29 20:28:15 +00:00
Nikita Keba	c18bff58b3	fix style	2023-05-29 20:08:18 +00:00
Nikita Keba	8cf79cdb6c	add SevenZipArchiveReader unit tests	2023-05-29 19:55:46 +00:00
Nikita Keba	636d50caa0	fix cmake + add unit tests for TarArchiveReader	2023-05-29 19:35:24 +00:00
Nikita Keba	19a0fbeccc	add SevenZipArchiveReader/TarArchiveReader	2023-05-29 17:15:33 +00:00
alesapin	6b7e2a0041	Merge pull request #49961 from CheSema/mem-consumption-s3-write ... Introduce a limitation of async calls inside s3 write buffer, make a test that memory is bounded even when s3 is slow	2023-05-29 13:29:01 +02:00
Alexey Milovidov	ef7e9e28b7	Merge pull request #50231 from ClickHouse/inverse-io-scheduler-priority-3 ... Unify priorities: rework IO scheduling subsystem	2023-05-29 04:34:23 +03:00
Vitaly Baranov	0e2156c776	Fix checksum calculation for backup entries (#50264) ... * Fix possible deadlock in calculating backup entries' checksums. * Fix data type used to store checksums. * Allow ReadBufferFromMemory::seek() to go to the end of the buffer (seek(0, SEEK_SET) must be allowed for an empty buffer) * Add tests. --------- Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>	2023-05-27 14:35:37 +02:00
Sema Checherinda	86f85d7b3b	cope with finalize in d-tors	2023-05-27 01:18:01 +02:00
avogar	ef9bae50b9	Fix bugs in Poco, use true non-blocking IO	2023-05-26 23:11:57 +00:00
Alexey Gerasimchuk	12105c3dfa	Merge branch 'master' into ADQM-830	2023-05-27 07:35:05 +10:00
Sema Checherinda	1588219ffd	comments	2023-05-26 19:03:59 +02:00
Sema Checherinda	8523cd54a0	working with review comments	2023-05-26 19:00:22 +02:00
Sema Checherinda	7c9d2ab6d0	working with review comments	2023-05-26 19:00:13 +02:00
Sema Checherinda	b0eff95388	update doc, add profile event WriteBufferFromS3WaitInflightLimitMicroseconds	2023-05-26 18:59:21 +02:00
Sema Checherinda	0b781bd4f1	fix test_merge_canceled_by_s3_errors	2023-05-26 18:59:21 +02:00
Sema Checherinda	206efee9b7	add test with merge tree table	2023-05-26 18:59:21 +02:00
Sema Checherinda	4249bda449	s3_max_inflight_parts_for_one_file with tests	2023-05-26 18:59:11 +02:00
HarryLeeIBM	fa3f8bc32e	Fix endian issue in native protocol	2023-05-26 07:04:44 -07:00
Sergei Trifonov	0d1f2e297b	Unify priorities: lower value means higher priority (#50205)	2023-05-26 15:55:30 +02:00
Antonio Andelic	8a2a63a7bd	Merge branch 'master' into keeper-with-disks	2023-05-26 10:48:41 +00:00
Kruglov Pavel	67c8c5c561	Add missing include	2023-05-26 12:44:43 +02:00
Kruglov Pavel	03652efe58	Add missing include	2023-05-26 12:44:28 +02:00
Aleksei Filatov	2a2c35e4c1	Fix changed IP for https session	2023-05-26 09:08:34 +03:00
Kruglov Pavel	18be731e30	Merge branch 'master' into fix-secure-async-read-write	2023-05-26 00:46:33 +02:00
Kruglov Pavel	f03ca41b08	Fix build	2023-05-26 00:21:46 +02:00
Kruglov Pavel	67b78829fc	Fix build	2023-05-26 00:21:14 +02:00
Kruglov Pavel	a9082b24b4	Fix build	2023-05-26 00:20:20 +02:00
Alexey Gerasimchuk	8d7cb7fc3b	Merge branch 'master' into ADQM-830	2023-05-26 07:49:51 +10:00
Kruglov Pavel	1964d1bb7e	Fix comment	2023-05-25 22:30:16 +02:00
avogar	42e1e3ae20	Fix working with secure socket after async connection	2023-05-25 20:24:03 +00:00
serxa	0ca526c603	Unify priorities: rework IO scheduling subsystem	2023-05-25 13:25:41 +00:00
serxa	3ef6cb7bdc	git-apply #50205	2023-05-25 13:24:45 +00:00
Kseniia Sumarokova	f1a3c9cfd5	Merge pull request #50109 from kssenii/abstract-async-prefetched-buffer ... Make async prefetched buffer work with arbitrary impl	2023-05-25 15:06:44 +02:00
Alexey Gerasimchuck	75791d7a63	Added input_format_csv_trim_whitespaces parameter	2023-05-25 07:51:32 +00:00
Antonio Andelic	fa3fb49d33	Merge branch 'master' into keeper-with-disks	2023-05-24 09:04:47 +00:00
Antonio Andelic	161afea266	Add support for changelog	2023-05-24 07:43:57 +00:00
avogar	3c1aeaaa79	Change default value of handshake_timeout to 10 sec, fix possible use-after-free	2023-05-23 11:39:40 +00:00
Antonio Andelic	3e6314675c	Merge pull request #49930 from AVMusorin/write-buffer-from-s3 ... Fix metrics `WriteBufferFromS3Bytes`, `WriteBufferFromS3Microseconds` and `WriteBufferFromS3RequestsErrors`	2023-05-23 12:26:05 +02:00