Merge branch 'master' into igor/replicated_merge_tree_some_polishing

CHANGELOG.md

@@ -1,6 +1,6 @@
 ### Table of Contents
 **[ClickHouse release v22.9, 2022-09-22](#229)**<br/>
-**[ClickHouse release v22.8, 2022-08-18](#228)**<br/>
+**[ClickHouse release v22.8-lts, 2022-08-18](#228)**<br/>
 **[ClickHouse release v22.7, 2022-07-21](#227)**<br/>
 **[ClickHouse release v22.6, 2022-06-16](#226)**<br/>
 **[ClickHouse release v22.5, 2022-05-19](#225)**<br/>
@@ -10,10 +10,10 @@
 **[ClickHouse release v22.1, 2022-01-18](#221)**<br/>
 **[Changelog for 2021](https://clickhouse.com/docs/en/whats-new/changelog/2021/)**<br/>
 
-
 ### <a id="229"></a> ClickHouse release 22.9, 2022-09-22
 
 #### Backward Incompatible Change
+
 * Upgrade from 20.3 and older to 22.9 and newer should be done through an intermediate version if there are any `ReplicatedMergeTree` tables, otherwise server with the new version will not start. [#40641](https://github.com/ClickHouse/ClickHouse/pull/40641) ([Alexander Tokmakov](https://github.com/tavplubix)).
 * Remove the functions `accurate_Cast` and `accurate_CastOrNull` (they are different to `accurateCast` and `accurateCastOrNull` by underscore in the name and they are not affected by the value of `cast_keep_nullable` setting). These functions were undocumented, untested, unused, and unneeded. They appeared to be alive due to code generalization. [#40682](https://github.com/ClickHouse/ClickHouse/pull/40682) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Add a test to ensure that every new table function will be documented. See [#40649](https://github.com/ClickHouse/ClickHouse/issues/40649). Rename table function `MeiliSearch` to `meilisearch`. [#40709](https://github.com/ClickHouse/ClickHouse/pull/40709) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
@@ -21,6 +21,7 @@
 * Make interpretation of YAML configs to be more conventional. [#41044](https://github.com/ClickHouse/ClickHouse/pull/41044) ([Vitaly Baranov](https://github.com/vitlibar)).
 
 #### New Feature
+
 * Support `insert_quorum = 'auto'` to use majority number. [#39970](https://github.com/ClickHouse/ClickHouse/pull/39970) ([Sachin](https://github.com/SachinSetiya)).
 * Add embedded dashboards to ClickHouse server. This is a demo project about how to achieve 90% results with 1% effort using ClickHouse features. [#40461](https://github.com/ClickHouse/ClickHouse/pull/40461) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Added new settings constraint writability kind `changeable_in_readonly`. [#40631](https://github.com/ClickHouse/ClickHouse/pull/40631) ([Sergei Trifonov](https://github.com/serxa)).
@@ -38,6 +39,7 @@
 * Improvement for in-memory data parts: remove completely processed WAL files. [#40592](https://github.com/ClickHouse/ClickHouse/pull/40592) ([Azat Khuzhin](https://github.com/azat)).
 
 #### Performance Improvement
+
 * Implement compression of marks and primary key. Close [#34437](https://github.com/ClickHouse/ClickHouse/issues/34437). [#37693](https://github.com/ClickHouse/ClickHouse/pull/37693) ([zhongyuankai](https://github.com/zhongyuankai)).
 * Allow to load marks with threadpool in advance. Regulated by setting `load_marks_asynchronously` (default: 0). [#40821](https://github.com/ClickHouse/ClickHouse/pull/40821) ([Kseniia Sumarokova](https://github.com/kssenii)).
 * Virtual filesystem over s3 will use random object names split into multiple path prefixes for better performance on AWS. [#40968](https://github.com/ClickHouse/ClickHouse/pull/40968) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
@@ -58,6 +60,7 @@
 * Parallel hash JOIN for Float data types might be suboptimal. Make it better. [#41183](https://github.com/ClickHouse/ClickHouse/pull/41183) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 
 #### Improvement
+
 * During startup and ATTACH call, `ReplicatedMergeTree` tables will be readonly until the ZooKeeper connection is made and the setup is finished. [#40148](https://github.com/ClickHouse/ClickHouse/pull/40148) ([Antonio Andelic](https://github.com/antonio2368)).
 * Add `enable_extended_results_for_datetime_functions` option to return results of type Date32 for functions toStartOfYear, toStartOfISOYear, toStartOfQuarter, toStartOfMonth, toStartOfWeek, toMonday and toLastDayOfMonth when argument is Date32 or DateTime64, otherwise results of Date type are returned. For compatibility reasons default value is ‘0’. [#41214](https://github.com/ClickHouse/ClickHouse/pull/41214) ([Roman Vasin](https://github.com/rvasin)).
 * For security and stability reasons, CatBoost models are no longer evaluated within the ClickHouse server. Instead, the evaluation is now done in the clickhouse-library-bridge, a separate process that loads the catboost library and communicates with the server process via HTTP. [#40897](https://github.com/ClickHouse/ClickHouse/pull/40897) ([Robert Schulze](https://github.com/rschu1ze)). [#39629](https://github.com/ClickHouse/ClickHouse/pull/39629) ([Robert Schulze](https://github.com/rschu1ze)).
@@ -108,6 +111,7 @@
 * Add `has_lightweight_delete` to system.parts. [#41564](https://github.com/ClickHouse/ClickHouse/pull/41564) ([Kseniia Sumarokova](https://github.com/kssenii)).
 
 #### Build/Testing/Packaging Improvement
+
 * Enforce documentation for every setting. [#40644](https://github.com/ClickHouse/ClickHouse/pull/40644) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Enforce documentation for every current metric. [#40645](https://github.com/ClickHouse/ClickHouse/pull/40645) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Enforce documentation for every profile event counter. Write the documentation where it was missing. [#40646](https://github.com/ClickHouse/ClickHouse/pull/40646) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
@@ -217,15 +221,16 @@
 * Fix read bytes/rows in X-ClickHouse-Summary with materialized views. [#41586](https://github.com/ClickHouse/ClickHouse/pull/41586) ([Raúl Marín](https://github.com/Algunenano)).
 * Fix possible `pipeline stuck` exception for queries with `OFFSET`. The error was found with `enable_optimize_predicate_expression = 0` and always false condition in `WHERE`. Fixes [#41383](https://github.com/ClickHouse/ClickHouse/issues/41383). [#41588](https://github.com/ClickHouse/ClickHouse/pull/41588) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
 
-
-### <a id="228"></a> ClickHouse release 22.8, 2022-08-18
+### <a id="228"></a> ClickHouse release 22.8-lts, 2022-08-18
 
 #### Backward Incompatible Change
+
 * Extended range of `Date32` and `DateTime64` to support dates from the year 1900 to 2299. In previous versions, the supported interval was only from the year 1925 to 2283. The implementation is using the proleptic Gregorian calendar (which is conformant with [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601):2004 (clause 3.2.1 The Gregorian calendar)) instead of accounting for historical transitions from the Julian to the Gregorian calendar. This change affects implementation-specific behavior for out-of-range arguments. E.g. if in previous versions the value of `1899-01-01` was clamped to `1925-01-01`, in the new version it will be clamped to `1900-01-01`. It changes the behavior of rounding with `toStartOfInterval` if you pass `INTERVAL 3 QUARTER` up to one quarter because the intervals are counted from an implementation-specific point of time. Closes [#28216](https://github.com/ClickHouse/ClickHouse/issues/28216), improves [#38393](https://github.com/ClickHouse/ClickHouse/issues/38393). [#39425](https://github.com/ClickHouse/ClickHouse/pull/39425) ([Roman Vasin](https://github.com/rvasin)).
 * Now, all relevant dictionary sources respect `remote_url_allow_hosts` setting. It was already done for HTTP, Cassandra, Redis. Added ClickHouse, MongoDB, MySQL, PostgreSQL. Host is checked only for dictionaries created from DDL. [#39184](https://github.com/ClickHouse/ClickHouse/pull/39184) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
 * Make the remote filesystem cache composable, allow not to evict certain files (regarding idx, mrk, ..), delete old cache version. Now it is possible to configure cache over Azure blob storage disk, over Local disk, over StaticWeb disk, etc. This PR is marked backward incompatible because cache configuration changes and in order for cache to work need to update the config file. Old cache will still be used with new configuration. The server will startup fine with the old cache configuration. Closes https://github.com/ClickHouse/ClickHouse/issues/36140. Closes https://github.com/ClickHouse/ClickHouse/issues/37889. ([Kseniia Sumarokova](https://github.com/kssenii)). [#36171](https://github.com/ClickHouse/ClickHouse/pull/36171))
 
 #### New Feature
+
 * Query parameters can be set in interactive mode as `SET param_abc = 'def'` and transferred via the native protocol as settings. [#39906](https://github.com/ClickHouse/ClickHouse/pull/39906) ([Nikita Taranov](https://github.com/nickitat)).
 * Quota key can be set in the native protocol ([Yakov Olkhovsky](https://github.com/ClickHouse/ClickHouse/pull/39874)).
 * Added a setting `exact_rows_before_limit` (0/1). When enabled, ClickHouse will provide exact value for `rows_before_limit_at_least` statistic, but with the cost that the data before limit will have to be read completely. This closes [#6613](https://github.com/ClickHouse/ClickHouse/issues/6613). [#25333](https://github.com/ClickHouse/ClickHouse/pull/25333) ([kevin wan](https://github.com/MaxWk)).
@@ -240,12 +245,14 @@
 * Add new setting schema_inference_hints that allows to specify structure hints in schema inference for specific columns. Closes [#39569](https://github.com/ClickHouse/ClickHouse/issues/39569). [#40068](https://github.com/ClickHouse/ClickHouse/pull/40068) ([Kruglov Pavel](https://github.com/Avogar)).
 
 #### Experimental Feature
+
 * Support SQL standard DELETE FROM syntax on merge tree tables and lightweight delete implementation for merge tree families. [#37893](https://github.com/ClickHouse/ClickHouse/pull/37893) ([Jianmei Zhang](https://github.com/zhangjmruc)) ([Alexander Gololobov](https://github.com/davenger)). Note: this new feature does not make ClickHouse an HTAP DBMS.
 
 #### Performance Improvement
+
 * Improved memory usage during memory efficient merging of aggregation results. [#39429](https://github.com/ClickHouse/ClickHouse/pull/39429) ([Nikita Taranov](https://github.com/nickitat)).
 * Added concurrency control logic to limit total number of concurrent threads created by queries. [#37558](https://github.com/ClickHouse/ClickHouse/pull/37558) ([Sergei Trifonov](https://github.com/serxa)). Add `concurrent_threads_soft_limit parameter` to increase performance in case of high QPS by means of limiting total number of threads for all queries. [#37285](https://github.com/ClickHouse/ClickHouse/pull/37285) ([Roman Vasin](https://github.com/rvasin)).
-* Add `SLRU` cache policy for uncompressed cache and marks cache. ([Kseniia Sumarokova](https://github.com/kssenii)). [#34651](https://github.com/ClickHouse/ClickHouse/pull/34651) ([alexX512](https://github.com/alexX512)). Decoupling local cache function and cache algorithm [#38048](https://github.com/ClickHouse/ClickHouse/pull/38048) ([Han Shukai](https://github.com/KinderRiven)). 
+* Add `SLRU` cache policy for uncompressed cache and marks cache. ([Kseniia Sumarokova](https://github.com/kssenii)). [#34651](https://github.com/ClickHouse/ClickHouse/pull/34651) ([alexX512](https://github.com/alexX512)). Decoupling local cache function and cache algorithm [#38048](https://github.com/ClickHouse/ClickHouse/pull/38048) ([Han Shukai](https://github.com/KinderRiven)).
 * Intel® In-Memory Analytics Accelerator (Intel® IAA) is a hardware accelerator available in the upcoming generation of Intel® Xeon® Scalable processors ("Sapphire Rapids"). Its goal is to speed up common operations in analytics like data (de)compression and filtering. ClickHouse gained the new "DeflateQpl" compression codec which utilizes the Intel® IAA offloading technology to provide a high-performance DEFLATE implementation. The codec uses the [Intel® Query Processing Library (QPL)](https://github.com/intel/qpl) which abstracts access to the hardware accelerator, respectively to a software fallback in case the hardware accelerator is not available. DEFLATE provides in general higher compression rates than ClickHouse's LZ4 default codec, and as a result, offers less disk I/O and lower main memory consumption. [#36654](https://github.com/ClickHouse/ClickHouse/pull/36654) ([jasperzhu](https://github.com/jinjunzh)). [#39494](https://github.com/ClickHouse/ClickHouse/pull/39494) ([Robert Schulze](https://github.com/rschu1ze)).
 * `DISTINCT` in order with `ORDER BY`: Deduce way to sort based on input stream sort description. Skip sorting if input stream is already sorted. [#38719](https://github.com/ClickHouse/ClickHouse/pull/38719) ([Igor Nikonov](https://github.com/devcrafter)). Improve memory usage (significantly) and query execution time + use `DistinctSortedChunkTransform` for final distinct when `DISTINCT` columns match `ORDER BY` columns, but rename to `DistinctSortedStreamTransform` in `EXPLAIN PIPELINE` → this improves memory usage significantly + remove unnecessary allocations in hot loop in `DistinctSortedChunkTransform`. [#39432](https://github.com/ClickHouse/ClickHouse/pull/39432) ([Igor Nikonov](https://github.com/devcrafter)). Use `DistinctSortedTransform` only when sort description is applicable to DISTINCT columns, otherwise fall back to ordinary DISTINCT implementation + it allows making less checks during `DistinctSortedTransform` execution. [#39528](https://github.com/ClickHouse/ClickHouse/pull/39528) ([Igor Nikonov](https://github.com/devcrafter)). Fix: `DistinctSortedTransform` didn't take advantage of sorting. It never cleared HashSet since clearing_columns were detected incorrectly (always empty). So, it basically worked as ordinary `DISTINCT` (`DistinctTransform`). The fix reduces memory usage significantly. [#39538](https://github.com/ClickHouse/ClickHouse/pull/39538) ([Igor Nikonov](https://github.com/devcrafter)).
 * Use local node as first priority to get structure of remote table when executing `cluster` and similar table functions. [#39440](https://github.com/ClickHouse/ClickHouse/pull/39440) ([Mingliang Pan](https://github.com/liangliangpan)).
@@ -256,6 +263,7 @@
 * Improve bytes to bits mask transform for SSE/AVX/AVX512. [#39586](https://github.com/ClickHouse/ClickHouse/pull/39586) ([Guo Wangyang](https://github.com/guowangy)).
 
 #### Improvement
+
 * Normalize `AggregateFunction` types and state representations because optimizations like [#35788](https://github.com/ClickHouse/ClickHouse/pull/35788) will treat `count(not null columns)` as `count()`, which might confuses distributed interpreters with the following error : `Conversion from AggregateFunction(count) to AggregateFunction(count, Int64) is not supported`. [#39420](https://github.com/ClickHouse/ClickHouse/pull/39420) ([Amos Bird](https://github.com/amosbird)). The functions with identical states can be used in materialized views interchangeably.
 * Rework and simplify the `system.backups` table, remove the `internal` column, allow user to set the ID of operation, add columns `num_files`, `uncompressed_size`, `compressed_size`, `start_time`, `end_time`. [#39503](https://github.com/ClickHouse/ClickHouse/pull/39503) ([Vitaly Baranov](https://github.com/vitlibar)).
 * Improved structure of DDL query result table for `Replicated` database (separate columns with shard and replica name, more clear status) - `CREATE TABLE ... ON CLUSTER` queries can be normalized on initiator first if `distributed_ddl_entry_format_version` is set to 3 (default value). It means that `ON CLUSTER` queries may not work if initiator does not belong to the cluster that specified in query. Fixes [#37318](https://github.com/ClickHouse/ClickHouse/issues/37318), [#39500](https://github.com/ClickHouse/ClickHouse/issues/39500) - Ignore `ON CLUSTER` clause if database is `Replicated` and cluster name equals to database name. Related to [#35570](https://github.com/ClickHouse/ClickHouse/issues/35570) - Miscellaneous minor fixes for `Replicated` database engine - Check metadata consistency when starting up `Replicated` database, start replica recovery in case of mismatch of local metadata and metadata in Keeper. Resolves [#24880](https://github.com/ClickHouse/ClickHouse/issues/24880). [#37198](https://github.com/ClickHouse/ClickHouse/pull/37198) ([Alexander Tokmakov](https://github.com/tavplubix)).
@@ -294,6 +302,7 @@
 * Add support for LARGE_BINARY/LARGE_STRING with Arrow (Closes [#32401](https://github.com/ClickHouse/ClickHouse/issues/32401)). [#40293](https://github.com/ClickHouse/ClickHouse/pull/40293) ([Josh Taylor](https://github.com/joshuataylor)).
 
 #### Build/Testing/Packaging Improvement
+
 * [ClickFiddle](https://fiddle.clickhouse.com/): A new tool for testing ClickHouse versions in read/write mode (**Igor Baliuk**).
 * ClickHouse binary is made self-extracting [#35775](https://github.com/ClickHouse/ClickHouse/pull/35775) ([Yakov Olkhovskiy, Arthur Filatenkov](https://github.com/yakov-olkhovskiy)).
 * Update tzdata to 2022b to support the new timezone changes. See https://github.com/google/cctz/pull/226. Chile's 2022 DST start is delayed from September 4 to September 11. Iran plans to stop observing DST permanently, after it falls back on 2022-09-21. There are corrections of the historical time zone of Asia/Tehran in the year 1977: Iran adopted standard time in 1935, not 1946. In 1977 it observed DST from 03-21 23:00 to 10-20 24:00; its 1978 transitions were on 03-24 and 08-05, not 03-20 and 10-20; and its spring 1979 transition was on 05-27, not 03-21 (https://data.iana.org/time-zones/tzdb/NEWS). ([Alexey Milovidov](https://github.com/alexey-milovidov)).
@@ -308,6 +317,7 @@
 * Docker: Now entrypoint.sh in docker image creates and executes chown for all folders it found in config for multidisk setup [#17717](https://github.com/ClickHouse/ClickHouse/issues/17717). [#39121](https://github.com/ClickHouse/ClickHouse/pull/39121) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
 
 #### Bug Fix
+
 * Fix possible segfault in `CapnProto` input format. This bug was found and send through ClickHouse bug-bounty [program](https://github.com/ClickHouse/ClickHouse/issues/38986) by *kiojj*. [#40241](https://github.com/ClickHouse/ClickHouse/pull/40241) ([Kruglov Pavel](https://github.com/Avogar)).
 * Fix a very rare case of incorrect behavior of array subscript operator. This closes [#28720](https://github.com/ClickHouse/ClickHouse/issues/28720). [#40185](https://github.com/ClickHouse/ClickHouse/pull/40185) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Fix insufficient argument check for encryption functions (found by query fuzzer). This closes [#39987](https://github.com/ClickHouse/ClickHouse/issues/39987). [#40194](https://github.com/ClickHouse/ClickHouse/pull/40194) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
@@ -358,16 +368,17 @@
 * A fix for reverse DNS resolution. [#40134](https://github.com/ClickHouse/ClickHouse/pull/40134) ([Arthur Passos](https://github.com/arthurpassos)).
 * Fix unexpected result `arrayDifference` of `Array(UInt32). [#40211](https://github.com/ClickHouse/ClickHouse/pull/40211) ([Duc Canh Le](https://github.com/canhld94)).
 
-
 ### <a id="227"></a> ClickHouse release 22.7, 2022-07-21
 
 #### Upgrade Notes
+
 * Enable setting `enable_positional_arguments` by default. It allows queries like `SELECT ... ORDER BY 1, 2` where 1, 2 are the references to the select clause. If you need to return the old behavior, disable this setting. [#38204](https://github.com/ClickHouse/ClickHouse/pull/38204) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Disable `format_csv_allow_single_quotes` by default. See [#37096](https://github.com/ClickHouse/ClickHouse/issues/37096). ([Kruglov Pavel](https://github.com/Avogar)).
 * `Ordinary` database engine and old storage definition syntax for `*MergeTree` tables are deprecated. By default it's not possible to create new databases with `Ordinary` engine. If `system` database has `Ordinary` engine it will be automatically converted to `Atomic` on server startup. There are settings to keep old behavior (`allow_deprecated_database_ordinary` and `allow_deprecated_syntax_for_merge_tree`), but these settings may be removed in future releases. [#38335](https://github.com/ClickHouse/ClickHouse/pull/38335) ([Alexander Tokmakov](https://github.com/tavplubix)).
 * Force rewriting comma join to inner by default (set default value `cross_to_inner_join_rewrite = 2`). To have old behavior set `cross_to_inner_join_rewrite = 1`. [#39326](https://github.com/ClickHouse/ClickHouse/pull/39326) ([Vladimir C](https://github.com/vdimir)). If you will face any incompatibilities, you can turn this setting back.
 
 #### New Feature
+
 * Support expressions with window functions. Closes [#19857](https://github.com/ClickHouse/ClickHouse/issues/19857). [#37848](https://github.com/ClickHouse/ClickHouse/pull/37848) ([Dmitry Novik](https://github.com/novikd)).
 * Add new `direct` join algorithm for `EmbeddedRocksDB` tables, see [#33582](https://github.com/ClickHouse/ClickHouse/issues/33582). [#35363](https://github.com/ClickHouse/ClickHouse/pull/35363) ([Vladimir C](https://github.com/vdimir)).
 * Added full sorting merge join algorithm. [#35796](https://github.com/ClickHouse/ClickHouse/pull/35796) ([Vladimir C](https://github.com/vdimir)).
@@ -395,9 +406,11 @@
 * Add `clickhouse-diagnostics` binary to the packages. [#38647](https://github.com/ClickHouse/ClickHouse/pull/38647) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
 
 #### Experimental Feature
+
 * Adds new setting `implicit_transaction` to run standalone queries inside a transaction. It handles both creation and closing (via COMMIT if the query succeeded or ROLLBACK if it didn't) of the transaction automatically. [#38344](https://github.com/ClickHouse/ClickHouse/pull/38344) ([Raúl Marín](https://github.com/Algunenano)).
 
 #### Performance Improvement
+
 * Distinct optimization for sorted columns. Use specialized distinct transformation in case input stream is sorted by column(s) in distinct. Optimization can be applied to pre-distinct, final distinct, or both. Initial implementation by @dimarub2000. [#37803](https://github.com/ClickHouse/ClickHouse/pull/37803) ([Igor Nikonov](https://github.com/devcrafter)).
 * Improve performance of `ORDER BY`, `MergeTree` merges, window functions using batch version of `BinaryHeap`. [#38022](https://github.com/ClickHouse/ClickHouse/pull/38022) ([Maksim Kita](https://github.com/kitaisreal)).
 * More parallel execution for queries with `FINAL` [#36396](https://github.com/ClickHouse/ClickHouse/pull/36396) ([Nikita Taranov](https://github.com/nickitat)).
@@ -407,7 +420,7 @@
 * Improve performance of insertion to columns of type `JSON`. [#38320](https://github.com/ClickHouse/ClickHouse/pull/38320) ([Anton Popov](https://github.com/CurtizJ)).
 * Optimized insertion and lookups in the HashTable. [#38413](https://github.com/ClickHouse/ClickHouse/pull/38413) ([Nikita Taranov](https://github.com/nickitat)).
 * Fix performance degradation from [#32493](https://github.com/ClickHouse/ClickHouse/issues/32493). [#38417](https://github.com/ClickHouse/ClickHouse/pull/38417) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
-* Improve performance of joining with numeric columns using SIMD instructions. [#37235](https://github.com/ClickHouse/ClickHouse/pull/37235) ([zzachimed](https://github.com/zzachimed)). [#38565](https://github.com/ClickHouse/ClickHouse/pull/38565) ([Maksim Kita](https://github.com/kitaisreal)). 
+* Improve performance of joining with numeric columns using SIMD instructions. [#37235](https://github.com/ClickHouse/ClickHouse/pull/37235) ([zzachimed](https://github.com/zzachimed)). [#38565](https://github.com/ClickHouse/ClickHouse/pull/38565) ([Maksim Kita](https://github.com/kitaisreal)).
 * Norm and Distance functions for arrays speed up 1.2-2 times. [#38740](https://github.com/ClickHouse/ClickHouse/pull/38740) ([Alexander Gololobov](https://github.com/davenger)).
 * Add AVX-512 VBMI optimized `copyOverlap32Shuffle` for LZ4 decompression. In other words, LZ4 decompression performance is improved. [#37891](https://github.com/ClickHouse/ClickHouse/pull/37891) ([Guo Wangyang](https://github.com/guowangy)).
 * `ORDER BY (a, b)` will use all the same benefits as `ORDER BY a, b`. [#38873](https://github.com/ClickHouse/ClickHouse/pull/38873) ([Igor Nikonov](https://github.com/devcrafter)).
@@ -419,6 +432,7 @@
 * The table `system.asynchronous_metric_log` is further optimized for storage space. This closes [#38134](https://github.com/ClickHouse/ClickHouse/issues/38134). See the [YouTube video](https://www.youtube.com/watch?v=0fSp9SF8N8A). [#38428](https://github.com/ClickHouse/ClickHouse/pull/38428) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 
 #### Improvement
+
 * Support SQL standard CREATE INDEX and DROP INDEX syntax. [#35166](https://github.com/ClickHouse/ClickHouse/pull/35166) ([Jianmei Zhang](https://github.com/zhangjmruc)).
 * Send profile events for INSERT queries (previously only SELECT was supported). [#37391](https://github.com/ClickHouse/ClickHouse/pull/37391) ([Azat Khuzhin](https://github.com/azat)).
 * Implement in order aggregation (`optimize_aggregation_in_order`) for fully materialized projections. [#37469](https://github.com/ClickHouse/ClickHouse/pull/37469) ([Azat Khuzhin](https://github.com/azat)).
@@ -464,6 +478,7 @@
 * Allow to declare `RabbitMQ` queue without default arguments `x-max-length` and `x-overflow`. [#39259](https://github.com/ClickHouse/ClickHouse/pull/39259) ([rnbondarenko](https://github.com/rnbondarenko)).
 
 #### Build/Testing/Packaging Improvement
+
 * Apply Clang Thread Safety Analysis (TSA) annotations to ClickHouse. [#38068](https://github.com/ClickHouse/ClickHouse/pull/38068) ([Robert Schulze](https://github.com/rschu1ze)).
 * Adapt universal installation script for FreeBSD. [#39302](https://github.com/ClickHouse/ClickHouse/pull/39302) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Preparation for building on `s390x` platform. [#39193](https://github.com/ClickHouse/ClickHouse/pull/39193) ([Harry Lee](https://github.com/HarryLeeIBM)).
@@ -473,6 +488,7 @@
 * Change `all|noarch` packages to architecture-dependent - Fix some documentation for it - Push aarch64|arm64 packages to artifactory and release assets - Fixes [#36443](https://github.com/ClickHouse/ClickHouse/issues/36443). [#38580](https://github.com/ClickHouse/ClickHouse/pull/38580) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
 
 #### Bug Fix (user-visible misbehavior in official stable or prestable release)
+
 * Fix rounding for `Decimal128/Decimal256` with more than 19-digits long scale. [#38027](https://github.com/ClickHouse/ClickHouse/pull/38027) ([Igor Nikonov](https://github.com/devcrafter)).
 * Fixed crash caused by data race in storage `Hive` (integration table engine). [#38887](https://github.com/ClickHouse/ClickHouse/pull/38887) ([lgbo](https://github.com/lgbo-ustc)).
 * Fix crash when executing GRANT ALL ON *.* with ON CLUSTER. It was broken in https://github.com/ClickHouse/ClickHouse/pull/35767. This closes [#38618](https://github.com/ClickHouse/ClickHouse/issues/38618). [#38674](https://github.com/ClickHouse/ClickHouse/pull/38674) ([Vitaly Baranov](https://github.com/vitlibar)).
@@ -529,6 +545,7 @@
 ### <a id="226"></a> ClickHouse release 22.6, 2022-06-16
 
 #### Backward Incompatible Change
+
 * Remove support for octal number literals in SQL. In previous versions they were parsed as Float64. [#37765](https://github.com/ClickHouse/ClickHouse/pull/37765) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)).
 * Changes how settings using `seconds` as type are parsed to support floating point values (for example: `max_execution_time=0.5`). Infinity or NaN values will throw an exception. [#37187](https://github.com/ClickHouse/ClickHouse/pull/37187) ([Raúl Marín](https://github.com/Algunenano)).
 * Changed format of binary serialization of columns of experimental type `Object`. New format is more convenient to implement by third-party clients. [#37482](https://github.com/ClickHouse/ClickHouse/pull/37482) ([Anton Popov](https://github.com/CurtizJ)).
@@ -537,6 +554,7 @@
 * If you run different ClickHouse versions on a cluster with AArch64 CPU or mix AArch64 and amd64 on a cluster, and use distributed queries with GROUP BY multiple keys of fixed-size type that fit in 256 bits but don't fit in 64 bits, and the size of the result is huge, the data will not be fully aggregated in the result of these queries during upgrade. Workaround: upgrade with downtime instead of a rolling upgrade.
 
 #### New Feature
+
 * Add `GROUPING` function. It allows to disambiguate the records in the queries with `ROLLUP`, `CUBE` or `GROUPING SETS`. Closes [#19426](https://github.com/ClickHouse/ClickHouse/issues/19426). [#37163](https://github.com/ClickHouse/ClickHouse/pull/37163) ([Dmitry Novik](https://github.com/novikd)).
 * A new codec [FPC](https://userweb.cs.txstate.edu/~burtscher/papers/dcc07a.pdf) algorithm for floating point data compression. [#37553](https://github.com/ClickHouse/ClickHouse/pull/37553) ([Mikhail Guzov](https://github.com/koloshmet)).
 * Add new columnar JSON formats: `JSONColumns`, `JSONCompactColumns`, `JSONColumnsWithMetadata`. Closes [#36338](https://github.com/ClickHouse/ClickHouse/issues/36338) Closes [#34509](https://github.com/ClickHouse/ClickHouse/issues/34509). [#36975](https://github.com/ClickHouse/ClickHouse/pull/36975) ([Kruglov Pavel](https://github.com/Avogar)).
@@ -557,11 +575,13 @@
 * Added `SYSTEM UNFREEZE` query that deletes the whole backup regardless if the corresponding table is deleted or not. [#36424](https://github.com/ClickHouse/ClickHouse/pull/36424) ([Vadim Volodin](https://github.com/PolyProgrammist)).
 
 #### Experimental Feature
+
 * Enables `POPULATE` for `WINDOW VIEW`. [#36945](https://github.com/ClickHouse/ClickHouse/pull/36945) ([vxider](https://github.com/Vxider)).
 * `ALTER TABLE ... MODIFY QUERY` support for `WINDOW VIEW`. [#37188](https://github.com/ClickHouse/ClickHouse/pull/37188) ([vxider](https://github.com/Vxider)).
 * This PR changes the behavior of the `ENGINE` syntax in `WINDOW VIEW`, to make it like in `MATERIALIZED VIEW`. [#37214](https://github.com/ClickHouse/ClickHouse/pull/37214) ([vxider](https://github.com/Vxider)).
 
 #### Performance Improvement
+
 * Added numerous optimizations for ARM NEON [#38093](https://github.com/ClickHouse/ClickHouse/pull/38093)([Daniel Kutenin](https://github.com/danlark1)), ([Alexandra Pilipyuk](https://github.com/chalice19)) Note: if you run different ClickHouse versions on a cluster with ARM CPU and use distributed queries with GROUP BY multiple keys of fixed-size type that fit in 256 bits but don't fit in 64 bits, the result of the aggregation query will be wrong during upgrade. Workaround: upgrade with downtime instead of a rolling upgrade.
 * Improve performance and memory usage for select of subset of columns for formats Native, Protobuf, CapnProto, JSONEachRow, TSKV, all formats with suffixes WithNames/WithNamesAndTypes. Previously while selecting only subset of columns from files in these formats all columns were read and stored in memory. Now only required columns are read. This PR enables setting `input_format_skip_unknown_fields` by default, because otherwise in case of select of subset of columns exception will be thrown. [#37192](https://github.com/ClickHouse/ClickHouse/pull/37192) ([Kruglov Pavel](https://github.com/Avogar)).
 * Now more filters can be pushed down for join. [#37472](https://github.com/ClickHouse/ClickHouse/pull/37472) ([Amos Bird](https://github.com/amosbird)).
@@ -592,6 +612,7 @@
 * In function: CompressedWriteBuffer::nextImpl(), there is an unnecessary write-copy step that would happen frequently during inserting data. Below shows the differentiation with this patch: - Before: 1. Compress "working_buffer" into "compressed_buffer" 2. write-copy into "out" - After: Directly Compress "working_buffer" into "out". [#37242](https://github.com/ClickHouse/ClickHouse/pull/37242) ([jasperzhu](https://github.com/jinjunzh)).
 
 #### Improvement
+
 * Support types with non-standard defaults in ROLLUP, CUBE, GROUPING SETS. Closes [#37360](https://github.com/ClickHouse/ClickHouse/issues/37360). [#37667](https://github.com/ClickHouse/ClickHouse/pull/37667) ([Dmitry Novik](https://github.com/novikd)).
 * Fix stack traces collection on ARM. Closes [#37044](https://github.com/ClickHouse/ClickHouse/issues/37044). Closes [#15638](https://github.com/ClickHouse/ClickHouse/issues/15638). [#37797](https://github.com/ClickHouse/ClickHouse/pull/37797) ([Maksim Kita](https://github.com/kitaisreal)).
 * Client will try every IP address returned by DNS resolution until successful connection. [#37273](https://github.com/ClickHouse/ClickHouse/pull/37273) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)).
@@ -633,6 +654,7 @@
 * Add implicit grants with grant option too. For example `GRANT CREATE TABLE ON test.* TO A WITH GRANT OPTION` now allows `A` to execute `GRANT CREATE VIEW ON test.* TO B`. [#38017](https://github.com/ClickHouse/ClickHouse/pull/38017) ([Vitaly Baranov](https://github.com/vitlibar)).
 
 #### Build/Testing/Packaging Improvement
+
 * Use `clang-14` and LLVM infrastructure version 14 for builds. This closes [#34681](https://github.com/ClickHouse/ClickHouse/issues/34681). [#34754](https://github.com/ClickHouse/ClickHouse/pull/34754) ([Alexey Milovidov](https://github.com/alexey-milovidov)). Note: `clang-14` has [a bug](https://github.com/google/sanitizers/issues/1540) in ThreadSanitizer that makes our CI work worse.
 * Allow to drop privileges at startup. This simplifies Docker images. Closes [#36293](https://github.com/ClickHouse/ClickHouse/issues/36293). [#36341](https://github.com/ClickHouse/ClickHouse/pull/36341) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
 * Add docs spellcheck to CI. [#37790](https://github.com/ClickHouse/ClickHouse/pull/37790) ([Vladimir C](https://github.com/vdimir)).
@@ -690,7 +712,6 @@
 * Fix possible heap-use-after-free error when reading system.projection_parts and system.projection_parts_columns . This fixes [#37184](https://github.com/ClickHouse/ClickHouse/issues/37184). [#37185](https://github.com/ClickHouse/ClickHouse/pull/37185) ([Amos Bird](https://github.com/amosbird)).
 * Fixed `DateTime64` fractional seconds behavior prior to Unix epoch. [#37697](https://github.com/ClickHouse/ClickHouse/pull/37697) ([Andrey Zvonov](https://github.com/zvonand)). [#37039](https://github.com/ClickHouse/ClickHouse/pull/37039) ([李扬](https://github.com/taiyang-li)).
 
-
 ### <a id="225"></a> ClickHouse release 22.5, 2022-05-19
 
 #### Upgrade Notes
@@ -743,7 +764,7 @@
 * Implement partial GROUP BY key for optimize_aggregation_in_order. [#35111](https://github.com/ClickHouse/ClickHouse/pull/35111) ([Azat Khuzhin](https://github.com/azat)).
 
 #### Improvement
- 
+
 * Show names of erroneous files in case of parsing errors while executing table functions `file`, `s3` and `url`. [#36314](https://github.com/ClickHouse/ClickHouse/pull/36314) ([Anton Popov](https://github.com/CurtizJ)).
 * Allowed to increase the number of threads for executing background operations (merges, mutations, moves and fetches) at runtime if they are specified at top level config. [#36425](https://github.com/ClickHouse/ClickHouse/pull/36425) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
 * Now date time conversion functions that generates time before 1970-01-01 00:00:00 with partial hours/minutes timezones will be saturated to zero instead of overflow. This is the continuation of https://github.com/ClickHouse/ClickHouse/pull/29953 which addresses https://github.com/ClickHouse/ClickHouse/pull/29953#discussion_r800550280 . Mark as improvement because it's implementation defined behavior (and very rare case) and we are allowed to break it. [#36656](https://github.com/ClickHouse/ClickHouse/pull/36656) ([Amos Bird](https://github.com/amosbird)).
@@ -852,7 +873,6 @@
 * Fix ALTER DROP COLUMN of nested column with compact parts (i.e. `ALTER TABLE x DROP COLUMN n`, when there is column `n.d`). [#35797](https://github.com/ClickHouse/ClickHouse/pull/35797) ([Azat Khuzhin](https://github.com/azat)).
 * Fix substring function range error length when `offset` and `length` is negative constant and `s` is not constant. [#33861](https://github.com/ClickHouse/ClickHouse/pull/33861) ([RogerYK](https://github.com/RogerYK)).
 
-
 ### <a id="224"></a> ClickHouse release 22.4, 2022-04-19
 
 #### Backward Incompatible Change
@@ -1004,8 +1024,7 @@
 * Fix mutations in tables with enabled sparse columns. [#35284](https://github.com/ClickHouse/ClickHouse/pull/35284) ([Anton Popov](https://github.com/CurtizJ)).
 * Do not delay final part writing by default (fixes possible `Memory limit exceeded` during `INSERT` by adding `max_insert_delayed_streams_for_parallel_write` with default to 1000 for writes to s3 and disabled as before otherwise). [#34780](https://github.com/ClickHouse/ClickHouse/pull/34780) ([Azat Khuzhin](https://github.com/azat)).
 
-
-## <a id="223"></a> ClickHouse release v22.3-lts, 2022-03-17
+### <a id="223"></a> ClickHouse release v22.3-lts, 2022-03-17
 
 #### Backward Incompatible Change
 
@@ -1132,7 +1151,6 @@
 * Fix incorrect result of trivial count query when part movement feature is used [#34089](https://github.com/ClickHouse/ClickHouse/issues/34089). [#34385](https://github.com/ClickHouse/ClickHouse/pull/34385) ([nvartolomei](https://github.com/nvartolomei)).
 * Fix inconsistency of `max_query_size` limitation in distributed subqueries. [#34078](https://github.com/ClickHouse/ClickHouse/pull/34078) ([Chao Ma](https://github.com/godliness)).
 
-
 ### <a id="222"></a> ClickHouse release v22.2, 2022-02-17
 
 #### Upgrade Notes
@@ -1308,7 +1326,6 @@
 * Fix issue [#18206](https://github.com/ClickHouse/ClickHouse/issues/18206). [#33977](https://github.com/ClickHouse/ClickHouse/pull/33977) ([Vitaly Baranov](https://github.com/vitlibar)).
 * This PR allows using multiple LDAP storages in the same list of user directories. It worked earlier but was broken because LDAP tests are disabled (they are part of the testflows tests). [#33574](https://github.com/ClickHouse/ClickHouse/pull/33574) ([Vitaly Baranov](https://github.com/vitlibar)).
 
-
 ### <a id="221"></a> ClickHouse release v22.1, 2022-01-18
 
 #### Upgrade Notes
@@ -1335,7 +1352,6 @@
 * Add function `decodeURLFormComponent` slightly different to `decodeURLComponent`. Close [#10298](https://github.com/ClickHouse/ClickHouse/issues/10298). [#33451](https://github.com/ClickHouse/ClickHouse/pull/33451) ([SuperDJY](https://github.com/cmsxbc)).
 * Allow to split `GraphiteMergeTree` rollup rules for plain/tagged metrics (optional rule_type field). [#33494](https://github.com/ClickHouse/ClickHouse/pull/33494) ([Michail Safronov](https://github.com/msaf1980)).
 
-
 #### Performance Improvement
 
 * Support moving conditions to `PREWHERE` (setting `optimize_move_to_prewhere`) for tables of `Merge` engine if its all underlying tables supports `PREWHERE`. [#33300](https://github.com/ClickHouse/ClickHouse/pull/33300) ([Anton Popov](https://github.com/CurtizJ)).
@@ -1351,7 +1367,6 @@
 * Optimize selecting of MergeTree parts that can be moved between volumes. [#33225](https://github.com/ClickHouse/ClickHouse/pull/33225) ([OnePiece](https://github.com/zhongyuankai)).
 * Fix `sparse_hashed` dict performance with sequential keys (wrong hash function). [#32536](https://github.com/ClickHouse/ClickHouse/pull/32536) ([Azat Khuzhin](https://github.com/azat)).
 
-
 #### Experimental Feature
 
 * Parallel reading from multiple replicas within a shard during distributed query without using sample key. To enable this, set `allow_experimental_parallel_reading_from_replicas = 1` and `max_parallel_replicas` to any number. This closes [#26748](https://github.com/ClickHouse/ClickHouse/issues/26748). [#29279](https://github.com/ClickHouse/ClickHouse/pull/29279) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
@@ -1364,7 +1379,6 @@
 * Fix ACL with explicit digit hash in `clickhouse-keeper`: now the behavior consistent with ZooKeeper and generated digest is always accepted. [#33249](https://github.com/ClickHouse/ClickHouse/pull/33249) ([小路](https://github.com/nicelulu)). [#33246](https://github.com/ClickHouse/ClickHouse/pull/33246).
 * Fix unexpected projection removal when detaching parts. [#32067](https://github.com/ClickHouse/ClickHouse/pull/32067) ([Amos Bird](https://github.com/amosbird)).
 
-
 #### Improvement
 
 * Now date time conversion functions that generates time before `1970-01-01 00:00:00` will be saturated to zero instead of overflow. [#29953](https://github.com/ClickHouse/ClickHouse/pull/29953) ([Amos Bird](https://github.com/amosbird)). It also fixes a bug in index analysis if date truncation function would yield result before the Unix epoch.
@@ -1411,7 +1425,6 @@
 * Updating `modification_time` for data part in `system.parts` after part movement [#32964](https://github.com/ClickHouse/ClickHouse/issues/32964). [#32965](https://github.com/ClickHouse/ClickHouse/pull/32965) ([save-my-heart](https://github.com/save-my-heart)).
 * Potential issue, cannot be exploited: integer overflow may happen in array resize. [#33024](https://github.com/ClickHouse/ClickHouse/pull/33024) ([varadarajkumar](https://github.com/varadarajkumar)).
 
-
 #### Build/Testing/Packaging Improvement
 
 * Add packages, functional tests and Docker builds for AArch64 (ARM) version of ClickHouse. [#32911](https://github.com/ClickHouse/ClickHouse/pull/32911) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). [#32415](https://github.com/ClickHouse/ClickHouse/pull/32415)
@@ -1426,7 +1439,6 @@
 * Inject git information into clickhouse binary file. So we can get source code revision easily from clickhouse binary file. [#33124](https://github.com/ClickHouse/ClickHouse/pull/33124) ([taiyang-li](https://github.com/taiyang-li)).
 * Remove obsolete code from ConfigProcessor. Yandex specific code is not used anymore. The code contained one minor defect. This defect was reported by [Mallik Hassan](https://github.com/SadiHassan) in [#33032](https://github.com/ClickHouse/ClickHouse/issues/33032). This closes [#33032](https://github.com/ClickHouse/ClickHouse/issues/33032). [#33026](https://github.com/ClickHouse/ClickHouse/pull/33026) ([alexey-milovidov](https://github.com/alexey-milovidov)).
 
-
 #### Bug Fix (user-visible misbehavior in official stable or prestable release)
 
 * Several fixes for format parsing. This is relevant if `clickhouse-server` is open for write access to adversary. Specifically crafted input data for `Native` format may lead to reading uninitialized memory or crash. This is relevant if `clickhouse-server` is open for write access to adversary. [#33050](https://github.com/ClickHouse/ClickHouse/pull/33050) ([Heena Bansal](https://github.com/HeenaBansal2009)). Fixed Apache Avro Union type index out of boundary issue in Apache Avro binary format. [#33022](https://github.com/ClickHouse/ClickHouse/pull/33022) ([Harry Lee](https://github.com/HarryLeeIBM)). Fix null pointer dereference in `LowCardinality` data when deserializing `LowCardinality` data in the Native format. [#33021](https://github.com/ClickHouse/ClickHouse/pull/33021) ([Harry Lee](https://github.com/HarryLeeIBM)).
@@ -1485,5 +1497,4 @@
 * Fix possible crash (or incorrect result) in case of `LowCardinality` arguments of window function. Fixes [#31114](https://github.com/ClickHouse/ClickHouse/issues/31114). [#31888](https://github.com/ClickHouse/ClickHouse/pull/31888) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
 * Fix hang up with command `DROP TABLE system.query_log sync`. [#33293](https://github.com/ClickHouse/ClickHouse/pull/33293) ([zhanghuajie](https://github.com/zhanghuajieHIT)).
 
-
 ## [Changelog for 2021](https://clickhouse.com/docs/en/whats-new/changelog/2021)

base/base/safeExit.cpp

@@ -1,10 +1,8 @@
 #if defined(OS_LINUX)
 #    include <sys/syscall.h>
 #endif
-#include <cstdlib>
 #include <unistd.h>
 #include <base/safeExit.h>
-#include <base/defines.h>
 
 [[noreturn]] void safeExit(int code)
 {

cmake/ld.lld.in

@@ -3,15 +3,15 @@
 # This is a workaround for bug in llvm/clang,
 # that does not produce .debug_aranges with LTO
 #
-# NOTE: this is a temporary solution, that should be removed once [1] will be
-# resolved.
+# NOTE: this is a temporary solution, that should be removed after upgrading to
+# clang-16/llvm-16.
 #
-#   [1]: https://discourse.llvm.org/t/clang-does-not-produce-full-debug-aranges-section-with-thinlto/64898/8
+# Refs: https://reviews.llvm.org/D133092
 
 # NOTE: only -flto=thin is supported.
 # NOTE: it is not possible to check was there -gdwarf-aranges initially or not.
 if [[ "$*" =~ -plugin-opt=thinlto ]]; then
-    exec "@LLD_PATH@" -mllvm -generate-arange-section "$@"
+    exec "@LLD_PATH@" -plugin-opt=-generate-arange-section "$@"
 else
     exec "@LLD_PATH@" "$@"
 fi

cmake/tools.cmake

@@ -117,7 +117,7 @@ endif()
 # Archiver
 
 if (COMPILER_GCC)
-    find_program (LLVM_AR_PATH NAMES "llvm-ar" "llvm-ar-14" "llvm-ar-13" "llvm-ar-12")
+    find_program (LLVM_AR_PATH NAMES "llvm-ar" "llvm-ar-15" "llvm-ar-14" "llvm-ar-13" "llvm-ar-12")
 else ()
     find_program (LLVM_AR_PATH NAMES "llvm-ar-${COMPILER_VERSION_MAJOR}" "llvm-ar")
 endif ()
@@ -131,7 +131,7 @@ message(STATUS "Using archiver: ${CMAKE_AR}")
 # Ranlib
 
 if (COMPILER_GCC)
-    find_program (LLVM_RANLIB_PATH NAMES "llvm-ranlib" "llvm-ranlib-14" "llvm-ranlib-13" "llvm-ranlib-12")
+    find_program (LLVM_RANLIB_PATH NAMES "llvm-ranlib" "llvm-ranlib-15" "llvm-ranlib-14" "llvm-ranlib-13" "llvm-ranlib-12")
 else ()
     find_program (LLVM_RANLIB_PATH NAMES "llvm-ranlib-${COMPILER_VERSION_MAJOR}" "llvm-ranlib")
 endif ()
@@ -145,7 +145,7 @@ message(STATUS "Using ranlib: ${CMAKE_RANLIB}")
 # Install Name Tool
 
 if (COMPILER_GCC)
-    find_program (LLVM_INSTALL_NAME_TOOL_PATH NAMES "llvm-install-name-tool" "llvm-install-name-tool-14" "llvm-install-name-tool-13" "llvm-install-name-tool-12")
+    find_program (LLVM_INSTALL_NAME_TOOL_PATH NAMES "llvm-install-name-tool" "llvm-install-name-tool-15" "llvm-install-name-tool-14" "llvm-install-name-tool-13" "llvm-install-name-tool-12")
 else ()
     find_program (LLVM_INSTALL_NAME_TOOL_PATH NAMES "llvm-install-name-tool-${COMPILER_VERSION_MAJOR}" "llvm-install-name-tool")
 endif ()
@@ -159,7 +159,7 @@ message(STATUS "Using install-name-tool: ${CMAKE_INSTALL_NAME_TOOL}")
 # Objcopy
 
 if (COMPILER_GCC)
-    find_program (OBJCOPY_PATH NAMES "llvm-objcopy" "llvm-objcopy-14" "llvm-objcopy-13" "llvm-objcopy-12" "objcopy")
+    find_program (OBJCOPY_PATH NAMES "llvm-objcopy" "llvm-objcopy-15" "llvm-objcopy-14" "llvm-objcopy-13" "llvm-objcopy-12" "objcopy")
 else ()
     find_program (OBJCOPY_PATH NAMES "llvm-objcopy-${COMPILER_VERSION_MAJOR}" "llvm-objcopy" "objcopy")
 endif ()
@@ -173,7 +173,7 @@ endif ()
 # Strip
 
 if (COMPILER_GCC)
-    find_program (STRIP_PATH NAMES "llvm-strip" "llvm-strip-14" "llvm-strip-13" "llvm-strip-12" "strip")
+    find_program (STRIP_PATH NAMES "llvm-strip" "llvm-strip-15" "llvm-strip-14" "llvm-strip-13" "llvm-strip-12" "strip")
 else ()
     find_program (STRIP_PATH NAMES "llvm-strip-${COMPILER_VERSION_MAJOR}" "llvm-strip" "strip")
 endif ()

docker/test/stress/run.sh

@@ -47,7 +47,6 @@ function install_packages()
 
 function configure()
 {
-    export ZOOKEEPER_FAULT_INJECTION=1
     # install test configs
     export USE_DATABASE_ORDINARY=1
     export EXPORT_S3_STORAGE_POLICIES=1
@@ -203,6 +202,7 @@ quit
 
 install_packages package_folder
 
+export ZOOKEEPER_FAULT_INJECTION=1
 configure
 
 azurite-blob --blobHost 0.0.0.0 --blobPort 10000 --debug /azurite_log &
@@ -243,6 +243,7 @@ stop
 
 # Let's enable S3 storage by default
 export USE_S3_STORAGE_FOR_MERGE_TREE=1
+export ZOOKEEPER_FAULT_INJECTION=1
 configure
 
 # But we still need default disk because some tables loaded only into it
@@ -375,6 +376,8 @@ else
     install_packages previous_release_package_folder
 
     # Start server from previous release
+    # Previous version may not be ready for fault injections
+    export ZOOKEEPER_FAULT_INJECTION=0
     configure
 
     # Avoid "Setting s3_check_objects_after_upload is neither a builtin setting..."
@@ -389,12 +392,23 @@ else
 
     clickhouse-client --query="SELECT 'Server version: ', version()"
 
-    # Install new package before running stress test because we should use new clickhouse-client and new clickhouse-test
-    # But we should leave old binary in /usr/bin/ for gdb (so it will print sane stacktarces)
+    # Install new package before running stress test because we should use new
+    # clickhouse-client and new clickhouse-test.
+    #
+    # But we should leave old binary in /usr/bin/ and debug symbols in
+    # /usr/lib/debug/usr/bin (if any) for gdb and internal DWARF parser, so it
+    # will print sane stacktraces and also to avoid possible crashes.
+    #
+    # FIXME: those files can be extracted directly from debian package, but
+    # actually better solution will be to use different PATH instead of playing
+    # games with files from packages.
     mv /usr/bin/clickhouse previous_release_package_folder/
+    mv /usr/lib/debug/usr/bin/clickhouse.debug previous_release_package_folder/
     install_packages package_folder
     mv /usr/bin/clickhouse package_folder/
+    mv /usr/lib/debug/usr/bin/clickhouse.debug package_folder/
     mv previous_release_package_folder/clickhouse /usr/bin/
+    mv previous_release_package_folder/clickhouse.debug /usr/lib/debug/usr/bin/clickhouse.debug
 
     mkdir tmp_stress_output
 
@@ -410,6 +424,8 @@ else
 
     # Start new server
     mv package_folder/clickhouse /usr/bin/
+    mv package_folder/clickhouse.debug /usr/lib/debug/usr/bin/clickhouse.debug
+    export ZOOKEEPER_FAULT_INJECTION=1
     configure
     start 500
     clickhouse-client --query "SELECT 'Backward compatibility check: Server successfully started', 'OK'" >> /test_output/test_results.tsv \

docs/en/development/architecture.md

@@ -49,27 +49,13 @@ When we calculate some function over columns in a block, we add another column w
 
 Blocks are created for every processed chunk of data. Note that for the same type of calculation, the column names and types remain the same for different blocks, and only column data changes. It is better to split block data from the block header because small block sizes have a high overhead of temporary strings for copying shared_ptrs and column names.
 
-## Block Streams {#block-streams}
+## Processors
 
-Block streams are for processing data. We use streams of blocks to read data from somewhere, perform data transformations, or write data to somewhere. `IBlockInputStream` has the `read` method to fetch the next block while available. `IBlockOutputStream` has the `write` method to push the block somewhere.
-
-Streams are responsible for:
-
-1.  Reading or writing to a table. The table just returns a stream for reading or writing blocks.
-2.  Implementing data formats. For example, if you want to output data to a terminal in `Pretty` format, you create a block output stream where you push blocks, and it formats them.
-3.  Performing data transformations. Let’s say you have `IBlockInputStream` and want to create a filtered stream. You create `FilterBlockInputStream` and initialize it with your stream. Then when you pull a block from `FilterBlockInputStream`, it pulls a block from your stream, filters it, and returns the filtered block to you. Query execution pipelines are represented this way.
-
-There are more sophisticated transformations. For example, when you pull from `AggregatingBlockInputStream`, it reads all data from its source, aggregates it, and then returns a stream of aggregated data for you. Another example: `UnionBlockInputStream` accepts many input sources in the constructor and also a number of threads. It launches multiple threads and reads from multiple sources in parallel.
-
-> Block streams use the “pull” approach to control flow: when you pull a block from the first stream, it consequently pulls the required blocks from nested streams, and the entire execution pipeline will work. Neither “pull” nor “push” is the best solution, because control flow is implicit, and that limits the implementation of various features like simultaneous execution of multiple queries (merging many pipelines together). This limitation could be overcome with coroutines or just running extra threads that wait for each other. We may have more possibilities if we make control flow explicit: if we locate the logic for passing data from one calculation unit to another outside of those calculation units. Read this [article](http://journal.stuffwithstuff.com/2013/01/13/iteration-inside-and-out/) for more thoughts.
-
-We should note that the query execution pipeline creates temporary data at each step. We try to keep block size small enough so that temporary data fits in the CPU cache. With that assumption, writing and reading temporary data is almost free in comparison with other calculations. We could consider an alternative, which is to fuse many operations in the pipeline together. It could make the pipeline as short as possible and remove much of the temporary data, which could be an advantage, but it also has drawbacks. For example, a split pipeline makes it easy to implement caching intermediate data, stealing intermediate data from similar queries running at the same time, and merging pipelines for similar queries.
+See the description at [https://github.com/ClickHouse/ClickHouse/blob/master/src/Processors/IProcessor.h](https://github.com/ClickHouse/ClickHouse/blob/master/src/Processors/IProcessor.h).
 
 ## Formats {#formats}
 
-Data formats are implemented with block streams. There are “presentational” formats only suitable for the output of data to the client, such as `Pretty` format, which provides only `IBlockOutputStream`. And there are input/output formats, such as `TabSeparated` or `JSONEachRow`.
-
-There are also row streams: `IRowInputStream` and `IRowOutputStream`. They allow you to pull/push data by individual rows, not by blocks. And they are only needed to simplify the implementation of row-oriented formats. The wrappers `BlockInputStreamFromRowInputStream` and `BlockOutputStreamFromRowOutputStream` allow you to convert row-oriented streams to regular block-oriented streams.
+Data formats are implemented with processors.
 
 ## I/O {#io}
 

docs/en/engines/table-engines/mergetree-family/mergetree.md

@@ -419,6 +419,8 @@ Supported data types: `Int*`, `UInt*`, `Float*`, `Enum`, `Date`, `DateTime`, `St
 
 For `Map` data type client can specify if index should be created for keys or values using [mapKeys](../../../sql-reference/functions/tuple-map-functions.md#mapkeys) or [mapValues](../../../sql-reference/functions/tuple-map-functions.md#mapvalues) function.
 
+There are also special-purpose and experimental indexes to support approximate nearest neighbor (ANN) queries. See [here](annindexes.md) for details.
+
 The following functions can use the filter: [equals](../../../sql-reference/functions/comparison-functions.md), [notEquals](../../../sql-reference/functions/comparison-functions.md), [in](../../../sql-reference/functions/in-functions), [notIn](../../../sql-reference/functions/in-functions), [has](../../../sql-reference/functions/array-functions#hasarr-elem), [hasAny](../../../sql-reference/functions/array-functions#hasany), [hasAll](../../../sql-reference/functions/array-functions#hasall).
 
 Example of index creation for `Map` data type

docs/en/sql-reference/functions/date-time-functions.md

@@ -271,11 +271,7 @@ Result:
 The return type of `toStartOf*`, `toLastDayOfMonth`, `toMonday`, `timeSlot` functions described below is determined by the configuration parameter [enable_extended_results_for_datetime_functions](../../operations/settings/settings#enable-extended-results-for-datetime-functions) which is `0` by default.
 
 Behavior for
-* `enable_extended_results_for_datetime_functions = 0`: Functions `toStartOfYear`, `toStartOfISOYear`, `toStartOfQuarter`, `toStartOfMonth`, `toStartOfWeek`, `toLastDayOfMonth`, `toMonday` return `Date` or `DateTime`. Functions `toStartOfDay`, `toStartOfHour`, `toStartOfFifteenMinutes`, `toStartOfTenMinutes`, `toStartOfFiveMinutes`, `toStartOfMinute`, `timeSlot` return `DateTime`. Though these functions can take values of the extended types `Date32` and `DateTime64` as an argument, passing them a time outside the normal range (year 1970 to 2149 for `Date` / 2106 for `DateTime`) will produce wrong results. In case argument is out of normal range:
-  * If the argument is smaller than 1970, the result will be calculated from the argument `1970-01-01 (00:00:00)` instead.
-  * If the return type is `DateTime` and the argument is larger than `2106-02-07 08:28:15`, the result will be calculated from the argument `2106-02-07 08:28:15` instead.
-  * If the return type is `Date` and the argument is larger than `2149-06-06`, the result will be calculated from the argument `2149-06-06` instead.
-  *  If `toLastDayOfMonth` is called with an argument greater then `2149-05-31`, the result will be calculated from the argument `2149-05-31` instead.
+* `enable_extended_results_for_datetime_functions = 0`: Functions `toStartOfYear`, `toStartOfISOYear`, `toStartOfQuarter`, `toStartOfMonth`, `toStartOfWeek`, `toLastDayOfMonth`, `toMonday` return `Date` or `DateTime`. Functions `toStartOfDay`, `toStartOfHour`, `toStartOfFifteenMinutes`, `toStartOfTenMinutes`, `toStartOfFiveMinutes`, `toStartOfMinute`, `timeSlot` return `DateTime`. Though these functions can take values of the extended types `Date32` and `DateTime64` as an argument, passing them a time outside the normal range (year 1970 to 2149 for `Date` / 2106 for `DateTime`) will produce wrong results.
 * `enable_extended_results_for_datetime_functions = 1`:
   * Functions `toStartOfYear`, `toStartOfISOYear`, `toStartOfQuarter`, `toStartOfMonth`, `toStartOfWeek`, `toLastDayOfMonth`, `toMonday` return `Date` or `DateTime` if their argument is a `Date` or `DateTime`, and they return `Date32` or `DateTime64` if their argument is a `Date32` or `DateTime64`.
   * Functions `toStartOfDay`, `toStartOfHour`, `toStartOfFifteenMinutes`, `toStartOfTenMinutes`, `toStartOfFiveMinutes`, `toStartOfMinute`, `timeSlot` return `DateTime` if their argument is a `Date` or `DateTime`, and they return `DateTime64` if their argument is a `Date32` or `DateTime64`.
@@ -302,25 +298,22 @@ Returns the date.
 Rounds down a date or date with time to the first day of the month.
 Returns the date.
 
-## toLastDayOfMonth
-
-Rounds up a date or date with time to the last day of the month.
-Returns the date.
+:::note
+The behavior of parsing incorrect dates is implementation specific. ClickHouse may return zero date, throw an exception or do “natural” overflow.
+:::
 
 If `toLastDayOfMonth` is called with an argument of type `Date` greater then 2149-05-31, the result will be calculated from the argument 2149-05-31 instead.
 
 ## toMonday
 
 Rounds down a date or date with time to the nearest Monday.
-As a special case, date arguments `1970-01-01`, `1970-01-02`, `1970-01-03` and `1970-01-04` return date `1970-01-01`.
 Returns the date.
 
 ## toStartOfWeek(t\[,mode\])
 
 Rounds down a date or date with time to the nearest Sunday or Monday by mode.
 Returns the date.
-As a special case, date arguments `1970-01-01`, `1970-01-02`, `1970-01-03` and `1970-01-04` (and `1970-01-05` if `mode` is `1`) return date `1970-01-01`.
-The `mode` argument works exactly like the mode argument to toWeek(). For the single-argument syntax, a mode value of 0 is used.
+The mode argument works exactly like the mode argument to toWeek(). For the single-argument syntax, a mode value of 0 is used.
 
 ## toStartOfDay
 
@@ -671,9 +664,9 @@ Aliases: `dateDiff`, `DATE_DIFF`.
     - `quarter`
     - `year`
 
--   `startdate` — The first time value to subtract (the subtrahend). [Date](../../sql-reference/data-types/date.md) or [DateTime](../../sql-reference/data-types/datetime.md).
+-   `startdate` — The first time value to subtract (the subtrahend). [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
 
--   `enddate` — The second time value to subtract from (the minuend). [Date](../../sql-reference/data-types/date.md) or [DateTime](../../sql-reference/data-types/datetime.md).
+-   `enddate` — The second time value to subtract from (the minuend). [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
 
 -   `timezone` — [Timezone name](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (optional). If specified, it is applied to both `startdate` and `enddate`. If not specified, timezones of `startdate` and `enddate` are used. If they are not the same, the result is unspecified. [String](../../sql-reference/data-types/string.md).
 
@@ -1075,7 +1068,7 @@ Example:
 SELECT timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600));
 SELECT timeSlots(toDateTime('1980-12-12 21:01:02', 'UTC'), toUInt32(600), 299);
 SELECT timeSlots(toDateTime64('1980-12-12 21:01:02.1234', 4, 'UTC'), toDecimal64(600.1, 1), toDecimal64(299, 0));
-```
+``` 
 ``` text
 ┌─timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600))─┐
 │ ['2012-01-01 12:00:00','2012-01-01 12:30:00']               │
@@ -1163,7 +1156,7 @@ dateName(date_part, date)
 **Arguments**
 
 -   `date_part` — Date part. Possible values: 'year', 'quarter', 'month', 'week', 'dayofyear', 'day', 'weekday', 'hour', 'minute', 'second'. [String](../../sql-reference/data-types/string.md).
--   `date` — Date. [Date](../../sql-reference/data-types/date.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
+-   `date` — Date. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
 -   `timezone` — Timezone. Optional. [String](../../sql-reference/data-types/string.md).
 
 **Returned value**

docs/en/sql-reference/functions/other-functions.md

@@ -571,7 +571,7 @@ Example:
 
 ``` sql
 SELECT
-    transform(domain(Referer), ['yandex.ru', 'google.ru', 'vk.com'], ['www.yandex', 'example.com']) AS s,
+    transform(domain(Referer), ['yandex.ru', 'google.ru', 'vkontakte.ru'], ['www.yandex', 'example.com', 'vk.com']) AS s,
     count() AS c
 FROM test.hits
 GROUP BY domain(Referer)

docs/en/sql-reference/statements/misc.md

@@ -1,21 +0,0 @@
----
-slug: /en/sql-reference/statements/misc
-toc_hidden: true
-sidebar_position: 70
----
-
-# Miscellaneous Statements
-
--   [ATTACH](../../sql-reference/statements/attach.md)
--   [CHECK TABLE](../../sql-reference/statements/check-table.md)
--   [DESCRIBE TABLE](../../sql-reference/statements/describe-table.md)
--   [DETACH](../../sql-reference/statements/detach.md)
--   [DROP](../../sql-reference/statements/drop.md)
--   [EXISTS](../../sql-reference/statements/exists.md)
--   [KILL](../../sql-reference/statements/kill.md)
--   [OPTIMIZE](../../sql-reference/statements/optimize.md)
--   [RENAME](../../sql-reference/statements/rename.md)
--   [SET](../../sql-reference/statements/set.md)
--   [SET ROLE](../../sql-reference/statements/set-role.md)
--   [TRUNCATE](../../sql-reference/statements/truncate.md)
--   [USE](../../sql-reference/statements/use.md)

docs/ru/sql-reference/data-types/date.md

@@ -6,7 +6,7 @@ sidebar_label: Date
 
 # Date {#data-type-date}
 
-Дата. Хранится в двух байтах в виде (беззнакового) числа дней, прошедших от 1970-01-01. Позволяет хранить значения от чуть больше, чем начала unix-эпохи до верхнего порога, определяющегося константой на этапе компиляции (сейчас - до 2149 года, последний полностью поддерживаемый год - 2148).
+Дата. Хранится в двух байтах в виде (беззнакового) числа дней, прошедших от 1970-01-01. Позволяет хранить значения от чуть больше, чем начала unix-эпохи до верхнего порога, определяющегося константой на этапе компиляции (сейчас - до 2106 года, последний полностью поддерживаемый год - 2105).
 
 Диапазон значений: \[1970-01-01, 2149-06-06\].
 

docs/ru/sql-reference/functions/date-time-functions.md

@@ -272,15 +272,9 @@ SELECT toUnixTimestamp('2017-11-05 08:07:47', 'Asia/Tokyo') AS unix_timestamp;
 
 Поведение для
 * `enable_extended_results_for_datetime_functions = 0`: Функции `toStartOf*`, `toLastDayOfMonth`, `toMonday` возвращают `Date` или `DateTime`. Функции `toStartOfDay`, `toStartOfHour`, `toStartOfFifteenMinutes`, `toStartOfTenMinutes`, `toStartOfFiveMinutes`, `toStartOfMinute`, `timeSlot` возвращают `DateTime`. Хотя эти функции могут принимать значения типа `Date32` или `DateTime64` в качестве аргумента, при обработке аргумента вне нормального диапазона значений (`1970` - `2148` для `Date` и `1970-01-01 00:00:00`-`2106-02-07 08:28:15` для `DateTime`) будет получен некорректный результат.
-В случае если значение аргумента вне нормального диапазона:
-    * `1970-01-01 (00:00:00)` будет возвращён для моментов времени до 1970 года,
-    * `2106-02-07 08:28:15` будет взят в качестве аргумента, если полученный аргумент превосходит данное значение и возвращаемый тип - `DateTime`,
-    * `2149-06-06` будет взят в качестве аргумента, если полученный аргумент превосходит данное значение и возвращаемый тип - `Date`,
-    * `2149-05-31` будет результатом функции `toLastDayOfMonth` при обработке аргумента больше `2149-05-31`.
 * `enable_extended_results_for_datetime_functions = 1`:
     * Функции `toStartOfYear`, `toStartOfISOYear`, `toStartOfQuarter`, `toStartOfMonth`, `toStartOfWeek`, `toLastDayOfMonth`, `toMonday` возвращают `Date` или `DateTime` если их аргумент `Date` или `DateTime` и они возвращают `Date32` или `DateTime64` если их аргумент `Date32` или `DateTime64`.
     * Функции `toStartOfDay`, `toStartOfHour`, `toStartOfFifteenMinutes`, `toStartOfTenMinutes`, `toStartOfFiveMinutes`, `toStartOfMinute`, `timeSlot` возвращают `DateTime` если их аргумент `Date` или `DateTime` и они возвращают `DateTime64` если их аргумент `Date32` или `DateTime64`.
-
 :::
 
 ## toStartOfYear {#tostartofyear}
@@ -321,20 +315,20 @@ SELECT toStartOfISOYear(toDate('2017-01-01')) AS ISOYear20170101;
 Округляет дату или дату-с-временем до последнего числа месяца.
 Возвращается дата.
 
-Если `toLastDayOfMonth` вызывается с аргументом типа `Date` большим чем 2149-05-31, то результат будет вычислен от аргумента 2149-05-31.
-
+:::note "Attention"
+    Возвращаемое значение для некорректных дат зависит от реализации. ClickHouse может вернуть нулевую дату, выбросить исключение, или выполнить «естественное» перетекание дат между месяцами.
+:::
+    
 ## toMonday {#tomonday}
 
 Округляет дату или дату-с-временем вниз до ближайшего понедельника.
-Частный случай: для дат `1970-01-01`, `1970-01-02`, `1970-01-03` и `1970-01-04` результатом будет `1970-01-01`.
 Возвращается дата.
 
 ## toStartOfWeek(t[,mode]) {#tostartofweek}
 
 Округляет дату или дату со временем до ближайшего воскресенья или понедельника в соответствии с mode.
 Возвращается дата.
-Частный случай: для дат `1970-01-01`, `1970-01-02`, `1970-01-03` и `1970-01-04` (и `1970-01-05`, если `mode` равен `1`) результатом будет `1970-01-01`.
-Аргумент `mode` работает точно так же, как аргумент mode [toWeek()](#toweek). Если аргумент mode опущен, то используется режим 0.
+Аргумент mode работает точно так же, как аргумент mode [toWeek()](#toweek). Если аргумент mode опущен, то используется режим 0.
 
 ## toStartOfDay {#tostartofday}
 
@@ -721,9 +715,9 @@ date_diff('unit', startdate, enddate, [timezone])
     - `quarter`
     - `year`
 
--   `startdate` — первая дата или дата со временем, которая вычитается из `enddate`. [Date](../../sql-reference/data-types/date.md) или [DateTime](../../sql-reference/data-types/datetime.md).
+-   `startdate` — первая дата или дата со временем, которая вычитается из `enddate`. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
 
--   `enddate` — вторая дата или дата со временем, из которой вычитается `startdate`. [Date](../../sql-reference/data-types/date.md) или [DateTime](../../sql-reference/data-types/datetime.md).
+-   `enddate` — вторая дата или дата со временем, из которой вычитается `startdate`. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
 
 -   `timezone` — [часовой пояс](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (необязательно). Если этот аргумент указан, то он применяется как для `startdate`, так и для `enddate`. Если этот аргумент не указан, то используются часовые пояса аргументов `startdate` и `enddate`. Если часовые пояса аргументов `startdate` и `enddate` не совпадают, то результат не определен. [String](../../sql-reference/data-types/string.md).
 
@@ -975,8 +969,7 @@ SELECT now('Europe/Moscow');
 
 ## timeSlots(StartTime, Duration,\[, Size\]) {#timeslotsstarttime-duration-size}
 Для интервала, начинающегося в `StartTime` и длящегося `Duration` секунд, возвращает массив моментов времени, кратных `Size`. Параметр `Size` указывать необязательно, по умолчанию он равен 1800 секундам (30 минутам) - необязательный параметр.
-Данная функция может использоваться, например, для анализа количества просмотров страницы за соответствующую сессию.
-Аргумент `StartTime` может иметь тип `DateTime` или `DateTime64`. В случае, если используется `DateTime`, аргументы `Duration` и `Size` должны иметь тип `UInt32`; Для DateTime64 они должны быть типа `Decimal64`.
+
 Возвращает массив DateTime/DateTime64 (тип будет совпадать с типом параметра ’StartTime’). Для DateTime64 масштаб(scale) возвращаемой величины может отличаться от масштаба фргумента ’StartTime’ --- результат будет иметь наибольший масштаб среди всех данных аргументов.
 
 Пример использования:
@@ -1085,7 +1078,7 @@ dateName(date_part, date)
 **Аргументы**
 
 -   `date_part` — часть даты. Возможные значения: 'year', 'quarter', 'month', 'week', 'dayofyear', 'day', 'weekday', 'hour', 'minute', 'second'. [String](../../sql-reference/data-types/string.md).
--   `date` — дата. [Date](../../sql-reference/data-types/date.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
+-   `date` — дата. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
 -   `timezone` — часовой пояс. Необязательный аргумент. [String](../../sql-reference/data-types/string.md).
 
 **Возвращаемое значение**

docs/ru/sql-reference/functions/other-functions.md

@@ -568,7 +568,7 @@ ORDER BY c DESC
 
 ``` sql
 SELECT
-    transform(domain(Referer), ['yandex.ru', 'google.ru', 'vk.com'], ['www.yandex', 'example.com']) AS s,
+    transform(domain(Referer), ['yandex.ru', 'google.ru', 'vkontakte.ru'], ['www.yandex', 'example.com', 'vk.com']) AS s,
     count() AS c
 FROM test.hits
 GROUP BY domain(Referer)

docs/ru/sql-reference/statements/misc.md

@@ -1,21 +0,0 @@
----
-slug: /ru/sql-reference/statements/misc
-sidebar_position: 41
----
-
-# Прочие виды запросов {#prochie-vidy-zaprosov}
-
--   [ATTACH](../../sql-reference/statements/attach.md)
--   [CHECK TABLE](../../sql-reference/statements/check-table.md)
--   [DESCRIBE TABLE](../../sql-reference/statements/describe-table.md)
--   [DETACH](../../sql-reference/statements/detach.md)
--   [DROP](../../sql-reference/statements/drop.md)
--   [EXISTS](../../sql-reference/statements/exists.md)
--   [KILL](../../sql-reference/statements/kill.md)
--   [OPTIMIZE](../../sql-reference/statements/optimize.md)
--   [RENAME](../../sql-reference/statements/rename.md)
--   [SET](../../sql-reference/statements/set.md)
--   [SET ROLE](../../sql-reference/statements/set-role.md)
--   [TRUNCATE](../../sql-reference/statements/truncate.md)
--   [USE](../../sql-reference/statements/use.md)
-

docs/zh/sql-reference/data-types/date.md

@@ -3,7 +3,7 @@ slug: /zh/sql-reference/data-types/date
 ---
 # 日期 {#date}
 
-日期类型，用两个字节存储，表示从 1970-01-01 (无符号) 到当前的日期值。允许存储从 Unix 纪元开始到编译阶段定义的上限阈值常量（目前上限是2149年，但最终完全支持的年份为2148）。最小值输出为1970-01-01。
+日期类型，用两个字节存储，表示从 1970-01-01 (无符号) 到当前的日期值。允许存储从 Unix 纪元开始到编译阶段定义的上限阈值常量（目前上限是2106年，但最终完全支持的年份为2105）。最小值输出为1970-01-01。
 
 值的范围: \[1970-01-01, 2149-06-06\]。
 

docs/zh/sql-reference/functions/other-functions.md

@@ -237,7 +237,7 @@ ORDER BY c DESC
 
 ``` sql
 SELECT
-    transform(domain(Referer), ['yandex.ru', 'google.ru', 'vk.com'], ['www.yandex', 'example.com']) AS s,
+    transform(domain(Referer), ['yandex.ru', 'google.ru', 'vkontakte.ru'], ['www.yandex', 'example.com', 'vk.com']) AS s,
     count() AS c
 FROM test.hits
 GROUP BY domain(Referer)

programs/client/clickhouse-client.xml

@@ -19,7 +19,6 @@
             {host}
             {port}
             {user}
-            {database}
             {display_name}
         Terminal colors: https://misc.flogisoft.com/bash/tip_colors_and_formatting
         See also: https://wiki.hackzine.org/development/misc/readline-color-prompt.html

programs/disks/DisksApp.cpp

@@ -57,7 +57,7 @@ void DisksApp::addOptions(
         ("config-file,C", po::value<String>(), "Set config file")
         ("disk", po::value<String>(), "Set disk name")
         ("command_name", po::value<String>(), "Name for command to do")
-        ("send-logs", "Send logs")
+        ("save-logs", "Save logs to a file")
         ("log-level", po::value<String>(), "Logging level")
         ;
 
@@ -82,10 +82,10 @@ void DisksApp::processOptions()
         config().setString("config-file", options["config-file"].as<String>());
     if (options.count("disk"))
         config().setString("disk", options["disk"].as<String>());
-    if (options.count("send-logs"))
-        config().setBool("send-logs", true);
+    if (options.count("save-logs"))
+        config().setBool("save-logs", true);
     if (options.count("log-level"))
-        Poco::Logger::root().setLevel(options["log-level"].as<std::string>());
+        config().setString("log-level", options["log-level"].as<String>());
 }
 
 void DisksApp::init(std::vector<String> & common_arguments)
@@ -149,15 +149,6 @@ void DisksApp::parseAndCheckOptions(
 
 int DisksApp::main(const std::vector<String> & /*args*/)
 {
-    if (config().has("send-logs"))
-    {
-        auto log_level = config().getString("log-level", "trace");
-        Poco::Logger::root().setLevel(Poco::Logger::parseLevel(log_level));
-
-        auto log_path = config().getString("logger.clickhouse-disks", "/var/log/clickhouse-server/clickhouse-disks.log");
-        Poco::Logger::root().setChannel(Poco::AutoPtr<Poco::FileChannel>(new Poco::FileChannel(log_path)));
-    }
-
     if (config().has("config-file") || fs::exists(getDefaultConfigFileName()))
     {
         String config_path = config().getString("config-file", getDefaultConfigFileName());
@@ -171,6 +162,20 @@ int DisksApp::main(const std::vector<String> & /*args*/)
         throw Exception(ErrorCodes::BAD_ARGUMENTS, "No config-file specifiged");
     }
 
+    if (config().has("save-logs"))
+    {
+        auto log_level = config().getString("log-level", "trace");
+        Poco::Logger::root().setLevel(Poco::Logger::parseLevel(log_level));
+
+        auto log_path = config().getString("logger.clickhouse-disks", "/var/log/clickhouse-server/clickhouse-disks.log");
+        Poco::Logger::root().setChannel(Poco::AutoPtr<Poco::FileChannel>(new Poco::FileChannel(log_path)));
+    }
+    else
+    {
+        auto log_level = config().getString("log-level", "none");
+        Poco::Logger::root().setLevel(Poco::Logger::parseLevel(log_level));
+    }
+
     registerDisks();
     registerFormats();
 

programs/keeper/CMakeLists.txt

@@ -45,6 +45,7 @@ if (BUILD_STANDALONE_KEEPER)
         ${CMAKE_CURRENT_SOURCE_DIR}/../../src/Coordination/KeeperLogStore.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/../../src/Coordination/KeeperServer.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/../../src/Coordination/KeeperSnapshotManager.cpp
+        ${CMAKE_CURRENT_SOURCE_DIR}/../../src/Coordination/KeeperSnapshotManagerS3.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/../../src/Coordination/KeeperStateMachine.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/../../src/Coordination/KeeperStateManager.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/../../src/Coordination/KeeperStorage.cpp

src/AggregateFunctions/AggregateFunctionQuantile.cpp

@@ -46,7 +46,7 @@ AggregateFunctionPtr createAggregateFunctionQuantile(
     if (which.idx == TypeIndex::DateTime64) return std::make_shared<Function<DateTime64, false>>(argument_types, params);
 
     if (which.idx == TypeIndex::Int128) return std::make_shared<Function<Int128, true>>(argument_types, params);
-    if (which.idx == TypeIndex::UInt128) return std::make_shared<Function<Int128, true>>(argument_types, params);
+    if (which.idx == TypeIndex::UInt128) return std::make_shared<Function<UInt128, true>>(argument_types, params);
     if (which.idx == TypeIndex::Int256) return std::make_shared<Function<Int256, true>>(argument_types, params);
     if (which.idx == TypeIndex::UInt256) return std::make_shared<Function<UInt256, true>>(argument_types, params);
 

src/AggregateFunctions/AggregateFunctionQuantileDeterministic.cpp

@@ -40,7 +40,7 @@ AggregateFunctionPtr createAggregateFunctionQuantile(
     if (which.idx == TypeIndex::DateTime) return std::make_shared<Function<DataTypeDateTime::FieldType, false>>(argument_types, params);
 
     if (which.idx == TypeIndex::Int128) return std::make_shared<Function<Int128, true>>(argument_types, params);
-    if (which.idx == TypeIndex::UInt128) return std::make_shared<Function<Int128, true>>(argument_types, params);
+    if (which.idx == TypeIndex::UInt128) return std::make_shared<Function<UInt128, true>>(argument_types, params);
     if (which.idx == TypeIndex::Int256) return std::make_shared<Function<Int256, true>>(argument_types, params);
     if (which.idx == TypeIndex::UInt256) return std::make_shared<Function<UInt256, true>>(argument_types, params);
 

src/AggregateFunctions/AggregateFunctionQuantileExact.cpp

@@ -47,7 +47,7 @@ AggregateFunctionPtr createAggregateFunctionQuantile(
     if (which.idx == TypeIndex::DateTime64) return std::make_shared<Function<DateTime64, false>>(argument_types, params);
 
     if (which.idx == TypeIndex::Int128) return std::make_shared<Function<Int128, true>>(argument_types, params);
-    if (which.idx == TypeIndex::UInt128) return std::make_shared<Function<Int128, true>>(argument_types, params);
+    if (which.idx == TypeIndex::UInt128) return std::make_shared<Function<UInt128, true>>(argument_types, params);
     if (which.idx == TypeIndex::Int256) return std::make_shared<Function<Int256, true>>(argument_types, params);
     if (which.idx == TypeIndex::UInt256) return std::make_shared<Function<UInt256, true>>(argument_types, params);
 

src/AggregateFunctions/AggregateFunctionQuantileExactWeighted.cpp

@@ -46,7 +46,7 @@ AggregateFunctionPtr createAggregateFunctionQuantile(
     if (which.idx == TypeIndex::DateTime64) return std::make_shared<Function<DateTime64, false>>(argument_types, params);
 
     if (which.idx == TypeIndex::Int128) return std::make_shared<Function<Int128, true>>(argument_types, params);
-    if (which.idx == TypeIndex::UInt128) return std::make_shared<Function<Int128, true>>(argument_types, params);
+    if (which.idx == TypeIndex::UInt128) return std::make_shared<Function<UInt128, true>>(argument_types, params);
     if (which.idx == TypeIndex::Int256) return std::make_shared<Function<Int256, true>>(argument_types, params);
     if (which.idx == TypeIndex::UInt256) return std::make_shared<Function<UInt256, true>>(argument_types, params);
 

src/AggregateFunctions/AggregateFunctionWelchTTest.cpp

@@ -40,7 +40,15 @@ struct WelchTTestData : public TTestMoments<Float64>
         Float64 denominator_x = sx2 * sx2 / (nx * nx * (nx - 1));
         Float64 denominator_y = sy2 * sy2 / (ny * ny * (ny - 1));
 
-        return numerator / (denominator_x + denominator_y);
+        auto result = numerator / (denominator_x + denominator_y);
+
+        if (result <= 0 || std::isinf(result) || isNaN(result))
+            throw Exception(
+                ErrorCodes::BAD_ARGUMENTS,
+                "Cannot calculate p_value, because the t-distribution \
+                has inappropriate value of degrees of freedom (={}). It should be > 0", result);
+
+        return result;
     }
 
     std::tuple<Float64, Float64> getResult() const

src/AggregateFunctions/IAggregateFunction.cpp

@@ -53,9 +53,12 @@ String IAggregateFunction::getDescription() const
 
 bool IAggregateFunction::haveEqualArgumentTypes(const IAggregateFunction & rhs) const
 {
-    return std::equal(argument_types.begin(), argument_types.end(),
-                      rhs.argument_types.begin(), rhs.argument_types.end(),
-                      [](const auto & t1, const auto & t2) { return t1->equals(*t2); });
+    return std::equal(
+        argument_types.begin(),
+        argument_types.end(),
+        rhs.argument_types.begin(),
+        rhs.argument_types.end(),
+        [](const auto & t1, const auto & t2) { return t1->equals(*t2); });
 }
 
 bool IAggregateFunction::haveSameStateRepresentation(const IAggregateFunction & rhs) const
@@ -67,11 +70,7 @@ bool IAggregateFunction::haveSameStateRepresentation(const IAggregateFunction &
 
 bool IAggregateFunction::haveSameStateRepresentationImpl(const IAggregateFunction & rhs) const
 {
-    bool res = getName() == rhs.getName()
-        && parameters == rhs.parameters
-        && haveEqualArgumentTypes(rhs);
-    assert(res == (getStateType()->getName() == rhs.getStateType()->getName()));
-    return res;
+    return getStateType()->equals(*rhs.getStateType());
 }
 
 }

src/Client/ClientBase.cpp

@@ -1,7 +1,6 @@
 #include <Client/ClientBase.h>
 
 #include <iostream>
-#include <iomanip>
 #include <filesystem>
 #include <map>
 #include <unordered_map>
@@ -9,7 +8,6 @@
 #include "config.h"
 
 #include <Common/DateLUT.h>
-#include <Common/LocalDate.h>
 #include <Common/MemoryTracker.h>
 #include <base/argsToConfig.h>
 #include <base/LineReader.h>
@@ -32,7 +30,6 @@
 #include <Common/clearPasswordFromCommandLine.h>
 #include <Common/StringUtils/StringUtils.h>
 #include <Common/filesystemHelpers.h>
-#include <Common/Config/configReadClient.h>
 #include <Common/NetException.h>
 #include <Storages/ColumnsDescription.h>
 
@@ -70,10 +67,10 @@
 #include <IO/WriteBufferFromOStream.h>
 #include <IO/CompressionMethod.h>
 #include <Client/InternalTextLogs.h>
-#include <boost/algorithm/string/replace.hpp>
 #include <IO/ForkWriteBuffer.h>
 #include <Parsers/Kusto/ParserKQLStatement.h>
 
+
 namespace fs = std::filesystem;
 using namespace std::literals;
 
@@ -1925,7 +1922,7 @@ bool ClientBase::processQueryText(const String & text)
 
 String ClientBase::prompt() const
 {
-    return boost::replace_all_copy(prompt_by_server_display_name, "{database}", config().getString("database", "default"));
+    return prompt_by_server_display_name;
 }
 
 

src/Client/LocalConnection.cpp

@@ -6,8 +6,6 @@
 #include <Processors/Executors/PushingAsyncPipelineExecutor.h>
 #include <Storages/IStorage.h>
 #include <Core/Protocol.h>
-#include <DataTypes/DataTypesNumber.h>
-#include <DataTypes/DataTypeString.h>
 
 
 namespace DB

src/Client/MultiplexedConnections.cpp

@@ -393,24 +393,38 @@ MultiplexedConnections::ReplicaState & MultiplexedConnections::getReplicaForRead
         Poco::Net::Socket::SocketList write_list;
         Poco::Net::Socket::SocketList except_list;
 
-        for (const ReplicaState & state : replica_states)
-        {
-            Connection * connection = state.connection;
-            if (connection != nullptr)
-                read_list.push_back(*connection->socket);
-        }
-
         auto timeout = is_draining ? drain_timeout : receive_timeout;
-        int n = Poco::Net::Socket::select(
-            read_list,
-            write_list,
-            except_list,
-            timeout);
+        int n = 0;
+
+        /// EINTR loop
+        while (true)
+        {
+            read_list.clear();
+            for (const ReplicaState & state : replica_states)
+            {
+                Connection * connection = state.connection;
+                if (connection != nullptr)
+                    read_list.push_back(*connection->socket);
+            }
+
+            /// poco returns 0 on EINTR, let's reset errno to ensure that EINTR came from select().
+            errno = 0;
+
+            n = Poco::Net::Socket::select(
+                read_list,
+                write_list,
+                except_list,
+                timeout);
+            if (n <= 0 && errno == EINTR)
+                continue;
+            break;
+        }
 
         /// We treat any error as timeout for simplicity.
         /// And we also check if read_list is still empty just in case.
         if (n <= 0 || read_list.empty())
         {
+            const auto & addresses = dumpAddressesUnlocked();
             for (ReplicaState & state : replica_states)
             {
                 Connection * connection = state.connection;
@@ -423,7 +437,7 @@ MultiplexedConnections::ReplicaState & MultiplexedConnections::getReplicaForRead
             throw Exception(ErrorCodes::TIMEOUT_EXCEEDED,
                 "Timeout ({} ms) exceeded while reading from {}",
                 timeout.totalMilliseconds(),
-                dumpAddressesUnlocked());
+                addresses);
         }
     }
 

src/Common/DateLUTImpl.h

@@ -895,6 +895,19 @@ public:
         return toRelativeHourNum(lut[toLUTIndex(v)].date);
     }
 
+    /// The same formula is used for positive time (after Unix epoch) and negative time (before Unix epoch).
+    /// It’s needed for correct work of dateDiff function.
+    inline Time toStableRelativeHourNum(Time t) const
+    {
+        return (t + DATE_LUT_ADD + 86400 - offset_at_start_of_epoch) / 3600 - (DATE_LUT_ADD / 3600);
+    }
+
+    template <typename DateOrTime>
+    inline Time toStableRelativeHourNum(DateOrTime v) const
+    {
+        return toStableRelativeHourNum(lut[toLUTIndex(v)].date);
+    }
+
     inline Time toRelativeMinuteNum(Time t) const /// NOLINT
     {
         return (t + DATE_LUT_ADD) / 60 - (DATE_LUT_ADD / 60);

src/Common/OvercommitTracker.cpp

@@ -5,6 +5,7 @@
 #include <Common/ProfileEvents.h>
 #include <Interpreters/ProcessList.h>
 
+
 namespace ProfileEvents
 {
     extern const Event MemoryOvercommitWaitTimeMicroseconds;
@@ -170,7 +171,8 @@ void UserOvercommitTracker::pickQueryToExcludeImpl()
 
 GlobalOvercommitTracker::GlobalOvercommitTracker(DB::ProcessList * process_list_)
     : OvercommitTracker(process_list_)
-{}
+{
+}
 
 void GlobalOvercommitTracker::pickQueryToExcludeImpl()
 {
@@ -180,16 +182,16 @@ void GlobalOvercommitTracker::pickQueryToExcludeImpl()
     // This is guaranteed by locking global_mutex in OvercommitTracker::needToStopQuery.
     for (auto const & query : process_list->processes)
     {
-        if (query.isKilled())
+        if (query->isKilled())
             continue;
 
         Int64 user_soft_limit = 0;
-        if (auto const * user_process_list = query.getUserProcessList())
+        if (auto const * user_process_list = query->getUserProcessList())
             user_soft_limit = user_process_list->user_memory_tracker.getSoftLimit();
         if (user_soft_limit == 0)
             continue;
 
-        auto * memory_tracker = query.getMemoryTracker();
+        auto * memory_tracker = query->getMemoryTracker();
         if (!memory_tracker)
             continue;
         auto ratio = memory_tracker->getOvercommitRatio(user_soft_limit);

src/Common/getNumberOfPhysicalCPUCores.cpp

@@ -48,7 +48,7 @@ static unsigned getNumberOfPhysicalCPUCoresImpl()
     /// Let's limit ourself to the number of physical cores.
     /// But if the number of logical cores is small - maybe it is a small machine
     /// or very limited cloud instance and it is reasonable to use all the cores.
-    if (cpu_count >= 8)
+    if (cpu_count >= 32)
         cpu_count /= 2;
 #endif
 

src/Common/intExp.h

@@ -47,6 +47,11 @@ namespace common
 
 constexpr inline int exp10_i32(int x)
 {
+    if (x < 0)
+        return 0;
+    if (x > 9)
+        return std::numeric_limits<int>::max();
+
     constexpr int values[] =
     {
         1,
@@ -65,6 +70,11 @@ constexpr inline int exp10_i32(int x)
 
 constexpr inline int64_t exp10_i64(int x)
 {
+    if (x < 0)
+        return 0;
+    if (x > 18)
+        return std::numeric_limits<int64_t>::max();
+
     constexpr int64_t values[] =
     {
         1LL,
@@ -92,6 +102,11 @@ constexpr inline int64_t exp10_i64(int x)
 
 constexpr inline Int128 exp10_i128(int x)
 {
+    if (x < 0)
+        return 0;
+    if (x > 38)
+        return std::numeric_limits<Int128>::max();
+
     constexpr Int128 values[] =
     {
         static_cast<Int128>(1LL),
@@ -140,6 +155,11 @@ constexpr inline Int128 exp10_i128(int x)
 
 inline Int256 exp10_i256(int x)
 {
+    if (x < 0)
+        return 0;
+    if (x > 76)
+        return std::numeric_limits<Int256>::max();
+
     using Int256 = Int256;
     static constexpr Int256 i10e18{1000000000000000000ll};
     static const Int256 values[] = {
@@ -231,8 +251,10 @@ inline Int256 exp10_i256(int x)
 template <typename T>
 constexpr inline T intExp10OfSize(int x)
 {
-    if constexpr (sizeof(T) <= 8)
-        return intExp10(x);
+    if constexpr (sizeof(T) <= 4)
+        return common::exp10_i32(x);
+    else if constexpr (sizeof(T) <= 8)
+        return common::exp10_i64(x);
     else if constexpr (sizeof(T) <= 16)
         return common::exp10_i128(x);
     else

src/Common/tests/gtest_DateLUTImpl.cpp

@@ -134,6 +134,7 @@ TEST(DateLUTTest, TimeValuesInMiddleOfRange)
     EXPECT_EQ(lut.toRelativeMonthNum(time), 24237 /*unsigned*/);
     EXPECT_EQ(lut.toRelativeQuarterNum(time), 8078 /*unsigned*/);
     EXPECT_EQ(lut.toRelativeHourNum(time), 435736 /*time_t*/);
+    EXPECT_EQ(lut.toStableRelativeHourNum(time), 435757 /*time_t*/);
     EXPECT_EQ(lut.toRelativeMinuteNum(time), 26144180 /*time_t*/);
     EXPECT_EQ(lut.toStartOfMinuteInterval(time, 6), 1568650680 /*time_t*/);
     EXPECT_EQ(lut.toStartOfSecondInterval(time, 7), 1568650811 /*time_t*/);
@@ -196,6 +197,7 @@ TEST(DateLUTTest, TimeValuesAtLeftBoderOfRange)
     EXPECT_EQ(lut.toRelativeMonthNum(time), 23641 /*unsigned*/); // ?
     EXPECT_EQ(lut.toRelativeQuarterNum(time), 7880 /*unsigned*/); // ?
     EXPECT_EQ(lut.toRelativeHourNum(time), 0 /*time_t*/);
+    EXPECT_EQ(lut.toStableRelativeHourNum(time), 24 /*time_t*/);
     EXPECT_EQ(lut.toRelativeMinuteNum(time), 0 /*time_t*/);
     EXPECT_EQ(lut.toStartOfMinuteInterval(time, 6), 0 /*time_t*/);
     EXPECT_EQ(lut.toStartOfSecondInterval(time, 7), 0 /*time_t*/);
@@ -259,6 +261,7 @@ TEST(DateLUTTest, TimeValuesAtRightBoderOfRangeOfOldLUT)
     EXPECT_EQ(lut.toRelativeMonthNum(time), 25273 /*unsigned*/);
     EXPECT_EQ(lut.toRelativeQuarterNum(time), 8424 /*unsigned*/);
     EXPECT_EQ(lut.toRelativeHourNum(time), 1192873 /*time_t*/);
+    EXPECT_EQ(lut.toStableRelativeHourNum(time), 1192897 /*time_t*/);
     EXPECT_EQ(lut.toRelativeMinuteNum(time), 71572397 /*time_t*/);
     EXPECT_EQ(lut.toStartOfMinuteInterval(time, 6), 4294343520 /*time_t*/);
     EXPECT_EQ(lut.toStartOfSecondInterval(time, 7), 4294343872 /*time_t*/);

src/Coordination/KeeperDispatcher.cpp

@@ -1,14 +1,21 @@
 #include <Coordination/KeeperDispatcher.h>
+
+#include <Poco/Path.h>
+#include <Poco/Util/AbstractConfiguration.h>
+
+#include <Common/hex.h>
 #include <Common/setThreadName.h>
 #include <Common/ZooKeeper/KeeperException.h>
-#include <future>
-#include <chrono>
-#include <Poco/Path.h>
-#include <Common/hex.h>
-#include <filesystem>
 #include <Common/checkStackSize.h>
 #include <Common/CurrentMetrics.h>
 
+
+#include <future>
+#include <chrono>
+#include <filesystem>
+#include <iterator>
+#include <limits>
+
 namespace CurrentMetrics
 {
     extern const Metric KeeperAliveConnections;
@@ -32,9 +39,7 @@ KeeperDispatcher::KeeperDispatcher()
     : responses_queue(std::numeric_limits<size_t>::max())
     , configuration_and_settings(std::make_shared<KeeperConfigurationAndSettings>())
     , log(&Poco::Logger::get("KeeperDispatcher"))
-{
-}
-
+{}
 
 void KeeperDispatcher::requestThread()
 {
@@ -191,7 +196,13 @@ void KeeperDispatcher::snapshotThread()
 
         try
         {
-            task.create_snapshot(std::move(task.snapshot));
+            auto snapshot_path = task.create_snapshot(std::move(task.snapshot));
+
+            if (snapshot_path.empty())
+                continue;
+
+            if (isLeader())
+                snapshot_s3.uploadSnapshot(snapshot_path);
         }
         catch (...)
         {
@@ -285,7 +296,9 @@ void KeeperDispatcher::initialize(const Poco::Util::AbstractConfiguration & conf
     responses_thread = ThreadFromGlobalPool([this] { responseThread(); });
     snapshot_thread = ThreadFromGlobalPool([this] { snapshotThread(); });
 
-    server = std::make_unique<KeeperServer>(configuration_and_settings, config, responses_queue, snapshots_queue);
+    snapshot_s3.startup(config);
+
+    server = std::make_unique<KeeperServer>(configuration_and_settings, config, responses_queue, snapshots_queue, snapshot_s3);
 
     try
     {
@@ -312,7 +325,6 @@ void KeeperDispatcher::initialize(const Poco::Util::AbstractConfiguration & conf
     /// Start it after keeper server start
     session_cleaner_thread = ThreadFromGlobalPool([this] { sessionCleanerTask(); });
     update_configuration_thread = ThreadFromGlobalPool([this] { updateConfigurationThread(); });
-    updateConfiguration(config);
 
     LOG_DEBUG(log, "Dispatcher initialized");
 }
@@ -415,6 +427,8 @@ void KeeperDispatcher::shutdown()
         if (server)
             server->shutdown();
 
+        snapshot_s3.shutdown();
+
         CurrentMetrics::set(CurrentMetrics::KeeperAliveConnections, 0);
 
     }
@@ -678,6 +692,8 @@ void KeeperDispatcher::updateConfiguration(const Poco::Util::AbstractConfigurati
         if (!push_result)
             throw Exception(ErrorCodes::SYSTEM_ERROR, "Cannot push configuration update to queue");
     }
+
+    snapshot_s3.updateS3Configuration(config);
 }
 
 void KeeperDispatcher::updateKeeperStatLatency(uint64_t process_time_ms)

src/Coordination/KeeperDispatcher.h

@@ -14,6 +14,7 @@
 #include <Coordination/CoordinationSettings.h>
 #include <Coordination/Keeper4LWInfo.h>
 #include <Coordination/KeeperConnectionStats.h>
+#include <Coordination/KeeperSnapshotManagerS3.h>
 
 namespace DB
 {
@@ -76,6 +77,8 @@ private:
     /// Counter for new session_id requests.
     std::atomic<int64_t> internal_session_id_counter{0};
 
+    KeeperSnapshotManagerS3 snapshot_s3;
+
     /// Thread put requests to raft
     void requestThread();
     /// Thread put responses for subscribed sessions

src/Coordination/KeeperServer.cpp

@@ -8,6 +8,7 @@
 #include <string>
 #include <Coordination/KeeperStateMachine.h>
 #include <Coordination/KeeperStateManager.h>
+#include <Coordination/KeeperSnapshotManagerS3.h>
 #include <Coordination/LoggerWrapper.h>
 #include <Coordination/ReadBufferFromNuraftBuffer.h>
 #include <Coordination/WriteBufferFromNuraftBuffer.h>
@@ -105,7 +106,8 @@ KeeperServer::KeeperServer(
     const KeeperConfigurationAndSettingsPtr & configuration_and_settings_,
     const Poco::Util::AbstractConfiguration & config,
     ResponsesQueue & responses_queue_,
-    SnapshotsQueue & snapshots_queue_)
+    SnapshotsQueue & snapshots_queue_,
+    KeeperSnapshotManagerS3 & snapshot_manager_s3)
     : server_id(configuration_and_settings_->server_id)
     , coordination_settings(configuration_and_settings_->coordination_settings)
     , log(&Poco::Logger::get("KeeperServer"))
@@ -125,6 +127,7 @@ KeeperServer::KeeperServer(
         configuration_and_settings_->snapshot_storage_path,
         coordination_settings,
         keeper_context,
+        config.getBool("keeper_server.upload_snapshot_on_exit", true) ? &snapshot_manager_s3 : nullptr,
         checkAndGetSuperdigest(configuration_and_settings_->super_digest));
 
     state_manager = nuraft::cs_new<KeeperStateManager>(

src/Coordination/KeeperServer.h

@@ -71,7 +71,8 @@ public:
         const KeeperConfigurationAndSettingsPtr & settings_,
         const Poco::Util::AbstractConfiguration & config_,
         ResponsesQueue & responses_queue_,
-        SnapshotsQueue & snapshots_queue_);
+        SnapshotsQueue & snapshots_queue_,
+        KeeperSnapshotManagerS3 & snapshot_manager_s3);
 
     /// Load state machine from the latest snapshot and load log storage. Start NuRaft with required settings.
     void startup(const Poco::Util::AbstractConfiguration & config, bool enable_ipv6 = true);

src/Coordination/KeeperSnapshotManager.h

@@ -87,7 +87,7 @@ public:
 };
 
 using KeeperStorageSnapshotPtr = std::shared_ptr<KeeperStorageSnapshot>;
-using CreateSnapshotCallback = std::function<void(KeeperStorageSnapshotPtr &&)>;
+using CreateSnapshotCallback = std::function<std::string(KeeperStorageSnapshotPtr &&)>;
 
 
 using SnapshotMetaAndStorage = std::pair<SnapshotMetadataPtr, KeeperStoragePtr>;

src/Coordination/KeeperSnapshotManagerS3.cpp

@@ -0,0 +1,311 @@
+#include <Coordination/KeeperSnapshotManagerS3.h>
+
+#if USE_AWS_S3
+#include <Core/UUID.h>
+
+#include <Common/Exception.h>
+#include <Common/setThreadName.h>
+
+#include <IO/S3Common.h>
+#include <IO/WriteBufferFromS3.h>
+#include <IO/ReadBufferFromS3.h>
+#include <IO/ReadBufferFromFile.h>
+#include <IO/ReadHelpers.h>
+#include <IO/S3/PocoHTTPClient.h>
+#include <IO/WriteHelpers.h>
+#include <IO/copyData.h>
+
+#include <aws/core/auth/AWSCredentials.h>
+#include <aws/s3/S3Client.h>
+#include <aws/s3/S3Errors.h>
+#include <aws/s3/model/HeadObjectRequest.h>
+#include <aws/s3/model/DeleteObjectRequest.h>
+
+#include <filesystem>
+
+namespace fs = std::filesystem;
+
+namespace DB
+{
+
+struct KeeperSnapshotManagerS3::S3Configuration
+{
+    S3Configuration(S3::URI uri_, S3::AuthSettings auth_settings_, std::shared_ptr<const Aws::S3::S3Client> client_)
+        : uri(std::move(uri_))
+        , auth_settings(std::move(auth_settings_))
+        , client(std::move(client_))
+    {}
+
+    S3::URI uri;
+    S3::AuthSettings auth_settings;
+    std::shared_ptr<const Aws::S3::S3Client> client;
+};
+
+KeeperSnapshotManagerS3::KeeperSnapshotManagerS3()
+    : snapshots_s3_queue(std::numeric_limits<size_t>::max())
+    , log(&Poco::Logger::get("KeeperSnapshotManagerS3"))
+    , uuid(UUIDHelpers::generateV4())
+{}
+
+void KeeperSnapshotManagerS3::updateS3Configuration(const Poco::Util::AbstractConfiguration & config)
+{
+    try
+    {
+        const std::string config_prefix = "keeper_server.s3_snapshot";
+
+        if (!config.has(config_prefix))
+        {
+            std::lock_guard client_lock{snapshot_s3_client_mutex};
+            if (snapshot_s3_client)
+                LOG_INFO(log, "S3 configuration was removed");
+            snapshot_s3_client = nullptr;
+            return;
+        }
+
+        auto auth_settings = S3::AuthSettings::loadFromConfig(config_prefix, config);
+
+        auto endpoint = config.getString(config_prefix + ".endpoint");
+        auto new_uri = S3::URI{Poco::URI(endpoint)};
+
+        {
+            std::lock_guard client_lock{snapshot_s3_client_mutex};
+            // if client is not changed (same auth settings, same endpoint) we don't need to update
+            if (snapshot_s3_client && snapshot_s3_client->client && auth_settings == snapshot_s3_client->auth_settings
+                && snapshot_s3_client->uri.uri == new_uri.uri)
+                return;
+        }
+
+        LOG_INFO(log, "S3 configuration was updated");
+
+        auto credentials = Aws::Auth::AWSCredentials(auth_settings.access_key_id, auth_settings.secret_access_key);
+        HeaderCollection headers = auth_settings.headers;
+
+        static constexpr size_t s3_max_redirects = 10;
+        static constexpr bool enable_s3_requests_logging = false;
+
+        if (!new_uri.key.empty())
+        {
+            LOG_ERROR(log, "Invalid endpoint defined for S3, it shouldn't contain key, endpoint: {}", endpoint);
+            return;
+        }
+
+        S3::PocoHTTPClientConfiguration client_configuration = S3::ClientFactory::instance().createClientConfiguration(
+            auth_settings.region,
+            RemoteHostFilter(), s3_max_redirects,
+            enable_s3_requests_logging,
+            /* for_disk_s3 = */ false);
+
+        client_configuration.endpointOverride = new_uri.endpoint;
+
+        auto client = S3::ClientFactory::instance().create(
+            client_configuration,
+            new_uri.is_virtual_hosted_style,
+            credentials.GetAWSAccessKeyId(),
+            credentials.GetAWSSecretKey(),
+            auth_settings.server_side_encryption_customer_key_base64,
+            std::move(headers),
+            auth_settings.use_environment_credentials.value_or(false),
+            auth_settings.use_insecure_imds_request.value_or(false));
+
+        auto new_client = std::make_shared<KeeperSnapshotManagerS3::S3Configuration>(std::move(new_uri), std::move(auth_settings), std::move(client));
+
+        {
+            std::lock_guard client_lock{snapshot_s3_client_mutex};
+            snapshot_s3_client = std::move(new_client);
+        }
+        LOG_INFO(log, "S3 client was updated");
+    }
+    catch (...)
+    {
+        LOG_ERROR(log, "Failed to create an S3 client for snapshots");
+        tryLogCurrentException(__PRETTY_FUNCTION__);
+    }
+}
+std::shared_ptr<KeeperSnapshotManagerS3::S3Configuration> KeeperSnapshotManagerS3::getSnapshotS3Client() const
+{
+    std::lock_guard lock{snapshot_s3_client_mutex};
+    return snapshot_s3_client;
+}
+
+void KeeperSnapshotManagerS3::uploadSnapshotImpl(const std::string & snapshot_path)
+{
+    try
+    {
+        auto s3_client = getSnapshotS3Client();
+        if (s3_client == nullptr)
+            return;
+
+        S3Settings::ReadWriteSettings read_write_settings;
+        read_write_settings.upload_part_size_multiply_parts_count_threshold = 10000;
+
+        const auto create_writer = [&](const auto & key)
+        {
+            return WriteBufferFromS3
+            {
+                s3_client->client,
+                s3_client->uri.bucket,
+                key,
+                read_write_settings
+            };
+        };
+
+        const auto file_exists = [&](const auto & key)
+        {
+            Aws::S3::Model::HeadObjectRequest request;
+            request.SetBucket(s3_client->uri.bucket);
+            request.SetKey(key);
+            auto outcome = s3_client->client->HeadObject(request);
+
+            if (outcome.IsSuccess())
+                return true;
+
+            const auto & error = outcome.GetError();
+            if (error.GetErrorType() != Aws::S3::S3Errors::NO_SUCH_KEY && error.GetErrorType() != Aws::S3::S3Errors::RESOURCE_NOT_FOUND)
+                throw S3Exception(error.GetErrorType(), "Failed to verify existence of lock file: {}", error.GetMessage());
+
+            return false;
+        };
+
+
+        LOG_INFO(log, "Will try to upload snapshot on {} to S3", snapshot_path);
+        ReadBufferFromFile snapshot_file(snapshot_path);
+
+        auto snapshot_name = fs::path(snapshot_path).filename().string();
+        auto lock_file = fmt::format(".{}_LOCK", snapshot_name);
+
+        if (file_exists(snapshot_name))
+        {
+            LOG_ERROR(log, "Snapshot {} already exists", snapshot_name);
+            return;
+        }
+
+        // First we need to verify that there isn't already a lock file for the snapshot we want to upload
+        // Only leader uploads a snapshot, but there can be a rare case where we have 2 leaders in NuRaft
+        if (file_exists(lock_file))
+        {
+            LOG_ERROR(log, "Lock file for {} already, exists. Probably a different node is already uploading the snapshot", snapshot_name);
+            return;
+        }
+
+        // We write our UUID to lock file
+        LOG_DEBUG(log, "Trying to create a lock file");
+        WriteBufferFromS3 lock_writer = create_writer(lock_file);
+        writeUUIDText(uuid, lock_writer);
+        lock_writer.finalize();
+
+        // We read back the written UUID, if it's the same we can upload the file
+        ReadBufferFromS3 lock_reader
+        {
+            s3_client->client,
+            s3_client->uri.bucket,
+            lock_file,
+            "",
+            1,
+            {}
+        };
+
+        std::string read_uuid;
+        readStringUntilEOF(read_uuid, lock_reader);
+
+        if (read_uuid != toString(uuid))
+        {
+            LOG_ERROR(log, "Failed to create a lock file");
+            return;
+        }
+
+        SCOPE_EXIT(
+        {
+            LOG_INFO(log, "Removing lock file");
+            try
+            {
+                Aws::S3::Model::DeleteObjectRequest delete_request;
+                delete_request.SetBucket(s3_client->uri.bucket);
+                delete_request.SetKey(lock_file);
+                auto delete_outcome = s3_client->client->DeleteObject(delete_request);
+                if (!delete_outcome.IsSuccess())
+                    throw S3Exception(delete_outcome.GetError().GetMessage(), delete_outcome.GetError().GetErrorType());
+            }
+            catch (...)
+            {
+                LOG_INFO(log, "Failed to delete lock file for {} from S3", snapshot_path);
+                tryLogCurrentException(__PRETTY_FUNCTION__);
+            }
+        });
+
+        WriteBufferFromS3 snapshot_writer = create_writer(snapshot_name);
+        copyData(snapshot_file, snapshot_writer);
+        snapshot_writer.finalize();
+
+        LOG_INFO(log, "Successfully uploaded {} to S3", snapshot_path);
+    }
+    catch (...)
+    {
+        LOG_INFO(log, "Failure during upload of {} to S3", snapshot_path);
+        tryLogCurrentException(__PRETTY_FUNCTION__);
+    }
+}
+
+void KeeperSnapshotManagerS3::snapshotS3Thread()
+{
+    setThreadName("KeeperS3SnpT");
+
+    while (!shutdown_called)
+    {
+        std::string snapshot_path;
+        if (!snapshots_s3_queue.pop(snapshot_path))
+            break;
+
+        if (shutdown_called)
+            break;
+
+        uploadSnapshotImpl(snapshot_path);
+    }
+}
+
+void KeeperSnapshotManagerS3::uploadSnapshot(const std::string & path, bool async_upload)
+{
+    if (getSnapshotS3Client() == nullptr)
+        return;
+
+    if (async_upload)
+    {
+        if (!snapshots_s3_queue.push(path))
+            LOG_WARNING(log, "Failed to add snapshot {} to S3 queue", path);
+
+        return;
+    }
+
+    uploadSnapshotImpl(path);
+}
+
+void KeeperSnapshotManagerS3::startup(const Poco::Util::AbstractConfiguration & config)
+{
+    updateS3Configuration(config);
+    snapshot_s3_thread = ThreadFromGlobalPool([this] { snapshotS3Thread(); });
+}
+
+void KeeperSnapshotManagerS3::shutdown()
+{
+    if (shutdown_called)
+        return;
+
+    LOG_DEBUG(log, "Shutting down KeeperSnapshotManagerS3");
+    shutdown_called = true;
+
+    try
+    {
+        snapshots_s3_queue.finish();
+        if (snapshot_s3_thread.joinable())
+            snapshot_s3_thread.join();
+    }
+    catch (...)
+    {
+        tryLogCurrentException(__PRETTY_FUNCTION__);
+    }
+
+    LOG_INFO(log, "KeeperSnapshotManagerS3 shut down");
+}
+
+}
+
+#endif

src/Coordination/KeeperSnapshotManagerS3.h

@@ -0,0 +1,68 @@
+#pragma once
+
+#include "config.h"
+
+#include <Poco/Util/AbstractConfiguration.h>
+
+#if USE_AWS_S3
+#include <Common/ConcurrentBoundedQueue.h>
+#include <Common/ThreadPool.h>
+#include <Common/logger_useful.h>
+
+#include <string>
+#endif
+
+namespace DB
+{
+
+#if USE_AWS_S3
+class KeeperSnapshotManagerS3
+{
+public:
+    KeeperSnapshotManagerS3();
+
+    void updateS3Configuration(const Poco::Util::AbstractConfiguration & config);
+    void uploadSnapshot(const std::string & path, bool async_upload = true);
+
+    void startup(const Poco::Util::AbstractConfiguration & config);
+    void shutdown();
+private:
+    using SnapshotS3Queue = ConcurrentBoundedQueue<std::string>;
+    SnapshotS3Queue snapshots_s3_queue;
+
+    /// Upload new snapshots to S3
+    ThreadFromGlobalPool snapshot_s3_thread;
+
+    struct S3Configuration;
+    mutable std::mutex snapshot_s3_client_mutex;
+    std::shared_ptr<S3Configuration> snapshot_s3_client;
+
+    std::atomic<bool> shutdown_called{false};
+
+    Poco::Logger * log;
+
+    UUID uuid;
+
+    std::shared_ptr<S3Configuration> getSnapshotS3Client() const;
+
+    void uploadSnapshotImpl(const std::string & snapshot_path);
+
+    /// Thread upload snapshots to S3 in the background
+    void snapshotS3Thread();
+};
+#else
+class KeeperSnapshotManagerS3
+{
+public:
+    KeeperSnapshotManagerS3() = default;
+
+    void updateS3Configuration(const Poco::Util::AbstractConfiguration &) {}
+    void uploadSnapshot(const std::string &, [[maybe_unused]] bool async_upload = true) {}
+
+    void startup(const Poco::Util::AbstractConfiguration &) {}
+
+    void shutdown() {}
+};
+#endif
+
+}

src/Coordination/KeeperStateMachine.cpp

@@ -44,6 +44,7 @@ KeeperStateMachine::KeeperStateMachine(
     const std::string & snapshots_path_,
     const CoordinationSettingsPtr & coordination_settings_,
     const KeeperContextPtr & keeper_context_,
+    KeeperSnapshotManagerS3 * snapshot_manager_s3_,
     const std::string & superdigest_)
     : coordination_settings(coordination_settings_)
     , snapshot_manager(
@@ -59,6 +60,7 @@ KeeperStateMachine::KeeperStateMachine(
     , log(&Poco::Logger::get("KeeperStateMachine"))
     , superdigest(superdigest_)
     , keeper_context(keeper_context_)
+    , snapshot_manager_s3(snapshot_manager_s3_)
 {
 }
 
@@ -400,13 +402,22 @@ void KeeperStateMachine::create_snapshot(nuraft::snapshot & s, nuraft::async_res
         }
 
         when_done(ret, exception);
+
+        return ret ? latest_snapshot_path : "";
     };
 
 
     if (keeper_context->server_state == KeeperContext::Phase::SHUTDOWN)
     {
         LOG_INFO(log, "Creating a snapshot during shutdown because 'create_snapshot_on_exit' is enabled.");
-        snapshot_task.create_snapshot(std::move(snapshot_task.snapshot));
+        auto snapshot_path = snapshot_task.create_snapshot(std::move(snapshot_task.snapshot));
+
+        if (!snapshot_path.empty() && snapshot_manager_s3)
+        {
+            LOG_INFO(log, "Uploading snapshot {} during shutdown because 'upload_snapshot_on_exit' is enabled.", snapshot_path);
+            snapshot_manager_s3->uploadSnapshot(snapshot_path, /* asnyc_upload */ false);
+        }
+
         return;
     }
 

src/Coordination/KeeperStateMachine.h

@@ -2,11 +2,13 @@
 
 #include <Coordination/CoordinationSettings.h>
 #include <Coordination/KeeperSnapshotManager.h>
+#include <Coordination/KeeperSnapshotManagerS3.h>
+#include <Coordination/KeeperContext.h>
 #include <Coordination/KeeperStorage.h>
+
 #include <libnuraft/nuraft.hxx>
 #include <Common/ConcurrentBoundedQueue.h>
 #include <Common/logger_useful.h>
-#include <Coordination/KeeperContext.h>
 
 
 namespace DB
@@ -26,6 +28,7 @@ public:
         const std::string & snapshots_path_,
         const CoordinationSettingsPtr & coordination_settings_,
         const KeeperContextPtr & keeper_context_,
+        KeeperSnapshotManagerS3 * snapshot_manager_s3_,
         const std::string & superdigest_ = "");
 
     /// Read state from the latest snapshot
@@ -146,6 +149,8 @@ private:
     const std::string superdigest;
 
     KeeperContextPtr keeper_context;
+
+    KeeperSnapshotManagerS3 * snapshot_manager_s3;
 };
 
 }

src/Coordination/tests/gtest_coordination.cpp

@@ -1318,7 +1318,7 @@ void testLogAndStateMachine(Coordination::CoordinationSettingsPtr settings, uint
 
     ResponsesQueue queue(std::numeric_limits<size_t>::max());
     SnapshotsQueue snapshots_queue{1};
-    auto state_machine = std::make_shared<KeeperStateMachine>(queue, snapshots_queue, "./snapshots", settings, keeper_context);
+    auto state_machine = std::make_shared<KeeperStateMachine>(queue, snapshots_queue, "./snapshots", settings, keeper_context, nullptr);
     state_machine->init();
     DB::KeeperLogStore changelog("./logs", settings->rotate_log_storage_interval, true, enable_compression);
     changelog.init(state_machine->last_commit_index() + 1, settings->reserved_log_items);
@@ -1359,7 +1359,7 @@ void testLogAndStateMachine(Coordination::CoordinationSettingsPtr settings, uint
     }
 
     SnapshotsQueue snapshots_queue1{1};
-    auto restore_machine = std::make_shared<KeeperStateMachine>(queue, snapshots_queue1, "./snapshots", settings, keeper_context);
+    auto restore_machine = std::make_shared<KeeperStateMachine>(queue, snapshots_queue1, "./snapshots", settings, keeper_context, nullptr);
     restore_machine->init();
     EXPECT_EQ(restore_machine->last_commit_index(), total_logs - total_logs % settings->snapshot_distance);
 
@@ -1471,7 +1471,7 @@ TEST_P(CoordinationTest, TestEphemeralNodeRemove)
 
     ResponsesQueue queue(std::numeric_limits<size_t>::max());
     SnapshotsQueue snapshots_queue{1};
-    auto state_machine = std::make_shared<KeeperStateMachine>(queue, snapshots_queue, "./snapshots", settings, keeper_context);
+    auto state_machine = std::make_shared<KeeperStateMachine>(queue, snapshots_queue, "./snapshots", settings, keeper_context, nullptr);
     state_machine->init();
 
     std::shared_ptr<ZooKeeperCreateRequest> request_c = std::make_shared<ZooKeeperCreateRequest>();

src/Core/Settings.h

@@ -331,8 +331,8 @@ static constexpr UInt64 operator""_GiB(unsigned long long value)
     M(UInt64, max_bytes_before_remerge_sort, 1000000000, "In case of ORDER BY with LIMIT, when memory usage is higher than specified threshold, perform additional steps of merging blocks before final merge to keep just top LIMIT rows.", 0) \
     M(Float, remerge_sort_lowered_memory_bytes_ratio, 2., "If memory usage after remerge does not reduced by this ratio, remerge will be disabled.", 0) \
     \
-    M(UInt64, max_result_rows, 0, "Limit on result size in rows. Also checked for intermediate data sent from remote servers.", 0) \
-    M(UInt64, max_result_bytes, 0, "Limit on result size in bytes (uncompressed). Also checked for intermediate data sent from remote servers.", 0) \
+    M(UInt64, max_result_rows, 0, "Limit on result size in rows. The query will stop after processing a block of data if the threshold is met, but it will not cut the last block of the result, therefore the result size can be larger than the threshold.", 0) \
+    M(UInt64, max_result_bytes, 0, "Limit on result size in bytes (uncompressed).  The query will stop after processing a block of data if the threshold is met, but it will not cut the last block of the result, therefore the result size can be larger than the threshold. Caveats: the result size in memory is taken into account for this threshold. Even if the result size is small, it can reference larger data structures in memory, representing dictionaries of LowCardinality columns, and Arenas of AggregateFunction columns, so the threshold can be exceeded despite the small result size. The setting is fairly low level and should be used with caution.", 0) \
     M(OverflowMode, result_overflow_mode, OverflowMode::THROW, "What to do when the limit is exceeded.", 0) \
     \
     /* TODO: Check also when merging and finalizing aggregate functions. */ \

src/Formats/EscapingRuleUtils.cpp

@@ -11,6 +11,7 @@
 #include <DataTypes/DataTypeTuple.h>
 #include <DataTypes/DataTypeDate.h>
 #include <DataTypes/DataTypeDateTime64.h>
+#include <DataTypes/DataTypeLowCardinality.h>
 #include <DataTypes/DataTypeMap.h>
 #include <DataTypes/DataTypeObject.h>
 #include <DataTypes/getLeastSupertype.h>
@@ -875,4 +876,19 @@ String getAdditionalFormatInfoByEscapingRule(const FormatSettings & settings, Fo
     return result;
 }
 
+
+void checkSupportedDelimiterAfterField(FormatSettings::EscapingRule escaping_rule, const String & delimiter, const DataTypePtr & type)
+{
+    if (escaping_rule != FormatSettings::EscapingRule::Escaped)
+        return;
+
+    bool is_supported_delimiter_after_string = !delimiter.empty() && (delimiter.front() == '\t' || delimiter.front() == '\n');
+    if (is_supported_delimiter_after_string)
+        return;
+
+    /// Nullptr means that field is skipped and it's equivalent to String
+    if (!type || isString(removeNullable(removeLowCardinality(type))))
+        throw Exception(ErrorCodes::BAD_ARGUMENTS, "'Escaped' serialization requires delimiter after String field to start with '\\t' or '\\n'");
+}
+
 }

src/Formats/EscapingRuleUtils.h

@@ -77,6 +77,8 @@ void transformInferredTypesIfNeeded(DataTypePtr & first, DataTypePtr & second, c
 void transformInferredJSONTypesIfNeeded(DataTypes & types, const FormatSettings & settings, const std::unordered_set<const IDataType *> * numbers_parsed_from_json_strings = nullptr);
 void transformInferredJSONTypesIfNeeded(DataTypePtr & first, DataTypePtr & second, const FormatSettings & settings);
 
-String getAdditionalFormatInfoByEscapingRule(const FormatSettings & settings,FormatSettings::EscapingRule escaping_rule);
+String getAdditionalFormatInfoByEscapingRule(const FormatSettings & settings, FormatSettings::EscapingRule escaping_rule);
+
+void checkSupportedDelimiterAfterField(FormatSettings::EscapingRule escaping_rule, const String & delimiter, const DataTypePtr & type);
 
 }

src/Formats/FormatFactory.cpp

@@ -303,7 +303,7 @@ InputFormatPtr FormatFactory::getInputFormat(
 
 static void addExistingProgressToOutputFormat(OutputFormatPtr format, ContextPtr context)
 {
-    auto * element_id = context->getProcessListElement();
+    auto element_id = context->getProcessListElement();
     if (element_id)
     {
         /// While preparing the query there might have been progress (for example in subscalar subqueries) so add it here

src/Functions/CustomWeekTransforms.h

@@ -62,10 +62,7 @@ struct ToStartOfWeekImpl
 
     static inline UInt16 execute(Int64 t, UInt8 week_mode, const DateLUTImpl & time_zone)
     {
-        if (t < 0)
-            return 0;
-
-        return time_zone.toFirstDayNumOfWeek(DayNum(std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM))), week_mode);
+        return time_zone.toFirstDayNumOfWeek(time_zone.toDayNum(t), week_mode);
     }
     static inline UInt16 execute(UInt32 t, UInt8 week_mode, const DateLUTImpl & time_zone)
     {
@@ -73,10 +70,7 @@ struct ToStartOfWeekImpl
     }
     static inline UInt16 execute(Int32 d, UInt8 week_mode, const DateLUTImpl & time_zone)
     {
-        if (d < 0)
-            return 0;
-
-        return time_zone.toFirstDayNumOfWeek(DayNum(std::min(d, Int32(DATE_LUT_MAX_DAY_NUM))), week_mode);
+        return time_zone.toFirstDayNumOfWeek(ExtendedDayNum(d), week_mode);
     }
     static inline UInt16 execute(UInt16 d, UInt8 week_mode, const DateLUTImpl & time_zone)
     {

src/Functions/DateTimeTransforms.h

@@ -55,15 +55,15 @@ struct ToDateImpl
 
     static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return t < 0 ? 0 : std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM));
+        return UInt16(time_zone.toDayNum(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toDayNum(t);
+        return UInt16(time_zone.toDayNum(t));
     }
-    static inline UInt16 execute(Int32 t, const DateLUTImpl &)
+    static inline UInt16 execute(Int32, const DateLUTImpl &)
     {
-        return t < 0 ? 0 : std::min(t, Int32(DATE_LUT_MAX_DAY_NUM));
+        throwDateIsNotSupported(name);
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl &)
     {
@@ -104,10 +104,7 @@ struct ToStartOfDayImpl
 
     static inline UInt32 execute(const DecimalUtils::DecimalComponents<DateTime64> & t, const DateLUTImpl & time_zone)
     {
-        if (t.whole < 0 || (t.whole >= 0 && t.fractional < 0))
-            return 0;
-
-        return time_zone.toDate(std::min<Int64>(t.whole, Int64(0xffffffff)));
+        return time_zone.toDate(static_cast<time_t>(t.whole));
     }
     static inline UInt32 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
@@ -115,19 +112,11 @@ struct ToStartOfDayImpl
     }
     static inline UInt32 execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        if (d < 0)
-            return 0;
-
-        auto date_time = time_zone.fromDayNum(ExtendedDayNum(d));
-        if (date_time <= 0xffffffff)
-            return date_time;
-        else
-            return time_zone.toDate(0xffffffff);
+        return time_zone.toDate(ExtendedDayNum(d));
     }
     static inline UInt32 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
-        auto date_time = time_zone.fromDayNum(ExtendedDayNum(d));
-        return date_time < 0xffffffff ? date_time : time_zone.toDate(0xffffffff);
+        return time_zone.toDate(DayNum(d));
     }
     static inline DecimalUtils::DecimalComponents<DateTime64> executeExtendedResult(const DecimalUtils::DecimalComponents<DateTime64> & t, const DateLUTImpl & time_zone)
     {
@@ -147,16 +136,17 @@ struct ToMondayImpl
 
     static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return t < 0 ? 0 : time_zone.toFirstDayNumOfWeek(ExtendedDayNum(
-                   std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM))));
+        //return time_zone.toFirstDayNumOfWeek(time_zone.toDayNum(t));
+        return time_zone.toFirstDayNumOfWeek(t);
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
+        //return time_zone.toFirstDayNumOfWeek(time_zone.toDayNum(t));
         return time_zone.toFirstDayNumOfWeek(t);
     }
     static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return d < 0 ? 0 : time_zone.toFirstDayNumOfWeek(ExtendedDayNum(std::min(d, Int32(DATE_LUT_MAX_DAY_NUM))));
+        return time_zone.toFirstDayNumOfWeek(ExtendedDayNum(d));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
@@ -179,15 +169,15 @@ struct ToStartOfMonthImpl
 
     static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return t < 0 ? 0 : time_zone.toFirstDayNumOfMonth(ExtendedDayNum(std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM))));
+        return time_zone.toFirstDayNumOfMonth(time_zone.toDayNum(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toFirstDayNumOfMonth(ExtendedDayNum(time_zone.toDayNum(t)));
+        return time_zone.toFirstDayNumOfMonth(time_zone.toDayNum(t));
     }
     static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return d < 0 ? 0 : time_zone.toFirstDayNumOfMonth(ExtendedDayNum(std::min(d, Int32(DATE_LUT_MAX_DAY_NUM))));
+        return time_zone.toFirstDayNumOfMonth(ExtendedDayNum(d));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
@@ -211,11 +201,7 @@ struct ToLastDayOfMonthImpl
 
     static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        if (t < 0)
-            return 0;
-
-        /// 0xFFF9 is Int value for 2149-05-31 -- the last day where we can actually find LastDayOfMonth. This will also be the return value.
-        return time_zone.toLastDayNumOfMonth(ExtendedDayNum(std::min(Int32(time_zone.toDayNum(t)), Int32(0xFFF9))));
+        return time_zone.toLastDayNumOfMonth(time_zone.toDayNum(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
@@ -223,16 +209,11 @@ struct ToLastDayOfMonthImpl
     }
     static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        if (d < 0)
-            return 0;
-
-        /// 0xFFF9 is Int value for 2149-05-31 -- the last day where we can actually find LastDayOfMonth. This will also be the return value.
-        return time_zone.toLastDayNumOfMonth(ExtendedDayNum(std::min(d, Int32(0xFFF9))));
+        return time_zone.toLastDayNumOfMonth(ExtendedDayNum(d));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
-        /// 0xFFF9 is Int value for 2149-05-31 -- the last day where we can actually find LastDayOfMonth. This will also be the return value.
-        return time_zone.toLastDayNumOfMonth(DayNum(std::min(d, UInt16(0xFFF9))));
+        return time_zone.toLastDayNumOfMonth(DayNum(d));
     }
     static inline Int64 executeExtendedResult(Int64 t, const DateLUTImpl & time_zone)
     {
@@ -251,7 +232,7 @@ struct ToStartOfQuarterImpl
 
     static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return t < 0 ? 0 : time_zone.toFirstDayNumOfQuarter(ExtendedDayNum(std::min<Int64>(Int64(time_zone.toDayNum(t)), Int64(DATE_LUT_MAX_DAY_NUM))));
+        return time_zone.toFirstDayNumOfQuarter(time_zone.toDayNum(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
@@ -259,7 +240,7 @@ struct ToStartOfQuarterImpl
     }
     static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return d < 0 ? 0 : time_zone.toFirstDayNumOfQuarter(ExtendedDayNum(std::min<Int32>(d, Int32(DATE_LUT_MAX_DAY_NUM))));
+        return time_zone.toFirstDayNumOfQuarter(ExtendedDayNum(d));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
@@ -282,7 +263,7 @@ struct ToStartOfYearImpl
 
     static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return t < 0 ? 0 : time_zone.toFirstDayNumOfYear(ExtendedDayNum(std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM))));
+        return time_zone.toFirstDayNumOfYear(time_zone.toDayNum(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
@@ -290,7 +271,7 @@ struct ToStartOfYearImpl
     }
     static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return d < 0 ? 0 : time_zone.toFirstDayNumOfYear(ExtendedDayNum(std::min(d, Int32(DATE_LUT_MAX_DAY_NUM))));
+        return time_zone.toFirstDayNumOfYear(ExtendedDayNum(d));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
@@ -340,10 +321,7 @@ struct ToStartOfMinuteImpl
 
     static inline UInt32 execute(const DecimalUtils::DecimalComponents<DateTime64> & t, const DateLUTImpl & time_zone)
     {
-        if (t.whole < 0 || (t.whole >= 0 && t.fractional < 0))
-            return 0;
-
-        return time_zone.toStartOfMinute(std::min<Int64>(t.whole, Int64(0xffffffff)));
+        return time_zone.toStartOfMinute(t.whole);
     }
     static inline UInt32 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
@@ -677,10 +655,7 @@ struct ToStartOfHourImpl
 
     static inline UInt32 execute(const DecimalUtils::DecimalComponents<DateTime64> & t, const DateLUTImpl & time_zone)
     {
-        if (t.whole < 0 || (t.whole >= 0 && t.fractional < 0))
-            return 0;
-
-        return time_zone.toStartOfHour(std::min<Int64>(t.whole, Int64(0xffffffff)));
+        return time_zone.toStartOfHour(t.whole);
     }
 
     static inline UInt32 execute(UInt32 t, const DateLUTImpl & time_zone)
@@ -1034,21 +1009,39 @@ struct ToISOWeekImpl
     using FactorTransform = ToISOYearImpl;
 };
 
+enum class ResultPrecision
+{
+    Standard,
+    Extended
+};
+
+/// Standard precision results (precision_ == ResultPrecision::Standard) potentially lead to overflows when returning values.
+/// This mode is used by SQL functions "toRelative*Num()" which cannot easily be changed due to backward compatibility.
+/// According to documentation, these functions merely need to compute the time difference to a deterministic, fixed point in the past.
+/// As a future TODO, we should fix their behavior in a backwards-compatible way.
+/// See https://github.com/ClickHouse/ClickHouse/issues/41977#issuecomment-1267536814.
+template <ResultPrecision precision_>
 struct ToRelativeYearNumImpl
 {
     static constexpr auto name = "toRelativeYearNum";
 
-    static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
+    static inline auto execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toYear(t);
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int16>(time_zone.toYear(t));
+        else
+            return static_cast<UInt16>(time_zone.toYear(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
         return time_zone.toYear(static_cast<time_t>(t));
     }
-    static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone)
+    static inline auto execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return time_zone.toYear(ExtendedDayNum(d));
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int16>(time_zone.toYear(ExtendedDayNum(d)));
+        else
+            return static_cast<UInt16>(time_zone.toYear(ExtendedDayNum(d)));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
@@ -1058,21 +1051,28 @@ struct ToRelativeYearNumImpl
     using FactorTransform = ZeroTransform;
 };
 
+template <ResultPrecision precision_>
 struct ToRelativeQuarterNumImpl
 {
     static constexpr auto name = "toRelativeQuarterNum";
 
-    static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
+    static inline auto execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeQuarterNum(t);
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int32>(time_zone.toRelativeQuarterNum(t));
+        else
+            return static_cast<UInt16>(time_zone.toRelativeQuarterNum(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
         return time_zone.toRelativeQuarterNum(static_cast<time_t>(t));
     }
-    static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone)
+    static inline auto execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeQuarterNum(ExtendedDayNum(d));
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int32>(time_zone.toRelativeQuarterNum(ExtendedDayNum(d)));
+        else
+            return static_cast<UInt16>(time_zone.toRelativeQuarterNum(ExtendedDayNum(d)));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
@@ -1082,21 +1082,28 @@ struct ToRelativeQuarterNumImpl
     using FactorTransform = ZeroTransform;
 };
 
+template <ResultPrecision precision_>
 struct ToRelativeMonthNumImpl
 {
     static constexpr auto name = "toRelativeMonthNum";
 
-    static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
+    static inline auto execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeMonthNum(t);
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int32>(time_zone.toRelativeMonthNum(t));
+        else
+            return static_cast<UInt16>(time_zone.toRelativeMonthNum(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
         return time_zone.toRelativeMonthNum(static_cast<time_t>(t));
     }
-    static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone)
+    static inline auto execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeMonthNum(ExtendedDayNum(d));
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int32>(time_zone.toRelativeMonthNum(ExtendedDayNum(d)));
+        else
+            return static_cast<UInt16>(time_zone.toRelativeMonthNum(ExtendedDayNum(d)));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
@@ -1106,21 +1113,28 @@ struct ToRelativeMonthNumImpl
     using FactorTransform = ZeroTransform;
 };
 
+template <ResultPrecision precision_>
 struct ToRelativeWeekNumImpl
 {
     static constexpr auto name = "toRelativeWeekNum";
 
-    static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
+    static inline auto execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeWeekNum(t);
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int32>(time_zone.toRelativeWeekNum(t));
+        else
+            return static_cast<UInt16>(time_zone.toRelativeWeekNum(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
         return time_zone.toRelativeWeekNum(static_cast<time_t>(t));
     }
-    static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone)
+    static inline auto execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeWeekNum(ExtendedDayNum(d));
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int32>(time_zone.toRelativeWeekNum(ExtendedDayNum(d)));
+        else
+            return static_cast<UInt16>(time_zone.toRelativeWeekNum(ExtendedDayNum(d)));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
@@ -1130,21 +1144,28 @@ struct ToRelativeWeekNumImpl
     using FactorTransform = ZeroTransform;
 };
 
+template <ResultPrecision precision_>
 struct ToRelativeDayNumImpl
 {
     static constexpr auto name = "toRelativeDayNum";
 
-    static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone)
+    static inline auto execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toDayNum(t);
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int64>(time_zone.toDayNum(t));
+        else
+            return static_cast<UInt16>(time_zone.toDayNum(t));
     }
     static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
         return time_zone.toDayNum(static_cast<time_t>(t));
     }
-    static inline UInt16 execute(Int32 d, const DateLUTImpl &)
+    static inline auto execute(Int32 d, const DateLUTImpl &)
     {
-        return static_cast<ExtendedDayNum>(d);
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int32>(static_cast<ExtendedDayNum>(d));
+        else
+            return static_cast<UInt16>(static_cast<ExtendedDayNum>(d));
     }
     static inline UInt16 execute(UInt16 d, const DateLUTImpl &)
     {
@@ -1154,46 +1175,65 @@ struct ToRelativeDayNumImpl
     using FactorTransform = ZeroTransform;
 };
 
-
+template <ResultPrecision precision_>
 struct ToRelativeHourNumImpl
 {
     static constexpr auto name = "toRelativeHourNum";
 
-    static inline UInt32 execute(Int64 t, const DateLUTImpl & time_zone)
+    static inline auto execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeHourNum(t);
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int64>(time_zone.toStableRelativeHourNum(t));
+        else
+            return static_cast<UInt32>(time_zone.toRelativeHourNum(t));
     }
     static inline UInt32 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeHourNum(static_cast<time_t>(t));
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return time_zone.toStableRelativeHourNum(static_cast<time_t>(t));
+        else
+            return time_zone.toRelativeHourNum(static_cast<time_t>(t));
     }
-    static inline UInt32 execute(Int32 d, const DateLUTImpl & time_zone)
+    static inline auto execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeHourNum(ExtendedDayNum(d));
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int64>(time_zone.toStableRelativeHourNum(ExtendedDayNum(d)));
+        else
+            return static_cast<UInt32>(time_zone.toRelativeHourNum(ExtendedDayNum(d)));
     }
     static inline UInt32 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeHourNum(DayNum(d));
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return time_zone.toStableRelativeHourNum(DayNum(d));
+        else
+            return time_zone.toRelativeHourNum(DayNum(d));
     }
 
     using FactorTransform = ZeroTransform;
 };
 
+template <ResultPrecision precision_>
 struct ToRelativeMinuteNumImpl
 {
     static constexpr auto name = "toRelativeMinuteNum";
 
-    static inline UInt32 execute(Int64 t, const DateLUTImpl & time_zone)
+    static inline auto execute(Int64 t, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeMinuteNum(t);
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int64>(time_zone.toRelativeMinuteNum(t));
+        else
+            return static_cast<UInt32>(time_zone.toRelativeMinuteNum(t));
     }
     static inline UInt32 execute(UInt32 t, const DateLUTImpl & time_zone)
     {
         return time_zone.toRelativeMinuteNum(static_cast<time_t>(t));
     }
-    static inline UInt32 execute(Int32 d, const DateLUTImpl & time_zone)
+    static inline auto execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return time_zone.toRelativeMinuteNum(ExtendedDayNum(d));
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int64>(time_zone.toRelativeMinuteNum(ExtendedDayNum(d)));
+        else
+            return static_cast<UInt32>(time_zone.toRelativeMinuteNum(ExtendedDayNum(d)));
     }
     static inline UInt32 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
@@ -1203,6 +1243,7 @@ struct ToRelativeMinuteNumImpl
     using FactorTransform = ZeroTransform;
 };
 
+template <ResultPrecision precision_>
 struct ToRelativeSecondNumImpl
 {
     static constexpr auto name = "toRelativeSecondNum";
@@ -1215,9 +1256,12 @@ struct ToRelativeSecondNumImpl
     {
         return t;
     }
-    static inline UInt32 execute(Int32 d, const DateLUTImpl & time_zone)
+    static inline auto execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        return time_zone.fromDayNum(ExtendedDayNum(d));
+        if constexpr (precision_ == ResultPrecision::Extended)
+            return static_cast<Int64>(time_zone.fromDayNum(ExtendedDayNum(d)));
+        else
+            return static_cast<UInt32>(time_zone.fromDayNum(ExtendedDayNum(d)));
     }
     static inline UInt32 execute(UInt16 d, const DateLUTImpl & time_zone)
     {

src/Functions/FunctionsConversion.h

@@ -302,11 +302,6 @@ struct ConvertImpl
     }
 };
 
-/** Conversion of Date32 to Date: check bounds.
-  */
-template <typename Name> struct ConvertImpl<DataTypeDate32, DataTypeDate, Name, ConvertDefaultBehaviorTag>
-    : DateTimeTransformImpl<DataTypeDate32, DataTypeDate, ToDateImpl> {};
-
 /** Conversion of DateTime to Date: throw off time component.
   */
 template <typename Name> struct ConvertImpl<DataTypeDateTime, DataTypeDate, Name, ConvertDefaultBehaviorTag>
@@ -325,17 +320,12 @@ struct ToDateTimeImpl
 
     static UInt32 execute(UInt16 d, const DateLUTImpl & time_zone)
     {
-        auto date_time = time_zone.fromDayNum(ExtendedDayNum(d));
-        return date_time <= 0xffffffff ? UInt32(date_time) : UInt32(0xffffffff);
+        return time_zone.fromDayNum(DayNum(d));
     }
 
-    static UInt32 execute(Int32 d, const DateLUTImpl & time_zone)
+    static Int64 execute(Int32 d, const DateLUTImpl & time_zone)
     {
-        if (d < 0)
-            return 0;
-
-        auto date_time = time_zone.fromDayNum(ExtendedDayNum(d));
-        return date_time <= 0xffffffff ? date_time : 0xffffffff;
+        return time_zone.fromDayNum(ExtendedDayNum(d));
     }
 
     static UInt32 execute(UInt32 dt, const DateLUTImpl & /*time_zone*/)
@@ -343,21 +333,10 @@ struct ToDateTimeImpl
         return dt;
     }
 
-    static UInt32 execute(Int64 d, const DateLUTImpl & time_zone)
+    // TODO: return UInt32 ???
+    static Int64 execute(Int64 dt64, const DateLUTImpl & /*time_zone*/)
     {
-        if (d < 0)
-            return 0;
-
-        auto date_time = time_zone.toDate(d);
-        return date_time <= 0xffffffff ? date_time : 0xffffffff;
-    }
-
-    static UInt32 execute(const DecimalUtils::DecimalComponents<DateTime64> & t, const DateLUTImpl & /*time_zone*/)
-    {
-        if (t.whole < 0 || (t.whole >= 0 && t.fractional < 0))
-            return 0;
-
-        return std::min<Int64>(t.whole, Int64(0xFFFFFFFF));
+        return dt64;
     }
 };
 
@@ -377,12 +356,9 @@ struct ToDateTransform32Or64
     static NO_SANITIZE_UNDEFINED ToType execute(const FromType & from, const DateLUTImpl & time_zone)
     {
         // since converting to Date, no need in values outside of default LUT range.
-        if (from < 0)
-            return 0;
-
         return (from < DATE_LUT_MAX_DAY_NUM)
             ? from
-            : std::min<Int32>(Int32(time_zone.toDayNum(from)), Int32(DATE_LUT_MAX_DAY_NUM));
+            : time_zone.toDayNum(std::min(time_t(from), time_t(0xFFFFFFFF)));
     }
 };
 
@@ -397,14 +373,9 @@ struct ToDateTransform32Or64Signed
         /// The function should be monotonic (better for query optimizations), so we saturate instead of overflow.
         if (from < 0)
             return 0;
-
-        auto day_num = time_zone.toDayNum(ExtendedDayNum(static_cast<Int32>(from)));
-        return day_num < DATE_LUT_MAX_DAY_NUM ? day_num : DATE_LUT_MAX_DAY_NUM;
-
         return (from < DATE_LUT_MAX_DAY_NUM)
-            ? from
-            : std::min<Int32>(Int32(time_zone.toDayNum(static_cast<UInt16>(from))), Int32(0xFFFFFFFF));
-
+            ? static_cast<ToType>(from)
+            : time_zone.toDayNum(std::min(time_t(from), time_t(0xFFFFFFFF)));
     }
 };
 
@@ -435,7 +406,7 @@ struct ToDate32Transform32Or64
     {
         return (from < DATE_LUT_MAX_EXTEND_DAY_NUM)
             ? from
-            : std::min<Int32>(Int32(time_zone.toDayNum(from)), Int32(DATE_LUT_MAX_EXTEND_DAY_NUM));
+            : time_zone.toDayNum(std::min(time_t(from), time_t(0xFFFFFFFF)));
     }
 };
 
@@ -451,7 +422,7 @@ struct ToDate32Transform32Or64Signed
             return daynum_min_offset;
         return (from < DATE_LUT_MAX_EXTEND_DAY_NUM)
             ? static_cast<ToType>(from)
-            : time_zone.toDayNum(std::min<Int64>(Int64(from), Int64(0xFFFFFFFF)));
+            : time_zone.toDayNum(std::min(time_t(Int64(from)), time_t(0xFFFFFFFF)));
     }
 };
 
@@ -477,49 +448,35 @@ struct ToDate32Transform8Or16Signed
   */
 template <typename Name> struct ConvertImpl<DataTypeUInt32, DataTypeDate, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeUInt32, DataTypeDate, ToDateTransform32Or64<UInt32, UInt16>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeUInt64, DataTypeDate, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeUInt64, DataTypeDate, ToDateTransform32Or64<UInt64, UInt16>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeInt8, DataTypeDate, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeInt8, DataTypeDate, ToDateTransform8Or16Signed<Int8, UInt16>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeInt16, DataTypeDate, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeInt16, DataTypeDate, ToDateTransform8Or16Signed<Int16, UInt16>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeInt32, DataTypeDate, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeInt32, DataTypeDate, ToDateTransform32Or64Signed<Int32, UInt16>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeInt64, DataTypeDate, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeInt64, DataTypeDate, ToDateTransform32Or64Signed<Int64, UInt16>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeFloat32, DataTypeDate, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeFloat32, DataTypeDate, ToDateTransform32Or64Signed<Float32, UInt16>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeFloat64, DataTypeDate, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeFloat64, DataTypeDate, ToDateTransform32Or64Signed<Float64, UInt16>> {};
 
 template <typename Name> struct ConvertImpl<DataTypeUInt32, DataTypeDate32, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeUInt32, DataTypeDate32, ToDate32Transform32Or64<UInt32, Int32>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeUInt64, DataTypeDate32, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeUInt64, DataTypeDate32, ToDate32Transform32Or64<UInt64, Int32>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeInt8, DataTypeDate32, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeInt8, DataTypeDate32, ToDate32Transform8Or16Signed<Int8, Int32>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeInt16, DataTypeDate32, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeInt16, DataTypeDate32, ToDate32Transform8Or16Signed<Int16, Int32>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeInt32, DataTypeDate32, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeInt32, DataTypeDate32, ToDate32Transform32Or64Signed<Int32, Int32>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeInt64, DataTypeDate32, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeInt64, DataTypeDate32, ToDate32Transform32Or64Signed<Int64, Int32>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeFloat32, DataTypeDate32, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeFloat32, DataTypeDate32, ToDate32Transform32Or64Signed<Float32, Int32>> {};
-
 template <typename Name> struct ConvertImpl<DataTypeFloat64, DataTypeDate32, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeFloat64, DataTypeDate32, ToDate32Transform32Or64Signed<Float64, Int32>> {};
 
@@ -531,7 +488,7 @@ struct ToDateTimeTransform64
 
     static NO_SANITIZE_UNDEFINED ToType execute(const FromType & from, const DateLUTImpl &)
     {
-        return std::min<Int64>(Int64(from), Int64(0xFFFFFFFF));
+        return std::min(time_t(from), time_t(0xFFFFFFFF));
     }
 };
 
@@ -553,12 +510,11 @@ struct ToDateTimeTransform64Signed
 {
     static constexpr auto name = "toDateTime";
 
-    static NO_SANITIZE_UNDEFINED ToType execute(const FromType & from, const DateLUTImpl & /* time_zone */)
+    static NO_SANITIZE_UNDEFINED ToType execute(const FromType & from, const DateLUTImpl &)
     {
         if (from < 0)
             return 0;
-
-        return std::min<Int64>(Int64(from), Int64(0xFFFFFFFF));
+        return std::min(time_t(from), time_t(0xFFFFFFFF));
     }
 };
 
@@ -678,6 +634,8 @@ struct FromDateTime64Transform
     }
 };
 
+/** Conversion of DateTime64 to Date or DateTime: discards fractional part.
+ */
 template <typename Name> struct ConvertImpl<DataTypeDateTime64, DataTypeDate, Name, ConvertDefaultBehaviorTag>
     : DateTimeTransformImpl<DataTypeDateTime64, DataTypeDate, TransformDateTime64<ToDateImpl>> {};
 template <typename Name> struct ConvertImpl<DataTypeDateTime64, DataTypeDateTime, Name, ConvertDefaultBehaviorTag>
@@ -701,7 +659,7 @@ struct ToDateTime64Transform
 
     DateTime64::NativeType execute(Int32 d, const DateLUTImpl & time_zone) const
     {
-        const auto dt = time_zone.fromDayNum(ExtendedDayNum(d));
+        const auto dt = ToDateTimeImpl::execute(d, time_zone);
         return DecimalUtils::decimalFromComponentsWithMultiplier<DateTime64>(dt, 0, scale_multiplier);
     }
 
@@ -1855,7 +1813,7 @@ private:
                 {
                     /// Account for optional timezone argument.
                     if (arguments.size() != 2 && arguments.size() != 3)
-                        throw Exception{"Function " + getName() + " expects 2 or 3 arguments for DateTime64.",
+                        throw Exception{"Function " + getName() + " expects 2 or 3 arguments for DataTypeDateTime64.",
                             ErrorCodes::TOO_FEW_ARGUMENTS_FOR_FUNCTION};
                 }
                 else if (arguments.size() != 2)

src/Functions/GregorianDate.h

@@ -38,7 +38,7 @@ namespace DB
           * integral type which should be at least 32 bits wide, and
           * should preferably signed.
           */
-        explicit GregorianDate(is_integer auto mjd);
+        explicit GregorianDate(is_integer auto modified_julian_day);
 
         /** Convert to Modified Julian Day. The type T is an integral type
           * which should be at least 32 bits wide, and should preferably
@@ -89,7 +89,8 @@ namespace DB
           * integral type which should be at least 32 bits wide, and
           * should preferably signed.
           */
-        explicit OrdinalDate(is_integer auto mjd);
+        template <is_integer DayT>
+        explicit OrdinalDate(DayT modified_julian_day);
 
         /** Convert to Modified Julian Day. The type T is an integral
           * type which should be at least 32 bits wide, and should
@@ -257,9 +258,9 @@ namespace DB
     }
 
     template <typename YearT>
-    GregorianDate<YearT>::GregorianDate(is_integer auto mjd)
+    GregorianDate<YearT>::GregorianDate(is_integer auto modified_julian_day)
     {
-        const OrdinalDate<YearT> ord(mjd);
+        const OrdinalDate<YearT> ord(modified_julian_day);
         const MonthDay md(gd::is_leap_year(ord.year()), ord.dayOfYear());
         year_       = ord.year();
         month_      = md.month();
@@ -329,9 +330,24 @@ namespace DB
     }
 
     template <typename YearT>
-    OrdinalDate<YearT>::OrdinalDate(is_integer auto mjd)
+    template <is_integer DayT>
+    OrdinalDate<YearT>::OrdinalDate(DayT modified_julian_day)
     {
-        const auto a         = mjd + 678575;
+        /// This function supports day number from -678941 to 2973119 (which represent 0000-01-01 and 9999-12-31 respectively).
+
+        if constexpr (is_signed_v<DayT> && std::numeric_limits<DayT>::lowest() < -678941)
+            if (modified_julian_day < -678941)
+                throw Exception(
+                    ErrorCodes::CANNOT_FORMAT_DATETIME,
+                    "Value cannot be represented as date because it's out of range");
+
+        if constexpr (std::numeric_limits<DayT>::max() > 2973119)
+            if (modified_julian_day > 2973119)
+                throw Exception(
+                    ErrorCodes::CANNOT_FORMAT_DATETIME,
+                    "Value cannot be represented as date because it's out of range");
+
+        const auto a         = modified_julian_day + 678575;
         const auto quad_cent = gd::div(a, 146097);
         const auto b         = gd::mod(a, 146097);
         const auto cent      = gd::min(gd::div(b, 36524), 3);
@@ -339,8 +355,9 @@ namespace DB
         const auto quad      = gd::div(c, 1461);
         const auto d         = gd::mod(c, 1461);
         const auto y         = gd::min(gd::div(d, 365), 3);
+
         day_of_year_ = d - y * 365 + 1;
-        year_      = quad_cent * 400 + cent * 100 + quad * 4 + y + 1;
+        year_ = quad_cent * 400 + cent * 100 + quad * 4 + y + 1;
     }
 
     template <typename YearT>

src/Functions/dateDiff.cpp

@@ -61,25 +61,30 @@ public:
     DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
     {
         if (arguments.size() != 3 && arguments.size() != 4)
-            throw Exception("Number of arguments for function " + getName() + " doesn't match: passed "
-                + toString(arguments.size()) + ", should be 3 or 4",
-                ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
+                "Number of arguments for function {} doesn't match: passed {}, should be 3 or 4",
+                getName(), arguments.size());
 
         if (!isString(arguments[0]))
-            throw Exception("First argument for function " + getName() + " (unit) must be String",
-                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "First argument for function {} (unit) must be String",
+                getName());
 
-        if (!isDate(arguments[1]) && !isDateTime(arguments[1]) && !isDateTime64(arguments[1]))
-            throw Exception("Second argument for function " + getName() + " must be Date or DateTime",
-                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
+        if (!isDate(arguments[1]) && !isDate32(arguments[1]) && !isDateTime(arguments[1]) && !isDateTime64(arguments[1]))
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Second argument for function {} must be Date, Date32, DateTime or DateTime64",
+                getName());
 
-        if (!isDate(arguments[2]) && !isDateTime(arguments[2]) && !isDateTime64(arguments[2]))
-            throw Exception("Third argument for function " + getName() + " must be Date or DateTime",
-                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
+        if (!isDate(arguments[2]) && !isDate32(arguments[2]) && !isDateTime(arguments[2]) && !isDateTime64(arguments[2]))
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Third argument for function {} must be Date, Date32, DateTime or DateTime64",
+                getName()
+                );
 
         if (arguments.size() == 4 && !isString(arguments[3]))
-            throw Exception("Fourth argument for function " + getName() + " (timezone) must be String",
-                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Fourth argument for function {} (timezone) must be String",
+                getName());
 
         return std::make_shared<DataTypeInt64>();
     }
@@ -91,7 +96,9 @@ public:
     {
         const auto * unit_column = checkAndGetColumnConst<ColumnString>(arguments[0].column.get());
         if (!unit_column)
-            throw Exception("First argument for function " + getName() + " must be constant String", ErrorCodes::ILLEGAL_COLUMN);
+            throw Exception(ErrorCodes::ILLEGAL_COLUMN,
+                "First argument for function {} must be constant String",
+                getName());
 
         String unit = Poco::toLower(unit_column->getValue<String>());
 
@@ -105,23 +112,24 @@ public:
         const auto & timezone_y = extractTimeZoneFromFunctionArguments(arguments, 3, 2);
 
         if (unit == "year" || unit == "yy" || unit == "yyyy")
-            dispatchForColumns<ToRelativeYearNumImpl>(x, y, timezone_x, timezone_y, res->getData());
+            dispatchForColumns<ToRelativeYearNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
         else if (unit == "quarter" || unit == "qq" || unit == "q")
-            dispatchForColumns<ToRelativeQuarterNumImpl>(x, y, timezone_x, timezone_y, res->getData());
+            dispatchForColumns<ToRelativeQuarterNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
         else if (unit == "month" || unit == "mm" || unit == "m")
-            dispatchForColumns<ToRelativeMonthNumImpl>(x, y, timezone_x, timezone_y, res->getData());
+            dispatchForColumns<ToRelativeMonthNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
         else if (unit == "week" || unit == "wk" || unit == "ww")
-            dispatchForColumns<ToRelativeWeekNumImpl>(x, y, timezone_x, timezone_y, res->getData());
+            dispatchForColumns<ToRelativeWeekNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
         else if (unit == "day" || unit == "dd" || unit == "d")
-            dispatchForColumns<ToRelativeDayNumImpl>(x, y, timezone_x, timezone_y, res->getData());
+            dispatchForColumns<ToRelativeDayNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
         else if (unit == "hour" || unit == "hh" || unit == "h")
-            dispatchForColumns<ToRelativeHourNumImpl>(x, y, timezone_x, timezone_y, res->getData());
+            dispatchForColumns<ToRelativeHourNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
         else if (unit == "minute" || unit == "mi" || unit == "n")
-            dispatchForColumns<ToRelativeMinuteNumImpl>(x, y, timezone_x, timezone_y, res->getData());
+            dispatchForColumns<ToRelativeMinuteNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
         else if (unit == "second" || unit == "ss" || unit == "s")
-            dispatchForColumns<ToRelativeSecondNumImpl>(x, y, timezone_x, timezone_y, res->getData());
+            dispatchForColumns<ToRelativeSecondNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
         else
-            throw Exception("Function " + getName() + " does not support '" + unit + "' unit", ErrorCodes::BAD_ARGUMENTS);
+            throw Exception(ErrorCodes::BAD_ARGUMENTS,
+                "Function {} does not support '{}' unit", getName(), unit);
 
         return res;
     }
@@ -137,16 +145,22 @@ private:
             dispatchForSecondColumn<Transform>(*x_vec_16, y, timezone_x, timezone_y, result);
         else if (const auto * x_vec_32 = checkAndGetColumn<ColumnUInt32>(&x))
             dispatchForSecondColumn<Transform>(*x_vec_32, y, timezone_x, timezone_y, result);
+        else if (const auto * x_vec_32_s = checkAndGetColumn<ColumnInt32>(&x))
+            dispatchForSecondColumn<Transform>(*x_vec_32_s, y, timezone_x, timezone_y, result);
         else if (const auto * x_vec_64 = checkAndGetColumn<ColumnDateTime64>(&x))
             dispatchForSecondColumn<Transform>(*x_vec_64, y, timezone_x, timezone_y, result);
         else if (const auto * x_const_16 = checkAndGetColumnConst<ColumnUInt16>(&x))
             dispatchConstForSecondColumn<Transform>(x_const_16->getValue<UInt16>(), y, timezone_x, timezone_y, result);
         else if (const auto * x_const_32 = checkAndGetColumnConst<ColumnUInt32>(&x))
             dispatchConstForSecondColumn<Transform>(x_const_32->getValue<UInt32>(), y, timezone_x, timezone_y, result);
+        else if (const auto * x_const_32_s = checkAndGetColumnConst<ColumnInt32>(&x))
+            dispatchConstForSecondColumn<Transform>(x_const_32_s->getValue<Int32>(), y, timezone_x, timezone_y, result);
         else if (const auto * x_const_64 = checkAndGetColumnConst<ColumnDateTime64>(&x))
             dispatchConstForSecondColumn<Transform>(x_const_64->getValue<DecimalField<DateTime64>>(), y, timezone_x, timezone_y, result);
         else
-            throw Exception("Illegal column for first argument of function " + getName() + ", must be Date, DateTime or DateTime64", ErrorCodes::ILLEGAL_COLUMN);
+            throw Exception(ErrorCodes::ILLEGAL_COLUMN,
+                "Illegal column for first argument of function {}, must be Date, Date32, DateTime or DateTime64",
+                getName());
     }
 
     template <typename Transform, typename LeftColumnType>
@@ -159,16 +173,22 @@ private:
             vectorVector<Transform>(x, *y_vec_16, timezone_x, timezone_y, result);
         else if (const auto * y_vec_32 = checkAndGetColumn<ColumnUInt32>(&y))
             vectorVector<Transform>(x, *y_vec_32, timezone_x, timezone_y, result);
+        else if (const auto * y_vec_32_s = checkAndGetColumn<ColumnInt32>(&y))
+            vectorVector<Transform>(x, *y_vec_32_s, timezone_x, timezone_y, result);
         else if (const auto * y_vec_64 = checkAndGetColumn<ColumnDateTime64>(&y))
             vectorVector<Transform>(x, *y_vec_64, timezone_x, timezone_y, result);
         else if (const auto * y_const_16 = checkAndGetColumnConst<ColumnUInt16>(&y))
             vectorConstant<Transform>(x, y_const_16->getValue<UInt16>(), timezone_x, timezone_y, result);
         else if (const auto * y_const_32 = checkAndGetColumnConst<ColumnUInt32>(&y))
             vectorConstant<Transform>(x, y_const_32->getValue<UInt32>(), timezone_x, timezone_y, result);
+        else if (const auto * y_const_32_s = checkAndGetColumnConst<ColumnInt32>(&y))
+            vectorConstant<Transform>(x, y_const_32_s->getValue<Int32>(), timezone_x, timezone_y, result);
         else if (const auto * y_const_64 = checkAndGetColumnConst<ColumnDateTime64>(&y))
             vectorConstant<Transform>(x, y_const_64->getValue<DecimalField<DateTime64>>(), timezone_x, timezone_y, result);
         else
-            throw Exception("Illegal column for second argument of function " + getName() + ", must be Date or DateTime", ErrorCodes::ILLEGAL_COLUMN);
+            throw Exception(ErrorCodes::ILLEGAL_COLUMN,
+                "Illegal column for second argument of function {}, must be Date, Date32, DateTime or DateTime64",
+                getName());
     }
 
     template <typename Transform, typename T1>
@@ -181,10 +201,14 @@ private:
             constantVector<Transform>(x, *y_vec_16, timezone_x, timezone_y, result);
         else if (const auto * y_vec_32 = checkAndGetColumn<ColumnUInt32>(&y))
             constantVector<Transform>(x, *y_vec_32, timezone_x, timezone_y, result);
+        else if (const auto * y_vec_32_s = checkAndGetColumn<ColumnInt32>(&y))
+            constantVector<Transform>(x, *y_vec_32_s, timezone_x, timezone_y, result);
         else if (const auto * y_vec_64 = checkAndGetColumn<ColumnDateTime64>(&y))
             constantVector<Transform>(x, *y_vec_64, timezone_x, timezone_y, result);
         else
-            throw Exception("Illegal column for second argument of function " + getName() + ", must be Date or DateTime", ErrorCodes::ILLEGAL_COLUMN);
+            throw Exception(ErrorCodes::ILLEGAL_COLUMN,
+                "Illegal column for second argument of function {}, must be Date, Date32, DateTime or DateTime64",
+                getName());
     }
 
     template <typename Transform, typename LeftColumnType, typename RightColumnType>

src/Functions/dateName.cpp

@@ -4,6 +4,7 @@
 #include <IO/WriteHelpers.h>
 
 #include <DataTypes/DataTypeDate.h>
+#include <DataTypes/DataTypeDate32.h>
 #include <DataTypes/DataTypeDateTime.h>
 #include <DataTypes/DataTypeDateTime64.h>
 #include <DataTypes/DataTypeString.h>
@@ -34,6 +35,11 @@ template <> struct DataTypeToTimeTypeMap<DataTypeDate>
     using TimeType = UInt16;
 };
 
+template <> struct DataTypeToTimeTypeMap<DataTypeDate32>
+{
+    using TimeType = Int32;
+};
+
 template <> struct DataTypeToTimeTypeMap<DataTypeDateTime>
 {
     using TimeType = UInt32;
@@ -72,7 +78,7 @@ public:
                 ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
                 "Number of arguments for function {} doesn't match: passed {}",
                 getName(),
-                toString(arguments.size()));
+                arguments.size());
 
         if (!WhichDataType(arguments[0].type).isString())
             throw Exception(
@@ -83,7 +89,7 @@ public:
 
         WhichDataType first_argument_type(arguments[1].type);
 
-        if (!(first_argument_type.isDate() || first_argument_type.isDateTime() || first_argument_type.isDateTime64()))
+        if (!(first_argument_type.isDate() || first_argument_type.isDateTime() || first_argument_type.isDate32() || first_argument_type.isDateTime64()))
             throw Exception(
                 ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
                 "Illegal type {} of 2 argument of function {}. Must be a date or a date with time",
@@ -108,6 +114,7 @@ public:
         ColumnPtr res;
 
         if (!((res = executeType<DataTypeDate>(arguments, result_type))
+            || (res = executeType<DataTypeDate32>(arguments, result_type))
             || (res = executeType<DataTypeDateTime>(arguments, result_type))
             || (res = executeType<DataTypeDateTime64>(arguments, result_type))))
             throw Exception(

src/Functions/randDistribution.cpp

@@ -0,0 +1,472 @@
+#include <Functions/IFunction.h>
+#include <Functions/FunctionHelpers.h>
+#include <Functions/FunctionFactory.h>
+#include "Common/Exception.h"
+#include <Common/NaNUtils.h>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnsNumber.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Common/FieldVisitorConvertToNumber.h>
+#include <Common/ProfileEvents.h>
+#include <Common/assert_cast.h>
+#include <IO/WriteHelpers.h>
+#include <Interpreters/Context_fwd.h>
+
+#include <random>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int ILLEGAL_COLUMN;
+    extern const int BAD_ARGUMENTS;
+    extern const int LOGICAL_ERROR;
+}
+
+namespace
+{
+struct UniformDistribution
+{
+    using ReturnType = DataTypeFloat64;
+    static constexpr const char * getName() { return "randUniform"; }
+    static constexpr size_t getNumberOfArguments() { return 2; }
+
+    static void generate(Float64 min, Float64 max, ColumnFloat64::Container & container)
+    {
+        auto distribution = std::uniform_real_distribution<>(min, max);
+        for (auto & elem : container)
+            elem = distribution(thread_local_rng);
+    }
+};
+
+struct NormalDistribution
+{
+    using ReturnType = DataTypeFloat64;
+    static constexpr const char * getName() { return "randNormal"; }
+    static constexpr size_t getNumberOfArguments() { return 2; }
+
+    static void generate(Float64 mean, Float64 variance, ColumnFloat64::Container & container)
+    {
+        auto distribution = std::normal_distribution<>(mean, variance);
+        for (auto & elem : container)
+            elem = distribution(thread_local_rng);
+    }
+};
+
+struct LogNormalDistribution
+{
+    using ReturnType = DataTypeFloat64;
+    static constexpr const char * getName() { return "randLogNormal"; }
+    static constexpr size_t getNumberOfArguments() { return 2; }
+
+    static void generate(Float64 mean, Float64 variance, ColumnFloat64::Container & container)
+    {
+        auto distribution = std::lognormal_distribution<>(mean, variance);
+        for (auto & elem : container)
+            elem = distribution(thread_local_rng);
+    }
+};
+
+struct ExponentialDistribution
+{
+    using ReturnType = DataTypeFloat64;
+    static constexpr const char * getName() { return "randExponential"; }
+    static constexpr size_t getNumberOfArguments() { return 1; }
+
+    static void generate(Float64 lambda, ColumnFloat64::Container & container)
+    {
+        auto distribution = std::exponential_distribution<>(lambda);
+        for (auto & elem : container)
+            elem = distribution(thread_local_rng);
+    }
+};
+
+struct ChiSquaredDistribution
+{
+    using ReturnType = DataTypeFloat64;
+    static constexpr const char * getName() { return "randChiSquared"; }
+    static constexpr size_t getNumberOfArguments() { return 1; }
+
+    static void generate(Float64 degree_of_freedom, ColumnFloat64::Container & container)
+    {
+        auto distribution = std::chi_squared_distribution<>(degree_of_freedom);
+        for (auto & elem : container)
+            elem = distribution(thread_local_rng);
+    }
+};
+
+struct StudentTDistribution
+{
+    using ReturnType = DataTypeFloat64;
+    static constexpr const char * getName() { return "randStudentT"; }
+    static constexpr size_t getNumberOfArguments() { return 1; }
+
+    static void generate(Float64 degree_of_freedom, ColumnFloat64::Container & container)
+    {
+        auto distribution = std::student_t_distribution<>(degree_of_freedom);
+        for (auto & elem : container)
+            elem = distribution(thread_local_rng);
+    }
+};
+
+struct FisherFDistribution
+{
+    using ReturnType = DataTypeFloat64;
+    static constexpr const char * getName() { return "randFisherF"; }
+    static constexpr size_t getNumberOfArguments() { return 2; }
+
+    static void generate(Float64 d1, Float64 d2, ColumnFloat64::Container & container)
+    {
+        auto distribution = std::fisher_f_distribution<>(d1, d2);
+        for (auto & elem : container)
+            elem = distribution(thread_local_rng);
+    }
+};
+
+struct BernoulliDistribution
+{
+    using ReturnType = DataTypeUInt8;
+    static constexpr const char * getName() { return "randBernoulli"; }
+    static constexpr size_t getNumberOfArguments() { return 1; }
+
+    static void generate(Float64 p, ColumnUInt8::Container & container)
+    {
+        if (p < 0.0f || p > 1.0f)
+            throw Exception(ErrorCodes::BAD_ARGUMENTS, "Argument of function {} should be inside [0, 1] because it is a probability", getName());
+
+        auto distribution = std::bernoulli_distribution(p);
+        for (auto & elem : container)
+            elem = static_cast<UInt8>(distribution(thread_local_rng));
+    }
+};
+
+struct BinomialDistribution
+{
+    using ReturnType = DataTypeUInt64;
+    static constexpr const char * getName() { return "randBinomial"; }
+    static constexpr size_t getNumberOfArguments() { return 2; }
+
+    static void generate(UInt64 t, Float64 p, ColumnUInt64::Container & container)
+    {
+        if (p < 0.0f || p > 1.0f)
+            throw Exception(ErrorCodes::BAD_ARGUMENTS, "Argument of function {} should be inside [0, 1] because it is a probability", getName());
+
+        auto distribution = std::binomial_distribution(t, p);
+        for (auto & elem : container)
+            elem = static_cast<UInt64>(distribution(thread_local_rng));
+    }
+};
+
+struct NegativeBinomialDistribution
+{
+    using ReturnType = DataTypeUInt64;
+    static constexpr const char * getName() { return "randNegativeBinomial"; }
+    static constexpr size_t getNumberOfArguments() { return 2; }
+
+    static void generate(UInt64 t, Float64 p, ColumnUInt64::Container & container)
+    {
+        if (p < 0.0f || p > 1.0f)
+            throw Exception(ErrorCodes::BAD_ARGUMENTS, "Argument of function {} should be inside [0, 1] because it is a probability", getName());
+
+        auto distribution = std::negative_binomial_distribution(t, p);
+        for (auto & elem : container)
+            elem = static_cast<UInt64>(distribution(thread_local_rng));
+    }
+};
+
+struct PoissonDistribution
+{
+    using ReturnType = DataTypeUInt64;
+    static constexpr const char * getName() { return "randPoisson"; }
+    static constexpr size_t getNumberOfArguments() { return 1; }
+
+    static void generate(UInt64 n, ColumnUInt64::Container & container)
+    {
+        auto distribution = std::poisson_distribution(n);
+        for (auto & elem : container)
+            elem = static_cast<UInt64>(distribution(thread_local_rng));
+    }
+};
+
+}
+
+/** Function which will generate values according to the specified distribution
+  * Accepts only constant arguments
+  * Similar to the functions rand and rand64 an additional 'tag' argument could be added to the
+  * end of arguments list (this argument will be ignored) which will guarantee that functions are not sticked together
+  * during optimisations.
+  * Example: SELECT randNormal(0, 1, 1), randNormal(0, 1, 2) FROM numbers(10)
+  * This query will return two different columns
+  */
+template <typename Distribution>
+class FunctionRandomDistribution : public IFunction
+{
+private:
+
+    template <typename ResultType>
+    ResultType getParameterFromConstColumn(size_t parameter_number, const ColumnsWithTypeAndName & arguments) const
+    {
+        if (parameter_number >= arguments.size())
+            throw Exception(
+                ErrorCodes::LOGICAL_ERROR, "Parameter number ({}) is greater than the size of arguments ({}). This is a bug", parameter_number, arguments.size());
+
+        const IColumn * col = arguments[parameter_number].column.get();
+
+        if (!isColumnConst(*col))
+            throw Exception(ErrorCodes::ILLEGAL_COLUMN, "Parameter number {} of function must be constant.", parameter_number, getName());
+
+        auto parameter = applyVisitor(FieldVisitorConvertToNumber<ResultType>(), assert_cast<const ColumnConst &>(*col).getField());
+
+        if (isNaN(parameter) || !std::isfinite(parameter))
+            throw Exception(ErrorCodes::BAD_ARGUMENTS, "Parameter number {} of function {} cannot be NaN of infinite", parameter_number, getName());
+
+        return parameter;
+    }
+
+public:
+    static FunctionPtr create(ContextPtr)
+    {
+        return std::make_shared<FunctionRandomDistribution<Distribution>>();
+    }
+
+    static constexpr auto name = Distribution::getName();
+    String getName() const override { return name; }
+    size_t getNumberOfArguments() const override { return Distribution::getNumberOfArguments(); }
+    bool isVariadic() const override { return true; }
+    bool isDeterministic() const override { return false; }
+    bool isDeterministicInScopeOfQuery() const override { return false; }
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo & /*arguments*/) const override { return false; }
+
+    DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
+    {
+        auto desired = Distribution::getNumberOfArguments();
+        if (arguments.size() != desired && arguments.size() != desired + 1)
+            throw Exception(ErrorCodes::BAD_ARGUMENTS, "Wrong number of arguments for function {}. Should be {} or {}", getName(), desired, desired + 1);
+
+        for (size_t i = 0; i < Distribution::getNumberOfArguments(); ++i)
+        {
+            const auto & type = arguments[i];
+            WhichDataType which(type);
+            if (!which.isFloat() && !which.isNativeUInt())
+                throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                    "Illegal type {} of argument of function {}, expected Float64 or integer", type->getName(), getName());
+        }
+
+        return std::make_shared<typename Distribution::ReturnType>();
+    }
+
+    ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr & /*result_type*/, size_t input_rows_count) const override
+    {
+        if constexpr (std::is_same_v<Distribution, BernoulliDistribution>)
+        {
+            auto res_column = ColumnUInt8::create(input_rows_count);
+            auto & res_data = res_column->getData();
+            Distribution::generate(getParameterFromConstColumn<Float64>(0, arguments), res_data);
+            return res_column;
+        }
+        else if constexpr (std::is_same_v<Distribution, BinomialDistribution> || std::is_same_v<Distribution, NegativeBinomialDistribution>)
+        {
+            auto res_column = ColumnUInt64::create(input_rows_count);
+            auto & res_data = res_column->getData();
+            Distribution::generate(getParameterFromConstColumn<UInt64>(0, arguments), getParameterFromConstColumn<Float64>(1, arguments), res_data);
+            return res_column;
+        }
+        else if constexpr (std::is_same_v<Distribution, PoissonDistribution>)
+        {
+            auto res_column = ColumnUInt64::create(input_rows_count);
+            auto & res_data = res_column->getData();
+            Distribution::generate(getParameterFromConstColumn<UInt64>(0, arguments), res_data);
+            return res_column;
+        }
+        else
+        {
+            auto res_column = ColumnFloat64::create(input_rows_count);
+            auto & res_data = res_column->getData();
+            if constexpr (Distribution::getNumberOfArguments() == 1)
+            {
+                Distribution::generate(getParameterFromConstColumn<Float64>(0, arguments), res_data);
+            }
+            else if constexpr (Distribution::getNumberOfArguments() == 2)
+            {
+                Distribution::generate(getParameterFromConstColumn<Float64>(0, arguments), getParameterFromConstColumn<Float64>(1, arguments), res_data);
+            }
+            else
+            {
+                throw Exception(ErrorCodes::BAD_ARGUMENTS, "More than two argument specified for function {}", getName());
+            }
+
+            return res_column;
+        }
+    }
+};
+
+
+REGISTER_FUNCTION(Distribution)
+{
+    factory.registerFunction<FunctionRandomDistribution<UniformDistribution>>(
+    {
+    R"(
+Returns a random number from the uniform distribution in the specified range.
+Accepts two parameters - minimum bound and maximum bound.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randUniform(0, 1) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+    factory.registerFunction<FunctionRandomDistribution<NormalDistribution>>(
+    {
+    R"(
+Returns a random number from the normal distribution.
+Accepts two parameters - mean and variance.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randNormal(0, 5) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+
+    factory.registerFunction<FunctionRandomDistribution<LogNormalDistribution>>(
+    {
+    R"(
+Returns a random number from the lognormal distribution (a distribution of a random variable whose logarithm is normally distributed).
+Accepts two parameters - mean and variance.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randLogNormal(0, 5) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+
+    factory.registerFunction<FunctionRandomDistribution<ExponentialDistribution>>(
+    {
+    R"(
+Returns a random number from the exponential distribution.
+Accepts one parameter.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randExponential(0, 5) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+
+    factory.registerFunction<FunctionRandomDistribution<ChiSquaredDistribution>>(
+    {
+    R"(
+Returns a random number from the chi-squared distribution (a distribution of a sum of the squares of k independent standard normal random variables).
+Accepts one parameter - degree of freedom.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randChiSquared(5) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+    factory.registerFunction<FunctionRandomDistribution<StudentTDistribution>>(
+    {
+    R"(
+Returns a random number from the t-distribution.
+Accepts one parameter - degree of freedom.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randStudentT(5) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+
+    factory.registerFunction<FunctionRandomDistribution<FisherFDistribution>>(
+    {
+    R"(
+Returns a random number from the f-distribution.
+The F-distribution is the distribution of X = (S1 / d1) / (S2 / d2) where d1 and d2 are degrees of freedom.
+Accepts two parameters - degrees of freedom.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randFisherF(5) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+
+    factory.registerFunction<FunctionRandomDistribution<BernoulliDistribution>>(
+    {
+    R"(
+Returns a random number from the Bernoulli distribution.
+Accepts two parameters - probability of success.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randBernoulli(0.1) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+
+    factory.registerFunction<FunctionRandomDistribution<BinomialDistribution>>(
+    {
+    R"(
+Returns a random number from the binomial distribution.
+Accepts two parameters - number of experiments and probability of success in each experiment.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randBinomial(10, 0.1) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+
+    factory.registerFunction<FunctionRandomDistribution<NegativeBinomialDistribution>>(
+    {
+    R"(
+Returns a random number from the negative binomial distribution.
+Accepts two parameters - number of experiments and probability of success in each experiment.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randNegativeBinomial(10, 0.1) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+
+
+    factory.registerFunction<FunctionRandomDistribution<PoissonDistribution>>(
+    {
+    R"(
+Returns a random number from the poisson distribution.
+Accepts two parameters - the mean number of occurrences.
+
+Typical usage:
+[example:typical]
+)",
+    Documentation::Examples{
+        {"typical", "SELECT randPoisson(3) FROM numbers(100000);"}},
+    Documentation::Categories{"Distribution"}
+    });
+}
+
+}

src/Functions/timeSlots.cpp

@@ -19,6 +19,7 @@ namespace ErrorCodes
     extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
     extern const int ILLEGAL_TYPE_OF_ARGUMENT;
     extern const int ILLEGAL_COLUMN;
+    extern const int BAD_ARGUMENTS;
 }
 
 namespace
@@ -41,6 +42,9 @@ struct TimeSlotsImpl
         const PaddedPODArray<UInt32> & starts, const PaddedPODArray<UInt32> & durations, UInt32 time_slot_size,
         PaddedPODArray<UInt32> & result_values, ColumnArray::Offsets & result_offsets)
     {
+        if (time_slot_size == 0)
+            throw Exception("Time slot size cannot be zero", ErrorCodes::BAD_ARGUMENTS);
+
         size_t size = starts.size();
 
         result_offsets.resize(size);
@@ -63,6 +67,9 @@ struct TimeSlotsImpl
         const PaddedPODArray<UInt32> & starts, UInt32 duration, UInt32 time_slot_size,
         PaddedPODArray<UInt32> & result_values, ColumnArray::Offsets & result_offsets)
     {
+        if (time_slot_size == 0)
+            throw Exception("Time slot size cannot be zero", ErrorCodes::BAD_ARGUMENTS);
+
         size_t size = starts.size();
 
         result_offsets.resize(size);
@@ -85,6 +92,9 @@ struct TimeSlotsImpl
         UInt32 start, const PaddedPODArray<UInt32> & durations, UInt32 time_slot_size,
         PaddedPODArray<UInt32> & result_values, ColumnArray::Offsets & result_offsets)
     {
+        if (time_slot_size == 0)
+            throw Exception("Time slot size cannot be zero", ErrorCodes::BAD_ARGUMENTS);
+
         size_t size = durations.size();
 
         result_offsets.resize(size);
@@ -125,6 +135,9 @@ struct TimeSlotsImpl
 
         ColumnArray::Offset current_offset = 0;
         time_slot_size = time_slot_size.value * ts_multiplier;
+        if (time_slot_size == 0)
+            throw Exception("Time slot size cannot be zero", ErrorCodes::BAD_ARGUMENTS);
+
         for (size_t i = 0; i < size; ++i)
         {
             for (DateTime64 value = (starts[i] * dt_multiplier) / time_slot_size, end = (starts[i] * dt_multiplier + durations[i] * dur_multiplier) / time_slot_size; value <= end; value += 1)
@@ -155,6 +168,9 @@ struct TimeSlotsImpl
         ColumnArray::Offset current_offset = 0;
         duration = duration * dur_multiplier;
         time_slot_size = time_slot_size.value * ts_multiplier;
+        if (time_slot_size == 0)
+            throw Exception("Time slot size cannot be zero", ErrorCodes::BAD_ARGUMENTS);
+
         for (size_t i = 0; i < size; ++i)
         {
             for (DateTime64 value = (starts[i] * dt_multiplier) / time_slot_size, end = (starts[i] * dt_multiplier + duration) / time_slot_size; value <= end; value += 1)
@@ -185,6 +201,9 @@ struct TimeSlotsImpl
         ColumnArray::Offset current_offset = 0;
         start = dt_multiplier * start;
         time_slot_size = time_slot_size.value * ts_multiplier;
+        if (time_slot_size == 0)
+            throw Exception("Time slot size cannot be zero", ErrorCodes::BAD_ARGUMENTS);
+
         for (size_t i = 0; i < size; ++i)
         {
             for (DateTime64 value = start / time_slot_size, end = (start + durations[i] * dur_multiplier) / time_slot_size; value <= end; value += 1)

src/Functions/toRelativeDayNum.cpp

@@ -7,7 +7,7 @@
 namespace DB
 {
 
-using FunctionToRelativeDayNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeDayNumImpl>;
+using FunctionToRelativeDayNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeDayNumImpl<ResultPrecision::Standard>>;
 
 REGISTER_FUNCTION(ToRelativeDayNum)
 {

src/Functions/toRelativeHourNum.cpp

@@ -7,7 +7,7 @@
 namespace DB
 {
 
-using FunctionToRelativeHourNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeHourNumImpl>;
+using FunctionToRelativeHourNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeHourNumImpl<ResultPrecision::Standard>>;
 
 REGISTER_FUNCTION(ToRelativeHourNum)
 {

src/Functions/toRelativeMinuteNum.cpp

@@ -7,7 +7,7 @@
 namespace DB
 {
 
-using FunctionToRelativeMinuteNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeMinuteNumImpl>;
+using FunctionToRelativeMinuteNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeMinuteNumImpl<ResultPrecision::Standard>>;
 
 REGISTER_FUNCTION(ToRelativeMinuteNum)
 {

src/Functions/toRelativeMonthNum.cpp

@@ -7,7 +7,7 @@
 namespace DB
 {
 
-using FunctionToRelativeMonthNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeMonthNumImpl>;
+using FunctionToRelativeMonthNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeMonthNumImpl<ResultPrecision::Standard>>;
 
 REGISTER_FUNCTION(ToRelativeMonthNum)
 {

src/Functions/toRelativeQuarterNum.cpp

@@ -7,7 +7,7 @@
 namespace DB
 {
 
-using FunctionToRelativeQuarterNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeQuarterNumImpl>;
+using FunctionToRelativeQuarterNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeQuarterNumImpl<ResultPrecision::Standard>>;
 
 REGISTER_FUNCTION(ToRelativeQuarterNum)
 {

src/Functions/toRelativeSecondNum.cpp

@@ -7,7 +7,7 @@
 namespace DB
 {
 
-using FunctionToRelativeSecondNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeSecondNumImpl>;
+using FunctionToRelativeSecondNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeSecondNumImpl<ResultPrecision::Standard>>;
 
 REGISTER_FUNCTION(ToRelativeSecondNum)
 {

src/Functions/toRelativeWeekNum.cpp

@@ -7,7 +7,7 @@
 namespace DB
 {
 
-using FunctionToRelativeWeekNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeWeekNumImpl>;
+using FunctionToRelativeWeekNum = FunctionDateOrDateTimeToSomething<DataTypeUInt32, ToRelativeWeekNumImpl<ResultPrecision::Standard>>;
 
 REGISTER_FUNCTION(ToRelativeWeekNum)
 {

src/Functions/toRelativeYearNum.cpp

@@ -7,7 +7,7 @@
 namespace DB
 {
 
-using FunctionToRelativeYearNum = FunctionDateOrDateTimeToSomething<DataTypeUInt16, ToRelativeYearNumImpl>;
+using FunctionToRelativeYearNum = FunctionDateOrDateTimeToSomething<DataTypeUInt16, ToRelativeYearNumImpl<ResultPrecision::Standard>>;
 
 REGISTER_FUNCTION(ToRelativeYearNum)
 {

src/IO/S3/PocoHTTPClient.h

@@ -2,20 +2,22 @@
 
 #include "config.h"
 
+#include <string>
+#include <vector>
+
 #if USE_AWS_S3
 
 #include <Common/RemoteHostFilter.h>
 #include <IO/ConnectionTimeouts.h>
 #include <IO/HTTPCommon.h>
 #include <IO/S3/SessionAwareIOStream.h>
-#include <Storages/StorageS3Settings.h>
+#include <Storages/HeaderCollection.h>
 
 #include <aws/core/client/ClientConfiguration.h>
 #include <aws/core/http/HttpClient.h>
 #include <aws/core/http/HttpRequest.h>
 #include <aws/core/http/standard/StandardHttpResponse.h>
 
-
 namespace Aws::Http::Standard
 {
 class StandardHttpResponse;
@@ -23,6 +25,7 @@ class StandardHttpResponse;
 
 namespace DB
 {
+
 class Context;
 }
 

src/IO/S3Common.cpp

@@ -1,9 +1,11 @@
+#include <IO/S3Common.h>
+
+#include <Common/Exception.h>
+#include <Poco/Util/AbstractConfiguration.h>
 #include "config.h"
 
 #if USE_AWS_S3
 
-#    include <IO/S3Common.h>
-
 #    include <Common/quoteString.h>
 
 #    include <IO/WriteBufferFromString.h>
@@ -780,25 +782,16 @@ namespace S3
 
             boost::to_upper(name);
             if (name != S3 && name != COS && name != OBS && name != OSS)
-            {
                 throw Exception(ErrorCodes::BAD_ARGUMENTS, "Object storage system name is unrecognized in virtual hosted style S3 URI: {}", quoteString(name));
-            }
+
             if (name == S3)
-            {
                 storage_name = name;
-            }
             else if (name == OBS)
-            {
                 storage_name = OBS;
-            }
             else if (name == OSS)
-            {
                 storage_name = OSS;
-            }
             else
-            {
                 storage_name = COSN;
-            }
         }
         else if (re2::RE2::PartialMatch(uri.getPath(), path_style_pattern, &bucket, &key))
         {
@@ -851,8 +844,82 @@ namespace S3
     {
         return getObjectInfo(client_ptr, bucket, key, version_id, throw_on_error, for_disk_s3).size;
     }
+
 }
 
 }
 
 #endif
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+    extern const int INVALID_CONFIG_PARAMETER;
+}
+
+namespace S3
+{
+
+AuthSettings AuthSettings::loadFromConfig(const std::string & config_elem, const Poco::Util::AbstractConfiguration & config)
+{
+    auto access_key_id = config.getString(config_elem + ".access_key_id", "");
+    auto secret_access_key = config.getString(config_elem + ".secret_access_key", "");
+    auto region = config.getString(config_elem + ".region", "");
+    auto server_side_encryption_customer_key_base64 = config.getString(config_elem + ".server_side_encryption_customer_key_base64", "");
+
+    std::optional<bool> use_environment_credentials;
+    if (config.has(config_elem + ".use_environment_credentials"))
+        use_environment_credentials = config.getBool(config_elem + ".use_environment_credentials");
+
+    std::optional<bool> use_insecure_imds_request;
+    if (config.has(config_elem + ".use_insecure_imds_request"))
+        use_insecure_imds_request = config.getBool(config_elem + ".use_insecure_imds_request");
+
+    HeaderCollection headers;
+    Poco::Util::AbstractConfiguration::Keys subconfig_keys;
+    config.keys(config_elem, subconfig_keys);
+    for (const std::string & subkey : subconfig_keys)
+    {
+        if (subkey.starts_with("header"))
+        {
+            auto header_str = config.getString(config_elem + "." + subkey);
+            auto delimiter = header_str.find(':');
+            if (delimiter == std::string::npos)
+                throw Exception("Malformed s3 header value", ErrorCodes::INVALID_CONFIG_PARAMETER);
+            headers.emplace_back(HttpHeader{header_str.substr(0, delimiter), header_str.substr(delimiter + 1, String::npos)});
+        }
+    }
+
+    return AuthSettings
+    {
+        std::move(access_key_id), std::move(secret_access_key),
+        std::move(region),
+        std::move(server_side_encryption_customer_key_base64),
+        std::move(headers),
+        use_environment_credentials,
+        use_insecure_imds_request
+    };
+}
+
+
+void AuthSettings::updateFrom(const AuthSettings & from)
+{
+    /// Update with check for emptyness only parameters which
+    /// can be passed not only from config, but via ast.
+
+    if (!from.access_key_id.empty())
+        access_key_id = from.access_key_id;
+    if (!from.secret_access_key.empty())
+        secret_access_key = from.secret_access_key;
+
+    headers = from.headers;
+    region = from.region;
+    server_side_encryption_customer_key_base64 = from.server_side_encryption_customer_key_base64;
+    use_environment_credentials = from.use_environment_credentials;
+    use_insecure_imds_request = from.use_insecure_imds_request;
+}
+
+}
+}

src/IO/S3Common.h

@@ -1,5 +1,11 @@
 #pragma once
 
+#include <Storages/HeaderCollection.h>
+#include <IO/S3/PocoHTTPClient.h>
+
+#include <string>
+#include <optional>
+
 #include "config.h"
 
 #if USE_AWS_S3
@@ -8,7 +14,6 @@
 #include <aws/core/Aws.h>
 #include <aws/core/client/ClientConfiguration.h>
 #include <aws/s3/S3Errors.h>
-#include <IO/S3/PocoHTTPClient.h>
 #include <Poco/URI.h>
 
 #include <Common/Exception.h>
@@ -27,8 +32,6 @@ namespace ErrorCodes
 }
 
 class RemoteHostFilter;
-struct HttpHeader;
-using HeaderCollection = std::vector<HttpHeader>;
 
 class S3Exception : public Exception
 {
@@ -130,5 +133,33 @@ S3::ObjectInfo getObjectInfo(std::shared_ptr<const Aws::S3::S3Client> client_ptr
 size_t getObjectSize(std::shared_ptr<const Aws::S3::S3Client> client_ptr, const String & bucket, const String & key, const String & version_id, bool throw_on_error, bool for_disk_s3);
 
 }
-
 #endif
+
+namespace Poco::Util
+{
+class AbstractConfiguration;
+};
+
+namespace DB::S3
+{
+
+struct AuthSettings
+{
+    static AuthSettings loadFromConfig(const std::string & config_elem, const Poco::Util::AbstractConfiguration & config);
+
+    std::string access_key_id;
+    std::string secret_access_key;
+    std::string region;
+    std::string server_side_encryption_customer_key_base64;
+
+    HeaderCollection headers;
+
+    std::optional<bool> use_environment_credentials;
+    std::optional<bool> use_insecure_imds_request;
+
+    bool operator==(const AuthSettings & other) const = default;
+
+    void updateFrom(const AuthSettings & from);
+};
+
+}

src/Interpreters/Cache/FileCache.cpp

@@ -32,6 +32,8 @@ FileCache::FileCache(
     , allow_persistent_files(cache_settings_.do_not_evict_index_and_mark_files)
     , enable_cache_hits_threshold(cache_settings_.enable_cache_hits_threshold)
     , enable_filesystem_query_cache_limit(cache_settings_.enable_filesystem_query_cache_limit)
+    , enable_bypass_cache_with_threashold(cache_settings_.enable_bypass_cache_with_threashold)
+    , bypass_cache_threashold(cache_settings_.bypass_cache_threashold)
     , log(&Poco::Logger::get("FileCache"))
     , main_priority(std::make_unique<LRUFileCachePriority>())
     , stash_priority(std::make_unique<LRUFileCachePriority>())
@@ -185,6 +187,20 @@ FileSegments FileCache::getImpl(
     /// Given range = [left, right] and non-overlapping ordered set of file segments,
     /// find list [segment1, ..., segmentN] of segments which intersect with given range.
 
+    FileSegments result;
+
+    if (enable_bypass_cache_with_threashold && (range.size() > bypass_cache_threashold))
+    {
+        auto file_segment = std::make_shared<FileSegment>(
+            range.left, range.size(), key, this, FileSegment::State::SKIP_CACHE, CreateFileSegmentSettings{});
+        {
+            std::unique_lock segment_lock(file_segment->mutex);
+            file_segment->detachAssumeStateFinalized(segment_lock);
+        }
+        result.emplace_back(file_segment);
+        return result;
+    }
+
     auto it = files.find(key);
     if (it == files.end())
         return {};
@@ -197,7 +213,6 @@ FileSegments FileCache::getImpl(
         return {};
     }
 
-    FileSegments result;
     auto segment_it = file_segments.lower_bound(range.left);
     if (segment_it == file_segments.end())
     {
@@ -392,7 +407,6 @@ FileSegmentsHolder FileCache::getOrSet(const Key & key, size_t offset, size_t si
 #endif
 
     FileSegment::Range range(offset, offset + size - 1);
-
     /// Get all segments which intersect with the given range.
     auto file_segments = getImpl(key, range, cache_lock);
 
@@ -404,7 +418,6 @@ FileSegmentsHolder FileCache::getOrSet(const Key & key, size_t offset, size_t si
     {
         fillHolesWithEmptyFileSegments(file_segments, key, range, /* fill_with_detached */false, settings, cache_lock);
     }
-
     assert(!file_segments.empty());
     return FileSegmentsHolder(std::move(file_segments));
 }

src/Interpreters/Cache/FileCache.h

@@ -140,6 +140,9 @@ private:
     const size_t enable_cache_hits_threshold;
     const bool enable_filesystem_query_cache_limit;
 
+    const bool enable_bypass_cache_with_threashold;
+    const size_t bypass_cache_threashold;
+
     mutable std::mutex mutex;
     Poco::Logger * log;
 

src/Interpreters/Cache/FileCacheSettings.cpp

@@ -35,6 +35,13 @@ void FileCacheSettings::loadFromConfig(const Poco::Util::AbstractConfiguration &
     enable_filesystem_query_cache_limit = config.getUInt64(config_prefix + ".enable_filesystem_query_cache_limit", false);
     enable_cache_hits_threshold = config.getUInt64(config_prefix + ".enable_cache_hits_threshold", REMOTE_FS_OBJECTS_CACHE_ENABLE_HITS_THRESHOLD);
 
+    enable_bypass_cache_with_threashold = config.getUInt64(config_prefix + ".enable_bypass_cache_with_threashold", false);
+
+    if (config.has(config_prefix + ".bypass_cache_threashold"))
+        bypass_cache_threashold = parseWithSizeSuffix<uint64_t>(config.getString(config_prefix + ".bypass_cache_threashold"));
+    else
+         bypass_cache_threashold = REMOTE_FS_OBJECTS_CACHE_BYPASS_THRESHOLD;
+
     do_not_evict_index_and_mark_files = config.getUInt64(config_prefix + ".do_not_evict_index_and_mark_files", false);
 }
 

src/Interpreters/Cache/FileCacheSettings.h

@@ -20,6 +20,9 @@ struct FileCacheSettings
 
     bool do_not_evict_index_and_mark_files = true;
 
+    bool enable_bypass_cache_with_threashold = false;
+    size_t bypass_cache_threashold = REMOTE_FS_OBJECTS_CACHE_BYPASS_THRESHOLD;
+
     void loadFromConfig(const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix);
 };
 

src/Interpreters/Cache/FileCache_fwd.h

@@ -7,6 +7,7 @@ namespace DB
 static constexpr int REMOTE_FS_OBJECTS_CACHE_DEFAULT_MAX_FILE_SEGMENT_SIZE = 100 * 1024 * 1024;
 static constexpr int REMOTE_FS_OBJECTS_CACHE_DEFAULT_MAX_ELEMENTS = 1024 * 1024;
 static constexpr int REMOTE_FS_OBJECTS_CACHE_ENABLE_HITS_THRESHOLD = 0;
+static constexpr size_t REMOTE_FS_OBJECTS_CACHE_BYPASS_THRESHOLD = 256 * 1024 * 1024;;
 
 class FileCache;
 using FileCachePtr = std::shared_ptr<FileCache>;

src/Interpreters/ClusterProxy/executeQuery.cpp

@@ -7,6 +7,7 @@
 #include <Interpreters/ProcessList.h>
 #include <Interpreters/OptimizeShardingKeyRewriteInVisitor.h>
 #include <QueryPipeline/Pipe.h>
+#include <Parsers/queryToString.h>
 #include <Processors/QueryPlan/QueryPlan.h>
 #include <Processors/QueryPlan/ReadFromRemote.h>
 #include <Processors/QueryPlan/UnionStep.h>
@@ -26,7 +27,7 @@ namespace ErrorCodes
 namespace ClusterProxy
 {
 
-ContextMutablePtr updateSettingsForCluster(const Cluster & cluster, ContextPtr context, const Settings & settings, Poco::Logger * log)
+ContextMutablePtr updateSettingsForCluster(const Cluster & cluster, ContextPtr context, const Settings & settings, const StorageID & main_table, const SelectQueryInfo * query_info, Poco::Logger * log)
 {
     Settings new_settings = settings;
     new_settings.queue_max_wait_ms = Cluster::saturate(new_settings.queue_max_wait_ms, settings.max_execution_time);
@@ -96,6 +97,20 @@ ContextMutablePtr updateSettingsForCluster(const Cluster & cluster, ContextPtr c
         new_settings.limit.changed = false;
     }
 
+    /// Setting additional_table_filters may be applied to Distributed table.
+    /// In case if query is executed up to WithMergableState on remote shard, it is impossible to filter on initiator.
+    /// We need to propagate the setting, but change the table name from distributed to source.
+    ///
+    /// Here we don't try to analyze setting again. In case if query_info->additional_filter_ast is not empty, some filter was applied.
+    /// It's just easier to add this filter for a source table.
+    if (query_info && query_info->additional_filter_ast)
+    {
+        Tuple tuple;
+        tuple.push_back(main_table.getShortName());
+        tuple.push_back(queryToString(query_info->additional_filter_ast));
+        new_settings.additional_table_filters.value.push_back(std::move(tuple));
+    }
+
     auto new_context = Context::createCopy(context);
     new_context->setSettings(new_settings);
     return new_context;
@@ -121,12 +136,12 @@ void executeQuery(
     std::vector<QueryPlanPtr> plans;
     SelectStreamFactory::Shards remote_shards;
 
-    auto new_context = updateSettingsForCluster(*query_info.getCluster(), context, settings, log);
+    auto new_context = updateSettingsForCluster(*query_info.getCluster(), context, settings, main_table, &query_info, log);
 
     new_context->getClientInfo().distributed_depth += 1;
 
     ThrottlerPtr user_level_throttler;
-    if (auto * process_list_element = context->getProcessListElement())
+    if (auto process_list_element = context->getProcessListElement())
         user_level_throttler = process_list_element->getUserNetworkThrottler();
 
     /// Network bandwidth limit, if needed.
@@ -228,7 +243,7 @@ void executeQueryWithParallelReplicas(
     const Settings & settings = context->getSettingsRef();
 
     ThrottlerPtr user_level_throttler;
-    if (auto * process_list_element = context->getProcessListElement())
+    if (auto process_list_element = context->getProcessListElement())
         user_level_throttler = process_list_element->getUserNetworkThrottler();
 
     /// Network bandwidth limit, if needed.

src/Interpreters/ClusterProxy/executeQuery.h

@@ -35,7 +35,7 @@ class SelectStreamFactory;
 ///
 /// @return new Context with adjusted settings
 ContextMutablePtr updateSettingsForCluster(
-    const Cluster & cluster, ContextPtr context, const Settings & settings, Poco::Logger * log = nullptr);
+    const Cluster & cluster, ContextPtr context, const Settings & settings, const StorageID & main_table, const SelectQueryInfo * query_info = nullptr, Poco::Logger * log = nullptr);
 
 /// Execute a distributed query, creating a query plan, from which the query pipeline can be built.
 /// `stream_factory` object encapsulates the logic of creating plans for a different type of query

src/Interpreters/Context.cpp

@@ -1463,10 +1463,8 @@ void Context::setCurrentQueryId(const String & query_id)
 
 void Context::killCurrentQuery()
 {
-    if (process_list_elem)
-    {
-        process_list_elem->cancelQuery(true);
-    }
+    if (auto elem = process_list_elem.lock())
+        elem->cancelQuery(true);
 }
 
 String Context::getDefaultFormat() const
@@ -1707,15 +1705,15 @@ ProgressCallback Context::getProgressCallback() const
 }
 
 
-void Context::setProcessListElement(ProcessList::Element * elem)
+void Context::setProcessListElement(QueryStatusPtr elem)
 {
     /// Set to a session or query. In the session, only one query is processed at a time. Therefore, the lock is not needed.
     process_list_elem = elem;
 }
 
-ProcessList::Element * Context::getProcessListElement() const
+QueryStatusPtr Context::getProcessListElement() const
 {
-    return process_list_elem;
+    return process_list_elem.lock();
 }
 
 

src/Interpreters/Context.h

@@ -68,6 +68,7 @@ class MMappedFileCache;
 class UncompressedCache;
 class ProcessList;
 class QueryStatus;
+using QueryStatusPtr = std::shared_ptr<QueryStatus>;
 class Macros;
 struct Progress;
 struct FileProgress;
@@ -230,7 +231,7 @@ private:
     using FileProgressCallback = std::function<void(const FileProgress & progress)>;
     FileProgressCallback file_progress_callback; /// Callback for tracking progress of file loading.
 
-    QueryStatus * process_list_elem = nullptr;  /// For tracking total resource usage for query.
+    std::weak_ptr<QueryStatus> process_list_elem;  /// For tracking total resource usage for query.
     StorageID insertion_table = StorageID::createEmpty();  /// Saved insertion table in query context
     bool is_distributed = false;  /// Whether the current context it used for distributed query
 
@@ -750,9 +751,9 @@ public:
     /** Set in executeQuery and InterpreterSelectQuery. Then it is used in QueryPipeline,
       *  to update and monitor information about the total number of resources spent for the query.
       */
-    void setProcessListElement(QueryStatus * elem);
+    void setProcessListElement(QueryStatusPtr elem);
     /// Can return nullptr if the query was not inserted into the ProcessList.
-    QueryStatus * getProcessListElement() const;
+    QueryStatusPtr getProcessListElement() const;
 
     /// List all queries.
     ProcessList & getProcessList();

src/Interpreters/DDLWorker.cpp

@@ -114,7 +114,7 @@ DDLWorker::DDLWorker(
 void DDLWorker::startup()
 {
     [[maybe_unused]] bool prev_stop_flag = stop_flag.exchange(false);
-    chassert(true);
+    chassert(prev_stop_flag);
     main_thread = ThreadFromGlobalPool(&DDLWorker::runMainThread, this);
     cleanup_thread = ThreadFromGlobalPool(&DDLWorker::runCleanupThread, this);
 }

src/Interpreters/ProcessList.cpp

@@ -243,15 +243,15 @@ ProcessList::EntryPtr ProcessList::insert(const String & query_, const IAST * as
         }
 
         auto process_it = processes.emplace(processes.end(),
-            query_context, query_, client_info, priorities.insert(settings.priority), std::move(thread_group), query_kind);
+            std::make_shared<QueryStatus>(query_context, query_, client_info, priorities.insert(settings.priority), std::move(thread_group), query_kind));
 
         increaseQueryKindAmount(query_kind);
 
         res = std::make_shared<Entry>(*this, process_it);
 
-        process_it->setUserProcessList(&user_process_list);
+        (*process_it)->setUserProcessList(&user_process_list);
 
-        user_process_list.queries.emplace(client_info.current_query_id, &res->get());
+        user_process_list.queries.emplace(client_info.current_query_id, res->getQueryStatus());
 
         /// Track memory usage for all simultaneously running queries from single user.
         user_process_list.user_memory_tracker.setOrRaiseHardLimit(settings.max_memory_usage_for_user);
@@ -280,11 +280,11 @@ ProcessListEntry::~ProcessListEntry()
 {
     auto lock = parent.safeLock();
 
-    String user = it->getClientInfo().current_user;
-    String query_id = it->getClientInfo().current_query_id;
-    IAST::QueryKind query_kind = it->query_kind;
+    String user = (*it)->getClientInfo().current_user;
+    String query_id = (*it)->getClientInfo().current_query_id;
+    IAST::QueryKind query_kind = (*it)->query_kind;
 
-    const QueryStatus * process_list_element_ptr = &*it;
+    const QueryStatusPtr process_list_element_ptr = *it;
 
     auto user_process_list_it = parent.user_to_queries.find(user);
     if (user_process_list_it == parent.user_to_queries.end())
@@ -307,7 +307,7 @@ ProcessListEntry::~ProcessListEntry()
     }
 
     /// Wait for the query if it is in the cancellation right now.
-    parent.cancelled_cv.wait(lock.lock, [&]() { return it->is_cancelling == false; });
+    parent.cancelled_cv.wait(lock.lock, [&]() { return process_list_element_ptr->is_cancelling == false; });
 
     /// This removes the memory_tracker of one request.
     parent.processes.erase(it);
@@ -344,6 +344,7 @@ QueryStatus::QueryStatus(
     , client_info(client_info_)
     , thread_group(std::move(thread_group_))
     , priority_handle(std::move(priority_handle_))
+    , global_overcommit_tracker(context_->getGlobalOvercommitTracker())
     , query_kind(query_kind_)
     , num_queries_increment(CurrentMetrics::Query)
 {
@@ -360,8 +361,8 @@ QueryStatus::~QueryStatus()
     {
         if (user_process_list)
             user_process_list->user_overcommit_tracker.onQueryStop(memory_tracker);
-        if (auto shared_context = getContext())
-            shared_context->getGlobalOvercommitTracker()->onQueryStop(memory_tracker);
+        if (global_overcommit_tracker)
+            global_overcommit_tracker->onQueryStop(memory_tracker);
     }
 }
 
@@ -430,7 +431,7 @@ ThrottlerPtr QueryStatus::getUserNetworkThrottler()
 }
 
 
-QueryStatus * ProcessList::tryGetProcessListElement(const String & current_query_id, const String & current_user)
+QueryStatusPtr ProcessList::tryGetProcessListElement(const String & current_query_id, const String & current_user)
 {
     auto user_it = user_to_queries.find(current_user);
     if (user_it != user_to_queries.end())
@@ -442,13 +443,13 @@ QueryStatus * ProcessList::tryGetProcessListElement(const String & current_query
             return query_it->second;
     }
 
-    return nullptr;
+    return {};
 }
 
 
 CancellationCode ProcessList::sendCancelToQuery(const String & current_query_id, const String & current_user, bool kill)
 {
-    QueryStatus * elem;
+    QueryStatusPtr elem;
 
     /// Cancelling the query should be done without the lock.
     ///
@@ -484,7 +485,7 @@ CancellationCode ProcessList::sendCancelToQuery(const String & current_query_id,
 
 void ProcessList::killAllQueries()
 {
-    std::vector<QueryStatus *> cancelled_processes;
+    std::vector<QueryStatusPtr> cancelled_processes;
 
     SCOPE_EXIT({
         auto lock = safeLock();
@@ -498,8 +499,8 @@ void ProcessList::killAllQueries()
         cancelled_processes.reserve(processes.size());
         for (auto & process : processes)
         {
-            cancelled_processes.push_back(&process);
-            process.is_cancelling = true;
+            cancelled_processes.push_back(process);
+            process->is_cancelling = true;
         }
     }
 
@@ -558,7 +559,7 @@ ProcessList::Info ProcessList::getInfo(bool get_thread_list, bool get_profile_ev
 
     per_query_infos.reserve(processes.size());
     for (const auto & process : processes)
-        per_query_infos.emplace_back(process.getInfo(get_thread_list, get_profile_events, get_settings));
+        per_query_infos.emplace_back(process->getInfo(get_thread_list, get_profile_events, get_settings));
 
     return per_query_infos;
 }

src/Interpreters/ProcessList.h

@@ -133,6 +133,8 @@ protected:
 
     ProcessListForUser * user_process_list = nullptr;
 
+    OvercommitTracker * global_overcommit_tracker = nullptr;
+
     IAST::QueryKind query_kind;
 
     /// This field is unused in this class, but it
@@ -221,6 +223,8 @@ public:
     [[nodiscard]] bool checkTimeLimitSoft();
 };
 
+using QueryStatusPtr = std::shared_ptr<QueryStatus>;
+
 
 /// Information of process list for user.
 struct ProcessListForUserInfo
@@ -241,7 +245,7 @@ struct ProcessListForUser
     ProcessListForUser(ContextPtr global_context, ProcessList * global_process_list);
 
     /// query_id -> ProcessListElement(s). There can be multiple queries with the same query_id as long as all queries except one are cancelled.
-    using QueryToElement = std::unordered_map<String, QueryStatus *>;
+    using QueryToElement = std::unordered_map<String, QueryStatusPtr>;
     QueryToElement queries;
 
     ProfileEvents::Counters user_performance_counters{VariableContext::User, &ProfileEvents::global_counters};
@@ -278,7 +282,7 @@ class ProcessList;
 class ProcessListEntry
 {
 private:
-    using Container = std::list<QueryStatus>;
+    using Container = std::list<QueryStatusPtr>;
 
     ProcessList & parent;
     Container::iterator it;
@@ -289,11 +293,8 @@ public:
 
     ~ProcessListEntry();
 
-    QueryStatus * operator->() { return &*it; }
-    const QueryStatus * operator->() const { return &*it; }
-
-    QueryStatus & get() { return *it; }
-    const QueryStatus & get() const { return *it; }
+    QueryStatusPtr getQueryStatus() { return *it; }
+    const QueryStatusPtr getQueryStatus() const { return *it; }
 };
 
 
@@ -319,7 +320,7 @@ protected:
 class ProcessList : public ProcessListBase
 {
 public:
-    using Element = QueryStatus;
+    using Element = QueryStatusPtr;
     using Entry = ProcessListEntry;
     using QueryAmount = UInt64;
 
@@ -358,7 +359,7 @@ protected:
     ThrottlerPtr total_network_throttler;
 
     /// Call under lock. Finds process with specified current_user and current_query_id.
-    QueryStatus * tryGetProcessListElement(const String & current_query_id, const String & current_user);
+    QueryStatusPtr tryGetProcessListElement(const String & current_query_id, const String & current_user);
 
     /// limit for insert. 0 means no limit. Otherwise, when limit exceeded, an exception is thrown.
     size_t max_insert_queries_amount = 0;

src/Interpreters/executeQuery.cpp

@@ -537,7 +537,7 @@ static std::tuple<ASTPtr, BlockIO> executeQueryImpl(
         {
             /// processlist also has query masked now, to avoid secrets leaks though SHOW PROCESSLIST by other users.
             process_list_entry = context->getProcessList().insert(query_for_logging, ast.get(), context);
-            context->setProcessListElement(&process_list_entry->get());
+            context->setProcessListElement(process_list_entry->getQueryStatus());
         }
 
         /// Load external tables if they were provided
@@ -713,9 +713,9 @@ static std::tuple<ASTPtr, BlockIO> executeQueryImpl(
         if (process_list_entry)
         {
             /// Query was killed before execution
-            if ((*process_list_entry)->isKilled())
-                throw Exception("Query '" + (*process_list_entry)->getInfo().client_info.current_query_id + "' is killed in pending state",
-                    ErrorCodes::QUERY_WAS_CANCELLED);
+            if (process_list_entry->getQueryStatus()->isKilled())
+                throw Exception(ErrorCodes::QUERY_WAS_CANCELLED,
+                    "Query '{}' is killed in pending state", process_list_entry->getQueryStatus()->getInfo().client_info.current_query_id);
         }
 
         /// Hold element of process list till end of query execution.
@@ -859,7 +859,7 @@ static std::tuple<ASTPtr, BlockIO> executeQueryImpl(
                                     pulling_pipeline = pipeline.pulling(),
                                     query_span](QueryPipeline & query_pipeline) mutable
             {
-                QueryStatus * process_list_elem = context->getProcessListElement();
+                QueryStatusPtr process_list_elem = context->getProcessListElement();
 
                 if (process_list_elem)
                 {
@@ -1025,7 +1025,7 @@ static std::tuple<ASTPtr, BlockIO> executeQueryImpl(
                 elem.exception_code = getCurrentExceptionCode();
                 elem.exception = getCurrentExceptionMessage(false);
 
-                QueryStatus * process_list_elem = context->getProcessListElement();
+                QueryStatusPtr process_list_elem = context->getProcessListElement();
                 const Settings & current_settings = context->getSettingsRef();
 
                 /// Update performance counters before logging to query_log

src/Processors/Executors/CompletedPipelineExecutor.cpp

@@ -72,9 +72,9 @@ void CompletedPipelineExecutor::execute()
         data->executor = std::make_shared<PipelineExecutor>(pipeline.processors, pipeline.process_list_element);
         data->executor->setReadProgressCallback(pipeline.getReadProgressCallback());
 
-        /// Avoid passing this to labmda, copy ptr to data instead.
+        /// Avoid passing this to lambda, copy ptr to data instead.
         /// Destructor of unique_ptr copy raw ptr into local variable first, only then calls object destructor.
-        auto func = [data_ptr = data.get(), num_threads = pipeline.getNumThreads(), thread_group = CurrentThread::getGroup()]()
+        auto func = [data_ptr = data.get(), num_threads = pipeline.getNumThreads(), thread_group = CurrentThread::getGroup()]
         {
             threadFunction(*data_ptr, thread_group, num_threads);
         };

src/Processors/Executors/CompletedPipelineExecutor.h

@@ -1,7 +1,9 @@
 #pragma once
+
 #include <functional>
 #include <memory>
 
+
 namespace DB
 {
 

src/Processors/Executors/ExecutingGraph.cpp

@@ -10,17 +10,17 @@ namespace ErrorCodes
     extern const int LOGICAL_ERROR;
 }
 
-ExecutingGraph::ExecutingGraph(Processors & processors_, bool profile_processors_)
-    : processors(processors_)
+ExecutingGraph::ExecutingGraph(std::shared_ptr<Processors> processors_, bool profile_processors_)
+    : processors(std::move(processors_))
     , profile_processors(profile_processors_)
 {
-    uint64_t num_processors = processors.size();
+    uint64_t num_processors = processors->size();
     nodes.reserve(num_processors);
 
     /// Create nodes.
     for (uint64_t node = 0; node < num_processors; ++node)
     {
-        IProcessor * proc = processors[node].get();
+        IProcessor * proc = processors->at(node).get();
         processors_map[proc] = node;
         nodes.emplace_back(std::make_unique<Node>(proc, node));
     }
@@ -109,10 +109,10 @@ bool ExecutingGraph::expandPipeline(std::stack<uint64_t> & stack, uint64_t pid)
 
     {
         std::lock_guard guard(processors_mutex);
-        processors.insert(processors.end(), new_processors.begin(), new_processors.end());
+        processors->insert(processors->end(), new_processors.begin(), new_processors.end());
     }
 
-    uint64_t num_processors = processors.size();
+    uint64_t num_processors = processors->size();
     std::vector<uint64_t> back_edges_sizes(num_processors, 0);
     std::vector<uint64_t> direct_edge_sizes(num_processors, 0);
 
@@ -126,7 +126,7 @@ bool ExecutingGraph::expandPipeline(std::stack<uint64_t> & stack, uint64_t pid)
 
     while (nodes.size() < num_processors)
     {
-        auto * processor = processors[nodes.size()].get();
+        auto * processor = processors->at(nodes.size()).get();
         if (processors_map.contains(processor))
             throw Exception(ErrorCodes::LOGICAL_ERROR, "Processor {} was already added to pipeline", processor->getName());
 
@@ -386,7 +386,7 @@ bool ExecutingGraph::updateNode(uint64_t pid, Queue & queue, Queue & async_queue
 void ExecutingGraph::cancel()
 {
     std::lock_guard guard(processors_mutex);
-    for (auto & processor : processors)
+    for (auto & processor : *processors)
         processor->cancel();
 }
 

src/Processors/Executors/ExecutingGraph.h

@@ -1,4 +1,5 @@
 #pragma once
+
 #include <Processors/Port.h>
 #include <Processors/IProcessor.h>
 #include <Processors/Executors/UpgradableLock.h>
@@ -6,6 +7,7 @@
 #include <queue>
 #include <stack>
 
+
 namespace DB
 {
 
@@ -123,9 +125,9 @@ public:
     using ProcessorsMap = std::unordered_map<const IProcessor *, uint64_t>;
     ProcessorsMap processors_map;
 
-    explicit ExecutingGraph(Processors & processors_, bool profile_processors_);
+    explicit ExecutingGraph(std::shared_ptr<Processors> processors_, bool profile_processors_);
 
-    const Processors & getProcessors() const { return processors; }
+    const Processors & getProcessors() const { return *processors; }
 
     /// Traverse graph the first time to update all the childless nodes.
     void initializeExecution(Queue & queue);
@@ -149,7 +151,7 @@ private:
     /// All new nodes and nodes with updated ports are pushed into stack.
     bool expandPipeline(std::stack<uint64_t> & stack, uint64_t pid);
 
-    Processors & processors;
+    std::shared_ptr<Processors> processors;
     std::mutex processors_mutex;
 
     UpgradableMutex nodes_mutex;

src/Processors/Executors/PipelineExecutor.cpp

@@ -15,6 +15,7 @@
     #include <Common/Stopwatch.h>
 #endif
 
+
 namespace DB
 {
 
@@ -24,8 +25,8 @@ namespace ErrorCodes
 }
 
 
-PipelineExecutor::PipelineExecutor(Processors & processors, QueryStatus * elem)
-    : process_list_element(elem)
+PipelineExecutor::PipelineExecutor(std::shared_ptr<Processors> & processors, QueryStatusPtr elem)
+    : process_list_element(std::move(elem))
 {
     if (process_list_element)
     {
@@ -41,7 +42,7 @@ PipelineExecutor::PipelineExecutor(Processors & processors, QueryStatus * elem)
         /// If exception was thrown while pipeline initialization, it means that query pipeline was not build correctly.
         /// It is logical error, and we need more information about pipeline.
         WriteBufferFromOwnString buf;
-        printPipeline(processors, buf);
+        printPipeline(*processors, buf);
         buf.finalize();
         exception.addMessage("Query pipeline:\n" + buf.str());
 

src/Processors/Executors/PipelineExecutor.h

@@ -10,16 +10,19 @@
 #include <queue>
 #include <mutex>
 
+
 namespace DB
 {
 
 class QueryStatus;
+using QueryStatusPtr = std::shared_ptr<QueryStatus>;
 class ExecutingGraph;
 using ExecutingGraphPtr = std::unique_ptr<ExecutingGraph>;
 
 class ReadProgressCallback;
 using ReadProgressCallbackPtr = std::unique_ptr<ReadProgressCallback>;
 
+
 /// Executes query pipeline.
 class PipelineExecutor
 {
@@ -30,7 +33,7 @@ public:
     /// During pipeline execution new processors can appear. They will be added to existing set.
     ///
     /// Explicit graph representation is built in constructor. Throws if graph is not correct.
-    explicit PipelineExecutor(Processors & processors, QueryStatus * elem);
+    explicit PipelineExecutor(std::shared_ptr<Processors> & processors, QueryStatusPtr elem);
     ~PipelineExecutor();
 
     /// Execute pipeline in multiple threads. Must be called once.
@@ -79,7 +82,7 @@ private:
     Poco::Logger * log = &Poco::Logger::get("PipelineExecutor");
 
     /// Now it's used to check if query was killed.
-    QueryStatus * const process_list_element = nullptr;
+    QueryStatusPtr process_list_element;
 
     ReadProgressCallbackPtr read_progress_callback;
 

src/Processors/Executors/PushingAsyncPipelineExecutor.cpp

@@ -129,7 +129,7 @@ PushingAsyncPipelineExecutor::PushingAsyncPipelineExecutor(QueryPipeline & pipel
 
     pushing_source = std::make_shared<PushingAsyncSource>(pipeline.input->getHeader());
     connect(pushing_source->getPort(), *pipeline.input);
-    pipeline.processors.emplace_back(pushing_source);
+    pipeline.processors->emplace_back(pushing_source);
 }
 
 PushingAsyncPipelineExecutor::~PushingAsyncPipelineExecutor()

src/Processors/Executors/PushingPipelineExecutor.cpp

@@ -58,7 +58,7 @@ PushingPipelineExecutor::PushingPipelineExecutor(QueryPipeline & pipeline_) : pi
 
     pushing_source = std::make_shared<PushingSource>(pipeline.input->getHeader(), input_wait_flag);
     connect(pushing_source->getPort(), *pipeline.input);
-    pipeline.processors.emplace_back(pushing_source);
+    pipeline.processors->emplace_back(pushing_source);
 }
 
 PushingPipelineExecutor::~PushingPipelineExecutor()

src/Processors/Formats/Impl/CustomSeparatedRowInputFormat.cpp

@@ -67,6 +67,19 @@ CustomSeparatedRowInputFormat::CustomSeparatedRowInputFormat(
     }
 }
 
+void CustomSeparatedRowInputFormat::readPrefix()
+{
+    RowInputFormatWithNamesAndTypes::readPrefix();
+
+    /// Provide better error message for unsupported delimiters
+    for (const auto & column_index : column_mapping->column_indexes_for_input_fields)
+    {
+        if (column_index)
+            checkSupportedDelimiterAfterField(format_settings.custom.escaping_rule, format_settings.custom.field_delimiter, data_types[*column_index]);
+        else
+            checkSupportedDelimiterAfterField(format_settings.custom.escaping_rule, format_settings.custom.field_delimiter, nullptr);
+    }
+}
 
 bool CustomSeparatedRowInputFormat::allowSyncAfterError() const
 {

src/Processors/Formats/Impl/CustomSeparatedRowInputFormat.h

@@ -30,6 +30,7 @@ private:
 
     bool allowSyncAfterError() const override;
     void syncAfterError() override;
+    void readPrefix() override;
 
     std::unique_ptr<PeekableReadBuffer> buf;
     bool ignore_spaces;

src/Processors/Formats/Impl/MySQLOutputFormat.cpp

@@ -74,7 +74,7 @@ void MySQLOutputFormat::finalizeImpl()
 {
     size_t affected_rows = 0;
     std::string human_readable_info;
-    if (QueryStatus * process_list_elem = getContext()->getProcessListElement())
+    if (QueryStatusPtr process_list_elem = getContext()->getProcessListElement())
     {
         CurrentThread::finalizePerformanceCounters();
         QueryStatusInfo info = process_list_elem->getInfo();

src/Processors/Formats/Impl/TemplateRowInputFormat.cpp

@@ -53,18 +53,25 @@ TemplateRowInputFormat::TemplateRowInputFormat(const Block & header_, std::uniqu
     std::vector<UInt8> column_in_format(header_.columns(), false);
     for (size_t i = 0; i < row_format.columnsCount(); ++i)
     {
-        if (row_format.format_idx_to_column_idx[i])
+        const auto & column_index = row_format.format_idx_to_column_idx[i];
+        if (column_index)
         {
-            if (header_.columns() <= *row_format.format_idx_to_column_idx[i])
-                row_format.throwInvalidFormat("Column index " + std::to_string(*row_format.format_idx_to_column_idx[i]) +
+            if (header_.columns() <= *column_index)
+                row_format.throwInvalidFormat("Column index " + std::to_string(*column_index) +
                                               " must be less then number of columns (" + std::to_string(header_.columns()) + ")", i);
             if (row_format.escaping_rules[i] == EscapingRule::None)
                 row_format.throwInvalidFormat("Column is not skipped, but deserialization type is None", i);
 
-            size_t col_idx = *row_format.format_idx_to_column_idx[i];
+            size_t col_idx = *column_index;
             if (column_in_format[col_idx])
                 row_format.throwInvalidFormat("Duplicate column", i);
             column_in_format[col_idx] = true;
+
+            checkSupportedDelimiterAfterField(row_format.escaping_rules[i], row_format.delimiters[i + 1], data_types[*column_index]);
+        }
+        else
+        {
+            checkSupportedDelimiterAfterField(row_format.escaping_rules[i], row_format.delimiters[i + 1], nullptr);
         }
     }
 

src/Processors/Formats/RowInputFormatWithNamesAndTypes.h

@@ -41,6 +41,7 @@ protected:
     void resetParser() override;
     bool isGarbageAfterField(size_t index, ReadBuffer::Position pos) override;
     void setReadBuffer(ReadBuffer & in_) override;
+    void readPrefix() override;
 
     const FormatSettings format_settings;
     DataTypes data_types;
@@ -48,7 +49,6 @@ protected:
 
 private:
     bool readRow(MutableColumns & columns, RowReadExtension & ext) override;
-    void readPrefix() override;
 
     bool parseRowAndPrintDiagnosticInfo(MutableColumns & columns, WriteBuffer & out) override;
     void tryDeserializeField(const DataTypePtr & type, IColumn & column, size_t file_column) override;

src/Processors/QueryPlan/BuildQueryPipelineSettings.h

@@ -5,16 +5,18 @@
 
 #include <cstddef>
 
+
 namespace DB
 {
 
 struct Settings;
 class QueryStatus;
+using QueryStatusPtr = std::shared_ptr<QueryStatus>;
 
 struct BuildQueryPipelineSettings
 {
     ExpressionActionsSettings actions_settings;
-    QueryStatus * process_list_element = nullptr;
+    QueryStatusPtr process_list_element;
     ProgressCallback progress_callback = nullptr;
 
     const ExpressionActionsSettings & getActionsSettings() const { return actions_settings; }

src/Processors/Transforms/CountingTransform.h

@@ -9,6 +9,7 @@ namespace DB
 {
 
 class QueryStatus;
+using QueryStatusPtr = std::shared_ptr<QueryStatus>;
 class ThreadStatus;
 
 /// Proxy class which counts number of written block, rows, bytes
@@ -29,7 +30,7 @@ public:
         progress_callback = callback;
     }
 
-    void setProcessListElement(QueryStatus * elem)
+    void setProcessListElement(QueryStatusPtr elem)
     {
         process_elem = elem;
     }
@@ -50,7 +51,7 @@ public:
 protected:
     Progress progress;
     ProgressCallback progress_callback;
-    QueryStatus * process_elem = nullptr;
+    QueryStatusPtr process_elem;
     ThreadStatus * thread_status = nullptr;
 
     /// Quota is used to limit amount of written bytes.

src/Processors/Transforms/buildPushingToViewsChain.cpp

@@ -620,9 +620,10 @@ void PushingToLiveViewSink::consume(Chunk chunk)
 {
     Progress local_progress(chunk.getNumRows(), chunk.bytes(), 0);
     StorageLiveView::writeIntoLiveView(live_view, getHeader().cloneWithColumns(chunk.detachColumns()), context);
-    auto * process = context->getProcessListElement();
-    if (process)
+
+    if (auto process = context->getProcessListElement())
         process->updateProgressIn(local_progress);
+
     ProfileEvents::increment(ProfileEvents::SelectedRows, local_progress.read_rows);
     ProfileEvents::increment(ProfileEvents::SelectedBytes, local_progress.read_bytes);
 }
@@ -643,9 +644,10 @@ void PushingToWindowViewSink::consume(Chunk chunk)
     Progress local_progress(chunk.getNumRows(), chunk.bytes(), 0);
     StorageWindowView::writeIntoWindowView(
         window_view, getHeader().cloneWithColumns(chunk.detachColumns()), context);
-    auto * process = context->getProcessListElement();
-    if (process)
+
+    if (auto process = context->getProcessListElement())
         process->updateProgressIn(local_progress);
+
     ProfileEvents::increment(ProfileEvents::SelectedRows, local_progress.read_rows);
     ProfileEvents::increment(ProfileEvents::SelectedBytes, local_progress.read_bytes);
 }

src/Processors/tests/gtest_exception_on_incorrect_pipeline.cpp

@@ -23,11 +23,11 @@ TEST(Processors, PortsConnected)
 
     connect(source->getPort(), sink->getPort());
 
-    Processors processors;
-    processors.emplace_back(std::move(source));
-    processors.emplace_back(std::move(sink));
+    auto processors = std::make_shared<Processors>();
+    processors->emplace_back(std::move(source));
+    processors->emplace_back(std::move(sink));
 
-    QueryStatus * element = nullptr;
+    QueryStatusPtr element;
     PipelineExecutor executor(processors, element);
     executor.execute(1);
 }
@@ -46,14 +46,14 @@ TEST(Processors, PortsNotConnected)
 
     /// connect(source->getPort(), sink->getPort());
 
-    Processors processors;
-    processors.emplace_back(std::move(source));
-    processors.emplace_back(std::move(sink));
+    auto processors = std::make_shared<Processors>();
+    processors->emplace_back(std::move(source));
+    processors->emplace_back(std::move(sink));
 
 #ifndef ABORT_ON_LOGICAL_ERROR
     try
     {
-        QueryStatus * element = nullptr;
+        QueryStatusPtr element;
         PipelineExecutor executor(processors, element);
         executor.execute(1);
         ASSERT_TRUE(false) << "Should have thrown.";

src/QueryPipeline/BlockIO.cpp

@@ -53,9 +53,8 @@ void BlockIO::setAllDataSent() const
     /// - internal
     /// - SHOW PROCESSLIST
     if (process_list_entry)
-        (*process_list_entry)->setAllDataSent();
+        process_list_entry->getQueryStatus()->setAllDataSent();
 }
 
 
 }
-

src/QueryPipeline/BlockIO.h

@@ -34,9 +34,8 @@ struct BlockIO
     void onFinish()
     {
         if (finish_callback)
-        {
             finish_callback(pipeline);
-        }
+
         pipeline.reset();
     }
 

src/QueryPipeline/Pipe.cpp

@@ -102,7 +102,12 @@ static OutputPort * uniteTotals(const OutputPortRawPtrs & ports, const Block & h
     return totals_port;
 }
 
+Pipe::Pipe() : processors(std::make_shared<Processors>())
+{
+}
+
 Pipe::Pipe(ProcessorPtr source, OutputPort * output, OutputPort * totals, OutputPort * extremes)
+    : processors(std::make_shared<Processors>())
 {
     if (!source->getInputs().empty())
         throw Exception(
@@ -155,11 +160,12 @@ Pipe::Pipe(ProcessorPtr source, OutputPort * output, OutputPort * totals, Output
     totals_port = totals;
     extremes_port = extremes;
     output_ports.push_back(output);
-    processors.emplace_back(std::move(source));
+    processors->emplace_back(std::move(source));
     max_parallel_streams = 1;
 }
 
 Pipe::Pipe(ProcessorPtr source)
+    : processors(std::make_shared<Processors>())
 {
     checkSource(*source);
 
@@ -168,18 +174,18 @@ Pipe::Pipe(ProcessorPtr source)
 
     output_ports.push_back(&source->getOutputs().front());
     header = output_ports.front()->getHeader();
-    processors.emplace_back(std::move(source));
+    processors->emplace_back(std::move(source));
     max_parallel_streams = 1;
 }
 
-Pipe::Pipe(Processors processors_) : processors(std::move(processors_))
+Pipe::Pipe(std::shared_ptr<Processors> processors_) : processors(std::move(processors_))
 {
     /// Create hash table with processors.
     std::unordered_set<const IProcessor *> set;
-    for (const auto & processor : processors)
+    for (const auto & processor : *processors)
         set.emplace(processor.get());
 
-    for (auto & processor : processors)
+    for (auto & processor : *processors)
     {
         for (const auto & port : processor->getInputs())
         {
@@ -225,7 +231,7 @@ Pipe::Pipe(Processors processors_) : processors(std::move(processors_))
     max_parallel_streams = output_ports.size();
 
     if (collected_processors)
-        for (const auto & processor : processors)
+        for (const auto & processor : *processors)
             collected_processors->emplace_back(processor);
 }
 
@@ -311,7 +317,7 @@ Pipe Pipe::unitePipes(Pipes pipes, Processors * collected_processors, bool allow
         if (!allow_empty_header || pipe.header)
             assertCompatibleHeader(pipe.header, res.header, "Pipe::unitePipes");
 
-        res.processors.insert(res.processors.end(), pipe.processors.begin(), pipe.processors.end());
+        res.processors->insert(res.processors->end(), pipe.processors->begin(), pipe.processors->end());
         res.output_ports.insert(res.output_ports.end(), pipe.output_ports.begin(), pipe.output_ports.end());
 
         res.max_parallel_streams += pipe.max_parallel_streams;
@@ -323,15 +329,15 @@ Pipe Pipe::unitePipes(Pipes pipes, Processors * collected_processors, bool allow
             extremes.emplace_back(pipe.extremes_port);
     }
 
-    size_t num_processors = res.processors.size();
+    size_t num_processors = res.processors->size();
 
-    res.totals_port = uniteTotals(totals, res.header, res.processors);
-    res.extremes_port = uniteExtremes(extremes, res.header, res.processors);
+    res.totals_port = uniteTotals(totals, res.header, *res.processors);
+    res.extremes_port = uniteExtremes(extremes, res.header, *res.processors);
 
     if (res.collected_processors)
     {
-        for (; num_processors < res.processors.size(); ++num_processors)
-            res.collected_processors->emplace_back(res.processors[num_processors]);
+        for (; num_processors < res.processors->size(); ++num_processors)
+            res.collected_processors->emplace_back(res.processors->at(num_processors));
     }
 
     return res;
@@ -351,7 +357,7 @@ void Pipe::addSource(ProcessorPtr source)
         collected_processors->emplace_back(source);
 
     output_ports.push_back(&source->getOutputs().front());
-    processors.emplace_back(std::move(source));
+    processors->emplace_back(std::move(source));
 
     max_parallel_streams = std::max<size_t>(max_parallel_streams, output_ports.size());
 }
@@ -373,7 +379,7 @@ void Pipe::addTotalsSource(ProcessorPtr source)
         collected_processors->emplace_back(source);
 
     totals_port = &source->getOutputs().front();
-    processors.emplace_back(std::move(source));
+    processors->emplace_back(std::move(source));
 }
 
 void Pipe::addExtremesSource(ProcessorPtr source)
@@ -393,7 +399,7 @@ void Pipe::addExtremesSource(ProcessorPtr source)
         collected_processors->emplace_back(source);
 
     extremes_port = &source->getOutputs().front();
-    processors.emplace_back(std::move(source));
+    processors->emplace_back(std::move(source));
 }
 
 static void dropPort(OutputPort *& port, Processors & processors, Processors * collected_processors)
@@ -413,12 +419,12 @@ static void dropPort(OutputPort *& port, Processors & processors, Processors * c
 
 void Pipe::dropTotals()
 {
-    dropPort(totals_port, processors, collected_processors);
+    dropPort(totals_port, *processors, collected_processors);
 }
 
 void Pipe::dropExtremes()
 {
-    dropPort(extremes_port, processors, collected_processors);
+    dropPort(extremes_port, *processors, collected_processors);
 }
 
 void Pipe::addTransform(ProcessorPtr transform)
@@ -504,7 +510,7 @@ void Pipe::addTransform(ProcessorPtr transform, OutputPort * totals, OutputPort
     if (collected_processors)
         collected_processors->emplace_back(transform);
 
-    processors.emplace_back(std::move(transform));
+    processors->emplace_back(std::move(transform));
 
     max_parallel_streams = std::max<size_t>(max_parallel_streams, output_ports.size());
 }
@@ -595,7 +601,7 @@ void Pipe::addTransform(ProcessorPtr transform, InputPort * totals, InputPort *
     if (collected_processors)
         collected_processors->emplace_back(transform);
 
-    processors.emplace_back(std::move(transform));
+    processors->emplace_back(std::move(transform));
 
     max_parallel_streams = std::max<size_t>(max_parallel_streams, output_ports.size());
 }
@@ -647,7 +653,7 @@ void Pipe::addSimpleTransform(const ProcessorGetterWithStreamKind & getter)
             if (collected_processors)
                 collected_processors->emplace_back(transform);
 
-            processors.emplace_back(std::move(transform));
+            processors->emplace_back(std::move(transform));
         }
     };
 
@@ -698,7 +704,7 @@ void Pipe::addChains(std::vector<Chain> chains)
             if (collected_processors)
                 collected_processors->emplace_back(transform);
 
-            processors.emplace_back(std::move(transform));
+            processors->emplace_back(std::move(transform));
         }
     }
 
@@ -757,7 +763,7 @@ void Pipe::setSinks(const Pipe::ProcessorGetterWithStreamKind & getter)
             transform = std::make_shared<NullSink>(stream->getHeader());
 
         connect(*stream, transform->getInputs().front());
-        processors.emplace_back(std::move(transform));
+        processors->emplace_back(std::move(transform));
     };
 
     for (auto & port : output_ports)
@@ -858,7 +864,7 @@ void Pipe::transform(const Transformer & transformer, bool check_ports)
             collected_processors->emplace_back(processor);
     }
 
-    processors.insert(processors.end(), new_processors.begin(), new_processors.end());
+    processors->insert(processors->end(), new_processors.begin(), new_processors.end());
 
     max_parallel_streams = std::max<size_t>(max_parallel_streams, output_ports.size());
 }

src/QueryPipeline/Pipe.h

@@ -5,6 +5,7 @@
 #include <QueryPipeline/Chain.h>
 #include <QueryPipeline/SizeLimits.h>
 
+
 namespace DB
 {
 
@@ -27,13 +28,13 @@ class Pipe
 public:
     /// Default constructor creates empty pipe. Generally, you cannot do anything with it except to check it is empty().
     /// You cannot get empty pipe in any other way. All transforms check that result pipe is not empty.
-    Pipe() = default;
+    Pipe();
     /// Create from source. Source must have no input ports and single output.
     explicit Pipe(ProcessorPtr source);
     /// Create from source with specified totals end extremes (may be nullptr). Ports should be owned by source.
     explicit Pipe(ProcessorPtr source, OutputPort * output, OutputPort * totals, OutputPort * extremes);
     /// Create from processors. Use all not-connected output ports as output_ports. Check invariants.
-    explicit Pipe(Processors processors_);
+    explicit Pipe(std::shared_ptr<Processors> processors_);
 
     Pipe(const Pipe & other) = delete;
     Pipe(Pipe && other) = default;
@@ -41,7 +42,7 @@ public:
     Pipe & operator=(Pipe && other) = default;
 
     const Block & getHeader() const { return header; }
-    bool empty() const { return processors.empty(); }
+    bool empty() const { return processors->empty(); }
     size_t numOutputPorts() const { return output_ports.size(); }
     size_t maxParallelStreams() const { return max_parallel_streams; }
     OutputPort * getOutputPort(size_t pos) const { return output_ports[pos]; }
@@ -96,15 +97,15 @@ public:
     /// Unite several pipes together. They should have same header.
     static Pipe unitePipes(Pipes pipes);
 
-    /// Get processors from Pipe. Use it with cautious, it is easy to loss totals and extremes ports.
-    static Processors detachProcessors(Pipe pipe) { return std::move(pipe.processors); }
+    /// Get processors from Pipe. Use it with caution, it is easy to lose totals and extremes ports.
+    static Processors detachProcessors(Pipe pipe) { return *std::move(pipe.processors); }
     /// Get processors from Pipe without destroying pipe (used for EXPLAIN to keep QueryPlan).
-    const Processors & getProcessors() const { return processors; }
+    const Processors & getProcessors() const { return *processors; }
 
 private:
     /// Header is common for all output below.
     Block header;
-    Processors processors;
+    std::shared_ptr<Processors> processors;
 
     /// Output ports. Totals and extremes are allowed to be empty.
     OutputPortRawPtrs output_ports;