Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into hdfs-idisk

CHANGELOG.md

@@ -1,3 +1,142 @@
+## ClickHouse release 21.5, 2021-05-20
+
+#### Backward Incompatible Change
+
+* Change comparison of integers and floating point numbers when integer is not exactly representable in the floating point data type. In new version comparison will return false as the rounding error will occur. Example: `9223372036854775808.0 != 9223372036854775808`, because the number `9223372036854775808` is not representable as floating point number exactly (and `9223372036854775808.0` is rounded to `9223372036854776000.0`). But in previous version the comparison will return as the numbers are equal, because if the floating point number `9223372036854776000.0` get converted back to UInt64, it will yield `9223372036854775808`. For the reference, the Python programming language also treats these numbers as equal. But this behaviour was dependend on CPU model (different results on AMD64 and AArch64 for some out-of-range numbers), so we make the comparison more precise. It will treat int and float numbers equal only if int is represented in floating point type exactly. [#22595](https://github.com/ClickHouse/ClickHouse/pull/22595) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Remove support for `argMin` and `argMax` for single `Tuple` argument. The code was not memory-safe. The feature was added by mistake and it is confusing for people. These functions can be reintroduced under different names later. This fixes [#22384](https://github.com/ClickHouse/ClickHouse/issues/22384) and reverts [#17359](https://github.com/ClickHouse/ClickHouse/issues/17359). [#23393](https://github.com/ClickHouse/ClickHouse/pull/23393) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+
+#### New Feature
+
+* Added functions `dictGetChildren(dictionary, key)`, `dictGetDescendants(dictionary, key, level)`. Function `dictGetChildren` return all children as an array if indexes. It is a inverse transformation for `dictGetHierarchy`. Function `dictGetDescendants` return all descendants as if `dictGetChildren` was applied `level` times recursively. Zero `level` value is equivalent to infinity. Improved performance of `dictGetHierarchy`, `dictIsIn` functions. Closes [#14656](https://github.com/ClickHouse/ClickHouse/issues/14656). [#22096](https://github.com/ClickHouse/ClickHouse/pull/22096) ([Maksim Kita](https://github.com/kitaisreal)).
+* Added function `dictGetOrNull`. It works like `dictGet`, but return `Null` in case key was not found in dictionary. Closes [#22375](https://github.com/ClickHouse/ClickHouse/issues/22375). [#22413](https://github.com/ClickHouse/ClickHouse/pull/22413) ([Maksim Kita](https://github.com/kitaisreal)).
+* Added a table function `s3Cluster`, which allows to process files from `s3` in parallel on every node of a specified cluster. [#22012](https://github.com/ClickHouse/ClickHouse/pull/22012) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+* Added support for replicas and shards in MySQL/PostgreSQL table engine / table function. You can write `SELECT * FROM mysql('host{1,2}-{1|2}', ...)`. Closes [#20969](https://github.com/ClickHouse/ClickHouse/issues/20969). [#22217](https://github.com/ClickHouse/ClickHouse/pull/22217) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Added `ALTER TABLE ... FETCH PART ...` query. It's similar to `FETCH PARTITION`, but fetches only one part. [#22706](https://github.com/ClickHouse/ClickHouse/pull/22706) ([turbo jason](https://github.com/songenjie)).
+* Added a setting `max_distributed_depth` that limits the depth of recursive queries to `Distributed` tables. Closes [#20229](https://github.com/ClickHouse/ClickHouse/issues/20229). [#21942](https://github.com/ClickHouse/ClickHouse/pull/21942) ([flynn](https://github.com/ucasFL)).
+
+#### Performance Improvement
+
+* Improved performance of `intDiv` by dynamic dispatch for AVX2. This closes [#22314](https://github.com/ClickHouse/ClickHouse/issues/22314). [#23000](https://github.com/ClickHouse/ClickHouse/pull/23000) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Improved performance of reading from `ArrowStream` input format for sources other then local file (e.g. URL). [#22673](https://github.com/ClickHouse/ClickHouse/pull/22673) ([nvartolomei](https://github.com/nvartolomei)).
+* Disabled compression by default when interacting with localhost (with clickhouse-client or server to server with distributed queries) via native protocol. It may improve performance of some import/export operations. This closes [#22234](https://github.com/ClickHouse/ClickHouse/issues/22234). [#22237](https://github.com/ClickHouse/ClickHouse/pull/22237) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Exclude values that does not belong to the shard from right part of IN section for distributed queries (under `optimize_skip_unused_shards_rewrite_in`, enabled by default, since it still requires `optimize_skip_unused_shards`). [#21511](https://github.com/ClickHouse/ClickHouse/pull/21511) ([Azat Khuzhin](https://github.com/azat)).
+* Improved performance of reading a subset of columns with File-like table engine and column-oriented format like Parquet, Arrow or ORC. This closes [#issue:20129](https://github.com/ClickHouse/ClickHouse/issues/20129). [#21302](https://github.com/ClickHouse/ClickHouse/pull/21302) ([keenwolf](https://github.com/keen-wolf)).
+* Allow to move more conditions to `PREWHERE` as it was before version 21.1 (adjustment of internal heuristics). Insufficient number of moved condtions could lead to worse performance. [#23397](https://github.com/ClickHouse/ClickHouse/pull/23397) ([Anton Popov](https://github.com/CurtizJ)).
+* Improved performance of ODBC connections and fixed all the outstanding issues from the backlog. Using `nanodbc` library instead of `Poco::ODBC`. Closes [#9678](https://github.com/ClickHouse/ClickHouse/issues/9678). Add support for DateTime64 and Decimal* for ODBC table engine. Closes [#21961](https://github.com/ClickHouse/ClickHouse/issues/21961). Fixed issue with cyrillic text being truncated. Closes [#16246](https://github.com/ClickHouse/ClickHouse/issues/16246). Added connection pools for odbc bridge. [#21972](https://github.com/ClickHouse/ClickHouse/pull/21972) ([Kseniia Sumarokova](https://github.com/kssenii)).
+
+#### Improvement
+
+* Increase `max_uri_size` (the maximum size of URL in HTTP interface) to 1 MiB by default. This closes [#21197](https://github.com/ClickHouse/ClickHouse/issues/21197). [#22997](https://github.com/ClickHouse/ClickHouse/pull/22997) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Set `background_fetches_pool_size` to `8` that is better for production usage with frequent small insertions or slow ZooKeeper cluster. [#22945](https://github.com/ClickHouse/ClickHouse/pull/22945) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* FlatDictionary added `initial_array_size`, `max_array_size` options. [#22521](https://github.com/ClickHouse/ClickHouse/pull/22521) ([Maksim Kita](https://github.com/kitaisreal)).
+* Add new setting `non_replicated_deduplication_window` for non-replicated MergeTree inserts deduplication. [#22514](https://github.com/ClickHouse/ClickHouse/pull/22514) ([alesapin](https://github.com/alesapin)).
+* Update paths to the `CatBoost` model configs in config reloading. [#22434](https://github.com/ClickHouse/ClickHouse/pull/22434) ([Kruglov Pavel](https://github.com/Avogar)).
+* Added `Decimal256` type support in dictionaries. `Decimal256` is experimental feature. Closes [#20979](https://github.com/ClickHouse/ClickHouse/issues/20979). [#22960](https://github.com/ClickHouse/ClickHouse/pull/22960) ([Maksim Kita](https://github.com/kitaisreal)).
+* Enabled `async_socket_for_remote` by default (using less amount of OS threads for distributed queries). [#23683](https://github.com/ClickHouse/ClickHouse/pull/23683) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
+* Fixed `quantile(s)TDigest`. Added special handling of singleton centroids according to tdunning/t-digest 3.2+. Also a bug with over-compression of centroids in implementation of earlier version of the algorithm was fixed. [#23314](https://github.com/ClickHouse/ClickHouse/pull/23314) ([Vladimir Chebotarev](https://github.com/excitoon)).
+* Make function name `unhex` case insensitive for compatibility with MySQL. [#23229](https://github.com/ClickHouse/ClickHouse/pull/23229) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Implement functions `arrayHasAny`, `arrayHasAll`, `has`, `indexOf`, `countEqual` for generic case when types of array elements are different. In previous versions the functions `arrayHasAny`, `arrayHasAll` returned false and `has`, `indexOf`, `countEqual` thrown exception. Also add support for `Decimal` and big integer types in functions `has` and similar. This closes [#20272](https://github.com/ClickHouse/ClickHouse/issues/20272). [#23044](https://github.com/ClickHouse/ClickHouse/pull/23044) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Raised the threshold on max number of matches in result of the function `extractAllGroupsHorizontal`. [#23036](https://github.com/ClickHouse/ClickHouse/pull/23036) ([Vasily Nemkov](https://github.com/Enmk)).
+* Do not perform `optimize_skip_unused_shards` for cluster with one node. [#22999](https://github.com/ClickHouse/ClickHouse/pull/22999) ([Azat Khuzhin](https://github.com/azat)).
+* Added ability to run clickhouse-keeper (experimental drop-in replacement to ZooKeeper) with SSL. Config settings `keeper_server.tcp_port_secure` can be used for secure interaction between client and keeper-server. `keeper_server.raft_configuration.secure` can be used to enable internal secure communication between nodes. [#22992](https://github.com/ClickHouse/ClickHouse/pull/22992) ([alesapin](https://github.com/alesapin)).
+* Added ability to flush buffer only in background for `Buffer` tables. [#22986](https://github.com/ClickHouse/ClickHouse/pull/22986) ([Azat Khuzhin](https://github.com/azat)).
+* When selecting from MergeTree table with NULL in WHERE condition, in rare cases, exception was thrown. This closes [#20019](https://github.com/ClickHouse/ClickHouse/issues/20019). [#22978](https://github.com/ClickHouse/ClickHouse/pull/22978) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Fix error handling in Poco HTTP Client for AWS. [#22973](https://github.com/ClickHouse/ClickHouse/pull/22973) ([kreuzerkrieg](https://github.com/kreuzerkrieg)).
+* Respect `max_part_removal_threads` for `ReplicatedMergeTree`. [#22971](https://github.com/ClickHouse/ClickHouse/pull/22971) ([Azat Khuzhin](https://github.com/azat)).
+* Fix obscure corner case of MergeTree settings inactive_parts_to_throw_insert = 0 with inactive_parts_to_delay_insert > 0. [#22947](https://github.com/ClickHouse/ClickHouse/pull/22947) ([Azat Khuzhin](https://github.com/azat)).
+* `dateDiff` now works with `DateTime64` arguments (even for values outside of `DateTime` range) [#22931](https://github.com/ClickHouse/ClickHouse/pull/22931) ([Vasily Nemkov](https://github.com/Enmk)).
+* MaterializeMySQL (experimental feature): added an ability to replicate MySQL databases containing views without failing. This is accomplished by ignoring the views. [#22760](https://github.com/ClickHouse/ClickHouse/pull/22760) ([Christian](https://github.com/cfroystad)).
+* Allow RBAC row policy via postgresql protocol. Closes [#22658](https://github.com/ClickHouse/ClickHouse/issues/22658). PostgreSQL protocol is enabled in configuration by default. [#22755](https://github.com/ClickHouse/ClickHouse/pull/22755) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Add metric to track how much time is spend during waiting for Buffer layer lock. [#22725](https://github.com/ClickHouse/ClickHouse/pull/22725) ([Azat Khuzhin](https://github.com/azat)).
+* Allow to use CTE in VIEW definition. This closes [#22491](https://github.com/ClickHouse/ClickHouse/issues/22491). [#22657](https://github.com/ClickHouse/ClickHouse/pull/22657) ([Amos Bird](https://github.com/amosbird)).
+* Clear the rest of the screen and show cursor in `clickhouse-client` if previous program has left garbage in terminal. This closes [#16518](https://github.com/ClickHouse/ClickHouse/issues/16518). [#22634](https://github.com/ClickHouse/ClickHouse/pull/22634) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Make `round` function to behave consistently on non-x86_64 platforms. Rounding half to nearest even (Banker's rounding) is used. [#22582](https://github.com/ClickHouse/ClickHouse/pull/22582) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Correctly check structure of blocks of data that are sending by Distributed tables. [#22325](https://github.com/ClickHouse/ClickHouse/pull/22325) ([Azat Khuzhin](https://github.com/azat)).
+* Allow publishing Kafka errors to a virtual column of Kafka engine, controlled by the `kafka_handle_error_mode` setting. [#21850](https://github.com/ClickHouse/ClickHouse/pull/21850) ([fastio](https://github.com/fastio)).
+* Add aliases `simpleJSONExtract/simpleJSONHas` to `visitParam/visitParamExtract{UInt, Int, Bool, Float, Raw, String}`. Fixes [#21383](https://github.com/ClickHouse/ClickHouse/issues/21383). [#21519](https://github.com/ClickHouse/ClickHouse/pull/21519) ([fastio](https://github.com/fastio)).
+* Add `clickhouse-library-bridge` for library dictionary source. Closes [#9502](https://github.com/ClickHouse/ClickHouse/issues/9502). [#21509](https://github.com/ClickHouse/ClickHouse/pull/21509) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Forbid to drop a column if it's referenced by materialized view. Closes [#21164](https://github.com/ClickHouse/ClickHouse/issues/21164). [#21303](https://github.com/ClickHouse/ClickHouse/pull/21303) ([flynn](https://github.com/ucasFL)).
+* Support dynamic interserver credentials (rotating credentials without downtime). [#14113](https://github.com/ClickHouse/ClickHouse/pull/14113) ([johnskopis](https://github.com/johnskopis)).
+* Add support for Kafka storage with `Arrow` and `ArrowStream` format messages. [#23415](https://github.com/ClickHouse/ClickHouse/pull/23415) ([Chao Ma](https://github.com/godliness)).
+* Fixed missing semicolon in exception message. The user may find this exception message unpleasant to read. [#23208](https://github.com/ClickHouse/ClickHouse/pull/23208) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Fixed missing whitespace in some exception messages about `LowCardinality` type. [#23207](https://github.com/ClickHouse/ClickHouse/pull/23207) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Some values were formatted with alignment in center in table cells in `Markdown` format. Not anymore. [#23096](https://github.com/ClickHouse/ClickHouse/pull/23096) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Remove non-essential details from suggestions in clickhouse-client. This closes [#22158](https://github.com/ClickHouse/ClickHouse/issues/22158). [#23040](https://github.com/ClickHouse/ClickHouse/pull/23040) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Correct calculation of `bytes_allocated` field in system.dictionaries for sparse_hashed dictionaries. [#22867](https://github.com/ClickHouse/ClickHouse/pull/22867) ([Azat Khuzhin](https://github.com/azat)).
+* Fixed approximate total rows accounting for reverse reading from MergeTree. [#22726](https://github.com/ClickHouse/ClickHouse/pull/22726) ([Azat Khuzhin](https://github.com/azat)).
+* Fix the case when it was possible to configure dictionary with clickhouse source that was looking to itself that leads to infinite loop. Closes [#14314](https://github.com/ClickHouse/ClickHouse/issues/14314). [#22479](https://github.com/ClickHouse/ClickHouse/pull/22479) ([Maksim Kita](https://github.com/kitaisreal)).
+
+#### Bug Fix
+
+* Multiple fixes for hedged requests. Fixed an error `Can't initialize pipeline with empty pipe` for queries with `GLOBAL IN/JOIN` when the setting `use_hedged_requests` is enabled. Fixes [#23431](https://github.com/ClickHouse/ClickHouse/issues/23431). [#23805](https://github.com/ClickHouse/ClickHouse/pull/23805) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). Fixed a race condition in hedged connections which leads to crash. This fixes [#22161](https://github.com/ClickHouse/ClickHouse/issues/22161). [#22443](https://github.com/ClickHouse/ClickHouse/pull/22443) ([Kruglov Pavel](https://github.com/Avogar)). Fix possible crash in case if `unknown packet` was received from remote query (with `async_socket_for_remote` enabled). Fixes [#21167](https://github.com/ClickHouse/ClickHouse/issues/21167). [#23309](https://github.com/ClickHouse/ClickHouse/pull/23309) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
+* Fixed the behavior when disabling `input_format_with_names_use_header ` setting discards all the input with CSVWithNames format. This fixes [#22406](https://github.com/ClickHouse/ClickHouse/issues/22406). [#23202](https://github.com/ClickHouse/ClickHouse/pull/23202) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+* Fixed remote JDBC bridge timeout connection issue. Closes [#9609](https://github.com/ClickHouse/ClickHouse/issues/9609). [#23771](https://github.com/ClickHouse/ClickHouse/pull/23771) ([Maksim Kita](https://github.com/kitaisreal), [alexey-milovidov](https://github.com/alexey-milovidov)).
+* Fix the logic of initial load of `complex_key_hashed` if `update_field` is specified. Closes [#23800](https://github.com/ClickHouse/ClickHouse/issues/23800). [#23824](https://github.com/ClickHouse/ClickHouse/pull/23824) ([Maksim Kita](https://github.com/kitaisreal)).
+* Fixed crash when `PREWHERE` and row policy filter are both in effect with empty result. [#23763](https://github.com/ClickHouse/ClickHouse/pull/23763) ([Amos Bird](https://github.com/amosbird)).
+* Avoid possible "Cannot schedule a task" error (in case some exception had been occurred) on INSERT into Distributed. [#23744](https://github.com/ClickHouse/ClickHouse/pull/23744) ([Azat Khuzhin](https://github.com/azat)).
+* Added an exception in case of completely the same values in both samples in aggregate function `mannWhitneyUTest`. This fixes [#23646](https://github.com/ClickHouse/ClickHouse/issues/23646). [#23654](https://github.com/ClickHouse/ClickHouse/pull/23654) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+* Fixed server fault when inserting data through HTTP caused an exception. This fixes [#23512](https://github.com/ClickHouse/ClickHouse/issues/23512). [#23643](https://github.com/ClickHouse/ClickHouse/pull/23643) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+* Fixed misinterpretation of some `LIKE` expressions with escape sequences. [#23610](https://github.com/ClickHouse/ClickHouse/pull/23610) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Fixed restart / stop command hanging. Closes [#20214](https://github.com/ClickHouse/ClickHouse/issues/20214). [#23552](https://github.com/ClickHouse/ClickHouse/pull/23552) ([filimonov](https://github.com/filimonov)).
+* Fixed `COLUMNS` matcher in case of multiple JOINs in select query. Closes [#22736](https://github.com/ClickHouse/ClickHouse/issues/22736). [#23501](https://github.com/ClickHouse/ClickHouse/pull/23501) ([Maksim Kita](https://github.com/kitaisreal)).
+* Fixed a crash when modifying column's default value when a column itself is used as `ReplacingMergeTree`'s parameter. [#23483](https://github.com/ClickHouse/ClickHouse/pull/23483) ([hexiaoting](https://github.com/hexiaoting)).
+* Fixed corner cases in vertical merges with `ReplacingMergeTree`. In rare cases they could lead to fails of merges with exceptions like `Incomplete granules are not allowed while blocks are granules size`. [#23459](https://github.com/ClickHouse/ClickHouse/pull/23459) ([Anton Popov](https://github.com/CurtizJ)).
+* Fixed bug that does not allow cast from empty array literal, to array with dimensions greater than 1, e.g. `CAST([] AS Array(Array(String)))`. Closes [#14476](https://github.com/ClickHouse/ClickHouse/issues/14476). [#23456](https://github.com/ClickHouse/ClickHouse/pull/23456) ([Maksim Kita](https://github.com/kitaisreal)).
+* Fixed a bug when `deltaSum` aggregate function produced incorrect result after resetting the counter. [#23437](https://github.com/ClickHouse/ClickHouse/pull/23437) ([Russ Frank](https://github.com/rf)).
+* Fixed `Cannot unlink file` error on unsuccessful creation of ReplicatedMergeTree table with multidisk configuration. This closes [#21755](https://github.com/ClickHouse/ClickHouse/issues/21755). [#23433](https://github.com/ClickHouse/ClickHouse/pull/23433) ([tavplubix](https://github.com/tavplubix)).
+* Fixed incompatible constant expression generation during partition pruning based on virtual columns. This fixes https://github.com/ClickHouse/ClickHouse/pull/21401#discussion_r611888913. [#23366](https://github.com/ClickHouse/ClickHouse/pull/23366) ([Amos Bird](https://github.com/amosbird)).
+* Fixed a crash when setting join_algorithm is set to 'auto' and Join is performed with a Dictionary. Close [#23002](https://github.com/ClickHouse/ClickHouse/issues/23002). [#23312](https://github.com/ClickHouse/ClickHouse/pull/23312) ([Vladimir](https://github.com/vdimir)).
+* Don't relax NOT conditions during partition pruning. This fixes [#23305](https://github.com/ClickHouse/ClickHouse/issues/23305) and [#21539](https://github.com/ClickHouse/ClickHouse/issues/21539). [#23310](https://github.com/ClickHouse/ClickHouse/pull/23310) ([Amos Bird](https://github.com/amosbird)).
+* Fixed very rare race condition on background cleanup of old blocks. It might cause a block not to be deduplicated if it's too close to the end of deduplication window. [#23301](https://github.com/ClickHouse/ClickHouse/pull/23301) ([tavplubix](https://github.com/tavplubix)).
+* Fixed very rare (distributed) race condition between creation and removal of ReplicatedMergeTree tables. It might cause exceptions like `node doesn't exist` on attempt to create replicated table. Fixes [#21419](https://github.com/ClickHouse/ClickHouse/issues/21419). [#23294](https://github.com/ClickHouse/ClickHouse/pull/23294) ([tavplubix](https://github.com/tavplubix)).
+* Fixed simple key dictionary from DDL creation if primary key is not first attribute. Fixes [#23236](https://github.com/ClickHouse/ClickHouse/issues/23236). [#23262](https://github.com/ClickHouse/ClickHouse/pull/23262) ([Maksim Kita](https://github.com/kitaisreal)).
+* Fixed reading from ODBC when there are many long column names in a table. Closes [#8853](https://github.com/ClickHouse/ClickHouse/issues/8853). [#23215](https://github.com/ClickHouse/ClickHouse/pull/23215) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* MaterializeMySQL (experimental feature): fixed `Not found column` error when selecting from `MaterializeMySQL` with condition on key column. Fixes [#22432](https://github.com/ClickHouse/ClickHouse/issues/22432). [#23200](https://github.com/ClickHouse/ClickHouse/pull/23200) ([tavplubix](https://github.com/tavplubix)).
+* Correct aliases handling if subquery was optimized to constant. Fixes [#22924](https://github.com/ClickHouse/ClickHouse/issues/22924). Fixes [#10401](https://github.com/ClickHouse/ClickHouse/issues/10401). [#23191](https://github.com/ClickHouse/ClickHouse/pull/23191) ([Maksim Kita](https://github.com/kitaisreal)).
+* Server might fail to start if `data_type_default_nullable` setting is enabled in default profile, it's fixed. Fixes [#22573](https://github.com/ClickHouse/ClickHouse/issues/22573). [#23185](https://github.com/ClickHouse/ClickHouse/pull/23185) ([tavplubix](https://github.com/tavplubix)).
+* Fixed a crash on shutdown which happened because of wrong accounting of current connections. [#23154](https://github.com/ClickHouse/ClickHouse/pull/23154) ([Vitaly Baranov](https://github.com/vitlibar)).
+* Fixed `Table .inner_id... doesn't exist` error when selecting from Materialized View after detaching it from Atomic database and attaching back. [#23047](https://github.com/ClickHouse/ClickHouse/pull/23047) ([tavplubix](https://github.com/tavplubix)).
+* Fix error `Cannot find column in ActionsDAG result` which may happen if subquery uses `untuple`. Fixes [#22290](https://github.com/ClickHouse/ClickHouse/issues/22290). [#22991](https://github.com/ClickHouse/ClickHouse/pull/22991) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
+* Fix usage of constant columns of type `Map` with nullable values. [#22939](https://github.com/ClickHouse/ClickHouse/pull/22939) ([Anton Popov](https://github.com/CurtizJ)).
+* fixed `formatDateTime()` on `DateTime64` and "%C" format specifier fixed `toDateTime64()` for large values and non-zero scale. [#22937](https://github.com/ClickHouse/ClickHouse/pull/22937) ([Vasily Nemkov](https://github.com/Enmk)).
+* Fixed a crash when using `mannWhitneyUTest` and `rankCorr` with window functions. This fixes [#22728](https://github.com/ClickHouse/ClickHouse/issues/22728). [#22876](https://github.com/ClickHouse/ClickHouse/pull/22876) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+* LIVE VIEW (experimental feature): fixed possible hanging in concurrent DROP/CREATE of TEMPORARY LIVE VIEW in `TemporaryLiveViewCleaner`, [see](https://gist.github.com/vzakaznikov/0c03195960fc86b56bfe2bc73a90019e). [#22858](https://github.com/ClickHouse/ClickHouse/pull/22858) ([Vitaly Baranov](https://github.com/vitlibar)).
+* Fixed pushdown of `HAVING` in case, when filter column is used in aggregation. [#22763](https://github.com/ClickHouse/ClickHouse/pull/22763) ([Anton Popov](https://github.com/CurtizJ)).
+* Fixed possible hangs in Zookeeper requests in case of OOM exception. Fixes [#22438](https://github.com/ClickHouse/ClickHouse/issues/22438). [#22684](https://github.com/ClickHouse/ClickHouse/pull/22684) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
+* Fixed wait for mutations on several replicas for ReplicatedMergeTree table engines. Previously, mutation/alter query may finish before mutation actually executed on other replicas. [#22669](https://github.com/ClickHouse/ClickHouse/pull/22669) ([alesapin](https://github.com/alesapin)).
+* Fixed exception for Log with nested types without columns in the SELECT clause. [#22654](https://github.com/ClickHouse/ClickHouse/pull/22654) ([Azat Khuzhin](https://github.com/azat)).
+* Fix unlimited wait for auxiliary AWS requests. [#22594](https://github.com/ClickHouse/ClickHouse/pull/22594) ([Vladimir Chebotarev](https://github.com/excitoon)).
+* Fixed a crash when client closes connection very early [#22579](https://github.com/ClickHouse/ClickHouse/issues/22579). [#22591](https://github.com/ClickHouse/ClickHouse/pull/22591) ([nvartolomei](https://github.com/nvartolomei)).
+* `Map` data type (experimental feature): fixed an incorrect formatting of function `map` in distributed queries. [#22588](https://github.com/ClickHouse/ClickHouse/pull/22588) ([foolchi](https://github.com/foolchi)).
+* Fixed deserialization of empty string without newline at end of TSV format. This closes [#20244](https://github.com/ClickHouse/ClickHouse/issues/20244). Possible workaround without version update: set `input_format_null_as_default` to zero. It was zero in old versions. [#22527](https://github.com/ClickHouse/ClickHouse/pull/22527) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Fixed wrong cast of a column of `LowCardinality` type in Merge Join algorithm. Close [#22386](https://github.com/ClickHouse/ClickHouse/issues/22386), close [#22388](https://github.com/ClickHouse/ClickHouse/issues/22388). [#22510](https://github.com/ClickHouse/ClickHouse/pull/22510) ([Vladimir](https://github.com/vdimir)).
+* Buffer overflow (on read) was possible in `tokenbf_v1` full text index. The excessive bytes are not used but the read operation may lead to crash in rare cases. This closes [#19233](https://github.com/ClickHouse/ClickHouse/issues/19233). [#22421](https://github.com/ClickHouse/ClickHouse/pull/22421) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Do not limit HTTP chunk size. Fixes [#21907](https://github.com/ClickHouse/ClickHouse/issues/21907). [#22322](https://github.com/ClickHouse/ClickHouse/pull/22322) ([Ivan](https://github.com/abyss7)).
+* Fixed a bug, which leads to underaggregation of data in case of enabled `optimize_aggregation_in_order` and many parts in table. Slightly improve performance of aggregation with enabled `optimize_aggregation_in_order`. [#21889](https://github.com/ClickHouse/ClickHouse/pull/21889) ([Anton Popov](https://github.com/CurtizJ)).
+* Check if table function view is used as a column. This complements #20350. [#21465](https://github.com/ClickHouse/ClickHouse/pull/21465) ([Amos Bird](https://github.com/amosbird)).
+* Fix "unknown column" error for tables with `Merge` engine in queris with `JOIN` and aggregation. Closes [#18368](https://github.com/ClickHouse/ClickHouse/issues/18368), close [#22226](https://github.com/ClickHouse/ClickHouse/issues/22226). [#21370](https://github.com/ClickHouse/ClickHouse/pull/21370) ([Vladimir](https://github.com/vdimir)).
+* Fixed name clashes in pushdown optimization. It caused incorrect `WHERE` filtration after FULL JOIN. Close [#20497](https://github.com/ClickHouse/ClickHouse/issues/20497). [#20622](https://github.com/ClickHouse/ClickHouse/pull/20622) ([Vladimir](https://github.com/vdimir)).
+* Fixed very rare bug when quorum insert with `quorum_parallel=1` is not really "quorum" because of deduplication. [#18215](https://github.com/ClickHouse/ClickHouse/pull/18215) ([filimonov](https://github.com/filimonov) - reported, [alesapin](https://github.com/alesapin) - fixed).
+
+#### Build/Testing/Packaging Improvement
+
+* Run stateless tests in parallel in CI. [#22300](https://github.com/ClickHouse/ClickHouse/pull/22300) ([alesapin](https://github.com/alesapin)).
+* Simplify debian packages. This fixes [#21698](https://github.com/ClickHouse/ClickHouse/issues/21698). [#22976](https://github.com/ClickHouse/ClickHouse/pull/22976) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Added support for ClickHouse build on Apple M1. [#21639](https://github.com/ClickHouse/ClickHouse/pull/21639) ([changvvb](https://github.com/changvvb)).
+* Fixed ClickHouse Keeper build for MacOS. [#22860](https://github.com/ClickHouse/ClickHouse/pull/22860) ([alesapin](https://github.com/alesapin)).
+* Fixed some tests on AArch64 platform. [#22596](https://github.com/ClickHouse/ClickHouse/pull/22596) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Added function alignment for possibly better performance. [#21431](https://github.com/ClickHouse/ClickHouse/pull/21431) ([Danila Kutenin](https://github.com/danlark1)).
+* Adjust some tests to output identical results on amd64 and aarch64 (qemu). The result was depending on implementation specific CPU behaviour. [#22590](https://github.com/ClickHouse/ClickHouse/pull/22590) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Allow query profiling only on x86_64. See [#15174](https://github.com/ClickHouse/ClickHouse/issues/15174#issuecomment-812954965) and [#15638](https://github.com/ClickHouse/ClickHouse/issues/15638#issuecomment-703805337). This closes [#15638](https://github.com/ClickHouse/ClickHouse/issues/15638). [#22580](https://github.com/ClickHouse/ClickHouse/pull/22580) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Allow building with unbundled xz (lzma) using `USE_INTERNAL_XZ_LIBRARY=OFF` CMake option. [#22571](https://github.com/ClickHouse/ClickHouse/pull/22571) ([Kfir Itzhak](https://github.com/mastertheknife)).
+* Enable bundled `openldap` on `ppc64le` [#22487](https://github.com/ClickHouse/ClickHouse/pull/22487) ([Kfir Itzhak](https://github.com/mastertheknife)).
+* Disable incompatible libraries (platform specific typically) on `ppc64le` [#22475](https://github.com/ClickHouse/ClickHouse/pull/22475) ([Kfir Itzhak](https://github.com/mastertheknife)).
+* Add Jepsen test in CI for clickhouse Keeper. [#22373](https://github.com/ClickHouse/ClickHouse/pull/22373) ([alesapin](https://github.com/alesapin)).
+* Build `jemalloc` with support for [heap profiling](https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling). [#22834](https://github.com/ClickHouse/ClickHouse/pull/22834) ([nvartolomei](https://github.com/nvartolomei)).
+* Avoid UB in `*Log` engines for rwlock unlock due to unlock from another thread. [#22583](https://github.com/ClickHouse/ClickHouse/pull/22583) ([Azat Khuzhin](https://github.com/azat)).
+* Fixed UB by unlocking the rwlock of the TinyLog from the same thread. [#22560](https://github.com/ClickHouse/ClickHouse/pull/22560) ([Azat Khuzhin](https://github.com/azat)).
+
+
 ## ClickHouse release 21.4
 
 ### ClickHouse release 21.4.1 2021-04-12

base/daemon/BaseDaemon.cpp

@@ -468,7 +468,7 @@ void BaseDaemon::reloadConfiguration()
       *  instead of using files specified in config.xml.
       * (It's convenient to log in console when you start server without any command line parameters.)
       */
-    config_path = config().getString("config-file", "config.xml");
+    config_path = config().getString("config-file", getDefaultConfigFileName());
     DB::ConfigProcessor config_processor(config_path, false, true);
     config_processor.setConfigPath(Poco::Path(config_path).makeParent().toString());
     loaded_config = config_processor.loadConfig(/* allow_zk_includes = */ true);
@@ -516,6 +516,11 @@ std::string BaseDaemon::getDefaultCorePath() const
     return "/opt/cores/";
 }
 
+std::string BaseDaemon::getDefaultConfigFileName() const
+{
+    return "config.xml";
+}
+
 void BaseDaemon::closeFDs()
 {
 #if defined(OS_FREEBSD) || defined(OS_DARWIN)

base/daemon/BaseDaemon.h

@@ -149,6 +149,8 @@ protected:
 
     virtual std::string getDefaultCorePath() const;
 
+    virtual std::string getDefaultConfigFileName() const;
+
     std::optional<DB::StatusFile> pid_file;
 
     std::atomic_bool is_cancelled{false};

base/mysqlxx/PoolWithFailover.cpp

@@ -78,6 +78,8 @@ PoolWithFailover::PoolWithFailover(
         const RemoteDescription & addresses,
         const std::string & user,
         const std::string & password,
+        unsigned default_connections_,
+        unsigned max_connections_,
         size_t max_tries_)
     : max_tries(max_tries_)
     , shareable(false)
@@ -85,7 +87,13 @@ PoolWithFailover::PoolWithFailover(
     /// Replicas have the same priority, but traversed replicas are moved to the end of the queue.
     for (const auto & [host, port] : addresses)
     {
-        replicas_by_priority[0].emplace_back(std::make_shared<Pool>(database, host, user, password, port));
+        replicas_by_priority[0].emplace_back(std::make_shared<Pool>(database,
+            host, user, password, port,
+            /* socket_ = */ "",
+            MYSQLXX_DEFAULT_TIMEOUT,
+            MYSQLXX_DEFAULT_RW_TIMEOUT,
+            default_connections_,
+            max_connections_));
     }
 }
 

base/mysqlxx/PoolWithFailover.h

@@ -115,6 +115,8 @@ namespace mysqlxx
             const RemoteDescription & addresses,
             const std::string & user,
             const std::string & password,
+            unsigned default_connections_ = MYSQLXX_POOL_WITH_FAILOVER_DEFAULT_START_CONNECTIONS,
+            unsigned max_connections_ = MYSQLXX_POOL_WITH_FAILOVER_DEFAULT_MAX_CONNECTIONS,
             size_t max_tries_ = MYSQLXX_POOL_WITH_FAILOVER_DEFAULT_MAX_TRIES);
 
         PoolWithFailover(const PoolWithFailover & other);

cmake/autogenerated_versions.txt

@@ -1,9 +1,9 @@
 # This strings autochanged from release_lib.sh:
-SET(VERSION_REVISION 54451)
+SET(VERSION_REVISION 54452)
 SET(VERSION_MAJOR 21)
-SET(VERSION_MINOR 6)
+SET(VERSION_MINOR 7)
 SET(VERSION_PATCH 1)
-SET(VERSION_GITHASH 96fced4c3cf432fb0b401d2ab01f0c56e5f74a96)
-SET(VERSION_DESCRIBE v21.6.1.1-prestable)
-SET(VERSION_STRING 21.6.1.1)
+SET(VERSION_GITHASH 976ccc2e908ac3bc28f763bfea8134ea0a121b40)
+SET(VERSION_DESCRIBE v21.7.1.1-prestable)
+SET(VERSION_STRING 21.7.1.1)
 # end of autochange

contrib/re2

@@ -1 +1 @@
-Subproject commit 7cf8b88e8f70f97fd4926b56aa87e7f53b2717e0
+Subproject commit 13ebb377c6ad763ca61d12dd6f88b1126bd0b911

contrib/re2_st/re2_transform.cmake

@@ -1,7 +1,7 @@
 file (READ ${SOURCE_FILENAME} CONTENT)
 string (REGEX REPLACE "using re2::RE2;" "" CONTENT "${CONTENT}")
 string (REGEX REPLACE "using re2::LazyRE2;" "" CONTENT "${CONTENT}")
-string (REGEX REPLACE "namespace re2" "namespace re2_st" CONTENT "${CONTENT}")
+string (REGEX REPLACE "namespace re2 {" "namespace re2_st {" CONTENT "${CONTENT}")
 string (REGEX REPLACE "re2::" "re2_st::" CONTENT "${CONTENT}")
 string (REGEX REPLACE "\"re2/" "\"re2_st/" CONTENT "${CONTENT}")
 string (REGEX REPLACE "(.\\*?_H)" "\\1_ST" CONTENT "${CONTENT}")

debian/changelog

@@ -1,5 +1,5 @@
-clickhouse (21.6.1.1) unstable; urgency=low
+clickhouse (21.7.1.1) unstable; urgency=low
 
   * Modified source code
 
- -- clickhouse-release <clickhouse-release@yandex-team.ru>  Tue, 20 Apr 2021 01:48:16 +0300
+ -- clickhouse-release <clickhouse-release@yandex-team.ru>  Thu, 20 May 2021 22:23:29 +0300

docker/client/Dockerfile

@@ -1,7 +1,7 @@
 FROM ubuntu:18.04
 
 ARG repository="deb https://repo.clickhouse.tech/deb/stable/ main/"
-ARG version=21.6.1.*
+ARG version=21.7.1.*
 
 RUN apt-get update \
     && apt-get install --yes --no-install-recommends \

docker/server/Dockerfile

@@ -1,7 +1,7 @@
 FROM ubuntu:20.04
 
 ARG repository="deb https://repo.clickhouse.tech/deb/stable/ main/"
-ARG version=21.6.1.*
+ARG version=21.7.1.*
 ARG gosu_ver=1.10
 
 # set non-empty deb_location_url url to create a docker image

docker/test/Dockerfile

@@ -1,7 +1,7 @@
 FROM ubuntu:18.04
 
 ARG repository="deb https://repo.clickhouse.tech/deb/stable/ main/"
-ARG version=21.6.1.*
+ARG version=21.7.1.*
 
 RUN apt-get update && \
     apt-get install -y apt-transport-https dirmngr && \

docker/test/fasttest/run.sh

@@ -73,7 +73,7 @@ function start_server
         --path "$FASTTEST_DATA"
         --user_files_path "$FASTTEST_DATA/user_files"
         --top_level_domains_path "$FASTTEST_DATA/top_level_domains"
-        --keeper_server.log_storage_path "$FASTTEST_DATA/coordination"
+        --keeper_server.storage_path "$FASTTEST_DATA/coordination"
     )
     clickhouse-server "${opts[@]}" &>> "$FASTTEST_OUTPUT/server.log" &
     server_pid=$!
@@ -376,35 +376,14 @@ function run_tests
         # Depends on LLVM JIT
         01852_jit_if
         01865_jit_comparison_constant_result
+        01871_merge_tree_compile_expressions
     )
 
-    (time clickhouse-test --hung-check -j 8 --order=random --use-skip-list --no-long --testname --shard --zookeeper --skip "${TESTS_TO_SKIP[@]}" -- "$FASTTEST_FOCUS" 2>&1 ||:) | ts '%Y-%m-%d %H:%M:%S' | tee "$FASTTEST_OUTPUT/test_log.txt"
-
-    # substr is to remove semicolon after test name
-    readarray -t FAILED_TESTS < <(awk '/\[ FAIL|TIMEOUT|ERROR \]/ { print substr($3, 1, length($3)-1) }' "$FASTTEST_OUTPUT/test_log.txt" | tee "$FASTTEST_OUTPUT/failed-parallel-tests.txt")
-
-    # We will rerun sequentially any tests that have failed during parallel run.
-    # They might have failed because there was some interference from other tests
-    # running concurrently. If they fail even in seqential mode, we will report them.
-    # FIXME All tests that require exclusive access to the server must be
-    # explicitly marked as `sequential`, and `clickhouse-test` must detect them and
-    # run them in a separate group after all other tests. This is faster and also
-    # explicit instead of guessing.
-    if [[ -n "${FAILED_TESTS[*]}" ]]
-    then
-        stop_server ||:
-
-        # Clean the data so that there is no interference from the previous test run.
-        rm -rf "$FASTTEST_DATA"/{{meta,}data,user_files,coordination} ||:
-
-        start_server
-
-        echo "Going to run again: ${FAILED_TESTS[*]}"
-
-        clickhouse-test --hung-check --order=random --no-long --testname --shard --zookeeper "${FAILED_TESTS[@]}" 2>&1 | ts '%Y-%m-%d %H:%M:%S' | tee -a "$FASTTEST_OUTPUT/test_log.txt"
-    else
-        echo "No failed tests"
-    fi
+    time clickhouse-test --hung-check -j 8 --order=random --use-skip-list \
+            --no-long --testname --shard --zookeeper --skip "${TESTS_TO_SKIP[@]}" \
+            -- "$FASTTEST_FOCUS" 2>&1 \
+        | ts '%Y-%m-%d %H:%M:%S' \
+        | tee "$FASTTEST_OUTPUT/test_log.txt"
 }
 
 case "$stage" in

docker/test/integration/runner/compose/docker_compose_keeper.yml

@@ -0,0 +1,92 @@
+version: '2.3'
+services:
+    zoo1:
+        image: ${image:-yandex/clickhouse-integration-test}
+        restart: always
+        user: ${user:-}
+        volumes:
+            - type: bind
+              source: ${keeper_binary:-}
+              target: /usr/bin/clickhouse
+            - type: bind
+              source: ${keeper_config_dir1:-}
+              target: /etc/clickhouse-keeper
+            - type: bind
+              source: ${keeper_logs_dir1:-}
+              target: /var/log/clickhouse-keeper
+            - type: ${keeper_fs:-tmpfs}
+              source: ${keeper_db_dir1:-}
+              target: /var/lib/clickhouse-keeper
+        entrypoint: "clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config1.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log"
+        cap_add:
+            - SYS_PTRACE
+            - NET_ADMIN
+            - IPC_LOCK
+            - SYS_NICE
+        security_opt:
+            - label:disable
+        dns_opt:
+            - attempts:2
+            - timeout:1
+            - inet6
+            - rotate
+    zoo2:
+        image: ${image:-yandex/clickhouse-integration-test}
+        restart: always
+        user: ${user:-}
+        volumes:
+            - type: bind
+              source: ${keeper_binary:-}
+              target: /usr/bin/clickhouse
+            - type: bind
+              source: ${keeper_config_dir2:-}
+              target: /etc/clickhouse-keeper
+            - type: bind
+              source: ${keeper_logs_dir2:-}
+              target: /var/log/clickhouse-keeper
+            - type: ${keeper_fs:-tmpfs}
+              source: ${keeper_db_dir2:-}
+              target: /var/lib/clickhouse-keeper
+        entrypoint: "clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config2.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log"
+        cap_add:
+            - SYS_PTRACE
+            - NET_ADMIN
+            - IPC_LOCK
+            - SYS_NICE
+        security_opt:
+            - label:disable
+        dns_opt:
+            - attempts:2
+            - timeout:1
+            - inet6
+            - rotate
+    zoo3:
+        image: ${image:-yandex/clickhouse-integration-test}
+        restart: always
+        user: ${user:-}
+        volumes:
+            - type: bind
+              source: ${keeper_binary:-}
+              target: /usr/bin/clickhouse
+            - type: bind
+              source: ${keeper_config_dir3:-}
+              target: /etc/clickhouse-keeper
+            - type: bind
+              source: ${keeper_logs_dir3:-}
+              target: /var/log/clickhouse-keeper
+            - type: ${keeper_fs:-tmpfs}
+              source: ${keeper_db_dir3:-}
+              target: /var/lib/clickhouse-keeper
+        entrypoint: "clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config3.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log"
+        cap_add:
+            - SYS_PTRACE
+            - NET_ADMIN
+            - IPC_LOCK
+            - SYS_NICE
+        security_opt:
+            - label:disable
+        dns_opt:
+            - attempts:2
+            - timeout:1
+            - inet6
+            - rotate

docs/en/engines/table-engines/integrations/mysql.md

@@ -15,7 +15,12 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster]
     name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1] [TTL expr1],
     name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2] [TTL expr2],
     ...
-) ENGINE = MySQL('host:port', 'database', 'table', 'user', 'password'[, replace_query, 'on_duplicate_clause']);
+) ENGINE = MySQL('host:port', 'database', 'table', 'user', 'password'[, replace_query, 'on_duplicate_clause'])
+SETTINGS
+    [connection_pool_size=16, ]
+    [connection_max_tries=3, ]
+    [connection_auto_close=true ]
+;
 ```
 
 See a detailed description of the [CREATE TABLE](../../../sql-reference/statements/create/table.md#create-table-query) query.

docs/en/operations/external-authenticators/ldap.md

@@ -17,6 +17,7 @@ To define LDAP server you must add `ldap_servers` section to the `config.xml`.
 <yandex>
     <!- ... -->
     <ldap_servers>
+        <!- Typical LDAP server. -->
         <my_ldap_server>
             <host>localhost</host>
             <port>636</port>
@@ -31,6 +32,18 @@ To define LDAP server you must add `ldap_servers` section to the `config.xml`.
             <tls_ca_cert_dir>/path/to/tls_ca_cert_dir</tls_ca_cert_dir>
             <tls_cipher_suite>ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:AES256-GCM-SHA384</tls_cipher_suite>
         </my_ldap_server>
+
+        <!- Typical Active Directory with configured user DN detection for further role mapping. -->
+        <my_ad_server>
+            <host>localhost</host>
+            <port>389</port>
+            <bind_dn>EXAMPLE\{user_name}</bind_dn>
+            <user_dn_detection>
+                <base_dn>CN=Users,DC=example,DC=com</base_dn>
+                <search_filter>(&amp;(objectClass=user)(sAMAccountName={user_name}))</search_filter>
+            </user_dn_detection>
+            <enable_tls>no</enable_tls>
+        </my_ad_server>
     </ldap_servers>
 </yandex>
 ```
@@ -43,6 +56,15 @@ Note, that you can define multiple LDAP servers inside the `ldap_servers` sectio
 - `port` — LDAP server port, default is `636` if `enable_tls` is set to `true`, `389` otherwise.
 - `bind_dn` — Template used to construct the DN to bind to.
     - The resulting DN will be constructed by replacing all `{user_name}` substrings of the template with the actual user name during each authentication attempt.
+- `user_dn_detection` - Section with LDAP search parameters for detecting the actual user DN of the bound user.
+    - This is mainly used in search filters for further role mapping when the server is Active Directory. The resulting user DN will be used when replacing `{user_dn}` substrings wherever they are allowed. By default, user DN is set equal to bind DN, but once search is performed, it will be updated with to the actual detected user DN value.
+        - `base_dn` - Template used to construct the base DN for the LDAP search.
+            - The resulting DN will be constructed by replacing all `{user_name}` and `{bind_dn}` substrings of the template with the actual user name and bind DN during the LDAP search.
+        - `scope` - Scope of the LDAP search.
+            - Accepted values are: `base`, `one_level`, `children`, `subtree` (the default).
+        - `search_filter` - Template used to construct the search filter for the LDAP search.
+            - The resulting filter will be constructed by replacing all `{user_name}`, `{bind_dn}`, and `{base_dn}` substrings of the template with the actual user name, bind DN, and base DN during the LDAP search.
+            - Note, that the special characters must be escaped properly in XML.
 - `verification_cooldown` — A period of time, in seconds, after a successful bind attempt, during which the user will be assumed to be successfully authenticated for all consecutive requests without contacting the LDAP server.
     - Specify `0` (the default) to disable caching and force contacting the LDAP server for each authentication request.
 - `enable_tls` — A flag to trigger the use of the secure connection to the LDAP server.
@@ -107,7 +129,7 @@ Goes into `config.xml`.
 <yandex>
     <!- ... -->
     <user_directories>
-        <!- ... -->
+        <!- Typical LDAP server. -->
         <ldap>
             <server>my_ldap_server</server>
             <roles>
@@ -122,6 +144,18 @@ Goes into `config.xml`.
                 <prefix>clickhouse_</prefix>
             </role_mapping>
         </ldap>
+
+        <!- Typical Active Directory with role mapping that relies on the detected user DN. -->
+        <ldap>
+            <server>my_ad_server</server>
+            <role_mapping>
+                <base_dn>CN=Users,DC=example,DC=com</base_dn>
+                <attribute>CN</attribute>
+                <scope>subtree</scope>
+                <search_filter>(&amp;(objectClass=group)(member={user_dn}))</search_filter>
+                <prefix>clickhouse_</prefix>
+            </role_mapping>
+        </ldap>
     </user_directories>
 </yandex>
 ```
@@ -137,13 +171,13 @@ Note that `my_ldap_server` referred in the `ldap` section inside the `user_direc
     - When a user authenticates, while still bound to LDAP, an LDAP search is performed using `search_filter` and the name of the logged-in user. For each entry found during that search, the value of the specified attribute is extracted. For each attribute value that has the specified prefix, the prefix is removed, and the rest of the value becomes the name of a local role defined in ClickHouse, which is expected to be created beforehand by the [CREATE ROLE](../../sql-reference/statements/create/role.md#create-role-statement) statement.
     - There can be multiple `role_mapping` sections defined inside the same `ldap` section. All of them will be applied.
         - `base_dn` — Template used to construct the base DN for the LDAP search.
-           - The resulting DN will be constructed by replacing all `{user_name}` and `{bind_dn}` substrings of the template with the actual user name and bind DN during each LDAP search.
+            - The resulting DN will be constructed by replacing all `{user_name}`, `{bind_dn}`, and `{user_dn}` substrings of the template with the actual user name, bind DN, and user DN during each LDAP search.
         - `scope` — Scope of the LDAP search.
             - Accepted values are: `base`, `one_level`, `children`, `subtree` (the default).
         - `search_filter` — Template used to construct the search filter for the LDAP search.
-            - The resulting filter will be constructed by replacing all `{user_name}`, `{bind_dn}` and `{base_dn}` substrings of the template with the actual user name, bind DN and base DN during each LDAP search.
+            - The resulting filter will be constructed by replacing all `{user_name}`, `{bind_dn}`, `{user_dn}`, and `{base_dn}` substrings of the template with the actual user name, bind DN, user DN, and base DN during each LDAP search.
             - Note, that the special characters must be escaped properly in XML.
-        - `attribute` — Attribute name whose values will be returned by the LDAP search.
+        - `attribute` — Attribute name whose values will be returned by the LDAP search. `cn`, by default.
         - `prefix` — Prefix, that will be expected to be in front of each string in the original list of strings returned by the LDAP search. The prefix will be removed from the original strings and the resulting strings will be treated as local role names. Empty by default.
 
 [Original article](https://clickhouse.tech/docs/en/operations/external-authenticators/ldap/) <!--hide-->

docs/en/operations/settings/settings.md

@@ -1520,8 +1520,8 @@ Do not merge aggregation states from different servers for distributed query pro
 Possible values:
 
 -   0 — Disabled (final query processing is done on the initiator node).
--   1 - Do not merge aggregation states from different servers for distributed query processing (query completelly processed on the shard, initiator only proxy the data).
--   2 - Same as 1 but apply `ORDER BY` and `LIMIT` on the initiator (can be used for queries with `ORDER BY` and/or `LIMIT`).
+-   1 - Do not merge aggregation states from different servers for distributed query processing (query completelly processed on the shard, initiator only proxy the data), can be used in case it is for certain that there are different keys on different shards.
+-   2 - Same as `1` but applies `ORDER BY` and `LIMIT` (it is not possilbe when the query processed completelly on the remote node, like for `distributed_group_by_no_merge=1`) on the initiator (can be used for queries with `ORDER BY` and/or `LIMIT`).
 
 **Example**
 

docs/ru/engines/table-engines/integrations/hdfs.md

@@ -183,7 +183,7 @@ CREATE TABLE big_table (name String, value UInt32) ENGINE = HDFS('hdfs://hdfs1:9
 #### Ограничения {#limitations}
   * hadoop\_security\_kerberos\_ticket\_cache\_path могут быть определены только на глобальном уровне
 
-## Поддержика Kerberos {#kerberos-support}
+## Поддержка Kerberos {#kerberos-support}
 
 Если hadoop\_security\_authentication параметр имеет значение 'kerberos', ClickHouse аутентифицируется с помощью Kerberos.
 [Расширенные параметры](#clickhouse-extras) и hadoop\_security\_kerberos\_ticket\_cache\_path помогают сделать это.

docs/tools/blog.py

@@ -40,7 +40,7 @@ def build_for_lang(lang, args):
 
         site_names = {
             'en': 'ClickHouse Blog',
-            'ru': 'Блог ClickHouse '
+            'ru': 'Блог ClickHouse'
         }
 
         assert len(site_names) == len(languages)
@@ -62,7 +62,7 @@ def build_for_lang(lang, args):
             strict=True,
             theme=theme_cfg,
             nav=blog_nav,
-            copyright='©2016–2020 Yandex LLC',
+            copyright='©2016–2021 Yandex LLC',
             use_directory_urls=True,
             repo_name='ClickHouse/ClickHouse',
             repo_url='https://github.com/ClickHouse/ClickHouse/',

docs/tools/build.py

@@ -94,7 +94,7 @@ def build_for_lang(lang, args):
             site_dir=site_dir,
             strict=True,
             theme=theme_cfg,
-            copyright='©2016–2020 Yandex LLC',
+            copyright='©2016–2021 Yandex LLC',
             use_directory_urls=True,
             repo_name='ClickHouse/ClickHouse',
             repo_url='https://github.com/ClickHouse/ClickHouse/',

docs/tools/nav.py

@@ -31,7 +31,16 @@ def build_nav_entry(root, args):
                 result_items.append((prio, title, payload))
         elif filename.endswith('.md'):
             path = os.path.join(root, filename)
-            meta, content = util.read_md_file(path)
+
+            meta = ''
+            content = ''
+
+            try:
+                meta, content = util.read_md_file(path)
+            except:
+                print('Error in file: {}'.format(path))
+                raise
+
             path = path.split('/', 2)[-1]
             title = meta.get('toc_title', find_first_header(content))
             if title:

docs/tools/website.py

@@ -155,10 +155,6 @@ def build_website(args):
         os.path.join(args.src_dir, 'utils', 'list-versions', 'version_date.tsv'),
         os.path.join(args.output_dir, 'data', 'version_date.tsv'))
 
-    shutil.copy2(
-        os.path.join(args.website_dir, 'js', 'embedd.min.js'),
-        os.path.join(args.output_dir, 'js', 'embedd.min.js'))
-
     for root, _, filenames in os.walk(args.output_dir):
         for filename in filenames:
             if filename == 'main.html':

docs/zh/operations/backup.md

@@ -7,37 +7,37 @@ toc_title: "\u6570\u636E\u5907\u4EFD"
 
 # 数据备份 {#data-backup}
 
-尽管[副本](../engines/table-engines/mergetree-family/replication.md) 可以预防硬件错误带来的数据丢失, 但是它不能防止人为操作的错误: 意外删除数据, 删除错误的 table 或者删除错误 cluster 上的 table, 可以导致错误数据处理错误或者数据损坏的 bugs. 这类意外可能会影响所有的副本. ClickHouse 有内建的保障措施可以预防一些错误 — 例如, 默认情况下[您不能使用类似MergeTree的引擎删除包含超过50Gb数据的表](server-configuration-parameters/settings.md#max-table-size-to-drop). 但是，这些保障措施不能涵盖所有可能的情况，并且可以规避。
+尽管 [副本] (../engines/table-engines/mergetree-family/replication.md) 可以提供针对硬件的错误防护, 但是它不能预防人为操作失误: 数据的意外删除, 错误表的删除或者错误集群上表的删除, 以及导致错误数据处理或者数据损坏的软件bug. 在很多案例中，这类意外可能会影响所有的副本. ClickHouse 有内置的保护措施可以预防一些错误 — 例如, 默认情况下 [不能人工删除使用带有MergeTree引擎且包含超过50Gb数据的表] (server-configuration-parameters/settings.md#max-table-size-to-drop). 但是，这些保护措施不能覆盖所有可能情况，并且这些措施可以被绕过。
 
-为了有效地减少可能的人为错误，您应该 **提前**准备备份和还原数据的策略.
+为了有效地减少可能的人为错误，您应该 **提前** 仔细的准备备份和数据还原的策略.
 
-不同公司有不同的可用资源和业务需求，因此没有适合各种情况的ClickHouse备份和恢复通用解决方案。 适用于 1GB 的数据的方案可能并不适用于几十 PB 数据的情况。 有多种可能的并有自己优缺点的方法，这将在下面讨论。 好的主意是同时结合使用多种方法而不是仅使用一种，这样可以弥补不同方法各自的缺点。
+不同公司有不同的可用资源和业务需求，因此不存在一个通用的解决方案可以应对各种情况下的ClickHouse备份和恢复。 适用于 1GB 数据的方案可能并不适用于几十 PB 数据的情况。 有多种具备各自优缺点的可能方法，将在下面对其进行讨论。最好使用几种方法而不是仅仅使用一种方法来弥补它们的各种缺点。。
 
 !!! note "注"
-    请记住，如果您备份了某些内容并且从未尝试过还原它，那么当您实际需要它时（或者至少需要比业务能够容忍的时间更长），恢复可能无法正常工作。 因此，无论您选择哪种备份方法，请确保自动还原过程，并定期在备用ClickHouse群集上练习。
+    需要注意的是，如果您备份了某些内容并且从未尝试过还原它，那么当您实际需要它时可能无法正常恢复（或者至少需要的时间比业务能够容忍的时间更长）。 因此，无论您选择哪种备份方法，请确保自动还原过程，并定期在备用ClickHouse群集上演练。
 
-## 将源数据复制到其他地方 {#duplicating-source-data-somewhere-else}
+## 将源数据复制到其它地方 {#duplicating-source-data-somewhere-else}
 
-通常被聚集到ClickHouse的数据是通过某种持久队列传递的，例如 [Apache Kafka](https://kafka.apache.org). 在这种情况下，可以配置一组额外的订阅服务器，这些订阅服务器将在写入ClickHouse时读取相同的数据流，并将其存储在冷存储中。 大多数公司已经有一些默认的推荐冷存储，可能是对象存储或分布式文件系统，如 [HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html).
+通常摄入到ClickHouse的数据是通过某种持久队列传递的，例如 [Apache Kafka] (https://kafka.apache.org). 在这种情况下，可以配置一组额外的订阅服务器，这些订阅服务器将在写入ClickHouse时读取相同的数据流，并将其存储在冷存储中。 大多数公司已经有一些默认推荐的冷存储，可能是对象存储或分布式文件系统，如 [HDFS] (https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html).
 
 ## 文件系统快照 {#filesystem-snapshots}
 
-某些本地文件系统提供快照功能（例如, [ZFS](https://en.wikipedia.org/wiki/ZFS)），但它们可能不是提供实时查询的最佳选择。 一个可能的解决方案是使用这种文件系统创建额外的副本，并将它们从 [分布](../engines/table-engines/special/distributed.md) 用于以下目的的表 `SELECT` 查询。 任何修改数据的查询都无法访问此类副本上的快照。 作为奖励，这些副本可能具有特殊的硬件配置，每个服务器附加更多的磁盘，这将是经济高效的。
+某些本地文件系统提供快照功能（例如, [ZFS] (https://en.wikipedia.org/wiki/ZFS)），但它们可能不是提供实时查询的最佳选择。 一个可能的解决方案是使用这种文件系统创建额外的副本，并将它们与用于`SELECT` 查询的 [分布式] (../engines/table-engines/special/distributed.md) 表分离。 任何修改数据的查询都无法访问此类副本上的快照。 作为回报，这些副本可能具有特殊的硬件配置，每个服务器附加更多的磁盘，这将是经济高效的。
 
 ## clickhouse-copier {#clickhouse-copier}
 
-[clickhouse-copier](utilities/clickhouse-copier.md) 是一个多功能工具，最初创建用于重新分片pb大小的表。 因为它可以在ClickHouse表和集群之间可靠地复制数据，所以它还可用于备份和还原数据。
+[clickhouse-copier] (utilities/clickhouse-copier.md) 是一个多功能工具，最初创建它是为了用于重新切分pb大小的表。 因为它能够在ClickHouse表和集群之间可靠地复制数据，所以它也可用于备份和还原数据。
 
 对于较小的数据量，一个简单的 `INSERT INTO ... SELECT ...` 到远程表也可以工作。
 
-## 部件操作 {#manipulations-with-parts}
+## part操作 {#manipulations-with-parts}
 
-ClickHouse允许使用 `ALTER TABLE ... FREEZE PARTITION ...` 查询以创建表分区的本地副本。 这是利用硬链接(hardlink)到 `/var/lib/clickhouse/shadow/` 文件夹中实现的，所以它通常不会占用旧数据的额外磁盘空间。 创建的文件副本不由ClickHouse服务器处理，所以你可以把它们留在那里：你将有一个简单的备份，不需要任何额外的外部系统，但它仍然会容易出现硬件问题。 出于这个原因，最好将它们远程复制到另一个位置，然后删除本地副本。 分布式文件系统和对象存储仍然是一个不错的选择，但是具有足够大容量的正常附加文件服务器也可以工作（在这种情况下，传输将通过网络文件系统 [rsync](https://en.wikipedia.org/wiki/Rsync)).
+ClickHouse允许使用 `ALTER TABLE ... FREEZE PARTITION ...` 查询以创建表分区的本地副本。 这是利用硬链接(hardlink)到 `/var/lib/clickhouse/shadow/` 文件夹中实现的，所以它通常不会因为旧数据而占用额外的磁盘空间。 创建的文件副本不由ClickHouse服务器处理，所以你可以把它们留在那里：你将有一个简单的备份，不需要任何额外的外部系统，但它仍然容易出现硬件问题。 出于这个原因，最好将它们远程复制到另一个位置，然后删除本地副本。 分布式文件系统和对象存储仍然是一个不错的选择，但是具有足够大容量的正常附加文件服务器也可以工作（在这种情况下，传输将通过网络文件系统或者也许是 [rsync] (https://en.wikipedia.org/wiki/Rsync) 来进行).
 
 数据可以使用 `ALTER TABLE ... ATTACH PARTITION ...` 从备份中恢复。
 
-有关与分区操作相关的查询的详细信息，请参阅 [更改文档](../sql-reference/statements/alter.md#alter_manipulations-with-partitions).
+有关与分区操作相关的查询的详细信息，请参阅 [更改文档] (../sql-reference/statements/alter.md#alter_manipulations-with-partitions).
 
-第三方工具可用于自动化此方法: [clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup).
+第三方工具可用于自动化此方法: [clickhouse-backup] (https://github.com/AlexAkulov/clickhouse-backup).
 
-[原始文章](https://clickhouse.tech/docs/en/operations/backup/) <!--hide-->
+[原始文章] (https://clickhouse.tech/docs/en/operations/backup/) <!--hide-->

docs/zh/sql-reference/data-types/special-data-types/interval.md

@@ -5,9 +5,9 @@ toc_priority: 61
 toc_title: "\u95F4\u9694"
 ---
 
-# 间隔 {#data-type-interval}
+# Interval类型 {#data-type-interval}
 
-表示时间和日期间隔的数据类型族。 由此产生的类型 [INTERVAL](../../../sql-reference/operators/index.md#operator-interval) 接线员
+表示时间和日期间隔的数据类型家族。  [INTERVAL](../../../sql-reference/operators/index.md#operator-interval) 运算的结果类型。
 
 !!! warning "警告"
     `Interval` 数据类型值不能存储在表中。
@@ -15,7 +15,7 @@ toc_title: "\u95F4\u9694"
 结构:
 
 -   时间间隔作为无符号整数值。
--   间隔的类型。
+-   时间间隔的类型。
 
 支持的时间间隔类型:
 
@@ -28,7 +28,7 @@ toc_title: "\u95F4\u9694"
 -   `QUARTER`
 -   `YEAR`
 
-对于每个间隔类型，都有一个单独的数据类型。 例如， `DAY` 间隔对应于 `IntervalDay` 数据类型:
+对于每个时间间隔类型，都有一个单独的数据类型。 例如， `DAY` 间隔对应于 `IntervalDay` 数据类型:
 
 ``` sql
 SELECT toTypeName(INTERVAL 4 DAY)
@@ -42,7 +42,7 @@ SELECT toTypeName(INTERVAL 4 DAY)
 
 ## 使用说明 {#data-type-interval-usage-remarks}
 
-您可以使用 `Interval`-在算术运算类型值 [日期](../../../sql-reference/data-types/date.md) 和 [日期时间](../../../sql-reference/data-types/datetime.md)-类型值。 例如，您可以将4天添加到当前时间:
+您可以在与 [日期](../../../sql-reference/data-types/date.md) 和 [日期时间](../../../sql-reference/data-types/datetime.md) 类型值的算术运算中使用 `Interval` 类型值。 例如，您可以将4天添加到当前时间:
 
 ``` sql
 SELECT now() as current_date_time, current_date_time + INTERVAL 4 DAY
@@ -54,10 +54,10 @@ SELECT now() as current_date_time, current_date_time + INTERVAL 4 DAY
 └─────────────────────┴───────────────────────────────┘
 ```
 
-不同类型的间隔不能合并。 你不能使用间隔，如 `4 DAY 1 HOUR`. 以小于或等于间隔的最小单位的单位指定间隔，例如，间隔 `1 day and an hour` 间隔可以表示为 `25 HOUR` 或 `90000 SECOND`.
-
-你不能执行算术运算 `Interval`-类型值，但你可以添加不同类型的时间间隔，因此值 `Date` 或 `DateTime` 数据类型。 例如:
+不同类型的间隔不能合并。 你不能使用诸如 `4 DAY 1 HOUR` 的时间间隔. 以小于或等于时间间隔最小单位的单位来指定间隔，例如，时间间隔 `1 day and an hour` 可以表示为 `25 HOUR` 或 `90000 SECOND`.
 
+你不能对 `Interval` 类型的值执行算术运算，但你可以向 `Date` 或 `DateTime` 数据类型的值添加不同类型的时间间隔，例如:
+ 
 ``` sql
 SELECT now() AS current_date_time, current_date_time + INTERVAL 4 DAY + INTERVAL 3 HOUR
 ```
@@ -81,5 +81,5 @@ Code: 43. DB::Exception: Received from localhost:9000. DB::Exception: Wrong argu
 
 ## 另请参阅 {#see-also}
 
--   [INTERVAL](../../../sql-reference/operators/index.md#operator-interval) 接线员
+-   [INTERVAL](../../../sql-reference/operators/index.md#operator-interval) 操作
 -   [toInterval](../../../sql-reference/functions/type-conversion-functions.md#function-tointerval) 类型转换函数

docs/zh/sql-reference/syntax.md

@@ -14,7 +14,7 @@ INSERT INTO t VALUES (1, 'Hello, world'), (2, 'abc'), (3, 'def')
 
 含`INSERT INTO t VALUES` 的部分由完整SQL解析器处理，包含数据的部分 `(1, 'Hello, world'), (2, 'abc'), (3, 'def')` 交给快速流式解析器解析。通过设置参数 [input_format_values_interpret_expressions](../operations/settings/settings.md#settings-input_format_values_interpret_expressions)，你也可以对数据部分开启完整SQL解析器。当 `input_format_values_interpret_expressions = 1` 时，CH优先采用快速流式解析器来解析数据。如果失败，CH再尝试用完整SQL解析器来处理，就像处理SQL [expression](#syntax-expressions) 一样。
 
-数据可以采用任何格式。当CH接受到请求时，服务端先在内存中计算不超过 [max_query_size](../operations/settings/settings.md#settings-max_query_size) 字节的请求数据（默认1 mb），然后剩下部分交给快速流式解析器。
+数据可以采用任何格式。当CH接收到请求时，服务端先在内存中计算不超过 [max_query_size](../operations/settings/settings.md#settings-max_query_size) 字节的请求数据（默认1 mb），然后剩下部分交给快速流式解析器。
 
 这将避免在处理大型的 `INSERT`语句时出现问题。
 

programs/CMakeLists.txt

@@ -47,6 +47,9 @@ option (ENABLE_CLICKHOUSE_LIBRARY_BRIDGE "HTTP-server working like a proxy to Li
 option (ENABLE_CLICKHOUSE_GIT_IMPORT "A tool to analyze Git repositories"
     ${ENABLE_CLICKHOUSE_ALL})
 
+
+option (ENABLE_CLICKHOUSE_KEEPER "ClickHouse alternative to ZooKeeper" ${ENABLE_CLICKHOUSE_ALL})
+
 if (CLICKHOUSE_SPLIT_BINARY)
     option(ENABLE_CLICKHOUSE_INSTALL "Install ClickHouse without .deb/.rpm/.tgz packages (having the binary only)" OFF)
 else ()
@@ -134,6 +137,12 @@ else()
     message(STATUS "ClickHouse git-import: OFF")
 endif()
 
+if (ENABLE_CLICKHOUSE_KEEPER)
+    message(STATUS "ClickHouse keeper mode: ON")
+else()
+    message(STATUS "ClickHouse keeper mode: OFF")
+endif()
+
 if(NOT (MAKE_STATIC_LIBRARIES OR SPLIT_SHARED_LIBRARIES))
     set(CLICKHOUSE_ONE_SHARED ON)
 endif()
@@ -189,6 +198,54 @@ macro(clickhouse_program_add name)
     clickhouse_program_add_executable(${name})
 endmacro()
 
+# Embed default config files as a resource into the binary.
+# This is needed for two purposes:
+# 1. Allow to run the binary without download of any other files.
+# 2. Allow to implement "sudo clickhouse install" tool.
+#
+# Arguments: target (server, client, keeper, etc.) and list of files
+#
+# Also dependency on TARGET_FILE is required, look at examples in programs/server and programs/keeper
+macro(clickhouse_embed_binaries)
+    # TODO We actually need this on Mac, FreeBSD.
+    if (OS_LINUX)
+
+        set(arguments_list "${ARGN}")
+        list(GET arguments_list 0 target)
+
+        # for some reason cmake iterates loop including <stop>
+        math(EXPR arguments_count "${ARGC}-1")
+
+        foreach(RESOURCE_POS RANGE 1 "${arguments_count}")
+            list(GET arguments_list "${RESOURCE_POS}" RESOURCE_FILE)
+            set(RESOURCE_OBJ ${RESOURCE_FILE}.o)
+            set(RESOURCE_OBJS ${RESOURCE_OBJS} ${RESOURCE_OBJ})
+
+            # https://stackoverflow.com/questions/14776463/compile-and-add-an-object-file-from-a-binary-with-cmake
+            # PPC64LE fails to do this with objcopy, use ld or lld instead
+            if (ARCH_PPC64LE)
+                add_custom_command(OUTPUT ${RESOURCE_OBJ}
+                    COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && ${CMAKE_LINKER} -m elf64lppc -r -b binary -o "${CMAKE_CURRENT_BINARY_DIR}/${RESOURCE_OBJ}" ${RESOURCE_FILE})
+            else()
+                add_custom_command(OUTPUT ${RESOURCE_OBJ}
+                    COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && ${OBJCOPY_PATH} -I binary ${OBJCOPY_ARCH_OPTIONS} ${RESOURCE_FILE} "${CMAKE_CURRENT_BINARY_DIR}/${RESOURCE_OBJ}"
+                    COMMAND ${OBJCOPY_PATH} --rename-section .data=.rodata,alloc,load,readonly,data,contents
+                        "${CMAKE_CURRENT_BINARY_DIR}/${RESOURCE_OBJ}" "${CMAKE_CURRENT_BINARY_DIR}/${RESOURCE_OBJ}")
+            endif()
+            set_source_files_properties(${RESOURCE_OBJ} PROPERTIES EXTERNAL_OBJECT true GENERATED true)
+        endforeach()
+
+        add_library(clickhouse_${target}_configs STATIC ${RESOURCE_OBJS})
+        set_target_properties(clickhouse_${target}_configs PROPERTIES LINKER_LANGUAGE C)
+
+        # whole-archive prevents symbols from being discarded for unknown reason
+        # CMake can shuffle each of target_link_libraries arguments with other
+        # libraries in linker command. To avoid this we hardcode whole-archive
+        # library into single string.
+        add_dependencies(clickhouse-${target}-lib clickhouse_${target}_configs)
+    endif ()
+endmacro()
+
 
 add_subdirectory (server)
 add_subdirectory (client)
@@ -202,6 +259,7 @@ add_subdirectory (obfuscator)
 add_subdirectory (install)
 add_subdirectory (git-import)
 add_subdirectory (bash-completion)
+add_subdirectory (keeper)
 
 if (ENABLE_CLICKHOUSE_ODBC_BRIDGE)
     add_subdirectory (odbc-bridge)
@@ -212,15 +270,15 @@ if (ENABLE_CLICKHOUSE_LIBRARY_BRIDGE)
 endif ()
 
 if (CLICKHOUSE_ONE_SHARED)
-    add_library(clickhouse-lib SHARED ${CLICKHOUSE_SERVER_SOURCES} ${CLICKHOUSE_CLIENT_SOURCES} ${CLICKHOUSE_LOCAL_SOURCES} ${CLICKHOUSE_BENCHMARK_SOURCES} ${CLICKHOUSE_COPIER_SOURCES} ${CLICKHOUSE_EXTRACT_FROM_CONFIG_SOURCES} ${CLICKHOUSE_COMPRESSOR_SOURCES} ${CLICKHOUSE_FORMAT_SOURCES} ${CLICKHOUSE_OBFUSCATOR_SOURCES} ${CLICKHOUSE_GIT_IMPORT_SOURCES} ${CLICKHOUSE_ODBC_BRIDGE_SOURCES})
-    target_link_libraries(clickhouse-lib ${CLICKHOUSE_SERVER_LINK} ${CLICKHOUSE_CLIENT_LINK} ${CLICKHOUSE_LOCAL_LINK} ${CLICKHOUSE_BENCHMARK_LINK} ${CLICKHOUSE_COPIER_LINK} ${CLICKHOUSE_EXTRACT_FROM_CONFIG_LINK} ${CLICKHOUSE_COMPRESSOR_LINK} ${CLICKHOUSE_FORMAT_LINK} ${CLICKHOUSE_OBFUSCATOR_LINK} ${CLICKHOUSE_GIT_IMPORT_LINK} ${CLICKHOUSE_ODBC_BRIDGE_LINK})
-    target_include_directories(clickhouse-lib ${CLICKHOUSE_SERVER_INCLUDE} ${CLICKHOUSE_CLIENT_INCLUDE} ${CLICKHOUSE_LOCAL_INCLUDE} ${CLICKHOUSE_BENCHMARK_INCLUDE} ${CLICKHOUSE_COPIER_INCLUDE} ${CLICKHOUSE_EXTRACT_FROM_CONFIG_INCLUDE} ${CLICKHOUSE_COMPRESSOR_INCLUDE} ${CLICKHOUSE_FORMAT_INCLUDE} ${CLICKHOUSE_OBFUSCATOR_INCLUDE} ${CLICKHOUSE_GIT_IMPORT_INCLUDE} ${CLICKHOUSE_ODBC_BRIDGE_INCLUDE})
+    add_library(clickhouse-lib SHARED ${CLICKHOUSE_SERVER_SOURCES} ${CLICKHOUSE_CLIENT_SOURCES} ${CLICKHOUSE_LOCAL_SOURCES} ${CLICKHOUSE_BENCHMARK_SOURCES} ${CLICKHOUSE_COPIER_SOURCES} ${CLICKHOUSE_EXTRACT_FROM_CONFIG_SOURCES} ${CLICKHOUSE_COMPRESSOR_SOURCES} ${CLICKHOUSE_FORMAT_SOURCES} ${CLICKHOUSE_OBFUSCATOR_SOURCES} ${CLICKHOUSE_GIT_IMPORT_SOURCES} ${CLICKHOUSE_ODBC_BRIDGE_SOURCES} ${CLICKHOUSE_KEEPER_SOURCES})
+    target_link_libraries(clickhouse-lib ${CLICKHOUSE_SERVER_LINK} ${CLICKHOUSE_CLIENT_LINK} ${CLICKHOUSE_LOCAL_LINK} ${CLICKHOUSE_BENCHMARK_LINK} ${CLICKHOUSE_COPIER_LINK} ${CLICKHOUSE_EXTRACT_FROM_CONFIG_LINK} ${CLICKHOUSE_COMPRESSOR_LINK} ${CLICKHOUSE_FORMAT_LINK} ${CLICKHOUSE_OBFUSCATOR_LINK} ${CLICKHOUSE_GIT_IMPORT_LINK} ${CLICKHOUSE_ODBC_BRIDGE_LINK} ${CLICKHOUSE_KEEPER_LINK})
+    target_include_directories(clickhouse-lib ${CLICKHOUSE_SERVER_INCLUDE} ${CLICKHOUSE_CLIENT_INCLUDE} ${CLICKHOUSE_LOCAL_INCLUDE} ${CLICKHOUSE_BENCHMARK_INCLUDE} ${CLICKHOUSE_COPIER_INCLUDE} ${CLICKHOUSE_EXTRACT_FROM_CONFIG_INCLUDE} ${CLICKHOUSE_COMPRESSOR_INCLUDE} ${CLICKHOUSE_FORMAT_INCLUDE} ${CLICKHOUSE_OBFUSCATOR_INCLUDE} ${CLICKHOUSE_GIT_IMPORT_INCLUDE} ${CLICKHOUSE_ODBC_BRIDGE_INCLUDE} ${CLICKHOUSE_KEEPER_INCLUDE})
     set_target_properties(clickhouse-lib PROPERTIES SOVERSION ${VERSION_MAJOR}.${VERSION_MINOR} VERSION ${VERSION_SO} OUTPUT_NAME clickhouse DEBUG_POSTFIX "")
     install (TARGETS clickhouse-lib LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} COMPONENT clickhouse)
 endif()
 
 if (CLICKHOUSE_SPLIT_BINARY)
-    set (CLICKHOUSE_ALL_TARGETS clickhouse-server clickhouse-client clickhouse-local clickhouse-benchmark clickhouse-extract-from-config clickhouse-compressor clickhouse-format clickhouse-obfuscator clickhouse-git-import clickhouse-copier)
+    set (CLICKHOUSE_ALL_TARGETS clickhouse-server clickhouse-client clickhouse-local clickhouse-benchmark clickhouse-extract-from-config clickhouse-compressor clickhouse-format clickhouse-obfuscator clickhouse-git-import clickhouse-copier clickhouse-keeper)
 
     if (ENABLE_CLICKHOUSE_ODBC_BRIDGE)
         list (APPEND CLICKHOUSE_ALL_TARGETS clickhouse-odbc-bridge)
@@ -277,6 +335,9 @@ else ()
     if (ENABLE_CLICKHOUSE_GIT_IMPORT)
         clickhouse_target_link_split_lib(clickhouse git-import)
     endif ()
+    if (ENABLE_CLICKHOUSE_KEEPER)
+        clickhouse_target_link_split_lib(clickhouse keeper)
+    endif()
     if (ENABLE_CLICKHOUSE_INSTALL)
         clickhouse_target_link_split_lib(clickhouse install)
     endif ()
@@ -332,6 +393,11 @@ else ()
         install (FILES "${CMAKE_CURRENT_BINARY_DIR}/clickhouse-git-import" DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse)
         list(APPEND CLICKHOUSE_BUNDLE clickhouse-git-import)
     endif ()
+    if (ENABLE_CLICKHOUSE_KEEPER)
+        add_custom_target (clickhouse-keeper ALL COMMAND ${CMAKE_COMMAND} -E create_symlink clickhouse clickhouse-keeper DEPENDS clickhouse)
+        install (FILES "${CMAKE_CURRENT_BINARY_DIR}/clickhouse-keeper" DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse)
+        list(APPEND CLICKHOUSE_BUNDLE clickhouse-keeper)
+    endif ()
 
     install (TARGETS clickhouse RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse)
 

programs/client/Client.cpp

@@ -1366,6 +1366,27 @@ private:
             {
                 const auto * exception = server_exception ? server_exception.get() : client_exception.get();
                 fmt::print(stderr, "Error on processing query '{}': {}\n", ast_to_process->formatForErrorMessage(), exception->message());
+
+                // Try to reconnect after errors, for two reasons:
+                // 1. We might not have realized that the server died, e.g. if
+                //    it sent us a <Fatal> trace and closed connection properly.
+                // 2. The connection might have gotten into a wrong state and
+                //    the next query will get false positive about
+                //    "Unknown packet from server".
+                try
+                {
+                    connection->forceConnected(connection_parameters.timeouts);
+                }
+                catch (...)
+                {
+                    // Just report it, we'll terminate below.
+                    fmt::print(stderr,
+                        "Error while reconnecting to the server: Code: {}: {}\n",
+                        getCurrentExceptionCode(),
+                        getCurrentExceptionMessage(true));
+
+                    assert(!connection->isConnected());
+                }
             }
 
             if (!connection->isConnected())
@@ -1469,11 +1490,6 @@ private:
                 server_exception.reset();
                 client_exception.reset();
                 have_error = false;
-
-                // We have to reinitialize connection after errors, because it
-                // might have gotten into a wrong state and we'll get false
-                // positives about "Unknown packet from server".
-                connection->forceConnected(connection_parameters.timeouts);
             }
             else if (ast_to_process->formatForErrorMessage().size() > 500)
             {

programs/config_tools.h.in

@@ -16,3 +16,4 @@
 #cmakedefine01 ENABLE_CLICKHOUSE_INSTALL
 #cmakedefine01 ENABLE_CLICKHOUSE_ODBC_BRIDGE
 #cmakedefine01 ENABLE_CLICKHOUSE_LIBRARY_BRIDGE
+#cmakedefine01 ENABLE_CLICKHOUSE_KEEPER

programs/keeper/CMakeLists.txt

@@ -0,0 +1,24 @@
+set(CLICKHOUSE_KEEPER_SOURCES
+    Keeper.cpp
+)
+
+if (OS_LINUX)
+    set (LINK_RESOURCE_LIB INTERFACE "-Wl,${WHOLE_ARCHIVE} $<TARGET_FILE:clickhouse_keeper_configs> -Wl,${NO_WHOLE_ARCHIVE}")
+endif ()
+
+set (CLICKHOUSE_KEEPER_LINK
+    PRIVATE
+        clickhouse_common_config
+        clickhouse_common_io
+        clickhouse_common_zookeeper
+        daemon
+        dbms
+
+    ${LINK_RESOURCE_LIB}
+)
+
+clickhouse_program_add(keeper)
+
+install (FILES keeper_config.xml DESTINATION "${CLICKHOUSE_ETC_DIR}/clickhouse-keeper" COMPONENT clickhouse-keeper)
+
+clickhouse_embed_binaries(keeper keeper_config.xml keeper_embedded.xml)

programs/keeper/Keeper.cpp

@@ -0,0 +1,474 @@
+#include "Keeper.h"
+
+#include <sys/stat.h>
+#include <pwd.h>
+#include <Common/ClickHouseRevision.h>
+#include <Server/ProtocolServerAdapter.h>
+#include <Common/DNSResolver.h>
+#include <Interpreters/DNSCacheUpdater.h>
+#include <Poco/Net/NetException.h>
+#include <Poco/Net/TCPServerParams.h>
+#include <Poco/Net/TCPServer.h>
+#include <common/defines.h>
+#include <common/logger_useful.h>
+#include <common/ErrorHandlers.h>
+#include <ext/scope_guard.h>
+#include <Poco/Util/HelpFormatter.h>
+#include <Poco/Version.h>
+#include <Poco/Environment.h>
+#include <Common/getMultipleKeysFromConfig.h>
+#include <filesystem>
+#include <IO/UseSSL.h>
+
+#if !defined(ARCADIA_BUILD)
+#   include "config_core.h"
+#   include "Common/config_version.h"
+#endif
+
+#if USE_SSL
+#    include <Poco/Net/Context.h>
+#    include <Poco/Net/SecureServerSocket.h>
+#endif
+
+#if USE_NURAFT
+#   include <Server/KeeperTCPHandlerFactory.h>
+#endif
+
+#if defined(OS_LINUX)
+#    include <unistd.h>
+#    include <sys/syscall.h>
+#endif
+
+
+int mainEntryClickHouseKeeper(int argc, char ** argv)
+{
+    DB::Keeper app;
+
+    try
+    {
+        return app.run(argc, argv);
+    }
+    catch (...)
+    {
+        std::cerr << DB::getCurrentExceptionMessage(true) << "\n";
+        auto code = DB::getCurrentExceptionCode();
+        return code ? code : 1;
+    }
+}
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+    extern const int NO_ELEMENTS_IN_CONFIG;
+    extern const int SUPPORT_IS_DISABLED;
+    extern const int NETWORK_ERROR;
+    extern const int MISMATCHING_USERS_FOR_PROCESS_AND_DATA;
+    extern const int FAILED_TO_GETPWUID;
+}
+
+namespace
+{
+
+int waitServersToFinish(std::vector<DB::ProtocolServerAdapter> & servers, size_t seconds_to_wait)
+{
+    const int sleep_max_ms = 1000 * seconds_to_wait;
+    const int sleep_one_ms = 100;
+    int sleep_current_ms = 0;
+    int current_connections = 0;
+    for (;;)
+    {
+        current_connections = 0;
+
+        for (auto & server : servers)
+        {
+            server.stop();
+            current_connections += server.currentConnections();
+        }
+
+        if (!current_connections)
+            break;
+
+        sleep_current_ms += sleep_one_ms;
+        if (sleep_current_ms < sleep_max_ms)
+            std::this_thread::sleep_for(std::chrono::milliseconds(sleep_one_ms));
+        else
+            break;
+    }
+    return current_connections;
+}
+
+Poco::Net::SocketAddress makeSocketAddress(const std::string & host, UInt16 port, Poco::Logger * log)
+{
+    Poco::Net::SocketAddress socket_address;
+    try
+    {
+        socket_address = Poco::Net::SocketAddress(host, port);
+    }
+    catch (const Poco::Net::DNSException & e)
+    {
+        const auto code = e.code();
+        if (code == EAI_FAMILY
+#if defined(EAI_ADDRFAMILY)
+                    || code == EAI_ADDRFAMILY
+#endif
+           )
+        {
+            LOG_ERROR(log, "Cannot resolve listen_host ({}), error {}: {}. "
+                "If it is an IPv6 address and your host has disabled IPv6, then consider to "
+                "specify IPv4 address to listen in <listen_host> element of configuration "
+                "file. Example: <listen_host>0.0.0.0</listen_host>",
+                host, e.code(), e.message());
+        }
+
+        throw;
+    }
+    return socket_address;
+}
+
+[[noreturn]] void forceShutdown()
+{
+#if defined(THREAD_SANITIZER) && defined(OS_LINUX)
+    /// Thread sanitizer tries to do something on exit that we don't need if we want to exit immediately,
+    /// while connection handling threads are still run.
+    (void)syscall(SYS_exit_group, 0);
+    __builtin_unreachable();
+#else
+    _exit(0);
+#endif
+}
+
+std::string getUserName(uid_t user_id)
+{
+    /// Try to convert user id into user name.
+    auto buffer_size = sysconf(_SC_GETPW_R_SIZE_MAX);
+    if (buffer_size <= 0)
+        buffer_size = 1024;
+    std::string buffer;
+    buffer.reserve(buffer_size);
+
+    struct passwd passwd_entry;
+    struct passwd * result = nullptr;
+    const auto error = getpwuid_r(user_id, &passwd_entry, buffer.data(), buffer_size, &result);
+
+    if (error)
+        throwFromErrno("Failed to find user name for " + toString(user_id), ErrorCodes::FAILED_TO_GETPWUID, error);
+    else if (result)
+        return result->pw_name;
+    return toString(user_id);
+}
+
+}
+
+Poco::Net::SocketAddress Keeper::socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure) const
+{
+    auto address = makeSocketAddress(host, port, &logger());
+#if !defined(POCO_CLICKHOUSE_PATCH) || POCO_VERSION < 0x01090100
+    if (secure)
+        /// Bug in old (<1.9.1) poco, listen() after bind() with reusePort param will fail because have no implementation in SecureServerSocketImpl
+        /// https://github.com/pocoproject/poco/pull/2257
+        socket.bind(address, /* reuseAddress = */ true);
+    else
+#endif
+#if POCO_VERSION < 0x01080000
+    socket.bind(address, /* reuseAddress = */ true);
+#else
+    socket.bind(address, /* reuseAddress = */ true, /* reusePort = */ config().getBool("listen_reuse_port", false));
+#endif
+
+    socket.listen(/* backlog = */ config().getUInt("listen_backlog", 64));
+
+    return address;
+}
+
+void Keeper::createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func) const
+{
+    /// For testing purposes, user may omit tcp_port or http_port or https_port in configuration file.
+    if (!config().has(port_name))
+        return;
+
+    auto port = config().getInt(port_name);
+    try
+    {
+        func(port);
+    }
+    catch (const Poco::Exception &)
+    {
+        std::string message = "Listen [" + listen_host + "]:" + std::to_string(port) + " failed: " + getCurrentExceptionMessage(false);
+
+        if (listen_try)
+        {
+            LOG_WARNING(&logger(), "{}. If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to "
+                "specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration "
+                "file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> ."
+                " Example for disabled IPv4: <listen_host>::</listen_host>",
+                message);
+        }
+        else
+        {
+            throw Exception{message, ErrorCodes::NETWORK_ERROR};
+        }
+    }
+}
+
+void Keeper::uninitialize()
+{
+    logger().information("shutting down");
+    BaseDaemon::uninitialize();
+}
+
+int Keeper::run()
+{
+    if (config().hasOption("help"))
+    {
+        Poco::Util::HelpFormatter help_formatter(Keeper::options());
+        auto header_str = fmt::format("{} [OPTION] [-- [ARG]...]\n"
+                                      "positional arguments can be used to rewrite config.xml properties, for example, --http_port=8010",
+                                      commandName());
+        help_formatter.setHeader(header_str);
+        help_formatter.format(std::cout);
+        return 0;
+    }
+    if (config().hasOption("version"))
+    {
+        std::cout << DBMS_NAME << " keeper version " << VERSION_STRING << VERSION_OFFICIAL << "." << std::endl;
+        return 0;
+    }
+
+    return Application::run(); // NOLINT
+}
+
+void Keeper::initialize(Poco::Util::Application & self)
+{
+    BaseDaemon::initialize(self);
+    logger().information("starting up");
+
+    LOG_INFO(&logger(), "OS Name = {}, OS Version = {}, OS Architecture = {}",
+        Poco::Environment::osName(),
+        Poco::Environment::osVersion(),
+        Poco::Environment::osArchitecture());
+}
+
+std::string Keeper::getDefaultConfigFileName() const
+{
+    return "keeper_config.xml";
+}
+
+void Keeper::defineOptions(Poco::Util::OptionSet & options)
+{
+    options.addOption(
+        Poco::Util::Option("help", "h", "show help and exit")
+            .required(false)
+            .repeatable(false)
+            .binding("help"));
+    options.addOption(
+        Poco::Util::Option("version", "V", "show version and exit")
+            .required(false)
+            .repeatable(false)
+            .binding("version"));
+    BaseDaemon::defineOptions(options);
+}
+
+int Keeper::main(const std::vector<std::string> & /*args*/)
+{
+    Poco::Logger * log = &logger();
+
+    UseSSL use_ssl;
+
+    MainThreadStatus::getInstance();
+
+#if !defined(NDEBUG) || !defined(__OPTIMIZE__)
+    LOG_WARNING(log, "Keeper was built in debug mode. It will work slowly.");
+#endif
+
+#if defined(SANITIZER)
+    LOG_WARNING(log, "Keeper was built with sanitizer. It will work slowly.");
+#endif
+
+    auto shared_context = Context::createShared();
+    global_context = Context::createGlobal(shared_context.get());
+
+    global_context->makeGlobalContext();
+    global_context->setApplicationType(Context::ApplicationType::KEEPER);
+
+    if (!config().has("keeper_server"))
+        throw Exception(ErrorCodes::NO_ELEMENTS_IN_CONFIG, "Keeper configuration (<keeper_server> section) not found in config");
+
+
+    std::string path;
+
+    if (config().has("keeper_server.storage_path"))
+        path = config().getString("keeper_server.storage_path");
+    else if (config().has("keeper_server.log_storage_path"))
+        path = config().getString("keeper_server.log_storage_path");
+    else if (config().has("keeper_server.snapshot_storage_path"))
+        path = config().getString("keeper_server.snapshot_storage_path");
+    else
+        path = std::filesystem::path{KEEPER_DEFAULT_PATH};
+
+
+    /// Check that the process user id matches the owner of the data.
+    const auto effective_user_id = geteuid();
+    struct stat statbuf;
+    if (stat(path.c_str(), &statbuf) == 0 && effective_user_id != statbuf.st_uid)
+    {
+        const auto effective_user = getUserName(effective_user_id);
+        const auto data_owner = getUserName(statbuf.st_uid);
+        std::string message = "Effective user of the process (" + effective_user +
+            ") does not match the owner of the data (" + data_owner + ").";
+        if (effective_user_id == 0)
+        {
+            message += " Run under 'sudo -u " + data_owner + "'.";
+            throw Exception(message, ErrorCodes::MISMATCHING_USERS_FOR_PROCESS_AND_DATA);
+        }
+        else
+        {
+            LOG_WARNING(log, message);
+        }
+    }
+
+    const Settings & settings = global_context->getSettingsRef();
+
+    GlobalThreadPool::initialize(config().getUInt("max_thread_pool_size", 100));
+
+    static ServerErrorHandler error_handler;
+    Poco::ErrorHandler::set(&error_handler);
+
+    /// Initialize DateLUT early, to not interfere with running time of first query.
+    LOG_DEBUG(log, "Initializing DateLUT.");
+    DateLUT::instance();
+    LOG_TRACE(log, "Initialized DateLUT with time zone '{}'.", DateLUT::instance().getTimeZone());
+
+    /// Don't want to use DNS cache
+    DNSResolver::instance().setDisableCacheFlag();
+
+    Poco::ThreadPool server_pool(3, config().getUInt("max_connections", 1024));
+
+    std::vector<std::string> listen_hosts = DB::getMultipleValuesFromConfig(config(), "", "listen_host");
+
+    bool listen_try = config().getBool("listen_try", false);
+    if (listen_hosts.empty())
+    {
+        listen_hosts.emplace_back("::1");
+        listen_hosts.emplace_back("127.0.0.1");
+        listen_try = true;
+    }
+
+    auto servers = std::make_shared<std::vector<ProtocolServerAdapter>>();
+
+#if USE_NURAFT
+    /// Initialize test keeper RAFT. Do nothing if no nu_keeper_server in config.
+    global_context->initializeKeeperStorageDispatcher();
+    for (const auto & listen_host : listen_hosts)
+    {
+        /// TCP Keeper
+        const char * port_name = "keeper_server.tcp_port";
+        createServer(listen_host, port_name, listen_try, [&](UInt16 port)
+        {
+            Poco::Net::ServerSocket socket;
+            auto address = socketBindListen(socket, listen_host, port);
+            socket.setReceiveTimeout(settings.receive_timeout);
+            socket.setSendTimeout(settings.send_timeout);
+            servers->emplace_back(
+                port_name,
+                std::make_unique<Poco::Net::TCPServer>(
+                    new KeeperTCPHandlerFactory(*this, false), server_pool, socket, new Poco::Net::TCPServerParams));
+
+            LOG_INFO(log, "Listening for connections to Keeper (tcp): {}", address.toString());
+        });
+
+        const char * secure_port_name = "keeper_server.tcp_port_secure";
+        createServer(listen_host, secure_port_name, listen_try, [&](UInt16 port)
+        {
+#if USE_SSL
+            Poco::Net::SecureServerSocket socket;
+            auto address = socketBindListen(socket, listen_host, port, /* secure = */ true);
+            socket.setReceiveTimeout(settings.receive_timeout);
+            socket.setSendTimeout(settings.send_timeout);
+            servers->emplace_back(
+                secure_port_name,
+                std::make_unique<Poco::Net::TCPServer>(
+                    new KeeperTCPHandlerFactory(*this, true), server_pool, socket, new Poco::Net::TCPServerParams));
+            LOG_INFO(log, "Listening for connections to Keeper with secure protocol (tcp_secure): {}", address.toString());
+#else
+            UNUSED(port);
+            throw Exception{"SSL support for TCP protocol is disabled because Poco library was built without NetSSL support.",
+                ErrorCodes::SUPPORT_IS_DISABLED};
+#endif
+        });
+    }
+#else
+    throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "ClickHouse keeper built without NuRaft library. Cannot use coordination.");
+#endif
+
+    for (auto & server : *servers)
+        server.start();
+
+    SCOPE_EXIT({
+        LOG_INFO(log, "Shutting down.");
+
+        global_context->shutdown();
+
+        LOG_DEBUG(log, "Waiting for current connections to Keeper to finish.");
+        int current_connections = 0;
+        for (auto & server : *servers)
+        {
+            server.stop();
+            current_connections += server.currentConnections();
+        }
+
+        if (current_connections)
+            LOG_INFO(log, "Closed all listening sockets. Waiting for {} outstanding connections.", current_connections);
+        else
+            LOG_INFO(log, "Closed all listening sockets.");
+
+        if (current_connections > 0)
+            current_connections = waitServersToFinish(*servers, config().getInt("shutdown_wait_unfinished", 5));
+
+        if (current_connections)
+            LOG_INFO(log, "Closed connections to Keeper. But {} remain. Probably some users cannot finish their connections after context shutdown.", current_connections);
+        else
+            LOG_INFO(log, "Closed connections to Keeper.");
+
+        global_context->shutdownKeeperStorageDispatcher();
+
+        /// Wait server pool to avoid use-after-free of destroyed context in the handlers
+        server_pool.joinAll();
+
+        /** Explicitly destroy Context. It is more convenient than in destructor of Server, because logger is still available.
+          * At this moment, no one could own shared part of Context.
+          */
+        global_context.reset();
+        shared_context.reset();
+
+        LOG_DEBUG(log, "Destroyed global context.");
+
+        if (current_connections)
+        {
+            LOG_INFO(log, "Will shutdown forcefully.");
+            forceShutdown();
+        }
+    });
+
+
+    buildLoggers(config(), logger());
+
+    LOG_INFO(log, "Ready for connections.");
+
+    waitForTerminationRequest();
+
+    return Application::EXIT_OK;
+}
+
+
+void Keeper::logRevision() const
+{
+    Poco::Logger::root().information("Starting ClickHouse Keeper " + std::string{VERSION_STRING}
+        + " with revision " + std::to_string(ClickHouseRevision::getVersionRevision())
+        + ", " + build_id_info
+        + ", PID " + std::to_string(getpid()));
+}
+
+
+}

programs/keeper/Keeper.h

@@ -0,0 +1,69 @@
+#pragma once
+
+#include <Server/IServer.h>
+#include <daemon/BaseDaemon.h>
+
+namespace Poco
+{
+    namespace Net
+    {
+        class ServerSocket;
+    }
+}
+
+namespace DB
+{
+
+/// standalone clickhouse-keeper server (replacement for ZooKeeper). Uses the same
+/// config as clickhouse-server. Serves requests on TCP ports with or without
+/// SSL using ZooKeeper protocol.
+class Keeper : public BaseDaemon, public IServer
+{
+public:
+    using ServerApplication::run;
+
+    Poco::Util::LayeredConfiguration & config() const override
+    {
+        return BaseDaemon::config();
+    }
+
+    Poco::Logger & logger() const override
+    {
+        return BaseDaemon::logger();
+    }
+
+    ContextPtr context() const override
+    {
+        return global_context;
+    }
+
+    bool isCancelled() const override
+    {
+        return BaseDaemon::isCancelled();
+    }
+
+    void defineOptions(Poco::Util::OptionSet & _options) override;
+
+protected:
+    void logRevision() const override;
+
+    int run() override;
+
+    void initialize(Application & self) override;
+
+    void uninitialize() override;
+
+    int main(const std::vector<std::string> & args) override;
+
+    std::string getDefaultConfigFileName() const override;
+
+private:
+    ContextPtr global_context;
+
+    Poco::Net::SocketAddress socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure = false) const;
+
+    using CreateServerFunc = std::function<void(UInt16)>;
+    void createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func) const;
+};
+
+}

programs/keeper/clickhouse-keeper.cpp

@@ -0,0 +1,6 @@
+int mainEntryClickHouseKeeper(int argc, char ** argv);
+
+int main(int argc_, char ** argv_)
+{
+    return mainEntryClickHouseKeeper(argc_, argv_);
+}

programs/keeper/keeper_config.xml

@@ -0,0 +1,81 @@
+<yandex>
+    <logger>
+        <!-- Possible levels [1]:
+
+          - none (turns off logging)
+          - fatal
+          - critical
+          - error
+          - warning
+          - notice
+          - information
+          - debug
+          - trace
+
+            [1]: https://github.com/pocoproject/poco/blob/poco-1.9.4-release/Foundation/include/Poco/Logger.h#L105-L114
+        -->
+        <level>trace</level>
+        <log>/var/log/clickhouse-keeper/clickhouse-keeper.log</log>
+        <errorlog>/var/log/clickhouse-keeper/clickhouse-keeper.err.log</errorlog>
+        <!-- Rotation policy
+             See https://github.com/pocoproject/poco/blob/poco-1.9.4-release/Foundation/include/Poco/FileChannel.h#L54-L85
+          -->
+        <size>1000M</size>
+        <count>10</count>
+        <!-- <console>1</console> --> <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
+    </logger>
+
+    <max_connections>4096</max_connections>
+
+    <keeper_server>
+            <tcp_port>9181</tcp_port>
+
+            <!-- Must be unique among all keeper serves -->
+            <server_id>1</server_id>
+
+            <log_storage_path>/var/lib/clickhouse/coordination/logs</log_storage_path>
+            <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
+
+            <coordination_settings>
+                <operation_timeout_ms>10000</operation_timeout_ms>
+                <session_timeout_ms>30000</session_timeout_ms>
+                <raft_logs_level>information</raft_logs_level>
+                <!-- All settings listed in https://github.com/ClickHouse/ClickHouse/blob/master/src/Coordination/CoordinationSettings.h -->
+            </coordination_settings>
+
+            <raft_configuration>
+                <server>
+                    <id>1</id>
+
+                    <!-- Internal port and hostname -->
+                    <hostname>localhost</hostname>
+                    <port>44444</port>
+                </server>
+
+                <!-- Add more servers here -->
+
+            </raft_configuration>
+    </keeper_server>
+
+
+    <openSSL>
+      <server>
+            <!-- Used for secure tcp port -->
+            <!-- openssl req -subj "/CN=localhost" -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout /etc/clickhouse-server/server.key -out /etc/clickhouse-server/server.crt -->
+            <certificateFile>/etc/clickhouse-keeper/server.crt</certificateFile>
+            <privateKeyFile>/etc/clickhouse-keeper/server.key</privateKeyFile>
+            <!-- dhparams are optional. You can delete the <dhParamsFile> element.
+                 To generate dhparams, use the following command:
+                  openssl dhparam -out /etc/clickhouse-keeper/dhparam.pem 4096
+                 Only file format with BEGIN DH PARAMETERS is supported.
+              -->
+            <dhParamsFile>/etc/clickhouse-keeper/dhparam.pem</dhParamsFile>
+            <verificationMode>none</verificationMode>
+            <loadDefaultCAFile>true</loadDefaultCAFile>
+            <cacheSessions>true</cacheSessions>
+            <disableProtocols>sslv2,sslv3</disableProtocols>
+            <preferServerCiphers>true</preferServerCiphers>
+        </server>
+    </openSSL>
+
+</yandex>

programs/keeper/keeper_embedded.xml

@@ -0,0 +1,21 @@
+<yandex>
+    <logger>
+        <level>trace</level>
+        <console>true</console>
+    </logger>
+
+    <keeper_server>
+        <tcp_port>9181</tcp_port>
+        <server_id>1</server_id>
+        <log_storage_path>./keeper_log</log_storage_path>
+        <snapshot_storage_path>./keeper_snapshot</snapshot_storage_path>
+
+        <raft_configuration>
+            <server>
+                <id>1</id>
+                <hostname>localhost</hostname>
+                <port>44444</port>
+            </server>
+        </raft_configuration>
+    </keeper_server>
+</yandex>

programs/main.cpp

@@ -55,6 +55,9 @@ int mainEntryClickHouseObfuscator(int argc, char ** argv);
 #if ENABLE_CLICKHOUSE_GIT_IMPORT
 int mainEntryClickHouseGitImport(int argc, char ** argv);
 #endif
+#if ENABLE_CLICKHOUSE_KEEPER
+int mainEntryClickHouseKeeper(int argc, char ** argv);
+#endif
 #if ENABLE_CLICKHOUSE_INSTALL
 int mainEntryClickHouseInstall(int argc, char ** argv);
 int mainEntryClickHouseStart(int argc, char ** argv);
@@ -112,6 +115,9 @@ std::pair<const char *, MainFunc> clickhouse_applications[] =
 #if ENABLE_CLICKHOUSE_GIT_IMPORT
     {"git-import", mainEntryClickHouseGitImport},
 #endif
+#if ENABLE_CLICKHOUSE_KEEPER
+    {"keeper", mainEntryClickHouseKeeper},
+#endif
 #if ENABLE_CLICKHOUSE_INSTALL
     {"install", mainEntryClickHouseInstall},
     {"start", mainEntryClickHouseStart},

programs/server/CMakeLists.txt

@@ -31,37 +31,4 @@ clickhouse_program_add(server)
 
 install(FILES config.xml users.xml DESTINATION "${CLICKHOUSE_ETC_DIR}/clickhouse-server" COMPONENT clickhouse)
 
-# TODO We actually need this on Mac, FreeBSD.
-if (OS_LINUX)
-    # Embed default config files as a resource into the binary.
-    # This is needed for two purposes:
-    # 1. Allow to run the binary without download of any other files.
-    # 2. Allow to implement "sudo clickhouse install" tool.
-
-    foreach(RESOURCE_FILE config.xml users.xml embedded.xml play.html)
-        set(RESOURCE_OBJ ${RESOURCE_FILE}.o)
-        set(RESOURCE_OBJS ${RESOURCE_OBJS} ${RESOURCE_OBJ})
-
-        # https://stackoverflow.com/questions/14776463/compile-and-add-an-object-file-from-a-binary-with-cmake
-        # PPC64LE fails to do this with objcopy, use ld or lld instead
-        if (ARCH_PPC64LE)
-            add_custom_command(OUTPUT ${RESOURCE_OBJ}
-                COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && ${CMAKE_LINKER} -m elf64lppc -r -b binary -o "${CMAKE_CURRENT_BINARY_DIR}/${RESOURCE_OBJ}" ${RESOURCE_FILE})
-        else()
-            add_custom_command(OUTPUT ${RESOURCE_OBJ}
-                COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && ${OBJCOPY_PATH} -I binary ${OBJCOPY_ARCH_OPTIONS} ${RESOURCE_FILE} "${CMAKE_CURRENT_BINARY_DIR}/${RESOURCE_OBJ}"
-                COMMAND ${OBJCOPY_PATH} --rename-section .data=.rodata,alloc,load,readonly,data,contents
-                    "${CMAKE_CURRENT_BINARY_DIR}/${RESOURCE_OBJ}" "${CMAKE_CURRENT_BINARY_DIR}/${RESOURCE_OBJ}")
-        endif()
-        set_source_files_properties(${RESOURCE_OBJ} PROPERTIES EXTERNAL_OBJECT true GENERATED true)
-    endforeach(RESOURCE_FILE)
-
-    add_library(clickhouse_server_configs STATIC ${RESOURCE_OBJS})
-    set_target_properties(clickhouse_server_configs PROPERTIES LINKER_LANGUAGE C)
-
-    # whole-archive prevents symbols from being discarded for unknown reason
-    # CMake can shuffle each of target_link_libraries arguments with other
-    # libraries in linker command. To avoid this we hardcode whole-archive
-    # library into single string.
-    add_dependencies(clickhouse-server-lib clickhouse_server_configs)
-endif ()
+clickhouse_embed_binaries(server config.xml users.xml embedded.xml play.html)

programs/server/config.xml

@@ -362,6 +362,20 @@
                 bind_dn - template used to construct the DN to bind to.
                         The resulting DN will be constructed by replacing all '{user_name}' substrings of the template with the actual
                          user name during each authentication attempt.
+                user_dn_detection - section with LDAP search parameters for detecting the actual user DN of the bound user.
+                        This is mainly used in search filters for further role mapping when the server is Active Directory. The
+                         resulting user DN will be used when replacing '{user_dn}' substrings wherever they are allowed. By default,
+                         user DN is set equal to bind DN, but once search is performed, it will be updated with to the actual detected
+                         user DN value.
+                    base_dn - template used to construct the base DN for the LDAP search.
+                            The resulting DN will be constructed by replacing all '{user_name}' and '{bind_dn}' substrings
+                             of the template with the actual user name and bind DN during the LDAP search.
+                    scope - scope of the LDAP search.
+                            Accepted values are: 'base', 'one_level', 'children', 'subtree' (the default).
+                    search_filter - template used to construct the search filter for the LDAP search.
+                            The resulting filter will be constructed by replacing all '{user_name}', '{bind_dn}', and '{base_dn}'
+                             substrings of the template with the actual user name, bind DN, and base DN during the LDAP search.
+                            Note, that the special characters must be escaped properly in XML.
                 verification_cooldown - a period of time, in seconds, after a successful bind attempt, during which a user will be assumed
                          to be successfully authenticated for all consecutive requests without contacting the LDAP server.
                         Specify 0 (the default) to disable caching and force contacting the LDAP server for each authentication request.
@@ -393,6 +407,17 @@
                     <tls_ca_cert_dir>/path/to/tls_ca_cert_dir</tls_ca_cert_dir>
                     <tls_cipher_suite>ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:AES256-GCM-SHA384</tls_cipher_suite>
                 </my_ldap_server>
+             Example (typical Active Directory with configured user DN detection for further role mapping):
+                <my_ad_server>
+                    <host>localhost</host>
+                    <port>389</port>
+                    <bind_dn>EXAMPLE\{user_name}</bind_dn>
+                    <user_dn_detection>
+                        <base_dn>CN=Users,DC=example,DC=com</base_dn>
+                        <search_filter>(&amp;(objectClass=user)(sAMAccountName={user_name}))</search_filter>
+                    </user_dn_detection>
+                    <enable_tls>no</enable_tls>
+                </my_ad_server>
         -->
     </ldap_servers>
 
@@ -444,15 +469,16 @@
                         There can be multiple 'role_mapping' sections defined inside the same 'ldap' section. All of them will be
                          applied.
                     base_dn - template used to construct the base DN for the LDAP search.
-                            The resulting DN will be constructed by replacing all '{user_name}' and '{bind_dn}' substrings
-                             of the template with the actual user name and bind DN during each LDAP search.
+                            The resulting DN will be constructed by replacing all '{user_name}', '{bind_dn}', and '{user_dn}'
+                             substrings of the template with the actual user name, bind DN, and user DN during each LDAP search.
                     scope - scope of the LDAP search.
                             Accepted values are: 'base', 'one_level', 'children', 'subtree' (the default).
                     search_filter - template used to construct the search filter for the LDAP search.
-                            The resulting filter will be constructed by replacing all '{user_name}', '{bind_dn}', and '{base_dn}'
-                             substrings of the template with the actual user name, bind DN, and base DN during each LDAP search.
+                            The resulting filter will be constructed by replacing all '{user_name}', '{bind_dn}', '{user_dn}', and
+                             '{base_dn}' substrings of the template with the actual user name, bind DN, user DN, and base DN during
+                             each LDAP search.
                             Note, that the special characters must be escaped properly in XML.
-                    attribute - attribute name whose values will be returned by the LDAP search.
+                    attribute - attribute name whose values will be returned by the LDAP search. 'cn', by default.
                     prefix - prefix, that will be expected to be in front of each string in the original list of strings returned by
                              the LDAP search. Prefix will be removed from the original strings and resulting strings will be treated
                              as local role names. Empty, by default.
@@ -471,6 +497,17 @@
                         <prefix>clickhouse_</prefix>
                     </role_mapping>
                 </ldap>
+             Example (typical Active Directory with role mapping that relies on the detected user DN):
+                <ldap>
+                    <server>my_ad_server</server>
+                    <role_mapping>
+                        <base_dn>CN=Users,DC=example,DC=com</base_dn>
+                        <attribute>CN</attribute>
+                        <scope>subtree</scope>
+                        <search_filter>(&amp;(objectClass=group)(member={user_dn}))</search_filter>
+                        <prefix>clickhouse_</prefix>
+                    </role_mapping>
+                </ldap>
         -->
     </user_directories>
 

src/Access/ExternalAuthenticators.cpp

@@ -20,13 +20,42 @@ namespace ErrorCodes
 namespace
 {
 
-auto parseLDAPServer(const Poco::Util::AbstractConfiguration & config, const String & name)
+void parseLDAPSearchParams(LDAPClient::SearchParams & params, const Poco::Util::AbstractConfiguration & config, const String & prefix)
+{
+    const bool has_base_dn = config.has(prefix + ".base_dn");
+    const bool has_search_filter = config.has(prefix + ".search_filter");
+    const bool has_attribute = config.has(prefix + ".attribute");
+    const bool has_scope = config.has(prefix + ".scope");
+
+    if (has_base_dn)
+        params.base_dn = config.getString(prefix + ".base_dn");
+
+    if (has_search_filter)
+        params.search_filter = config.getString(prefix + ".search_filter");
+
+    if (has_attribute)
+        params.attribute = config.getString(prefix + ".attribute");
+
+    if (has_scope)
+    {
+        auto scope = config.getString(prefix + ".scope");
+        boost::algorithm::to_lower(scope);
+
+        if (scope == "base")           params.scope = LDAPClient::SearchParams::Scope::BASE;
+        else if (scope == "one_level") params.scope = LDAPClient::SearchParams::Scope::ONE_LEVEL;
+        else if (scope == "subtree")   params.scope = LDAPClient::SearchParams::Scope::SUBTREE;
+        else if (scope == "children")  params.scope = LDAPClient::SearchParams::Scope::CHILDREN;
+        else
+            throw Exception("Invalid value for 'scope' field of LDAP search parameters in '" + prefix +
+                "' section, must be one of 'base', 'one_level', 'subtree', or 'children'", ErrorCodes::BAD_ARGUMENTS);
+    }
+}
+
+void parseLDAPServer(LDAPClient::Params & params, const Poco::Util::AbstractConfiguration & config, const String & name)
 {
     if (name.empty())
         throw Exception("LDAP server name cannot be empty", ErrorCodes::BAD_ARGUMENTS);
 
-    LDAPClient::Params params;
-
     const String ldap_server_config = "ldap_servers." + name;
 
     const bool has_host = config.has(ldap_server_config + ".host");
@@ -34,6 +63,7 @@ auto parseLDAPServer(const Poco::Util::AbstractConfiguration & config, const Str
     const bool has_bind_dn = config.has(ldap_server_config + ".bind_dn");
     const bool has_auth_dn_prefix = config.has(ldap_server_config + ".auth_dn_prefix");
     const bool has_auth_dn_suffix = config.has(ldap_server_config + ".auth_dn_suffix");
+    const bool has_user_dn_detection = config.has(ldap_server_config + ".user_dn_detection");
     const bool has_verification_cooldown = config.has(ldap_server_config + ".verification_cooldown");
     const bool has_enable_tls = config.has(ldap_server_config + ".enable_tls");
     const bool has_tls_minimum_protocol_version = config.has(ldap_server_config + ".tls_minimum_protocol_version");
@@ -66,6 +96,17 @@ auto parseLDAPServer(const Poco::Util::AbstractConfiguration & config, const Str
         params.bind_dn = auth_dn_prefix + "{user_name}" + auth_dn_suffix;
     }
 
+    if (has_user_dn_detection)
+    {
+        if (!params.user_dn_detection)
+        {
+            params.user_dn_detection.emplace();
+            params.user_dn_detection->attribute = "dn";
+        }
+
+        parseLDAPSearchParams(*params.user_dn_detection, config, ldap_server_config + ".user_dn_detection");
+    }
+
     if (has_verification_cooldown)
         params.verification_cooldown = std::chrono::seconds{config.getUInt64(ldap_server_config + ".verification_cooldown")};
 
@@ -143,14 +184,10 @@ auto parseLDAPServer(const Poco::Util::AbstractConfiguration & config, const Str
     }
     else
         params.port = (params.enable_tls == LDAPClient::Params::TLSEnable::YES ? 636 : 389);
-
-    return params;
 }
 
-auto parseKerberosParams(const Poco::Util::AbstractConfiguration & config)
+void parseKerberosParams(GSSAcceptorContext::Params & params, const Poco::Util::AbstractConfiguration & config)
 {
-    GSSAcceptorContext::Params params;
-
     Poco::Util::AbstractConfiguration::Keys keys;
     config.keys("kerberos", keys);
 
@@ -180,12 +217,20 @@ auto parseKerberosParams(const Poco::Util::AbstractConfiguration & config)
 
     params.realm = config.getString("kerberos.realm", "");
     params.principal = config.getString("kerberos.principal", "");
-
-    return params;
 }
 
 }
 
+void parseLDAPRoleSearchParams(LDAPClient::RoleSearchParams & params, const Poco::Util::AbstractConfiguration & config, const String & prefix)
+{
+    parseLDAPSearchParams(params, config, prefix);
+
+    const bool has_prefix = config.has(prefix + ".prefix");
+
+    if (has_prefix)
+        params.prefix = config.getString(prefix + ".prefix");
+}
+
 void ExternalAuthenticators::reset()
 {
     std::scoped_lock lock(mutex);
@@ -229,7 +274,8 @@ void ExternalAuthenticators::setConfiguration(const Poco::Util::AbstractConfigur
     {
         try
         {
-            ldap_client_params_blueprint.insert_or_assign(ldap_server_name, parseLDAPServer(config, ldap_server_name));
+            ldap_client_params_blueprint.erase(ldap_server_name);
+            parseLDAPServer(ldap_client_params_blueprint.emplace(ldap_server_name, LDAPClient::Params{}).first->second, config, ldap_server_name);
         }
         catch (...)
         {
@@ -240,7 +286,7 @@ void ExternalAuthenticators::setConfiguration(const Poco::Util::AbstractConfigur
     try
     {
         if (kerberos_keys_count > 0)
-            kerberos_params = parseKerberosParams(config);
+            parseKerberosParams(kerberos_params.emplace(), config);
     }
     catch (...)
     {
@@ -249,7 +295,7 @@ void ExternalAuthenticators::setConfiguration(const Poco::Util::AbstractConfigur
 }
 
 bool ExternalAuthenticators::checkLDAPCredentials(const String & server, const BasicCredentials & credentials,
-    const LDAPClient::SearchParamsList * search_params, LDAPClient::SearchResultsList * search_results) const
+    const LDAPClient::RoleSearchParamsList * role_search_params, LDAPClient::SearchResultsList * role_search_results) const
 {
     std::optional<LDAPClient::Params> params;
     std::size_t params_hash = 0;
@@ -267,9 +313,9 @@ bool ExternalAuthenticators::checkLDAPCredentials(const String & server, const B
         params->password = credentials.getPassword();
 
         params->combineCoreHash(params_hash);
-        if (search_params)
+        if (role_search_params)
         {
-            for (const auto & params_instance : *search_params)
+            for (const auto & params_instance : *role_search_params)
             {
                 params_instance.combineHash(params_hash);
             }
@@ -301,14 +347,14 @@ bool ExternalAuthenticators::checkLDAPCredentials(const String & server, const B
 
                         // Ensure that search_params are compatible.
                         (
-                            search_params == nullptr ?
-                            entry.last_successful_search_results.empty() :
-                            search_params->size() == entry.last_successful_search_results.size()
+                            role_search_params == nullptr ?
+                            entry.last_successful_role_search_results.empty() :
+                            role_search_params->size() == entry.last_successful_role_search_results.size()
                         )
                     )
                     {
-                        if (search_results)
-                            *search_results = entry.last_successful_search_results;
+                        if (role_search_results)
+                            *role_search_results = entry.last_successful_role_search_results;
 
                         return true;
                     }
@@ -326,7 +372,7 @@ bool ExternalAuthenticators::checkLDAPCredentials(const String & server, const B
     }
 
     LDAPSimpleAuthClient client(params.value());
-    const auto result = client.authenticate(search_params, search_results);
+    const auto result = client.authenticate(role_search_params, role_search_results);
     const auto current_check_timestamp = std::chrono::steady_clock::now();
 
     // Update the cache, but only if this is the latest check and the server is still configured in a compatible way.
@@ -345,9 +391,9 @@ bool ExternalAuthenticators::checkLDAPCredentials(const String & server, const B
 
         std::size_t new_params_hash = 0;
         new_params.combineCoreHash(new_params_hash);
-        if (search_params)
+        if (role_search_params)
         {
-            for (const auto & params_instance : *search_params)
+            for (const auto & params_instance : *role_search_params)
             {
                 params_instance.combineHash(new_params_hash);
             }
@@ -363,17 +409,17 @@ bool ExternalAuthenticators::checkLDAPCredentials(const String & server, const B
             entry.last_successful_params_hash = params_hash;
             entry.last_successful_authentication_timestamp = current_check_timestamp;
 
-            if (search_results)
-                entry.last_successful_search_results = *search_results;
+            if (role_search_results)
+                entry.last_successful_role_search_results = *role_search_results;
             else
-                entry.last_successful_search_results.clear();
+                entry.last_successful_role_search_results.clear();
         }
         else if (
             entry.last_successful_params_hash != params_hash ||
             (
-                search_params == nullptr ?
-                !entry.last_successful_search_results.empty() :
-                search_params->size() != entry.last_successful_search_results.size()
+                role_search_params == nullptr ?
+                !entry.last_successful_role_search_results.empty() :
+                role_search_params->size() != entry.last_successful_role_search_results.size()
             )
         )
         {

src/Access/ExternalAuthenticators.h

@@ -34,7 +34,7 @@ public:
 
     // The name and readiness of the credentials must be verified before calling these.
     bool checkLDAPCredentials(const String & server, const BasicCredentials & credentials,
-        const LDAPClient::SearchParamsList * search_params = nullptr, LDAPClient::SearchResultsList * search_results = nullptr) const;
+        const LDAPClient::RoleSearchParamsList * role_search_params = nullptr, LDAPClient::SearchResultsList * role_search_results = nullptr) const;
     bool checkKerberosCredentials(const String & realm, const GSSAcceptorContext & credentials) const;
 
     GSSAcceptorContext::Params getKerberosParams() const;
@@ -44,7 +44,7 @@ private:
     {
         std::size_t last_successful_params_hash = 0;
         std::chrono::steady_clock::time_point last_successful_authentication_timestamp;
-        LDAPClient::SearchResultsList last_successful_search_results;
+        LDAPClient::SearchResultsList last_successful_role_search_results;
     };
 
     using LDAPCache = std::unordered_map<String, LDAPCacheEntry>; // user name   -> cache entry
@@ -58,4 +58,6 @@ private:
     std::optional<GSSAcceptorContext::Params> kerberos_params;
 };
 
+void parseLDAPRoleSearchParams(LDAPClient::RoleSearchParams & params, const Poco::Util::AbstractConfiguration & config, const String & prefix);
+
 }

src/Access/LDAPAccessStorage.cpp

@@ -68,34 +68,15 @@ void LDAPAccessStorage::setConfiguration(AccessControlManager * access_control_m
         common_roles_cfg.insert(role_names.begin(), role_names.end());
     }
 
-    LDAPClient::SearchParamsList role_search_params_cfg;
+    LDAPClient::RoleSearchParamsList role_search_params_cfg;
     if (has_role_mapping)
     {
         Poco::Util::AbstractConfiguration::Keys all_keys;
         config.keys(prefix, all_keys);
         for (const auto & key : all_keys)
         {
-            if (key != "role_mapping" && key.find("role_mapping[") != 0)
-                continue;
-
-            const String rm_prefix = prefix_str + key;
-            const String rm_prefix_str = rm_prefix + '.';
-            role_search_params_cfg.emplace_back();
-            auto & rm_params = role_search_params_cfg.back();
-
-            rm_params.base_dn = config.getString(rm_prefix_str + "base_dn", "");
-            rm_params.search_filter = config.getString(rm_prefix_str + "search_filter", "");
-            rm_params.attribute = config.getString(rm_prefix_str + "attribute", "cn");
-            rm_params.prefix = config.getString(rm_prefix_str + "prefix", "");
-
-            auto scope = config.getString(rm_prefix_str + "scope", "subtree");
-            boost::algorithm::to_lower(scope);
-            if (scope == "base")           rm_params.scope = LDAPClient::SearchParams::Scope::BASE;
-            else if (scope == "one_level") rm_params.scope = LDAPClient::SearchParams::Scope::ONE_LEVEL;
-            else if (scope == "subtree")   rm_params.scope = LDAPClient::SearchParams::Scope::SUBTREE;
-            else if (scope == "children")  rm_params.scope = LDAPClient::SearchParams::Scope::CHILDREN;
-            else
-                throw Exception("Invalid value of 'scope' field in '" + key + "' section of LDAP user directory, must be one of 'base', 'one_level', 'subtree', or 'children'", ErrorCodes::BAD_ARGUMENTS);
+            if (key == "role_mapping" || key.find("role_mapping[") == 0)
+                parseLDAPRoleSearchParams(role_search_params_cfg.emplace_back(), config, prefix_str + key);
         }
     }
 
@@ -364,7 +345,7 @@ std::set<String> LDAPAccessStorage::mapExternalRolesNoLock(const LDAPClient::Sea
 
 
 bool LDAPAccessStorage::areLDAPCredentialsValidNoLock(const User & user, const Credentials & credentials,
-    const ExternalAuthenticators & external_authenticators, LDAPClient::SearchResultsList & search_results) const
+    const ExternalAuthenticators & external_authenticators, LDAPClient::SearchResultsList & role_search_results) const
 {
     if (!credentials.isReady())
         return false;
@@ -373,7 +354,7 @@ bool LDAPAccessStorage::areLDAPCredentialsValidNoLock(const User & user, const C
         return false;
 
     if (const auto * basic_credentials = dynamic_cast<const BasicCredentials *>(&credentials))
-        return external_authenticators.checkLDAPCredentials(ldap_server_name, *basic_credentials, &role_search_params, &search_results);
+        return external_authenticators.checkLDAPCredentials(ldap_server_name, *basic_credentials, &role_search_params, &role_search_results);
 
     return false;
 }

src/Access/LDAPAccessStorage.h

@@ -68,12 +68,12 @@ private:
     void updateAssignedRolesNoLock(const UUID & id, const String & user_name, const LDAPClient::SearchResultsList & external_roles) const;
     std::set<String> mapExternalRolesNoLock(const LDAPClient::SearchResultsList & external_roles) const;
     bool areLDAPCredentialsValidNoLock(const User & user, const Credentials & credentials,
-        const ExternalAuthenticators & external_authenticators, LDAPClient::SearchResultsList & search_results) const;
+        const ExternalAuthenticators & external_authenticators, LDAPClient::SearchResultsList & role_search_results) const;
 
     mutable std::recursive_mutex mutex;
     AccessControlManager * access_control_manager = nullptr;
     String ldap_server_name;
-    LDAPClient::SearchParamsList role_search_params;
+    LDAPClient::RoleSearchParamsList role_search_params;
     std::set<String> common_role_names;                         // role name that should be granted to all users at all times
     mutable std::map<String, std::size_t> external_role_hashes; // user name -> LDAPClient::SearchResultsList hash (most recently retrieved and processed)
     mutable std::map<String, std::set<String>> users_per_roles; // role name -> user names (...it should be granted to; may but don't have to exist for common roles)

src/Access/LDAPClient.cpp

@@ -32,6 +32,11 @@ void LDAPClient::SearchParams::combineHash(std::size_t & seed) const
     boost::hash_combine(seed, static_cast<int>(scope));
     boost::hash_combine(seed, search_filter);
     boost::hash_combine(seed, attribute);
+}
+
+void LDAPClient::RoleSearchParams::combineHash(std::size_t & seed) const
+{
+    SearchParams::combineHash(seed);
     boost::hash_combine(seed, prefix);
 }
 
@@ -42,6 +47,9 @@ void LDAPClient::Params::combineCoreHash(std::size_t & seed) const
     boost::hash_combine(seed, bind_dn);
     boost::hash_combine(seed, user);
     boost::hash_combine(seed, password);
+
+    if (user_dn_detection)
+        user_dn_detection->combineHash(seed);
 }
 
 LDAPClient::LDAPClient(const Params & params_)
@@ -286,18 +294,33 @@ void LDAPClient::openConnection()
     if (params.enable_tls == LDAPClient::Params::TLSEnable::YES_STARTTLS)
         diag(ldap_start_tls_s(handle, nullptr, nullptr));
 
+    final_user_name = escapeForLDAP(params.user);
+    final_bind_dn = replacePlaceholders(params.bind_dn, { {"{user_name}", final_user_name} });
+    final_user_dn = final_bind_dn; // The default value... may be updated right after a successful bind.
+
     switch (params.sasl_mechanism)
     {
         case LDAPClient::Params::SASLMechanism::SIMPLE:
         {
-            const auto escaped_user_name = escapeForLDAP(params.user);
-            const auto bind_dn = replacePlaceholders(params.bind_dn, { {"{user_name}", escaped_user_name} });
-
             ::berval cred;
             cred.bv_val = const_cast<char *>(params.password.c_str());
             cred.bv_len = params.password.size();
 
-            diag(ldap_sasl_bind_s(handle, bind_dn.c_str(), LDAP_SASL_SIMPLE, &cred, nullptr, nullptr, nullptr));
+            diag(ldap_sasl_bind_s(handle, final_bind_dn.c_str(), LDAP_SASL_SIMPLE, &cred, nullptr, nullptr, nullptr));
+
+            // Once bound, run the user DN search query and update the default value, if asked.
+            if (params.user_dn_detection)
+            {
+                const auto user_dn_search_results = search(*params.user_dn_detection);
+
+                if (user_dn_search_results.empty())
+                    throw Exception("Failed to detect user DN: empty search results", ErrorCodes::LDAP_ERROR);
+
+                if (user_dn_search_results.size() > 1)
+                    throw Exception("Failed to detect user DN: more than one entry in the search results", ErrorCodes::LDAP_ERROR);
+
+                final_user_dn = *user_dn_search_results.begin();
+            }
 
             break;
         }
@@ -316,6 +339,9 @@ void LDAPClient::closeConnection() noexcept
 
     ldap_unbind_ext_s(handle, nullptr, nullptr);
     handle = nullptr;
+    final_user_name.clear();
+    final_bind_dn.clear();
+    final_user_dn.clear();
 }
 
 LDAPClient::SearchResults LDAPClient::search(const SearchParams & search_params)
@@ -333,10 +359,19 @@ LDAPClient::SearchResults LDAPClient::search(const SearchParams & search_params)
         case SearchParams::Scope::CHILDREN:  scope = LDAP_SCOPE_CHILDREN; break;
     }
 
-    const auto escaped_user_name = escapeForLDAP(params.user);
-    const auto bind_dn = replacePlaceholders(params.bind_dn, { {"{user_name}", escaped_user_name} });
-    const auto base_dn = replacePlaceholders(search_params.base_dn, { {"{user_name}", escaped_user_name}, {"{bind_dn}", bind_dn} });
-    const auto search_filter = replacePlaceholders(search_params.search_filter, { {"{user_name}", escaped_user_name}, {"{bind_dn}", bind_dn}, {"{base_dn}", base_dn} });
+    const auto final_base_dn = replacePlaceholders(search_params.base_dn, {
+        {"{user_name}", final_user_name},
+        {"{bind_dn}", final_bind_dn},
+        {"{user_dn}", final_user_dn}
+    });
+
+    const auto final_search_filter = replacePlaceholders(search_params.search_filter, {
+        {"{user_name}", final_user_name},
+        {"{bind_dn}", final_bind_dn},
+        {"{user_dn}", final_user_dn},
+        {"{base_dn}", final_base_dn}
+    });
+
     char * attrs[] = { const_cast<char *>(search_params.attribute.c_str()), nullptr };
     ::timeval timeout = { params.search_timeout.count(), 0 };
     LDAPMessage* msgs = nullptr;
@@ -349,7 +384,7 @@ LDAPClient::SearchResults LDAPClient::search(const SearchParams & search_params)
         }
     });
 
-    diag(ldap_search_ext_s(handle, base_dn.c_str(), scope, search_filter.c_str(), attrs, 0, nullptr, nullptr, &timeout, params.search_limit, &msgs));
+    diag(ldap_search_ext_s(handle, final_base_dn.c_str(), scope, final_search_filter.c_str(), attrs, 0, nullptr, nullptr, &timeout, params.search_limit, &msgs));
 
     for (
          auto * msg = ldap_first_message(handle, msgs);
@@ -361,6 +396,27 @@ LDAPClient::SearchResults LDAPClient::search(const SearchParams & search_params)
         {
             case LDAP_RES_SEARCH_ENTRY:
             {
+                // Extract DN separately, if the requested attribute is DN.
+                if (boost::iequals("dn", search_params.attribute))
+                {
+                    BerElement * ber = nullptr;
+
+                    SCOPE_EXIT({
+                        if (ber)
+                        {
+                            ber_free(ber, 0);
+                            ber = nullptr;
+                        }
+                    });
+
+                    ::berval bv;
+
+                    diag(ldap_get_dn_ber(handle, msg, &ber, &bv));
+
+                    if (bv.bv_val && bv.bv_len > 0)
+                        result.emplace(bv.bv_val, bv.bv_len);
+                }
+
                 BerElement * ber = nullptr;
 
                 SCOPE_EXIT({
@@ -471,12 +527,12 @@ LDAPClient::SearchResults LDAPClient::search(const SearchParams & search_params)
     return result;
 }
 
-bool LDAPSimpleAuthClient::authenticate(const SearchParamsList * search_params, SearchResultsList * search_results)
+bool LDAPSimpleAuthClient::authenticate(const RoleSearchParamsList * role_search_params, SearchResultsList * role_search_results)
 {
     if (params.user.empty())
         throw Exception("LDAP authentication of a user with empty name is not allowed", ErrorCodes::BAD_ARGUMENTS);
 
-    if (!search_params != !search_results)
+    if (!role_search_params != !role_search_results)
         throw Exception("Cannot return LDAP search results", ErrorCodes::BAD_ARGUMENTS);
 
     // Silently reject authentication attempt if the password is empty as if it didn't match.
@@ -489,21 +545,21 @@ bool LDAPSimpleAuthClient::authenticate(const SearchParamsList * search_params,
     openConnection();
 
     // While connected, run search queries and save the results, if asked.
-    if (search_params)
+    if (role_search_params)
     {
-        search_results->clear();
-        search_results->reserve(search_params->size());
+        role_search_results->clear();
+        role_search_results->reserve(role_search_params->size());
 
         try
         {
-            for (const auto & single_search_params : *search_params)
+            for (const auto & params_instance : *role_search_params)
             {
-                search_results->emplace_back(search(single_search_params));
+                role_search_results->emplace_back(search(params_instance));
             }
         }
         catch (...)
         {
-            search_results->clear();
+            role_search_results->clear();
             throw;
         }
     }
@@ -532,7 +588,7 @@ LDAPClient::SearchResults LDAPClient::search(const SearchParams &)
     throw Exception("ClickHouse was built without LDAP support", ErrorCodes::FEATURE_IS_NOT_ENABLED_AT_BUILD_TIME);
 }
 
-bool LDAPSimpleAuthClient::authenticate(const SearchParamsList *, SearchResultsList *)
+bool LDAPSimpleAuthClient::authenticate(const RoleSearchParamsList *, SearchResultsList *)
 {
     throw Exception("ClickHouse was built without LDAP support", ErrorCodes::FEATURE_IS_NOT_ENABLED_AT_BUILD_TIME);
 }

src/Access/LDAPClient.h

@@ -38,12 +38,20 @@ public:
         Scope scope = Scope::SUBTREE;
         String search_filter;
         String attribute = "cn";
+
+        void combineHash(std::size_t & seed) const;
+    };
+
+    struct RoleSearchParams
+        : public SearchParams
+    {
         String prefix;
 
         void combineHash(std::size_t & seed) const;
     };
 
-    using SearchParamsList = std::vector<SearchParams>;
+    using RoleSearchParamsList = std::vector<RoleSearchParams>;
+
     using SearchResults = std::set<String>;
     using SearchResultsList = std::vector<SearchResults>;
 
@@ -105,6 +113,8 @@ public:
         String user;
         String password;
 
+        std::optional<SearchParams> user_dn_detection;
+
         std::chrono::seconds verification_cooldown{0};
 
         std::chrono::seconds operation_timeout{40};
@@ -134,6 +144,9 @@ protected:
 #if USE_LDAP
     LDAP * handle = nullptr;
 #endif
+    String final_user_name;
+    String final_bind_dn;
+    String final_user_dn;
 };
 
 class LDAPSimpleAuthClient
@@ -141,7 +154,7 @@ class LDAPSimpleAuthClient
 {
 public:
     using LDAPClient::LDAPClient;
-    bool authenticate(const SearchParamsList * search_params, SearchResultsList * search_results);
+    bool authenticate(const RoleSearchParamsList * role_search_params, SearchResultsList * role_search_results);
 };
 
 }

src/CMakeLists.txt

@@ -188,6 +188,7 @@ add_object_library(clickhouse_interpreters_clusterproxy Interpreters/ClusterProx
 add_object_library(clickhouse_interpreters_jit Interpreters/JIT)
 add_object_library(clickhouse_columns Columns)
 add_object_library(clickhouse_storages Storages)
+add_object_library(clickhouse_storages_mysql Storages/MySQL)
 add_object_library(clickhouse_storages_distributed Storages/Distributed)
 add_object_library(clickhouse_storages_mergetree Storages/MergeTree)
 add_object_library(clickhouse_storages_liveview Storages/LiveView)

src/Common/Allocator.h

@@ -99,9 +99,17 @@ public:
     /// Free memory range.
     void free(void * buf, size_t size)
     {
-        checkSize(size);
-        freeNoTrack(buf, size);
-        CurrentMemoryTracker::free(size);
+        try
+        {
+            checkSize(size);
+            freeNoTrack(buf, size);
+            CurrentMemoryTracker::free(size);
+        }
+        catch (...)
+        {
+            DB::tryLogCurrentException("Allocator::free");
+            throw;
+        }
     }
 
     /** Enlarge memory range.

src/Common/Config/ConfigProcessor.cpp

@@ -462,10 +462,19 @@ XMLDocumentPtr ConfigProcessor::processConfig(
     }
     else
     {
-        /// When we can use config embedded in binary.
+        /// These embedded files added during build with some cmake magic.
+        /// Look at the end of programs/sever/CMakeLists.txt.
+        std::string embedded_name;
         if (path == "config.xml")
+            embedded_name = "embedded.xml";
+
+        if (path == "keeper_config.xml")
+            embedded_name = "keeper_embedded.xml";
+
+        /// When we can use config embedded in binary.
+        if (!embedded_name.empty())
         {
-            auto resource = getResource("embedded.xml");
+            auto resource = getResource(embedded_name);
             if (resource.empty())
                 throw Exception(ErrorCodes::FILE_DOESNT_EXIST, "Configuration file {} doesn't exist and there is no embedded config", path);
             LOG_DEBUG(log, "There is no file '{}', will use embedded config.", path);

src/Common/NamePrompter.h

@@ -90,17 +90,16 @@ private:
     }
 };
 
-template <size_t MaxNumHints, class Self>
+
+template <size_t MaxNumHints, typename Self>
 class IHints
 {
 public:
-
     virtual std::vector<String> getAllRegisteredNames() const = 0;
 
     std::vector<String> getHints(const String & name) const
     {
-        static const auto registered_names = getAllRegisteredNames();
-        return prompter.getHints(name, registered_names);
+        return prompter.getHints(name, getAllRegisteredNames());
     }
 
     virtual ~IHints() = default;

src/Common/PODArray.h

@@ -513,7 +513,7 @@ public:
         insertPrepare(from_begin, from_end);
 
         if (unlikely(bytes_to_move))
-            memcpy(this->c_end + bytes_to_copy - bytes_to_move, this->c_end - bytes_to_move, bytes_to_move);
+            memmove(this->c_end + bytes_to_copy - bytes_to_move, this->c_end - bytes_to_move, bytes_to_move);
 
         memcpy(this->c_end - bytes_to_move, reinterpret_cast<const void *>(&*from_begin), bytes_to_copy);
 

src/Common/StringUtils/StringUtils.h

@@ -123,7 +123,7 @@ inline bool isWhitespaceASCII(char c)
 /// Since |isWhiteSpaceASCII()| is used inside algorithms it's easier to implement another function than add extra argument.
 inline bool isWhitespaceASCIIOneLine(char c)
 {
-    return c == ' ' || c == '\t' || c == '\r' || c == '\f' || c == '\v';
+    return c == ' ' || c == '\t' || c == '\f' || c == '\v';
 }
 
 inline bool isControlASCII(char c)

src/Common/examples/integer_hash_tables_benchmark.cpp

@@ -61,6 +61,10 @@ static void NO_INLINE testForType(size_t method, size_t rows_size)
         test<Key, ::absl::flat_hash_map<Key, UInt64>>(data.data(), data.size(), "Abseil HashMap");
     }
     else if (method == 3)
+    {
+        test<Key, ::absl::flat_hash_map<Key, UInt64, DefaultHash<Key>>>(data.data(), data.size(), "Abseil HashMap with CH Hash");
+    }
+    else if (method == 4)
     {
         test<Key, std::unordered_map<Key, UInt64>>(data.data(), data.size(), "std::unordered_map");
     }
@@ -81,50 +85,110 @@ static void NO_INLINE testForType(size_t method, size_t rows_size)
  * ./integer_hash_tables_benchmark 1 $2 100000000 < $1
  * ./integer_hash_tables_benchmark 2 $2 100000000 < $1
  * ./integer_hash_tables_benchmark 3 $2 100000000 < $1
+ * ./integer_hash_tables_benchmark 4 $2 100000000 < $1
  *
- * Results of this benchmark on hits_100m_obfuscated
+ * Results of this benchmark on hits_100m_obfuscated X86-64
  *
  * File hits_100m_obfuscated/201307_1_96_4/WatchID.bin
- * CH HashMap: Elapsed: 7.366 (13575745.933 elem/sec.), map size: 99997493
- * Google DenseMap: Elapsed: 10.089 (9911817.125 elem/sec.), map size: 99997493
- * Abseil HashMap: Elapsed: 9.011 (11097794.073 elem/sec.), map size: 99997493
- * std::unordered_map: Elapsed: 44.758 (2234223.189 elem/sec.), map size: 99997493
+ * CH HashMap: Elapsed: 7.416 (13484217.815 elem/sec.), map size: 99997493
+ * Google DenseMap: Elapsed: 10.303 (9706022.031 elem/sec.), map size: 99997493
+ * Abseil HashMap: Elapsed: 9.106 (10982139.229 elem/sec.), map size: 99997493
+ * Abseil HashMap with CH Hash: Elapsed: 9.221 (10845360.669 elem/sec.), map size: 99997493
+ * std::unordered_map: Elapsed: 45.213 (2211758.706 elem/sec.), map size: 9999749
  *
  * File hits_100m_obfuscated/201307_1_96_4/URLHash.bin
- * CH HashMap: Elapsed: 2.672 (37421588.347 elem/sec.), map size: 20714865
- * Google DenseMap: Elapsed: 3.409 (29333308.209 elem/sec.), map size: 20714865
- * Abseil HashMap: Elapsed: 2.778 (36000540.035 elem/sec.), map size: 20714865
- * std::unordered_map: Elapsed: 8.643 (11570012.207 elem/sec.), map size: 20714865
+ * CH HashMap: Elapsed: 2.620 (38168135.308 elem/sec.), map size: 20714865
+ * Google DenseMap: Elapsed: 3.426 (29189309.058 elem/sec.), map size: 20714865
+ * Abseil HashMap: Elapsed: 2.788 (35870495.097 elem/sec.), map size: 20714865
+ * Abseil HashMap with CH Hash: Elapsed: 2.991 (33428850.155 elem/sec.), map size: 20714865
+ * std::unordered_map: Elapsed: 8.503 (11760331.346 elem/sec.), map size: 20714865
  *
  * File hits_100m_obfuscated/201307_1_96_4/UserID.bin
- * CH HashMap: Elapsed: 2.116 (47267659.076 elem/sec.), map size: 17630976
- * Google DenseMap: Elapsed: 2.722 (36740693.786 elem/sec.), map size: 17630976
- * Abseil HashMap: Elapsed: 2.597 (38509988.663 elem/sec.), map size: 17630976
- * std::unordered_map: Elapsed: 7.327 (13647271.471 elem/sec.), map size: 17630976
+ * CH HashMap: Elapsed: 2.157 (46352039.753 elem/sec.), map size: 17630976
+ * Google DenseMap: Elapsed: 2.725 (36694226.782 elem/sec.), map size: 17630976
+ * Abseil HashMap: Elapsed: 2.590 (38604284.187 elem/sec.), map size: 17630976
+ * Abseil HashMap with CH Hash: Elapsed: 2.785 (35904856.137 elem/sec.), map size: 17630976
+ * std::unordered_map: Elapsed: 7.268 (13759557.609 elem/sec.), map size: 17630976
  *
  * File hits_100m_obfuscated/201307_1_96_4/RegionID.bin
- * CH HashMap: Elapsed: 0.201 (498144193.695 elem/sec.), map size: 9040
- * Google DenseMap: Elapsed: 0.261 (382656387.016 elem/sec.), map size: 9046
- * Abseil HashMap: Elapsed: 0.307 (325874545.117 elem/sec.), map size: 9040
- * std::unordered_map: Elapsed: 0.466 (214379083.420 elem/sec.), map size: 9040
+ * CH HashMap: Elapsed: 0.192 (521583315.810 elem/sec.), map size: 9040
+ * Google DenseMap: Elapsed: 0.297 (337081407.799 elem/sec.), map size: 9046
+ * Abseil HashMap: Elapsed: 0.295 (338805623.511 elem/sec.), map size: 9040
+ * Abseil HashMap with CH Hash: Elapsed: 0.331 (302155391.036 elem/sec.), map size: 9040
+ * std::unordered_map: Elapsed: 0.455 (219971555.390 elem/sec.), map size: 9040
  *
  * File hits_100m_obfuscated/201307_1_96_4/CounterID.bin
- * CH HashMap: Elapsed: 0.220 (455344735.648 elem/sec.), map size: 6506
- * Google DenseMap: Elapsed: 0.297 (336187522.818 elem/sec.), map size: 6506
- * Abseil HashMap: Elapsed: 0.307 (325264214.480 elem/sec.), map size: 6506
- * std::unordered_map: Elapsed: 0.389 (257195996.114 elem/sec.), map size: 6506
+ * CH HashMap: Elapsed: 0.217 (460216823.609 elem/sec.), map size: 6506
+ * Google DenseMap: Elapsed: 0.373 (267838665.098 elem/sec.), map size: 6506
+ * Abseil HashMap: Elapsed: 0.325 (308124728.989 elem/sec.), map size: 6506
+ * Abseil HashMap with CH Hash: Elapsed: 0.354 (282167144.801 elem/sec.), map size: 6506
+ * std::unordered_map: Elapsed: 0.390 (256573354.171 elem/sec.), map size: 6506
  *
  * File hits_100m_obfuscated/201307_1_96_4/TraficSourceID.bin
- * CH HashMap: Elapsed: 0.274 (365196673.729 elem/sec.), map size: 10
- * Google DenseMap: Elapsed: 0.782 (127845746.927 elem/sec.), map size: 1565609 /// Broken because there is 0 key in dataset
- * Abseil HashMap: Elapsed: 0.303 (330461565.053 elem/sec.), map size: 10
- * std::unordered_map: Elapsed: 0.843 (118596530.649 elem/sec.), map size: 10
+ * CH HashMap: Elapsed: 0.246 (406714566.282 elem/sec.), map size: 10
+ * Google DenseMap: Elapsed: 0.760 (131615151.233 elem/sec.), map size: 1565609 /// Broken because there is 0 key in dataset
+ * Abseil HashMap: Elapsed: 0.309 (324068156.680 elem/sec.), map size: 10
+ * Abseil HashMap with CH Hash: Elapsed: 0.339 (295108223.814 elem/sec.), map size: 10
+ * std::unordered_map: Elapsed: 0.811 (123304031.195 elem/sec.), map size: 10
  *
  * File hits_100m_obfuscated/201307_1_96_4/AdvEngineID.bin
- * CH HashMap: Elapsed: 0.160 (623399865.019 elem/sec.), map size: 19
- * Google DenseMap: Elapsed: 1.673 (59757144.027 elem/sec.), map size: 32260732 /// Broken because there is 0 key in dataset
- * Abseil HashMap: Elapsed: 0.297 (336589258.845 elem/sec.), map size: 19
- * std::unordered_map: Elapsed: 0.332 (301114451.384 elem/sec.), map size: 19
+ * CH HashMap: Elapsed: 0.155 (643245257.748 elem/sec.), map size: 19
+ * Google DenseMap: Elapsed: 1.629 (61395025.417 elem/sec.), map size: 32260732 // Broken because there is 0 key in dataset
+ * Abseil HashMap: Elapsed: 0.292 (342765027.204 elem/sec.), map size: 19
+ * Abseil HashMap with CH Hash: Elapsed: 0.330 (302822020.210 elem/sec.), map size: 19
+ * std::unordered_map: Elapsed: 0.308 (325059333.730 elem/sec.), map size: 19
+ *
+ *
+ * Results of this benchmark on hits_100m_obfuscated AARCH64
+ *
+ * File hits_100m_obfuscated/201307_1_96_4/WatchID.bin
+ * CH HashMap: Elapsed: 9.530 (10493528.533 elem/sec.), map size: 99997493
+ * Google DenseMap: Elapsed: 14.436 (6927091.135 elem/sec.), map size: 99997493
+ * Abseil HashMap: Elapsed: 16.671 (5998504.085 elem/sec.), map size: 99997493
+ * Abseil HashMap with CH Hash: Elapsed: 16.803 (5951365.711 elem/sec.), map size: 99997493
+ * std::unordered_map: Elapsed: 50.805 (1968305.658 elem/sec.), map size: 99997493
+ *
+ * File hits_100m_obfuscated/201307_1_96_4/URLHash.bin
+ * CH HashMap: Elapsed: 3.693 (27076878.092 elem/sec.), map size: 20714865
+ * Google DenseMap: Elapsed: 5.051 (19796401.694 elem/sec.), map size: 20714865
+ * Abseil HashMap: Elapsed: 5.617 (17804528.625 elem/sec.), map size: 20714865
+ * Abseil HashMap with CH Hash: Elapsed: 5.702 (17537013.639 elem/sec.), map size: 20714865
+ * std::unordered_map: Elapsed: 10.757 (9296040.953 elem/sec.), map size: 2071486
+ *
+ * File hits_100m_obfuscated/201307_1_96_4/UserID.bin
+ * CH HashMap: Elapsed: 2.982 (33535795.695 elem/sec.), map size: 17630976
+ * Google DenseMap: Elapsed: 3.940 (25381557.959 elem/sec.), map size: 17630976
+ * Abseil HashMap: Elapsed: 4.493 (22259078.458 elem/sec.), map size: 17630976
+ * Abseil HashMap with CH Hash: Elapsed: 4.596 (21759738.710 elem/sec.), map size: 17630976
+ * std::unordered_map: Elapsed: 9.035 (11067903.596 elem/sec.), map size: 17630976
+ *
+ * File hits_100m_obfuscated/201307_1_96_4/RegionID.bin
+ * CH HashMap: Elapsed: 0.302 (331026285.361 elem/sec.), map size: 9040
+ * Google DenseMap: Elapsed: 0.623 (160419421.840 elem/sec.), map size: 9046
+ * Abseil HashMap: Elapsed: 0.981 (101971186.758 elem/sec.), map size: 9040
+ * Abseil HashMap with CH Hash: Elapsed: 0.991 (100932993.199 elem/sec.), map size: 9040
+ * std::unordered_map: Elapsed: 0.809 (123541402.715 elem/sec.), map size: 9040
+ *
+ * File hits_100m_obfuscated/201307_1_96_4/CounterID.bin
+ * CH HashMap: Elapsed: 0.343 (291821742.078 elem/sec.), map size: 6506
+ * Google DenseMap: Elapsed: 0.718 (139191105.450 elem/sec.), map size: 6506
+ * Abseil HashMap: Elapsed: 1.019 (98148285.278 elem/sec.), map size: 6506
+ * Abseil HashMap with CH Hash: Elapsed: 1.048 (95446843.667 elem/sec.), map size: 6506
+ * std::unordered_map: Elapsed: 0.701 (142701070.085 elem/sec.), map size: 6506
+ *
+ * File hits_100m_obfuscated/201307_1_96_4/TraficSourceID.bin
+ * CH HashMap: Elapsed: 0.376 (265905243.103 elem/sec.), map size: 10
+ * Google DenseMap: Elapsed: 1.309 (76420707.298 elem/sec.), map size: 1565609 /// Broken because there is 0 key in dataset
+ * Abseil HashMap: Elapsed: 0.955 (104668109.775 elem/sec.), map size: 10
+ * Abseil HashMap with CH Hash: Elapsed: 0.967 (103456305.391 elem/sec.), map size: 10
+ * std::unordered_map: Elapsed: 1.241 (80591305.890 elem/sec.), map size: 10
+ *
+ * File hits_100m_obfuscated/201307_1_96_4/AdvEngineID.bin
+ * CH HashMap: Elapsed: 0.213 (470208130.105 elem/sec.), map size: 19
+ * Google DenseMap: Elapsed: 2.525 (39607131.523 elem/sec.), map size: 32260732 /// Broken because there is 0 key in dataset
+ * Abseil HashMap: Elapsed: 0.950 (105233678.618 elem/sec.), map size: 19
+ * Abseil HashMap with CH Hash: Elapsed: 0.962 (104001230.717 elem/sec.), map size: 19
+ * std::unordered_map: Elapsed: 0.585 (171059989.837 elem/sec.), map size: 19
  */
 
 int main(int argc, char ** argv)

src/Common/isLocalAddress.cpp

@@ -1,29 +1,88 @@
 #include <Common/isLocalAddress.h>
 
+#include <ifaddrs.h>
 #include <cstring>
+#include <optional>
 #include <common/types.h>
-#include <Poco/Util/Application.h>
-#include <Poco/Net/NetworkInterface.h>
+#include <Common/Exception.h>
+#include <Poco/Net/IPAddress.h>
 #include <Poco/Net/SocketAddress.h>
 
 
 namespace DB
 {
 
+namespace ErrorCodes
+{
+    extern const int SYSTEM_ERROR;
+}
+
+namespace
+{
+
+struct NetworkInterfaces
+{
+    ifaddrs * ifaddr;
+    NetworkInterfaces()
+    {
+        if (getifaddrs(&ifaddr) == -1)
+        {
+            throwFromErrno("Cannot getifaddrs", ErrorCodes::SYSTEM_ERROR);
+        }
+    }
+
+    bool hasAddress(const Poco::Net::IPAddress & address) const
+    {
+        ifaddrs * iface;
+        for (iface = ifaddr; iface != nullptr; iface = iface->ifa_next)
+        {
+            /// Point-to-point (VPN) addresses may have NULL ifa_addr
+            if (!iface->ifa_addr)
+                continue;
+
+            auto family = iface->ifa_addr->sa_family;
+            std::optional<Poco::Net::IPAddress> interface_address;
+            switch (family)
+            {
+                /// We interested only in IP-adresses
+                case AF_INET:
+                {
+                    interface_address.emplace(*(iface->ifa_addr));
+                    break;
+                }
+                case AF_INET6:
+                {
+                    interface_address.emplace(&reinterpret_cast<const struct sockaddr_in6*>(iface->ifa_addr)->sin6_addr, sizeof(struct in6_addr));
+                    break;
+                }
+                default:
+                    continue;
+            }
+
+            /** Compare the addresses without taking into account `scope`.
+              * Theoretically, this may not be correct - depends on `route` setting
+              *  - through which interface we will actually access the specified address.
+              */
+            if (interface_address->length() == address.length()
+                && 0 == memcmp(interface_address->addr(), address.addr(), address.length()))
+                return true;
+        }
+        return false;
+    }
+
+    ~NetworkInterfaces()
+    {
+        freeifaddrs(ifaddr);
+    }
+};
+
+}
+
+
 bool isLocalAddress(const Poco::Net::IPAddress & address)
 {
-    static auto interfaces = Poco::Net::NetworkInterface::list();
-
-    return interfaces.end() != std::find_if(interfaces.begin(), interfaces.end(),
-                [&] (const Poco::Net::NetworkInterface & interface)
-                {
-                    /** Compare the addresses without taking into account `scope`.
-                      * Theoretically, this may not be correct - depends on `route` setting
-                      *  - through which interface we will actually access the specified address.
-                      */
-                    return interface.address().length() == address.length()
-                        && 0 == memcmp(interface.address().addr(), address.addr(), address.length());
-                });
+    NetworkInterfaces interfaces;
+    return interfaces.hasAddress(address);
 }
 
 bool isLocalAddress(const Poco::Net::SocketAddress & address, UInt16 clickhouse_port)

src/Common/tests/gtest_local_address.cpp

@@ -0,0 +1,19 @@
+#include <gtest/gtest.h>
+#include <Common/isLocalAddress.h>
+#include <Common/ShellCommand.h>
+#include <Poco/Net/IPAddress.h>
+#include <IO/ReadHelpers.h>
+
+
+TEST(LocalAddress, SmokeTest)
+{
+    auto cmd = DB::ShellCommand::executeDirect("/bin/hostname", {"-i"});
+    std::string address_str;
+    DB::readString(address_str, cmd->out);
+    cmd->wait();
+    std::cerr << "Got Address:" << address_str << std::endl;
+
+    Poco::Net::IPAddress address(address_str);
+
+    EXPECT_TRUE(DB::isLocalAddress(address));
+}

src/Common/tests/gtest_pod_array.cpp

@@ -419,31 +419,56 @@ TEST(Common, PODArrayBasicSwapMoveConstructor)
 
 TEST(Common, PODArrayInsert)
 {
-    std::string str = "test_string_abacaba";
-    PODArray<char> chars;
-    chars.insert(chars.end(), str.begin(), str.end());
-    EXPECT_EQ(str, std::string(chars.data(), chars.size()));
-
-    std::string insert_in_the_middle = "insert_in_the_middle";
-    auto pos = str.size() / 2;
-    str.insert(str.begin() + pos, insert_in_the_middle.begin(), insert_in_the_middle.end());
-    chars.insert(chars.begin() + pos, insert_in_the_middle.begin(), insert_in_the_middle.end());
-    EXPECT_EQ(str, std::string(chars.data(), chars.size()));
-
-    std::string insert_with_resize;
-    insert_with_resize.reserve(chars.capacity() * 2);
-    char cur_char = 'a';
-    while (insert_with_resize.size() < insert_with_resize.capacity())
     {
-        insert_with_resize += cur_char;
-        if (cur_char == 'z')
-            cur_char = 'a';
-        else
-            ++cur_char;
+        std::string str = "test_string_abacaba";
+        PODArray<char> chars;
+        chars.insert(chars.end(), str.begin(), str.end());
+        EXPECT_EQ(str, std::string(chars.data(), chars.size()));
+
+        std::string insert_in_the_middle = "insert_in_the_middle";
+        auto pos = str.size() / 2;
+        str.insert(str.begin() + pos, insert_in_the_middle.begin(), insert_in_the_middle.end());
+        chars.insert(chars.begin() + pos, insert_in_the_middle.begin(), insert_in_the_middle.end());
+        EXPECT_EQ(str, std::string(chars.data(), chars.size()));
+
+        std::string insert_with_resize;
+        insert_with_resize.reserve(chars.capacity() * 2);
+        char cur_char = 'a';
+        while (insert_with_resize.size() < insert_with_resize.capacity())
+        {
+            insert_with_resize += cur_char;
+            if (cur_char == 'z')
+                cur_char = 'a';
+            else
+                ++cur_char;
+        }
+        str.insert(str.begin(), insert_with_resize.begin(), insert_with_resize.end());
+        chars.insert(chars.begin(), insert_with_resize.begin(), insert_with_resize.end());
+        EXPECT_EQ(str, std::string(chars.data(), chars.size()));
+    }
+    {
+        PODArray<UInt64> values;
+        PODArray<UInt64> values_to_insert;
+
+        for (size_t i = 0; i < 120; ++i)
+            values.emplace_back(i);
+
+        values.insert(values.begin() + 1, values_to_insert.begin(), values_to_insert.end());
+        ASSERT_EQ(values.size(), 120);
+
+        values_to_insert.emplace_back(0);
+        values_to_insert.emplace_back(1);
+
+        values.insert(values.begin() + 1, values_to_insert.begin(), values_to_insert.end());
+        ASSERT_EQ(values.size(), 122);
+
+        values_to_insert.clear();
+        for (size_t i = 0; i < 240; ++i)
+            values_to_insert.emplace_back(i);
+
+        values.insert(values.begin() + 1, values_to_insert.begin(), values_to_insert.end());
+        ASSERT_EQ(values.size(), 362);
     }
-    str.insert(str.begin(), insert_with_resize.begin(), insert_with_resize.end());
-    chars.insert(chars.begin(), insert_with_resize.begin(), insert_with_resize.end());
-    EXPECT_EQ(str, std::string(chars.data(), chars.size()));
 }
 
 TEST(Common, PODArrayInsertFromItself)

src/Common/tests/gtest_sensitive_data_masker.cpp

@@ -223,7 +223,7 @@ TEST(Common, SensitiveDataMasker)
     {
         EXPECT_EQ(
             std::string(e.message()),
-            "SensitiveDataMasker: cannot compile re2: ())(, error: missing ): ())(. Look at https://github.com/google/re2/wiki/Syntax for reference.: while adding query masking rule 'test'."
+            "SensitiveDataMasker: cannot compile re2: ())(, error: unexpected ): ())(. Look at https://github.com/google/re2/wiki/Syntax for reference.: while adding query masking rule 'test'."
         );
         EXPECT_EQ(e.code(), DB::ErrorCodes::CANNOT_COMPILE_REGEXP);
     }

src/Common/ya.make

@@ -18,6 +18,7 @@ PEERDIR(
     contrib/libs/openssl
     contrib/libs/poco/NetSSL_OpenSSL
     contrib/libs/re2
+    contrib/libs/cxxsupp/libcxxabi-parts
     contrib/restricted/dragonbox
 )
 

src/Common/ya.make.in

@@ -17,6 +17,7 @@ PEERDIR(
     contrib/libs/openssl
     contrib/libs/poco/NetSSL_OpenSSL
     contrib/libs/re2
+    contrib/libs/cxxsupp/libcxxabi-parts
     contrib/restricted/dragonbox
 )
 

src/Compression/tests/gtest_compressionCodec.cpp

@@ -345,10 +345,12 @@ CodecTestSequence operator*(CodecTestSequence && left, T times)
 
 std::ostream & operator<<(std::ostream & ostr, const Codec & codec)
 {
-    return ostr << "Codec{"
-                << "name: " << codec.codec_statement
-                << ", expected_compression_ratio: " << *codec.expected_compression_ratio
-                << "}";
+    ostr << "Codec{"
+         << "name: " << codec.codec_statement;
+    if (codec.expected_compression_ratio)
+        return ostr << ", expected_compression_ratio: " << *codec.expected_compression_ratio << "}";
+    else
+        return ostr << "}";
 }
 
 std::ostream & operator<<(std::ostream & ostr, const CodecTestSequence & seq)

src/Coordination/KeeperServer.cpp

@@ -14,6 +14,7 @@
 #include <chrono>
 #include <Common/ZooKeeper/ZooKeeperIO.h>
 #include <string>
+#include <filesystem>
 #include <Poco/Util/Application.h>
 
 namespace DB
@@ -59,6 +60,21 @@ void setSSLParams(nuraft::asio_service::options & asio_opts)
 }
 #endif
 
+std::string getSnapshotsPathFromConfig(const Poco::Util::AbstractConfiguration & config, bool standalone_keeper)
+{
+    /// the most specialized path
+    if (config.has("keeper_server.snapshot_storage_path"))
+        return config.getString("keeper_server.snapshot_storage_path");
+
+    if (config.has("keeper_server.storage_path"))
+        return std::filesystem::path{config.getString("keeper_server.storage_path")} / "snapshots";
+
+    if (standalone_keeper)
+        return std::filesystem::path{config.getString("path", KEEPER_DEFAULT_PATH)} / "snapshots";
+    else
+        return std::filesystem::path{config.getString("path", DBMS_DEFAULT_PATH)} / "coordination/snapshots";
+}
+
 }
 
 KeeperServer::KeeperServer(
@@ -66,14 +82,15 @@ KeeperServer::KeeperServer(
     const CoordinationSettingsPtr & coordination_settings_,
     const Poco::Util::AbstractConfiguration & config,
     ResponsesQueue & responses_queue_,
-    SnapshotsQueue & snapshots_queue_)
+    SnapshotsQueue & snapshots_queue_,
+    bool standalone_keeper)
     : server_id(server_id_)
     , coordination_settings(coordination_settings_)
     , state_machine(nuraft::cs_new<KeeperStateMachine>(
                         responses_queue_, snapshots_queue_,
-                        config.getString("keeper_server.snapshot_storage_path", config.getString("path", DBMS_DEFAULT_PATH) + "coordination/snapshots"),
+                        getSnapshotsPathFromConfig(config, standalone_keeper),
                         coordination_settings))
-    , state_manager(nuraft::cs_new<KeeperStateManager>(server_id, "keeper_server", config, coordination_settings))
+    , state_manager(nuraft::cs_new<KeeperStateManager>(server_id, "keeper_server", config, coordination_settings, standalone_keeper))
     , log(&Poco::Logger::get("KeeperServer"))
 {
     if (coordination_settings->quorum_reads)

src/Coordination/KeeperServer.h

@@ -55,7 +55,8 @@ public:
         const CoordinationSettingsPtr & coordination_settings_,
         const Poco::Util::AbstractConfiguration & config,
         ResponsesQueue & responses_queue_,
-        SnapshotsQueue & snapshots_queue_);
+        SnapshotsQueue & snapshots_queue_,
+        bool standalone_keeper);
 
     void startup();
 

src/Coordination/KeeperStateManager.cpp

@@ -1,5 +1,6 @@
 #include <Coordination/KeeperStateManager.h>
 #include <Common/Exception.h>
+#include <filesystem>
 
 namespace DB
 {
@@ -9,6 +10,26 @@ namespace ErrorCodes
     extern const int RAFT_ERROR;
 }
 
+namespace
+{
+    std::string getLogsPathFromConfig(
+        const std::string & config_prefix, const Poco::Util::AbstractConfiguration & config, bool standalone_keeper)
+{
+    /// the most specialized path
+    if (config.has(config_prefix + ".log_storage_path"))
+        return config.getString(config_prefix + ".log_storage_path");
+
+    if (config.has(config_prefix + ".storage_path"))
+        return std::filesystem::path{config.getString(config_prefix + ".storage_path")} / "logs";
+
+    if (standalone_keeper)
+        return std::filesystem::path{config.getString("path", KEEPER_DEFAULT_PATH)} / "logs";
+    else
+        return std::filesystem::path{config.getString("path", DBMS_DEFAULT_PATH)} / "coordination/logs";
+}
+
+}
+
 KeeperStateManager::KeeperStateManager(int server_id_, const std::string & host, int port, const std::string & logs_path)
     : my_server_id(server_id_)
     , my_port(port)
@@ -24,11 +45,12 @@ KeeperStateManager::KeeperStateManager(
     int my_server_id_,
     const std::string & config_prefix,
     const Poco::Util::AbstractConfiguration & config,
-    const CoordinationSettingsPtr & coordination_settings)
+    const CoordinationSettingsPtr & coordination_settings,
+    bool standalone_keeper)
     : my_server_id(my_server_id_)
     , secure(config.getBool(config_prefix + ".raft_configuration.secure", false))
     , log_store(nuraft::cs_new<KeeperLogStore>(
-                    config.getString(config_prefix + ".log_storage_path", config.getString("path", DBMS_DEFAULT_PATH) + "coordination/logs"),
+                    getLogsPathFromConfig(config_prefix, config, standalone_keeper),
                     coordination_settings->rotate_log_storage_interval, coordination_settings->force_sync))
     , cluster_config(nuraft::cs_new<nuraft::cluster_config>())
 {

src/Coordination/KeeperStateManager.h

@@ -17,7 +17,8 @@ public:
         int server_id_,
         const std::string & config_prefix,
         const Poco::Util::AbstractConfiguration & config,
-        const CoordinationSettingsPtr & coordination_settings);
+        const CoordinationSettingsPtr & coordination_settings,
+        bool standalone_keeper);
 
     KeeperStateManager(
         int server_id_,

src/Coordination/KeeperStorage.cpp

@@ -547,6 +547,17 @@ struct KeeperStorageCloseRequest final : public KeeperStorageRequest
     }
 };
 
+/// Dummy implementation TODO: implement simple ACL
+struct KeeperStorageAuthRequest final : public KeeperStorageRequest
+{
+    using KeeperStorageRequest::KeeperStorageRequest;
+    std::pair<Coordination::ZooKeeperResponsePtr, Undo> process(KeeperStorage::Container &, KeeperStorage::Ephemerals &, int64_t, int64_t) const override
+    {
+        Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
+        return { response_ptr, {} };
+    }
+};
+
 void KeeperStorage::finalize()
 {
     if (finalized)
@@ -611,7 +622,7 @@ KeeperWrapperFactory::KeeperWrapperFactory()
 {
     registerKeeperRequestWrapper<Coordination::OpNum::Heartbeat, KeeperStorageHeartbeatRequest>(*this);
     registerKeeperRequestWrapper<Coordination::OpNum::Sync, KeeperStorageSyncRequest>(*this);
-    //registerKeeperRequestWrapper<Coordination::OpNum::Auth, KeeperStorageAuthRequest>(*this);
+    registerKeeperRequestWrapper<Coordination::OpNum::Auth, KeeperStorageAuthRequest>(*this);
     registerKeeperRequestWrapper<Coordination::OpNum::Close, KeeperStorageCloseRequest>(*this);
     registerKeeperRequestWrapper<Coordination::OpNum::Create, KeeperStorageCreateRequest>(*this);
     registerKeeperRequestWrapper<Coordination::OpNum::Remove, KeeperStorageRemoveRequest>(*this);

src/Coordination/KeeperStorageDispatcher.cpp

@@ -234,7 +234,7 @@ bool KeeperStorageDispatcher::putRequest(const Coordination::ZooKeeperRequestPtr
     return true;
 }
 
-void KeeperStorageDispatcher::initialize(const Poco::Util::AbstractConfiguration & config)
+void KeeperStorageDispatcher::initialize(const Poco::Util::AbstractConfiguration & config, bool standalone_keeper)
 {
     LOG_DEBUG(log, "Initializing storage dispatcher");
     int myid = config.getInt("keeper_server.server_id");
@@ -246,7 +246,8 @@ void KeeperStorageDispatcher::initialize(const Poco::Util::AbstractConfiguration
     responses_thread = ThreadFromGlobalPool([this] { responseThread(); });
     snapshot_thread = ThreadFromGlobalPool([this] { snapshotThread(); });
 
-    server = std::make_unique<KeeperServer>(myid, coordination_settings, config, responses_queue, snapshots_queue);
+    server = std::make_unique<KeeperServer>(
+        myid, coordination_settings, config, responses_queue, snapshots_queue, standalone_keeper);
     try
     {
         LOG_DEBUG(log, "Waiting server to initialize");

src/Coordination/KeeperStorageDispatcher.h

@@ -86,7 +86,7 @@ private:
 public:
     KeeperStorageDispatcher();
 
-    void initialize(const Poco::Util::AbstractConfiguration & config);
+    void initialize(const Poco::Util::AbstractConfiguration & config, bool standalone_keeper);
 
     void shutdown();
 

src/Core/Defines.h

@@ -98,6 +98,8 @@
 
 #define DBMS_DEFAULT_PATH "/var/lib/clickhouse/"
 
+#define KEEPER_DEFAULT_PATH "/var/lib/clickhouse-keeper/"
+
 // more aliases: https://mailman.videolan.org/pipermail/x264-devel/2014-May/010660.html
 
 /// Marks that extra information is sent to a shard. It could be any magic numbers.

src/Core/Settings.h

@@ -115,7 +115,7 @@ class IColumn;
     M(Bool, skip_unavailable_shards, false, "If 1, ClickHouse silently skips unavailable shards and nodes unresolvable through DNS. Shard is marked as unavailable when none of the replicas can be reached.", 0) \
     \
     M(UInt64, parallel_distributed_insert_select, 0, "Process distributed INSERT SELECT query in the same cluster on local tables on every shard, if 1 SELECT is executed on each shard, if 2 SELECT and INSERT is executed on each shard", 0) \
-    M(UInt64, distributed_group_by_no_merge, 0, "If 1, Do not merge aggregation states from different servers for distributed query processing - in case it is for certain that there are different keys on different shards. If 2 - same as 1 but also apply ORDER BY and LIMIT stages", 0) \
+    M(UInt64, distributed_group_by_no_merge, 0, "If 1, Do not merge aggregation states from different servers for distributed queries (shards will process query up to the Complete stage, initiator just proxies the data from the shards). If 2 the initiator will apply ORDER BY and LIMIT stages (it is not in case when shard process query up to the Complete stage)", 0) \
     M(Bool, optimize_distributed_group_by_sharding_key, false, "Optimize GROUP BY sharding_key queries (by avoiding costly aggregation on the initiator server).", 0) \
     M(UInt64, optimize_skip_unused_shards_limit, 1000, "Limit for number of sharding key values, turns off optimize_skip_unused_shards if the limit is reached", 0) \
     M(Bool, optimize_skip_unused_shards, false, "Assumes that data is distributed by sharding_key. Optimization to skip unused shards if SELECT query filters by sharding_key.", 0) \

src/DataStreams/AddingDefaultBlockOutputStream.cpp

@@ -15,7 +15,7 @@ AddingDefaultBlockOutputStream::AddingDefaultBlockOutputStream(
     : output(output_), header(header_)
 {
     auto dag = addMissingDefaults(header_, output->getHeader().getNamesAndTypesList(), columns_, context_, null_as_default_);
-    adding_defaults_actions = std::make_shared<ExpressionActions>(std::move(dag), ExpressionActionsSettings::fromContext(context_));
+    adding_defaults_actions = std::make_shared<ExpressionActions>(std::move(dag), ExpressionActionsSettings::fromContext(context_, CompileExpressions::yes));
 }
 
 void AddingDefaultBlockOutputStream::write(const Block & block)

src/DataStreams/AddingDefaultsBlockInputStream.cpp

@@ -174,7 +174,7 @@ Block AddingDefaultsBlockInputStream::readImpl()
     auto dag = evaluateMissingDefaults(evaluate_block, header.getNamesAndTypesList(), columns, context, false);
     if (dag)
     {
-        auto actions = std::make_shared<ExpressionActions>(std::move(dag), ExpressionActionsSettings::fromContext(context));
+        auto actions = std::make_shared<ExpressionActions>(std::move(dag), ExpressionActionsSettings::fromContext(context, CompileExpressions::yes));
         actions->execute(evaluate_block);
     }
 

src/DataTypes/EnumValues.cpp

@@ -67,7 +67,7 @@ T EnumValues<T>::getValue(StringRef field_name, bool try_treat_as_id) const
                 return x;
         }
         auto hints = this->getHints(field_name.toString());
-        auto hints_string = !hints.empty() ? ", may be you meant: " + toString(hints) : "";
+        auto hints_string = !hints.empty() ? ", maybe you meant: " + toString(hints) : "";
         throw Exception{"Unknown element '" + field_name.toString() + "' for enum" + hints_string, ErrorCodes::BAD_ARGUMENTS};
     }
     return it->getMapped();

src/Databases/MySQL/DatabaseConnectionMySQL.cpp

@@ -20,6 +20,7 @@
 #    include <Parsers/parseQuery.h>
 #    include <Parsers/queryToString.h>
 #    include <Storages/StorageMySQL.h>
+#    include <Storages/MySQL/MySQLSettings.h>
 #    include <Common/escapeForFileName.h>
 #    include <Common/parseAddress.h>
 #    include <Common/setThreadName.h>
@@ -253,12 +254,13 @@ void DatabaseConnectionMySQL::fetchLatestTablesStructureIntoCache(
                 std::move(mysql_pool),
                 database_name_in_mysql,
                 table_name,
-                false,
-                "",
+                /* replace_query_ */ false,
+                /* on_duplicate_clause = */ "",
                 ColumnsDescription{columns_name_and_type},
                 ConstraintsDescription{},
                 String{},
-                getContext()));
+                getContext(),
+                MySQLSettings{}));
     }
 }
 

src/Functions/FunctionsComparison.h

@@ -1147,17 +1147,24 @@ public:
         /// NOTE: We consider NaN comparison to be implementation specific (and in our implementation NaNs are sometimes equal sometimes not).
         if (left_type->equals(*right_type) && !left_type->isNullable() && !isTuple(left_type) && col_left_untyped == col_right_untyped)
         {
+            ColumnPtr result_column;
+
             /// Always true: =, <=, >=
             if constexpr (IsOperation<Op>::equals
                 || IsOperation<Op>::less_or_equals
                 || IsOperation<Op>::greater_or_equals)
             {
-                return DataTypeUInt8().createColumnConst(input_rows_count, 1u);
+                result_column = DataTypeUInt8().createColumnConst(input_rows_count, 1u);
             }
             else
             {
-                return DataTypeUInt8().createColumnConst(input_rows_count, 0u);
+                result_column = DataTypeUInt8().createColumnConst(input_rows_count, 0u);
             }
+
+            if (!isColumnConst(*col_left_untyped))
+                result_column = result_column->convertToFullColumnIfConst();
+
+            return result_column;
         }
 
         WhichDataType which_left{left_type};

src/Functions/IFunction.cpp

@@ -61,13 +61,14 @@ ColumnPtr replaceLowCardinalityColumnsByNestedAndGetDictionaryIndexes(
         {
             /// Single LowCardinality column is supported now.
             if (indexes)
-                throw Exception("Expected single dictionary argument for function.", ErrorCodes::LOGICAL_ERROR);
+                throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected single dictionary argument for function.");
 
             const auto * low_cardinality_type = checkAndGetDataType<DataTypeLowCardinality>(column.type.get());
 
             if (!low_cardinality_type)
-                throw Exception("Incompatible type for low cardinality column: " + column.type->getName(),
-                                ErrorCodes::LOGICAL_ERROR);
+                throw Exception(ErrorCodes::LOGICAL_ERROR,
+                    "Incompatible type for low cardinality column: {}",
+                    column.type->getName());
 
             if (can_be_executed_on_default_arguments)
             {
@@ -121,7 +122,10 @@ ColumnPtr IExecutableFunction::defaultImplementationForConstantArguments(
     /// Check that these arguments are really constant.
     for (auto arg_num : arguments_to_remain_constants)
         if (arg_num < args.size() && !isColumnConst(*args[arg_num].column))
-            throw Exception("Argument at index " + toString(arg_num) + " for function " + getName() + " must be constant", ErrorCodes::ILLEGAL_COLUMN);
+            throw Exception(ErrorCodes::ILLEGAL_COLUMN,
+                "Argument at index {} for function {} must be constant",
+                toString(arg_num),
+                getName());
 
     if (args.empty() || !useDefaultImplementationForConstants() || !allArgumentsAreConstants(args))
         return nullptr;
@@ -150,8 +154,9 @@ ColumnPtr IExecutableFunction::defaultImplementationForConstantArguments(
       *  not in "arguments_to_remain_constants" set. Otherwise we get infinite recursion.
       */
     if (!have_converted_columns)
-        throw Exception("Number of arguments for function " + getName() + " doesn't match: the function requires more arguments",
-            ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
+        throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
+            "Number of arguments for function {} doesn't match: the function requires more arguments",
+            getName());
 
     ColumnPtr result_column = executeWithoutLowCardinalityColumns(temporary_columns, result_type, 1, dry_run);
 
@@ -266,9 +271,11 @@ void IFunctionOverloadResolver::checkNumberOfArguments(size_t number_of_argument
     size_t expected_number_of_arguments = getNumberOfArguments();
 
     if (number_of_arguments != expected_number_of_arguments)
-        throw Exception("Number of arguments for function " + getName() + " doesn't match: passed "
-                        + toString(number_of_arguments) + ", should be " + toString(expected_number_of_arguments),
-                        ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
+        throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
+            "Number of arguments for function {} doesn't match: passed {}, should be {}",
+            getName(),
+            toString(number_of_arguments),
+            toString(expected_number_of_arguments));
 }
 
 DataTypePtr IFunctionOverloadResolver::getReturnType(const ColumnsWithTypeAndName & arguments) const
@@ -299,11 +306,7 @@ DataTypePtr IFunctionOverloadResolver::getReturnType(const ColumnsWithTypeAndNam
                 ++num_full_ordinary_columns;
         }
 
-        for (auto & arg : args_without_low_cardinality)
-        {
-            arg.column = recursiveRemoveLowCardinality(arg.column);
-            arg.type = recursiveRemoveLowCardinality(arg.type);
-        }
+        convertLowCardinalityColumnsToFull(args_without_low_cardinality);
 
         auto type_without_low_cardinality = getReturnTypeWithoutLowCardinality(args_without_low_cardinality);
 

src/Functions/IFunction.h

@@ -245,6 +245,8 @@ public:
 
     void getLambdaArgumentTypes(DataTypes & arguments) const;
 
+    void checkNumberOfArguments(size_t number_of_arguments) const;
+
     /// Get the main function name.
     virtual String getName() const = 0;
 
@@ -319,8 +321,6 @@ protected:
 
 private:
 
-    void checkNumberOfArguments(size_t number_of_arguments) const;
-
     DataTypePtr getReturnTypeWithoutLowCardinality(const ColumnsWithTypeAndName & arguments) const;
 };
 

src/IO/HashingWriteBuffer.h

@@ -17,7 +17,7 @@ class IHashingBuffer : public BufferWithOwnMemory<Buffer>
 public:
     using uint128 = CityHash_v1_0_2::uint128;
 
-    IHashingBuffer<Buffer>(size_t block_size_ = DBMS_DEFAULT_HASHING_BLOCK_SIZE)
+    IHashingBuffer(size_t block_size_ = DBMS_DEFAULT_HASHING_BLOCK_SIZE)
         : BufferWithOwnMemory<Buffer>(block_size_), block_pos(0), block_size(block_size_), state(0, 0)
     {
     }

src/Interpreters/ActionsDAG.cpp

@@ -293,6 +293,17 @@ NamesAndTypesList ActionsDAG::getRequiredColumns() const
     return result;
 }
 
+Names ActionsDAG::getRequiredColumnsNames() const
+{
+    Names result;
+    result.reserve(inputs.size());
+
+    for (const auto & input : inputs)
+        result.emplace_back(input->result_name);
+
+    return result;
+}
+
 ColumnsWithTypeAndName ActionsDAG::getResultColumns() const
 {
     ColumnsWithTypeAndName result;
@@ -1041,11 +1052,19 @@ ActionsDAGPtr ActionsDAG::makeConvertingActions(
             {
                 auto & input = inputs[res_elem.name];
                 if (input.empty())
-                    throw Exception("Cannot find column " + backQuote(res_elem.name) + " in source stream",
-                                    ErrorCodes::THERE_IS_NO_COLUMN);
-
-                src_node = dst_node = actions_dag->inputs[input.front()];
-                input.pop_front();
+                {
+                    const auto * res_const = typeid_cast<const ColumnConst *>(res_elem.column.get());
+                    if (ignore_constant_values && res_const)
+                        src_node = dst_node = &actions_dag->addColumn(res_elem);
+                    else
+                        throw Exception("Cannot find column " + backQuote(res_elem.name) + " in source stream",
+                                        ErrorCodes::THERE_IS_NO_COLUMN);
+                }
+                else
+                {
+                    src_node = dst_node = actions_dag->inputs[input.front()];
+                    input.pop_front();
+                }
                 break;
             }
         }

src/Interpreters/ActionsDAG.h

@@ -121,6 +121,7 @@ public:
     const NodeRawConstPtrs & getInputs() const { return inputs; }
 
     NamesAndTypesList getRequiredColumns() const;
+    Names getRequiredColumnsNames() const;
     ColumnsWithTypeAndName getResultColumns() const;
     NamesAndTypesList getNamesAndTypesList() const;
 

src/Interpreters/ActionsVisitor.cpp

@@ -1015,7 +1015,7 @@ void ActionsMatcher::visit(const ASTFunction & node, const ASTPtr & ast, Data &
 
                     auto lambda_actions = std::make_shared<ExpressionActions>(
                         lambda_dag,
-                        ExpressionActionsSettings::fromContext(data.getContext()));
+                        ExpressionActionsSettings::fromContext(data.getContext(), CompileExpressions::yes));
 
                     DataTypePtr result_type = lambda_actions->getSampleBlock().getByName(result_name).type;
 

src/Interpreters/ClusterProxy/SelectStreamFactory.cpp

@@ -8,11 +8,13 @@
 #include <Common/checkStackSize.h>
 #include <TableFunctions/TableFunctionFactory.h>
 #include <IO/ConnectionTimeoutsContext.h>
+#include <Interpreters/RequiredSourceColumnsVisitor.h>
 
 #include <common/logger_useful.h>
 #include <Processors/Pipe.h>
 #include <Processors/Sources/RemoteSource.h>
 #include <Processors/Sources/DelayedSource.h>
+#include <Processors/Transforms/ExpressionTransform.h>
 #include <Processors/QueryPlan/QueryPlan.h>
 #include <Processors/QueryPlan/ExpressionStep.h>
 #include <Processors/QueryPlan/BuildQueryPipelineSettings.h>
@@ -73,6 +75,95 @@ SelectStreamFactory::SelectStreamFactory(
 namespace
 {
 
+/// Special support for the case when `_shard_num` column is used in GROUP BY key expression.
+/// This column is a constant for shard.
+/// Constant expression with this column may be removed from intermediate header.
+/// However, this column is not constant for initiator, and it expect intermediate header has it.
+///
+/// To fix it, the following trick is applied.
+/// We check all GROUP BY keys which depend only on `_shard_num`.
+/// Calculate such expression for current shard if it is used in header.
+/// Those columns will be added to modified header as already known constants.
+///
+/// For local shard, missed constants will be added by converting actions.
+/// For remote shard, RemoteQueryExecutor will automatically add missing constant.
+Block evaluateConstantGroupByKeysWithShardNumber(
+    const ContextPtr & context, const ASTPtr & query_ast, const Block & header, UInt32 shard_num)
+{
+    Block res;
+
+    ColumnWithTypeAndName shard_num_col;
+    shard_num_col.type = std::make_shared<DataTypeUInt32>();
+    shard_num_col.column = shard_num_col.type->createColumnConst(0, shard_num);
+    shard_num_col.name = "_shard_num";
+
+    if (auto group_by = query_ast->as<ASTSelectQuery &>().groupBy())
+    {
+        for (const auto & elem : group_by->children)
+        {
+            String key_name = elem->getColumnName();
+            if (header.has(key_name))
+            {
+                auto ast = elem->clone();
+
+                RequiredSourceColumnsVisitor::Data columns_context;
+                RequiredSourceColumnsVisitor(columns_context).visit(ast);
+
+                auto required_columns = columns_context.requiredColumns();
+                if (required_columns.size() != 1 || required_columns.count("_shard_num") == 0)
+                    continue;
+
+                Block block({shard_num_col});
+                auto syntax_result = TreeRewriter(context).analyze(ast, {NameAndTypePair{shard_num_col.name, shard_num_col.type}});
+                ExpressionAnalyzer(ast, syntax_result, context).getActions(true, false)->execute(block);
+
+                res.insert(block.getByName(key_name));
+            }
+        }
+    }
+
+    /// We always add _shard_num constant just in case.
+    /// For initial query it is considered as a column from table, and may be required by intermediate block.
+    if (!res.has(shard_num_col.name))
+        res.insert(std::move(shard_num_col));
+
+    return res;
+}
+
+ActionsDAGPtr getConvertingDAG(const Block & block, const Block & header)
+{
+    /// Convert header structure to expected.
+    /// Also we ignore constants from result and replace it with constants from header.
+    /// It is needed for functions like `now64()` or `randConstant()` because their values may be different.
+    return ActionsDAG::makeConvertingActions(
+        block.getColumnsWithTypeAndName(),
+        header.getColumnsWithTypeAndName(),
+        ActionsDAG::MatchColumnsMode::Name,
+        true);
+}
+
+void addConvertingActions(QueryPlan & plan, const Block & header)
+{
+    if (blocksHaveEqualStructure(plan.getCurrentDataStream().header, header))
+        return;
+
+    auto convert_actions_dag = getConvertingDAG(plan.getCurrentDataStream().header, header);
+    auto converting = std::make_unique<ExpressionStep>(plan.getCurrentDataStream(), convert_actions_dag);
+    plan.addStep(std::move(converting));
+}
+
+void addConvertingActions(Pipe & pipe, const Block & header)
+{
+    if (blocksHaveEqualStructure(pipe.getHeader(), header))
+        return;
+
+    auto convert_actions = std::make_shared<ExpressionActions>(getConvertingDAG(pipe.getHeader(), header));
+    pipe.addSimpleTransform([&](const Block & cur_header, Pipe::StreamType) -> ProcessorPtr
+    {
+        return std::make_shared<ExpressionTransform>(cur_header, convert_actions);
+    });
+}
+
 std::unique_ptr<QueryPlan> createLocalPlan(
     const ASTPtr & query_ast,
     const Block & header,
@@ -86,18 +177,7 @@ std::unique_ptr<QueryPlan> createLocalPlan(
     InterpreterSelectQuery interpreter(query_ast, context, SelectQueryOptions(processed_stage));
     interpreter.buildQueryPlan(*query_plan);
 
-    /// Convert header structure to expected.
-    /// Also we ignore constants from result and replace it with constants from header.
-    /// It is needed for functions like `now64()` or `randConstant()` because their values may be different.
-    auto convert_actions_dag = ActionsDAG::makeConvertingActions(
-            query_plan->getCurrentDataStream().header.getColumnsWithTypeAndName(),
-            header.getColumnsWithTypeAndName(),
-            ActionsDAG::MatchColumnsMode::Name,
-            true);
-
-    auto converting = std::make_unique<ExpressionStep>(query_plan->getCurrentDataStream(), convert_actions_dag);
-    converting->setStepDescription("Convert block structure for query from local replica");
-    query_plan->addStep(std::move(converting));
+    addConvertingActions(*query_plan, header);
 
     return query_plan;
 }
@@ -134,12 +214,25 @@ void SelectStreamFactory::createForShard(
     }
 
     auto modified_query_ast = query_ast->clone();
+    auto modified_header = header;
     if (has_virtual_shard_num_column)
+    {
         VirtualColumnUtils::rewriteEntityInAst(modified_query_ast, "_shard_num", shard_info.shard_num, "toUInt32");
+        auto shard_num_constants = evaluateConstantGroupByKeysWithShardNumber(context, query_ast, modified_header, shard_info.shard_num);
+
+        for (auto & col : shard_num_constants)
+        {
+            if (modified_header.has(col.name))
+                modified_header.getByName(col.name).column = std::move(col.column);
+            else
+                modified_header.insert(std::move(col));
+        }
+    }
 
     auto emplace_local_stream = [&]()
     {
-        plans.emplace_back(createLocalPlan(modified_query_ast, header, context, processed_stage));
+        plans.emplace_back(createLocalPlan(modified_query_ast, modified_header, context, processed_stage));
+        addConvertingActions(*plans.back(), header);
     };
 
     String modified_query = formattedAST(modified_query_ast);
@@ -147,7 +240,7 @@ void SelectStreamFactory::createForShard(
     auto emplace_remote_stream = [&]()
     {
         auto remote_query_executor = std::make_shared<RemoteQueryExecutor>(
-            shard_info.pool, modified_query, header, context, throttler, scalars, external_tables, processed_stage);
+            shard_info.pool, modified_query, modified_header, context, throttler, scalars, external_tables, processed_stage);
         remote_query_executor->setLogger(log);
 
         remote_query_executor->setPoolMode(PoolMode::GET_MANY);
@@ -156,6 +249,7 @@ void SelectStreamFactory::createForShard(
 
         remote_pipes.emplace_back(createRemoteSourcePipe(remote_query_executor, add_agg_info, add_totals, add_extremes, async_read));
         remote_pipes.back().addInterpreterContext(context);
+        addConvertingActions(remote_pipes.back(), header);
     };
 
     const auto & settings = context->getSettingsRef();
@@ -247,7 +341,7 @@ void SelectStreamFactory::createForShard(
         /// Do it lazily to avoid connecting in the main thread.
 
         auto lazily_create_stream = [
-                pool = shard_info.pool, shard_num = shard_info.shard_num, modified_query, header = header, modified_query_ast,
+                pool = shard_info.pool, shard_num = shard_info.shard_num, modified_query, header = modified_header, modified_query_ast,
                 context, throttler,
                 main_table = main_table, table_func_ptr = table_func_ptr, scalars = scalars, external_tables = external_tables,
                 stage = processed_stage, local_delay, add_agg_info, add_totals, add_extremes, async_read]()
@@ -302,8 +396,9 @@ void SelectStreamFactory::createForShard(
             }
         };
 
-        delayed_pipes.emplace_back(createDelayedPipe(header, lazily_create_stream, add_totals, add_extremes));
+        delayed_pipes.emplace_back(createDelayedPipe(modified_header, lazily_create_stream, add_totals, add_extremes));
         delayed_pipes.back().addInterpreterContext(context);
+        addConvertingActions(delayed_pipes.back(), header);
     }
     else
         emplace_remote_stream();

src/Interpreters/Context.cpp

@@ -314,8 +314,8 @@ struct ContextSharedPart
     ConfigurationPtr zookeeper_config;                      /// Stores zookeeper configs
 
 #if USE_NURAFT
-    mutable std::mutex nu_keeper_storage_dispatcher_mutex;
-    mutable std::shared_ptr<KeeperStorageDispatcher> nu_keeper_storage_dispatcher;
+    mutable std::mutex keeper_storage_dispatcher_mutex;
+    mutable std::shared_ptr<KeeperStorageDispatcher> keeper_storage_dispatcher;
 #endif
     mutable std::mutex auxiliary_zookeepers_mutex;
     mutable std::map<String, zkutil::ZooKeeperPtr> auxiliary_zookeepers;    /// Map for auxiliary ZooKeeper clients.
@@ -1678,16 +1678,16 @@ zkutil::ZooKeeperPtr Context::getZooKeeper() const
 void Context::initializeKeeperStorageDispatcher() const
 {
 #if USE_NURAFT
-    std::lock_guard lock(shared->nu_keeper_storage_dispatcher_mutex);
+    std::lock_guard lock(shared->keeper_storage_dispatcher_mutex);
 
-    if (shared->nu_keeper_storage_dispatcher)
+    if (shared->keeper_storage_dispatcher)
         throw Exception(ErrorCodes::LOGICAL_ERROR, "Trying to initialize Keeper multiple times");
 
     const auto & config = getConfigRef();
     if (config.has("keeper_server"))
     {
-        shared->nu_keeper_storage_dispatcher = std::make_shared<KeeperStorageDispatcher>();
-        shared->nu_keeper_storage_dispatcher->initialize(config);
+        shared->keeper_storage_dispatcher = std::make_shared<KeeperStorageDispatcher>();
+        shared->keeper_storage_dispatcher->initialize(config, getApplicationType() == ApplicationType::KEEPER);
     }
 #endif
 }
@@ -1695,22 +1695,22 @@ void Context::initializeKeeperStorageDispatcher() const
 #if USE_NURAFT
 std::shared_ptr<KeeperStorageDispatcher> & Context::getKeeperStorageDispatcher() const
 {
-    std::lock_guard lock(shared->nu_keeper_storage_dispatcher_mutex);
-    if (!shared->nu_keeper_storage_dispatcher)
+    std::lock_guard lock(shared->keeper_storage_dispatcher_mutex);
+    if (!shared->keeper_storage_dispatcher)
         throw Exception(ErrorCodes::LOGICAL_ERROR, "Keeper must be initialized before requests");
 
-    return shared->nu_keeper_storage_dispatcher;
+    return shared->keeper_storage_dispatcher;
 }
 #endif
 
 void Context::shutdownKeeperStorageDispatcher() const
 {
 #if USE_NURAFT
-    std::lock_guard lock(shared->nu_keeper_storage_dispatcher_mutex);
-    if (shared->nu_keeper_storage_dispatcher)
+    std::lock_guard lock(shared->keeper_storage_dispatcher_mutex);
+    if (shared->keeper_storage_dispatcher)
     {
-        shared->nu_keeper_storage_dispatcher->shutdown();
-        shared->nu_keeper_storage_dispatcher.reset();
+        shared->keeper_storage_dispatcher->shutdown();
+        shared->keeper_storage_dispatcher.reset();
     }
 #endif
 }

src/Interpreters/Context.h

@@ -734,7 +734,8 @@ public:
     {
         SERVER,         /// The program is run as clickhouse-server daemon (default behavior)
         CLIENT,         /// clickhouse-client
-        LOCAL           /// clickhouse-local
+        LOCAL,          /// clickhouse-local
+        KEEPER,         /// clickhouse-keeper (also daemon)
     };
 
     ApplicationType getApplicationType() const;

src/Interpreters/ExpressionActions.cpp

@@ -51,7 +51,7 @@ ExpressionActions::ExpressionActions(ActionsDAGPtr actions_dag_, const Expressio
     actions_dag = actions_dag_->clone();
 
 #if USE_EMBEDDED_COMPILER
-    if (settings.compile_expressions)
+    if (settings.can_compile_expressions && settings.compile_expressions == CompileExpressions::yes)
         actions_dag->compileExpressions(settings.min_count_to_compile_expression);
 #endif
 

src/Interpreters/ExpressionActions.h

@@ -30,7 +30,6 @@ using ArrayJoinActionPtr = std::shared_ptr<ArrayJoinAction>;
 class ExpressionActions;
 using ExpressionActionsPtr = std::shared_ptr<ExpressionActions>;
 
-
 /// Sequence of actions on the block.
 /// Is used to calculate expressions.
 ///

src/Interpreters/ExpressionActionsSettings.cpp

@@ -6,20 +6,21 @@
 namespace DB
 {
 
-ExpressionActionsSettings ExpressionActionsSettings::fromSettings(const Settings & from)
+ExpressionActionsSettings ExpressionActionsSettings::fromSettings(const Settings & from, CompileExpressions compile_expressions)
 {
     ExpressionActionsSettings settings;
-    settings.compile_expressions = from.compile_expressions;
+    settings.can_compile_expressions = from.compile_expressions;
     settings.min_count_to_compile_expression = from.min_count_to_compile_expression;
     settings.max_temporary_columns = from.max_temporary_columns;
     settings.max_temporary_non_const_columns = from.max_temporary_non_const_columns;
+    settings.compile_expressions = compile_expressions;
 
     return settings;
 }
 
-ExpressionActionsSettings ExpressionActionsSettings::fromContext(ContextPtr from)
+ExpressionActionsSettings ExpressionActionsSettings::fromContext(ContextPtr from, CompileExpressions compile_expressions)
 {
-    return fromSettings(from->getSettingsRef());
+    return fromSettings(from->getSettingsRef(), compile_expressions);
 }
 
 }

src/Interpreters/ExpressionActionsSettings.h

@@ -9,16 +9,24 @@ namespace DB
 
 struct Settings;
 
+enum class CompileExpressions: uint8_t
+{
+    no = 0,
+    yes = 1,
+};
+
 struct ExpressionActionsSettings
 {
-    bool compile_expressions = false;
+    bool can_compile_expressions = false;
     size_t min_count_to_compile_expression = 0;
 
     size_t max_temporary_columns = 0;
     size_t max_temporary_non_const_columns = 0;
 
-    static ExpressionActionsSettings fromSettings(const Settings & from);
-    static ExpressionActionsSettings fromContext(ContextPtr from);
+    CompileExpressions compile_expressions = CompileExpressions::no;
+
+    static ExpressionActionsSettings fromSettings(const Settings & from, CompileExpressions compile_expressions = CompileExpressions::no);
+    static ExpressionActionsSettings fromContext(ContextPtr from, CompileExpressions compile_expressions = CompileExpressions::no);
 };
 
 }

src/Interpreters/ExpressionAnalyzer.cpp

@@ -909,8 +909,8 @@ ActionsDAGPtr SelectQueryExpressionAnalyzer::appendPrewhere(
         auto tmp_actions_dag = std::make_shared<ActionsDAG>(sourceColumns());
         getRootActions(select_query->prewhere(), only_types, tmp_actions_dag);
         tmp_actions_dag->removeUnusedActions(NameSet{prewhere_column_name});
-        auto tmp_actions = std::make_shared<ExpressionActions>(tmp_actions_dag, ExpressionActionsSettings::fromContext(getContext()));
-        auto required_columns = tmp_actions->getRequiredColumns();
+
+        auto required_columns = tmp_actions_dag->getRequiredColumnsNames();
         NameSet required_source_columns(required_columns.begin(), required_columns.end());
         required_source_columns.insert(first_action_names.begin(), first_action_names.end());
 
@@ -1028,7 +1028,7 @@ bool SelectQueryExpressionAnalyzer::appendGroupBy(ExpressionActionsChain & chain
             auto actions_dag = std::make_shared<ActionsDAG>(columns_after_join);
             getRootActions(child, only_types, actions_dag);
             group_by_elements_actions.emplace_back(
-                std::make_shared<ExpressionActions>(actions_dag, ExpressionActionsSettings::fromContext(getContext())));
+                std::make_shared<ExpressionActions>(actions_dag, ExpressionActionsSettings::fromContext(getContext(), CompileExpressions::yes)));
         }
     }
 
@@ -1187,7 +1187,7 @@ ActionsDAGPtr SelectQueryExpressionAnalyzer::appendOrderBy(ExpressionActionsChai
             auto actions_dag = std::make_shared<ActionsDAG>(columns_after_join);
             getRootActions(child, only_types, actions_dag);
             order_by_elements_actions.emplace_back(
-                std::make_shared<ExpressionActions>(actions_dag, ExpressionActionsSettings::fromContext(getContext())));
+                std::make_shared<ExpressionActions>(actions_dag, ExpressionActionsSettings::fromContext(getContext(), CompileExpressions::yes)));
         }
     }
 
@@ -1345,13 +1345,12 @@ ActionsDAGPtr ExpressionAnalyzer::getActionsDAG(bool add_aliases, bool project_r
     return actions_dag;
 }
 
-ExpressionActionsPtr ExpressionAnalyzer::getActions(bool add_aliases, bool project_result)
+ExpressionActionsPtr ExpressionAnalyzer::getActions(bool add_aliases, bool project_result, CompileExpressions compile_expressions)
 {
     return std::make_shared<ExpressionActions>(
-        getActionsDAG(add_aliases, project_result), ExpressionActionsSettings::fromContext(getContext()));
+        getActionsDAG(add_aliases, project_result), ExpressionActionsSettings::fromContext(getContext(), compile_expressions));
 }
 
-
 ExpressionActionsPtr ExpressionAnalyzer::getConstActions(const ColumnsWithTypeAndName & constant_inputs)
 {
     auto actions = std::make_shared<ActionsDAG>(constant_inputs);

src/Interpreters/ExpressionAnalyzer.h

@@ -112,7 +112,7 @@ public:
     ///     If also project_result, than only aliases remain in the output block.
     /// Otherwise, only temporary columns will be deleted from the block.
     ActionsDAGPtr getActionsDAG(bool add_aliases, bool project_result = true);
-    ExpressionActionsPtr getActions(bool add_aliases, bool project_result = true);
+    ExpressionActionsPtr getActions(bool add_aliases, bool project_result = true, CompileExpressions compile_expressions = CompileExpressions::no);
 
     /// Actions that can be performed on an empty block: adding constants and applying functions that depend only on constants.
     /// Does not execute subqueries.

src/Interpreters/ExpressionJIT.cpp

@@ -315,28 +315,6 @@ static bool isCompilableConstant(const ActionsDAG::Node & node)
     return node.column && isColumnConst(*node.column) && canBeNativeType(*node.result_type) && node.allow_constant_folding;
 }
 
-static bool checkIfFunctionIsComparisonEdgeCase(const ActionsDAG::Node & node, const IFunctionBase & impl)
-{
-    static std::unordered_set<std::string_view> comparison_functions
-    {
-        NameEquals::name,
-        NameNotEquals::name,
-        NameLess::name,
-        NameGreater::name,
-        NameLessOrEquals::name,
-        NameGreaterOrEquals::name
-    };
-
-    auto it = comparison_functions.find(impl.getName());
-    if (it == comparison_functions.end())
-        return false;
-
-    const auto * lhs_node = node.children[0];
-    const auto * rhs_node = node.children[1];
-
-    return lhs_node == rhs_node && !isTuple(lhs_node->result_type);
-}
-
 static bool isCompilableFunction(const ActionsDAG::Node & node)
 {
     if (node.type != ActionsDAG::ActionType::FUNCTION)
@@ -353,9 +331,6 @@ static bool isCompilableFunction(const ActionsDAG::Node & node)
             return false;
     }
 
-    if (checkIfFunctionIsComparisonEdgeCase(node, *node.function_base))
-        return false;
-
     return function.isCompilable();
 }
 

src/Interpreters/InterpreterInsertQuery.cpp

@@ -325,7 +325,7 @@ BlockIO InterpreterInsertQuery::execute()
                 res.pipeline.getHeader().getColumnsWithTypeAndName(),
                 header.getColumnsWithTypeAndName(),
                 ActionsDAG::MatchColumnsMode::Position);
-        auto actions = std::make_shared<ExpressionActions>(actions_dag, ExpressionActionsSettings::fromContext(getContext()));
+        auto actions = std::make_shared<ExpressionActions>(actions_dag, ExpressionActionsSettings::fromContext(getContext(), CompileExpressions::yes));
 
         res.pipeline.addSimpleTransform([&](const Block & in_header) -> ProcessorPtr
         {

src/Interpreters/InterpreterSelectQuery.cpp

@@ -593,6 +593,7 @@ Block InterpreterSelectQuery::getSampleBlockImpl()
     OpenTelemetrySpanHolder span(__PRETTY_FUNCTION__);
 
     query_info.query = query_ptr;
+    query_info.has_window = query_analyzer->hasWindow();
 
     if (storage && !options.only_analyze)
     {
@@ -636,9 +637,7 @@ Block InterpreterSelectQuery::getSampleBlockImpl()
 
         if (analysis_result.prewhere_info)
         {
-            ExpressionActions(
-                analysis_result.prewhere_info->prewhere_actions,
-                ExpressionActionsSettings::fromContext(context)).execute(header);
+            header = analysis_result.prewhere_info->prewhere_actions->updateHeader(header);
             if (analysis_result.prewhere_info->remove_prewhere_column)
                 header.erase(analysis_result.prewhere_info->prewhere_column_name);
         }
@@ -1257,7 +1256,12 @@ void InterpreterSelectQuery::executeImpl(QueryPlan & query_plan, const BlockInpu
             // 4) preliminary distinct.
             // Some of these were already executed at the shards (first_stage),
             // see the counterpart code and comments there.
-            if (expressions.need_aggregate)
+            if (from_aggregation_stage)
+            {
+                if (query_analyzer->hasWindow())
+                    throw Exception("Window functions does not support processing from WithMergeableStateAfterAggregation", ErrorCodes::NOT_IMPLEMENTED);
+            }
+            else if (expressions.need_aggregate)
             {
                 executeExpression(query_plan, expressions.before_window,
                     "Before window functions");
@@ -1895,11 +1899,12 @@ void InterpreterSelectQuery::executeFetchColumns(QueryProcessingStage::Enum proc
 
         // TODO figure out how to make set for projections
         query_info.sets = query_analyzer->getPreparedSets();
-        auto actions_settings = ExpressionActionsSettings::fromContext(context);
         auto & prewhere_info = analysis_result.prewhere_info;
 
         if (prewhere_info)
         {
+            auto actions_settings = ExpressionActionsSettings::fromContext(context, CompileExpressions::yes);
+
             query_info.prewhere_info = std::make_shared<PrewhereInfo>();
             query_info.prewhere_info->prewhere_actions = std::make_shared<ExpressionActions>(prewhere_info->prewhere_actions, actions_settings);
 

src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp

@@ -124,8 +124,8 @@ static NamesAndTypesList getNames(const ASTFunction & expr, ContextPtr context,
 
     ASTPtr temp_ast = expr.clone();
     auto syntax = TreeRewriter(context).analyze(temp_ast, columns);
-    auto expression = ExpressionAnalyzer(temp_ast, syntax, context).getActions(false);
-    return expression->getRequiredColumnsWithTypes();
+    auto required_columns = ExpressionAnalyzer(temp_ast, syntax, context).getActionsDAG(false)->getRequiredColumns();
+    return required_columns;
 }
 
 static NamesAndTypesList modifyPrimaryKeysToNonNullable(const NamesAndTypesList & primary_keys, NamesAndTypesList & columns)

src/Interpreters/convertFieldToType.cpp

@@ -201,7 +201,13 @@ Field convertFieldToTypeImpl(const Field & src, const IDataType & type, const ID
             return src;
         }
 
-        /// TODO Conversion from integers to DateTime64
+        if (which_type.isDateTime64()
+            && (which_from_type.isNativeInt() || which_from_type.isNativeUInt() || which_from_type.isDateOrDateTime()))
+        {
+            const auto scale = static_cast<const DataTypeDateTime64 &>(type).getScale();
+            const auto decimal_value = DecimalUtils::decimalFromComponents<DateTime64>(src.reinterpret<Int64>(), 0, scale);
+            return Field(DecimalField<DateTime64>(decimal_value, scale));
+        }
     }
     else if (which_type.isUUID() && src.getType() == Field::Types::UUID)
     {

src/Interpreters/examples/jit_example.cpp

@@ -1,57 +1,57 @@
 #include <iostream>
 
-#include <llvm/IR/IRBuilder.h>
+// #include <llvm/IR/IRBuilder.h>
 
-#include <Interpreters/JIT/CHJIT.h>
+// #include <Interpreters/JIT/CHJIT.h>
 
-void test_function()
-{
-    std::cerr << "Test function" << std::endl;
-}
+// void test_function()
+// {
+//     std::cerr << "Test function" << std::endl;
+// }
 
 int main(int argc, char **argv)
 {
     (void)(argc);
     (void)(argv);
 
-    auto jit = DB::CHJIT();
+    // auto jit = DB::CHJIT();
 
-    jit.registerExternalSymbol("test_function", reinterpret_cast<void *>(&test_function));
+    // jit.registerExternalSymbol("test_function", reinterpret_cast<void *>(&test_function));
 
-    auto compiled_module_info = jit.compileModule([](llvm::Module & module)
-    {
-        auto & context = module.getContext();
-        llvm::IRBuilder<> b (context);
+    // auto compiled_module_info = jit.compileModule([](llvm::Module & module)
+    // {
+    //     auto & context = module.getContext();
+    //     llvm::IRBuilder<> b (context);
 
-        auto * func_declaration_type = llvm::FunctionType::get(b.getVoidTy(), { }, /*isVarArg=*/false);
-        auto * func_declaration = llvm::Function::Create(func_declaration_type, llvm::Function::ExternalLinkage, "test_function", module);
+    //     auto * func_declaration_type = llvm::FunctionType::get(b.getVoidTy(), { }, /*isVarArg=*/false);
+    //     auto * func_declaration = llvm::Function::Create(func_declaration_type, llvm::Function::ExternalLinkage, "test_function", module);
 
-        auto * value_type = b.getInt64Ty();
-        auto * pointer_type = value_type->getPointerTo();
+    //     auto * value_type = b.getInt64Ty();
+    //     auto * pointer_type = value_type->getPointerTo();
 
-        auto * func_type = llvm::FunctionType::get(b.getVoidTy(), { pointer_type }, /*isVarArg=*/false);
-        auto * function = llvm::Function::Create(func_type, llvm::Function::ExternalLinkage, "test_name", module);
-        auto * entry = llvm::BasicBlock::Create(context, "entry", function);
+    //     auto * func_type = llvm::FunctionType::get(b.getVoidTy(), { pointer_type }, /*isVarArg=*/false);
+    //     auto * function = llvm::Function::Create(func_type, llvm::Function::ExternalLinkage, "test_name", module);
+    //     auto * entry = llvm::BasicBlock::Create(context, "entry", function);
 
-        auto * argument = function->args().begin();
-        b.SetInsertPoint(entry);
+    //     auto * argument = function->args().begin();
+    //     b.SetInsertPoint(entry);
 
-        b.CreateCall(func_declaration);
+    //     b.CreateCall(func_declaration);
 
-        auto * load_argument = b.CreateLoad(argument);
-        auto * value = b.CreateAdd(load_argument, load_argument);
-        b.CreateRet(value);
-    });
+    //     auto * load_argument = b.CreateLoad(argument);
+    //     auto * value = b.CreateAdd(load_argument, load_argument);
+    //     b.CreateRet(value);
+    // });
 
-    for (const auto & compiled_function_name : compiled_module_info.compiled_functions)
-    {
-        std::cerr << compiled_function_name << std::endl;
-    }
+    // for (const auto & compiled_function_name : compiled_module_info.compiled_functions)
+    // {
+    //     std::cerr << compiled_function_name << std::endl;
+    // }
 
-    int64_t value = 5;
-    auto * test_name_function = reinterpret_cast<int64_t (*)(int64_t *)>(jit.findCompiledFunction(compiled_module_info, "test_name"));
-    auto result = test_name_function(&value);
-    std::cerr << "Result " << result << std::endl;
+    // int64_t value = 5;
+    // auto * test_name_function = reinterpret_cast<int64_t (*)(int64_t *)>(jit.findCompiledFunction(compiled_module_info, "test_name"));
+    // auto result = test_name_function(&value);
+    // std::cerr << "Result " << result << std::endl;
 
     return 0;
 }

src/Parsers/ASTSelectQuery.cpp

@@ -95,7 +95,7 @@ void ASTSelectQuery::formatImpl(const FormatSettings & s, FormatState & state, F
 
     if (tables())
     {
-        s.ostr << (s.hilite ? hilite_keyword : "") << s.nl_or_ws << indent_str << "FROM " << (s.hilite ? hilite_none : "");
+        s.ostr << (s.hilite ? hilite_keyword : "") << s.nl_or_ws << indent_str << "FROM" << (s.hilite ? hilite_none : "");
         tables()->formatImpl(s, state, frame);
     }
 

src/Parsers/ASTSelectWithUnionQuery.cpp

@@ -35,24 +35,24 @@ void ASTSelectWithUnionQuery::formatQueryImpl(const FormatSettings & settings, F
         if (mode == Mode::Unspecified)
             return "";
         else if (mode == Mode::ALL)
-            return "ALL";
+            return " ALL";
         else
-            return "DISTINCT";
+            return " DISTINCT";
     };
 
     for (ASTs::const_iterator it = list_of_selects->children.begin(); it != list_of_selects->children.end(); ++it)
     {
         if (it != list_of_selects->children.begin())
-            settings.ostr << settings.nl_or_ws << indent_str << (settings.hilite ? hilite_keyword : "") << "UNION "
+            settings.ostr << settings.nl_or_ws << indent_str << (settings.hilite ? hilite_keyword : "") << "UNION"
                           << mode_to_str((is_normalized) ? union_mode : list_of_modes[it - list_of_selects->children.begin() - 1])
                           << (settings.hilite ? hilite_none : "");
 
         if (auto * node = (*it)->as<ASTSelectWithUnionQuery>())
         {
+            settings.ostr << settings.nl_or_ws << indent_str;
+
             if (node->list_of_selects->children.size() == 1)
             {
-                if (it != list_of_selects->children.begin())
-                    settings.ostr << settings.nl_or_ws;
                 (node->list_of_selects->children.at(0))->formatImpl(settings, state, frame);
             }
             else

src/Parsers/ASTSubquery.cpp

@@ -29,6 +29,11 @@ void ASTSubquery::appendColumnNameImpl(WriteBuffer & ostr) const
 
 void ASTSubquery::formatImplWithoutAlias(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const
 {
+    /// NOTE: due to trickery of filling cte_name (in interpreters) it is hard
+    /// to print it w/o newline (for !oneline case), since if nl_or_ws
+    /// prepended here, then formatting will be incorrect with alias:
+    ///
+    ///   (select 1 in ((select 1) as sub))
     if (!cte_name.empty())
     {
         settings.ostr << (settings.hilite ? hilite_identifier : "");
@@ -40,7 +45,7 @@ void ASTSubquery::formatImplWithoutAlias(const FormatSettings & settings, Format
     std::string indent_str = settings.one_line ? "" : std::string(4u * frame.indent, ' ');
     std::string nl_or_nothing = settings.one_line ? "" : "\n";
 
-    settings.ostr << nl_or_nothing << indent_str << "(" << nl_or_nothing;
+    settings.ostr << "(" << nl_or_nothing;
     FormatStateStacked frame_nested = frame;
     frame_nested.need_parens = false;
     ++frame_nested.indent;

src/Parsers/ASTTablesInSelectQuery.cpp

@@ -109,14 +109,17 @@ void ASTTableExpression::formatImpl(const FormatSettings & settings, FormatState
 
     if (database_and_table_name)
     {
+        settings.ostr << " ";
         database_and_table_name->formatImpl(settings, state, frame);
     }
     else if (table_function)
     {
+        settings.ostr << " ";
         table_function->formatImpl(settings, state, frame);
     }
     else if (subquery)
     {
+        settings.ostr << settings.nl_or_ws << indent_str;
         subquery->formatImpl(settings, state, frame);
     }
 
@@ -142,9 +145,15 @@ void ASTTableExpression::formatImpl(const FormatSettings & settings, FormatState
 }
 
 
-void ASTTableJoin::formatImplBeforeTable(const FormatSettings & settings, FormatState &, FormatStateStacked) const
+void ASTTableJoin::formatImplBeforeTable(const FormatSettings & settings, FormatState &, FormatStateStacked frame) const
 {
     settings.ostr << (settings.hilite ? hilite_keyword : "");
+    std::string indent_str = settings.one_line ? "" : std::string(4 * frame.indent, ' ');
+
+    if (kind != Kind::Comma)
+    {
+        settings.ostr << settings.nl_or_ws << indent_str;
+    }
 
     switch (locality)
     {
@@ -241,6 +250,7 @@ void ASTArrayJoin::formatImpl(const FormatSettings & settings, FormatState & sta
     frame.expression_list_prepend_whitespace = true;
 
     settings.ostr << (settings.hilite ? hilite_keyword : "")
+        << settings.nl_or_ws
         << (kind == Kind::Left ? "LEFT " : "") << "ARRAY JOIN" << (settings.hilite ? hilite_none : "");
 
     settings.one_line
@@ -254,10 +264,7 @@ void ASTTablesInSelectQueryElement::formatImpl(const FormatSettings & settings,
     if (table_expression)
     {
         if (table_join)
-        {
             table_join->as<ASTTableJoin &>().formatImplBeforeTable(settings, state, frame);
-            settings.ostr << " ";
-        }
 
         table_expression->formatImpl(settings, state, frame);
 
@@ -275,13 +282,8 @@ void ASTTablesInSelectQuery::formatImpl(const FormatSettings & settings, FormatS
 {
     std::string indent_str = settings.one_line ? "" : std::string(4 * frame.indent, ' ');
 
-    for (ASTs::const_iterator it = children.begin(); it != children.end(); ++it)
-    {
-        if (it != children.begin())
-            settings.ostr << settings.nl_or_ws << indent_str;
-
-        (*it)->formatImpl(settings, state, frame);
-    }
+    for (const auto & child : children)
+        child->formatImpl(settings, state, frame);
 }
 
 }

src/Parsers/ASTWithElement.cpp

@@ -16,8 +16,11 @@ ASTPtr ASTWithElement::clone() const
 
 void ASTWithElement::formatImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const
 {
+    std::string indent_str = settings.one_line ? "" : std::string(4 * frame.indent, ' ');
+
     settings.writeIdentifier(name);
-    settings.ostr << (settings.hilite ? hilite_keyword : "") << " AS " << (settings.hilite ? hilite_none : "");
+    settings.ostr << (settings.hilite ? hilite_keyword : "") << " AS" << (settings.hilite ? hilite_none : "");
+    settings.ostr << settings.nl_or_ws << indent_str;
     dynamic_cast<const ASTWithAlias &>(*subquery).formatImplWithoutAlias(settings, state, frame);
 }
 

src/Processors/QueryPlan/ArrayJoinStep.cpp

@@ -60,6 +60,7 @@ void ArrayJoinStep::transformPipeline(QueryPipeline & pipeline, const BuildQuery
                 pipeline.getHeader().getColumnsWithTypeAndName(),
                 res_header.getColumnsWithTypeAndName(),
                 ActionsDAG::MatchColumnsMode::Name);
+
         auto actions = std::make_shared<ExpressionActions>(actions_dag, settings.getActionsSettings());
 
         pipeline.addSimpleTransform([&](const Block & header)

src/Processors/QueryPlan/BuildQueryPipelineSettings.cpp

@@ -9,7 +9,7 @@ namespace DB
 BuildQueryPipelineSettings BuildQueryPipelineSettings::fromSettings(const Settings & from)
 {
     BuildQueryPipelineSettings settings;
-    settings.actions_settings = ExpressionActionsSettings::fromSettings(from);
+    settings.actions_settings = ExpressionActionsSettings::fromSettings(from, CompileExpressions::yes);
     return settings;
 }
 

src/Processors/QueryPlan/ExpressionStep.cpp

@@ -55,6 +55,7 @@ void ExpressionStep::updateInputStream(DataStream input_stream, bool keep_header
 void ExpressionStep::transformPipeline(QueryPipeline & pipeline, const BuildQueryPipelineSettings & settings)
 {
     auto expression = std::make_shared<ExpressionActions>(actions_dag, settings.getActionsSettings());
+
     pipeline.addSimpleTransform([&](const Block & header)
     {
         return std::make_shared<ExpressionTransform>(header, expression);
@@ -80,7 +81,7 @@ void ExpressionStep::describeActions(FormatSettings & settings) const
     String prefix(settings.offset, ' ');
     bool first = true;
 
-    auto expression = std::make_shared<ExpressionActions>(actions_dag, ExpressionActionsSettings{});
+    auto expression = std::make_shared<ExpressionActions>(actions_dag);
     for (const auto & action : expression->getActions())
     {
         settings.out << prefix << (first ? "Actions: "
@@ -97,7 +98,7 @@ void ExpressionStep::describeActions(FormatSettings & settings) const
 
 void ExpressionStep::describeActions(JSONBuilder::JSONMap & map) const
 {
-    auto expression = std::make_shared<ExpressionActions>(actions_dag, ExpressionActionsSettings{});
+    auto expression = std::make_shared<ExpressionActions>(actions_dag);
     map.add("Expression", expression->toTree());
 }