Merge branch 'master' into nv/last-queue-update-exception

2024-11-23 08:02:02 +00:00 · 2021-08-18 07:00:50 +00:00 · 2021-08-18 07:00:50 +00:00 · f11e396151
commit f11e396151
parent 3f291b024a 3736d6de35
2085 changed files with 36987 additions and 15535 deletions
--- a/.github/ISSUE_TEMPLATE/10_question.md
+++ b/.github/ISSUE_TEMPLATE/10_question.md
@ -7,6 +7,6 @@ assignees: ''

 ---

-Make sure to check documentation https://clickhouse.yandex/docs/en/ first. If the question is concise and probably has a short answer, asking it in Telegram chat https://telegram.me/clickhouse_en is probably the fastest way to find the answer. For more complicated questions, consider asking them on StackOverflow with "clickhouse" tag https://stackoverflow.com/questions/tagged/clickhouse 
+> Make sure to check documentation https://clickhouse.yandex/docs/en/ first. If the question is concise and probably has a short answer, asking it in Telegram chat https://telegram.me/clickhouse_en is probably the fastest way to find the answer. For more complicated questions, consider asking them on StackOverflow with "clickhouse" tag https://stackoverflow.com/questions/tagged/clickhouse 

-If you still prefer GitHub issues, remove all this text and ask your question here.
+> If you still prefer GitHub issues, remove all this text and ask your question here.
--- a/.github/ISSUE_TEMPLATE/20_feature-request.md
+++ b/.github/ISSUE_TEMPLATE/20_feature-request.md
@ -7,16 +7,20 @@ assignees: ''

 ---

-(you don't have to strictly follow this form)
+> (you don't have to strictly follow this form)

 **Use case**
-A clear and concise description of what is the intended usage scenario is.
+
+> A clear and concise description of what is the intended usage scenario is.

 **Describe the solution you'd like**
-A clear and concise description of what you want to happen.
+
+> A clear and concise description of what you want to happen.

 **Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
+
+> A clear and concise description of any alternative solutions or features you've considered.

 **Additional context**
-Add any other context or screenshots about the feature request here.
+
+> Add any other context or screenshots about the feature request here.
--- a/.github/ISSUE_TEMPLATE/50_build-issue.md
+++ b/.github/ISSUE_TEMPLATE/50_build-issue.md
@ -7,10 +7,11 @@ assignees: ''

 ---

-Make sure that `git diff` result is empty and you've just pulled fresh master. Try cleaning up cmake cache. Just in case, official build instructions are published here: https://clickhouse.yandex/docs/en/development/build/
+> Make sure that `git diff` result is empty and you've just pulled fresh master. Try cleaning up cmake cache. Just in case, official build instructions are published here: https://clickhouse.yandex/docs/en/development/build/

 **Operating system**
-OS kind or distribution, specific version/release, non-standard kernel if any. If you are trying to build inside virtual machine, please mention it too.
+
+> OS kind or distribution, specific version/release, non-standard kernel if any. If you are trying to build inside virtual machine, please mention it too.

 **Cmake version**

--- a/.github/ISSUE_TEMPLATE/85_bug-report.md
+++ b/.github/ISSUE_TEMPLATE/85_bug-report.md
@ -1,17 +1,17 @@
 ---
 name: Bug report
-about: Create a report to help us improve ClickHouse
+about: Wrong behaviour (visible to users) in official ClickHouse release.
 title: ''
-labels: bug
+labels: 'potential bug'
 assignees: ''

 ---

-You have to provide the following information whenever possible.
+> You have to provide the following information whenever possible.

 **Describe the bug**

-A clear and concise description of what works not as it is supposed to.
+> A clear and concise description of what works not as it is supposed to.

 **Does it reproduce on recent release?**

@ -19,7 +19,7 @@ A clear and concise description of what works not as it is supposed to.

 **Enable crash reporting**

-If possible, change "enabled" to true in "send_crash_reports" section in `config.xml`:
+> If possible, change "enabled" to true in "send_crash_reports" section in `config.xml`:

 ```
 <send_crash_reports>
@ -39,12 +39,12 @@ If possible, change "enabled" to true in "send_crash_reports" section in `config

 **Expected behavior**

-A clear and concise description of what you expected to happen.
+> A clear and concise description of what you expected to happen.

 **Error message and/or stacktrace**

-If applicable, add screenshots to help explain your problem.
+> If applicable, add screenshots to help explain your problem.

 **Additional context**

-Add any other context about the problem here.
+> Add any other context about the problem here.
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@ -19,9 +19,9 @@ Detailed description / Documentation draft:
 ...


-By adding documentation, you'll allow users to try your new feature immediately, not when someone else will have time to document it later. Documentation is necessary for all features that affect user experience in any way. You can add brief documentation draft above, or add documentation right into your patch as Markdown files in [docs](https://github.com/ClickHouse/ClickHouse/tree/master/docs) folder.
+> By adding documentation, you'll allow users to try your new feature immediately, not when someone else will have time to document it later. Documentation is necessary for all features that affect user experience in any way. You can add brief documentation draft above, or add documentation right into your patch as Markdown files in [docs](https://github.com/ClickHouse/ClickHouse/tree/master/docs) folder.

-If you are doing this for the first time, it's recommended to read the lightweight [Contributing to ClickHouse Documentation](https://github.com/ClickHouse/ClickHouse/tree/master/docs/README.md) guide first.
+> If you are doing this for the first time, it's recommended to read the lightweight [Contributing to ClickHouse Documentation](https://github.com/ClickHouse/ClickHouse/tree/master/docs/README.md) guide first.


-Information about CI checks: https://clickhouse.tech/docs/en/development/continuous-integration/
+> Information about CI checks: https://clickhouse.tech/docs/en/development/continuous-integration/
--- a/.gitmodules
+++ b/.gitmodules
@ -225,6 +225,15 @@
 [submodule "contrib/yaml-cpp"]
 	path = contrib/yaml-cpp
 	url = https://github.com/ClickHouse-Extras/yaml-cpp.git
+[submodule "contrib/libstemmer_c"]
+	path = contrib/libstemmer_c
+	url = https://github.com/ClickHouse-Extras/libstemmer_c.git
+[submodule "contrib/wordnet-blast"]
+	path = contrib/wordnet-blast
+	url = https://github.com/ClickHouse-Extras/wordnet-blast.git
+[submodule "contrib/lemmagen-c"]
+	path = contrib/lemmagen-c
+	url = https://github.com/ClickHouse-Extras/lemmagen-c.git
 [submodule "contrib/libpqxx"]
 	path = contrib/libpqxx
 	url = https://github.com/ClickHouse-Extras/libpqxx.git
@ -234,3 +243,6 @@
 [submodule "contrib/s2geometry"]
 	path = contrib/s2geometry
 	url = https://github.com/ClickHouse-Extras/s2geometry.git
+[submodule "contrib/bzip2"]
+	path = contrib/bzip2
+	url = https://github.com/ClickHouse-Extras/bzip2.git
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -1,3 +1,102 @@
+### ClickHouse release v21.8, 2021-08-12
+
+#### New Features
+
+* Add support for a part of SQL/JSON standard. [#24148](https://github.com/ClickHouse/ClickHouse/pull/24148) ([l1tsolaiki](https://github.com/l1tsolaiki), [Kseniia Sumarokova](https://github.com/kssenii)).
+* Collect common system metrics (in `system.asynchronous_metrics` and `system.asynchronous_metric_log`) on CPU usage, disk usage, memory usage, IO, network, files, load average, CPU frequencies, thermal sensors, EDAC counters, system uptime; also added metrics about the scheduling jitter and the time spent collecting the metrics. It works similar to `atop` in ClickHouse and allows access to monitoring data even if you have no additional tools installed. Close [#9430](https://github.com/ClickHouse/ClickHouse/issues/9430). [#24416](https://github.com/ClickHouse/ClickHouse/pull/24416) ([alexey-milovidov](https://github.com/alexey-milovidov), [Yegor Levankov](https://github.com/elevankoff)).
+* Add MaterializedPostgreSQL table engine and database engine. This database engine allows replicating a whole database or any subset of database tables. [#20470](https://github.com/ClickHouse/ClickHouse/pull/20470) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Add new functions `leftPad()`, `rightPad()`, `leftPadUTF8()`, `rightPadUTF8()`. [#26075](https://github.com/ClickHouse/ClickHouse/pull/26075) ([Vitaly Baranov](https://github.com/vitlibar)).
+* Add the `FIRST` keyword to the `ADD INDEX` command to be able to add the index at the beginning of the indices list. [#25904](https://github.com/ClickHouse/ClickHouse/pull/25904) ([xjewer](https://github.com/xjewer)).
+* Introduce `system.data_skipping_indices` table containing information about existing data skipping indices. Close [#7659](https://github.com/ClickHouse/ClickHouse/issues/7659). [#25693](https://github.com/ClickHouse/ClickHouse/pull/25693) ([Dmitry Novik](https://github.com/novikd)).
+* Add `bin`/`unbin` functions. [#25609](https://github.com/ClickHouse/ClickHouse/pull/25609) ([zhaoyu](https://github.com/zxc111)).
+* Support `Map` and `UInt128`, `Int128`, `UInt256`, `Int256` types in `mapAdd` and `mapSubtract` functions. [#25596](https://github.com/ClickHouse/ClickHouse/pull/25596) ([Ildus Kurbangaliev](https://github.com/ildus)).
+* Support `DISTINCT ON (columns)` expression, close [#25404](https://github.com/ClickHouse/ClickHouse/issues/25404). [#25589](https://github.com/ClickHouse/ClickHouse/pull/25589) ([Zijie Lu](https://github.com/TszKitLo40)).
+* Add an ability to reset a custom setting to default and remove it from the table's metadata. It allows rolling back the change without knowing the system/config's default.  Closes [#14449](https://github.com/ClickHouse/ClickHouse/issues/14449). [#17769](https://github.com/ClickHouse/ClickHouse/pull/17769) ([xjewer](https://github.com/xjewer)).
+* Render pipelines as graphs in Web UI if `EXPLAIN PIPELINE graph = 1` query is submitted. [#26067](https://github.com/ClickHouse/ClickHouse/pull/26067) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+
+#### Performance Improvements
+
+* Compile aggregate functions. Use option `compile_aggregate_expressions` to enable it. [#24789](https://github.com/ClickHouse/ClickHouse/pull/24789) ([Maksim Kita](https://github.com/kitaisreal)).
+* Improve latency of short queries that require reading from tables with many columns. [#26371](https://github.com/ClickHouse/ClickHouse/pull/26371) ([Anton Popov](https://github.com/CurtizJ)).
+
+#### Improvements
+
+* Use `Map` data type for system logs tables (`system.query_log`, `system.query_thread_log`, `system.processes`, `system.opentelemetry_span_log`). These tables will be auto-created with new data types. Virtual columns are created to support old queries. Closes [#18698](https://github.com/ClickHouse/ClickHouse/issues/18698). [#23934](https://github.com/ClickHouse/ClickHouse/pull/23934), [#25773](https://github.com/ClickHouse/ClickHouse/pull/25773) ([hexiaoting](https://github.com/hexiaoting), [sundy-li](https://github.com/sundy-li), [Maksim Kita](https://github.com/kitaisreal)).
+* For a dictionary with a complex key containing only one attribute, allow not wrapping the key expression in tuple for functions `dictGet`, `dictHas`. [#26130](https://github.com/ClickHouse/ClickHouse/pull/26130) ([Maksim Kita](https://github.com/kitaisreal)).
+* Implement function `bin`/`hex` from `AggregateFunction` states. [#26094](https://github.com/ClickHouse/ClickHouse/pull/26094) ([zhaoyu](https://github.com/zxc111)).
+* Support arguments of `UUID` type for `empty` and `notEmpty` functions. `UUID` is empty if it is all zeros (nil UUID). Closes [#3446](https://github.com/ClickHouse/ClickHouse/issues/3446). [#25974](https://github.com/ClickHouse/ClickHouse/pull/25974) ([zhaoyu](https://github.com/zxc111)).
+* Add support for `SET SQL_SELECT_LIMIT` in MySQL protocol. Closes [#17115](https://github.com/ClickHouse/ClickHouse/issues/17115). [#25972](https://github.com/ClickHouse/ClickHouse/pull/25972) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* More instrumentation for network interaction: add counters for recv/send bytes; add gauges for recvs/sends. Added missing documentation. Close [#5897](https://github.com/ClickHouse/ClickHouse/issues/5897). [#25962](https://github.com/ClickHouse/ClickHouse/pull/25962) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Add setting `optimize_move_to_prewhere_if_final`. If query has `FINAL`, the optimization `move_to_prewhere` will be enabled only if both `optimize_move_to_prewhere` and `optimize_move_to_prewhere_if_final` are enabled. Closes [#8684](https://github.com/ClickHouse/ClickHouse/issues/8684). [#25940](https://github.com/ClickHouse/ClickHouse/pull/25940) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Allow complex quoted identifiers of JOINed tables. Close [#17861](https://github.com/ClickHouse/ClickHouse/issues/17861). [#25924](https://github.com/ClickHouse/ClickHouse/pull/25924) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Add support for Unicode (e.g. Chinese, Cyrillic) components in `Nested` data types. Close [#25594](https://github.com/ClickHouse/ClickHouse/issues/25594). [#25923](https://github.com/ClickHouse/ClickHouse/pull/25923) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Allow `quantiles*` functions to work with `aggregate_functions_null_for_empty`. Close [#25892](https://github.com/ClickHouse/ClickHouse/issues/25892). [#25919](https://github.com/ClickHouse/ClickHouse/pull/25919) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Allow parameters for parametric aggregate functions to be arbitrary constant expressions (e.g., `1 + 2`), not just literals. It also allows using the query parameters (in parameterized queries like `{param:UInt8}`) inside parametric aggregate functions. Closes [#11607](https://github.com/ClickHouse/ClickHouse/issues/11607). [#25910](https://github.com/ClickHouse/ClickHouse/pull/25910) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Correctly throw the exception on the attempt to parse an invalid `Date`. Closes [#6481](https://github.com/ClickHouse/ClickHouse/issues/6481). [#25909](https://github.com/ClickHouse/ClickHouse/pull/25909) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Support for multiple includes in configuration. It is possible to include users configuration, remote server configuration from multiple sources. Simply place `<include />` element with `from_zk`, `from_env` or `incl` attribute, and it will be replaced with the substitution. [#24404](https://github.com/ClickHouse/ClickHouse/pull/24404) ([nvartolomei](https://github.com/nvartolomei)).
+* Support for queries with a column named `"null"` (it must be specified in back-ticks or double quotes) and `ON CLUSTER`. Closes [#24035](https://github.com/ClickHouse/ClickHouse/issues/24035). [#25907](https://github.com/ClickHouse/ClickHouse/pull/25907) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Support `LowCardinality`, `Decimal`, and `UUID` for `JSONExtract`. Closes [#24606](https://github.com/ClickHouse/ClickHouse/issues/24606). [#25900](https://github.com/ClickHouse/ClickHouse/pull/25900) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Convert history file from `readline` format to `replxx` format. [#25888](https://github.com/ClickHouse/ClickHouse/pull/25888) ([Azat Khuzhin](https://github.com/azat)).
+* Fix an issue which can lead to intersecting parts after `DROP PART` or background deletion of an empty part. [#25884](https://github.com/ClickHouse/ClickHouse/pull/25884) ([alesapin](https://github.com/alesapin)).
+* Better handling of lost parts for `ReplicatedMergeTree` tables. Fixes rare inconsistencies in `ReplicationQueue`. Fixes [#10368](https://github.com/ClickHouse/ClickHouse/issues/10368). [#25820](https://github.com/ClickHouse/ClickHouse/pull/25820) ([alesapin](https://github.com/alesapin)).
+* Allow starting clickhouse-client with unreadable working directory. [#25817](https://github.com/ClickHouse/ClickHouse/pull/25817) ([ianton-ru](https://github.com/ianton-ru)).
+* Fix "No available columns" error for `Merge` storage. [#25801](https://github.com/ClickHouse/ClickHouse/pull/25801) ([Azat Khuzhin](https://github.com/azat)).
+* MySQL Engine now supports the exchange of column comments between MySQL and ClickHouse. [#25795](https://github.com/ClickHouse/ClickHouse/pull/25795) ([Storozhuk Kostiantyn](https://github.com/sand6255)).
+* Fix inconsistent behaviour of `GROUP BY` constant on empty set. Closes [#6842](https://github.com/ClickHouse/ClickHouse/issues/6842). [#25786](https://github.com/ClickHouse/ClickHouse/pull/25786) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Cancel already running merges in partition on `DROP PARTITION` and `TRUNCATE` for `ReplicatedMergeTree`. Resolves [#17151](https://github.com/ClickHouse/ClickHouse/issues/17151). [#25684](https://github.com/ClickHouse/ClickHouse/pull/25684) ([tavplubix](https://github.com/tavplubix)).
+* Support ENUM` data type for MaterializeMySQL. [#25676](https://github.com/ClickHouse/ClickHouse/pull/25676) ([Storozhuk Kostiantyn](https://github.com/sand6255)).
+* Support materialized and aliased columns in JOIN, close [#13274](https://github.com/ClickHouse/ClickHouse/issues/13274). [#25634](https://github.com/ClickHouse/ClickHouse/pull/25634) ([Vladimir C](https://github.com/vdimir)).
+* Fix possible logical race condition between `ALTER TABLE ... DETACH` and background merges. [#25605](https://github.com/ClickHouse/ClickHouse/pull/25605) ([Azat Khuzhin](https://github.com/azat)).
+* Make `NetworkReceiveElapsedMicroseconds` metric to correctly include the time spent waiting for data from the client to `INSERT`. Close [#9958](https://github.com/ClickHouse/ClickHouse/issues/9958). [#25602](https://github.com/ClickHouse/ClickHouse/pull/25602) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Support `TRUNCATE TABLE` for S3 and HDFS. Close [#25530](https://github.com/ClickHouse/ClickHouse/issues/25530). [#25550](https://github.com/ClickHouse/ClickHouse/pull/25550) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Support for dynamic reloading of config to change number of threads in pool for background jobs execution (merges, mutations, fetches). [#25548](https://github.com/ClickHouse/ClickHouse/pull/25548) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+* Allow extracting of non-string element as string using `JSONExtract`. This is for [#25414](https://github.com/ClickHouse/ClickHouse/issues/25414). [#25452](https://github.com/ClickHouse/ClickHouse/pull/25452) ([Amos Bird](https://github.com/amosbird)).
+* Support regular expression in `Database` argument for `StorageMerge`. Close [#776](https://github.com/ClickHouse/ClickHouse/issues/776). [#25064](https://github.com/ClickHouse/ClickHouse/pull/25064) ([flynn](https://github.com/ucasfl)).
+* Web UI: if the value looks like a URL, automatically generate a link. [#25965](https://github.com/ClickHouse/ClickHouse/pull/25965) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Make `sudo service clickhouse-server start` to work on systems with `systemd` like Centos 8. Close [#14298](https://github.com/ClickHouse/ClickHouse/issues/14298). Close [#17799](https://github.com/ClickHouse/ClickHouse/issues/17799). [#25921](https://github.com/ClickHouse/ClickHouse/pull/25921) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+
+#### Bug Fixes
+
+* Fix incorrect `SET ROLE` in some cases. [#26707](https://github.com/ClickHouse/ClickHouse/pull/26707) ([Vitaly Baranov](https://github.com/vitlibar)).
+* Fix potential `nullptr` dereference in window functions. Fix [#25276](https://github.com/ClickHouse/ClickHouse/issues/25276). [#26668](https://github.com/ClickHouse/ClickHouse/pull/26668) ([Alexander Kuzmenkov](https://github.com/akuzm)).
+* Fix incorrect function names of `groupBitmapAnd/Or/Xor`. Fix [#26557](https://github.com/ClickHouse/ClickHouse/pull/26557) ([Amos Bird](https://github.com/amosbird)).
+* Fix crash in RabbitMQ shutdown in case RabbitMQ setup was not started. Closes [#26504](https://github.com/ClickHouse/ClickHouse/issues/26504). [#26529](https://github.com/ClickHouse/ClickHouse/pull/26529) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Fix issues with `CREATE DICTIONARY` query if dictionary name or database name was quoted. Closes [#26491](https://github.com/ClickHouse/ClickHouse/issues/26491). [#26508](https://github.com/ClickHouse/ClickHouse/pull/26508) ([Maksim Kita](https://github.com/kitaisreal)).
+* Fix broken name resolution after rewriting column aliases. Fix [#26432](https://github.com/ClickHouse/ClickHouse/issues/26432). [#26475](https://github.com/ClickHouse/ClickHouse/pull/26475) ([Amos Bird](https://github.com/amosbird)).
+* Fix infinite non-joined block stream in `partial_merge_join` close [#26325](https://github.com/ClickHouse/ClickHouse/issues/26325). [#26374](https://github.com/ClickHouse/ClickHouse/pull/26374) ([Vladimir C](https://github.com/vdimir)).
+* Fix possible crash when login as dropped user. Fix [#26073](https://github.com/ClickHouse/ClickHouse/issues/26073). [#26363](https://github.com/ClickHouse/ClickHouse/pull/26363) ([Vitaly Baranov](https://github.com/vitlibar)).
+* Fix `optimize_distributed_group_by_sharding_key` for multiple columns (leads to incorrect result w/ `optimize_skip_unused_shards=1`/`allow_nondeterministic_optimize_skip_unused_shards=1` and multiple columns in sharding key expression). [#26353](https://github.com/ClickHouse/ClickHouse/pull/26353) ([Azat Khuzhin](https://github.com/azat)).
+* `CAST` from `Date` to `DateTime` (or `DateTime64`) was not using the timezone of the `DateTime` type. It can also affect the comparison between `Date` and `DateTime`. Inference of the common type for `Date` and `DateTime` also was not using the corresponding timezone. It affected the results of function `if` and array construction. Closes [#24128](https://github.com/ClickHouse/ClickHouse/issues/24128). [#24129](https://github.com/ClickHouse/ClickHouse/pull/24129) ([Maksim Kita](https://github.com/kitaisreal)).
+* Fixed rare bug in lost replica recovery that may cause replicas to diverge. [#26321](https://github.com/ClickHouse/ClickHouse/pull/26321) ([tavplubix](https://github.com/tavplubix)).
+* Fix zstd decompression in case there are escape sequences at the end of internal buffer. Closes [#26013](https://github.com/ClickHouse/ClickHouse/issues/26013). [#26314](https://github.com/ClickHouse/ClickHouse/pull/26314) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Fix logical error on join with totals, close [#26017](https://github.com/ClickHouse/ClickHouse/issues/26017). [#26250](https://github.com/ClickHouse/ClickHouse/pull/26250) ([Vladimir C](https://github.com/vdimir)).
+* Remove excessive newline in `thread_name` column in `system.stack_trace` table. Fix [#24124](https://github.com/ClickHouse/ClickHouse/issues/24124). [#26210](https://github.com/ClickHouse/ClickHouse/pull/26210) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Fix `joinGet` with `LowCarinality` columns, close [#25993](https://github.com/ClickHouse/ClickHouse/issues/25993). [#26118](https://github.com/ClickHouse/ClickHouse/pull/26118) ([Vladimir C](https://github.com/vdimir)).
+* Fix possible crash in `pointInPolygon` if the setting `validate_polygons` is turned off. [#26113](https://github.com/ClickHouse/ClickHouse/pull/26113) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Fix throwing exception when iterate over non-existing remote directory. [#26087](https://github.com/ClickHouse/ClickHouse/pull/26087) ([ianton-ru](https://github.com/ianton-ru)).
+* Fix rare server crash because of `abort` in ZooKeeper client. Fixes [#25813](https://github.com/ClickHouse/ClickHouse/issues/25813). [#26079](https://github.com/ClickHouse/ClickHouse/pull/26079) ([alesapin](https://github.com/alesapin)).
+* Fix wrong thread count estimation for right subquery join in some cases. Close [#24075](https://github.com/ClickHouse/ClickHouse/issues/24075). [#26052](https://github.com/ClickHouse/ClickHouse/pull/26052) ([Vladimir C](https://github.com/vdimir)).
+* Fixed incorrect `sequence_id` in MySQL protocol packets that ClickHouse sends on exception during query execution. It might cause MySQL client to reset connection to ClickHouse server. Fixes [#21184](https://github.com/ClickHouse/ClickHouse/issues/21184). [#26051](https://github.com/ClickHouse/ClickHouse/pull/26051) ([tavplubix](https://github.com/tavplubix)).
+* Fix possible mismatched header when using normal projection with `PREWHERE`. Fix [#26020](https://github.com/ClickHouse/ClickHouse/issues/26020). [#26038](https://github.com/ClickHouse/ClickHouse/pull/26038) ([Amos Bird](https://github.com/amosbird)).
+* Fix formatting of type `Map` with integer keys to `JSON`. [#25982](https://github.com/ClickHouse/ClickHouse/pull/25982) ([Anton Popov](https://github.com/CurtizJ)).
+* Fix possible deadlock during query profiler stack unwinding. Fix [#25968](https://github.com/ClickHouse/ClickHouse/issues/25968). [#25970](https://github.com/ClickHouse/ClickHouse/pull/25970) ([Maksim Kita](https://github.com/kitaisreal)).
+* Fix crash on call `dictGet()` with bad arguments. [#25913](https://github.com/ClickHouse/ClickHouse/pull/25913) ([Vitaly Baranov](https://github.com/vitlibar)).
+* Fixed `scram-sha-256` authentication for PostgreSQL engines. Closes [#24516](https://github.com/ClickHouse/ClickHouse/issues/24516). [#25906](https://github.com/ClickHouse/ClickHouse/pull/25906) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Fix extremely long backoff for background tasks when the background pool is full. Fixes [#25836](https://github.com/ClickHouse/ClickHouse/issues/25836). [#25893](https://github.com/ClickHouse/ClickHouse/pull/25893) ([alesapin](https://github.com/alesapin)).
+* Fix ARM exception handling with non default page size. Fixes [#25512](https://github.com/ClickHouse/ClickHouse/issues/25512), [#25044](https://github.com/ClickHouse/ClickHouse/issues/25044), [#24901](https://github.com/ClickHouse/ClickHouse/issues/24901), [#23183](https://github.com/ClickHouse/ClickHouse/issues/23183), [#20221](https://github.com/ClickHouse/ClickHouse/issues/20221), [#19703](https://github.com/ClickHouse/ClickHouse/issues/19703), [#19028](https://github.com/ClickHouse/ClickHouse/issues/19028), [#18391](https://github.com/ClickHouse/ClickHouse/issues/18391), [#18121](https://github.com/ClickHouse/ClickHouse/issues/18121), [#17994](https://github.com/ClickHouse/ClickHouse/issues/17994), [#12483](https://github.com/ClickHouse/ClickHouse/issues/12483). [#25854](https://github.com/ClickHouse/ClickHouse/pull/25854) ([Maksim Kita](https://github.com/kitaisreal)).
+* Fix sharding_key from column w/o function for `remote()` (before `select * from remote('127.1', system.one, dummy)` leads to `Unknown column: dummy, there are only columns .` error). [#25824](https://github.com/ClickHouse/ClickHouse/pull/25824) ([Azat Khuzhin](https://github.com/azat)).
+* Fixed `Not found column ...` and `Missing column ...` errors when selecting from `MaterializeMySQL`. Fixes [#23708](https://github.com/ClickHouse/ClickHouse/issues/23708), [#24830](https://github.com/ClickHouse/ClickHouse/issues/24830), [#25794](https://github.com/ClickHouse/ClickHouse/issues/25794). [#25822](https://github.com/ClickHouse/ClickHouse/pull/25822) ([tavplubix](https://github.com/tavplubix)).
+* Fix `optimize_skip_unused_shards_rewrite_in` for non-UInt64 types (may select incorrect shards eventually or throw `Cannot infer type of an empty tuple` or `Function tuple requires at least one argument`). [#25798](https://github.com/ClickHouse/ClickHouse/pull/25798) ([Azat Khuzhin](https://github.com/azat)).
+* Fix rare bug with `DROP PART` query for `ReplicatedMergeTree` tables which can lead to error message `Unexpected merged part intersecting drop range`. [#25783](https://github.com/ClickHouse/ClickHouse/pull/25783) ([alesapin](https://github.com/alesapin)).
+* Fix bug in `TTL` with `GROUP BY` expression which refuses to execute `TTL` after first execution in part. [#25743](https://github.com/ClickHouse/ClickHouse/pull/25743) ([alesapin](https://github.com/alesapin)).
+* Allow StorageMerge to access tables with aliases. Closes [#6051](https://github.com/ClickHouse/ClickHouse/issues/6051). [#25694](https://github.com/ClickHouse/ClickHouse/pull/25694) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Fix slow dict join in some cases, close [#24209](https://github.com/ClickHouse/ClickHouse/issues/24209). [#25618](https://github.com/ClickHouse/ClickHouse/pull/25618) ([Vladimir C](https://github.com/vdimir)).
+* Fix `ALTER MODIFY COLUMN` of columns, which participates in TTL expressions. [#25554](https://github.com/ClickHouse/ClickHouse/pull/25554) ([Anton Popov](https://github.com/CurtizJ)).
+* Fix assertion in `PREWHERE` with non-UInt8 type, close [#19589](https://github.com/ClickHouse/ClickHouse/issues/19589). [#25484](https://github.com/ClickHouse/ClickHouse/pull/25484) ([Vladimir C](https://github.com/vdimir)).
+* Fix some fuzzed msan crash. Fixes [#22517](https://github.com/ClickHouse/ClickHouse/issues/22517). [#26428](https://github.com/ClickHouse/ClickHouse/pull/26428) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
+* Update `chown` cmd check in `clickhouse-server` docker entrypoint. It fixes error 'cluster pod restart failed (or timeout)' on kubernetes. [#26545](https://github.com/ClickHouse/ClickHouse/pull/26545) ([Ky Li](https://github.com/Kylinrix)).
+
+
 ### ClickHouse release v21.7, 2021-07-09

 #### Backward Incompatible Change
@ -1183,13 +1282,6 @@
 * PODArray: Avoid call to memcpy with (nullptr, 0) arguments (Fix UBSan report). This fixes [#18525](https://github.com/ClickHouse/ClickHouse/issues/18525). [#18526](https://github.com/ClickHouse/ClickHouse/pull/18526) ([alexey-milovidov](https://github.com/alexey-milovidov)).
 * Minor improvement for path concatenation of zookeeper paths inside DDLWorker. [#17767](https://github.com/ClickHouse/ClickHouse/pull/17767) ([Bharat Nallan](https://github.com/bharatnc)).
 * Allow to reload symbols from debug file. This PR also fixes a build-id issue. [#17637](https://github.com/ClickHouse/ClickHouse/pull/17637) ([Amos Bird](https://github.com/amosbird)).
-* TestFlows: fixes to LDAP tests that fail due to slow test execution. [#18790](https://github.com/ClickHouse/ClickHouse/pull/18790) ([vzakaznikov](https://github.com/vzakaznikov)).
-* TestFlows: Merging requirements for AES encryption functions. Updating aes_encryption tests to use new requirements. Updating TestFlows version to 1.6.72. [#18221](https://github.com/ClickHouse/ClickHouse/pull/18221) ([vzakaznikov](https://github.com/vzakaznikov)).
-* TestFlows: Updating TestFlows version to the latest 1.6.72. Re-generating requirements.py. [#18208](https://github.com/ClickHouse/ClickHouse/pull/18208) ([vzakaznikov](https://github.com/vzakaznikov)).
-* TestFlows: Updating TestFlows README.md to include "How To Debug Why Test Failed" section. [#17808](https://github.com/ClickHouse/ClickHouse/pull/17808) ([vzakaznikov](https://github.com/vzakaznikov)).
-* TestFlows: tests for RBAC [ACCESS MANAGEMENT](https://clickhouse.tech/docs/en/sql-reference/statements/grant/#grant-access-management) privileges. [#17804](https://github.com/ClickHouse/ClickHouse/pull/17804) ([MyroTk](https://github.com/MyroTk)).
-* TestFlows: RBAC tests for SHOW, TRUNCATE, KILL, and OPTIMIZE. - Updates to old tests. - Resolved comments from #https://github.com/ClickHouse/ClickHouse/pull/16977. [#17657](https://github.com/ClickHouse/ClickHouse/pull/17657) ([MyroTk](https://github.com/MyroTk)).
-* TestFlows: Added RBAC tests for `ATTACH`, `CREATE`, `DROP`, and `DETACH`. [#16977](https://github.com/ClickHouse/ClickHouse/pull/16977) ([MyroTk](https://github.com/MyroTk)).


 ## [Changelog for 2020](https://github.com/ClickHouse/ClickHouse/blob/master/docs/en/whats-new/changelog/2020.md)
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@ -271,12 +271,6 @@ endif()

 include(cmake/cpu_features.cmake)

-option(ARCH_NATIVE "Add -march=native compiler flag. This makes your binaries non-portable but more performant code may be generated.")
-
-if (ARCH_NATIVE)
-    set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native")
-endif ()
-
 # Asynchronous unwind tables are needed for Query Profiler.
 # They are already by default on some platforms but possibly not on all platforms.
 # Enable it explicitly.
@ -401,9 +395,10 @@ endif ()
 # Turns on all external libs like s3, kafka, ODBC, ...
 option(ENABLE_LIBRARIES "Enable all external libraries by default" ON)

-# We recommend avoiding this mode for production builds because we can't guarantee all needed libraries exist in your
-# system.
+# We recommend avoiding this mode for production builds because we can't guarantee
+# all needed libraries exist in your system.
 # This mode exists for enthusiastic developers who are searching for trouble.
+# The whole idea of using unknown version of libraries from the OS distribution is deeply flawed.
 # Useful for maintainers of OS packages.
 option (UNBUNDLED "Use system libraries instead of ones in contrib/" OFF)

@ -542,6 +537,8 @@ include (cmake/find/libpqxx.cmake)
 include (cmake/find/nuraft.cmake)
 include (cmake/find/yaml-cpp.cmake)
 include (cmake/find/s2geometry.cmake)
+include (cmake/find/nlp.cmake)
+include (cmake/find/bzip2.cmake)

 if(NOT USE_INTERNAL_PARQUET_LIBRARY)
    set (ENABLE_ORC OFF CACHE INTERNAL "")
--- a/README.md
+++ b/README.md
@ -15,4 +15,4 @@ ClickHouse® is an open-source column-oriented database management system that a
 * You can also [fill this form](https://clickhouse.tech/#meet) to meet Yandex ClickHouse team in person.

 ## Upcoming Events
-* [ClickHouse Meetup by ByteDance (online)](https://www.meetup.com/ByteDanceDev-group/events/279543467/) on 23 July 2021.
+* [SF Bay Area ClickHouse August Community Meetup (online)](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/events/279109379/) on 25 August 2021.
--- a/base/common/DateLUTImpl.cpp
+++ b/base/common/DateLUTImpl.cpp
@ -60,6 +60,7 @@ DateLUTImpl::DateLUTImpl(const std::string & time_zone_)
    offset_at_start_of_epoch = cctz_time_zone.lookup(cctz_time_zone.lookup(epoch).pre).offset;
    offset_at_start_of_lut = cctz_time_zone.lookup(cctz_time_zone.lookup(lut_start).pre).offset;
    offset_is_whole_number_of_hours_during_epoch = true;
+    offset_is_whole_number_of_minutes_during_epoch = true;

    cctz::civil_day date = lut_start;

@ -108,6 +109,9 @@ DateLUTImpl::DateLUTImpl(const std::string & time_zone_)
        if (offset_is_whole_number_of_hours_during_epoch && start_of_day > 0 && start_of_day % 3600)
            offset_is_whole_number_of_hours_during_epoch = false;

+        if (offset_is_whole_number_of_minutes_during_epoch && start_of_day > 0 && start_of_day % 60)
+            offset_is_whole_number_of_minutes_during_epoch = false;
+
        /// If UTC offset was changed this day.
        /// Change in time zone without transition is possible, e.g. Moscow 1991 Sun, 31 Mar, 02:00 MSK to EEST
        cctz::time_zone::civil_transition transition{};
--- a/base/common/DateLUTImpl.h
+++ b/base/common/DateLUTImpl.h
@ -193,6 +193,7 @@ private:
    /// UTC offset at the beginning of the first supported year.
    Time offset_at_start_of_lut;
    bool offset_is_whole_number_of_hours_during_epoch;
+    bool offset_is_whole_number_of_minutes_during_epoch;

    /// Time zone name.
    std::string time_zone;
@ -251,18 +252,23 @@ private:
    }

    template <typename T, typename Divisor>
-    static inline T roundDown(T x, Divisor divisor)
+    inline T roundDown(T x, Divisor divisor) const
    {
        static_assert(std::is_integral_v<T> && std::is_integral_v<Divisor>);
        assert(divisor > 0);

-        if (likely(x >= 0))
-            return x / divisor * divisor;
+        if (likely(offset_is_whole_number_of_hours_during_epoch))
+        {
+            if (likely(x >= 0))
+                return x / divisor * divisor;

-        /// Integer division for negative numbers rounds them towards zero (up).
-        /// We will shift the number so it will be rounded towards -inf (down).
+            /// Integer division for negative numbers rounds them towards zero (up).
+            /// We will shift the number so it will be rounded towards -inf (down).
+            return (x + 1 - divisor) / divisor * divisor;
+        }

-        return (x + 1 - divisor) / divisor * divisor;
+        Time date = find(x).date;
+        return date + (x - date) / divisor * divisor;
    }

 public:
@ -459,10 +465,21 @@ public:

    inline unsigned toSecond(Time t) const
    {
-        auto res = t % 60;
-        if (likely(res >= 0))
-            return res;
-        return res + 60;
+        if (likely(offset_is_whole_number_of_minutes_during_epoch))
+        {
+            Time res = t % 60;
+            if (likely(res >= 0))
+                return res;
+            return res + 60;
+        }
+
+        LUTIndex index = findIndex(t);
+        Time time = t - lut[index].date;
+
+        if (time >= lut[index].time_at_offset_change())
+            time += lut[index].amount_of_offset_change();
+
+        return time % 60;
    }

    inline unsigned toMinute(Time t) const
@ -483,29 +500,11 @@ public:
    }

    /// NOTE: Assuming timezone offset is a multiple of 15 minutes.
-    inline Time toStartOfMinute(Time t) const { return roundDown(t, 60); }
-    inline Time toStartOfFiveMinute(Time t) const { return roundDown(t, 300); }
-    inline Time toStartOfFifteenMinutes(Time t) const { return roundDown(t, 900); }
-
-    inline Time toStartOfTenMinutes(Time t) const
-    {
-        if (t >= 0 && offset_is_whole_number_of_hours_during_epoch)
-            return t / 600 * 600;
-
-        /// More complex logic is for Nepal - it has offset 05:45. Australia/Eucla is also unfortunate.
-        Time date = find(t).date;
-        return date + (t - date) / 600 * 600;
-    }
-
-    /// NOTE: Assuming timezone transitions are multiple of hours. Lord Howe Island in Australia is a notable exception.
-    inline Time toStartOfHour(Time t) const
-    {
-        if (t >= 0 && offset_is_whole_number_of_hours_during_epoch)
-            return t / 3600 * 3600;
-
-        Time date = find(t).date;
-        return date + (t - date) / 3600 * 3600;
-    }
+    inline Time toStartOfMinute(Time t) const { return toStartOfMinuteInterval(t, 1); }
+    inline Time toStartOfFiveMinute(Time t) const { return toStartOfMinuteInterval(t, 5); }
+    inline Time toStartOfFifteenMinutes(Time t) const { return toStartOfMinuteInterval(t, 15); }
+    inline Time toStartOfTenMinutes(Time t) const { return toStartOfMinuteInterval(t, 10); }
+    inline Time toStartOfHour(Time t) const { return roundDown(t, 3600); }

    /** Number of calendar day since the beginning of UNIX epoch (1970-01-01 is zero)
      * We use just two bytes for it. It covers the range up to 2105 and slightly more.
@ -903,25 +902,24 @@ public:

    inline Time toStartOfMinuteInterval(Time t, UInt64 minutes) const
    {
-        if (minutes == 1)
-            return toStartOfMinute(t);
+        UInt64 divisor = 60 * minutes;
+        if (likely(offset_is_whole_number_of_minutes_during_epoch))
+        {
+            if (likely(t >= 0))
+                return t / divisor * divisor;
+            return (t + 1 - divisor) / divisor * divisor;
+        }

-        /** In contrast to "toStartOfHourInterval" function above,
-          * the minute intervals are not aligned to the midnight.
-          * You will get unexpected results if for example, you round down to 60 minute interval
-          * and there was a time shift to 30 minutes.
-          *
-          * But this is not specified in docs and can be changed in future.
-          */
-
-        UInt64 seconds = 60 * minutes;
-        return roundDown(t, seconds);
+        Time date = find(t).date;
+        return date + (t - date) / divisor * divisor;
    }

    inline Time toStartOfSecondInterval(Time t, UInt64 seconds) const
    {
        if (seconds == 1)
            return t;
+        if (seconds % 60 == 0)
+            return toStartOfMinuteInterval(t, seconds / 60);

        return roundDown(t, seconds);
    }
@ -955,7 +953,7 @@ public:
    inline Time makeDateTime(Int16 year, UInt8 month, UInt8 day_of_month, UInt8 hour, UInt8 minute, UInt8 second) const
    {
        size_t index = makeLUTIndex(year, month, day_of_month);
-        UInt32 time_offset = hour * 3600 + minute * 60 + second;
+        Time time_offset = hour * 3600 + minute * 60 + second;

        if (time_offset >= lut[index].time_at_offset_change())
            time_offset -= lut[index].amount_of_offset_change();
--- a/base/common/memory.h
+++ b/base/common/memory.h
@ -1,57 +0,0 @@
-#pragma once
-
-#include <new>
-#include "defines.h"
-
-#if USE_JEMALLOC
-#    include <jemalloc/jemalloc.h>
-#endif
-
-#if !USE_JEMALLOC || JEMALLOC_VERSION_MAJOR < 4
-#    include <cstdlib>
-#endif
-
-
-namespace Memory
-{
-
-inline ALWAYS_INLINE void * newImpl(std::size_t size)
-{
-    auto * ptr = malloc(size);
-    if (likely(ptr != nullptr))
-        return ptr;
-
-    /// @note no std::get_new_handler logic implemented
-    throw std::bad_alloc{};
-}
-
-inline ALWAYS_INLINE void * newNoExept(std::size_t size) noexcept
-{
-    return malloc(size);
-}
-
-inline ALWAYS_INLINE void deleteImpl(void * ptr) noexcept
-{
-    free(ptr);
-}
-
-#if USE_JEMALLOC && JEMALLOC_VERSION_MAJOR >= 4
-
-inline ALWAYS_INLINE void deleteSized(void * ptr, std::size_t size) noexcept
-{
-    if (unlikely(ptr == nullptr))
-        return;
-
-    sdallocx(ptr, size, 0);
-}
-
-#else
-
-inline ALWAYS_INLINE void deleteSized(void * ptr, std::size_t size [[maybe_unused]]) noexcept
-{
-    free(ptr);
-}
-
-#endif
-
-}
--- a/base/common/removeDuplicates.h
+++ b/base/common/removeDuplicates.h
@ -0,0 +1,24 @@
+#pragma once
+#include <vector>
+
+/// Removes duplicates from a container without changing the order of its elements.
+/// Keeps the last occurrence of each element.
+/// Should NOT be used for containers with a lot of elements because it has O(N^2) complexity.
+template <typename T>
+void removeDuplicatesKeepLast(std::vector<T> & vec)
+{
+    auto begin = vec.begin();
+    auto end = vec.end();
+    auto new_begin = end;
+    for (auto current = end; current != begin;)
+    {
+        --current;
+        if (std::find(new_begin, end, *current) == end)
+        {
+            --new_begin;
+            if (new_begin != current)
+                *new_begin = *current;
+        }
+    }
+    vec.erase(begin, new_begin);
+}
--- a/base/daemon/BaseDaemon.cpp
+++ b/base/daemon/BaseDaemon.cpp
@ -259,10 +259,25 @@ private:
    Poco::Logger * log;
    BaseDaemon & daemon;

-    void onTerminate(const std::string & message, UInt32 thread_num) const
+    void onTerminate(std::string_view message, UInt32 thread_num) const
    {
+        size_t pos = message.find('\n');
+
        LOG_FATAL(log, "(version {}{}, {}) (from thread {}) {}",
-            VERSION_STRING, VERSION_OFFICIAL, daemon.build_id_info, thread_num, message);
+            VERSION_STRING, VERSION_OFFICIAL, daemon.build_id_info, thread_num, message.substr(0, pos));
+
+        /// Print trace from std::terminate exception line-by-line to make it easy for grep.
+        while (pos != std::string_view::npos)
+        {
+            ++pos;
+            size_t next_pos = message.find('\n', pos);
+            size_t size = next_pos;
+            if (next_pos != std::string_view::npos)
+                size = next_pos - pos;
+
+            LOG_FATAL(log, "{}", message.substr(pos, size));
+            pos = next_pos;
+        }
    }

    void onFault(
--- a/base/glibc-compatibility/CMakeLists.txt
+++ b/base/glibc-compatibility/CMakeLists.txt
@ -9,10 +9,6 @@ if (GLIBC_COMPATIBILITY)

    check_include_file("sys/random.h" HAVE_SYS_RANDOM_H)

-    if(COMPILER_CLANG)
-        set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-builtin-requires-header")
-    endif()
-
    add_headers_and_sources(glibc_compatibility .)
    add_headers_and_sources(glibc_compatibility musl)
    if (ARCH_AARCH64)
@ -35,11 +31,9 @@ if (GLIBC_COMPATIBILITY)

    add_library(glibc-compatibility STATIC ${glibc_compatibility_sources})

-    if (COMPILER_CLANG)
-        target_compile_options(glibc-compatibility PRIVATE -Wno-unused-command-line-argument)
-    elseif (COMPILER_GCC)
-        target_compile_options(glibc-compatibility PRIVATE -Wno-unused-but-set-variable)
-    endif ()
+    target_no_warning(glibc-compatibility unused-command-line-argument)
+    target_no_warning(glibc-compatibility unused-but-set-variable)
+    target_no_warning(glibc-compatibility builtin-requires-header)

    target_include_directories(glibc-compatibility PRIVATE libcxxabi ${musl_arch_include_dir})

--- a/base/glibc-compatibility/musl/getauxval.c
+++ b/base/glibc-compatibility/musl/getauxval.c
@ -1,4 +1,5 @@
 #include <sys/auxv.h>
+#include "atomic.h"
 #include <unistd.h> // __environ
 #include <errno.h>

@ -17,18 +18,7 @@ static size_t __find_auxv(unsigned long type)
    return (size_t) -1;
 }

-__attribute__((constructor)) static void __auxv_init()
-{
-    size_t i;
-    for (i = 0; __environ[i]; i++);
-    __auxv = (unsigned long *) (__environ + i + 1);
-
-    size_t secure_idx = __find_auxv(AT_SECURE);
-    if (secure_idx != ((size_t) -1))
-        __auxv_secure = __auxv[secure_idx];
-}
-
-unsigned long getauxval(unsigned long type)
+unsigned long __getauxval(unsigned long type)
 {
    if (type == AT_SECURE)
        return __auxv_secure;
@ -43,3 +33,38 @@ unsigned long getauxval(unsigned long type)
    errno = ENOENT;
    return 0;
 }
+
+static void * volatile getauxval_func;
+
+static unsigned long  __auxv_init(unsigned long type)
+{
+    if (!__environ)
+    {
+        // __environ is not initialized yet so we can't initialize __auxv right now.
+        // That's normally occurred only when getauxval() is called from some sanitizer's internal code.
+        errno = ENOENT;
+        return 0;
+    }
+
+    // Initialize __auxv and __auxv_secure.
+    size_t i;
+    for (i = 0; __environ[i]; i++);
+    __auxv = (unsigned long *) (__environ + i + 1);
+
+    size_t secure_idx = __find_auxv(AT_SECURE);
+    if (secure_idx != ((size_t) -1))
+        __auxv_secure = __auxv[secure_idx];
+
+    // Now we've initialized __auxv, next time getauxval() will only call __get_auxval().
+    a_cas_p(&getauxval_func, (void *)__auxv_init, (void *)__getauxval);
+
+    return __getauxval(type);
+}
+
+// First time getauxval() will call __auxv_init().
+static void * volatile getauxval_func = (void *)__auxv_init;
+
+unsigned long getauxval(unsigned long type)
+{
+    return ((unsigned long (*)(unsigned long))getauxval_func)(type);
+}
--- a/base/mysqlxx/Pool.cpp
+++ b/base/mysqlxx/Pool.cpp
@ -296,7 +296,7 @@ void Pool::initialize()

 Pool::Connection * Pool::allocConnection(bool dont_throw_if_failed_first_time)
 {
-    std::unique_ptr<Connection> conn_ptr{new Connection};
+    std::unique_ptr conn_ptr = std::make_unique<Connection>();

    try
    {
--- a/benchmark/clickhouse/benchmark-new.sh
+++ b/benchmark/clickhouse/benchmark-new.sh
@ -4,13 +4,24 @@ QUERIES_FILE="queries.sql"
 TABLE=$1
 TRIES=3

+if [ -x ./clickhouse ]
+then
+    CLICKHOUSE_CLIENT="./clickhouse client"
+elif command -v clickhouse-client >/dev/null 2>&1
+then
+    CLICKHOUSE_CLIENT="clickhouse-client"
+else
+    echo "clickhouse-client is not found"
+    exit 1
+fi
+
 cat "$QUERIES_FILE" | sed "s/{table}/${TABLE}/g" | while read query; do
    sync
    echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null

    echo -n "["
    for i in $(seq 1 $TRIES); do
-        RES=$(clickhouse-client --time --format=Null --query="$query" 2>&1)
+        RES=$(${CLICKHOUSE_CLIENT} --time --format=Null --max_memory_usage=100G --query="$query" 2>&1)
        [[ "$?" == "0" ]] && echo -n "${RES}" || echo -n "null"
        [[ "$i" != $TRIES ]] && echo -n ", "
    done
--- a/benchmark/hardware.sh
+++ b/benchmark/hardware.sh
@ -11,8 +11,8 @@ DATASET="${TABLE}_v1.tar.xz"
 QUERIES_FILE="queries.sql"
 TRIES=3

-AMD64_BIN_URL="https://clickhouse-builds.s3.yandex.net/0/e29c4c3cc47ab2a6c4516486c1b77d57e7d42643/clickhouse_build_check/gcc-10_relwithdebuginfo_none_bundled_unsplitted_disable_False_binary/clickhouse"
-AARCH64_BIN_URL="https://clickhouse-builds.s3.yandex.net/0/e29c4c3cc47ab2a6c4516486c1b77d57e7d42643/clickhouse_special_build_check/clang-10-aarch64_relwithdebuginfo_none_bundled_unsplitted_disable_False_binary/clickhouse"
+AMD64_BIN_URL="https://builds.clickhouse.tech/master/amd64/clickhouse"
+AARCH64_BIN_URL="https://builds.clickhouse.tech/master/aarch64/clickhouse"

 # Note: on older Ubuntu versions, 'axel' does not support IPv6. If you are using IPv6-only servers on very old Ubuntu, just don't install 'axel'.

@ -89,7 +89,7 @@ cat "$QUERIES_FILE" | sed "s/{table}/${TABLE}/g" | while read query; do

    echo -n "["
    for i in $(seq 1 $TRIES); do
-        RES=$(./clickhouse client --max_memory_usage 100000000000 --time --format=Null --query="$query" 2>&1 ||:)
+        RES=$(./clickhouse client --max_memory_usage 100G --time --format=Null --query="$query" 2>&1 ||:)
        [[ "$?" == "0" ]] && echo -n "${RES}" || echo -n "null"
        [[ "$i" != $TRIES ]] && echo -n ", "
    done
--- a/cmake/add_warning.cmake
+++ b/cmake/add_warning.cmake
@ -27,3 +27,22 @@ endmacro ()
 macro (no_warning flag)
    add_warning(no-${flag})
 endmacro ()
+
+
+# The same but only for specified target.
+macro (target_add_warning target flag)
+    string (REPLACE "-" "_" underscored_flag ${flag})
+    string (REPLACE "+" "x" underscored_flag ${underscored_flag})
+
+    check_cxx_compiler_flag("-W${flag}" SUPPORTS_CXXFLAG_${underscored_flag})
+
+    if (SUPPORTS_CXXFLAG_${underscored_flag})
+        target_compile_options (${target} PRIVATE "-W${flag}")
+    else ()
+        message (WARNING "Flag -W${flag} is unsupported")
+    endif ()
+endmacro ()
+
+macro (target_no_warning target flag)
+    target_add_warning(${target} no-${flag})
+endmacro ()
--- a/cmake/autogenerated_versions.txt
+++ b/cmake/autogenerated_versions.txt
@ -6,7 +6,7 @@ SET(VERSION_REVISION 54454)
 SET(VERSION_MAJOR 21)
 SET(VERSION_MINOR 9)
 SET(VERSION_PATCH 1)
-SET(VERSION_GITHASH f48c5af90c2ad51955d1ee3b6b05d006b03e4238)
-SET(VERSION_DESCRIBE v21.9.1.1-prestable)
-SET(VERSION_STRING 21.9.1.1)
+SET(VERSION_GITHASH f063e44131a048ba2d9af8075f03700fd5ec3e69)
+SET(VERSION_DESCRIBE v21.9.1.7770-prestable)
+SET(VERSION_STRING 21.9.1.7770)
 # end of autochange
--- a/cmake/cpu_features.cmake
+++ b/cmake/cpu_features.cmake
@ -5,109 +5,128 @@ include (CMakePushCheckState)

 cmake_push_check_state ()

-# gcc -dM -E -mno-sse2 - < /dev/null | sort > gcc-dump-nosse2
-# gcc -dM -E -msse2 - < /dev/null | sort > gcc-dump-sse2
-#define __SSE2__ 1
-#define __SSE2_MATH__ 1
+# The variables HAVE_* determine if compiler has support for the flag to use the corresponding instruction set.
+# The options ENABLE_* determine if we will tell compiler to actually use the corresponding instruction set if compiler can do it.

-# gcc -dM -E -msse4.1 - < /dev/null | sort > gcc-dump-sse41
-#define __SSE4_1__ 1
+# All of them are unrelated to the instruction set at the host machine
+# (you can compile for newer instruction set on old machines and vice versa).

-set (TEST_FLAG "-msse4.1")
-set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
-check_cxx_source_compiles("
-    #include <smmintrin.h>
-    int main() {
-        auto a = _mm_insert_epi8(__m128i(), 0, 0);
-        (void)a;
-        return 0;
-    }
-" HAVE_SSE41)
-if (HAVE_SSE41)
-    set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
-endif ()
+option (ENABLE_SSSE3 "Use SSSE3 instructions on x86_64" 1)
+option (ENABLE_SSE41 "Use SSE4.1 instructions on x86_64" 1)
+option (ENABLE_SSE42 "Use SSE4.2 instructions on x86_64" 1)
+option (ENABLE_PCLMULQDQ "Use pclmulqdq instructions on x86_64" 1)
+option (ENABLE_POPCNT "Use popcnt instructions on x86_64" 1)
+option (ENABLE_AVX "Use AVX instructions on x86_64" 0)
+option (ENABLE_AVX2 "Use AVX2 instructions on x86_64" 0)

-if (ARCH_PPC64LE)
-    set (COMPILER_FLAGS "${COMPILER_FLAGS} -maltivec -D__SSE2__=1 -DNO_WARN_X86_INTRINSICS")
-endif ()
+option (ARCH_NATIVE "Add -march=native compiler flag. This makes your binaries non-portable but more performant code may be generated. This option overrides ENABLE_* options for specific instruction set. Highly not recommended to use." 0)

-# gcc -dM -E -msse4.2 - < /dev/null | sort > gcc-dump-sse42
-#define __SSE4_2__ 1
+if (ARCH_NATIVE)
+    set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native")

-set (TEST_FLAG "-msse4.2")
-set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
-check_cxx_source_compiles("
-    #include <nmmintrin.h>
-    int main() {
-        auto a = _mm_crc32_u64(0, 0);
-        (void)a;
-        return 0;
-    }
-" HAVE_SSE42)
-if (HAVE_SSE42)
-    set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
-endif ()
+else ()
+    set (TEST_FLAG "-mssse3")
+    set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
+    check_cxx_source_compiles("
+        #include <tmmintrin.h>
+        int main() {
+            __m64 a = _mm_abs_pi8(__m64());
+            (void)a;
+            return 0;
+        }
+    " HAVE_SSSE3)
+    if (HAVE_SSSE3 AND ENABLE_SSSE3)
+        set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
+    endif ()

-set (TEST_FLAG "-mssse3")
-set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
-check_cxx_source_compiles("
-    #include <tmmintrin.h>
-    int main() {
-        __m64 a = _mm_abs_pi8(__m64());
-        (void)a;
-        return 0;
-    }
-" HAVE_SSSE3)

-set (TEST_FLAG "-mavx")
-set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
-check_cxx_source_compiles("
-    #include <immintrin.h>
-    int main() {
-        auto a = _mm256_insert_epi8(__m256i(), 0, 0);
-        (void)a;
-        return 0;
-    }
-" HAVE_AVX)
+    set (TEST_FLAG "-msse4.1")
+    set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
+    check_cxx_source_compiles("
+        #include <smmintrin.h>
+        int main() {
+            auto a = _mm_insert_epi8(__m128i(), 0, 0);
+            (void)a;
+            return 0;
+        }
+    " HAVE_SSE41)
+    if (HAVE_SSE41 AND ENABLE_SSE41)
+        set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
+    endif ()

-set (TEST_FLAG "-mavx2")
-set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
-check_cxx_source_compiles("
-    #include <immintrin.h>
-    int main() {
-        auto a = _mm256_add_epi16(__m256i(), __m256i());
-        (void)a;
-        return 0;
-    }
-" HAVE_AVX2)
+    if (ARCH_PPC64LE)
+        set (COMPILER_FLAGS "${COMPILER_FLAGS} -maltivec -D__SSE2__=1 -DNO_WARN_X86_INTRINSICS")
+    endif ()

-set (TEST_FLAG "-mpclmul")
-set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
-check_cxx_source_compiles("
-    #include <wmmintrin.h>
-    int main() {
-        auto a = _mm_clmulepi64_si128(__m128i(), __m128i(), 0);
-        (void)a;
-        return 0;
-    }
-" HAVE_PCLMULQDQ)
+    set (TEST_FLAG "-msse4.2")
+    set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
+    check_cxx_source_compiles("
+        #include <nmmintrin.h>
+        int main() {
+            auto a = _mm_crc32_u64(0, 0);
+            (void)a;
+            return 0;
+        }
+    " HAVE_SSE42)
+    if (HAVE_SSE42 AND ENABLE_SSE42)
+        set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
+    endif ()

-# gcc -dM -E -mpopcnt - < /dev/null | sort > gcc-dump-popcnt
-#define __POPCNT__ 1
+    set (TEST_FLAG "-mpclmul")
+    set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
+    check_cxx_source_compiles("
+        #include <wmmintrin.h>
+        int main() {
+            auto a = _mm_clmulepi64_si128(__m128i(), __m128i(), 0);
+            (void)a;
+            return 0;
+        }
+    " HAVE_PCLMULQDQ)
+    if (HAVE_PCLMULQDQ AND ENABLE_PCLMULQDQ)
+        set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
+    endif ()

-set (TEST_FLAG "-mpopcnt")
+    set (TEST_FLAG "-mpopcnt")

-set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
-check_cxx_source_compiles("
-    int main() {
-        auto a = __builtin_popcountll(0);
-        (void)a;
-        return 0;
-    }
-" HAVE_POPCNT)
+    set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
+    check_cxx_source_compiles("
+        int main() {
+            auto a = __builtin_popcountll(0);
+            (void)a;
+            return 0;
+        }
+    " HAVE_POPCNT)
+    if (HAVE_POPCNT AND ENABLE_POPCNT)
+        set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
+    endif ()

-if (HAVE_POPCNT AND NOT ARCH_AARCH64)
-    set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
+    set (TEST_FLAG "-mavx")
+    set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
+    check_cxx_source_compiles("
+        #include <immintrin.h>
+        int main() {
+            auto a = _mm256_insert_epi8(__m256i(), 0, 0);
+            (void)a;
+            return 0;
+        }
+    " HAVE_AVX)
+    if (HAVE_AVX AND ENABLE_AVX)
+        set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
+    endif ()
+
+    set (TEST_FLAG "-mavx2")
+    set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
+    check_cxx_source_compiles("
+        #include <immintrin.h>
+        int main() {
+            auto a = _mm256_add_epi16(__m256i(), __m256i());
+            (void)a;
+            return 0;
+        }
+    " HAVE_AVX2)
+    if (HAVE_AVX2 AND ENABLE_AVX2)
+        set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
+    endif ()
 endif ()

 cmake_pop_check_state ()
--- a/cmake/find/bzip2.cmake
+++ b/cmake/find/bzip2.cmake
@ -0,0 +1,19 @@
+option(ENABLE_BZIP2 "Enable bzip2 compression support" ${ENABLE_LIBRARIES})
+
+if (NOT ENABLE_BZIP2)
+    message (STATUS "bzip2 compression disabled")
+    return()
+endif()
+
+if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/bzip2/bzlib.h")
+    message (WARNING "submodule contrib/bzip2 is missing. to fix try run: \n git submodule update --init --recursive")
+    message (${RECONFIGURE_MESSAGE_LEVEL} "Can't find internal bzip2 library")
+    set (USE_NLP 0)
+    return()
+endif ()
+
+set (USE_BZIP2 1)
+set (BZIP2_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/bzip2")
+set (BZIP2_LIBRARY bzip2)
+
+message (STATUS "Using bzip2=${USE_BZIP2}: ${BZIP2_INCLUDE_DIR} : ${BZIP2_LIBRARY}")
--- a/cmake/find/nlp.cmake
+++ b/cmake/find/nlp.cmake
@ -0,0 +1,32 @@
+option(ENABLE_NLP "Enable NLP functions support" ${ENABLE_LIBRARIES})
+
+if (NOT ENABLE_NLP)
+
+    message (STATUS "NLP functions disabled")
+    return()
+endif()
+
+if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/libstemmer_c/Makefile")
+    message (WARNING "submodule contrib/libstemmer_c is missing. to fix try run: \n git submodule update --init --recursive")
+    message (${RECONFIGURE_MESSAGE_LEVEL} "Can't find internal libstemmer_c library, NLP functions will be disabled")
+    set (USE_NLP 0)
+    return()
+endif ()
+
+if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/wordnet-blast/CMakeLists.txt")
+    message (WARNING "submodule contrib/wordnet-blast is missing. to fix try run: \n git submodule update --init --recursive")
+    message (${RECONFIGURE_MESSAGE_LEVEL} "Can't find internal wordnet-blast library, NLP functions will be disabled")
+    set (USE_NLP 0)
+    return()
+endif ()
+
+if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/lemmagen-c/README.md")
+    message (WARNING "submodule contrib/lemmagen-c is missing. to fix try run: \n git submodule update --init --recursive")
+    message (${RECONFIGURE_MESSAGE_LEVEL} "Can't find internal lemmagen-c library, NLP functions will be disabled")
+    set (USE_NLP 0)
+    return()
+endif ()
+
+set (USE_NLP 1)
+
+message (STATUS "Using Libraries for NLP functions: contrib/wordnet-blast, contrib/libstemmer_c, contrib/lemmagen-c")
--- a/contrib/AMQP-CPP
+++ b/contrib/AMQP-CPP
@ -1 +1 @@
-Subproject commit 03781aaff0f10ef41f902b8cf865fe0067180c10
+Subproject commit 1a6c51f4ac51ac56610fa95081bd2f349911375a
--- a/contrib/CMakeLists.txt
+++ b/contrib/CMakeLists.txt
@ -328,6 +328,16 @@ endif()

 add_subdirectory(fast_float)

+if (USE_NLP)
+    add_subdirectory(libstemmer-c-cmake)
+    add_subdirectory(wordnet-blast-cmake)
+    add_subdirectory(lemmagen-c-cmake)
+endif()
+
+if (USE_BZIP2)
+    add_subdirectory(bzip2-cmake)
+endif()
+
 if (USE_SQLITE)
    add_subdirectory(sqlite-cmake)
 endif()
--- a/contrib/NuRaft
+++ b/contrib/NuRaft
@ -1 +1 @@
-Subproject commit 976874b7aa7f422bf4ea595bb7d1166c617b1c26
+Subproject commit 7ecb16844af6a9c283ad432d85ecc2e7d1544676
--- a/contrib/amqpcpp-cmake/CMakeLists.txt
+++ b/contrib/amqpcpp-cmake/CMakeLists.txt
@ -10,11 +10,12 @@ set (SRCS
    "${LIBRARY_DIR}/src/deferredconsumer.cpp"
    "${LIBRARY_DIR}/src/deferredextreceiver.cpp"
    "${LIBRARY_DIR}/src/deferredget.cpp"
-    "${LIBRARY_DIR}/src/deferredpublisher.cpp"
+    "${LIBRARY_DIR}/src/deferredrecall.cpp"
    "${LIBRARY_DIR}/src/deferredreceiver.cpp"
    "${LIBRARY_DIR}/src/field.cpp"
    "${LIBRARY_DIR}/src/flags.cpp"
    "${LIBRARY_DIR}/src/linux_tcp/openssl.cpp"
+    "${LIBRARY_DIR}/src/linux_tcp/sslerrorprinter.cpp"
    "${LIBRARY_DIR}/src/linux_tcp/tcpconnection.cpp"
    "${LIBRARY_DIR}/src/inbuffer.cpp"
    "${LIBRARY_DIR}/src/receivedframe.cpp"
--- a/contrib/arrow
+++ b/contrib/arrow
@ -1 +1 @@
-Subproject commit debf751a129bdda9ff4d1e895e08957ff77000a1
+Subproject commit 078e21bad344747b7656ef2d7a4f7410a0a303eb
--- a/contrib/arrow-cmake/CMakeLists.txt
+++ b/contrib/arrow-cmake/CMakeLists.txt
@ -119,12 +119,9 @@ set(ORC_SRCS
        "${ORC_SOURCE_SRC_DIR}/ColumnWriter.cc"
        "${ORC_SOURCE_SRC_DIR}/Common.cc"
        "${ORC_SOURCE_SRC_DIR}/Compression.cc"
-        "${ORC_SOURCE_SRC_DIR}/Exceptions.cc"
        "${ORC_SOURCE_SRC_DIR}/Int128.cc"
        "${ORC_SOURCE_SRC_DIR}/LzoDecompressor.cc"
        "${ORC_SOURCE_SRC_DIR}/MemoryPool.cc"
-        "${ORC_SOURCE_SRC_DIR}/OrcFile.cc"
-        "${ORC_SOURCE_SRC_DIR}/Reader.cc"
        "${ORC_SOURCE_SRC_DIR}/RLE.cc"
        "${ORC_SOURCE_SRC_DIR}/RLEv1.cc"
        "${ORC_SOURCE_SRC_DIR}/RLEv2.cc"
@ -194,9 +191,18 @@ set(ARROW_SRCS
        "${LIBRARY_DIR}/compute/cast.cc"
        "${LIBRARY_DIR}/compute/exec.cc"
        "${LIBRARY_DIR}/compute/function.cc"
+        "${LIBRARY_DIR}/compute/function_internal.cc"
        "${LIBRARY_DIR}/compute/kernel.cc"
        "${LIBRARY_DIR}/compute/registry.cc"

+        "${LIBRARY_DIR}/compute/exec/exec_plan.cc"
+        "${LIBRARY_DIR}/compute/exec/expression.cc"
+        "${LIBRARY_DIR}/compute/exec/key_compare.cc"
+        "${LIBRARY_DIR}/compute/exec/key_encode.cc"
+        "${LIBRARY_DIR}/compute/exec/key_hash.cc"
+        "${LIBRARY_DIR}/compute/exec/key_map.cc"
+        "${LIBRARY_DIR}/compute/exec/util.cc"
+
        "${LIBRARY_DIR}/compute/kernels/aggregate_basic.cc"
        "${LIBRARY_DIR}/compute/kernels/aggregate_mode.cc"
        "${LIBRARY_DIR}/compute/kernels/aggregate_quantile.cc"
@ -207,6 +213,7 @@ set(ARROW_SRCS
        "${LIBRARY_DIR}/compute/kernels/scalar_arithmetic.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_boolean.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_cast_boolean.cc"
+        "${LIBRARY_DIR}/compute/kernels/scalar_cast_dictionary.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_cast_internal.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_cast_nested.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_cast_numeric.cc"
@ -214,15 +221,18 @@ set(ARROW_SRCS
        "${LIBRARY_DIR}/compute/kernels/scalar_cast_temporal.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_compare.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_fill_null.cc"
+        "${LIBRARY_DIR}/compute/kernels/scalar_if_else.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_nested.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_set_lookup.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_string.cc"
+        "${LIBRARY_DIR}/compute/kernels/scalar_temporal.cc"
        "${LIBRARY_DIR}/compute/kernels/scalar_validity.cc"
+        "${LIBRARY_DIR}/compute/kernels/util_internal.cc"
        "${LIBRARY_DIR}/compute/kernels/vector_hash.cc"
        "${LIBRARY_DIR}/compute/kernels/vector_nested.cc"
+        "${LIBRARY_DIR}/compute/kernels/vector_replace.cc"
        "${LIBRARY_DIR}/compute/kernels/vector_selection.cc"
        "${LIBRARY_DIR}/compute/kernels/vector_sort.cc"
-        "${LIBRARY_DIR}/compute/kernels/util_internal.cc"

        "${LIBRARY_DIR}/csv/chunker.cc"
        "${LIBRARY_DIR}/csv/column_builder.cc"
@ -231,6 +241,7 @@ set(ARROW_SRCS
        "${LIBRARY_DIR}/csv/options.cc"
        "${LIBRARY_DIR}/csv/parser.cc"
        "${LIBRARY_DIR}/csv/reader.cc"
+        "${LIBRARY_DIR}/csv/writer.cc"

        "${LIBRARY_DIR}/ipc/dictionary.cc"
        "${LIBRARY_DIR}/ipc/feather.cc"
@ -247,6 +258,7 @@ set(ARROW_SRCS
        "${LIBRARY_DIR}/io/interfaces.cc"
        "${LIBRARY_DIR}/io/memory.cc"
        "${LIBRARY_DIR}/io/slow.cc"
+        "${LIBRARY_DIR}/io/stdio.cc"
        "${LIBRARY_DIR}/io/transform.cc"

        "${LIBRARY_DIR}/tensor/coo_converter.cc"
@ -257,9 +269,9 @@ set(ARROW_SRCS
        "${LIBRARY_DIR}/util/bit_block_counter.cc"
        "${LIBRARY_DIR}/util/bit_run_reader.cc"
        "${LIBRARY_DIR}/util/bit_util.cc"
-        "${LIBRARY_DIR}/util/bitmap.cc"
        "${LIBRARY_DIR}/util/bitmap_builders.cc"
        "${LIBRARY_DIR}/util/bitmap_ops.cc"
+        "${LIBRARY_DIR}/util/bitmap.cc"
        "${LIBRARY_DIR}/util/bpacking.cc"
        "${LIBRARY_DIR}/util/cancel.cc"
        "${LIBRARY_DIR}/util/compression.cc"
--- a/contrib/boost
+++ b/contrib/boost
@ -1 +1 @@
-Subproject commit 1ccbb5a522a571ce83b606dbc2e1011c42ecccfb
+Subproject commit 9cf09dbfd55a5c6202dedbdf40781a51b02c2675
--- a/contrib/boost-cmake/CMakeLists.txt
+++ b/contrib/boost-cmake/CMakeLists.txt
@ -13,11 +13,12 @@ if (NOT USE_INTERNAL_BOOST_LIBRARY)
        regex
        context
        coroutine
+        graph
    )

    if(Boost_INCLUDE_DIR AND Boost_FILESYSTEM_LIBRARY AND Boost_FILESYSTEM_LIBRARY AND
        Boost_PROGRAM_OPTIONS_LIBRARY AND Boost_REGEX_LIBRARY AND Boost_SYSTEM_LIBRARY AND Boost_CONTEXT_LIBRARY AND
-        Boost_COROUTINE_LIBRARY)
+        Boost_COROUTINE_LIBRARY AND Boost_GRAPH_LIBRARY)

        set(EXTERNAL_BOOST_FOUND 1)

@ -32,6 +33,7 @@ if (NOT USE_INTERNAL_BOOST_LIBRARY)
        add_library (_boost_system INTERFACE)
        add_library (_boost_context INTERFACE)
        add_library (_boost_coroutine INTERFACE)
+        add_library (_boost_graph INTERFACE)

        target_link_libraries (_boost_filesystem INTERFACE ${Boost_FILESYSTEM_LIBRARY})
        target_link_libraries (_boost_iostreams INTERFACE ${Boost_IOSTREAMS_LIBRARY})
@ -40,6 +42,7 @@ if (NOT USE_INTERNAL_BOOST_LIBRARY)
        target_link_libraries (_boost_system INTERFACE ${Boost_SYSTEM_LIBRARY})
        target_link_libraries (_boost_context INTERFACE ${Boost_CONTEXT_LIBRARY})
        target_link_libraries (_boost_coroutine INTERFACE ${Boost_COROUTINE_LIBRARY})
+        target_link_libraries (_boost_graph INTERFACE ${Boost_GRAPH_LIBRARY})

        add_library (boost::filesystem ALIAS _boost_filesystem)
        add_library (boost::iostreams ALIAS _boost_iostreams)
@ -48,6 +51,7 @@ if (NOT USE_INTERNAL_BOOST_LIBRARY)
        add_library (boost::system ALIAS _boost_system)
        add_library (boost::context ALIAS _boost_context)
        add_library (boost::coroutine ALIAS _boost_coroutine)
+        add_library (boost::graph ALIAS _boost_graph)
    else()
        set(EXTERNAL_BOOST_FOUND 0)
        message (${RECONFIGURE_MESSAGE_LEVEL} "Can't find system boost")
@ -221,4 +225,17 @@ if (NOT EXTERNAL_BOOST_FOUND)
    add_library (boost::coroutine ALIAS _boost_coroutine)
    target_include_directories (_boost_coroutine PRIVATE ${LIBRARY_DIR})
    target_link_libraries(_boost_coroutine PRIVATE _boost_context)
+
+    # graph
+
+    set (SRCS_GRAPH
+        "${LIBRARY_DIR}/libs/graph/src/graphml.cpp"
+        "${LIBRARY_DIR}/libs/graph/src/read_graphviz_new.cpp"
+    )
+
+    add_library (_boost_graph ${SRCS_GRAPH})
+    add_library (boost::graph ALIAS _boost_graph)
+    target_include_directories (_boost_graph PRIVATE ${LIBRARY_DIR})
+    target_link_libraries(_boost_graph PRIVATE _boost_regex)
+
 endif ()
--- a/contrib/bzip2
+++ b/contrib/bzip2
@ -0,0 +1 @@
+Subproject commit bf905ea2251191ff9911ae7ec0cfc35d41f9f7f6
--- a/contrib/bzip2-cmake/CMakeLists.txt
+++ b/contrib/bzip2-cmake/CMakeLists.txt
@ -0,0 +1,23 @@
+set(BZIP2_SOURCE_DIR "${ClickHouse_SOURCE_DIR}/contrib/bzip2")
+set(BZIP2_BINARY_DIR "${ClickHouse_BINARY_DIR}/contrib/bzip2")
+
+set(SRCS
+    "${BZIP2_SOURCE_DIR}/blocksort.c"
+    "${BZIP2_SOURCE_DIR}/huffman.c"
+    "${BZIP2_SOURCE_DIR}/crctable.c"
+    "${BZIP2_SOURCE_DIR}/randtable.c"
+    "${BZIP2_SOURCE_DIR}/compress.c"
+    "${BZIP2_SOURCE_DIR}/decompress.c"
+    "${BZIP2_SOURCE_DIR}/bzlib.c"
+)
+
+# From bzip2/CMakeLists.txt 
+set(BZ_VERSION "1.0.7")
+configure_file (
+    "${BZIP2_SOURCE_DIR}/bz_version.h.in"
+    "${BZIP2_BINARY_DIR}/bz_version.h"
+)
+
+add_library(bzip2 ${SRCS})
+
+target_include_directories(bzip2 PUBLIC "${BZIP2_SOURCE_DIR}" "${BZIP2_BINARY_DIR}")
--- a/contrib/croaring-cmake/CMakeLists.txt
+++ b/contrib/croaring-cmake/CMakeLists.txt
@ -24,3 +24,15 @@ add_library(roaring ${SRCS})
 target_include_directories(roaring PRIVATE "${LIBRARY_DIR}/include/roaring")
 target_include_directories(roaring SYSTEM BEFORE PUBLIC "${LIBRARY_DIR}/include")
 target_include_directories(roaring SYSTEM BEFORE PUBLIC "${LIBRARY_DIR}/cpp")
+
+# We redirect malloc/free family of functions to different functions that will track memory in ClickHouse.
+# Also note that we exploit implicit function declarations.
+target_compile_definitions(roaring PRIVATE
+    -Dmalloc=clickhouse_malloc
+    -Dcalloc=clickhouse_calloc
+    -Drealloc=clickhouse_realloc
+    -Dreallocarray=clickhouse_reallocarray
+    -Dfree=clickhouse_free
+    -Dposix_memalign=clickhouse_posix_memalign)
+
+target_link_libraries(roaring PUBLIC clickhouse_common_io)
--- a/contrib/jemalloc-cmake/CMakeLists.txt
+++ b/contrib/jemalloc-cmake/CMakeLists.txt
@ -1,10 +1,9 @@
+# Disabled under OSX until https://github.com/ClickHouse/ClickHouse/issues/27568 is fixed
 if (SANITIZE OR NOT (
-    ((OS_LINUX OR OS_FREEBSD) AND (ARCH_AMD64 OR ARCH_ARM OR ARCH_PPC64LE)) OR
-    (OS_DARWIN AND (CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo" OR CMAKE_BUILD_TYPE STREQUAL "Debug"))
-))
+    ((OS_LINUX OR OS_FREEBSD) AND (ARCH_AMD64 OR ARCH_ARM OR ARCH_PPC64LE))))
    if (ENABLE_JEMALLOC)
        message (${RECONFIGURE_MESSAGE_LEVEL}
-                 "jemalloc is disabled implicitly: it doesn't work with sanitizers and can only be used with x86_64, aarch64, or ppc64le Linux or FreeBSD builds and RelWithDebInfo macOS builds.")
+                 "jemalloc is disabled implicitly: it doesn't work with sanitizers and can only be used with x86_64, aarch64, or ppc64le Linux or FreeBSD builds")
    endif ()
    set (ENABLE_JEMALLOC OFF)
 else ()
--- a/contrib/lemmagen-c
+++ b/contrib/lemmagen-c
@ -0,0 +1 @@
+Subproject commit 59537bdcf57bbed17913292cb4502d15657231f1
--- a/contrib/lemmagen-c-cmake/CMakeLists.txt
+++ b/contrib/lemmagen-c-cmake/CMakeLists.txt
@ -0,0 +1,9 @@
+set(LIBRARY_DIR "${ClickHouse_SOURCE_DIR}/contrib/lemmagen-c")
+set(LEMMAGEN_INCLUDE_DIR "${LIBRARY_DIR}/include")
+
+set(SRCS
+    "${LIBRARY_DIR}/src/RdrLemmatizer.cpp"
+)
+
+add_library(lemmagen STATIC ${SRCS})
+target_include_directories(lemmagen PUBLIC "${LEMMAGEN_INCLUDE_DIR}")
--- a/contrib/libmetrohash/CMakeLists.txt
+++ b/contrib/libmetrohash/CMakeLists.txt
@ -2,9 +2,5 @@ set (SRCS
    src/metrohash64.cpp
    src/metrohash128.cpp
 )
-if (HAVE_SSE42) # Not used. Pretty easy to port.
-    list (APPEND SRCS src/metrohash128crc.cpp)
-endif ()
-
 add_library(metrohash ${SRCS})
 target_include_directories(metrohash PUBLIC src)
--- a/contrib/libstemmer-c-cmake/CMakeLists.txt
+++ b/contrib/libstemmer-c-cmake/CMakeLists.txt
@ -0,0 +1,31 @@
+set(LIBRARY_DIR "${ClickHouse_SOURCE_DIR}/contrib/libstemmer_c")
+set(STEMMER_INCLUDE_DIR "${LIBRARY_DIR}/include")
+
+FILE ( READ "${LIBRARY_DIR}/mkinc.mak" _CONTENT )
+# replace '\ ' into one big line
+STRING ( REGEX REPLACE "\\\\\n  " " ${LIBRARY_DIR}/" _CONTENT "${_CONTENT}" )
+# escape ';' (if any)
+STRING ( REGEX REPLACE ";" "\\\\;" _CONTENT "${_CONTENT}" )
+# now replace lf into ';' (it makes list from the line)
+STRING ( REGEX REPLACE "\n" ";" _CONTENT "${_CONTENT}" )
+FOREACH ( LINE ${_CONTENT} )
+    # skip comments (beginning with #)
+    IF ( NOT "${LINE}" MATCHES "^#.*" )
+        # parse 'name=value1 value2..." - extract the 'name' part
+        STRING ( REGEX REPLACE "=.*$" "" _NAME "${LINE}" )
+        # extract the list of values part
+        STRING ( REGEX REPLACE "^.*=" "" _LIST "${LINE}" )
+        # replace (multi)spaces into ';' (it makes list from the line)
+        STRING ( REGEX REPLACE " +" ";" _LIST "${_LIST}" )
+        # finally get our two variables
+        IF ( "${_NAME}" MATCHES "snowball_sources" )
+            SET ( _SOURCES "${_LIST}" )
+        ELSEIF ( "${_NAME}" MATCHES "snowball_headers" )
+            SET ( _HEADERS "${_LIST}" )
+        ENDIF ()
+    endif ()
+endforeach ()
+
+# all the sources parsed. Now just add the lib
+add_library ( stemmer STATIC ${_SOURCES} ${_HEADERS} )
+target_include_directories (stemmer PUBLIC "${STEMMER_INCLUDE_DIR}")
--- a/contrib/libstemmer_c
+++ b/contrib/libstemmer_c
@ -0,0 +1 @@
+Subproject commit c753054304d87daf460057c1a649c482aa094835
--- a/contrib/nuraft-cmake/CMakeLists.txt
+++ b/contrib/nuraft-cmake/CMakeLists.txt
@ -22,6 +22,7 @@ set(SRCS
    "${LIBRARY_DIR}/src/launcher.cxx"
    "${LIBRARY_DIR}/src/srv_config.cxx"
    "${LIBRARY_DIR}/src/snapshot_sync_req.cxx"
+    "${LIBRARY_DIR}/src/snapshot_sync_ctx.cxx"
    "${LIBRARY_DIR}/src/handle_timeout.cxx"
    "${LIBRARY_DIR}/src/handle_append_entries.cxx"
    "${LIBRARY_DIR}/src/cluster_config.cxx"
--- a/contrib/protobuf
+++ b/contrib/protobuf
@ -1 +1 @@
-Subproject commit 73b12814204ad9068ba352914d0dc244648b48ee
+Subproject commit 75601841d172c73ae6bf4ce8121f42b875cdbabd
--- a/contrib/rocksdb
+++ b/contrib/rocksdb
@ -1 +1 @@
-Subproject commit dac0e9a68080c837d6b6223921f3fc151abbfcdc
+Subproject commit b6480c69bf3ab6e298e0d019a07fd4f69029b26a
--- a/contrib/simdjson-cmake/CMakeLists.txt
+++ b/contrib/simdjson-cmake/CMakeLists.txt
@ -4,3 +4,6 @@ set(SIMDJSON_SRC "${SIMDJSON_SRC_DIR}/simdjson.cpp")

 add_library(simdjson ${SIMDJSON_SRC})
 target_include_directories(simdjson SYSTEM PUBLIC "${SIMDJSON_INCLUDE_DIR}" PRIVATE "${SIMDJSON_SRC_DIR}")
+
+# simdjson is using its own CPU dispatching and get confused if we enable AVX/AVX2 flags.
+target_compile_options(simdjson PRIVATE -mno-avx -mno-avx2)
--- a/contrib/wordnet-blast
+++ b/contrib/wordnet-blast
@ -0,0 +1 @@
+Subproject commit 1d16ac28036e19fe8da7ba72c16a307fbdf8c87e
--- a/contrib/wordnet-blast-cmake/CMakeLists.txt
+++ b/contrib/wordnet-blast-cmake/CMakeLists.txt
@ -0,0 +1,13 @@
+set(LIBRARY_DIR "${ClickHouse_SOURCE_DIR}/contrib/wordnet-blast")
+
+set(SRCS
+    "${LIBRARY_DIR}/wnb/core/info_helper.cc"
+    "${LIBRARY_DIR}/wnb/core/load_wordnet.cc"
+    "${LIBRARY_DIR}/wnb/core/wordnet.cc"
+)
+
+add_library(wnb ${SRCS})
+
+target_link_libraries(wnb PRIVATE boost::headers_only boost::graph)
+
+target_include_directories(wnb PUBLIC "${LIBRARY_DIR}")
--- a/contrib/zlib-ng
+++ b/contrib/zlib-ng
@ -1 +1 @@
-Subproject commit db232d30b4c72fd58e6d7eae2d12cebf9c3d90db
+Subproject commit 6a5e93b9007782115f7f7e5235dedc81c4f1facb
--- a/docker/packager/packager
+++ b/docker/packager/packager
@ -151,8 +151,14 @@ def parse_env_variables(build_type, compiler, sanitizer, package_type, image_typ
        cmake_flags.append('-DENABLE_TESTS=1')
        cmake_flags.append('-DUSE_GTEST=1')

+    # "Unbundled" build is not suitable for any production usage.
+    # But it is occasionally used by some developers.
+    # The whole idea of using unknown version of libraries from the OS distribution is deeply flawed.
+    # We wish these developers good luck.
    if unbundled:
-        cmake_flags.append('-DUNBUNDLED=1 -DUSE_INTERNAL_RDKAFKA_LIBRARY=1 -DENABLE_ARROW=0 -DENABLE_AVRO=0 -DENABLE_ORC=0 -DENABLE_PARQUET=0')
+        # We also disable all CPU features except basic x86_64.
+        # It is only slightly related to "unbundled" build, but it is a good place to test if code compiles without these instruction sets.
+        cmake_flags.append('-DUNBUNDLED=1 -DUSE_INTERNAL_RDKAFKA_LIBRARY=1 -DENABLE_ARROW=0 -DENABLE_AVRO=0 -DENABLE_ORC=0 -DENABLE_PARQUET=0 -DENABLE_SSSE3=0 -DENABLE_SSE41=0 -DENABLE_SSE42=0 -DENABLE_PCLMULQDQ=0 -DENABLE_POPCNT=0 -DENABLE_AVX=0 -DENABLE_AVX2=0')

    if split_binary:
        cmake_flags.append('-DUSE_STATIC_LIBRARIES=0 -DSPLIT_SHARED_LIBRARIES=1 -DCLICKHOUSE_SPLIT_BINARY=1')
--- a/docker/packager/unbundled/Dockerfile
+++ b/docker/packager/unbundled/Dockerfile
@ -23,6 +23,7 @@ RUN apt-get update \
        libboost-regex-dev \
        libboost-context-dev \
        libboost-coroutine-dev \
+        libboost-graph-dev \
        zlib1g-dev \
        liblz4-dev \
        libdouble-conversion-dev \
--- a/docker/server/entrypoint.sh
+++ b/docker/server/entrypoint.sh
@ -164,6 +164,10 @@ fi

 # if no args passed to `docker run` or first argument start with `--`, then the user is passing clickhouse-server arguments
 if [[ $# -lt 1 ]] || [[ "$1" == "--"* ]]; then
+    # Watchdog is launched by default, but does not send SIGINT to the main process,
+    # so the container can't be finished by ctrl+c
+    CLICKHOUSE_WATCHDOG_ENABLE=${CLICKHOUSE_WATCHDOG_ENABLE:-0}
+    export CLICKHOUSE_WATCHDOG_ENABLE
    exec $gosu /usr/bin/clickhouse-server --config-file="$CLICKHOUSE_CONFIG" "$@"
 fi

--- a/docker/test/base/Dockerfile
+++ b/docker/test/base/Dockerfile
@ -61,4 +61,7 @@ ENV TSAN_OPTIONS='halt_on_error=1 history_size=7'
 ENV UBSAN_OPTIONS='print_stacktrace=1'
 ENV MSAN_OPTIONS='abort_on_error=1 poison_in_dtor=1'

+ENV TZ=Europe/Moscow
+RUN ln -snf "/usr/share/zoneinfo/$TZ" /etc/localtime && echo "$TZ" > /etc/timezone
+
 CMD sleep 1
--- a/docker/test/fasttest/run.sh
+++ b/docker/test/fasttest/run.sh
@ -279,6 +279,7 @@ function run_tests
        00926_multimatch
        00929_multi_match_edit_distance
        01681_hyperscan_debug_assertion
+        02004_max_hyperscan_regex_length

        01176_mysql_client_interactive          # requires mysql client
        01031_mutations_interpreter_and_context
@ -299,6 +300,7 @@ function run_tests
        01318_decrypt                           # Depends on OpenSSL
        01663_aes_msan                          # Depends on OpenSSL
        01667_aes_args_check                    # Depends on OpenSSL
+        01683_codec_encrypted                   # Depends on OpenSSL
        01776_decrypt_aead_size_check           # Depends on OpenSSL
        01811_filter_by_null                    # Depends on OpenSSL
        01281_unsucceeded_insert_select_queries_counter
@ -310,6 +312,8 @@ function run_tests
        01411_bayesian_ab_testing
        01798_uniq_theta_sketch
        01799_long_uniq_theta_sketch
+        01890_stem                               # depends on libstemmer_c
+        02003_compress_bz2                       # depends on bzip2
        collate
        collation
        _orc_
--- a/docker/test/fuzzer/run-fuzzer.sh
+++ b/docker/test/fuzzer/run-fuzzer.sh
@ -194,6 +194,10 @@ continue
    jobs
    pstree -aspgT

+    server_exit_code=0
+    wait $server_pid || server_exit_code=$?
+    echo "Server exit code is $server_exit_code"
+
    # Make files with status and description we'll show for this check on Github.
    task_exit_code=$fuzzer_exit_code
    if [ "$server_died" == 1 ]
@ -222,7 +226,7 @@ continue
        task_exit_code=$fuzzer_exit_code
        echo "failure" > status.txt
        { grep --text -o "Found error:.*" fuzzer.log \
-            || grep --text -o "Exception.*" fuzzer.log \
+            || grep --text -ao "Exception:.*" fuzzer.log \
            || echo "Fuzzer failed ($fuzzer_exit_code). See the logs." ; } \
            | tail -1 > description.txt
    fi
--- a/docker/test/integration/runner/compose/docker_compose_jdbc_bridge.yml
+++ b/docker/test/integration/runner/compose/docker_compose_jdbc_bridge.yml
@ -14,10 +14,14 @@ services:
      }
      EOF
      ./docker-entrypoint.sh'
-    ports:
-      - 9020:9019
+    expose:
+      - 9019
    healthcheck:
      test: ["CMD", "curl", "-s", "localhost:9019/ping"]
      interval: 5s
      timeout: 3s
      retries: 30
+    volumes:
+      - type: ${JDBC_BRIDGE_FS:-tmpfs}
+        source: ${JDBC_BRIDGE_LOGS:-}
+        target: /app/logs
--- a/docker/test/integration/runner/compose/docker_compose_mongo_secure.yml
+++ b/docker/test/integration/runner/compose/docker_compose_mongo_secure.yml
@ -0,0 +1,13 @@
+version: '2.3'
+services:
+    mongo1:
+        image: mongo:3.6
+        restart: always
+        environment:
+            MONGO_INITDB_ROOT_USERNAME: root
+            MONGO_INITDB_ROOT_PASSWORD: clickhouse
+        volumes:
+            - ${MONGO_CONFIG_PATH}:/mongo/
+        ports:
+            - ${MONGO_EXTERNAL_PORT}:${MONGO_INTERNAL_PORT}
+        command: --config /mongo/mongo_secure.conf --profile=2 --verbose
--- a/docker/test/integration/runner/compose/docker_compose_mysql_5_7_for_materialized_mysql.yml
+++ b/docker/test/integration/runner/compose/docker_compose_mysql_5_7_for_materialized_mysql.yml
--- a/docker/test/integration/runner/compose/docker_compose_rabbitmq.yml
+++ b/docker/test/integration/runner/compose/docker_compose_rabbitmq.yml
@ -2,7 +2,7 @@ version: '2.3'

 services:
    rabbitmq1:
-        image: rabbitmq:3-management-alpine
+        image: rabbitmq:3.8-management-alpine
        hostname: rabbitmq1
        expose:
            - ${RABBITMQ_PORT}
--- a/docker/test/performance-comparison/compare.sh
+++ b/docker/test/performance-comparison/compare.sh
@ -628,6 +628,9 @@ cat analyze/errors.log >> report/errors.log ||:
 cat profile-errors.log >> report/errors.log ||:

 clickhouse-local --query "
+-- We use decimals specifically to get fixed-point, fixed-width formatting.
+set output_format_decimal_trailing_zeros = 1;
+
 create view query_display_names as select * from
    file('analyze/query-display-names.tsv', TSV,
        'test text, query_index int, query_display_name text')
@ -975,6 +978,9 @@ for version in {right,left}
 do
    rm -rf data
    clickhouse-local --query "
+-- We use decimals specifically to get fixed-point, fixed-width formatting.
+set output_format_decimal_trailing_zeros = 1;
+
 create view query_profiles as
    with 0 as left, 1 as right
    select * from file('analyze/query-profiles.tsv', TSV,
@ -1170,6 +1176,9 @@ rm -rf metrics ||:
 mkdir metrics

 clickhouse-local --query "
+-- We use decimals specifically to get fixed-point, fixed-width formatting.
+set output_format_decimal_trailing_zeros = 1;
+
 create view right_async_metric_log as
    select * from file('right-async-metric-log.tsv', TSVWithNamesAndTypes,
        '$(cat right-async-metric-log.tsv.columns)')
@ -1196,7 +1205,7 @@ create table changes engine File(TSV, 'metrics/changes.tsv') as
            if(left > right, left / right, right / left) times_diff
        from metrics
        group by metric
-        having abs(diff) > 0.05 and isFinite(diff)
+        having abs(diff) > 0.05 and isFinite(diff) and isFinite(times_diff)
    )
    order by diff desc
    ;
--- a/docker/test/performance-comparison/perf.py
+++ b/docker/test/performance-comparison/perf.py
@ -183,6 +183,10 @@ for conn_index, c in enumerate(all_connections):
        # requires clickhouse-driver >= 1.1.5 to accept arbitrary new settings
        # (https://github.com/mymarilyn/clickhouse-driver/pull/142)
        c.settings[s.tag] = s.text
+    # We have to perform a query to make sure the settings work. Otherwise an
+    # unknown setting will lead to failing precondition check, and we will skip
+    # the test, which is wrong.
+    c.execute("select 1")

 reportStageEnd('settings')

--- a/docker/test/pvs/Dockerfile
+++ b/docker/test/pvs/Dockerfile
@ -28,7 +28,7 @@ RUN apt-get update --yes \
 ENV PKG_VERSION="pvs-studio-latest"

 RUN set -x \
-    && export PUBKEY_HASHSUM="486a0694c7f92e96190bbfac01c3b5ac2cb7823981db510a28f744c99eabbbf17a7bcee53ca42dc6d84d4323c2742761" \
+    && export PUBKEY_HASHSUM="686e5eb8b3c543a5c54442c39ec876b6c2d912fe8a729099e600017ae53c877dda3368fe38ed7a66024fe26df6b5892a" \
    && wget -nv https://files.viva64.com/etc/pubkey.txt -O /tmp/pubkey.txt \
    && echo "${PUBKEY_HASHSUM} /tmp/pubkey.txt" | sha384sum -c \
    && apt-key add /tmp/pubkey.txt \
--- a/docker/test/stateful/run.sh
+++ b/docker/test/stateful/run.sh
@ -2,6 +2,11 @@

 set -e -x

+# Choose random timezone for this test run
+TZ="$(grep -v '#' /usr/share/zoneinfo/zone.tab  | awk '{print $3}' | shuf | head -n1)"
+echo "Choosen random timezone $TZ"
+ln -snf "/usr/share/zoneinfo/$TZ" /etc/localtime && echo "$TZ" > /etc/timezone
+
 dpkg -i package_folder/clickhouse-common-static_*.deb;
 dpkg -i package_folder/clickhouse-common-static-dbg_*.deb
 dpkg -i package_folder/clickhouse-server_*.deb
--- a/docker/test/stateless/run.sh
+++ b/docker/test/stateless/run.sh
@ -3,6 +3,11 @@
 # fail on errors, verbose and export all env variables
 set -e -x -a

+# Choose random timezone for this test run.
+TZ="$(grep -v '#' /usr/share/zoneinfo/zone.tab  | awk '{print $3}' | shuf | head -n1)"
+echo "Choosen random timezone $TZ"
+ln -snf "/usr/share/zoneinfo/$TZ" /etc/localtime && echo "$TZ" > /etc/timezone
+
 dpkg -i package_folder/clickhouse-common-static_*.deb
 dpkg -i package_folder/clickhouse-common-static-dbg_*.deb
 dpkg -i package_folder/clickhouse-server_*.deb
@ -138,15 +143,18 @@ if [[ -n "$WITH_COVERAGE" ]] && [[ "$WITH_COVERAGE" -eq 1 ]]; then
 fi
 tar -chf /test_output/text_log_dump.tar /var/lib/clickhouse/data/system/text_log ||:
 tar -chf /test_output/query_log_dump.tar /var/lib/clickhouse/data/system/query_log ||:
+tar -chf /test_output/zookeeper_log_dump.tar /var/lib/clickhouse/data/system/zookeeper_log ||:
 tar -chf /test_output/coordination.tar /var/lib/clickhouse/coordination ||:

 if [[ -n "$USE_DATABASE_REPLICATED" ]] && [[ "$USE_DATABASE_REPLICATED" -eq 1 ]]; then
-  grep -Fa "Fatal" /var/log/clickhouse-server/clickhouse-server1.log ||:
-  grep -Fa "Fatal" /var/log/clickhouse-server/clickhouse-server2.log ||:
+    grep -Fa "Fatal" /var/log/clickhouse-server/clickhouse-server1.log ||:
+    grep -Fa "Fatal" /var/log/clickhouse-server/clickhouse-server2.log ||:
    pigz < /var/log/clickhouse-server/clickhouse-server1.log > /test_output/clickhouse-server1.log.gz ||:
    pigz < /var/log/clickhouse-server/clickhouse-server2.log > /test_output/clickhouse-server2.log.gz ||:
    mv /var/log/clickhouse-server/stderr1.log /test_output/ ||:
    mv /var/log/clickhouse-server/stderr2.log /test_output/ ||:
+    tar -chf /test_output/zookeeper_log_dump1.tar /var/lib/clickhouse1/data/system/zookeeper_log ||:
+    tar -chf /test_output/zookeeper_log_dump2.tar /var/lib/clickhouse2/data/system/zookeeper_log ||:
    tar -chf /test_output/coordination1.tar /var/lib/clickhouse1/coordination ||:
    tar -chf /test_output/coordination2.tar /var/lib/clickhouse2/coordination ||:
 fi
--- a/docker/test/stateless_unbundled/Dockerfile
+++ b/docker/test/stateless_unbundled/Dockerfile
@ -77,9 +77,6 @@ RUN mkdir -p /tmp/clickhouse-odbc-tmp \
   && odbcinst -i -s -l -f /tmp/clickhouse-odbc-tmp/share/doc/clickhouse-odbc/config/odbc.ini.sample \
   && rm -rf /tmp/clickhouse-odbc-tmp

-ENV TZ=Europe/Moscow
-RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
-
 COPY run.sh /
 CMD ["/bin/bash", "/run.sh"]

--- a/docker/test/stress/stress
+++ b/docker/test/stress/stress
@ -20,6 +20,7 @@ def get_skip_list_cmd(path):

 def get_options(i):
    options = []
+    client_options = []
    if 0 < i:
        options.append("--order=random")

@ -27,25 +28,29 @@ def get_options(i):
        options.append("--db-engine=Ordinary")

    if i % 3 == 2:
-        options.append('''--client-option='allow_experimental_database_replicated=1' --db-engine="Replicated('/test/db/test_{}', 's1', 'r1')"'''.format(i))
+        options.append('''--db-engine="Replicated('/test/db/test_{}', 's1', 'r1')"'''.format(i))
+        client_options.append('allow_experimental_database_replicated=1')

    # If database name is not specified, new database is created for each functional test.
    # Run some threads with one database for all tests.
    if i % 2 == 1:
        options.append(" --database=test_{}".format(i))

-    if i % 7 == 0:
-        options.append(" --client-option='join_use_nulls=1'")
+    if i % 5 == 1:
+        client_options.append("join_use_nulls=1")

-    if i % 14 == 0:
-        options.append(' --client-option="join_algorithm=\'partial_merge\'"')
+    if i % 15 == 6:
+        client_options.append("join_algorithm='partial_merge'")

-    if i % 21 == 0:
-        options.append(' --client-option="join_algorithm=\'auto\'"')
-        options.append(' --client-option="max_rows_in_join=1000"')
+    if i % 15 == 11:
+        client_options.append("join_algorithm='auto'")
+        client_options.append('max_rows_in_join=1000')

    if i == 13:
-        options.append(" --client-option='memory_tracker_fault_probability=0.00001'")
+        client_options.append('memory_tracker_fault_probability=0.001')
+
+    if client_options:
+        options.append(" --client-option " + ' '.join(client_options))

    return ' '.join(options)

--- a/docker/test/testflows/runner/Dockerfile
+++ b/docker/test/testflows/runner/Dockerfile
@ -35,7 +35,7 @@ RUN apt-get update \
 ENV TZ=Europe/Moscow
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

-RUN pip3 install urllib3 testflows==1.6.90 docker-compose==1.29.1 docker==5.0.0 dicttoxml kazoo tzlocal python-dateutil numpy
+RUN pip3 install urllib3 testflows==1.7.20 docker-compose==1.29.1 docker==5.0.0 dicttoxml kazoo tzlocal python-dateutil numpy

 ENV DOCKER_CHANNEL stable
 ENV DOCKER_VERSION 20.10.6
--- a/docker/test/unit/Dockerfile
+++ b/docker/test/unit/Dockerfile
@ -1,8 +1,6 @@
 # docker build -t yandex/clickhouse-unit-test .
 FROM yandex/clickhouse-stateless-test

-ENV TZ=Europe/Moscow
-RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 RUN apt-get install gdb

 COPY run.sh /
--- a/docs/README.md
+++ b/docs/README.md
@ -9,7 +9,7 @@ Many developers can say that the code is the best docs by itself, and they are r
 If you want to help ClickHouse with documentation you can face, for example, the following questions:

 - "I don't know how to write."
-    
+
    We have prepared some [recommendations](#what-to-write) for you.

 - "I know what I want to write, but I don't know how to contribute to docs."
@ -71,17 +71,17 @@ Contribute all new information in English language. Other languages are translat
    ```

 - Bold text: `**asterisks**` or `__underlines__`.
- Links: `[link text](uri)`. Examples: 
+- Links: `[link text](uri)`. Examples:

    - External link: `[ClickHouse repo](https://github.com/ClickHouse/ClickHouse)`
    - Cross link: `[How to build docs](tools/README.md)`

 - Images: `![Exclamation sign](uri)`. You can refer to local images as well as remote in internet.
 - Lists: Lists can be of two types:
-    
+
    - `- unordered`: Each item starts from the `-`.
    - `1. ordered`: Each item starts from the number.
-    
+
    A list must be separated from the text by an empty line. Nested lists must be indented with 4 spaces.

 - Inline code: `` `in backticks` ``.
@ -107,7 +107,7 @@ Contribute all new information in English language. Other languages are translat
 - Text hidden behind a cut (single sting that opens on click):

    ```text
-    <details markdown="1"> <summary>Visible text</summary> 
+    <details markdown="1"> <summary>Visible text</summary>
        Hidden content.
    </details>`.
    ```
--- a/docs/_description_templates/template-data-type.md
+++ b/docs/_description_templates/template-data-type.md
@ -1,6 +1,6 @@
 ---
-toc_priority: 
-toc_title: 
+toc_priority:
+toc_title:
 ---

 # data_type_name {#data_type-name}
--- a/docs/_description_templates/template-engine.md
+++ b/docs/_description_templates/template-engine.md
@ -58,6 +58,6 @@ Result:

 Follow up with any text to clarify the example.

-**See Also** 
+**See Also**

 -   [link](#)
--- a/docs/_description_templates/template-function.md
+++ b/docs/_description_templates/template-function.md
@ -14,8 +14,8 @@ More text (Optional).

 **Arguments** (Optional)

-   `x` — Description. Optional (only for optional arguments). Possible values: <values list>. Default value: <value>. [Type name](relative/path/to/type/dscr.md#type). 
-   `y` — Description. Optional (only for optional arguments). Possible values: <values list>.Default value: <value>. [Type name](relative/path/to/type/dscr.md#type). 
+-   `x` — Description. Optional (only for optional arguments). Possible values: <values list>. Default value: <value>. [Type name](relative/path/to/type/dscr.md#type).
+-   `y` — Description. Optional (only for optional arguments). Possible values: <values list>.Default value: <value>. [Type name](relative/path/to/type/dscr.md#type).

 **Parameters** (Optional, only for parametric aggregate functions)

@ -23,7 +23,7 @@ More text (Optional).

 **Returned value(s)**

-   Returned values list. 
+-   Returned values list.

 Type: [Type name](relative/path/to/type/dscr.md#type).

--- a/docs/_includes/cmake_in_clickhouse_footer.md
+++ b/docs/_includes/cmake_in_clickhouse_footer.md
@ -16,8 +16,8 @@ Better:
 option(ENABLE_TESTS "Provide unit_test_dbms target with Google.test unit tests" OFF)
 ```

-If the option's purpose can't be guessed by its name, or the purpose guess may be misleading, or option has some 
-pre-conditions, leave a comment above the `option()` line and explain what it does. 
+If the option's purpose can't be guessed by its name, or the purpose guess may be misleading, or option has some
+pre-conditions, leave a comment above the `option()` line and explain what it does.
 The best way would be linking the docs page (if it exists).
 The comment is parsed into a separate column (see below).

@ -33,7 +33,7 @@ option(ENABLE_TESTS "Provide unit_test_dbms target with Google.test unit tests"

 Suppose you have an option that may strip debug symbols from the ClickHouse's part.
 This can speed up the linking process, but produces a binary that cannot be debugged.
-In that case, prefer explicitly raising a warning telling the developer that he may be doing something wrong. 
+In that case, prefer explicitly raising a warning telling the developer that he may be doing something wrong.
 Also, such options should be disabled if applies.

 Bad:
--- a/docs/en/commercial/support.md
+++ b/docs/en/commercial/support.md
@ -7,7 +7,7 @@ toc_title: Support

 !!! info "Info"
    If you have launched a ClickHouse commercial support service, feel free to [open a pull-request](https://github.com/ClickHouse/ClickHouse/edit/master/docs/en/commercial/support.md) adding it to the following list.
-    
+
 ## Yandex.Cloud

 ClickHouse worldwide support from the authors of ClickHouse. Supports on-premise and cloud deployments. Ask details on clickhouse-support@yandex-team.com
--- a/docs/en/development/adding_test_queries.md
+++ b/docs/en/development/adding_test_queries.md
@ -4,11 +4,11 @@ ClickHouse has hundreds (or even thousands) of features. Every commit gets check

 The core functionality is very well tested, but some corner-cases and different combinations of features can be uncovered with ClickHouse CI.

-Most of the bugs/regressions we see happen in that 'grey area' where test coverage is poor. 
+Most of the bugs/regressions we see happen in that 'grey area' where test coverage is poor.

-And we are very interested in covering most of the possible scenarios and feature combinations used in real life by tests. 
+And we are very interested in covering most of the possible scenarios and feature combinations used in real life by tests.

-## Why adding tests 
+## Why adding tests

 Why/when you should add a test case into ClickHouse code:
 1) you use some complicated scenarios / feature combinations / you have some corner case which is probably not widely used
@ -17,18 +17,18 @@ Why/when you should add a test case into ClickHouse code:
 4) once the test is added/accepted, you can be sure the corner case you check will never be accidentally broken.
 5) you will be a part of great open-source community
 6) your name will be visible in the `system.contributors` table!
-7) you will make a world bit better :) 
+7) you will make a world bit better :)

 ### Steps to do

-#### Prerequisite 
+#### Prerequisite

-I assume you run some Linux machine (you can use docker / virtual machines on other OS) and any modern browser / internet connection, and you have some basic Linux & SQL skills. 
+I assume you run some Linux machine (you can use docker / virtual machines on other OS) and any modern browser / internet connection, and you have some basic Linux & SQL skills.

 Any highly specialized knowledge is not needed (so you don't need to know C++ or know something about how ClickHouse CI works).


-#### Preparation 
+#### Preparation

 1) [create GitHub account](https://github.com/join) (if you haven't one yet)
 2) [setup git](https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/set-up-git)
@ -54,17 +54,17 @@ git remote add upstream https://github.com/ClickHouse/ClickHouse

 #### New branch for the test

-1) create a new branch from the latest clickhouse master 
+1) create a new branch from the latest clickhouse master
 ```
 cd ~/workspace/ClickHouse
 git fetch upstream
-git checkout -b name_for_a_branch_with_my_test upstream/master 
+git checkout -b name_for_a_branch_with_my_test upstream/master
 ```

-#### Install & run clickhouse 
+#### Install & run clickhouse

 1) install `clickhouse-server` (follow [official docs](https://clickhouse.tech/docs/en/getting-started/install/))
-2) install test configurations (it will use Zookeeper mock implementation and adjust some settings) 
+2) install test configurations (it will use Zookeeper mock implementation and adjust some settings)
 ```
 cd ~/workspace/ClickHouse/tests/config
 sudo ./install.sh
@ -74,7 +74,7 @@ sudo ./install.sh
 sudo systemctl restart clickhouse-server
 ```

-#### Creating the test file 
+#### Creating the test file


 1) find the number for your test - find the file with the biggest number in `tests/queries/0_stateless/`
@ -86,7 +86,7 @@ tests/queries/0_stateless/01520_client_print_query_id.reference
 ```
 Currently, the last number for the test is `01520`, so my test will have the number `01521`

-2) create an SQL file with the next number and name of the feature you test 
+2) create an SQL file with the next number and name of the feature you test

 ```sh
 touch tests/queries/0_stateless/01521_dummy_test.sql
@ -112,16 +112,16 @@ clickhouse-client -nmT < tests/queries/0_stateless/01521_dummy_test.sql | tee te
 	- fast - should not take longer than a few seconds (better subseconds)
 	- correct - fails then feature is not working
        - deterministic
-	- isolated / stateless 
+	- isolated / stateless
 		- don't rely on some environment things
-		- don't rely on timing when possible 
- try to cover corner cases (zeros / Nulls / empty sets / throwing exceptions) 
+		- don't rely on timing when possible
+- try to cover corner cases (zeros / Nulls / empty sets / throwing exceptions)
 - to test that query return errors, you can put special comment after the query: `-- { serverError 60 }` or `-- { clientError 20 }`
 - don't switch databases (unless necessary)
 - you can create several table replicas on the same node if needed
 - you can use one of the test cluster definitions when needed (see system.clusters)
 - use `number` / `numbers_mt` / `zeros` / `zeros_mt` and similar for queries / to initialize data when applicable
- clean up the created objects after test and before the test (DROP IF EXISTS) - in case of some dirty state 
+- clean up the created objects after test and before the test (DROP IF EXISTS) - in case of some dirty state
 - prefer sync mode of operations (mutations, merges, etc.)
 - use other SQL files in the `0_stateless` folder as an example
 - ensure the feature / feature combination you want to test is not yet covered with existing tests
@ -138,7 +138,7 @@ It's important to name tests correctly, so one could turn some tests subset off

 #### Commit / push / create PR.

-1) commit & push your changes 
+1) commit & push your changes
 ```sh
 cd ~/workspace/ClickHouse
 git add tests/queries/0_stateless/01521_dummy_test.sql
@ -147,5 +147,5 @@ git commit # use some nice commit message when possible
 git push origin HEAD
 ```
 2) use a link which was shown during the push, to create a PR into the main repo
-3) adjust the PR title and contents, in `Changelog category (leave one)` keep 
-`Build/Testing/Packaging Improvement`, fill the rest of the fields if you want. 
+3) adjust the PR title and contents, in `Changelog category (leave one)` keep
+`Build/Testing/Packaging Improvement`, fill the rest of the fields if you want.
--- a/docs/en/development/build.md
+++ b/docs/en/development/build.md
@ -155,6 +155,10 @@ Normally ClickHouse is statically linked into a single static `clickhouse` binar
 -DUSE_STATIC_LIBRARIES=0 -DSPLIT_SHARED_LIBRARIES=1 -DCLICKHOUSE_SPLIT_BINARY=1
 ```

-Note that in this configuration there is no single `clickhouse` binary, and you have to run `clickhouse-server`, `clickhouse-client` etc.
+Note that the split build has several drawbacks:
+* There is no single `clickhouse` binary, and you have to run `clickhouse-server`, `clickhouse-client`, etc.
+* Risk of segfault if you run any of the programs while rebuilding the project.
+* You cannot run the integration tests since they only work a single complete binary.
+* You can't easily copy the binaries elsewhere. Instead of moving a single binary you'll need to copy all binaries and libraries.

 [Original article](https://clickhouse.tech/docs/en/development/build/) <!--hide-->
--- a/docs/en/development/contrib.md
+++ b/docs/en/development/contrib.md
@ -8,7 +8,7 @@ toc_title: Third-Party Libraries Used
 The list of third-party libraries can be obtained by the following query:

 ``` sql
-SELECT library_name, license_type, license_path FROM system.licenses ORDER BY library_name COLLATE 'en'
+SELECT library_name, license_type, license_path FROM system.licenses ORDER BY library_name COLLATE 'en';
 ```

 [Example](https://gh-api.clickhouse.tech/play?user=play#U0VMRUNUIGxpYnJhcnlfbmFtZSwgbGljZW5zZV90eXBlLCBsaWNlbnNlX3BhdGggRlJPTSBzeXN0ZW0ubGljZW5zZXMgT1JERVIgQlkgbGlicmFyeV9uYW1lIENPTExBVEUgJ2VuJw==)
--- a/docs/en/development/style.md
+++ b/docs/en/development/style.md
@ -749,7 +749,7 @@ If your code in the `master` branch is not buildable yet, exclude it from the bu

 **1.** The C++20 standard library is used (experimental extensions are allowed), as well as `boost` and `Poco` frameworks.

-**2.** It is not allowed to use libraries from OS packages. It is also not allowed to use pre-installed libraries. All libraries should be placed in form of source code in `contrib` directory and built with ClickHouse.
+**2.** It is not allowed to use libraries from OS packages. It is also not allowed to use pre-installed libraries. All libraries should be placed in form of source code in `contrib` directory and built with ClickHouse. See [Guidelines for adding new third-party libraries](contrib.md#adding-third-party-libraries) for details.

 **3.** Preference is always given to libraries that are already in use.

--- a/docs/en/development/tests.md
+++ b/docs/en/development/tests.md
@ -70,7 +70,13 @@ Note that integration of ClickHouse with third-party drivers is not tested. Also

 Unit tests are useful when you want to test not the ClickHouse as a whole, but a single isolated library or class. You can enable or disable build of tests with `ENABLE_TESTS` CMake option. Unit tests (and other test programs) are located in `tests` subdirectories across the code. To run unit tests, type `ninja test`. Some tests use `gtest`, but some are just programs that return non-zero exit code on test failure.

-It’s not necessarily to have unit tests if the code is already covered by functional tests (and functional tests are usually much more simple to use).
+It’s not necessary to have unit tests if the code is already covered by functional tests (and functional tests are usually much more simple to use).
+
+You can run individual gtest checks by calling the executable directly, for example:
+
+```bash
+$ ./src/unit_tests_dbms --gtest_filter=LocalAddress*
+```

 ## Performance Tests {#performance-tests}

--- a/docs/en/engines/database-engines/atomic.md
+++ b/docs/en/engines/database-engines/atomic.md
@ -5,19 +5,19 @@ toc_title: Atomic

 # Atomic {#atomic}

-It supports non-blocking [DROP TABLE](#drop-detach-table) and [RENAME TABLE](#rename-table) queries and atomic [EXCHANGE TABLES t1 AND t2](#exchange-tables) queries. `Atomic` database engine is used by default.
+It supports non-blocking [DROP TABLE](#drop-detach-table) and [RENAME TABLE](#rename-table) queries and atomic [EXCHANGE TABLES](#exchange-tables) queries. `Atomic` database engine is used by default.

 ## Creating a Database {#creating-a-database}

 ``` sql
-    CREATE DATABASE test[ ENGINE = Atomic];
+CREATE DATABASE test [ENGINE = Atomic];
 ```

 ## Specifics and recommendations {#specifics-and-recommendations}

 ### Table UUID {#table-uuid}

-All tables in database `Atomic` have persistent [UUID](../../sql-reference/data-types/uuid.md) and store data in directory `/clickhouse_path/store/xxx/xxxyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy/`, where `xxxyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy` is UUID of the table. 
+All tables in database `Atomic` have persistent [UUID](../../sql-reference/data-types/uuid.md) and store data in directory `/clickhouse_path/store/xxx/xxxyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy/`, where `xxxyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy` is UUID of the table.
 Usually, the UUID is generated automatically, but the user can also explicitly specify the UUID in the same way when creating the table (this is not recommended). To display the `SHOW CREATE` query with the UUID you can use setting [show_table_uuid_in_table_create_query_if_not_nil](../../operations/settings/settings.md#show_table_uuid_in_table_create_query_if_not_nil). For example:

 ```sql
@ -25,16 +25,16 @@ CREATE TABLE name UUID '28f1c61c-2970-457a-bffe-454156ddcfef' (n UInt64) ENGINE
 ```
 ### RENAME TABLE {#rename-table}

-`RENAME` queries are performed without changing UUID and moving table data. These queries do not wait for the completion of queries using the table and will be executed instantly.
+[RENAME](../../sql-reference/statements/rename.md) queries are performed without changing UUID and moving table data. These queries do not wait for the completion of queries using the table and are executed instantly.

 ### DROP/DETACH TABLE {#drop-detach-table}

-On `DROP TABLE` no data is removed, database `Atomic` just marks table as dropped by moving metadata to `/clickhouse_path/metadata_dropped/` and notifies background thread. Delay before final table data deletion is specify by [database_atomic_delay_before_drop_table_sec](../../operations/server-configuration-parameters/settings.md#database_atomic_delay_before_drop_table_sec) setting.
+On `DROP TABLE` no data is removed, database `Atomic` just marks table as dropped by moving metadata to `/clickhouse_path/metadata_dropped/` and notifies background thread. Delay before final table data deletion is specified by the [database_atomic_delay_before_drop_table_sec](../../operations/server-configuration-parameters/settings.md#database_atomic_delay_before_drop_table_sec) setting.
 You can specify synchronous mode using `SYNC` modifier. Use the [database_atomic_wait_for_drop_and_detach_synchronously](../../operations/settings/settings.md#database_atomic_wait_for_drop_and_detach_synchronously) setting to do this. In this case `DROP` waits for running `SELECT`, `INSERT` and other queries which are using the table to finish. Table will be actually removed when it's not in use.

-### EXCHANGE TABLES {#exchange-tables}
+### EXCHANGE TABLES/DICTIONARIES {#exchange-tables}

-`EXCHANGE` query swaps tables atomically. So instead of this non-atomic operation:
+[EXCHANGE](../../sql-reference/statements/exchange.md) query swaps tables or dictionaries atomically. For instance, instead of this non-atomic operation:

 ```sql
 RENAME TABLE new_table TO tmp, old_table TO new_table, tmp TO old_table;
@ -47,7 +47,7 @@ EXCHANGE TABLES new_table AND old_table;

 ### ReplicatedMergeTree in Atomic Database {#replicatedmergetree-in-atomic-database}

-For [ReplicatedMergeTree](../table-engines/mergetree-family/replication.md#table_engines-replication) tables, it is recommended to not specify engine parameters - path in ZooKeeper and replica name. In this case, configuration parameters will be used [default_replica_path](../../operations/server-configuration-parameters/settings.md#default_replica_path) and [default_replica_name](../../operations/server-configuration-parameters/settings.md#default_replica_name). If you want to specify engine parameters explicitly, it is recommended to use `{uuid}` macros. This is useful so that unique paths are automatically generated for each table in ZooKeeper.
+For [ReplicatedMergeTree](../table-engines/mergetree-family/replication.md#table_engines-replication) tables, it is recommended not to specify engine parameters - path in ZooKeeper and replica name. In this case, configuration parameters [default_replica_path](../../operations/server-configuration-parameters/settings.md#default_replica_path) and [default_replica_name](../../operations/server-configuration-parameters/settings.md#default_replica_name) will be used. If you want to specify engine parameters explicitly, it is recommended to use `{uuid}` macros. This is useful so that unique paths are automatically generated for each table in ZooKeeper.

 ## See Also

--- a/docs/en/engines/database-engines/index.md
+++ b/docs/en/engines/database-engines/index.md
@ -8,13 +8,13 @@ toc_title: Introduction

 Database engines allow you to work with tables.

-By default, ClickHouse uses database engine [Atomic](../../engines/database-engines/atomic.md). It is provides configurable [table engines](../../engines/table-engines/index.md) and an [SQL dialect](../../sql-reference/syntax.md).
+By default, ClickHouse uses database engine [Atomic](../../engines/database-engines/atomic.md). It provides configurable [table engines](../../engines/table-engines/index.md) and an [SQL dialect](../../sql-reference/syntax.md).

 You can also use the following database engines:

 -   [MySQL](../../engines/database-engines/mysql.md)

-   [MaterializeMySQL](../../engines/database-engines/materialize-mysql.md)
+-   [MaterializedMySQL](../../engines/database-engines/materialized-mysql.md)

 -   [Lazy](../../engines/database-engines/lazy.md)

--- a/docs/en/engines/database-engines/materialized-mysql.md
+++ b/docs/en/engines/database-engines/materialized-mysql.md
@ -1,21 +1,22 @@
 ---
 toc_priority: 29
-toc_title: MaterializeMySQL
+toc_title: "[experimental] MaterializedMySQL"
 ---

-# MaterializeMySQL {#materialize-mysql}
+# [experimental] MaterializedMySQL {#materialized-mysql}
+
+!!! warning "Warning"
+    This is an experimental feature that should not be used in production.

 Creates ClickHouse database with all the tables existing in MySQL, and all the data in those tables.

 ClickHouse server works as MySQL replica. It reads binlog and performs DDL and DML queries.

-This feature is experimental.
-
 ## Creating a Database {#creating-a-database}

 ``` sql
 CREATE DATABASE [IF NOT EXISTS] db_name [ON CLUSTER cluster]
-ENGINE = MaterializeMySQL('host:port', ['database' | database], 'user', 'password') [SETTINGS ...]
+ENGINE = MaterializedMySQL('host:port', ['database' | database], 'user', 'password') [SETTINGS ...]
 ```

 **Engine Parameters**
@ -25,13 +26,40 @@ ENGINE = MaterializeMySQL('host:port', ['database' | database], 'user', 'passwor
 -   `user` — MySQL user.
 -   `password` — User password.

-## Virtual columns {#virtual-columns}
+**Engine Settings**
+
+-   `max_rows_in_buffer` — Maximum number of rows that data is allowed to cache in memory (for single table and the cache data unable to query). When this number is exceeded, the data will be materialized. Default: `65 505`.
+-   `max_bytes_in_buffer` —  Maximum number of bytes that data is allowed to cache in memory (for single table and the cache data unable to query). When this number is exceeded, the data will be materialized. Default: `1 048 576`.
+-   `max_rows_in_buffers` — Maximum number of rows that data is allowed to cache in memory (for database and the cache data unable to query). When this number is exceeded, the data will be materialized. Default: `65 505`.
+-   `max_bytes_in_buffers` — Maximum number of bytes that data is allowed to cache in memory (for database and the cache data unable to query). When this number is exceeded, the data will be materialized. Default: `1 048 576`.
+-   `max_flush_data_time` — Maximum number of milliseconds that data is allowed to cache in memory (for database and the cache data unable to query). When this time is exceeded, the data will be materialized. Default: `1000`.
+-   `max_wait_time_when_mysql_unavailable` — Retry interval when MySQL is not available (milliseconds). Negative value disables retry. Default: `1000`.
+-   `allows_query_when_mysql_lost` — Allows to query a materialized table when MySQL is lost. Default: `0` (`false`).
+
+```sql
+CREATE DATABASE mysql ENGINE = MaterializedMySQL('localhost:3306', 'db', 'user', '***')
+     SETTINGS
+        allows_query_when_mysql_lost=true,
+        max_wait_time_when_mysql_unavailable=10000;
+```
+
+**Settings on MySQL-server Side**
+
+For the correct work of `MaterializedMySQL`, there are few mandatory `MySQL`-side configuration settings that must be set:
+
+- `default_authentication_plugin = mysql_native_password` since `MaterializedMySQL` can only authorize with this method.
+- `gtid_mode = on` since GTID based logging is a mandatory for providing correct `MaterializedMySQL` replication.
+
+!!! attention "Attention"
+    While turning on `gtid_mode` you should also specify `enforce_gtid_consistency = on`.
+
+## Virtual Columns {#virtual-columns}
+
+When working with the `MaterializedMySQL` database engine, [ReplacingMergeTree](../../engines/table-engines/mergetree-family/replacingmergetree.md) tables are used with virtual `_sign` and `_version` columns.

-When working with the `MaterializeMySQL` database engine, [ReplacingMergeTree](../../engines/table-engines/mergetree-family/replacingmergetree.md) tables are used with virtual `_sign` and `_version` columns.
- 
 - `_version` — Transaction counter. Type [UInt64](../../sql-reference/data-types/int-uint.md).
 - `_sign` — Deletion mark. Type [Int8](../../sql-reference/data-types/int-uint.md). Possible values:
-    - `1` — Row is not deleted, 
+    - `1` — Row is not deleted,
    - `-1` — Row is deleted.

 ## Data Types Support {#data_types-support}
@ -53,30 +81,39 @@ When working with the `MaterializeMySQL` database engine, [ReplacingMergeTree](.
 | STRING                  | [String](../../sql-reference/data-types/string.md)           |
 | VARCHAR, VAR_STRING     | [String](../../sql-reference/data-types/string.md)           |
 | BLOB                    | [String](../../sql-reference/data-types/string.md)           |
-
-Other types are not supported. If MySQL table contains a column of such type, ClickHouse throws exception "Unhandled data type" and stops replication.
+| BINARY                  | [FixedString](../../sql-reference/data-types/fixedstring.md) |

 [Nullable](../../sql-reference/data-types/nullable.md) is supported.

+Other types are not supported. If MySQL table contains a column of such type, ClickHouse throws exception "Unhandled data type" and stops replication.
+
 ## Specifics and Recommendations {#specifics-and-recommendations}

+### Compatibility Restrictions {#compatibility-restrictions}
+
+Apart of the data types limitations there are few restrictions comparing to `MySQL` databases, that should be resolved before replication will be possible:
+
+- Each table in `MySQL` should contain `PRIMARY KEY`.
+
+- Replication for tables, those are containing rows with `ENUM` field values out of range (specified in `ENUM` signature) will not work.
+
 ### DDL Queries {#ddl-queries}

 MySQL DDL queries are converted into the corresponding ClickHouse DDL queries ([ALTER](../../sql-reference/statements/alter/index.md), [CREATE](../../sql-reference/statements/create/index.md), [DROP](../../sql-reference/statements/drop.md), [RENAME](../../sql-reference/statements/rename.md)). If ClickHouse cannot parse some DDL query, the query is ignored.

 ### Data Replication {#data-replication}

-`MaterializeMySQL` does not support direct `INSERT`, `DELETE` and `UPDATE` queries. However, they are supported in terms of data replication:
+`MaterializedMySQL` does not support direct `INSERT`, `DELETE` and `UPDATE` queries. However, they are supported in terms of data replication:

 - MySQL `INSERT` query is converted into `INSERT` with `_sign=1`.

- MySQL `DELETE` query is converted into `INSERT` with `_sign=-1`. 
+- MySQL `DELETE` query is converted into `INSERT` with `_sign=-1`.

 - MySQL `UPDATE` query is converted into `INSERT` with `_sign=-1` and `INSERT` with `_sign=1`.

-### Selecting from MaterializeMySQL Tables {#select}
+### Selecting from MaterializedMySQL Tables {#select}

-`SELECT` query from `MaterializeMySQL` tables has some specifics:
+`SELECT` query from `MaterializedMySQL` tables has some specifics:

 - If `_version` is not specified in the `SELECT` query, [FINAL](../../sql-reference/statements/select/from.md#select-from-final) modifier is used. So only rows with `MAX(_version)` are selected.

@ -93,10 +130,10 @@ ClickHouse has only one physical order, which is determined by `ORDER BY` clause
 **Notes**

 - Rows with `_sign=-1` are not deleted physically from the tables.
- Cascade `UPDATE/DELETE` queries are not supported by the `MaterializeMySQL` engine.
+- Cascade `UPDATE/DELETE` queries are not supported by the `MaterializedMySQL` engine.
 - Replication can be easily broken.
 - Manual operations on database and tables are forbidden.
- `MaterializeMySQL` is influenced by [optimize_on_insert](../../operations/settings/settings.md#optimize-on-insert) setting. The data is merged in the corresponding table in the `MaterializeMySQL` database when a table in the MySQL server changes.
+- `MaterializedMySQL` is influenced by [optimize_on_insert](../../operations/settings/settings.md#optimize-on-insert) setting. The data is merged in the corresponding table in the `MaterializedMySQL` database when a table in the MySQL server changes.

 ## Examples of Use {#examples-of-use}

@ -113,9 +150,9 @@ mysql> SELECT * FROM test;
 ```

 ```text
-+---+------+------+ 
+---+------+------+
 | a |    b |    c |
-+---+------+------+ 
+---+------+------+
 | 2 |  222 | Wow! |
 +---+------+------+
 ```
@ -125,7 +162,7 @@ Database in ClickHouse, exchanging data with the MySQL server:
 The database and the table created:

 ``` sql
-CREATE DATABASE mysql ENGINE = MaterializeMySQL('localhost:3306', 'db', 'user', '***');
+CREATE DATABASE mysql ENGINE = MaterializedMySQL('localhost:3306', 'db', 'user', '***');
 SHOW TABLES FROM mysql;
 ```

@ -142,9 +179,9 @@ SELECT * FROM mysql.test;
 ```

 ``` text
-┌─a─┬──b─┐ 
-│ 1 │ 11 │ 
-│ 2 │ 22 │ 
+┌─a─┬──b─┐
+│ 1 │ 11 │
+│ 2 │ 22 │
 └───┴────┘
 ```

@ -155,9 +192,9 @@ SELECT * FROM mysql.test;
 ```

 ``` text
-┌─a─┬───b─┬─c────┐ 
-│ 2 │ 222 │ Wow! │ 
+┌─a─┬───b─┬─c────┐
+│ 2 │ 222 │ Wow! │
 └───┴─────┴──────┘
 ```

-[Original article](https://clickhouse.tech/docs/en/engines/database-engines/materialize-mysql/) <!--hide-->
+[Original article](https://clickhouse.tech/docs/en/engines/database-engines/materialized-mysql/) <!--hide-->
--- a/docs/en/engines/database-engines/materialized-postgresql.md
+++ b/docs/en/engines/database-engines/materialized-postgresql.md
@ -3,45 +3,52 @@ toc_priority: 30
 toc_title: MaterializedPostgreSQL
 ---

-# MaterializedPostgreSQL {#materialize-postgresql}
+# [experimental] MaterializedPostgreSQL {#materialize-postgresql}
+
+Creates ClickHouse database with an initial data dump of PostgreSQL database tables and starts replication process, i.e. executes background job to apply new changes as they happen on PostgreSQL database tables in the remote PostgreSQL database.
+
+ClickHouse server works as PostgreSQL replica. It reads WAL and performs DML queries. DDL is not replicated, but can be handled (described below).

 ## Creating a Database {#creating-a-database}

 ``` sql
-CREATE DATABASE test_database
-ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgres_database', 'postgres_user', 'postgres_password'
-
-SELECT * FROM test_database.postgres_table;
+CREATE DATABASE [IF NOT EXISTS] db_name [ON CLUSTER cluster]
+ENGINE = MaterializedPostgreSQL('host:port', ['database' | database], 'user', 'password') [SETTINGS ...]
 ```

+**Engine Parameters**
+
+-   `host:port` — PostgreSQL server endpoint.
+-   `database` — PostgreSQL database name.
+-   `user` — PostgreSQL user.
+-   `password` — User password.

 ## Settings {#settings}

-1. `materialized_postgresql_max_block_size` - Number of rows collected before flushing data into table. Default: `65536`.
+-   [materialized_postgresql_max_block_size](../../operations/settings/settings.md#materialized-postgresql-max-block-size)

-2. `materialized_postgresql_tables_list` - List of tables for MaterializedPostgreSQL database engine. Default: `whole database`.
+-   [materialized_postgresql_tables_list](../../operations/settings/settings.md#materialized-postgresql-tables-list)

-3. `materialized_postgresql_allow_automatic_update` - Allow to reload table in the background, when schema changes are detected. Default: `0` (`false`).
+-   [materialized_postgresql_allow_automatic_update](../../operations/settings/settings.md#materialized-postgresql-allow-automatic-update)

 ``` sql
-CREATE DATABASE test_database
-ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgres_database', 'postgres_user', 'postgres_password'
+CREATE DATABASE database1
+ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgres_database', 'postgres_user', 'postgres_password')
 SETTINGS materialized_postgresql_max_block_size = 65536,
         materialized_postgresql_tables_list = 'table1,table2,table3';

-SELECT * FROM test_database.table1;
+SELECT * FROM database1.table1;
 ```

-
 ## Requirements {#requirements}

- Setting `wal_level`to `logical` and `max_replication_slots` to at least `2` in the postgresql config file.
+1. The [wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html) setting must have a value `logical` and `max_replication_slots` parameter must have a value at least `2` in the PostgreSQL config file.

- Each replicated table must have one of the following **replica identity**:
+2. Each replicated table must have one of the following [replica identity](https://www.postgresql.org/docs/10/sql-altertable.html#SQL-CREATETABLE-REPLICA-IDENTITY):

-1. **default** (primary key)
+-   primary key (by default)

-2. **index**
+-   index

 ``` bash
 postgres# CREATE TABLE postgres_table (a Integer NOT NULL, b Integer, c Integer NOT NULL, d Integer, e Integer NOT NULL);
@ -49,9 +56,8 @@ postgres# CREATE unique INDEX postgres_table_index on postgres_table(a, c, e);
 postgres# ALTER TABLE postgres_table REPLICA IDENTITY USING INDEX postgres_table_index;
 ```

-
-Primary key is always checked first. If it is absent, then index, defined as replica identity index, is checked.
-If index is used as replica identity, there has to be only one such index in a table.
+The primary key is always checked first. If it is absent, then the index, defined as replica identity index, is checked.
+If the index is used as a replica identity, there has to be only one such index in a table.
 You can check what type is used for a specific table with the following command:

 ``` bash
@ -65,7 +71,14 @@ FROM pg_class
 WHERE oid = 'postgres_table'::regclass;
 ```

+!!! warning "Warning"
+    Replication of [**TOAST**](https://www.postgresql.org/docs/9.5/storage-toast.html) values is not supported. The default value for the data type will be used.
+	
+## Example of Use {#example-of-use}

-## Warning {#warning}
+``` sql
+CREATE DATABASE postgresql_db
+ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgres_database', 'postgres_user', 'postgres_password');

-1. **TOAST** values convertion is not supported. Default value for the data type will be used.
+SELECT * FROM postgresql_db.postgres_table;
+```
--- a/docs/en/engines/database-engines/mysql.md
+++ b/docs/en/engines/database-engines/mysql.md
@ -53,7 +53,7 @@ All other MySQL data types are converted into [String](../../sql-reference/data-

 ## Global Variables Support {#global-variables-support}

-For better compatibility you may address global variables in MySQL style, as `@@identifier`. 
+For better compatibility you may address global variables in MySQL style, as `@@identifier`.

 These variables are supported:
 - `version`
--- a/docs/en/engines/database-engines/postgresql.md
+++ b/docs/en/engines/database-engines/postgresql.md
@ -14,8 +14,8 @@ Supports table structure modifications (`ALTER TABLE ... ADD|DROP COLUMN`). If `
 ## Creating a Database {#creating-a-database}

 ``` sql
-CREATE DATABASE test_database 
-ENGINE = PostgreSQL('host:port', 'database', 'user', 'password'[, `use_table_cache`]);
+CREATE DATABASE test_database
+ENGINE = PostgreSQL('host:port', 'database', 'user', 'password'[, `schema`, `use_table_cache`]);
 ```

 **Engine Parameters**
@ -24,6 +24,7 @@ ENGINE = PostgreSQL('host:port', 'database', 'user', 'password'[, `use_table_cac
 -   `database` — Remote database name.
 -   `user` — PostgreSQL user.
 -   `password` — User password.
+-   `schema` — PostgreSQL schema.
 -   `use_table_cache` —  Defines if the database table structure is cached or not. Optional. Default value: `0`.

 ## Data Types Support {#data_types-support}
@ -43,14 +44,14 @@ ENGINE = PostgreSQL('host:port', 'database', 'user', 'password'[, `use_table_cac
 | TEXT, CHAR       | [String](../../sql-reference/data-types/string.md)           |
 | INTEGER          | Nullable([Int32](../../sql-reference/data-types/int-uint.md))|
 | ARRAY            | [Array](../../sql-reference/data-types/array.md)             |
- 
+

 ## Examples of Use {#examples-of-use}

 Database in ClickHouse, exchanging data with the PostgreSQL server:

 ``` sql
-CREATE DATABASE test_database 
+CREATE DATABASE test_database
 ENGINE = PostgreSQL('postgres1:5432', 'test_database', 'postgres', 'mysecretpassword', 1);
 ```

@ -102,7 +103,7 @@ SELECT * FROM test_database.test_table;
 └────────┴───────┘
 ```

-Consider the table structure was modified in PostgreSQL: 
+Consider the table structure was modified in PostgreSQL:

 ``` sql
 postgre> ALTER TABLE test_table ADD COLUMN data Text
--- a/docs/en/engines/database-engines/replicated.md
+++ b/docs/en/engines/database-engines/replicated.md
@ -1,6 +1,11 @@
+---
+toc_priority: 36
+toc_title: Replicated
+---
+
 # [experimental] Replicated {#replicated}

-The engine is based on the [Atomic](../../engines/database-engines/atomic.md) engine. It supports replication of metadata via DDL log being written to ZooKeeper and executed on all of the replicas for a given database. 
+The engine is based on the [Atomic](../../engines/database-engines/atomic.md) engine. It supports replication of metadata via DDL log being written to ZooKeeper and executed on all of the replicas for a given database.

 One ClickHouse server can have multiple replicated databases running and updating at the same time. But there can't be multiple replicas of the same replicated database.

@ -20,9 +25,9 @@ One ClickHouse server can have multiple replicated databases running and updatin

 ## Specifics and Recommendations {#specifics-and-recommendations}

-DDL queries with `Replicated` database work in a similar way to [ON CLUSTER](../../sql-reference/distributed-ddl.md) queries, but with minor differences. 
+DDL queries with `Replicated` database work in a similar way to [ON CLUSTER](../../sql-reference/distributed-ddl.md) queries, but with minor differences.

-First, the DDL request tries to execute on the initiator (the host that originally received the request from the user). If the request is not fulfilled, then the user immediately receives an error, other hosts do not try to fulfill it. If the request has been successfully completed on the initiator, then all other hosts will automatically retry until they complete it. The initiator will try to wait for the query to be completed on other hosts (no longer than [distributed_ddl_task_timeout](../../operations/settings/settings.md#distributed_ddl_task_timeout)) and will return a table with the query execution statuses on each host. 
+First, the DDL request tries to execute on the initiator (the host that originally received the request from the user). If the request is not fulfilled, then the user immediately receives an error, other hosts do not try to fulfill it. If the request has been successfully completed on the initiator, then all other hosts will automatically retry until they complete it. The initiator will try to wait for the query to be completed on other hosts (no longer than [distributed_ddl_task_timeout](../../operations/settings/settings.md#distributed_ddl_task_timeout)) and will return a table with the query execution statuses on each host.

 The behavior in case of errors is regulated by the [distributed_ddl_output_mode](../../operations/settings/settings.md#distributed_ddl_output_mode) setting, for a `Replicated` database it is better to set it to `null_status_on_timeout` — i.e. if some hosts did not have time to execute the request for [distributed_ddl_task_timeout](../../operations/settings/settings.md#distributed_ddl_task_timeout), then do not throw an exception, but show the `NULL` status for them in the table.

@ -47,8 +52,8 @@ CREATE TABLE r.rmt (n UInt64) ENGINE=ReplicatedMergeTree ORDER BY n;
 ```

 ``` text
-┌─────hosts────────────┬──status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐ 
-│ shard1|replica1      │    0    │       │          2          │        0         │ 
+┌─────hosts────────────┬──status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
+│ shard1|replica1      │    0    │       │          2          │        0         │
 │ shard1|other_replica │    0    │       │          1          │        0         │
 │ other_shard|r1       │    0    │       │          0          │        0         │
 └──────────────────────┴─────────┴───────┴─────────────────────┴──────────────────┘
@ -57,13 +62,13 @@ CREATE TABLE r.rmt (n UInt64) ENGINE=ReplicatedMergeTree ORDER BY n;
 Showing the system table:

 ``` sql
-SELECT cluster, shard_num, replica_num, host_name, host_address, port, is_local 
+SELECT cluster, shard_num, replica_num, host_name, host_address, port, is_local
 FROM system.clusters WHERE cluster='r';
 ```

 ``` text
-┌─cluster─┬─shard_num─┬─replica_num─┬─host_name─┬─host_address─┬─port─┬─is_local─┐ 
-│ r       │     1     │      1      │   node3   │  127.0.0.1   │ 9002 │     0    │ 
+┌─cluster─┬─shard_num─┬─replica_num─┬─host_name─┬─host_address─┬─port─┬─is_local─┐
+│ r       │     1     │      1      │   node3   │  127.0.0.1   │ 9002 │     0    │
 │ r       │     2     │      1      │   node2   │  127.0.0.1   │ 9001 │     0    │
 │ r       │     2     │      2      │   node1   │  127.0.0.1   │ 9000 │     1    │
 └─────────┴───────────┴─────────────┴───────────┴──────────────┴──────┴──────────┘
@ -78,9 +83,9 @@ node1 :) SELECT materialize(hostName()) AS host, groupArray(n) FROM r.d GROUP BY
 ```

 ``` text
-┌─hosts─┬─groupArray(n)─┐ 
-│ node1 │  [1,3,5,7,9]  │   
-│ node2 │  [0,2,4,6,8]  │    
+┌─hosts─┬─groupArray(n)─┐
+│ node1 │  [1,3,5,7,9]  │
+│ node2 │  [0,2,4,6,8]  │
 └───────┴───────────────┘
 ```

@ -93,8 +98,8 @@ node4 :) CREATE DATABASE r ENGINE=Replicated('some/path/r','other_shard','r2');
 The cluster configuration will look like this:

 ``` text
-┌─cluster─┬─shard_num─┬─replica_num─┬─host_name─┬─host_address─┬─port─┬─is_local─┐ 
-│ r       │     1     │      1      │   node3   │  127.0.0.1   │ 9002 │     0    │ 
+┌─cluster─┬─shard_num─┬─replica_num─┬─host_name─┬─host_address─┬─port─┬─is_local─┐
+│ r       │     1     │      1      │   node3   │  127.0.0.1   │ 9002 │     0    │
 │ r       │     1     │      2      │   node4   │  127.0.0.1   │ 9003 │     0    │
 │ r       │     2     │      1      │   node2   │  127.0.0.1   │ 9001 │     0    │
 │ r       │     2     │      2      │   node1   │  127.0.0.1   │ 9000 │     1    │
@ -108,8 +113,8 @@ node2 :) SELECT materialize(hostName()) AS host, groupArray(n) FROM r.d GROUP BY
 ```

 ```text
-┌─hosts─┬─groupArray(n)─┐ 
-│ node2 │  [1,3,5,7,9]  │   
-│ node4 │  [0,2,4,6,8]  │    
+┌─hosts─┬─groupArray(n)─┐
+│ node2 │  [1,3,5,7,9]  │
+│ node4 │  [0,2,4,6,8]  │
 └───────┴───────────────┘
 ```
--- a/docs/en/engines/table-engines/integrations/ExternalDistributed.md
+++ b/docs/en/engines/table-engines/integrations/ExternalDistributed.md
@ -35,7 +35,7 @@ The table structure can differ from the original table structure:
 -   `password` — User password.

 ## Implementation Details {#implementation-details}
-	
+
 Supports multiple replicas that must be listed by `|` and shards must be listed by `,`. For example:

 ```sql
--- a/docs/en/engines/table-engines/integrations/embedded-rocksdb.md
+++ b/docs/en/engines/table-engines/integrations/embedded-rocksdb.md
@ -20,7 +20,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster]

 Required parameters:

-  `primary_key_name` – any column name in the column list.
+- `primary_key_name` – any column name in the column list.
 - `primary key` must be specified, it supports only one column in the primary key. The primary key will be serialized in binary as a `rocksdb key`.
 - columns other than the primary key will be serialized in binary as `rocksdb` value in corresponding order.
 - queries with key `equals` or `in` filtering will be optimized to multi keys lookup from `rocksdb`.
@ -39,4 +39,46 @@ ENGINE = EmbeddedRocksDB
 PRIMARY KEY key
 ```

+## Metrics
+
+There is also `system.rocksdb` table, that expose rocksdb statistics:
+
+```sql
+SELECT
+    name,
+    value
+FROM system.rocksdb
+
+┌─name──────────────────────┬─value─┐
+│ no.file.opens             │     1 │
+│ number.block.decompressed │     1 │
+└───────────────────────────┴───────┘
+```
+
+## Configuration
+
+You can also change any [rocksdb options](https://github.com/facebook/rocksdb/wiki/Option-String-and-Option-Map) using config:
+
+```xml
+<rocksdb>
+    <options>
+        <max_background_jobs>8</max_background_jobs>
+    </options>
+    <column_family_options>
+        <num_levels>2</num_levels>
+    </column_family_options>
+    <tables>
+        <table>
+            <name>TABLE</name>
+            <options>
+                <max_background_jobs>8</max_background_jobs>
+            </options>
+            <column_family_options>
+                <num_levels>2</num_levels>
+            </column_family_options>
+        </table>
+    </tables>
+</rocksdb>
+```
+
 [Original article](https://clickhouse.tech/docs/en/engines/table-engines/integrations/embedded-rocksdb/) <!--hide-->
--- a/docs/en/engines/table-engines/integrations/hdfs.md
+++ b/docs/en/engines/table-engines/integrations/hdfs.md
@ -50,11 +50,11 @@ SELECT * FROM hdfs_engine_table LIMIT 2

 ## Implementation Details {#implementation-details}

-   Reads and writes can be parallel
+-   Reads and writes can be parallel.
+-   [Zero-copy](../../../operations/storing-data.md#zero-copy) replication is supported.  
 -   Not supported:
    -   `ALTER` and `SELECT...SAMPLE` operations.
    -   Indexes.
-    -   Replication.

 **Globs in path**

@ -71,12 +71,12 @@ Constructions with `{}` are similar to the [remote](../../../sql-reference/table

 1.  Suppose we have several files in TSV format with the following URIs on HDFS:

-   ‘hdfs://hdfs1:9000/some_dir/some_file_1’
-   ‘hdfs://hdfs1:9000/some_dir/some_file_2’
-   ‘hdfs://hdfs1:9000/some_dir/some_file_3’
-   ‘hdfs://hdfs1:9000/another_dir/some_file_1’
-   ‘hdfs://hdfs1:9000/another_dir/some_file_2’
-   ‘hdfs://hdfs1:9000/another_dir/some_file_3’
+-   'hdfs://hdfs1:9000/some_dir/some_file_1'
+-   'hdfs://hdfs1:9000/some_dir/some_file_2'
+-   'hdfs://hdfs1:9000/some_dir/some_file_3'
+-   'hdfs://hdfs1:9000/another_dir/some_file_1'
+-   'hdfs://hdfs1:9000/another_dir/some_file_2'
+-   'hdfs://hdfs1:9000/another_dir/some_file_3'

 1.  There are several ways to make a table consisting of all six files:

@ -126,8 +126,9 @@ Similar to GraphiteMergeTree, the HDFS engine supports extended configuration us
  </hdfs_root>
 ```

-### List of possible configuration options with default values
-#### Supported by libhdfs3
+### Configuration Options {#configuration-options}
+
+#### Supported by libhdfs3 {#supported-by-libhdfs3}


 | **parameter**                                         | **default value**       |
@ -184,7 +185,7 @@ Similar to GraphiteMergeTree, the HDFS engine supports extended configuration us
 |hadoop\_kerberos\_principal                            | ""                      |
 |hadoop\_kerberos\_kinit\_command                       | kinit                   |

-#### Limitations {#limitations}
+### Limitations {#limitations}
  * hadoop\_security\_kerberos\_ticket\_cache\_path can be global only, not user specific

 ## Kerberos support {#kerberos-support}
--- a/docs/en/engines/table-engines/integrations/materialized-postgresql.md
+++ b/docs/en/engines/table-engines/integrations/materialized-postgresql.md
@ -5,42 +5,52 @@ toc_title: MaterializedPostgreSQL

 # MaterializedPostgreSQL {#materialize-postgresql}

+Creates ClickHouse table with an initial data dump of PostgreSQL table and starts replication process, i.e. executes background job to apply new changes as they happen on PostgreSQL table in the remote PostgreSQL database.
+
+If more than one table is required, it is highly recommended to use the [MaterializedPostgreSQL](../../../engines/database-engines/materialized-postgresql.md) database engine instead of the table engine and use the [materialized_postgresql_tables_list](../../../operations/settings/settings.md#materialized-postgresql-tables-list) setting, which specifies the tables to be replicated. It will be much better in terms of CPU, fewer connections and fewer replication slots inside the remote PostgreSQL database.
+
 ## Creating a Table {#creating-a-table}

 ``` sql
-CREATE TABLE test.postgresql_replica (key UInt64, value UInt64)
+CREATE TABLE postgresql_db.postgresql_replica (key UInt64, value UInt64)
 ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgres_database', 'postgresql_replica', 'postgres_user', 'postgres_password')
 PRIMARY KEY key;
 ```

+**Engine Parameters**
+
+-   `host:port` — PostgreSQL server address.
+-   `database` — Remote database name.
+-   `table` — Remote table name.
+-   `user` — PostgreSQL user.
+-   `password` — User password.

 ## Requirements {#requirements}

- Setting `wal_level`to `logical` and `max_replication_slots` to at least `2` in the postgresql config file.
+1. The [wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html) setting must have a value `logical` and `max_replication_slots` parameter must have a value at least `2` in the PostgreSQL config file.

- A table with engine `MaterializedPostgreSQL` must have a primary key - the same as a replica identity index (default: primary key) of a postgres table (See [details on replica identity index](../../database-engines/materialized-postgresql.md#requirements)).
+2. A table with `MaterializedPostgreSQL` engine must have a primary key — the same as a replica identity index (by default: primary key) of a PostgreSQL table (see [details on replica identity index](../../../engines/database-engines/materialized-postgresql.md#requirements)).

- Only database `Atomic` is allowed.
+3. Only database [Atomic](https://en.wikipedia.org/wiki/Atomicity_(database_systems)) is allowed.

+## Virtual columns {#virtual-columns}

-## Virtual columns {#creating-a-table}
+-   `_version` — Transaction counter. Type: [UInt64](../../../sql-reference/data-types/int-uint.md).

- `_version` (`UInt64`)
+-   `_sign` — Deletion mark. Type: [Int8](../../../sql-reference/data-types/int-uint.md). Possible values:
+    - `1` — Row is not deleted, 
+    - `-1` — Row is deleted.

- `_sign` (`Int8`)
-
-These columns do not need to be added, when table is created. They are always accessible in `SELECT` query.
+These columns do not need to be added when a table is created. They are always accessible in `SELECT` query.
 `_version` column equals `LSN` position in `WAL`, so it might be used to check how up-to-date replication is.

 ``` sql
-CREATE TABLE test.postgresql_replica (key UInt64, value UInt64)
+CREATE TABLE postgresql_db.postgresql_replica (key UInt64, value UInt64)
 ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgres_database', 'postgresql_replica', 'postgres_user', 'postgres_password')
 PRIMARY KEY key;

-SELECT key, value, _version FROM test.postgresql_replica;
+SELECT key, value, _version FROM postgresql_db.postgresql_replica;
 ```

-
-## Warning {#warning}
-
-1. **TOAST** values convertion is not supported. Default value for the data type will be used.
+!!! warning "Warning"
+    Replication of [**TOAST**](https://www.postgresql.org/docs/9.5/storage-toast.html) values is not supported. The default value for the data type will be used.
--- a/docs/en/engines/table-engines/integrations/mongodb.md
+++ b/docs/en/engines/table-engines/integrations/mongodb.md
@ -15,7 +15,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name
    name1 [type1],
    name2 [type2],
    ...
-) ENGINE = MongoDB(host:port, database, collection, user, password);
+) ENGINE = MongoDB(host:port, database, collection, user, password [, options]);
 ```

 **Engine Parameters**
@ -30,18 +30,30 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name

 -   `password` — User password.

+-   `options` — MongoDB connection string options (optional parameter).
+
 ## Usage Example {#usage-example}

-Table in ClickHouse which allows to read data from MongoDB collection:
+Create a table in ClickHouse which allows to read data from MongoDB collection:

 ``` text
 CREATE TABLE mongo_table
 (
-    key UInt64, 
+    key UInt64,
    data String
 ) ENGINE = MongoDB('mongo1:27017', 'test', 'simple_table', 'testuser', 'clickhouse');
 ```

+To read from an SSL secured MongoDB server:
+
+``` text
+CREATE TABLE mongo_table_ssl
+(
+    key UInt64,
+    data String
+) ENGINE = MongoDB('mongo2:27017', 'test', 'simple_table', 'testuser', 'clickhouse', 'ssl=true');
+```
+
 Query:

 ``` sql
--- a/docs/en/engines/table-engines/integrations/postgresql.md
+++ b/docs/en/engines/table-engines/integrations/postgresql.md
@ -49,14 +49,14 @@ PostgreSQL `Array` types are converted into ClickHouse arrays.

 !!! info "Note"
    Be careful - in PostgreSQL an array data, created like a `type_name[]`, may contain multi-dimensional arrays of different dimensions in different table rows in same column. But in ClickHouse it is only allowed to have multidimensional arrays of the same count of dimensions in all table rows in same column.
-	
+
 Supports multiple replicas that must be listed by `|`. For example:

 ```sql
 CREATE TABLE test_replicas (id UInt32, name String) ENGINE = PostgreSQL(`postgres{2|3|4}:5432`, 'clickhouse', 'test_replicas', 'postgres', 'mysecretpassword');
 ```

-Replicas priority for PostgreSQL dictionary source is supported. The bigger the number in map, the less the priority. The highest priority is `0`. 
+Replicas priority for PostgreSQL dictionary source is supported. The bigger the number in map, the less the priority. The highest priority is `0`.

 In the example below replica `example01-1` has the highest priority:

--- a/docs/en/engines/table-engines/integrations/s3.md
+++ b/docs/en/engines/table-engines/integrations/s3.md
@ -57,10 +57,10 @@ For more information about virtual columns see [here](../../../engines/table-eng
 ## Implementation Details {#implementation-details}

 -   Reads and writes can be parallel
+-   [Zero-copy](../../../operations/storing-data.md#zero-copy) replication is supported.  
 -   Not supported:
    -   `ALTER` and `SELECT...SAMPLE` operations.
    -   Indexes.
-    -   Replication.

 ## Wildcards In Path {#wildcards-in-path}

@ -77,12 +77,12 @@ Constructions with `{}` are similar to the [remote](../../../sql-reference/table

 1. Suppose we have several files in CSV format with the following URIs on S3:

-   ‘https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_1.csv’
-   ‘https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_2.csv’
-   ‘https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_3.csv’
-   ‘https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_1.csv’
-   ‘https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_2.csv’
-   ‘https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_3.csv’
+-   'https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_1.csv'
+-   'https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_2.csv'
+-   'https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_3.csv'
+-   'https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_1.csv'
+-   'https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_2.csv'
+-   'https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_3.csv'

 There are several ways to make a table consisting of all six files:

--- a/docs/en/engines/table-engines/log-family/index.md
+++ b/docs/en/engines/table-engines/log-family/index.md
@ -14,6 +14,8 @@ Engines of the family:
 -   [Log](../../../engines/table-engines/log-family/log.md)
 -   [TinyLog](../../../engines/table-engines/log-family/tinylog.md)

+`Log` family table engines can store data to [HDFS](../../../engines/table-engines/mergetree-family/mergetree.md#table_engine-mergetree-hdfs) or [S3](../../../engines/table-engines/mergetree-family/mergetree.md#table_engine-mergetree-s3) distributed file systems.
+
 ## Common Properties {#common-properties}

 Engines:
--- a/docs/en/engines/table-engines/log-family/log.md
+++ b/docs/en/engines/table-engines/log-family/log.md
@ -5,10 +5,8 @@ toc_title: Log

 # Log {#log}

-Engine belongs to the family of log engines. See the common properties of log engines and their differences in the [Log Engine Family](../../../engines/table-engines/log-family/index.md) article.
+The engine belongs to the family of `Log` engines. See the common properties of `Log` engines and their differences in the [Log Engine Family](../../../engines/table-engines/log-family/index.md) article.

-Log differs from [TinyLog](../../../engines/table-engines/log-family/tinylog.md) in that a small file of “marks” resides with the column files. These marks are written on every data block and contain offsets that indicate where to start reading the file in order to skip the specified number of rows. This makes it possible to read table data in multiple threads.
+`Log` differs from [TinyLog](../../../engines/table-engines/log-family/tinylog.md) in that a small file of "marks" resides with the column files. These marks are written on every data block and contain offsets that indicate where to start reading the file in order to skip the specified number of rows. This makes it possible to read table data in multiple threads.
 For concurrent data access, the read operations can be performed simultaneously, while write operations block reads and each other.
-The Log engine does not support indexes. Similarly, if writing to a table failed, the table is broken, and reading from it returns an error. The Log engine is appropriate for temporary data, write-once tables, and for testing or demonstration purposes.
-
-[Original article](https://clickhouse.tech/docs/en/operations/table_engines/log/) <!--hide-->
+The `Log` engine does not support indexes. Similarly, if writing to a table failed, the table is broken, and reading from it returns an error. The `Log` engine is appropriate for temporary data, write-once tables, and for testing or demonstration purposes.
--- a/docs/en/engines/table-engines/mergetree-family/graphitemergetree.md
+++ b/docs/en/engines/table-engines/mergetree-family/graphitemergetree.md
@ -38,9 +38,7 @@ A table for the Graphite data should have the following columns for the followin

 -   Value of the metric. Data type: any numeric.

-   Version of the metric. Data type: any numeric.
-
-    ClickHouse saves the rows with the highest version or the last written if versions are the same. Other rows are deleted during the merge of data parts.
+-   Version of the metric. Data type: any numeric (ClickHouse saves the rows with the highest version or the last written if versions are the same. Other rows are deleted during the merge of data parts).

 The names of these columns should be set in the rollup configuration.

@ -132,7 +130,7 @@ Fields for `pattern` and `default` sections:
 -   `regexp`– A pattern for the metric name.
 -   `age` – The minimum age of the data in seconds.
 -   `precision`– How precisely to define the age of the data in seconds. Should be a divisor for 86400 (seconds in a day).
-   `function` – The name of the aggregating function to apply to data whose age falls within the range `[age, age + precision]`.
+-   `function` – The name of the aggregating function to apply to data whose age falls within the range `[age, age + precision]`. Accepted functions: min / max / any / avg. The average is calculated imprecisely, like the average of the averages. 

 ### Configuration Example {#configuration-example}

@ -169,4 +167,7 @@ Fields for `pattern` and `default` sections:
 </graphite_rollup>
 ```

+!!! warning "Warning"
+    Data rollup is performed during merges. Usually, for old partitions, merges are not started, so for rollup it is necessary to trigger an unscheduled merge using [optimize](../../../sql-reference/statements/optimize.md). Or use additional tools, for example [graphite-ch-optimizer](https://github.com/innogames/graphite-ch-optimizer).
+
 [Original article](https://clickhouse.tech/docs/en/operations/table_engines/graphitemergetree/) <!--hide-->
--- a/docs/en/engines/table-engines/mergetree-family/mergetree.md
+++ b/docs/en/engines/table-engines/mergetree-family/mergetree.md
@ -39,7 +39,10 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster]
    name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2] [TTL expr2],
    ...
    INDEX index_name1 expr1 TYPE type1(...) GRANULARITY value1,
-    INDEX index_name2 expr2 TYPE type2(...) GRANULARITY value2
+    INDEX index_name2 expr2 TYPE type2(...) GRANULARITY value2,
+    ...
+    PROJECTION projection_name_1 (SELECT <COLUMN LIST EXPR> [GROUP BY] [ORDER BY]),
+    PROJECTION projection_name_2 (SELECT <COLUMN LIST EXPR> [GROUP BY] [ORDER BY])
 ) ENGINE = MergeTree()
 ORDER BY expr
 [PARTITION BY expr]
@ -76,7 +79,7 @@ For a description of parameters, see the [CREATE query description](../../../sql

 -   `SAMPLE BY` — An expression for sampling. Optional.

-    If a sampling expression is used, the primary key must contain it. The result of sampling expression must be unsigned integer. Example: `SAMPLE BY intHash32(UserID) ORDER BY (CounterID, EventDate, intHash32(UserID))`.
+    If a sampling expression is used, the primary key must contain it. The result of a sampling expression must be an unsigned integer. Example: `SAMPLE BY intHash32(UserID) ORDER BY (CounterID, EventDate, intHash32(UserID))`.

 -   `TTL` — A list of rules specifying storage duration of rows and defining logic of automatic parts movement [between disks and volumes](#table_engine-mergetree-multiple-volumes). Optional.

@ -385,6 +388,24 @@ Functions with a constant argument that is less than ngram size can’t be used
    -   `s != 1`
    -   `NOT startsWith(s, 'test')`

+### Projections {#projections}
+Projections are like materialized views but defined in part-level. It provides consistency guarantees along with automatic usage in queries.
+
+#### Query {#projection-query}
+A projection query is what defines a projection. It has the following grammar:
+
+`SELECT <COLUMN LIST EXPR> [GROUP BY] [ORDER BY]`
+
+It implicitly selects data from the parent table.
+
+#### Storage {#projection-storage}
+Projections are stored inside the part directory. It's similar to an index but contains a subdirectory that stores an anonymous MergeTree table's part. The table is induced by the definition query of the projection. If there is a GROUP BY clause, the underlying storage engine becomes AggregatedMergeTree, and all aggregate functions are converted to AggregateFunction. If there is an ORDER BY clause, the MergeTree table will use it as its primary key expression. During the merge process, the projection part will be merged via its storage's merge routine. The checksum of the parent table's part will combine the projection's part. Other maintenance jobs are similar to skip indices.
+
+#### Query Analysis {#projection-query-analysis}
+1. Check if the projection can be used to answer the given query, that is, it generates the same answer as querying the base table.
+2. Select the best feasible match, which contains the least granules to read.
+3. The query pipeline which uses projections will be different from the one that uses the original parts. If the projection is absent in some parts, we can add the pipeline to "project" it on the fly.
+
 ## Concurrent Data Access {#concurrent-data-access}

 For concurrent table access, we use multi-versioning. In other words, when a table is simultaneously read and updated, data is read from a set of parts that is current at the time of the query. There are no lengthy locks. Inserts do not get in the way of read operations.
@ -728,7 +749,9 @@ During this time, they are not moved to other volumes or disks. Therefore, until

 ## Using S3 for Data Storage {#table_engine-mergetree-s3}

-`MergeTree` family table engines is able to store data to [S3](https://aws.amazon.com/s3/) using a disk with type `s3`.
+`MergeTree` family table engines can store data to [S3](https://aws.amazon.com/s3/) using a disk with type `s3`.
+
+This feature is under development and not ready for production. There are known drawbacks such as very low performance.

 Configuration markup:
 ``` xml
@ -762,11 +785,13 @@ Configuration markup:
 ```

 Required parameters:
-   `endpoint` — S3 endpoint url in `path` or `virtual hosted` [styles](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html). Endpoint url should contain bucket and root path to store data.
+
+-   `endpoint` — S3 endpoint URL in `path` or `virtual hosted` [styles](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html). Endpoint URL should contain a bucket and root path to store data.
 -   `access_key_id` — S3 access key id.
 -   `secret_access_key` — S3 secret access key.

 Optional parameters:
+
 -   `region` — S3 region name.
 -   `use_environment_credentials` — Reads AWS credentials from the Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN if they exist. Default value is `false`.
 -   `use_insecure_imds_request` — If set to `true`, S3 client will use insecure IMDS request while obtaining credentials from Amazon EC2 metadata. Default value is `false`.
@ -782,7 +807,6 @@ Optional parameters:
 -   `skip_access_check` — If true, disk access checks will not be performed on disk start-up. Default value is `false`.
 -   `server_side_encryption_customer_key_base64` — If specified, required headers for accessing S3 objects with SSE-C encryption will be set.

-
 S3 disk can be configured as `main` or `cold` storage:
 ``` xml
 <storage_configuration>
@ -820,5 +844,3 @@ S3 disk can be configured as `main` or `cold` storage:
 ```

 In case of `cold` option a data can be moved to S3 if local disk free size will be smaller than `move_factor * disk_size` or by TTL move rule.
-
-[Original article](https://clickhouse.tech/docs/ru/operations/table_engines/mergetree/) <!--hide-->
--- a/docs/en/engines/table-engines/mergetree-family/replication.md
+++ b/docs/en/engines/table-engines/mergetree-family/replication.md
@ -101,7 +101,7 @@ For very large clusters, you can use different ZooKeeper clusters for different

 Replication is asynchronous and multi-master. `INSERT` queries (as well as `ALTER`) can be sent to any available server. Data is inserted on the server where the query is run, and then it is copied to the other servers. Because it is asynchronous, recently inserted data appears on the other replicas with some latency. If part of the replicas are not available, the data is written when they become available. If a replica is available, the latency is the amount of time it takes to transfer the block of compressed data over the network. The number of threads performing background tasks for replicated tables can be set by [background_schedule_pool_size](../../../operations/settings/settings.md#background_schedule_pool_size) setting.

-`ReplicatedMergeTree` engine uses a separate thread pool for replicated fetches. Size of the pool is limited by the [background_fetches_pool_size](../../../operations/settings/settings.md#background_fetches_pool_size) setting which can be tuned with a server restart. 
+`ReplicatedMergeTree` engine uses a separate thread pool for replicated fetches. Size of the pool is limited by the [background_fetches_pool_size](../../../operations/settings/settings.md#background_fetches_pool_size) setting which can be tuned with a server restart.

 By default, an INSERT query waits for confirmation of writing the data from only one replica. If the data was successfully written to only one replica and the server with this replica ceases to exist, the stored data will be lost. To enable getting confirmation of data writes from multiple replicas, use the `insert_quorum` option.

@ -155,7 +155,7 @@ CREATE TABLE table_name

 </details>

-As the example shows, these parameters can contain substitutions in curly brackets. The substituted values are taken from the «[macros](../../../operations/server-configuration-parameters/settings/#macros) section of the configuration file. 
+As the example shows, these parameters can contain substitutions in curly brackets. The substituted values are taken from the «[macros](../../../operations/server-configuration-parameters/settings/#macros) section of the configuration file.

 Example:

@ -198,7 +198,7 @@ In this case, you can omit arguments when creating tables:
 ``` sql
 CREATE TABLE table_name (
 	x UInt32
-) ENGINE = ReplicatedMergeTree 
+) ENGINE = ReplicatedMergeTree
 ORDER BY x;
 ```

@ -207,7 +207,7 @@ It is equivalent to:
 ``` sql
 CREATE TABLE table_name (
 	x UInt32
-) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/{database}/table_name', '{replica}') 
+) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/{database}/table_name', '{replica}')
 ORDER BY x;
 ```

--- a/docs/en/engines/table-engines/special/join.md
+++ b/docs/en/engines/table-engines/special/join.md
@ -30,65 +30,27 @@ See the detailed description of the [CREATE TABLE](../../../sql-reference/statem

 Enter `join_strictness` and `join_type` parameters without quotes, for example, `Join(ANY, LEFT, col1)`. They must match the `JOIN` operation that the table will be used for. If the parameters do not match, ClickHouse does not throw an exception and may return incorrect data.

-## Table Usage {#table-usage}
+## Specifics and Recommendations {#specifics-and-recommendations}

-### Example {#example}
+### Data Storage {#data-storage}

-Creating the left-side table:
+`Join` table data is always located in the RAM. When inserting rows into a table, ClickHouse writes data blocks to the directory on the disk so that they can be restored when the server restarts.

-``` sql
-CREATE TABLE id_val(`id` UInt32, `val` UInt32) ENGINE = TinyLog
-```
-
-``` sql
-INSERT INTO id_val VALUES (1,11)(2,12)(3,13)
-```
-
-Creating the right-side `Join` table:
-
-``` sql
-CREATE TABLE id_val_join(`id` UInt32, `val` UInt8) ENGINE = Join(ANY, LEFT, id)
-```
-
-``` sql
-INSERT INTO id_val_join VALUES (1,21)(1,22)(3,23)
-```
-
-Joining the tables:
-
-``` sql
-SELECT * FROM id_val ANY LEFT JOIN id_val_join USING (id) SETTINGS join_use_nulls = 1
-```
-
-``` text
-┌─id─┬─val─┬─id_val_join.val─┐
-│  1 │  11 │              21 │
-│  2 │  12 │            ᴺᵁᴸᴸ │
-│  3 │  13 │              23 │
-└────┴─────┴─────────────────┘
-```
-
-As an alternative, you can retrieve data from the `Join` table, specifying the join key value:
-
-``` sql
-SELECT joinGet('id_val_join', 'val', toUInt32(1))
-```
-
-``` text
-┌─joinGet('id_val_join', 'val', toUInt32(1))─┐
-│                                         21 │
-└────────────────────────────────────────────┘
-```
+If the server restarts incorrectly, the data block on the disk might get lost or damaged. In this case, you may need to manually delete the file with damaged data.

 ### Selecting and Inserting Data {#selecting-and-inserting-data}

 You can use `INSERT` queries to add data to the `Join`-engine tables. If the table was created with the `ANY` strictness, data for duplicate keys are ignored. With the `ALL` strictness, all rows are added.

-You cannot perform a `SELECT` query directly from the table. Instead, use one of the following methods:
+Main use-cases for `Join`-engine tables are following:

 -   Place the table to the right side in a `JOIN` clause.
 -   Call the [joinGet](../../../sql-reference/functions/other-functions.md#joinget) function, which lets you extract data from the table the same way as from a dictionary.

+### Deleting Data {#deleting-data}
+
+`ALTER DELETE` queries for `Join`-engine tables are implemented as [mutations](../../../sql-reference/statements/alter/index.md#mutations). `DELETE` mutation reads filtered data and overwrites data of memory and disk.
+
 ### Limitations and Settings {#join-limitations-and-settings}

 When creating a table, the following settings are applied:
@ -102,12 +64,64 @@ When creating a table, the following settings are applied:

 The `Join`-engine tables can’t be used in `GLOBAL JOIN` operations.

-The `Join`-engine allows use [join_use_nulls](../../../operations/settings/settings.md#join_use_nulls) setting in the `CREATE TABLE` statement. And [SELECT](../../../sql-reference/statements/select/index.md) query allows use `join_use_nulls` too. If you have different `join_use_nulls` settings, you can get an error joining table. It depends on kind of JOIN. When you use [joinGet](../../../sql-reference/functions/other-functions.md#joinget) function, you have to use the same `join_use_nulls` setting in `CRATE TABLE` and `SELECT` statements.
+The `Join`-engine allows to specify [join_use_nulls](../../../operations/settings/settings.md#join_use_nulls) setting in the `CREATE TABLE` statement. [SELECT](../../../sql-reference/statements/select/index.md) query should have the same `join_use_nulls` value.

-## Data Storage {#data-storage}
+## Usage Examples {#example}

-`Join` table data is always located in the RAM. When inserting rows into a table, ClickHouse writes data blocks to the directory on the disk so that they can be restored when the server restarts.
+Creating the left-side table:

-If the server restarts incorrectly, the data block on the disk might get lost or damaged. In this case, you may need to manually delete the file with damaged data.
+``` sql
+CREATE TABLE id_val(`id` UInt32, `val` UInt32) ENGINE = TinyLog;
+```

-[Original article](https://clickhouse.tech/docs/en/operations/table_engines/join/) <!--hide-->
+``` sql
+INSERT INTO id_val VALUES (1,11)(2,12)(3,13);
+```
+
+Creating the right-side `Join` table:
+
+``` sql
+CREATE TABLE id_val_join(`id` UInt32, `val` UInt8) ENGINE = Join(ANY, LEFT, id);
+```
+
+``` sql
+INSERT INTO id_val_join VALUES (1,21)(1,22)(3,23);
+```
+
+Joining the tables:
+
+``` sql
+SELECT * FROM id_val ANY LEFT JOIN id_val_join USING (id);
+```
+
+``` text
+┌─id─┬─val─┬─id_val_join.val─┐
+│  1 │  11 │              21 │
+│  2 │  12 │               0 │
+│  3 │  13 │              23 │
+└────┴─────┴─────────────────┘
+```
+
+As an alternative, you can retrieve data from the `Join` table, specifying the join key value:
+
+``` sql
+SELECT joinGet('id_val_join', 'val', toUInt32(1));
+```
+
+``` text
+┌─joinGet('id_val_join', 'val', toUInt32(1))─┐
+│                                         21 │
+└────────────────────────────────────────────┘
+```
+
+Deleting a row from the `Join` table:
+
+```sql
+ALTER TABLE id_val_join DELETE WHERE id = 3;
+```
+
+```text
+┌─id─┬─val─┐
+│  1 │  21 │
+└────┴─────┘
+```
--- a/docs/en/getting-started/example-datasets/index.md
+++ b/docs/en/getting-started/example-datasets/index.md
@ -14,7 +14,10 @@ The list of documented datasets:
 -   [Anonymized Yandex.Metrica Dataset](../../getting-started/example-datasets/metrica.md)
 -   [Recipes](../../getting-started/example-datasets/recipes.md)
 -   [OnTime](../../getting-started/example-datasets/ontime.md)
+-   [OpenSky](../../getting-started/example-datasets/opensky.md)
 -   [New York Taxi Data](../../getting-started/example-datasets/nyc-taxi.md)
+-   [UK Property Price Paid](../../getting-started/example-datasets/uk-price-paid.md)
+-   [What's on the Menu?](../../getting-started/example-datasets/menus.md)
 -   [Star Schema Benchmark](../../getting-started/example-datasets/star-schema.md)
 -   [WikiStat](../../getting-started/example-datasets/wikistat.md)
 -   [Terabyte of Click Logs from Criteo](../../getting-started/example-datasets/criteo.md)
--- a/docs/en/getting-started/example-datasets/menus.md
+++ b/docs/en/getting-started/example-datasets/menus.md
@ -0,0 +1,324 @@
+---
+toc_priority: 21
+toc_title: Menus
+---
+
+# New York Public Library "What's on the Menu?" Dataset
+
+The dataset is created by the New York Public Library. It contains historical data on the menus of hotels, restaurants and cafes with the dishes along with their prices.
+
+Source: http://menus.nypl.org/data
+The data is in public domain.
+
+The data is from library's archive and it may be incomplete and difficult for statistical analysis. Nevertheless it is also very yummy.
+The size is just 1.3 million records about dishes in the menus (a very small data volume for ClickHouse, but it's still a good example).
+
+## Download the Dataset
+
+```
+wget https://s3.amazonaws.com/menusdata.nypl.org/gzips/2021_08_01_07_01_17_data.tgz
+```
+
+Replace the link to the up to date link from http://menus.nypl.org/data if needed.
+Download size is about 35 MB.
+
+## Unpack the Dataset
+
+```
+tar xvf 2021_08_01_07_01_17_data.tgz
+```
+
+Uncompressed size is about 150 MB.
+
+The data is normalized consisted of four tables:
+- Menu: information about menus: the name of the restaurant, the date when menu was seen, etc;
+- Dish: information about dishes: the name of the dish along with some characteristic;
+- MenuPage: information about the pages in the menus; every page belongs to some menu;
+- MenuItem: an item of the menu - a dish along with its price on some menu page: links to dish and menu page.
+
+## Create the Tables
+
+```
+CREATE TABLE dish
+(
+    id UInt32,
+    name String,
+    description String,
+    menus_appeared UInt32,
+    times_appeared Int32,
+    first_appeared UInt16,
+    last_appeared UInt16,
+    lowest_price Decimal64(3),
+    highest_price Decimal64(3)
+) ENGINE = MergeTree ORDER BY id;
+
+CREATE TABLE menu
+(
+    id UInt32,
+    name String,
+    sponsor String,
+    event String,
+    venue String,
+    place String,
+    physical_description String,
+    occasion String,
+    notes String,
+    call_number String,
+    keywords String,
+    language String,
+    date String,
+    location String,
+    location_type String,
+    currency String,
+    currency_symbol String,
+    status String,
+    page_count UInt16,
+    dish_count UInt16
+) ENGINE = MergeTree ORDER BY id;
+
+CREATE TABLE menu_page
+(
+    id UInt32,
+    menu_id UInt32,
+    page_number UInt16,
+    image_id String,
+    full_height UInt16,
+    full_width UInt16,
+    uuid UUID
+) ENGINE = MergeTree ORDER BY id;
+
+CREATE TABLE menu_item
+(
+    id UInt32,
+    menu_page_id UInt32,
+    price Decimal64(3),
+    high_price Decimal64(3),
+    dish_id UInt32,
+    created_at DateTime,
+    updated_at DateTime,
+    xpos Float64,
+    ypos Float64
+) ENGINE = MergeTree ORDER BY id;
+```
+
+We use `Decimal` data type to store prices. Everything else is quite straightforward.
+
+## Import Data
+
+Upload data into ClickHouse in parallel:
+
+```
+clickhouse-client --format_csv_allow_single_quotes 0 --input_format_null_as_default 0 --query "INSERT INTO dish FORMAT CSVWithNames" < Dish.csv
+clickhouse-client --format_csv_allow_single_quotes 0 --input_format_null_as_default 0 --query "INSERT INTO menu FORMAT CSVWithNames" < Menu.csv
+clickhouse-client --format_csv_allow_single_quotes 0 --input_format_null_as_default 0 --query "INSERT INTO menu_page FORMAT CSVWithNames" < MenuPage.csv
+clickhouse-client --format_csv_allow_single_quotes 0 --input_format_null_as_default 0 --date_time_input_format best_effort --query "INSERT INTO menu_item FORMAT CSVWithNames" < MenuItem.csv
+```
+
+We use `CSVWithNames` format as the data is represented by CSV with header.
+
+We disable `format_csv_allow_single_quotes` as only double quotes are used for data fields and single quotes can be inside the values and should not confuse the CSV parser.
+
+We disable `input_format_null_as_default` as our data does not have NULLs. Otherwise ClickHouse will try to parse `\N` sequences and can be confused with `\` in data.
+
+The setting `--date_time_input_format best_effort` allows to parse `DateTime` fields in wide variety of formats. For example, ISO-8601 without seconds like '2000-01-01 01:02' will be recognized. Without this setting only fixed DateTime format is allowed.
+
+## Denormalize the Data
+
+Data is presented in multiple tables in normalized form. It means you have to perform JOINs if you want to query, e.g. dish names from menu items.
+For typical analytical tasks it is way more efficient to deal with pre-JOINed data to avoid doing JOIN every time. It is called "denormalized" data.
+
+We will create a table that will contain all the data JOINed together:
+
+```
+CREATE TABLE menu_item_denorm
+ENGINE = MergeTree ORDER BY (dish_name, created_at)
+AS SELECT
+    price,
+    high_price,
+    created_at,
+    updated_at,
+    xpos,
+    ypos,
+    dish.id AS dish_id,
+    dish.name AS dish_name,
+    dish.description AS dish_description,
+    dish.menus_appeared AS dish_menus_appeared,
+    dish.times_appeared AS dish_times_appeared,
+    dish.first_appeared AS dish_first_appeared,
+    dish.last_appeared AS dish_last_appeared,
+    dish.lowest_price AS dish_lowest_price,
+    dish.highest_price AS dish_highest_price,
+    menu.id AS menu_id,
+    menu.name AS menu_name,
+    menu.sponsor AS menu_sponsor,
+    menu.event AS menu_event,
+    menu.venue AS menu_venue,
+    menu.place AS menu_place,
+    menu.physical_description AS menu_physical_description,
+    menu.occasion AS menu_occasion,
+    menu.notes AS menu_notes,
+    menu.call_number AS menu_call_number,
+    menu.keywords AS menu_keywords,
+    menu.language AS menu_language,
+    menu.date AS menu_date,
+    menu.location AS menu_location,
+    menu.location_type AS menu_location_type,
+    menu.currency AS menu_currency,
+    menu.currency_symbol AS menu_currency_symbol,
+    menu.status AS menu_status,
+    menu.page_count AS menu_page_count,
+    menu.dish_count AS menu_dish_count
+FROM menu_item
+    JOIN dish ON menu_item.dish_id = dish.id
+    JOIN menu_page ON menu_item.menu_page_id = menu_page.id
+    JOIN menu ON menu_page.menu_id = menu.id
+```
+
+## Validate the Data
+
+```
+SELECT count() FROM menu_item_denorm
+1329175
+```
+
+## Run Some Queries
+
+Averaged historical prices of dishes:
+
+```
+SELECT
+    round(toUInt32OrZero(extract(menu_date, '^\\d{4}')), -1) AS d,
+    count(),
+    round(avg(price), 2),
+    bar(avg(price), 0, 100, 100)
+FROM menu_item_denorm
+WHERE (menu_currency = 'Dollars') AND (d > 0) AND (d < 2022)
+GROUP BY d
+ORDER BY d ASC
+
+┌────d─┬─count()─┬─round(avg(price), 2)─┬─bar(avg(price), 0, 100, 100)─┐
+│ 1850 │     618 │                  1.5 │ █▍                           │
+│ 1860 │    1634 │                 1.29 │ █▎                           │
+│ 1870 │    2215 │                 1.36 │ █▎                           │
+│ 1880 │    3909 │                 1.01 │ █                            │
+│ 1890 │    8837 │                  1.4 │ █▍                           │
+│ 1900 │  176292 │                 0.68 │ ▋                            │
+│ 1910 │  212196 │                 0.88 │ ▊                            │
+│ 1920 │  179590 │                 0.74 │ ▋                            │
+│ 1930 │   73707 │                  0.6 │ ▌                            │
+│ 1940 │   58795 │                 0.57 │ ▌                            │
+│ 1950 │   41407 │                 0.95 │ ▊                            │
+│ 1960 │   51179 │                 1.32 │ █▎                           │
+│ 1970 │   12914 │                 1.86 │ █▋                           │
+│ 1980 │    7268 │                 4.35 │ ████▎                        │
+│ 1990 │   11055 │                 6.03 │ ██████                       │
+│ 2000 │    2467 │                11.85 │ ███████████▋                 │
+│ 2010 │     597 │                25.66 │ █████████████████████████▋   │
+└──────┴─────────┴──────────────────────┴──────────────────────────────┘
+
+17 rows in set. Elapsed: 0.044 sec. Processed 1.33 million rows, 54.62 MB (30.00 million rows/s., 1.23 GB/s.)
+```
+
+Take it with a grain of salt.
+
+### Burger Prices:
+
+```
+SELECT
+    round(toUInt32OrZero(extract(menu_date, '^\\d{4}')), -1) AS d,
+    count(),
+    round(avg(price), 2),
+    bar(avg(price), 0, 50, 100)
+FROM menu_item_denorm
+WHERE (menu_currency = 'Dollars') AND (d > 0) AND (d < 2022) AND (dish_name ILIKE '%burger%')
+GROUP BY d
+ORDER BY d ASC
+
+┌────d─┬─count()─┬─round(avg(price), 2)─┬─bar(avg(price), 0, 50, 100)───────────┐
+│ 1880 │       2 │                 0.42 │ ▋                                     │
+│ 1890 │       7 │                 0.85 │ █▋                                    │
+│ 1900 │     399 │                 0.49 │ ▊                                     │
+│ 1910 │     589 │                 0.68 │ █▎                                    │
+│ 1920 │     280 │                 0.56 │ █                                     │
+│ 1930 │      74 │                 0.42 │ ▋                                     │
+│ 1940 │     119 │                 0.59 │ █▏                                    │
+│ 1950 │     134 │                 1.09 │ ██▏                                   │
+│ 1960 │     272 │                 0.92 │ █▋                                    │
+│ 1970 │     108 │                 1.18 │ ██▎                                   │
+│ 1980 │      88 │                 2.82 │ █████▋                                │
+│ 1990 │     184 │                 3.68 │ ███████▎                              │
+│ 2000 │      21 │                 7.14 │ ██████████████▎                       │
+│ 2010 │       6 │                18.42 │ ████████████████████████████████████▋ │
+└──────┴─────────┴──────────────────────┴───────────────────────────────────────┘
+
+14 rows in set. Elapsed: 0.052 sec. Processed 1.33 million rows, 94.15 MB (25.48 million rows/s., 1.80 GB/s.)
+```
+
+### Vodka:
+
+```
+SELECT
+    round(toUInt32OrZero(extract(menu_date, '^\\d{4}')), -1) AS d,
+    count(),
+    round(avg(price), 2),
+    bar(avg(price), 0, 50, 100)
+FROM menu_item_denorm
+WHERE (menu_currency IN ('Dollars', '')) AND (d > 0) AND (d < 2022) AND (dish_name ILIKE '%vodka%')
+GROUP BY d
+ORDER BY d ASC
+
+┌────d─┬─count()─┬─round(avg(price), 2)─┬─bar(avg(price), 0, 50, 100)─┐
+│ 1910 │       2 │                    0 │                             │
+│ 1920 │       1 │                  0.3 │ ▌                           │
+│ 1940 │      21 │                 0.42 │ ▋                           │
+│ 1950 │      14 │                 0.59 │ █▏                          │
+│ 1960 │     113 │                 2.17 │ ████▎                       │
+│ 1970 │      37 │                 0.68 │ █▎                          │
+│ 1980 │      19 │                 2.55 │ █████                       │
+│ 1990 │      86 │                  3.6 │ ███████▏                    │
+│ 2000 │       2 │                 3.98 │ ███████▊                    │
+└──────┴─────────┴──────────────────────┴─────────────────────────────┘
+```
+
+To get vodka we have to write `ILIKE '%vodka%'` and this definitely makes a statement.
+
+### Caviar:
+
+Let's print caviar prices. Also let's print a name of any dish with caviar.
+
+```
+SELECT
+    round(toUInt32OrZero(extract(menu_date, '^\\d{4}')), -1) AS d,
+    count(),
+    round(avg(price), 2),
+    bar(avg(price), 0, 50, 100),
+    any(dish_name)
+FROM menu_item_denorm
+WHERE (menu_currency IN ('Dollars', '')) AND (d > 0) AND (d < 2022) AND (dish_name ILIKE '%caviar%')
+GROUP BY d
+ORDER BY d ASC
+
+┌────d─┬─count()─┬─round(avg(price), 2)─┬─bar(avg(price), 0, 50, 100)──────┬─any(dish_name)──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ 1090 │       1 │                    0 │                                  │ Caviar                                                                                                                              │
+│ 1880 │       3 │                    0 │                                  │ Caviar                                                                                                                              │
+│ 1890 │      39 │                 0.59 │ █▏                               │ Butter and caviar                                                                                                                   │
+│ 1900 │    1014 │                 0.34 │ ▋                                │ Anchovy Caviar on Toast                                                                                                             │
+│ 1910 │    1588 │                 1.35 │ ██▋                              │ 1/1 Brötchen Caviar                                                                                                                 │
+│ 1920 │     927 │                 1.37 │ ██▋                              │ ASTRAKAN CAVIAR                                                                                                                     │
+│ 1930 │     289 │                 1.91 │ ███▋                             │ Astrachan caviar                                                                                                                    │
+│ 1940 │     201 │                 0.83 │ █▋                               │ (SPECIAL) Domestic Caviar Sandwich                                                                                                  │
+│ 1950 │      81 │                 2.27 │ ████▌                            │ Beluga Caviar                                                                                                                       │
+│ 1960 │     126 │                 2.21 │ ████▍                            │ Beluga Caviar                                                                                                                       │
+│ 1970 │     105 │                 0.95 │ █▊                               │ BELUGA MALOSSOL CAVIAR AMERICAN DRESSING                                                                                            │
+│ 1980 │      12 │                 7.22 │ ██████████████▍                  │ Authentic Iranian Beluga Caviar the world's finest black caviar presented in ice garni and a sampling of chilled 100° Russian vodka │
+│ 1990 │      74 │                14.42 │ ████████████████████████████▋    │ Avocado Salad, Fresh cut avocado with caviare                                                                                       │
+│ 2000 │       3 │                 7.82 │ ███████████████▋                 │ Aufgeschlagenes Kartoffelsueppchen mit Forellencaviar                                                                               │
+│ 2010 │       6 │                15.58 │ ███████████████████████████████▏ │ "OYSTERS AND PEARLS" "Sabayon" of Pearl Tapioca with Island Creek Oysters and Russian Sevruga Caviar                                │
+└──────┴─────────┴──────────────────────┴──────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+```
+
+At least they have caviar with vodka. Very nice.
+
+### Test it in Playground
+
+The data is uploaded to ClickHouse Playground, [example](https://gh-api.clickhouse.tech/play?user=play#U0VMRUNUCiAgICByb3VuZCh0b1VJbnQzMk9yWmVybyhleHRyYWN0KG1lbnVfZGF0ZSwgJ15cXGR7NH0nKSksIC0xKSBBUyBkLAogICAgY291bnQoKSwKICAgIHJvdW5kKGF2ZyhwcmljZSksIDIpLAogICAgYmFyKGF2ZyhwcmljZSksIDAsIDUwLCAxMDApLAogICAgYW55KGRpc2hfbmFtZSkKRlJPTSBtZW51X2l0ZW1fZGVub3JtCldIRVJFIChtZW51X2N1cnJlbmN5IElOICgnRG9sbGFycycsICcnKSkgQU5EIChkID4gMCkgQU5EIChkIDwgMjAyMikgQU5EIChkaXNoX25hbWUgSUxJS0UgJyVjYXZpYXIlJykKR1JPVVAgQlkgZApPUkRFUiBCWSBkIEFTQw==).
--- a/Show More
+++ b/Show More
				`@ -0,0 +1 @@`
				`Subproject commit bf905ea2251191ff9911ae7ec0cfc35d41f9f7f6`
				`@ -0,0 +1 @@`
				`Subproject commit 59537bdcf57bbed17913292cb4502d15657231f1`
				`@ -0,0 +1 @@`
				`Subproject commit c753054304d87daf460057c1a649c482aa094835`
				`@ -0,0 +1 @@`
				`Subproject commit 1d16ac28036e19fe8da7ba72c16a307fbdf8c87e`