ClickHouse/docs/changelogs/v21.8.1.7409-prestable.md

ClickHouse release v21.8.1.7409-prestable FIXME as compared to v21.7.1.7283-prestable

Backward Incompatible Change

• • Backward Incompatible Change:. #23934 (hexiaoting).

New Feature

• Add an ability to reset custom setting to default and remove it from table's metadata. This will allow to rollback the change without knowing the system/config's default. Closes #14449. #17769 (xjewer).
• Add MaterializedPostgreSQL table engine and database engine. Database engine allows to replicate a whole database or any subset of database tables. #20470 (Kseniia Sumarokova).
• Adding support for a part of SQLJSON standard. #24148 (l1tsolaiki).
• Collect common system metrics (in system.asynchronous_metrics and system.asynchronous_metric_log) about CPU usage, disk usage, memory usage, IO, network, files, load average, CPU frequencies, thermal sensors, EDAC counters, system uptime; also added metrics about the scheduling jitter and the time spent collecting the metrics. It works like atop in ClickHouse and allows to get monitoring data even if you have no additional tools installed. This closes #9430. #24416 (Yegor Levankov).
• Add support DISTINCT ON (columns) expression, close #25404. #25589 (Zijie Lu).
• • Support Map type in mapAdd and mapSubtract functions * Support (U)Int128, U(Int256) types in mapAdd and mapSubtract functions. #25596 (Ildus Kurbangaliev).
• Add bin/unbin functions support. #25609 (zhaoyu).
• Introduce system.data_skipping_indices table containing information about existing data skipping indices. Closes #7659. #25693 (Dmitry Novik).
• in addition to https://github.com/ClickHouse/ClickHouse/pull/12073 add the FIRST keyword to the ADD INDEX command to be able to add index in the beginning of the indices list. #25904 (xjewer).
• Render pipelines as graphs in Web UI if EXPLAIN PIPELINE graph = 1 query is given. #26067 (Alexey Milovidov).
• Add new functions leftPad(), rightPad(), leftPadUTF8(), rightPadUTF8(). #26075 (Vitaly Baranov).

Performance Improvement

• Added option to compile aggregate functions if compile_aggregate_expressions settings is on. #24789 (Maksim Kita).

Improvement

• Database argument for StorageMerge support regular expression. This closes #776. #25064 (flynn).
• Allow extract non-string element as string using JSONExtract. This is for #25414. #25452 (Amos Bird).
• Support for dynamic reloading of config to change number of threads in pool for background jobs execution (merges, mutations, fetches). #25548 (Nikita Mikhaylov).
• Support TRUNCATE TABLE for StorageS3 and StorageHDFS. Closes #25530. #25550 (Kseniia Sumarokova).
• Make NetworkReceiveElapsedMicroseconds metric to correctly include the time spent waiting for data from the client to INSERT. This closes #9958. #25602 (Alexey Milovidov).
• Fix possible logical race condition between ALTER TABLE ... DETACH and background merges. #25605 (Azat Khuzhin).
• Support materialized and aliased columns in joins, close #13274. #25634 (Vladimir C).
• MaterializeMySQL now supports ENUM data type. #25676 (Storozhuk Kostiantyn).
• Cancel already running merges in partition on DROP PARTITION and TRUNCATE for ReplicatedMergeTree. Resolves #17151. #25684 (Alexander Tokmakov).
• Use Map data type for key-value dictionaries in system logs tables (system.query_log, system.query_thread_log, system.processes, system.opentelemetry_span_log). Virtual columns are created to support old queries. Closes #18698. Authors @hexiaoting, @sundy-li. #25773 (Maksim Kita).
• Fix inconsistent behaviour of GROUP BY constant on empty set. Closes #6842. #25786 (Kseniia Sumarokova).
• MySQL Engine now supports the exchange of column comments between MySQL and ClickHouse. #25795 (Storozhuk Kostiantyn).
• Fix "No available columns" for Merge() storage. #25801 (Azat Khuzhin).
• Allow to start clickhouse-client with unreadable working directory. #25817 (ianton-ru).
• Better handling of lost parts for ReplicatedMergeTree tables. Fixes rare inconsistencies in ReplicationQueue. Nothing should be visible to the user. Fixes #10368. #25820 (alesapin).
• Fix an extremely rare bug which can lead to intersecting parts after DROP PART or background deletion of an empty part. #25884 (alesapin).
• Convert history file from readline format to replxx format. #25888 (Azat Khuzhin).
• Support LowCardinality, Decimal and UUID for JSON extract. Closes #24606. #25900 (Kseniia Sumarokova).
• Add support for queries with a column named "null" (it must be specified in backticks or double quotes) and ON CLUSTER. This closes #24035. #25907 (Alexey Milovidov).
• Correctly throw exception on attempt to parse invalid Date. This closes #6481. #25909 (Alexey Milovidov).
• Allow parameters for parametric aggregate functions to be arbitrary constant expressions (e.g. 1 + 2), not just literals. This also allows to use query parameters (in parametrized queries like {param:UInt8}) for parameters of parametric aggregate functions. This closes #11607. #25910 (Alexey Milovidov).
• Allow quantiles* functions to work with aggregate_functions_null_for_empty. This closes #25892. #25919 (Alexey Milovidov).
• Make sudo service clickhouse-server start to work on systems with systemd like Centos 8. This closes #14298. This closes #17799. #25921 (Alexey Milovidov).
• Add support for unicode (e.g. Chinese, Cyrillic) components in Nested data types. This closes #25594. #25923 (Alexey Milovidov).
• Allow complex quoted identifiers of JOINed tables. This closes #17861. #25924 (Alexey Milovidov).
• Added setting optimize_move_to_prewhere_if_final. If query has FINAL, the optimization move_to_prewhere will be enabled only if both optimize_move_to_prewhere and optimize_move_to_prewhere_if_final are enabled. Closes #8684. #25940 (Kseniia Sumarokova).
• More instrumentation for network interaction: add counters for recv/send bytes; add gauges for recvs/sends. Added missing documentation. This closes #5897. #25962 (Alexey Milovidov).
• Web UI: if value looks like an URL, automatically generate a link. #25965 (Alexey Milovidov).
• Fix error with query SET SQL_SELECT_LIMIT in mysql protocol. Closes #17115. #25972 (Kseniia Sumarokova).
• Add support for argument of UUID type for empty and notEmpty functions. UUID is empty if it is all zeros (nil UUID). This closes #3446. #25974 (zhaoyu).
• Add support for argument of AggregateFunction type for bin and hex functions. #26094 (zhaoyu).
• For dictionary with complex key if complex key contains only one attribute allow to not wrap key expression in tuple for functions dictGet, dictHas. #26130 (Maksim Kita).

Bug Fix

• CAST from Date to DateTime (or DateTime64) was not using the timezone of the DateTime type. It can also affect the comparison between Date and DateTime. Inference of the common type for Date and DateTime also was not using the corresponding timezone. It affected the results of function if and array construction. Closes #24128. #24129 (Maksim Kita).
• Fix assertion in PREWHERE with non-uint8 type, close #19589. #25484 (Vladimir C).
• Fix ALTER MODIFY COLUMN of columns, which participates in TTL expressions. #25554 (Anton Popov).
• Fix slow dict join in some cases, close #24209. #25618 (Vladimir C).
• Allow StorageMerge to access tables with aliases. Closes #6051. #25694 (Kseniia Sumarokova).
• Fix bug in TTL with GROUP BY expression which refuses to execute TTL after first execution in part. #25743 (alesapin).
• Fix rare bug with DROP PART query for ReplicatedMergeTree tables which can lead to error message Unexpected merged part intersecting drop range. #25783 (alesapin).
• Fix ARM exception handling with non default page size. Fixes #25512. Fixes #25044. Fixes #24901. Fixes #23183. Fixes #20221. Fixes #19703. Fixes #19028. Fixes #18391. Fixes #18121. Fixes #17994. Fixes #12483. #25854 (Maksim Kita).
• Fix extremely long backoff for background tasks when the background pool is full. Fixes #25836. #25893 (alesapin).
• Fixed scram-sha-256 authentication for PostgreSQL engines. Closes #24516. #25906 (Kseniia Sumarokova).
• Fix crash on call dictGet() with bad arguments. #25913 (Vitaly Baranov).
• Fix possible deadlock during query profiler stack unwinding. Fixes #25968. #25970 (Maksim Kita).
• Fix formatting of type Map with integer keys to JSON. #25982 (Anton Popov).
• Fix wrong thread estimation for right subquery join in some cases. Close #24075. #26052 (Vladimir C).
• Fix rare server crash because of abort in ZooKeeper client. Fixes #25813. #26079 (alesapin).
• Fix throwing exception when iterate over non existing remote directory. #26087 (ianton-ru).
• Fix possible crash in pointInPolygon if the setting validate_polygons is turned off. #26113 (Alexey Milovidov).
• Fix joinGet with LowCarinality columns, close #25993. #26118 (Vladimir C).

Build/Testing/Packaging Improvement

• • Syntax update: changing 'is' to '==' when necessary. #25559 (MyroTk).
• Add new tests for checking access rights for columns used in filters (WHERE / PREWHERE / row policy) of the SELECT statement after changes in https://github.com/ClickHouse/ClickHouse/pull/24405. #25619 (Vitaly Baranov).
• Enabling all TestFlows modules and fixing some tests. #26011 (vzakaznikov).
• Disabling TestFlows LDAP module due to test fails. #26065 (vzakaznikov).

Other

• Add clickhouse-keeper-converter tool which allows converting zookeeper logs and snapshots into clickhouse-keeper snapshot format. #25428 (alesapin).

NO CL ENTRY

• NO CL ENTRY: '[ImgBot] Optimize images'. #26054 (imgbot[bot]).