ClickHouse/docs/changelogs/v21.5.1.6601-prestable.md

ClickHouse release v21.5.1.6601-prestable FIXME as compared to v21.4.1.6422-prestable

Backward Incompatible Change

• Change comparison of integers and floating point numbers when integer is not exactly representable in the floating point data type. In new version comparison will return false as the rounding error will occur. Example: 9223372036854775808.0 != 9223372036854775808, because the number 9223372036854775808 is not representable as floating point number exactly (and 9223372036854775808.0 is rounded to 9223372036854776000.0). But in previous version the comparison will return as the numbers are equal, because if the floating point number 9223372036854776000.0 get converted back to UInt64, it will yield 9223372036854775808. For the reference, the Python programming language also treats these numbers as equal. But this behaviour was dependend on CPU model (different results on AMD64 and AArch64 for some out-of-range numbers), so we make the comparison more precise. It will treat int and float numbers equal only if int is represented in floating point type exactly. #22595 (Alexey Milovidov).

New Feature

• Implement function arrayFold(x1,...,xn,accum -> expression, array1,...,arrayn, init_accum) that applies the expression to each element of the array (or set of parallel arrays) and collect result in accumulator. #21589 (Dmitry Krylov).
• • Support Apple m1. #21639 (changvvb).
• Add a setting max_distributed_depth that limits the depth of recursive queries to Distributed tables. Closes #20229. #21942 (flynn).
• Table function, which allows to process files from s3 in parallel from many nodes in a specified cluster. #22012 (Nikita Mikhaylov).
• Support for replicas in MySQL/PostgreSQL table engine / table function. Added wrapper storage over MySQL / PostgreSQL storages to allow shards. Closes #20969. #22217 (Kseniia Sumarokova).
• Update paths to the catboost model configs in config reloading. #22434 (Kruglov Pavel).
• Add new setting non_replicated_deduplication_window for non-replicated MergeTree inserts deduplication. #22514 (alesapin).
• FlatDictionary added initial_array_size, max_array_size options. #22521 (Maksim Kita).
• Added ALTER TABLE ... FETCH PART ... query. It's similar to FETCH PARTITION, but fetches only one part. #22706 (jasong).
• Added Decimal256 type support in dictionaries. Closes #20979. #22960 (Maksim Kita).

Performance Improvement

• Add function alignment for possibly better performance. #21431 (Daniel Kutenin).
• Exclude values that does not belong to the shard from right part of IN section for distributed queries (under optimize_skip_unused_shards_rewrite_in, enabled by default, since it still requires optimize_skip_unused_shards). #21511 (Azat Khuzhin).
• Disable compression by default when interacting with localhost (with clickhouse-client or server to server with distributed queries) via native protocol. It may improve performance of some import/export operations. This closes #22234. #22237 (Alexey Milovidov).
• Improve performance of reading from ArrowStream input format for sources other then local file (e.g. URL). #22673 (nvartolomei).
• Improve performance of intDiv by dynamic dispatch for AVX2. This closes #22314. #23000 (Alexey Milovidov).

Improvement

• Support dynamic interserver credentials. #14113 (johnskopis).
• Add clickhouse-library-bridge for library dictionary source. Closes #9502. #21509 (Kseniia Sumarokova).
• Allow publishing Kafka errors to a virtual column of Kafka engine, controlled by the kafka_handle_error_mode setting. #21850 (fastio).
• Use nanodbc instead of Poco::ODBC. Closes #9678. Add support for DateTime64 and Decimal* for ODBC table engine. Closes #21961. Fixed issue with cyrillic text being truncated. Closes #16246. Added connection pools for odbc bridge. #21972 (Kseniia Sumarokova).
• Speeded up reading subset of columns from File-like table engine with internal file written in column oriented data formats (Parquet, Arrow and ORC) This closes #20129 Done by @keen-wolf. #22299 (Nikita Mikhaylov).
• Correctly check structure of async distributed blocks. #22325 (Azat Khuzhin).
• Make round function to behave consistently on non-x86_64 platforms. Rounding half to nearest even (Banker's rounding) is used. #22582 (Alexey Milovidov).
• Clear the rest of the screen and show cursor in clickhouse-client if previous program has left garbage in terminal. This closes #16518. #22634 (Alexey Milovidov).
• Allow to use CTE in VIEW definition. This closes #22491. #22657 (Amos Bird).
• Add metric to track how much time is spend during waiting for Buffer layer lock. #22725 (Azat Khuzhin).
• Allow RBAC row policy via postgresql protocol. Closes #22658. PostgreSQL protocol is enabled in configuration by default. #22755 (Kseniia Sumarokova).
• MaterializeMySQL (experimental feature). Make Clickhouse to be able to replicate MySQL databases containing views without failing. This is accomplished by ignoring the views. ... #22760 (Christian Frøystad).
• dateDiff now works with DateTime64 arguments (even for values outside of DateTime range) ... #22931 (Vasily Nemkov).
• Set background_fetches_pool_size to 8 that is better for production usage with frequent small insertions or slow ZooKeeper cluster. #22945 (Alexey Milovidov).
• Fix inactive_parts_to_throw_insert=0 with inactive_parts_to_delay_insert>0. #22947 (Azat Khuzhin).
• Respect max_part_removal_threads for ReplicatedMergeTree. #22971 (Azat Khuzhin).
• Fix an error handling in Poco HTTP Client for AWS. #22973 (Ernest Zaslavsky).
• When selecting from MergeTree table with NULL in WHERE condition, in rare cases, exception was thrown. This closes #20019. #22978 (Alexey Milovidov).
• Add ability to flush buffer only in background for StorageBuffer. #22986 (Azat Khuzhin).
• Add ability to run clickhouse-keeper with SSL. Config settings keeper_server.tcp_port_secure can be used for secure interaction between client and keeper-server. keeper_server.raft_configuration.secure can be used to enable internal secure communication between nodes. #22992 (alesapin).
• Increase max_uri_size (the maximum size of URL in HTTP interface) to 1 MiB by default. This closes #21197. #22997 (Alexey Milovidov).
• Do not perform optimize_skip_unused_shards for cluster with one node. #22999 (Azat Khuzhin).
• Raised the threshold on max number of matches in result of the function extractAllGroupsHorizontal. #23036 (Vasily Nemkov).
• Implement functions arrayHasAny, arrayHasAll, has, indexOf, countEqual for generic case when types of array elements are different. In previous versions the functions arrayHasAny, arrayHasAll returned false and has, indexOf, countEqual thrown exception. Also add support for Decimal and big integer types in functions has and similar. This closes #20272. #23044 (Alexey Milovidov).
• Fix memory tracking with min_bytes_to_use_mmap_io. #23211 (Azat Khuzhin).
• Make function unhex case insensitive for compatibility with MySQL. #23229 (Alexey Milovidov).

Bug Fix

• Fix very rare bug when quorum insert with quorum_parallel=1 is not really "quorum" because of deduplication. #18215 (filimonov).
• • Fix "unknown column" error for tables with Merge engine in queris with JOIN and aggregation. Closes #18368, close #22226. #21370 (Vladimir C).
• Check if table function view is used as a column. This complements https://github.com/ClickHouse/ClickHouse/pull/20350. #21465 (Amos Bird).
• Fix bug, which leads to underaggregation of data in case of enabled optimize_aggregation_in_order and many parts in table. Slightly improve performance of aggregation with enabled optimize_aggregation_in_order. #21889 (Anton Popov).
• Do not limit HTTP chunk size. Fixes #21907. #22322 (Ivan).
• Buffer overflow (on read) was possible in tokenbf_v1 full text index. The excessive bytes are not used but the read operation may lead to crash in rare cases. This closes #19233. #22421 (Alexey Milovidov).
• Fix ClickHouseDictionarySource configuration loop. Closes #14314. #22479 (Maksim Kita).
• Fix bug in partial merge join with LowCardinality. Close #22386, close #22388. #22510 (Vladimir C).
• Follow-up fix for #21936. Also fixes #22433. #22518 (Ivan).
• Fix deserialization of empty string without newline at end of TSV format. This closes #20244. Possible workaround without version update: set input_format_null_as_default to zero. It was zero in old versions. #22527 (Alexey Milovidov).
• Fix UB by unlocking the rwlock of the TinyLog from the same thread. #22560 (Azat Khuzhin).
• Avoid UB in *Log engines for rwlock unlock due to unlock from another thread. #22583 (Azat Khuzhin).
• Fix usage of function map in distributed queries. #22588 (foolchi).
• Try flush write buffer only if it is initialized. Fixes segfault when client closes connection very early #22579. #22591 (nvartolomei).
• Fixed a bug with unlimited wait for auxiliary AWS requests. #22594 (Vladimir Chebotarev).
• Fix LOGICAL_ERROR for Log with nested types w/o columns in the SELECT clause. #22654 (Azat Khuzhin).
• Fix wait for mutations on several replicas for ReplicatedMergeTree table engines. Previously, mutation/alter query may finish before mutation actually executed on other replicas. #22669 (alesapin).
• Fix possible hangs in zk requests in case of OOM exception. Fixes #22438. #22684 (Nikolai Kochetov).
• Fix approx total rows accounting for reverse reading from MergeTree. #22726 (Azat Khuzhin).
• • Revert "Move conditions from JOIN ON to WHERE" (ClickHouse/ClickHouse#19685), close #22399, close #21671. #22753 (Vladimir C).
• Fix pushdown of HAVING in case, when filter column is used in aggregation. #22763 (Anton Popov).
• LIVE VIEW (experimental feature). Fix possible hanging in concurrent DROP/CREATE of TEMPORARY LIVE VIEW in TemporaryLiveViewCleaner, see https://gist.github.com/vzakaznikov/0c03195960fc86b56bfe2bc73a90019e. #22858 (Vitaly Baranov).
• Fix bytes_allocated for sparse_hashed dictionaries. #22867 (Azat Khuzhin).
• Fixed a crash when using mannWhitneyUTest and rankCorr with window functions. This fixes #22728. #22876 (Nikita Mikhaylov).
• fixed formatDateTime() on DateTime64 and "%C" format specifier fixed toDateTime64() for large values and non-zero scale. ... #22937 (Vasily Nemkov).
• Fix usage of constant columns of type Map with nullable values. #22939 (Anton Popov).
• Simplify debian packages. This fixes #21698. #22976 (Alexey Milovidov).
• Fix error Cannot find column in ActionsDAG result which may happen if subquery uses untuple. Fixes #22290. #22991 (Nikolai Kochetov).
• Remove non-essential details from suggestions in clickhouse-client. This closes #22158. #23040 (Alexey Milovidov).
• Fixed Table .inner_id... doesn't exist error when selecting from Materialized View after detaching it from Atomic database and attaching back. #23047 (Alexander Tokmakov).
• Some values were formatted with alignment in center in table cells in Markdown format. Not anymore. #23096 (Alexey Milovidov).
• Server might fail to start if data_type_default_nullable setting is enabled in default profile, it's fixed. Fixes #22573. #23185 (Alexander Tokmakov).
• Fix missing whitespace in some exception messages about LowCardinality type. #23207 (Alexey Milovidov).
• Fixed missing semicolon in exception message. The user may find this exception message unpleasant to read. #23208 (Alexey Milovidov).
• Fix reading from ODBC when there are many long column names in a table. Closes #8853. #23215 (Kseniia Sumarokova).

Build/Testing/Packaging Improvement

• Add on-demand check for clickhouse Keeper. #22373 (alesapin).
• Disable incompatible libraries (platform specific typically) on ppc64le ... #22475 (Kfir Itzhak).
• Allow building with unbundled xz (lzma) using USE_INTERNAL_XZ_LIBRARY=OFF ... #22571 (Kfir Itzhak).
• Allow query profiling only on x86_64. See #15174#issuecomment-812954965 and #15638#issuecomment-703805337. This closes #15638. #22580 (Alexey Milovidov).
• Adjust some tests to output identical results on amd64 and aarch64 (qemu). The result was depending on implementation specific CPU behaviour. #22590 (Alexey Milovidov).
• Fix some tests on AArch64 platform. #22596 (Alexey Milovidov).
• Fix ClickHouse Keeper build for MacOS. #22860 (alesapin).

Other

• Fix some points from this comment https://github.com/ClickHouse/ClickHouse/pull/19516#issuecomment-782047840. #22323 (Ivan).

Build/Packaging/Testing Improvement

• Build jemalloc with support for heap profiling. #22834 (nvartolomei).

NO CL ENTRY

• NO CL ENTRY: 'Error message reads better'. #22983 (Igor O'sten).