ClickHouse/docs/changelogs/v21.5.1.6601-prestable.md

sidebar_position	sidebar_label
1	2022

2022 Changelog

ClickHouse release v21.5.1.6601-prestable FIXME as compared to v21.4.1.6422-prestable

Backward Incompatible Change

• Change comparison of integers and floating point numbers when integer is not exactly representable in the floating point data type. In new version comparison will return false as the rounding error will occur. Example: 9223372036854775808.0 != 9223372036854775808, because the number 9223372036854775808 is not representable as floating point number exactly (and 9223372036854775808.0 is rounded to 9223372036854776000.0). But in previous version the comparison will return as the numbers are equal, because if the floating point number 9223372036854776000.0 get converted back to UInt64, it will yield 9223372036854775808. For the reference, the Python programming language also treats these numbers as equal. But this behaviour was dependend on CPU model (different results on AMD64 and AArch64 for some out-of-range numbers), so we make the comparison more precise. It will treat int and float numbers equal only if int is represented in floating point type exactly. #22595 (Alexey Milovidov).

New Feature

• Implement function arrayFold(x1,...,xn,accum -> expression, array1,...,arrayn, init_accum) that applies the expression to each element of the array (or set of parallel arrays) and collect result in accumulator. #21589 (Dmitry Krylov).
• • Support Apple m1. #21639 (changvvb).
• Add a setting max_distributed_depth that limits the depth of recursive queries to Distributed tables. Closes #20229. #21942 (flynn).
• Table function, which allows to process files from s3 in parallel from many nodes in a specified cluster. #22012 (Nikita Mikhaylov).
• Support for replicas in MySQL/PostgreSQL table engine / table function. Added wrapper storage over MySQL / PostgreSQL storages to allow shards. Closes #20969. #22217 (Kseniia Sumarokova).
• Update paths to the catboost model configs in config reloading. #22434 (Kruglov Pavel).
• Add new setting non_replicated_deduplication_window for non-replicated MergeTree inserts deduplication. #22514 (alesapin).
• FlatDictionary added initial_array_size, max_array_size options. #22521 (Maksim Kita).
• Added ALTER TABLE ... FETCH PART ... query. It's similar to FETCH PARTITION, but fetches only one part. #22706 (jasong).
• Added Decimal256 type support in dictionaries. Closes #20979. #22960 (Maksim Kita).

Performance Improvement

• Add function alignment for possibly better performance. #21431 (Daniel Kutenin).
• Exclude values that does not belong to the shard from right part of IN section for distributed queries (under optimize_skip_unused_shards_rewrite_in, enabled by default, since it still requires optimize_skip_unused_shards). #21511 (Azat Khuzhin).
• Disable compression by default when interacting with localhost (with clickhouse-client or server to server with distributed queries) via native protocol. It may improve performance of some import/export operations. This closes #22234. #22237 (Alexey Milovidov).
• Improve performance of reading from ArrowStream input format for sources other then local file (e.g. URL). #22673 (nvartolomei).
• Improve performance of intDiv by dynamic dispatch for AVX2. This closes #22314. #23000 (Alexey Milovidov).

Improvement

• Support dynamic interserver credentials. #14113 (johnskopis).
• Add clickhouse-library-bridge for library dictionary source. Closes #9502. #21509 (Kseniia Sumarokova).
• Allow publishing Kafka errors to a virtual column of Kafka engine, controlled by the kafka_handle_error_mode setting. #21850 (fastio).
• Use nanodbc instead of Poco::ODBC. Closes #9678. Add support for DateTime64 and Decimal* for ODBC table engine. Closes #21961. Fixed issue with cyrillic text being truncated. Closes #16246. Added connection pools for odbc bridge. #21972 (Kseniia Sumarokova).
• Speeded up reading subset of columns from File-like table engine with internal file written in column oriented data formats (Parquet, Arrow and ORC) This closes #20129 Done by @keen-wolf. #22299 (Nikita Mikhaylov).
• Correctly check structure of async distributed blocks. #22325 (Azat Khuzhin).
• Make round function to behave consistently on non-x86_64 platforms. Rounding half to nearest even (Banker's rounding) is used. #22582 (Alexey Milovidov).
• Clear the rest of the screen and show cursor in clickhouse-client if previous program has left garbage in terminal. This closes #16518. #22634 (Alexey Milovidov).
• Allow to use CTE in VIEW definition. This closes #22491. #22657 (Amos Bird).
• Add metric to track how much time is spend during waiting for Buffer layer lock. #22725 (Azat Khuzhin).
• Allow RBAC row policy via postgresql protocol. Closes #22658. PostgreSQL protocol is enabled in configuration by default. #22755 (Kseniia Sumarokova).
• MaterializeMySQL (experimental feature). Make Clickhouse to be able to replicate MySQL databases containing views without failing. This is accomplished by ignoring the views. ... #22760 (Christian Frøystad).
• dateDiff now works with DateTime64 arguments (even for values outside of DateTime range) ... #22931 (Vasily Nemkov).
• Set background_fetches_pool_size to 8 that is better for production usage with frequent small insertions or slow ZooKeeper cluster. #22945 (Alexey Milovidov).
• Fix inactive_parts_to_throw_insert=0 with inactive_parts_to_delay_insert>0. #22947 (Azat Khuzhin).
• Respect max_part_removal_threads for ReplicatedMergeTree. #22971 (Azat Khuzhin).
• Fix an error handling in Poco HTTP Client for AWS. #22973 (Ernest Zaslavsky).
• When selecting from MergeTree table with NULL in WHERE condition, in rare cases, exception was thrown. This closes #20019. #22978 (Alexey Milovidov).
• Add ability to flush buffer only in background for StorageBuffer. #22986 (Azat Khuzhin).
• Add ability to run clickhouse-keeper with SSL. Config settings keeper_server.tcp_port_secure can be used for secure interaction between client and keeper-server. keeper_server.raft_configuration.secure can be used to enable internal secure communication between nodes. #22992 (alesapin).
• Increase max_uri_size (the maximum size of URL in HTTP interface) to 1 MiB by default. This closes #21197. #22997 (Alexey Milovidov).
• Do not perform optimize_skip_unused_shards for cluster with one node. #22999 (Azat Khuzhin).
• Raised the threshold on max number of matches in result of the function extractAllGroupsHorizontal. #23036 (Vasily Nemkov).
• Implement functions arrayHasAny, arrayHasAll, has, indexOf, countEqual for generic case when types of array elements are different. In previous versions the functions arrayHasAny, arrayHasAll returned false and has, indexOf, countEqual thrown exception. Also add support for Decimal and big integer types in functions has and similar. This closes #20272. #23044 (Alexey Milovidov).
• Fix memory tracking with min_bytes_to_use_mmap_io. #23211 (Azat Khuzhin).
• Make function unhex case insensitive for compatibility with MySQL. #23229 (Alexey Milovidov).

Bug Fix

• Fix very rare bug when quorum insert with quorum_parallel=1 is not really "quorum" because of deduplication. #18215 (filimonov).
• • Fix "unknown column" error for tables with Merge engine in queris with JOIN and aggregation. Closes #18368, close #22226. #21370 (Vladimir C).
• Check if table function view is used as a column. This complements https://github.com/ClickHouse/ClickHouse/pull/20350. #21465 (Amos Bird).
• Fix bug, which leads to underaggregation of data in case of enabled optimize_aggregation_in_order and many parts in table. Slightly improve performance of aggregation with enabled optimize_aggregation_in_order. #21889 (Anton Popov).
• Do not limit HTTP chunk size. Fixes #21907. #22322 (Ivan).
• Buffer overflow (on read) was possible in tokenbf_v1 full text index. The excessive bytes are not used but the read operation may lead to crash in rare cases. This closes #19233. #22421 (Alexey Milovidov).
• Fix ClickHouseDictionarySource configuration loop. Closes #14314. #22479 (Maksim Kita).
• Fix bug in partial merge join with LowCardinality. Close #22386, close #22388. #22510 (Vladimir C).
• Follow-up fix for #21936. Also fixes #22433. #22518 (Ivan).
• Fix deserialization of empty string without newline at end of TSV format. This closes #20244. Possible workaround without version update: set input_format_null_as_default to zero. It was zero in old versions. #22527 (Alexey Milovidov).
• Fix UB by unlocking the rwlock of the TinyLog from the same thread. #22560 (Azat Khuzhin).
• Avoid UB in *Log engines for rwlock unlock due to unlock from another thread. #22583 (Azat Khuzhin).
• Fix usage of function map in distributed queries. #22588 (foolchi).
• Try flush write buffer only if it is initialized. Fixes segfault when client closes connection very early #22579. #22591 (nvartolomei).
• Fixed a bug with unlimited wait for auxiliary AWS requests. #22594 (Vladimir Chebotarev).
• Fix LOGICAL_ERROR for Log with nested types w/o columns in the SELECT clause. #22654 (Azat Khuzhin).
• Fix wait for mutations on several replicas for ReplicatedMergeTree table engines. Previously, mutation/alter query may finish before mutation actually executed on other replicas. #22669 (alesapin).
• Fix possible hangs in zk requests in case of OOM exception. Fixes #22438. #22684 (Nikolai Kochetov).
• Fix approx total rows accounting for reverse reading from MergeTree. #22726 (Azat Khuzhin).
• • Revert "Move conditions from JOIN ON to WHERE" (ClickHouse/ClickHouse#19685), close #22399, close #21671. #22753 (Vladimir C).
• Fix pushdown of HAVING in case, when filter column is used in aggregation. #22763 (Anton Popov).
• LIVE VIEW (experimental feature). Fix possible hanging in concurrent DROP/CREATE of TEMPORARY LIVE VIEW in TemporaryLiveViewCleaner, see https://gist.github.com/vzakaznikov/0c03195960fc86b56bfe2bc73a90019e. #22858 (Vitaly Baranov).
• Fix bytes_allocated for sparse_hashed dictionaries. #22867 (Azat Khuzhin).
• Fixed a crash when using mannWhitneyUTest and rankCorr with window functions. This fixes #22728. #22876 (Nikita Mikhaylov).
• fixed formatDateTime() on DateTime64 and "%C" format specifier fixed toDateTime64() for large values and non-zero scale. ... #22937 (Vasily Nemkov).
• Fix usage of constant columns of type Map with nullable values. #22939 (Anton Popov).
• Simplify debian packages. This fixes #21698. #22976 (Alexey Milovidov).
• Fix error Cannot find column in ActionsDAG result which may happen if subquery uses untuple. Fixes #22290. #22991 (Nikolai Kochetov).
• Remove non-essential details from suggestions in clickhouse-client. This closes #22158. #23040 (Alexey Milovidov).
• Fixed Table .inner_id... doesn't exist error when selecting from Materialized View after detaching it from Atomic database and attaching back. #23047 (Alexander Tokmakov).
• Some values were formatted with alignment in center in table cells in Markdown format. Not anymore. #23096 (Alexey Milovidov).
• Server might fail to start if data_type_default_nullable setting is enabled in default profile, it's fixed. Fixes #22573. #23185 (Alexander Tokmakov).
• Fix missing whitespace in some exception messages about LowCardinality type. #23207 (Alexey Milovidov).
• Fixed missing semicolon in exception message. The user may find this exception message unpleasant to read. #23208 (Alexey Milovidov).
• Fix reading from ODBC when there are many long column names in a table. Closes #8853. #23215 (Kseniia Sumarokova).

Build/Testing/Packaging Improvement

• Add on-demand check for clickhouse Keeper. #22373 (alesapin).
• Disable incompatible libraries (platform specific typically) on ppc64le ... #22475 (Kfir Itzhak).
• Allow building with unbundled xz (lzma) using USE_INTERNAL_XZ_LIBRARY=OFF ... #22571 (Kfir Itzhak).
• Allow query profiling only on x86_64. See #15174#issuecomment-812954965 and #15638#issuecomment-703805337. This closes #15638. #22580 (Alexey Milovidov).
• Adjust some tests to output identical results on amd64 and aarch64 (qemu). The result was depending on implementation specific CPU behaviour. #22590 (Alexey Milovidov).
• Fix some tests on AArch64 platform. #22596 (Alexey Milovidov).
• Fix ClickHouse Keeper build for MacOS. #22860 (alesapin).

Other

• Fix some points from this comment https://github.com/ClickHouse/ClickHouse/pull/19516#issuecomment-782047840. #22323 (Ivan).

Build/Packaging/Testing Improvement

• Build jemalloc with support for heap profiling. #22834 (nvartolomei).

NO CL ENTRY

• NO CL ENTRY: 'Error message reads better'. #22983 (Igor O'sten).

NOT FOR CHANGELOG / INSIGNIFICANT

• Fix SIGSEGV by waiting servers thread pool #21318 (Azat Khuzhin).
• Better filter push down #22087 (Nikolai Kochetov).
• Better tests for finalize in nested writers #22110 (Vladimir Chebotarev).
• Replace all Context references with std::weak_ptr #22297 (Ivan).
• make some perf test queries more stable #22324 (Alexander Kuzmenkov).
• fix ccache broken by prlimit #22356 (Alexander Kuzmenkov).
• Remove old MSan suppressions (part 5) #22359 (Alexey Milovidov).
• Add test for copier #22441 (Nikita Mikhaylov).
• fix window frame offset check and add more tests #22459 (Alexander Kuzmenkov).
• FormatSettings null_as_default default value fix #22528 (Maksim Kita).
• Minor fixes in tests for AArch64 #22534 (Alexey Milovidov).
• Filter removed/renamed tests from ci-changed-files.txt for fuzzer #22542 (Azat Khuzhin).
• Try fix flaky test #22543 (Alexander Tokmakov).
• AppleClang compilation fix #22561 (Denis Glazachev).
• Fix assert in Arena when doing GROUP BY Array of Nothing of non-zero size. #22565 (Alexey Milovidov).
• Minor improvement in index deserialization #22586 (Alexey Milovidov).
• Fix flaky test after #22427 #22587 (Alexey Milovidov).
• Fix comments #22589 (Alexey Milovidov).
• Fix some uncaught exceptions (in SCOPE_EXIT) under memory pressure #22592 (Azat Khuzhin).
• Introduce IStorage::distributedWrite method for distributed INSERT SELECTS #22593 (Max Akhmedov).
• Fix flaky test 00816_long_concurrent_alter_column #22628 (Alexey Milovidov).
• FlatDictionary fix perf test #22629 (Maksim Kita).
• DirectDictionary dictGet multiple columns optimization #22630 (Maksim Kita).
• Better retries on ZK errors in sh tests #22633 (Alexey Milovidov).
• Add log_comment setting for DROP/CREATE DATABASE in clickhouse-test #22646 (Azat Khuzhin).
• Add retires for docker-compose pull in integration tests #22647 (Azat Khuzhin).
• Fix impossible invalid-read for system.errors accounting #22655 (Azat Khuzhin).
• Skip compiling xz if we're using system xz (unbundled) #22656 (Kfir Itzhak).
• Fix test 01294_create_settings_profile #22662 (Vitaly Baranov).
• Fix test 01039_row_policy_dcl #22663 (Anton Popov).
• Another attempt to enable pytest #22664 (Ivan).
• Fix random failures of tests that are using query_log #22666 (Anton Popov).
• fix build error 'always_inline' function might not be inlinable #22667 (flynn).
• Add bool type in postgres engine #22668 (Kseniia Sumarokova).
• Fix mutation killers tests #22670 (alesapin).
• Change Aggregatingmergetree to AggregatingMergeTree in docs #22687 (Kruglov Pavel).
• fix window functions with multiple input streams and no sorting #22704 (Alexander Kuzmenkov).
• Remove redundant fsync on coordination logs rotation #22707 (alesapin).
• Fix test 01702_system_query_log #22708 (Anton Popov).
• MemoryStorage sync comments and code #22721 (Maksim Kita).
• Fix potential segfault on Keeper startup #22743 (alesapin).
• Avoid using harmful function rand() #22744 (Amos Bird).
• Fix flaky hedged tests #22746 (Kruglov Pavel).
• add more messages when flushing the logs #22761 (Alexander Kuzmenkov).
• Moved BorrowedObjectPool to common #22764 (Maksim Kita).
• Functions ExternalDictionaries standardize exception throw #22821 (Maksim Kita).
• FileDictionarySource fix absolute file path #22822 (Maksim Kita).
• Small change in replicated database tests run #22826 (alesapin).
• Slightly improve logging messages for Distributed async sends #22829 (Azat Khuzhin).
• Fix what looks like a trivial mistake #22833 (Alexey Milovidov).
• Add a test for already fixed issue #22855 (Alexey Milovidov).
• DataTypeLowCardinality format tsv parsing issue #22863 (Maksim Kita).
• Updated MariaDB connector fix cmake #22865 (Maksim Kita).
• Prettify logs of integration tests #22868 (Anton Popov).
• Util memcpy-bench is built only when position independent code is disabled #22875 (Maksim Kita).
• Fix vanilla GCC compilation in macOS #22885 (Denis Glazachev).
• Better diagnostics for OOM in stress test #22894 (Alexey Milovidov).
• Dictionaries updated performance tests #22925 (Maksim Kita).
• IAggreagteFunction allocatesMemoryInArena removed default implementation #22938 (Maksim Kita).
• Fix flapping test_merge_tree_s3 test #22942 (ianton-ru).
• less flaky test #22944 (Alexander Kuzmenkov).
• Dictionaries standardize exceptions #22961 (Maksim Kita).
• StorageExternalDistributed arcadia fix #22962 (Maksim Kita).
• Check out of range values in FieldVisitorConverToNumber #22964 (Alexey Milovidov).
• Fix combinators with common prefix name (State and SimpleState) with libstdc++ #22977 (Azat Khuzhin).
• Fix arcadia #22982 (Kseniia Sumarokova).
• Fix excessive warning in StorageDistributed with cross-replication #22990 (Alexander Tokmakov).
• Update MergeTreeData.cpp Better error message. #22994 (Denny Crane).
• Fix assertion when filtering tables in StorageMerge #22998 (Alexander Tokmakov).
• Add a test for #2719 #23008 (Alexey Milovidov).
• Fix multi response in TestKeeper #23041 (Alexander Tokmakov).
• Improve hung check in Stress tests #23043 (Alexander Tokmakov).
• blog article about code review #23045 (Alexander Kuzmenkov).
• LibraryDictionary bridge library interface #23048 (Maksim Kita).
• Remove useless files #23049 (Alexey Milovidov).
• Upload keeper logs from stateless tests #23077 (alesapin).
• CI runner intergation tests logs update to tar.gz #23078 (Maksim Kita).
• Tiny logging improvements #23083 (Azat Khuzhin).
• Block all memory tracking limits in dtors/SCOPE_EXIT_*SAFE/tryLogCurrentException #23086 (Azat Khuzhin).
• jemalloc tuning #23088 (Azat Khuzhin).
• Rename strange tests #23111 (alesapin).
• Fix arcadia build S3 #23114 (Maksim Kita).
• Updated zlib #23153 (Maksim Kita).
• [RFC] Change logging from trace to debug for messages with rows/bytes #23160 (Azat Khuzhin).
• More verbose logs for debuging test failures with Replicated and Keeper #23161 (Alexander Tokmakov).
• Fix exception message for "parts_to_throw_insert" #23177 (madianjun).
• Fix flapping tests test_s3_zero_copy_replication #23184 (ianton-ru).
• Add a test for #14610 #23209 (Alexey Milovidov).
• Add test for fixed issue #9363 #23216 (Kseniia Sumarokova).
• Disable postgresql_port in perf tests #23218 (Azat Khuzhin).