ClickHouse/docs/changelogs/v20.12.2.1-stable.md

ClickHouse release v20.12.2.1-stable FIXME as compared to v20.11.1.5109-prestable

Backward Incompatible Change

• Accept user settings related to file formats (e.g. format_csv_delimiter) in the SETTINGS clause when creating a table that uses File engine, and use these settings in all INSERTs and SELECTs. The file format settings changed in the current user session, or in the SETTINGS clause of a DML query itself, no longer affect the query. #16591 (Alexander Kuzmenkov).
• Enable use_compact_format_in_distributed_parts_names by default (see the documentation for the reference). #16728 (Azat Khuzhin).

New Feature

• Added new ALTER UPDATE/DELETE IN PARTITION syntax. #13403 (Vladimir Chebotarev).
• Add StorageEmbeddedRocksdb Engine. #15073 (sundyli).
• Introduce the query ALTER TABLE ... DROP|DETACH PART 'part_name'. #15511 (nvartolomei).
• Make it possible to change the path to history file in clickhouse-client using the --history_file parameter. #15960 (Maksim Kita).
• Updated DateTime, DateTime64 formatting to accept string Date literal format. #16040 (Maksim Kita).
• Add setting aggregate_functions_null_for_empty, this option will rewrite all aggregate functions in a query, adding -OrNull suffix to them. fix 10273. #16123 (flynn).
• Add COLLATE support for Nullable, LowCardinality, Array and Tuple, where nested type is String. Also refactor the code associated with collations in ColumnString.cpp. #16273 (Kruglov Pavel).
• Possibility to distribute the merges between different replicas. Introduces the execute_merges_on_single_replica_time_threshold mergetree setting. #16424 (filimonov).
• add *.xz compression/decompression support.It enables using *.xz in file() function.This closes #8828. #16578 (Abi Palagashvili).
• Add new cmath functions: - acosh - asinh - atan2 - atanh - cosh - hypot - log1p - sinh. #16636 (Konstantin Malanchev).
• Add a possibility to input enum value as it's id in TSV and CSV formats by default. #16834 (Kruglov Pavel).
• New tcpPort() function returns TCP port listened by this server. #17134 (Ivan).

Performance Improvement

• Now we can safely prune partitions with exact match. Useful case: Suppose table is partitioned by intHash64(x) % 100 and the query has condition on intHash64(x) % 100 verbatim, not on x. #16253 (Amos Bird).
• Use Floyd-Rivest algorithm, it should be the best for the ClickHouse use case of partial sorting. Bechmarks are in https://github.com/danlark1/miniselect and here. #16825 (Daniel Kutenin).
• Backported in #17589: Fix performance of reading from Merge tables over huge number of MergeTree tables. Fixes #7748. #16988 (Anton Popov).

Improvement

• Add VIEW subquery description to EXPLAIN. Limit push down optimisation for VIEW. Add local replicas of Distributed to query plan. #14936 (Nikolai Kochetov).
• Made indexOf() use BloomFilter. #14977 (achimbab).
• Throw exception about right sync privileges when MySQL sync user has error privileges. #15977 (TCeason).
• Fix possible stack overflow if a loop of materialized views is created. This closes #15732. #16048 (Alexey Milovidov).
• Support SNI in https connections to remote resources. This will allow to connect to Cloudflare servers that require SNI. This fixes #10055. #16252 (Alexey Milovidov).
• Now ReplicatedMergeTree tree engines family uses a separate thread pool for replicated fetches. Size of the pool limited by setting background_fetches_pool_size which can be tuned with a server restart. The default value of the setting is 3 and it means that the maximum amount of parallel fetches is equal to 3 (and it allows to utilize 10G network). Fixes #520. #16390 (alesapin).
• Now, <auxiliary_zookeepers> configuration can be changed in config.xml and reloaded without server startup. #16627 (Amos Bird).
• Allow reinterpret between integers and floats of the same size. fix 16640. #16657 (flynn).
• Workaround for use S3 with nginx server as proxy. Nginx currenty does not accept urls with empty path like http://domain.com?delete, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, like http://domain.com/?delete. #16709 (ianton-ru).
• Remove empty directories for async INSERT at start of Distributed engine. #16729 (Azat Khuzhin).
• Usability improvement: better suggestions in syntax error message when CODEC expression is misplaced in CREATE TABLE query. This fixes #12493. #16768 (Alexey Milovidov).
• Better exception message when configuration for distributed DDL is absent. This fixes #5075. #16769 (Nikita Mikhaylov).
• Apply use_compact_format_in_distributed_parts_names for each INSERT (with internal_replication). #16788 (Azat Khuzhin).
• Server refused to startup with exception message if wrong config is given (metric_log.collect_interval_milliseconds is missing). #16815 (Ivan).
• Add cutToFirstSignificantSubdomainWithWWW(). #16845 (Azat Khuzhin).
• Throw an informative error message when doing ATTACH/DETACH TABLE . Before this PR, detach table <dict> works but leads to an ill-formed in-memory metadata. #16885 (Amos Bird).
• Remove empty parts after they were pruned by TTL, mutation, or collapsing merge algorithm. #16895 (Anton Popov).
• Make it possible to connect to clickhouse-server secure endpoint which requires SNI. This is possible when clickhouse-server is hosted behind TLS proxy. #16938 (filimonov).
• Set default host and port parameters for SOURCE(CLICKHOUSE(...)) to current instance and set default user value to 'default'. #16997 (Vladimir C).
• Add ability to output all rows as a JSON array in the JSONEachRow format, controlled by the output_format_json_array_of_rows setting. #17152 (Alexander Kuzmenkov).
• Allow formatting named tuples as JSON objects when using JSON input/output formats, controlled by the output_format_json_named_tuples_as_objects setting, disabled by default. #17175 (Alexander Kuzmenkov).
• Correct grammar in error message in JSONEachRow, JSONCompactEachRow, and RegexpRow input formats. #17205 (nico piderman).

Bug Fix

• Backported in #17620: Throw error when use ColumnTransformer replace non exist column. #16183 (hexiaoting).
• fixes #16574 fixes #16231 fix remote query failure when using 'if' suffix aggregate function. #16610 (Winter Zhang).
• Fixed #16081. #16613 (Nikita Mikhaylov).
• This will fix optimize_read_in_order/optimize_aggregation_in_order with max_threads>0 and expression in ORDER BY. #16637 (Azat Khuzhin).
• Fixed uncontrolled growth of TDigest. #16680 (hrissan).
• Backported in #17315: Return number of affected rows for INSERT queries via MySQL protocol. Previously ClickHouse used to always return 0, it's fixed. Fixes #16605. #16715 (Winter Zhang).
• Turn off parallel parsing when there is no enough memory for all threads to work simultaneously. Also there could be exceptions like "Memory limit exceeded" when somebody will try to insert extremely huge rows (> min_chunk_bytes_for_parallel_parsing), because each piece to parse has to be independent set of strings (one or more). #16721 (Nikita Mikhaylov).
• Fix IN operator over several columns and tuples with enabled transform_null_in setting. Fixes #15310. #16722 (Anton Popov).
• Mask password in data_path in the system.distribution_queue. #16727 (Azat Khuzhin).
• Not for changelog. #16757 (Alexander Tokmakov).
• Backported in #17520: Fix optimize_trivial_count_query with partition predicate. #16767 (Azat Khuzhin).
• If no memory can be allocated while writing table metadata on disk, broken metadata file can be written. #16772 (Alexey Milovidov).
• Fix crash when using any without any arguments. This is for #16803 . cc @azat. #16826 (Amos Bird).
• Abort multipart upload if no data was written to WriteBufferFromS3. #16840 (Pavel Kovalenko).
• Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes #15301. This fixes #13098. #16846 (Alexey Milovidov).
• Prevent clickhouse server crashes when using TimeSeriesGroupSum. #16865 (filimonov).
• Backported in #17340: TODO. #16866 (Alexander Tokmakov).
• Fix possible error Illegal type of argument for queries with ORDER BY. Fixes #16580. #16928 (Nikolai Kochetov).
• Install script should always create subdirs in config folders. This is only relevant for Docker build with custom config. #16936 (filimonov).
• Backported in #17586: Fix optimization of group by with enabled setting optimize_aggregators_of_group_by_keys and joins. Fixes #12604. #16951 (Anton Popov).
• Backported in #17593: Fix order by optimization with monotonous functions. Fixes #16107. #16956 (Anton Popov).
• Blame info was not calculated correctly in clickhouse-git-import. #16959 (Alexey Milovidov).
• Fix possible server crash after ALTER TABLE ... MODIFY COLUMN ... NewType when SELECT have WHERE expression on altering column and alter doesn't finished yet. #16968 (Amos Bird).
• Reresolve the IP of the format_avro_schema_registry_url in case of errors. #16985 (filimonov).
• Fixed wrong result in big integers (128, 256 bit) when casting from double. #16986 (Mike Kot).
• Backported in #17338: Fix Merge(Distributed()) with JOIN. #16993 (Azat Khuzhin).
• Backported in #17746: - Fix optimize_distributed_group_by_sharding_key for query with OFFSET only. #16996 (Azat Khuzhin).
• Avoid unnecessary network errors for remote queries which may be cancelled while execution, like queries with LIMIT. #17006 (Azat Khuzhin).
• Fix LLVM's libunwind in the case when CFA register is RAX. This is the bug in LLVM's libunwind. We already have workarounds for this bug. #17046 (Alexey Milovidov).
• Backported in #17429: Bug fix for funciton fuzzBits, related issue: #16980. #17051 (hexiaoting).
• Fixed crash on CREATE TABLE ... AS some_table query when some_table was created AS table_function() Fixes #16944. #17072 (Alexander Tokmakov).
• Fix bug when ON CLUSTER queries may hang forever for non-leader ReplicatedMergeTreeTables. #17089 (alesapin).
• fixes #16923 fixes #15883 Fix MaterializeMySQL SYNC failure when the modify MySQL binlog_checksum. #17091 (Winter Zhang).
• Improve adaptive index granularity calculation when incoming blocks of data differ in bytes size a lot. #17120 (alesapin).
• Fix ColumnConst comparison which leads to crash. This fixed #17088 . #17135 (Amos Bird).
• Backported in #17560: Fix possible wrong index analysis when the types of the index comparison are different. This fixes #17122. #17145 (Amos Bird).
• Backported in #17394: Fix #15235. When clickhouse-copier handle non-partitioned table, throws segfault error. #17248 (Qi Chen).
• Backported in #17406: Fix set index invalidation when there are const columns in the subquery. This fixes #17246 . #17249 (Amos Bird).
• Backported in #17422: Fix possible Unexpected packet Data received from client error for Distributed queries with LIMIT. #17254 (Azat Khuzhin).
• fix toInt256(inf) stack overflow. close #17235. #17257 (flynn).
• Backported in #17489: Fix crash while reading from JOIN table with LowCardinality types. Fixes #17228. #17397 (Nikolai Kochetov).
• Backported in #17451: Fixed high CPU usage in background tasks of *MergeTree tables. #17416 (Alexander Tokmakov).
• Backported in #17493: Fix duplicates after DISTINCT which were possible because of incorrect optimization. Fixes #17294. #17296 (li chengxiang). #17439 (Nikolai Kochetov).
• Backported in #17523: Fix ORDER BY with enabled setting optimize_redundant_functions_in_order_by. #17471 (Anton Popov).
• Backported in #17533: Fix bug when mark cache size was underestimated by clickhouse. It may happen when there are a lot of tiny files with marks. #17496 (alesapin).
• Backported in #17625: Fix alter query hang when the corresponding mutation was killed on the different replica. Fixes #16953. #17499 (alesapin).
• Backported in #17611: Fix the issue when server can stop accepting connections in very rare cases. #17542 (Alexey Milovidov).
• Backported in #17607: When clickhouse-client is used in interactive mode with multiline queries, single line comment was erronously extended till the end of query. This fixes #13654. #17565 (Alexey Milovidov).
• Backported in #17683: Exception fmt::v7::format_error can be logged in background for MergeTree tables. This fixes #17613. #17615 (Alexey Milovidov).
• Backported in #17697: In might be determined incorrectly if cluster is circular- (cross-) replicated or not when executing ON CLUSTER query due to race condition when pool_size > 1. It's fixed. #17640 (Alexander Tokmakov).
• Backported in #17727: Fixed Function not implemented error when executing RENAME query in Atomic database with ClickHouse running on Windows Subsystem for Linux. Fixes #17661. #17664 (Alexander Tokmakov).
• Backported in #17784: Fixed problem when ClickHouse fails to resume connection to MySQL servers. #17681 (Alexander Kazakov).
• Backported in #17780: Exception message about max table size to drop was displayed incorrectly. #17764 (Alexey Milovidov).
• Backported in #17817: Do not restore parts from WAL if in_memory_parts_enable_wal is disabled. #17802 (detailyang).
• Backported in #17842: fix incorrect initialize max_compress_block_size of MergeTreeWriterSettings with min_compress_block_size. #17833 (flynn).

Build/Testing/Packaging Improvement

• Fix UBSan report when trying to convert infinite floating point number to integer. This closes #14190. #16677 (Alexey Milovidov).
• Fix UBSan report in cache dictionaries. This closes #12641. #16763 (Alexey Milovidov).
• Do not instrument 3rd-party libraries with UBSan. #16764 (Alexey Milovidov).
• Fix UBSan report in Poco. This closes #12719. #16765 (Alexey Milovidov).
• fix query_db_generate build error. #16859 (hhhhhzhen).
• Try fix fasttest submodule clone #16132 https://clickhouse-test-reports.s3.yandex.net/16132/ad569f6d1bd2ce545db280daf7fbb9b8335de87b/fast_test.html#fail1. #16908 (Winter Zhang).
• Fixing unstable test in tests/testflows/ldap/external_user_directory/tests/authentications.py. #17161 (vzakaznikov).
• bump up rocksdb version to v6.14.5. #17179 (sundyli).
• Update embedded timezone data to version 2020d (also update cctz to the latest master). #17204 (filimonov).
• Improvements in coverage building images. #17233 (alesapin).
• std::logic_error is used at line 294 of base/common/StringRef.h, so the appropriate <stdexcept> header is required. #17256 (Matwey V. Kornilov).

NO CL ENTRY

• NO CL ENTRY: 'sync MySQL DDL atomicly'. #16704 (TCeason).
• NO CL ENTRY: 'RBAC Testflows - Server log intrumentation for debug and new ALTER tests'. #16719 (MyroTk).
• NO CL ENTRY: 'Enabling existing testflows RBAC tests.'. #16773 (MyroTk).
• NO CL ENTRY: 'Bump protobuf from 3.13.0 to 3.14.0 in /docs/tools'. #17056 (dependabot-preview[bot]).
• NO CL ENTRY: 'Fixed a problem with the translation of the document'. #17218 (qianmoQ).