ClickHouse/docs/changelogs/v20.10.2.20-stable.md

ClickHouse release v20.10.2.20-stable FIXME as compared to v20.9.1.4585-prestable

Backward Incompatible Change

• Add support for nested multiline comments /* comment /* comment */ */ in SQL. This conforms to the SQL standard. #14655 (Alexey Milovidov).
• Change default value of format_regexp_escaping_rule setting (it's related to Regexp format) to Raw (it means - read whole subpattern as a value) to make the behaviour more like to what users expect. #15426 (Alexey Milovidov).
• Make multiple_joins_rewriter_version obsolete. Remove first version of joins rewriter. #15472 (Artem Zuikov).

New Feature

• Add new feature: format LineAsString that accepts a sequence of line separated by newlines, spaces and/or commas. #14703 (Nikita Mikhaylov).
• Added formatReadableQuantity function. It is useful for reading big numbers by human. #14725 (Artem Hnilov).
• Add the ability to remove column properties and table TTLs. Introduced queries ALTER TABLE MODIFY COLUMN col_name REMOVE what_to_remove and ALTER TABLE REMOVE TTL. Both operations are lightweight and executed at the metadata level. #14742 (alesapin).
• Introduce event_time_microseconds field to system.text_log, system.trace_log, system.query_log and system.query_thread_log tables. #14760 (Bharat Nallan).
• Now we support WITH <identifier> AS (subquery) ... to introduce named subqueries in the query context. This closes #2416. This closes #4967. #14771 (Amos Bird).
• Allow to omit arguments for Replicated table engine if defaults are specified in config. #14791 (vxider).
• Add table function null('structure'). #14797 (vxider).
• Added query obfuscation tool. It allows to share more queries for better testing. This closes #15268. #15321 (Alexey Milovidov).
• Added format RawBLOB. It is intended for input or output a single value without any escaping and delimiters. This closes #15349. #15364 (Alexey Milovidov).
• fix 15350. #15443 (flynn).
• Introduce enable_global_with_statement setting which propagates the first select's WITH statements to other select queries at the same level, and makes aliases in WITH statements visible to subqueries. #15451 (Amos Bird).
• Add the reinterpretAsUUID function that allows to convert a big-endian byte string to UUID. #15480 (Alexander Kuzmenkov).
• Add parallel quorum inserts. This closes #15601. #15601 (Latysheva Alexandra).

Performance Improvement

• Improve performance of 256-bit bytes using (u)int64_t as base type for wide integers. Original wide integers use 8-bit types as base. #14859 (Artem Zuikov).
• Only mlock code segment when starting clickhouse-server. In previous versions, all mapped regions were locked in memory, including debug info. Debug info is usually splitted to a separate file but if it isn't, it led to +2..3 GiB memory usage. #14929 (Alexey Milovidov).
• We used to choose fixed key method to group by one fixed string. It's unnecessary since we have StringHashTable which do the similar packedFix optimization for FixedString columns. And we should use low_cardinality_key_fixed_string if possible. #15034 (Amos Bird).
• Fix DateTime <op> DateTime mistakenly choosing the slow generic implementation. This fixes #15153 . #15178 (Amos Bird).
• Use one S3 DeleteObjects request instead of multiple DeleteObject in cycle. No any functionality changes, so covered by existing tests like integration/test_log_family_s3. #15238 (ianton-ru).
• Faster 256-bit multiplication. #15418 (Artem Zuikov).
• Improve quantileTDigest performance. This fixes #2668. #15542 (Kruglov Pavel).
• Explicitly use a temporary disk to store vertical merge temporary data. #15639 (Pervakov Grigorii).

Improvement

• Add QueryMemoryLimitExceeded event. This closes #14589. #14647 (fastio).
• Fixed the backward-incompatible change by providing the options to build without debug info for functions. #14657 (Mike Kot).
• dynamic reload zookeeper config. #14678 (sundyli).
• Allow parallel execution of distributed DDL. #14684 (Azat Khuzhin).
• Fix potential memory leak caused by zookeeper exists watch. #14693 (hustnn).
• Fixed "Packet payload is not fully read" error in MaterializeMySQL database engine. #14696 (BohuTANG).
• Fix crash in bitShiftLeft() when called with negative big integer. #14697 (Artem Zuikov).
• Add merge_algorithm to system.merges table to improve merging inspections. #14705 (Amos Bird).
• Less unneded code generated by DecimalBinaryOperation template in FunctionBinaryArithmetic. #14743 (Artem Zuikov).
• Now columns can be used to wrap over a list of columns and apply column transformers afterwards. #14775 (Amos Bird).
• Support for disabling persistency for StorageJoin and StorageSet, this feature is controlled by setting disable_set_and_join_persistency. And this PR solved issue #6318. #14776 (vxider).
• Construct query_start_time and query_start_time_microseconds from the same timespec. #14831 (Bharat Nallan).
• Allow using multi-volume storage configuration in storage Distributed. #14839 (Pavel Kovalenko).
• Show subqueries for SET and JOIN in EXPLAIN result. #14856 (Nikolai Kochetov).
• Provide a load_balancing_first_offset query setting to explicitly state what the first replica is. It's used together with FIRST_OR_RANDOM load balancing strategy, which allows to control replicas workload. #14867 (Amos Bird).
• Fixed excessive settings constraint violation when running SELECT with SETTINGS from a distributed table. #14876 (Amos Bird).
• Allow to drop Replicated table if previous drop attempt was failed due to ZooKeeper session expiration. This fixes #11891. #14926 (Alexey Milovidov).
• Avoid deadlock when executing INSERT SELECT into itself from a table with TinyLog or Log table engines. This closes #6802. #14962 (Alexey Milovidov).
• Ignore key constraints when doing mutations. Without this pr, it's not possible to do mutations when force_index_by_date = 1 or force_primary_key = 1. #14973 (Amos Bird).
• Add option to disable TTL move on data part insert. #15000 (Pavel Kovalenko).
• Enable Atomic database engine by default. #15003 (Alexander Tokmakov).
• Proper exception message for wrong number of arguments of CAST. This closes #13992. #15029 (Alexey Milovidov).
• Add the ability to specify specialized codecs like Delta, T64, etc. for columns with subtypes. Implements #12551, fixes #11397, fixes #4609. #15089 (alesapin).
• Added optimize setting to EXPLAIN PLAN query. If enabled, query plan level optimisations are applied. Enabled by default. #15201 (Nikolai Kochetov).
• Do not allow connections to ClickHouse server until all scripts in /docker-entrypoint-initdb.d/ are executed. #15244 (Aleksei Kozharin).
• fix 15264. #15285 (flynn).
• Unfold {database}, {table} and {uuid} macros in zookeeper_path on replicated table creation. Do not allow RENAME TABLE if it may break zookeeper_path after server restart. Fixes #6917. #15348 (Alexander Tokmakov).
• Add support for "Raw" column format for Regexp format. It allows to simply extract subpatterns as a whole without any escaping rules. #15363 (Alexey Milovidov).
• Now it's possible to change the type of version column for VersionedCollapsingMergeTree with ALTER query. #15442 (alesapin).
• Wait for DROP/DETACH TABLE to actually finish if NO DELAY or SYNC is specified for Atomic database. #15448 (Alexander Tokmakov).
• Pass through *_for_user settings via Distributed with cluster-secure. #15551 (Azat Khuzhin).
• Use experimental pass manager by default. #15608 (Daniel Kutenin).
• Implement force_data_skipping_indices setting. #15642 (Azat Khuzhin).
• Backported in #16102: Now it's allowed to execute ALTER ... ON CLUSTER queries regardless of the <internal_replication> setting in cluster config. #16075 (alesapin).

Bug Fix

• Fix wrong Decimal multiplication result caused wrong decimal scale of result column. #14603 (Artem Zuikov).
• Stuff the query into ASTFunction's argument list so that we don't break the presumptions of some AST visitors. This fixes #14608. #14611 (Amos Bird).
• Fix bug when ALTER UPDATE mutation with Nullable column in assignment expression and constant value (like UPDATE x = 42) leads to incorrect value in column or segfault. Fixes #13634, #14045. #14646 (alesapin).
• Fixed missed default database name in metadata of materialized view when executing ALTER ... MODIFY QUERY. #14664 (Alexander Tokmakov).
• Replace column transformer should replace identifiers with cloned ASTs. This fixes #14695 . #14734 (Amos Bird).
• Fix wrong monotonicity detection for shrunk Int -> Int cast of signed types. It might lead to incorrect query result. This bug is unveiled in #14513. #14783 (Amos Bird).
• Fix unreleased bug for LineAsString Format. #14842 (hexiaoting).
• Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the from_zk include option). This fixes #14814. #14843 (Alexander Kuzmenkov).
• Fix rare error in SELECT queries when the queried column has DEFAULT expression which depends on the other column which also has DEFAULT and not present in select query and not exists on disk. Partially fixes #14531. #14845 (alesapin).
• Fixed bug in parsing MySQL binlog events, which causes Attempt to read after eof and Packet payload is not fully read in MaterializeMySQL database engine. #14852 (Winter Zhang).
• Fixed segfault in CacheDictionary #14837. #14879 (Nikita Mikhaylov).
• Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). #14887 (Azat Khuzhin).
• Fix the issue when some invocations of extractAllGroups function may trigger "Memory limit exceeded" error. This fixes #13383. #14889 (Alexey Milovidov).
• Fixed .metadata.tmp File exists error when using MaterializeMySQL database engine. #14898 (Winter Zhang).
• Publish CPU frequencies per logical core in system.asynchronous_metrics. This fixes #14923. #14924 (Alexander Kuzmenkov).
• Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes #14847. #14937 (filimonov).
• Update jemalloc to fix possible issues with percpu arena. #14957 (Azat Khuzhin).
• Now settings number_of_free_entries_in_pool_to_execute_mutation and number_of_free_entries_in_pool_to_lower_max_size_of_merge can be equal to background_pool_size. #14975 (alesapin).
• Fix crash in RIGHT or FULL JOIN with join_algorith='auto' when memory limit exceeded and we should change HashJoin with MergeJoin. #15002 (Artem Zuikov).
• Fixed Cannot rename ... errno: 22, strerror: Invalid argument error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. #15024 (Alexander Tokmakov).
• If function bar was called with specifically crafted arguments, buffer overflow was possible. This closes #13926. #15028 (Alexey Milovidov).
• We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes #14908. #15033 (Amos Bird).
• Backported in #15829: Update jemalloc to fix percpu_arena with affinity mask. #15035 (Azat Khuzhin).
• Fixes Data compressed with different methods in join_algorithm='auto'. Keep LowCardinality as type for left table join key in join_algorithm='partial_merge'. #15088 (Artem Zuikov).
• Adjust decimals field size in mysql column definition packet. #15152 (maqroll).
• Fix bug in table engine Buffer which doesn't allow to insert data of new structure into Buffer after ALTER query. Fixes #15117. #15192 (alesapin).
• Fix instance crash when using joinGet with LowCardinality types. This fixes #15214. #15220 (Amos Bird).
• Fix 'Unknown identifier' in GROUP BY when query has JOIN over Merge table. #15242 (Artem Zuikov).
• Fix MSan report in QueryLog. Uninitialized memory can be used for the field memory_usage. #15258 (Alexey Milovidov).
• Fix hang of queries with a lot of subqueries to same table of MySQL engine. Previously, if there were more than 16 subqueries to same MySQL table in query, it hang forever. #15299 (Anton Popov).
• Fix rare race condition on server startup when system.logs are enabled. #15300 (alesapin).
• Fix race condition during MergeTree table rename and background cleanup. #15304 (alesapin).
• Fix bug where queries like SELECT toStartOfDay(today()) fail complaining about empty time_zone argument. #15319 (Bharat Nallan).
• Fixed compression in S3 storage. #15376 (Vladimir Chebotarev).
• Fix multiple occurrences of column transformers in a select query. #15378 (Amos Bird).
• fixes #15365 fix attach mysql database engine throw exception(no query context). #15384 (Winter Zhang).
• Report proper error when the second argument of boundingRatio aggregate function has a wrong type. #15407 (detailyang).
• Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in ON CLUSTER. Introduced in #13450. #15477 (alesapin).
• Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. #15516 (nvartolomei).
• Fix Missing columns errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes #15530. #15532 (alesapin).
• Fix bug when ILIKE operator stops being case insensitive if LIKE with the same pattern was executed. #15536 (alesapin).
• Mutation might hang waiting for some non-existent part after MOVE or REPLACE PARTITION or, in rare cases, after DETACH or DROP PARTITION. It's fixed. #15537 (Alexander Tokmakov).
• Fix 'Database doesn't exist.' in queries with IN and Distributed table when there's no database on initiator. #15538 (Artem Zuikov).
• Significantly reduce memory usage in AggregatingInOrderTransform/optimize_aggregation_in_order. #15543 (Azat Khuzhin).
• Prevent the possibility of error message Could not calculate available disk space (statvfs), errno: 4, strerror: Interrupted system call. This fixes #15541. #15557 (Alexey Milovidov).
• Query is finished faster in case of exception. Cancel execution on remote replicas if exception happens. #15578 (Azat Khuzhin).
• Fixed Element ... is not a constant expression error when using JSON* function result in VALUES, LIMIT or right side of IN operator. #15589 (Alexander Tokmakov).
• Fix the order of destruction for resources in ReadFromStorage step of query plan. It might cause crashes in rare cases. Possibly connected with #15610. #15645 (Nikolai Kochetov).
• Backported in #16009: Fixed bug with globs in S3 table function, region from URL was not applied to S3 client configuration. #15646 (Vladimir Chebotarev).
• Proper error handling during insert into MergeTree with S3. #15657 (Pavel Kovalenko).
• Backported in #15869: Fix error Cannot add simple transform to empty Pipe which happened while reading from Buffer table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes #15529. #15662 (Nikolai Kochetov).
• Fix race condition in AMQP-CPP. #15667 (alesapin).
• Fix rare race condition in dictionaries and tables from MySQL. #15686 (alesapin).
• Fixed too low default value of max_replicated_logs_to_keep setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. #15701 (Alexander Tokmakov).
• Fix error Cannot find column which may happen at insertion into MATERIALIZED VIEW in case if query for MV containes ARRAY JOIN. #15717 (Nikolai Kochetov).
• Backported in #15926: Fix drop of materialized view with inner table in Atomic database (hangs all subsequent DROP TABLE due to hang of the worker thread, due to recursive DROP TABLE for inner table of MV). #15743 (Azat Khuzhin).
• Fix some cases of queries, in which only virtual columns are selected. Previously Not found column _nothing in block exception may be thrown. Fixes #12298. #15756 (Anton Popov).
• Backported in #15867: Fix select count() inaccuracy for MaterializeMySQL. #15767 (Alexander Tokmakov).
• Backported in #15902: Fix exception Block structure mismatch in SELECT ... ORDER BY DESC queries which were executed after ALTER MODIFY COLUMN query. Fixes #15800. #15852 (alesapin).
• Backported in #15922: Now exception will be thrown when ALTER MODIFY COLUMN ... DEFAULT ... has incompatible default with column type. Fixes #15854. #15858 (alesapin).
• Backported in #15917: Fix possible deadlocks in RBAC. #15875 (Vitaly Baranov).
• Backported in #15950: fixes #12513 fix difference expressions with same alias when analyze queries again. #15886 (Winter Zhang).
• Backported in #16169: Fix incorrect empty result for query from Distributed table if query has WHERE, PREWHERE and GLOBAL IN. Fixes #15792. #15933 (Nikolai Kochetov).
• Backported in #15969: Fix a crash when database creation fails. #15954 (Winter Zhang).
• Backported in #16023: Fix ambiguity in parsing of settings profiles: CREATE USER ... SETTINGS profile readonly is now considered as using a profile named readonly, not a setting named profile with the readonly constraint. This fixes #15628. #15982 (Vitaly Baranov).
• Backported in #16217: Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). #15984 (Alexander Tokmakov).
• Backported in #16027: Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. #15987 (filimonov).
• Backported in #16089: Allow to use direct layout for dictionaries with complex keys. #16007 (Anton Popov).
• Backported in #16142: Fix ALTER MODIFY ... ORDER BY query hang for ReplicatedVersionedCollapsingMergeTree. This fixes #15980. #16011 (alesapin).
• Backported in #16077: Fixes #15780 regression, e.g. indexOf([1, 2, 3], toLowCardinality(1)) now is prohibited but it should not be. #16038 (Mike Kot).
• Backported in #16120: Fix segfault in some cases of wrong aggregation in lambdas. #16082 (Anton Popov).

Build/Testing/Packaging Improvement

• Now ClickHouse uses gcc-10 for the release build. Fixes #11138. #14609 (alesapin).
• Attempt to make performance test more reliable. It is done by remapping the executable memory of the process on the fly with madvise to use transparent huge pages - it can lower the number of iTLB misses which is the main source of instabilities in performance tests. #14685 (Alexey Milovidov).
• 1. In CMake files: - Moved some options' descriptions' parts to comments above. - Replace 0 -> OFF, 1 -> ON in options default values. - Added some descriptions and links to docs to the options. - Replaced FUZZER option (there is another option ENABLE_FUZZING which also enables same functionality). - Removed ENABLE_GTEST_LIBRARY option as there is ENABLE_TESTS. #14711 (Mike Kot).
• Speed up build a little by removing unused headers. #14714 (Alexey Milovidov).
• Fix build failure in OSX. #14761 (Winter Zhang).
• Attempt to speed up build a little. #14808 (Alexey Milovidov).
• Now we use clang-11 to build ClickHouse in CI. #14846 (alesapin).
• #14809 fix MaterializeMySQL empty transaction unstable test case found in CI. #14854 (Winter Zhang).
• Reformat and cleanup code in all integration test *.py files. #14864 (Bharat Nallan).
• Fixing tests/integration/test_distributed_over_live_view/test.py. #14892 (vzakaznikov).
• Switch from clang-tidy-10 to clang-tidy-11. #14922 (Alexey Milovidov).
• Convert to python3. This closes #14886. #15007 (Azat Khuzhin).
• Make performance test more stable and representative by splitting test runs and profile runs. #15027 (Alexey Milovidov).
• Maybe fix MSan report in base64 (on servers with AVX-512). This fixes #14006. #15030 (Alexey Milovidov).
• Don't allow any C++ translation unit to build more than 10 minutes or to use more than 10 GB or memory. This fixes #14925. #15060 (Alexey Milovidov).
• Now all test images use llvm-symbolizer-11. #15069 (alesapin).
• Splitted huge test test_dictionaries_all_layouts_and_sources into smaller ones. #15110 (Nikita Mikhaylov).
• Added a script to perform hardware benchmark in a single command. #15115 (Alexey Milovidov).
• Fix CMake options forwarding in fast test script. Fixes error in #14711. #15155 (alesapin).
• Improvements in CI docker images: get rid of ZooKeeper and single script for test configs installation. #15215 (alesapin).
• Now we use clang-11 for production ClickHouse build. #15239 (alesapin).
• Allow to run AArch64 version of clickhouse-server without configs. This facilitates #15174. #15266 (Alexey Milovidov).
• Fail early in functional tests if server failed to respond. This closes #15262. #15267 (Alexey Milovidov).
• fix bug for build error: #15272. #15297 (hexiaoting).
• fix bug for building query_db_generator.cpp. #15353 (hexiaoting).
• Allow to build with llvm-11. #15366 (Alexey Milovidov).
• Switch binary builds(Linux, Darwin, AArch64, FreeDSD) to clang-11. #15622 (Ilya Yatsishin).

NO CL ENTRY

• NO CL ENTRY: 'Bump numpy from 1.19.1 to 1.19.2 in /docs/tools'. #14733 (dependabot-preview[bot]).
• NO CL ENTRY: 'Modify the minimum version of the Clang compiler'. #14882 (Simon Liu).
• NO CL ENTRY: 'fix a syntax error bug while using copier'. #14890 (HyaZz).
• NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.9 to 0.4.13 in /docs/tools'. #15067 (dependabot-preview[bot]).
• NO CL ENTRY: 'Revert "Test and doc for PR12771 krb5 + cyrus-sasl + kerberized kafka"'. #15232 (Alexander Tokmakov).
• NO CL ENTRY: 'Revert "Avoid deadlocks in Log/TinyLog"'. #15259 (Alexey Milovidov).
• NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.13 to 0.4.17 in /docs/tools'. #15460 (dependabot-preview[bot]).