ClickHouse/docs/changelogs/v20.6.1.4066-prestable.md

ClickHouse release v20.6.1.4066-prestable FIXME as compared to v20.5.1.3833-prestable

Backward Incompatible Change

• clickhouse-local now uses an unique temporary data directory by default, not the current directory as before. If needed, the data directory can be explicitly specified with the -- --path option. #11931 (Alexander Kuzmenkov).

New Feature

• Add Alter table drop replica replica_name support. This fixes #7080. #10679 (sundyli).
• Added new in-memory format of parts in MergeTree-family tables, which stores data in memory. Parts are written on disk at first merge. Part will be created in in-memory format if its size in rows or bytes is below thresholds min_rows_for_compact_part and min_bytes_for_compact_part. Also optional support of Write-Ahead-Log is available, which is enabled by default and is controlled by setting in_memory_parts_enable_wal. #10697 (Anton Popov).
• Add -Distinct combinator for aggregate functions. #10930 (Sofia Antipushina).
• Support table engine mongo(host:port, database, collection, user, password). #10931 (ageraab).
• Add storage RabbitMQ. #11069 (Kseniia Sumarokova).
• Opt-in settings to send crash reports to the ClickHouse core team via Sentry. #11300 (Ivan Blinkov).
• Add ORCBlockOutputFormat. #11662 (Kruglov Pavel).
• max_thread_pool_size config for changing the maximum number of Threads in Global Thread Pool. #11668 (Bharat Nallan).
• Initial implementation of EXPLAIN query. Syntax: EXPLAIN SELECT .... This fixes #1118. #11873 (Nikolai Kochetov).
• Switched paths in S3 metadata to relative which allows to handle S3 blobs more easily. #11892 (Vladimir Chebotarev).
• Avro UUID input/output support. #11954 (Andrew Onyshchuk).
• Added read-only MongoDB table engine. Allows reading flat (primitive types, not nested) fields. #11983 (alesapin).
• Add setting to fields not found in Avro schema. #12007 (Andrew Onyshchuk).
• add function parseDateTimeBestEffortUS. #12028 (flynn).
• #4006 Support ALTER TABLE ... [ADD|MODIFY] COLUMN ... FIRST. #12073 (Winter Zhang).
• Add a function initializedAggregation to initialize an aggregation based on a single value. #12109 (Guillaume Tassery).
• Support RIGHT and FULL JOIN with set join_algorithm=partial_merge. Only ALL strictness is supported (ANY, SEMI, ANTI, ASOF are not). #12118 (Artem Zuikov).
• Implementation of PostgreSQL-like ILIKE operator for #11710. #12125 (Mike Kot).
• Allow Nullable types as keys in MergeTree tables. #5319. #12433 (Amos Bird).

Performance Improvement

• Allow to use direct_io and mmap_io for secondary indices if the settings min_bytes_to_use_direct_io or min_bytes_to_use_mmap_io are configured. #11955 (Alexey Milovidov).
• Fix estimation of the number of marks while reading from MergeTree. This is needed to correctly handle the settings merge_tree_max_rows_to_use_cache, merge_tree_max_bytes_to_use_cache, merge_tree_min_rows_for_concurrent_read, merge_tree_min_bytes_for_concurrent_read, merge_tree_min_rows_for_seek, merge_tree_min_bytes_for_seek. Now settings min_bytes_to_use_mmap_io also applied to read index and compact parts in MergeTree table engines family. #11970 (Alexey Milovidov).
• Fix "#10574 Index not used for IN operator with literals", performance regression introduced around v19.3. #12062 (nvartolomei).
• Remove injective functions inside uniq*() if set optimize_injective_functions_inside_uniq=1. #12337 (Artem Zuikov).
• Add order by optimisation that rewrites ORDER BY x, f(x) with ORDER by x if set optimize_redundant_functions_in_order_by = 1. #12404 (Artem Zuikov).
• Lower memory usage for some operations up to 2 times. #12424 (Alexey Milovidov).

Improvement

• Add number of errors to ignore while choosing replicas (distributed_replica_error_ignore). #11669 (Azat Khuzhin).
• Improved performace of 'ORDER BY' and 'GROUP BY' by prefix of sorting key. #11696 (Anton Popov).
• • Add optimize_skip_unused_shards_nesting (allows control nesting level for shards skipping optimization) - Add force_skip_optimize_shards_nesting (allows control nesting level for checking was shards skipped or not) - Deprecate force_optimize_skip_unused_shards_no_nested (force_skip_optimize_shards_nesting should be used instead) - Disable optimize_skip_unused_shards if sharding_key has non-deterministic func (i.e. rand(), note that this does not changes anything for INSERT side). #11715 (Azat Khuzhin).
• Multiversion metadata for storages without structure locks. #11745 (alesapin).
• Slightly relax the validation of ODBC connection string. If the hostname or username contains only word characters along with . and -, don't put it into curly braces. It is needed, because some ODBC drivers (e.g. PostgreSQL) don't understand when hostname is enclosed in curly braces. #11845 (Alexey Milovidov).
• Support parse UUID without separator(separators are always removed in most implementations, this is helpful for users to write data). #11856 (Winter Zhang).
• Support SIGNED and UNSIGNED modifiers of standard integer types (BIGINT, INT, ...) for compatibility with MySQL. #11858 (Alexander Tokmakov).
• Allow to use sumWithOverflow as SimpleAggregateFunction. Closes #8053. #11865 (Alexander Kuzmenkov).
• Add FixedString support in Hashing functions. #11878 (flynn).
• Rewrite code for optimize_arithmetic_operations_in_aggregate_functions optimisation. #11899 (Artem Zuikov).
• Improve path concatenation and fix double slashed paths using std::filesystem::path instead of std::string in DatabaseOrdinary.cpp. #11900 (Bharat Nallan).
• Deprecate the old regular style and use the new globalVariable method #11832. #11901 (BohuTANG).
• related to issue 9797. #11923 (flynn).
• system.tables now considers column capacities for Memory and Buffer table engines, which is better approximation for resident memory size. #11935 (Max Akhmedov).
• Add CPU frequencies to system.asynchronous_metrics. Make the metric collection period configurable. #11972 (Alexander Kuzmenkov).
• Allow to perform "metadata-only" ALTER of partition key such as adding more elements to Enum data type. This fixes #7513. #11973 (Alexey Milovidov).
• Add replica priority for load_balancing (for manual prioritization of the load balancing). #11995 (Azat Khuzhin).
• Support MySQL engine reading Enums type #3985. #11996 (BohuTANG).
• Implemented single part uploads for DiskS3. #12026 (Vladimir Chebotarev).
• Moved useless S3 logging to TRACE level. #12067 (Vladimir Chebotarev).
• Improves REVOKE command: now it requires grant/admin option for only access which will be revoked. For example, to execute REVOKE ALL ON *.* FROM user1 now it doesn't require to have full access rights granted with grant option. Added command REVOKE ALL FROM user1 - it revokes all granted roles from user1. #12083 (Vitaly Baranov).
• Add 'type' column in system.disks. #12115 (ianton-ru).
• Added support for %g (two digit ISO year) and %G (four digit ISO year) substitutions in formatDateTime function. #12136 (vivarum).
• Add KILL QUERY [connection_id] for the MySQL client/driver to cancel the long query, issue #12038. #12152 (BohuTANG).
• 1. Support MySQL 'SELECT DATABASE()' #9336 2. Add MySQL replacement query integration test. #12314 (BohuTANG).
• This setting allows to chose charset for printing grids (either utf8 or ascii). #12372 (Sabyanin Maxim).
• Write the detail exception message to the client instead of 'MySQL server has gone away'. #12383 (BohuTANG).
• lifetime_rows/lifetime_bytes for Buffer engine. #12421 (Azat Khuzhin).
• Use correct default secure port for clickhouse-benchmark with --secure argument. This fixes #11044. #12440 (Alexey Milovidov).

Bug Fix

• Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. #11753 (hexiaoting).
• Fix wrong result for if() with NULLs in condition. #11807 (Artem Zuikov).
• Fix memory accounting via HTTP interface (can be significant with wait_end_of_query=1). #11840 (Azat Khuzhin).
• Fix rare crash caused by using Nullable column in prewhere condition. Continuation of #11608. #11869 (Nikolai Kochetov).
• Fix potential floating point exception when parsing DateTime64. This fixes #11374. #11875 (Alexey Milovidov).
• Fixed bug with no moves when changing storage policy from default one. #11893 (Vladimir Chebotarev).
• Fix rare crash caused by using Nullable column in prewhere condition. Continuation of #11869. #11895 (Nikolai Kochetov).
• Keep aliases for substitutions in query (parametrized queries). This fixes #11914. #11916 (Alexey Milovidov).
• Fix unitialized memory in partitions serialization. #11919 (alesapin).
• Use the correct current database for checking access rights after statement USE database. #11920 (Vitaly Baranov).
• Fixed Query parameter was not set in Values format. Fixes #11918. #11936 (Alexander Tokmakov).
• Fix race condition in extractAllGroups* functions. #11949 (Alexey Milovidov).
• Make mmap IO work again (experimental). Continuation of #8520. #11953 (Alexey Milovidov).
• Fix wrong setting name in log message at server startup. #11997 (Alexey Milovidov).
• Partial revokes work correctly in complex cases as well, for example. #12002 (Vitaly Baranov).
• Fix potential floating point exception. This closes #11378. #12005 (Alexey Milovidov).
• Avoid returning wrong number of geohashes in function geoHashesInBox due to accumulation of floating point error. This fixes #11369. #12006 (Alexey Milovidov).
• Fix potential array size overflow in generateRandom that may lead to crash. This fixes #11371. #12013 (Alexey Milovidov).
• Fix calculation of access rights when allow_ddl=0. #12015 (Vitaly Baranov).
• When adding floating point number of intervals to date/datetime, the result may be calculated incorrectly. This fixes #11377. #12018 (Alexey Milovidov).
• A query with function neighbor as the only returned expression may return empty result if the function is called with offset -9223372036854775808. This fixes #11367. #12019 (Alexey Milovidov).
• Do not mention in changelog, because the bug did not come to release. Fix potential crash when doing ORDER BY multiple columns with specified COLLATE on one of the column when this column is constant. This fixes #11379. The bug was introduced in #11006 in version 20.5. #12020 (Alexey Milovidov).
• Fix wrong result and potential crash when invoking function if with arguments of type FixedString with different sizes. This fixes #11362. #12021 (Alexey Milovidov).
• Fix calculation of access rights when allow_introspection_functions=0. #12031 (Vitaly Baranov).
• Fix crash in JOIN with LowCardinality type with join_algorithm=partial_merge. #12035 (Artem Zuikov).
• Fix incorrect comparison of tuples with Nullable columns. Fixes #11985. #12039 (Nikolai Kochetov).
• Fix constraints check if constraint is a constant expression. This fixes #11360. #12042 (Alexey Milovidov).
• Make topK aggregate function return Enum for Enum types. This fixes #3740. #12043 (Alexey Milovidov).
• Parse tables metadata in parallel when loading database. This fixes slow server startup when there are large number of tables. #12045 (Alexander Tokmakov).
• Fix error Cannot capture column for higher-order functions with Tuple(LowCardinality) argument. Fixes #9766. #12055 (Nikolai Kochetov).
• Fix error Expected single dictionary argument for function for function defaultValueOfArgumentType with LowCardinality type. Fixes #11808. #12056 (Nikolai Kochetov).
• Fix possible crash while using wrong type for PREWHERE. Fixes #12053, #12060. #12060 (Nikolai Kochetov).
• Fix SIGSEGV in StorageKafka on DROP TABLE. #12075 (Azat Khuzhin).
• Fix unnecessary limiting the number of threads for selects from VIEW. Fixes #11937. #12085 (Nikolai Kochetov).
• Fix empty result_rows and result_bytes metrics in system.quey_log for selects. Fixes #11595. #12089 (Nikolai Kochetov).
• Fix segfault with -StateResample combinators. #12092 (Anton Popov).
• Fix performance for selects with UNION caused by wrong limit for the total number of threads. Fixes #12030. #12103 (Nikolai Kochetov).
• Format Parquet now properly works with LowCardinality and LowCardinality(Nullable) types. Fixes #12086, #8406. #12108 (Nikolai Kochetov).
• Fix handling dependency of table with ENGINE=Dictionary on dictionary. This fixes #10994. This fixes #10397. #12116 (Vitaly Baranov).
• Avoid "There is no query" exception for materialized views with joins or with subqueries attached to system logs (system.query_log, metric_log, etc) or to engine=Buffer underlying table. #12120 (filimonov).
• Fix bug which leads to incorrect table metadata in ZooKeepeer for ReplicatedVersionedCollapsingMergeTree tables. Fixes #12093. #12121 (alesapin).
• Normalize "pid" file handling. In previous versions the server may refuse to start if it was killed without proper shutdown and if there is another process that has the same pid as previously runned server. Also pid file may be removed in unsuccessful server startup even if there is another server running. This fixes #3501. #12133 (Alexey Milovidov).
• Fix potential infinite loop in greatCircleDistance, geoDistance. This fixes #12117. #12137 (Alexey Milovidov).
• Fix potential overflow in integer division. This fixes #12119. #12140 (Alexey Milovidov).
• Fix bad code in redundant ORDER BY optimization. The bug was introduced in #10067. #12148 (Alexey Milovidov).
• Fix transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. #12151 (Alexey Milovidov).
• Fix wrong logic in ALTER DELETE that leads to deleting of records when condition evaluates to NULL. This fixes #9088. This closes #12106. #12153 (Alexey Milovidov).
• Don't split the dictionary source's table name into schema and table name itself if ODBC connection doesn't support schema. #12165 (Vitaly Baranov).
• Fixed the behaviour when SummingMergeTree engine sums up columns from partition key. Added an exception in case of explicit definition of columns to sum which intersects with partition key columns. This fixes #7867. #12173 (Nikita Mikhaylov).
• Fix dictGet arguments check during GROUP BY injective functions elimination. #12179 (Azat Khuzhin).
• Cap max_memory_usage* limits to the process resident memory. #12182 (Azat Khuzhin).
• Fixed logical functions for UInt8 values when they are not equal to 0 or 1. #12196 (Alexander Kazakov).
• Fixed the behaviour when during multiple sequential inserts in StorageFile header for some special types was written more than once. This fixed #6155. #12197 (Nikita Mikhaylov).
• Fixed behaviour on reaching redirect limit in request to S3 storage. #12256 (ianton-ru).
• Not for changelog. Cherry-pick after #12196. #12271 (Alexey Milovidov).
• Implement conversions to the common type for LowCardinality types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes #8212. This fixes #4342. #12275 (Alexey Milovidov).
• The function arrayFill worked incorrectly for empty arrays that may lead to crash. This fixes #12263. #12279 (Alexey Milovidov).
• Show error after TrieDictionary failed to load. #12290 (Vitaly Baranov).
• Fix typo in setting name. #12292 (Alexey Milovidov).
• Some threads might randomly hang for a few seconds during DNS cache updating. It's fixed. #12296 (Alexander Tokmakov).
• Fix TTL after renaming column, on which depends TTL expression. #12304 (Anton Popov).
• Avoid "bad cast" exception when there is an expression that filters data by virtual columns (like _table in Merge tables) or by "index" columns in system tables such as filtering by database name when querying from system.tables, and this expression returns Nullable type. This fixes #12166. #12305 (Alexey Milovidov).
• Fix order of columns in WITH FILL modifier. Previously order of columns of ORDER BY statement wasn't respected. #12306 (Anton Popov).
• When using codec Delta or DoubleDelta with non fixed width types, exception with code LOGICAL_ERROR was returned instead of exception with code BAD_ARGUMENTS (we ensure that exceptions with code logical error never happen). This fixes #12110. #12308 (Alexey Milovidov).
• Fix very rare race condition in ReplicatedMergeTreeQueue. #12315 (Alexey Milovidov).
• Fix error message and exit codes for ALTER RENAME COLUMN queries, when RENAME is not allowed. Fixes #12301 and #12303. #12335 (alesapin).
• Fix TOTALS/ROLLUP/CUBE for aggregate functions with -State and Nullable arguments. This fixes #12163. #12376 (Alexey Milovidov).
• Allow to CLEAR column even if there are depending DEFAULT expressions. This fixes #12333. #12378 (Alexey Milovidov).
• Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables. This fixes #11905. #12384 (Alexey Milovidov).
• Additional check for arguments of bloom filter index. This fixes #11408. #12388 (Alexey Milovidov).
• Reverts change introduced in #11079 to resolve #12098. #12397 (Mike Kot).
• Fixed possible segfault if StorageMerge. Closes #12054. #12401 (Alexander Tokmakov).
• Fix overflow when very large LIMIT or OFFSET is specified. This fixes #10470. This fixes #11372. #12427 (Alexey Milovidov).
• Fix UBSan report in base64 if tests were run on server with AVX-512. This fixes #12318. Author: @qoega. #12441 (Alexey Milovidov).

Build/Testing/Packaging Improvement

• Add simple GitHub hook script for the serverless environment. #11605 (alesapin).
• Send logs to client on fatal errors if possible. This will make test results more readable. #11826 (Alexey Milovidov).
• Allow ClickHouse to run on Android. #11894 (Alexey Milovidov).
• Clean up unused header files from Obfuscator.cpp and DatabaseAtomic.cpp. #11922 (Bharat Nallan).
• Fix some typos in code. #12003 (Alexey Milovidov).
• Scripts for automated scheduled backporting based on PR labels. #12029 (Ivan).
• Add new type of tests based on Testflows framework. #12090 (vzakaznikov).
• Install ca-certificates before the first apt-get update in Dockerfile. #12095 (Ivan Blinkov).
• Daily check by GitHub CodeQL security analysis tool that looks for CWE. #12101 (Ivan Blinkov).
• Regular check by Anchore Container Analysis security analysis tool that looks for CVE in clickhouse-server Docker image. Also confirms that Dockerfile is buildable. Runs daily on master and on pull-requests to Dockerfile. #12102 (Ivan Blinkov).
• Add UNBUNDLED flag to system.build_options table. Move skip lists for clickhouse-test to clickhouse repo. #12107 (alesapin).
• Implement AST-based query fuzzing mode for clickhouse-client. See this label for the list of issues we recently found by fuzzing. Most of them were found by this tool, and a couple by SQLancer and 00746_sql_fuzzy.pl. #12111 (Alexander Kuzmenkov).
• Perform an upgrade of system packages in the clickhouse-server docker image. #12124 (Ivan Blinkov).
• Added a showcase of the minimal Docker image without using any Linux distribution. #12126 (Alexey Milovidov).
• Minor CMake fixes for UNBUNDLED build. #12131 (Matwey V. Kornilov).
• Missed <atomic> is required for std::atomic<>. #12134 (Matwey V. Kornilov).
• Fix warnings from CodeQL. CodeQL is another static analyzer that we will use along with clang-tidy and PVS-Studio that we use already. #12138 (Alexey Milovidov).
• Record additional detail on Dockerfile scan reports. #12159 (Ivan Blinkov).
• Place common docker compose files to integration docker container. #12168 (Ilya Yatsishin).
• Remove verbosity from the binary builds. #12174 (alesapin).
• Remove strange file creation during build in orc. #12258 (Nikita Mikhaylov).
• Now functional and stress tests will be able to run with old version of clickhouse-test script. #12287 (alesapin).
• Log sanitizer trap messages from separate thread. This will prevent possible deadlock under thread sanitizer. #12313 (Alexey Milovidov).
• Added S3 HTTPS integration test. #12412 (Pavel Kovalenko).
• Fix UBSan report in HDFS library. This closes #12330. #12453 (Alexey Milovidov).

Other

• Update word break characters to match readline default - all non-alphanumeric characters. ... #11975 (Andrew Onyshchuk).

NO CL CATEGORY

• • Not for changelog. #12265 (Alexander Kuzmenkov).
• ... #12431 (Tom Bombadil).

NO CL ENTRY

• NO CL ENTRY: 'Bump certifi from 2020.4.5.2 to 2020.6.20 in /docs/tools/translate'. #11853 (dependabot-preview[bot]).
• NO CL ENTRY: 'Merging #10679'. #11896 (Alexander Tokmakov).
• NO CL ENTRY: 'Revert "[experiment] maybe fix warnings in integration tests"'. #12011 (alesapin).
• NO CL ENTRY: 'Bump idna from 2.9 to 2.10 in /docs/tools'. #12024 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools'. #12049 (dependabot-preview[bot]).