ClickHouse/docs/changelogs/v20.9.1.4585-prestable.md

ClickHouse release v20.9.1.4585-prestable FIXME as compared to v20.8.1.4513-prestable

Backward Incompatible Change

• Added MergeTree settings (max_replicated_merges_with_ttl_in_queue and max_number_of_merges_with_ttl_in_pool) to control the number of merges with TTL in the background pool and replicated queue. This change breaks compatibility with older versions only if you use delete TTL. Otherwise, replication will stay compatible. You can avoid incompatibility issues if you update all shard replicas at once or execute SYSTEM STOP TTL MERGES until you finish the update of all replicas. If you'll get an incompatible entry in the replication queue, first of all, execute SYSTEM STOP TTL MERGES and after ALTER TABLE ... DETACH PARTITION ... the partition where incompatible TTL merge was assigned. Attach it back on a single replica. #14490 (alesapin).

New Feature

• Add table function view which turns an subquery into a table object. This helps passing queries around. For instance, it can be used in remote/cluster table functions. #12567 (Amos Bird).
• Now we can write select * apply(length) apply(max) from wide_string_table to find out the maxium length of all string columns. And the follow two variants are provided too:. #14233 (Amos Bird).
• Add query_start_time_microseconds field to system.query_log & system.query_thread_log tables. #14252 (Bharat Nallan).
• Added an aggregate function RankCorrelationSpearman which simply computes a rank correlation coefficient. Continuation of #11769. #14411 (Nikita Mikhaylov).
• Added database generation by query util. Continuation of #10973. #14442 (Nikita Mikhaylov).

Performance Improvement

• Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key (under optimize_skip_unused_shards and optimize_distributed_group_by_sharding_key). #10373 (Azat Khuzhin).

Improvement

• Improvements in StorageRabbitMQ: Added connection and channels failure handling, proper commits, insert failures handling, better exchanges, queue durability and queue resume opportunity, new queue settings. Fixed tests. #12761 (Kseniia Sumarokova).
• Added Redis requirepass authorization. #13688 (Ivan Torgashov).
• Add precision argument for DateTime type. #13761 (Winter Zhang).
• Improve the Kafka engine performance by providing independent thread for each consumer. Separate thread pool for streaming engines (like Kafka). #13939 (fastio).
• Add default compression codec for parts in system.part_log with name default_compression_codec. #14116 (alesapin).
• Replace wide integers from boost multiprecision with implementation from https://github.com/cerevra/int. #14229 (Artem Zuikov).
• Implicitly convert primary key to not null in MaterializeMySQL(Same as MySQL). Fixes #14114. #14397 (Winter Zhang).
• Added new setting system_events_show_zero_values as proposed in #11384. #14404 (Dmitry Rubashkin).
• Now obfuscator supports UUID type as proposed in #13163. #14409 (Dmitry Rubashkin).
• Creating sets for multiple JOIN and IN in parallel. It may slightly improve performance for queries with several different IN subquery expressions. #14412 (Nikolai Kochetov).
• Now TTLs will be applied during merge if they were not previously materialized. #14438 (alesapin).
• MySQL handler returns OK for queries like SET @@var = value. Such statement is ignored. It is needed because some MySQL drivers send SET @@ query for setup after handshake #9336#issuecomment-686222422 . #14469 (BohuTANG).
• Speed up server shutdown process if there are ongoing S3 requests. #14496 (Pavel Kovalenko).
• Disallow empty time_zone argument in toStartOf* type of functions. #14509 (Bharat Nallan).
• ... #14523 (BohuTANG).
• Use std::filesystem::path in ConfigProcessor for concatenating file paths. #14558 (Bharat Nallan).

Bug Fix

• Fix arrayJoin() capturing in lambda (LOGICAL_ERROR). #13792 (Azat Khuzhin).
• Fix GRANT ALL statement when executed on a non-global level. #13987 (Vitaly Baranov).
• Disallows CODEC on ALIAS column type. Fixes #13911. #14263 (Bharat Nallan).
• Better check for tuple size in SSD cache complex key external dictionaries. This fixes #13981. #14313 (Alexey Milovidov).
• Fix QueryPlan lifetime (for EXPLAIN PIPELINE graph=1) for queries with nested interpreter. #14315 (Azat Khuzhin).
• Fix exception during ALTER LIVE VIEW query with REFRESH command. #14320 (Bharat Nallan).
• Fix crash during ALTER query for table which was created AS table_function. Fixes #14212. #14326 (alesapin).
• Stop query execution if exception happened in PipelineExecutor itself. This could prevent rare possible query hung. #14334 (Nikolai Kochetov).
• Stop query execution if exception happened in PipelineExecutor itself. This could prevent rare possible query hung. Continuation of #14334. #14402 (Nikolai Kochetov).
• Fix bug which leads to wrong merges assignment if table has partitions with a single part. #14444 (alesapin).
• Proxy restart/start/stop/reload of SysVinit to systemd (if it is used). #14460 (Azat Khuzhin).
• Check for array size overflow in topK aggregate function. Without this check the user may send a query with carefully crafter parameters that will lead to server crash. This closes #14452. #14467 (Alexey Milovidov).

Build/Testing/Packaging Improvement

• Integration tests use default base config. All config changes are explicit with main_configs, user_configs and dictionaries parameters for instance. #13647 (Ilya Yatsishin).
• ... #14368 (BohuTANG).
• Fix the logic in backport script. In previous versions it was triggered for any labels of 100% red color. It was strange. #14433 (Alexey Milovidov).
• Fix missed #include <atomic>. #14440 (Matwey V. Kornilov).
• Prepare for build with clang 11. #14455 (Alexey Milovidov).
• Lower binary size in debug build by removing debug info from Functions. This is needed only for one internal project in Yandex who is using very old linker. #14549 (Alexey Milovidov).
• Enable ccache by default in cmake if it's found in OS. #14575 (alesapin).

Other

• Changelog for 20.7 #13499. #14420 (Alexander Kazakov).

NO CL ENTRY

• NO CL ENTRY: 'Revert "Less number of threads in builder"'. #14421 (Alexey Milovidov).