ClickHouse/docs/changelogs/v20.7.1.4310-prestable.md

ClickHouse release v20.7.1.4310-prestable FIXME as compared to v20.6.1.4066-prestable

Backward Incompatible Change

• Deprecate special printing of zero Date/DateTime values as 0000-00-00 and 0000-00-00 00:00:00. #12442 (Alexey Milovidov).
• Function modulo (operator %) with at least one floating point number as argument will calculate remainder of division directly on floating point numbers without converting both arguments to integers. It makes behaviour compatible with most of DBMS. This also applicable for Date and DateTime data types. Added alias mod. This closes #7323. #12585 (Alexey Milovidov).

New Feature

• • Added support of LDAP authentication for preconfigured users ("Simple Bind" method). #11234 (Denis Glazachev).
• Add mapAdd and mapSubtract functions for working with tuple maps. #11735 (Ildus Kurbangaliev).
• Added system.crash_log table into which stack traces for fatal errors are collected. #12316 (Alexey Milovidov).
• Add bayesAB function for bayesian-ab-testing. #12327 (achimbab).
• Integration with COS. #12386 (fastio).
• • Add FROM_UNIXTIME function, related to 12149. #12484 (flynn).
• A function formatRow is added to support turning arbitrary expressions into a string via given format. It's useful for manipulating SQL outputs and is quite versatile combined with the columns function. #12574 (Amos Bird).
• Add setting allow_non_metadata_alters which restricts to execute ALTER queries which modify data on disk. Disabled be default. Closes #11547. #12635 (alesapin).
• Add minMap and maxMap functions support to SimpleAggregateFunction. #12662 (Ildus Kurbangaliev).
• Added http headers X-ClickHouse-Database and X-ClickHouse-Format which may be used to set default database and output format. #12981 (hcz).
• Implement user-defined settings. #13013 (Vitaly Baranov).

Performance Improvement

• Optimize PK lookup for queries that match exact PK range. #12277 (Ivan Babrou).
• Attempt to implement streaming optimization in DiskS3. #12434 (Vladimir Chebotarev).
• Replaces monotonous functions with its argument in ORDER BY if set optimize_monotonous_functions_in_order_by=1. #12467 (Artem Zuikov).
• Converts String-type arguments of function "if" and "transform" into enum if set optimize_if_transform_strings_to_enum = 1. #12515 (Artem Zuikov).
• Parallel PK lookup and skipping index stages on parts, as described in #11564. #12589 (Ivan Babrou).
• Push down LIMIT step for query plan. #13016 (Nikolai Kochetov).
• Slightly improve performance of aggregation by UInt8/UInt16 keys. #13055 (Alexey Milovidov).
• Slightly improve performance of aggregation by UInt8/UInt16 keys. #13091 (Alexey Milovidov).

Improvement

• Adds a new type of polygon dictionary which uses a recursively built grid to reduce the number of polygons which need to be checked for each point. #9278 (achulkov2).
• Allow TabSeparatedRaw as an input format. #12009 (hcz).
• Separated AWSAuthV4Signer into different logger, removed "AWSClient: AWSClient". #12320 (Vladimir Chebotarev).
• • Implement RENAME DATABASE and RENAME DICTIONARY for Atomic database engine - Add implicit {uuid} macro, which can be used in ZooKeeper path for ReplicatedMergeTree. It works with CREATE ... ON CLUSTER ... queries. Set show_table_uuid_in_table_create_query_if_not_nil to true to use it. - Make ReplicatedMergeTree engine arguments optional, /clickhouse/tables/{uuid}/{shard}/ and {replica} are used by default. Closes #12135. - Minor fixes. - These changes break backward compatibility of Atomic database engine. Previously created Atomic databases must be manually converted to new format. #12343 (Alexander Tokmakov).
• Now joinGet supports multi-key lookup. #12418 (Amos Bird).
• Rollback insertion errors in Log, TinyLog, StripeLog engines. In previous versions insertion error lead to inconsisent table state (this works as documented and it is normal for these table engines). This fixes #12402. #12426 (Alexey Milovidov).
• Avoid overflow in parsing of DateTime values that will lead to negative unix timestamp in their timezone (for example, 1970-01-01 00:00:00 in Moscow). Saturate to zero instead. This fixes #3470. This fixes #4172. #12443 (Alexey Milovidov).
• Add several metrics for requests to S3 storages. #12464 (ianton-ru).
• Changes default value for multiple_joins_rewriter_version to 2. It enables new multiple joins rewriter that knows about column names. #12469 (Artem Zuikov).
• Allow to set JOIN kind and type in more standad way: LEFT SEMI JOIN instead of SEMI LEFT JOIN. For now both are correct. #12520 (Artem Zuikov).
• Fix explain query format overwrite by default, issue #12432. #12541 (BohuTANG).
• Add SelectedRows and SelectedBytes events. #12638 (ianton-ru).
• Add current_database information to system.query_log. #12652 (Amos Bird).
• Support truncate table without table keyword. #12653 (Winter Zhang).
• Now exceptions forwarded to the client if an error happened during ALTER or mutation. Closes #11329. #12666 (alesapin).
• Protect from the cases when user may set background_pool_size to value lower than number_of_free_entries_in_pool_to_execute_mutation or number_of_free_entries_in_pool_to_lower_max_size_of_merge. In these cases ALTERs won't work or the maximum size of merge will be too limited. Saturate values instead. This closes #10897. #12728 (Alexey Milovidov).
• This change makes skipping index efficiency more obvious by showing both skipped and total examined granules. #12754 (Ivan Babrou).
• This change makes skipping index efficiency more obvious by showing total marks before and after skipping indices. #12755 (Ivan Babrou).
• Introduce setting alter_partition_verbose_result which outputs information about touched parts for some types of ALTER TABLE ... PARTITION ... queries (currently ATTACH and FREEZE). Closes #8076. #13017 (alesapin).
• Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQueryTimeMicroseconds to system.events. #13028 (ianton-ru).
• break-ing out of a loop because it makes sense to do so. #13058 (Mark Papadakis).
• Keep less amount of logs in ZooKeeper. Avoid too large growth of ZooKeeper nodes in case of offline replicas when having many servers/tables/inserts. #13100 (Alexey Milovidov).
• Add sanity check for MergeTree settings. If the settings are incorrect, the server will refuse to start or to create a table, printing detailed explanation to the user. #13153 (Alexey Milovidov).
• Allow *Map aggregate functions to work on Arrays with NULLs. Fixes #13157. #13225 (Alexey Milovidov).
• Fix assert in parseDateTimeBestEffort. This fixes #12649. #13227 (Alexey Milovidov).
• Fix assert in geohashesInBox. This fixes #12554. #13229 (Alexey Milovidov).
• Now broken parts are also reported when encountered in compact part processing. #13282 (Amos Bird).
• Fix a 'Week'-interval formatting for ATTACH/ALTER/CREATE QUOTA-statements. #13417 (vladimir-golovchenko).
• Updated gitignore-files. #13447 (vladimir-golovchenko).

Bug Fix

• kafka: fix SIGSEGV if there is an message with error in the middle of the batch. #12302 (Azat Khuzhin).
• If MergeTree table does not contain ORDER BY or PARTITION BY, it was possible to request ALTER to CLEAR all the columns and ALTER will stuck. Fixed #7941. #12382 (Alexey Milovidov).
• SystemLog: do not write to ordinary server log under mutex. This can lead to deadlock if text_log is enabled. #12452 (Alexey Milovidov).
• Fix crash in JOIN with dictionary when we are joining over expression of dictionary key: t JOIN dict ON expr(dict.id) = t.id. Disable dictionary join optimisation for this case. #12458 (Artem Zuikov).
• Fix SETTINGS parse after FORMAT. #12480 (Azat Khuzhin).
• Fix backwards compatibility in binary format of AggregateFunction(avg, ...) values. This fixes #12342. #12486 (Alexey Milovidov).
• Fixed performance issue, while reading from compact parts. #12492 (Anton Popov).
• Fixing race condition in live view tables which could cause data duplication. #12519 (vzakaznikov).
• Better exception for function in with invalid number of arguments. #12529 (Anton Popov).
• Fix bug which lead to broken old parts after ALTER DELETE query when enable_mixed_granularity_parts=1. Fixes #12536. #12543 (alesapin).
• Now ClickHouse will recalculate checksums for parts when file checksums.txt is absent. Broken since #9827. #12545 (alesapin).
• Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 (Azat Khuzhin).
• Fix race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
• Fix lack of aliases with function any. #12593 (Anton Popov).
• Fix error Output of TreeExecutor is not sorted for OPTIMIZE DEDUPLICATE. Fixes #11572. #12613 (Nikolai Kochetov).
• Fix possible Pipeline stuck error for queries with external sorting. Fixes #12617. #12618 (Nikolai Kochetov).
• The function groupArrayMoving* was not working for distributed queries. It's result was calculated within incorrect data type (without promotion to the largest type). The function groupArrayMovingAvg was returning integer number that was inconsistent with the avg function. This fixes #12568. #12622 (Alexey Milovidov).
• Fix error message about adaptive granularity. #12624 (alesapin).
• Better exception message in disk access storage. #12625 (alesapin).
• Exception There is no supertype... can be thrown during ALTER ... UPDATE in unexpected cases (e.g. when subtracting from UInt64 column). This fixes #7306. This fixes #4165. #12633 (Alexey Milovidov).
• CREATE USER IF NOT EXISTS now doesn't throw exception if the user exists. This fixes #12507. #12646 (Vitaly Baranov).
• Add support for function if with Array(UUID) arguments. This fixes #11066. #12648 (Alexey Milovidov).
• Fix SIGSEGV in StorageKafka when broker is unavailable (and not only). #12658 (Azat Khuzhin).
• fixes #10572 fix bloom filter index with const expression. #12659 (Winter Zhang).
• fixes #12293 allow push predicate when subquery contains with clause. #12663 (Winter Zhang).
• Fix optimization optimize_move_functions_out_of_any=1 in case of any(func(<lambda>)). #12664 (Artem Zuikov).
• Fix memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 (Azat Khuzhin).
• Fix performance with large tuples, which are interpreted as functions in IN section. The case when user write WHERE x IN tuple(1, 2, ...) instead of WHERE x IN (1, 2, ...) for some obscure reason. #12700 (Anton Popov).
• Fix CAST(Nullable(String), Enum()). #12745 (Azat Khuzhin).
• Fix rare bug when ALTER DELETE and ALTER MODIFY COLUMN queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows in count.txt and as a consequence incorrect data in part. Also, fix a small bug with simultaneous ALTER RENAME COLUMN and ALTER ADD COLUMN. #12760 (alesapin).
• Fix unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
• Fix columns duplication for range hashed dictionary created from DDL query. This fixes #10605. #12857 (alesapin).
• Corrected merge_with_ttl_timeout logic which did not work well when expiration affected more than one partition over one time interval. (Authored by @excitoon). #12982 (Alexander Kazakov).
• Fix Block structure mismatch error for queries with UNION and JOIN. Fixes #12602. #12989 (Nikolai Kochetov).
• Fix crash which was possible for queries with ORDER BY tuple and small LIMIT. Fixes #12623. #13009 (Nikolai Kochetov).
• Add logging when the mutation is not running because of limited disk space or free threads in the background pool. #13068 (alesapin).
• Fix error Cannot convert column because it is constant but values of constants are different in source and result for remote queries which use deterministic functions in scope of query, but not deterministic between queries, like now(), now64(), randConstant(). Fixes #11327. #13075 (Nikolai Kochetov).
• Fix wrong index analysis with functions. It could lead to pruning wrong parts, while reading from MergeTree tables. Fixes #13060. Fixes #12406. #13081 (Anton Popov).
• Fix segfault when mutation killed and the server tries to send the exception to the client. #13169 (alesapin).
• AvroConfluent: Skip Kafka tombstone records AvroConfluent: Support skipping broken records ... #13203 (Andrew Onyshchuk).
• Fix DateTime64 conversion functions with constant argument. #13205 (Azat Khuzhin).
• Fix assert in arrayElement function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. #13224 (Alexey Milovidov).
• Fix function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. #13226 (Alexey Milovidov).
• Return passed number for numbers with MSB set in roundUpToPowerOfTwoOrZero(). #13234 (Azat Khuzhin).
• Fix assertion in KeyCondition when primary key contains expression with monotonic function and query contains comparison with constant whose type is different. This fixes #12465. #13251 (Alexey Milovidov).
• Fix potentially low performance and slightly incorrect result for uniqExact, topK, sumDistinct and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes #12491. #13254 (Alexey Milovidov).
• The server may crash if user passed specifically crafted arguments to the function h3ToChildren. This fixes #13275. #13277 (Alexey Milovidov).
• Fix possible error Totals having transform was already added to pipeline in case of a query from delayed replica. #13290 (Nikolai Kochetov).
• Fix crash in LEFT ASOF JOIN with join_use_nulls=1. #13291 (Artem Zuikov).
• Throw error on arrayJoin() function in JOIN ON section. #13330 (Artem Zuikov).
• Fix segfault when function groupArrayMovingSum deserializes empty state. Fixes #13339. #13341 (alesapin).
• Fixed the deadlock in textlog. It is a part of #12339. This fixes #12325. #13386 (Nikita Mikhaylov).
• Fix PrettyCompactMonoBlock for clickhouse-local. Fix extremes/totals with PrettyCompactMonoBlock. Fixes #7746. #13394 (Azat Khuzhin).
• Fix queries with constant columns and ORDER BY prefix of primary key. #13396 (Anton Popov).
• Fix empty output for Arrow and Parquet formats in case if query return zero rows. It was done because empty output is not valid for this formats. #13399 (hcz).
• Fix aggregate function any(x) is found inside another aggregate function in query error with SET optimize_move_functions_out_of_any = 1 and aliases inside any(). #13419 (Artem Zuikov).
• Fix invalid return type for comparison of tuples with NULL elements. Fixes #12461. #13420 (Nikolai Kochetov).
• Fix error in parseDateTimeBestEffort function when unix timestamp was passed as an argument. This fixes #13362. #13441 (Alexey Milovidov).

Build/Testing/Packaging Improvement

• Add compiler option to control that stack frames are not too large. This will help to run the code in fibers with small stack size. #11524 (Alexey Milovidov).
• Rework integrational configuration paths. #12285 (Ilya Yatsishin).
• Add docker image for fast tests. #12294 (alesapin).
• Update fmtlib to master (7.0.1). #12446 (Alexey Milovidov).
• Add PEERDIR(protoc) as protobuf format parses .proto file in runtime. #12475 (Yuriy Chernyshov).
• Now we check that server is able to start after stress tests run. This fixes #12473. #12496 (alesapin).
• Improve performance of TestKeeper. This will speedup tests with heavy usage of Replicated tables. #12505 (Alexey Milovidov).
• Adding RBAC syntax tests in TestFlows. #12642 (vzakaznikov).
• Apply random query mutations (fuzzing) in stress tests. #12734 (Alexander Kuzmenkov).
• Connector-ODBC updated to mysql-connector-odbc-8.0.21. #12739 (Ilya Yatsishin).
• Fix build of test under Mac OS X. This closes #12767. #12772 (Alexey Milovidov).
• increasing timeouts in testflows tests. #12949 (vzakaznikov).
• Add a test for ALTER TABLE CLEAR COLUMN query for primary key columns. #12951 (alesapin).
• Do not build helper_container image inside integrational tests. Build docker container in CI and use pre-built helper_container in integration tests. #12953 (Ilya Yatsishin).
• Check an ability that we able to restore the backup from an old version to the new version. This closes #8979. #12959 (alesapin).
• Fix MSan error in "rdkafka" library. This closes #12990. Updated rdkafka to version 1.5 (master). #12991 (Alexey Milovidov).
• Rerun some tests in fast test check. #12992 (alesapin).
• Adding extra xfails for some ldap tests. #13054 (vzakaznikov).
• Added tests for RBAC functionality of SELECT privilege in TestFlows. #13061 (Ritaank Tiwari).
• Rewrote Function tests to gtest. Removed useless includes from tests. #13073 (Nikita Mikhaylov).
• Improve cache purge in documentation deploy script. #13107 (alesapin).
• Fixing 00960_live_view_watch_events_live.py test. #13108 (vzakaznikov).
• Small fixes to the RBAC SRS. #13152 (vzakaznikov).
• Even more retries in zkutil gtest to prevent test flakiness. #13165 (Alexey Milovidov).
• Add script which set labels for pull requests in GitHub hook. #13183 (alesapin).
• Use shellcheck for sh tests linting. #13200 (Alexey Milovidov).
• Use shellcheck for sh tests linting. #13207 (Alexey Milovidov).
• Fix libunwind build in AArch64. This fixes #13204. #13208 (Alexey Milovidov).
• Function materialize (the function for ClickHouse testing) will work for NULL as expected - by transforming it to non-constant column. #13212 (Alexey Milovidov).
• Now fast test will wait server with retries. #13284 (alesapin).
• Applying LDAP authentication test fixes. #13310 (vzakaznikov).
• Fix timeout error during server restart in the stress test. #13321 (alesapin).
• • Added testing for RBAC functionality of INSERT privilege in TestFlows. - Expanded tables on which SELECT is being tested. - Added Requirements to match new table engine tests. #13340 (MyroTk).
• Remove some of recursive submodules. See #13378. #13379 (Alexey Milovidov).
• Ensure that all the submodules are from proper URLs. Continuation of #13379. This fixes #13378. #13397 (Alexey Milovidov).

Other

• Avoid re-loading completion from the history file after each query (to avoid history overlaps with other client sessions). #13086 (Azat Khuzhin).

NO CL ENTRY

• NO CL ENTRY: 'Bump numpy from 1.18.5 to 1.19.1 in /docs/tools'. #12655 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump urllib3 from 1.25.9 to 1.25.10 in /docs/tools'. #12703 (dependabot-preview[bot]).
• NO CL ENTRY: 'Revert "Refactor joinGet and implement multi-key lookup."'. #12708 (alesapin).
• NO CL ENTRY: 'Revert "Abort on std::out_of_range in debug builds"'. #12752 (Alexey Milovidov).
• NO CL ENTRY: 'Bump protobuf from 3.12.2 to 3.12.4 in /docs/tools'. #13102 (dependabot-preview[bot]).
• NO CL ENTRY: 'Merge #12574'. #13158 (Alexey Milovidov).
• NO CL ENTRY: 'Revert "Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQuer..."'. #13303 (Alexey Milovidov).