ClickHouse/docs/changelogs/v20.5.1.3833-prestable.md

ClickHouse release v20.5.1.3833-prestable FIXME as compared to v20.4.1.3177-prestable

Backward Incompatible Change

• Remove experimental_use_processors setting. It is enabled by default. #10924 (Nikolai Kochetov).
• The setting input_format_with_names_use_header is enabled by default. It will affect parsing of input formats -WithNames and -WithNamesAndTypes. #10937 (Alexey Milovidov).
• Added a check for the case when user-level setting is specified in a wrong place. User-level settings should be specified in users.xml inside <profile> section for specific user profile (or in <default> for default settings). The server won't start with exception message in log. This fixes #9051. If you want to skip the check, you can either move settings to the appropriate place or add <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> to config.xml. #11449 (Alexey Milovidov).
• Return non-Nullable result from COUNT(DISTINCT), and uniq aggregate functions family. If all passed values are NULL, return zero instead. This improves SQL compatibility. #11661 (Alexey Milovidov).

New Feature

• Support Cassandra as external dictionary. #4978 (favstovol).
• SimpleAggregateFunction now also supports sumMap. #10000 (Ildus Kurbangaliev).
• Added OpenCl support and bitonic sort algorithm, which can be used for sorting integer types of data in single column. Needs to be build with flag -DENABLE_OPENCL=1. For using bitonic sort algorithm instead of others you need to set bitonic_sort for Setting's option special_sort and make sure that OpenCL is available. #10232 (Margarita Konnova [MARK]).
• Implementation of PostgreSQL wire protocol. #10242 (Movses Elbakian).
• Selects with final are executed in parallel. Added setting max_final_threads to limit the number of threads used. #10463 (Nikolai Kochetov).
• Function that extracts from haystack all matching non-overlapping groups with regular expressions, and put those into Array(Array(String)) column. #10534 (Vasily Nemkov).
• Added ability to delete a subset of expired rows, which satisfies the condition in WHERE clause. Added ability to replace expired rows with aggregates of them specified in GROUP BY clause. #10537 (expl0si0nn).
• (Only Linux) ClickHouse server now tries to fallback to ProcfsMetricsProvider when clickhouse binary is not attributed with CAP_NET_ADMIN capability to collect per-query system metrics (for CPU and I/O). #10544 (Alexander Kazakov).
• • Add Arrow IPC File format (Input and Output) - Fix incorrect work of resetParser() for Parquet Input Format - Add zero-copy optimization for ORC for RandomAccessFiles - Add missing halffloat type for input parquet and ORC formats ... #10580 (Zhanna).
• Allowed to profile memory with finer granularity steps than 4 MiB. Added sampling memory profiler to capture random allocations/deallocations. #10598 (Alexey Milovidov).
• Add new input format JSONAsString that accepts a sequence of JSON objects separated by newlines, spaces and/or commas. #10607 (Kruglov Pavel).
• Added a new layout direct which loads all the data directly from the source for each query, without storing or caching data. #10622 (Artem Streltsov).
• Default user and database creation on docker image starting. #10637 (Paramtamtam).
• Add data type Point (Tuple(Float64, Float64)) and Polygon (Array(Array(Tuple(Float64, Float64))). #10678 (Alexey Ilyukhov).
• New function function toStartOfSecond(DateTime64) -> DateTime64 that nullifies sub-second part of DateTime64 value. #10722 (Vasily Nemkov).
• Added a function randomString that generates binary string with random bytes (including zero bytes). #10733 (Andrei Nekrashevich).
• Added system.licenses table. This table contains licenses of third-party libraries that are located in contrib directory. This closes #2890. #10795 (Alexey Milovidov).
• OFFSET keyword can now be used without an affiliated LIMIT clause. #10802 (Guillaume Tassery).
• Added new complex key direct layout to dictionaries, that does not store anything locally during query execution. #10850 (Artem Streltsov).
• Added function randomFixedString. #10866 (Andrei Nekrashevich).
• Support ALTER RENAME COLUMN for the distributed table engine. Continuation of #10727. Fixes #10747. #10887 (alesapin).
• Support writes in ODBC Table function #10554 (ageraab). #10901 (Alexander Tokmakov).
• Allow specifying mongodb:// URI for MongoDB dictionaries. #10915 (Alexander Kuzmenkov).
• Added new functions to import/export DateTime64 as Int64 with various precision: to-/fromUnixTimestamp64Milli/-Micro/-Nano. #10923 (Vasily Nemkov).
• Now support NULL and NOT NULL modifiers for data types in create query. #11057 (Павел Потемкин).
• Add ArrowStream input and output format. #11088 (hcz).
• Default S3 credentials and custom auth headers. #11134 (Pervakov Grigorii).
• Add query performance metrics based on Linux perf_events. #9545 Andrey Skobtsov. #11226 (Alexander Kuzmenkov).
• Use HTTP client for S3 based on Poco. #11230 (Pavel Kovalenko).
• Add function fuzzBits that randomly flips bits in a string with given probability. #11237 (Andrei Nekrashevich).
• Add _timestamp_ms virtual column for Kafka engine (type is Nullable(DateTime64(3))). #11260 (filimonov).
• Add 2 more virtual columns for engine=Kafka to access message headers. #11283 (filimonov).
• Add netloc function for extracting network location, similar to urlparse(url), netloc in python. #11356 (Guillaume Tassery).
• Added syntax highligting to clickhouse-client using ReplXX. #11422 (Tagir Kuskarov).
• Add SHOW CLUSTER(S) queries. #11467 (hexiaoting).
• Add functions extractAllGroupsHorizontal(haystack, re) and extractAllGroupsVertical(haystack, re). #11554 (Vasily Nemkov).
• Add the system.asynchronous_metric_log table that logs historical metrics from system.asynchronous_metrics. #11588 (Alexander Kuzmenkov).
• minMap and maxMap functions were added. #11603 (Ildus Kurbangaliev).
• Added support for MySQL style global variables syntax (stub). This is needed for compatibility of MySQL protocol. #11832 (Alexey Milovidov).

Performance Improvement

• Optimization of GROUP BY with respect to table sorting key. #9113 (Dmitry Rubashkin).
• New optimization that takes arithmetic operations out of aggregate functions. #10047 (Ruslan).
• This feature eliminates functions of other keys in GROUP BY section. #10051 (Victor Grishanin).
• Add runtime CPU detection to select and dispatch the best function implementation. Add support for codegeneration for multiple targets. This closes #1017. #10058 (DimasKovas).
• Remove duplicate ORDER BY and DISTINCT from subqueries. #10067 (Mikhail Malafeev).
• Sort bigger parts of the left table in MergeJoin. Buffer left blocks in memory. Add partial_merge_join_left_table_buffer_bytes setting to manage the left blocks buffers sizes. #10601 (Artem Zuikov).
• Get dictionary and check access rights only once per each call of any function reading external dictionaries. #10928 (Vitaly Baranov).
• Improving radix sort by removing some redundant data moves. #10981 (Arslan Gumerov).
• Make queries with sum aggregate function and without GROUP BY keys to run multiple times faster. #10992 (Alexey Milovidov).
• Enable mlock of clickhouse binary by default. It will prevent clickhouse executable from being paged out under high IO load. #11139 (Alexey Milovidov).
• Improved performance for queries with ORDER BY and small LIMIT (less, then max_block_size). #11171 (Provet).
• Improve performance for INSERT queries via INSERT SELECT or INSERT with clickhouse-client when small blocks are generated (typical case with parallel parsing). This fixes #11275. Fix the issue that CONSTRAINTs were not working for DEFAULT fields. This fixes #11273. Fix the issue that CONSTRAINTS were ignored for TEMPORARY tables. This fixes #11274. #11276 (Alexey Milovidov).
• Improve performance of clickhouse-client in interactive mode when Pretty formats are used. In previous versions, significant amount of time can be spent calculating visible width of UTF-8 string. This closes #11323. #11323 (Alexey Milovidov).
• New optimization that takes all operations out of "any" function. #11529 (Ruslan).
• Speed up merging in AggregatingMergeTree. This fixes performance regression that was introduced more than a year ago in #4348 (in version 19.3). #11534 (Alexey Milovidov).
• Allow multiple replicas to assign merges, mutations, partition drop, move and replace concurrently. This closes #10367. #11639 (Alexey Milovidov).
• This optimization eliminates min/max/any aggregators of GROUP BY keys in SELECT section. #11667 (Victor Grishanin).

Improvement

• Allow to pass quota_key in clickhouse-client. This closes #10227. #10270 (Alexey Milovidov).
• Added system tables for users, roles, grants, settings profiles, quotas, row policies; added commands SHOW USER, SHOW [CURRENT|ENABLED] ROLES, SHOW SETTINGS PROFILES. #10387 (Vitaly Baranov).
• • Adding support for INSERT INTO [db.]table WATCH query. #10498 (vzakaznikov).
• Possibility to work with S3 through proxies. #10576 (Pavel Kovalenko).
• Added move_ttl_info to system.parts in order to provide introspection of move TTL functionality. #10591 (Vladimir Chebotarev).
• Make pointInPolygon work with non-constant polygon. PointInPolygon now can take Array(Array(Tuple(..., ...))) as second argument, array of polygon and holes. #10623 (Alexey Ilyukhov).
• Print a message if clickhouse-client is newer than clickhouse-server. #10627 (Alexey Milovidov).
• Change HTTP response code in case of some parse errors to 400 Bad Request. This fix #10636. #10640 (Alexey Milovidov).
• Added a check for meaningless codecs and a setting allow_suspicious_codecs to control this check. This closes #4966. #10645 (Alexey Milovidov).
• Adding support for ALTER RENAME COLUMN query to Distributed table engine. #10727 (vzakaznikov).
• Possibility to configure proxy-resolver for DiskS3. #10744 (Pavel Kovalenko).
• Better DNS exception message. This fixes #10813. #10828 (Alexey Milovidov).
• Ensure that varSamp, varPop cannot return negative results due to numerical errors and that stddevSamp, stddevPop cannot be calculated from negative variance. This fixes #10532. #10829 (Alexey Milovidov).
• Get rid of exception from replicated queue during server shutdown. Fixes #10819. #10841 (alesapin).
• The clickhouse-format tool is now able to format multiple queries when the -n argument is used. #10852 (Darío).
• Provide synonims for some data types. #10856 (Павел Потемкин).
• Introduce min_insert_block_size_rows_for_materialized_views , min_insert_block_size_bytes_for_materialized_views settings. This settings are similar to min_insert_block_size_rows and min_insert_block_size_bytes, but applied only for blocks inserted into MATERIALIZED VIEW. It helps to control blocks squashing while pushing to MVs and avoid excessive memory usage. #10858 (Azat Khuzhin).
• Respect prefer_localhost_replica/load_balancing on INSERT into Distributed. #10867 (Azat Khuzhin).
• Allow large UInt types as the index in function tupleElement. #10874 (hcz).
• Support for unicode whitespaces in queries. This helps when queries are copy-pasted from Word or from web page. This fixes #10896. #10903 (Alexey Milovidov).
• Some additions and cleanup for #10232. #10934 (Artem Zuikov).
• Set thread names for internal threads of rdkafka library. Make logs from rdkafka available in server logs. #10983 (Azat Khuzhin).
• Remove data on explicit DROP DATABASE for Memory database engine. Fixes #10557. #11021 (Alexander Tokmakov).
• Add NCHAR and NVARCHAR synonims for data types. #11025 (Alexey Milovidov).
• Apply TTL for old data, after ALTER MODIFY TTL query. This behaviour is controlled by setting materialize_ttl_after_modify, which is enabled by default. #11042 (Anton Popov).
• Now dictGet* functions accept table names. #11050 (Vitaly Baranov).
• Show authentication type in table system.users and while executing SHOW CREATE USER query. #11080 (Vitaly Baranov).
• Enable percpu_arena:percpu for jemalloc (This will reduce memory fragmentation due to thread pool). #11084 (Azat Khuzhin).
• Add port() function (to extract port from URL). #11120 (Azat Khuzhin).
• Resolved #7224: added FailedQuery, FailedSelectQuery and FailedInsertQuery metrics to system.events table. #11151 (Nikita Orlov).
• The query log is now enabled by default. #11184 (Ivan Blinkov).
• When parsing C-style backslash escapes in string literals, VALUES and various text formats (this is an extension to SQL standard that is endemic for ClickHouse and MySQL), keep backslash if unknown escape sequence is found (e.g. \% or \w) that will make usage of LIKE and match regular expressions more convenient (it's enough to write name LIKE 'used\_cars' instead of name LIKE 'used\\_cars') and more compatible at the same time. This fixes #10922. #11208 (Alexey Milovidov).
• Add support for multi-word data type names (such as DOUBLE PRECISION and CHAR VARYING) for better SQL compatibility. #11214 (Павел Потемкин).
• Keep the value of DistributedFilesToInsert metric on exceptions. In previous versions, the value was set when we are going to send some files, but it is zero, if there was an exception and some files are still pending. Now it corresponds to the number of pending files in filesystem. #11220 (Alexey Milovidov).
• Support kafka_client_id parameter for Kafka tables. It also changes the default client.id used by ClickHouse when communicating with Kafka to be more verbose and usable. #11252 (filimonov).
• Update librdkafka to version 1.4.2. #11256 (filimonov).
• Support (U)Int8, (U)Int16, Date in ASOF JOIN. #11301 (Artem Zuikov).
• Better exception message in case when there is shortage of memory mappings. This closes #11027. #11316 (Alexey Milovidov).
• Add setting "output_format_pretty_max_value_width". If value is longer, it will be cut to avoid output of too large values in terminal. This closes #11140. #11324 (Alexey Milovidov).
• Remove trailing whitespaces from formatted queries in clickhouse-client or clickhouse-format in some cases. #11325 (Alexey Milovidov).
• Better log messages in while reloading configuration. #11341 (Alexey Milovidov).
• Suppress output of cancelled queries in clickhouse-client. In previous versions result may continue to print in terminal even after you press Ctrl+C to cancel query. This closes #9473. #11342 (Alexey Milovidov).
• OPTIMIZE FINAL will force merge even if concurrent merges are performed. This closes #11309 and closes #11322. #11346 (Alexey Milovidov).
• Support for all format settings in Kafka, expose some setting on table level, adjust the defaults for better performance. #11388 (filimonov).
• Add system.distribution_queue table. #11394 (Azat Khuzhin).
• ON CLUSTER support for SYSTEM {FLUSH DISTRIBUTED,STOP/START DISTRIBUTED SEND}. #11415 (Azat Khuzhin).
• Now history file is updated after each query and there is no race condition if multiple clients use one history file. This fixes #9897. #11453 (Tagir Kuskarov).
• Automatically update DNS cache, which is used to check if user is allowed to connect from an address. #11487 (Alexander Tokmakov).
• Adding support for PREWHERE in live view tables. #11495 (vzakaznikov).
• Improve enable_optimize_predicate_expression=1 logic for VIEW. #11513 (Artem Zuikov).
• Better exception message when cannot parse columns declaration list. This closes #10403. #11537 (Alexey Milovidov).
• Optimize memory usage when reading a response from an S3 HTTP client. #11561 (Pavel Kovalenko).
• Improve multiple_joins_rewriter_version=2 logic. Fix unknown columns error for lambda aliases. #11587 (Artem Zuikov).
• Make more input format work with Kafka engine. Fix the issue with premature flushes. Fix the performance issue when kafka_num_consumers is greater than number of partitions in topic. #11599 (filimonov).
• https://github.com/ClickHouse/ClickHouse/pull/7572#issuecomment-642815377 Support config default HTTPHandlers. #11628 (Winter Zhang).
• Add round_robin load_balancing. #11645 (Azat Khuzhin).
• Allow comparison of numbers with constant string in comparison operators, IN and VALUES sections. #11647 (Alexey Milovidov).
• Allow comparison with constant strings by implicit conversions when analysing index conditions on other types. This may close #11630. #11648 (Alexey Milovidov).
• Allow using groupArrayArray and groupUniqArrayArray as SimpleAggregateFunction. #11650 (Volodymyr Kuznetsov).
• Skip empty parameters in requested URL. They may appear when you write http://localhost:8123/?&a=b or http://localhost:8123/?a=b&&c=d. This closes #10749. #11651 (Alexey Milovidov).
• Allow to DROP replicated table if the metadata in ZooKeeper was already removed and does not exist (this is also the case when using TestKeeper for testing and the server was restarted). Allow to RENAME replicated table even if there is an error communicating with ZooKeeper. This fixes #10720. #11652 (Alexey Milovidov).
• Return NULL/zero when value is not parsed completely in parseDateTimeBestEffortOrNull/Zero functions. This fixes #7876. #11653 (Alexey Milovidov).
• Added column position to system.columns table and column_position to system.parts_columns table. It contains ordinal position of a column in a table starting with 1. This closes #7744. #11655 (Alexey Milovidov).
• Don't use debug info from ELF file if it doesn't correspond to the running binary. It is needed to avoid printing wrong function names and source locations in stack traces. This fixes #7514. #11657 (Alexey Milovidov).
• Clear password from command line in clickhouse-client and clickhouse-benchmark if the user has specified it with explicit value. This prevents password exposure by ps and similar tools. #11665 (Alexey Milovidov).
• Multiple names are now allowed in commands: CREATE USER, CREATE ROLE, ALTER USER, SHOW CREATE USER, SHOW GRANTS and so on. #11670 (Vitaly Baranov).
• When multiline query is printed to server log, the lines are joined. Make it to work correct in case of multiline string literals, identifiers and single-line comments. This fixes #3853. #11686 (Alexey Milovidov).
• Emit warning instead of error in server log at startup if we cannot listen one of the listen addresses (e.g. IPv6 is unavailable inside Docker). Note that if server fails to listen all listed addresses, it will refuse to startup as before. This fixes #4406. #11687 (Alexey Milovidov).
• Added support for distributed DDL (update/delete/drop partition) on cross replication clusters. #11703 (Nikita Mikhaylov).
• Add cast_keep_nullable setting. If set CAST(something_nullable AS Type) return Nullable(Type). #11733 (Artem Zuikov).
• Add more jemalloc statistics to system.asynchronous_metrics, and ensure that we see up-to-date values for them. #11748 (Alexander Kuzmenkov).
• Add ability to set MATERIALIZED default type for primary key columns and columns with secondary indices. #11786 (alesapin).
• Remove leader election, step 3: remove yielding of leadership; remove sending queries to leader. #11795 (Alexey Milovidov).
• Added hostname as an alias to function hostName. This feature was suggested by Victor Tarnavskiy from Yandex.Metrica. #11821 (Alexey Milovidov).
• Fix sleep invocation in signal handler. It was sleeping for less amount of time than expected. #11825 (Alexey Milovidov).
• Slightly improve diagnostic of reading decimal from string. This closes #10202. #11829 (Alexey Milovidov).
• When reading Decimal value, cut extra digits after point. This behaviour is more compatible with MySQL and PostgreSQL. This fixes #10202. #11831 (Alexey Milovidov).

Bug Fix

• Fixed error with "max_rows_to_sort" limit. #10268 (Alexey Milovidov).
• Fixes: #10263 (after that PR dist send via INSERT had been postponing on each INSERT) Fixes: #8756 (that PR breaks distributed sends with all of the following conditions met (unlikely setup for now I guess): internal_replication == false, multiple local shards (activates the hardlinking code) and distributed_storage_policy (makes link(2) fails on EXDEV)). #10486 (Azat Khuzhin).
• Disable GROUP BY sharding_key optimization by default (optimize_distributed_group_by_sharding_key had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example is if in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. #10516 (Azat Khuzhin).
• Fix index corruption, which may accur in some cases after merge compact parts into another compact part. #10531 (Anton Popov).
• Implemented comparison between DateTime64 and String values (just like for DateTime). #10560 (Vasily Nemkov).
• Fix SELECT of column ALIAS which default expression type different from column type. #10563 (Azat Khuzhin).
• Fix error the BloomFilter false positive must be a double number between 0 and 1 #10551. #10569 (Winter Zhang).
• This PR fixes possible crash when createDictionary() is called before loadStoredObject() has finished. #10587 (Vitaly Baranov).
• Fixed handling condition variable for synchronous mutations. In some cases signals to that condition variable could be lost. #10588 (Vladimir Chebotarev).
• Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table. #10603 (Nikolai Kochetov).
• On SYSTEM DROP DNS CACHE query also drop caches, which are used to check if user is allowed to connect from some IP addresses. #10608 (Alexander Tokmakov).
• Fix segfault in StorageBuffer when exception on server startup. Fixes #10550. #10609 (Alexander Tokmakov).
• Fix optimize_skip_unused_shards with LowCardinality. #10611 (Azat Khuzhin).
• Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin).
• Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 (Alexey Milovidov).
• Fix bug which locks concurrent alters when table has a lot of parts. #10659 (alesapin).
• Fix possible incorrect number of rows for queries with LIMIT. Fixes #10566, #10709. #10660 (Nikolai Kochetov).
• Fix the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes #10655. #10664 (Nikolai Kochetov).
• Fix order of parameters in AggregateTransform constructor. #10667 (palasonic1).
• Fixed bug, which causes http requests stuck on client close when readonly=2 and cancel_http_readonly_queries_on_client_close=1. Fixes #7939, #7019, #7736, #7091. #10684 (Alexander Tokmakov).
• Fix multiple usages of IN operator with the identical set in one query. #10686 (Anton Popov).
• Fix atomicity of HTTP insert. This fixes #9666. #10687 (Andrew Onyshchuk).
• Fix disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes #10674. #10698 (Nikolai Kochetov).
• Fix possible buffer overflow in function h3EdgeAngle. #10711 (Alexey Milovidov).
• Fix usage of primary key wrapped into a function with 'FINAL' modifier and 'ORDER BY' optimization. #10715 (Anton Popov).
• Fix data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov).
• Fix crash in generateRandom with nested types. Fixes #10583. #10734 (Nikolai Kochetov).
• Fix combinator -OrNull and -OrDefault when combined with -State. #10741 (hcz).
• Fix parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov).
• Fix avgWeighted when using floating-point weight over multiple shards. #10758 (Baudouin Giard).
• Get rid of old libunwind patches. 500aa22791 (r39048012) This allows to disable -fno-omit-frame-pointer in clang builds that improves performance at least by 1% in average. #10761 (Amos Bird).
• Make use of src_type for correct type conversion in key conditions. Fixes #6287. #10791 (Andrew Onyshchuk).
• Fix UBSan and MSan report in DateLUT. #10798 (Alexey Milovidov).
• Fix the issue with ODBC bridge when no quoting of identifiers is requested. This fixes #7984. #10821 (Alexey Milovidov).
• Fix columns order after Block::sortColumns() (also add a test that shows that it affects some real use case - Buffer engine). #10826 (Azat Khuzhin).
• Fix potential read of uninitialized memory in cache dictionary. #10834 (Alexey Milovidov).
• Now constraints are updated if the column participating in CONSTRAINT expression was renamed. Fixes #10844. #10847 (alesapin).
• Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. #10849 (Alexander Tokmakov).
• Fixed WATCH hangs after LiveView table was dropped from database with Atomic engine. #10859 (Alexander Tokmakov).
• Fix SIGSEGV in StringHashTable (if such key does not exist). #10870 (Azat Khuzhin).
• Fix backward compatibility with tuples in Distributed tables. #10889 (Anton Popov).
• Fix possible race which could happen when you get result from aggregate function state from multiple thread for the same column. The only way (which I found) it can happen is when you use finalizeAggregation function while reading from table with Memory engine which stores AggregateFunction state for quanite* function. #10890 (Nikolai Kochetov).
• Now it's possible to execute multiple ALTER RENAME like a TO b, c TO a. #10895 (alesapin).
• Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #10910 (filimonov).
• Fix crash in SELECT count(notNullIn(NULL, [])). #10920 (Nikolai Kochetov).
• Avoid sending partially written files by the DistributedBlockOutputStream. #10940 (Azat Khuzhin).
• Fix incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 (Alexey Milovidov).
• Fix incorrect raw data size in method getRawData(). #10964 (Igr).
• Fix server crash on concurrent ALTER and DROP DATABASE queries with Atomic database engine. #10968 (Alexander Tokmakov).
• Fix metadata (relative path for rename) and data (relative path for symlink) handling for Atomic database. #10980 (Azat Khuzhin).
• Fix very rare potential use-after-free error in MergeTree if table was not created successfully. #10986 (Alexey Milovidov).
• Fix memory tracking for two-level GROUP BY when not all rows read from Aggregator (TCP). #11022 (Azat Khuzhin).
• Fixed parsing of S3 URLs. #11036 (Vladimir Chebotarev).
• Now it's possible to ADD/DROP and RENAME the same one column in a single ALTER query. Exception message for simultaneous MODIFY and RENAME became more clear. Partially fixes #10669. #11037 (alesapin).
• Fixed parseDateTime64BestEffort argument resolution bugs. #10925. #11038 (Vasily Nemkov).
• Fixes the potential missed data during termination of Kafka engine table. #11048 (filimonov).
• Fix error No such name in Block::erase() when JOIN appears with PREWHERE or optimize_move_to_prewhere makes PREWHERE from WHERE. #11051 (Artem Zuikov).
• Fixed memory leak in registerDiskS3. #11074 (Pavel Kovalenko).
• Fixed deadlock during server startup after update with changes in structure of system log tables. #11106 (alesapin).
• Remove logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
• Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 (Azat Khuzhin).
• Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #11145 (filimonov).
• Fix Kafka performance issue related to reschedules based on limits, which were always applied. #11149 (filimonov).
• If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
• Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 (Vladimir Chebotarev).
• Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. #11185 (Nikolai Kochetov).
• Now primary.idx will be checked if it's defined in CREATE query. #11199 (alesapin).
• Fix possible exception Invalid status for associated output. #11200 (Nikolai Kochetov).
• Fix error Block structure mismatch in QueryPipeline while reading from VIEW with constants in inner query. Fixes #11181. #11205 (Nikolai Kochetov).
• Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 (Nikita Mikhaylov).
• Fix crash while reading malformed data in Protobuf format. This fixes #5957, fixes #11203. #11258 (Vitaly Baranov).
• Fix wrong markup in documentation. #11263 (Alexey Milovidov).
• Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes #10586. #11278 (Vitaly Baranov).
• Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. #11296 (Alexey Milovidov).
• Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes #11286. #11303 (Alexey Milovidov).
• Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). #11311 (Alexey Milovidov).
• Fix insignificant data race in clickhouse-copier. Found by integration tests. #11313 (Alexey Milovidov).
• Fix very rare race condition in ThreadPool. #11314 (Alexey Milovidov).
• Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 (Ewout).
• Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes #10241. #11330 (Nikolai Kochetov).
• Now merges stopped before change metadata in ALTER queries. #11335 (alesapin).
• Fix crash in quantilesExactWeightedArray. #11337 (Nikolai Kochetov).
• Fix crash in direct selects from StorageJoin (without JOIN) and wrong nullability. #11340 (Artem Zuikov).
• Fix potential segfault when using Lazy database. #11348 (Alexey Milovidov).
• Fix support for \G (vertical output) in clickhouse-client in multiline mode. This closes #9933. #11350 (Alexey Milovidov).
• Remove redundant lock during parts send in ReplicatedMergeTree. #11354 (alesapin).
• Fix possible Pipeline stuck error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov).
• Better errors for joinGet() functions. #11389 (Artem Zuikov).
• Fixed geohashesInBox with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov).
• Fix error code for wrong USING key. #11373. #11404 (Artem Zuikov).
• Fix crash in JOIN over LowCarinality(T) and Nullable(T). #11380. #11414 (Artem Zuikov).
• Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. #11420 (Alexey Milovidov).
• Fix pointInPolygon with nan as point. Fixes #11375. #11421 (Alexey Ilyukhov).
• Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
• Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
• Fix wrong result in queries like select count() from t, u. #11454 (Artem Zuikov).
• Fix Pipeline stuck exception for INSERT SELECT FINAL where SELECT (max_threads>1) has multiple streams but INSERT has only one (max_insert_threads==0). #11455 (Azat Khuzhin).
• Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. #11496 (Alexey Milovidov).
• Fix race condition which may lead to an exception during table drop. It's a bit tricky and not dangerous at all. If you want an explanation, just notice me in telegram. #11523 (alesapin).
• Fix async INSERT into Distributed for prefer_localhost_replica=0 and w/o internal_replication. #11527 (Azat Khuzhin).
• Fix shard_num/replica_num for <node> (breaks use_compact_format_in_distributed_parts_names). #11528 (Azat Khuzhin).
• Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (Alexey Milovidov).
• Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
• All queries in HTTP session have had the same query_id. It is fixed. #11578 (Alexander Tokmakov).
• Fixed rare segfault in SHOW CREATE TABLE Fixes #11490. #11579 (Alexander Tokmakov).
• Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. #11580 (Nikolai Kochetov).
• Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (Alexey Milovidov).
• Fix race conditions in CREATE/DROP of different replicas of ReplicatedMergeTree. Continue to work if the table was not removed completely from ZooKeeper or not created successfully. This fixes #11432. #11592 (Alexey Milovidov).
• Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. #11601 (filimonov).
• Fix error Block structure mismatch for queries with sampling reading from Buffer table. #11602 (Nikolai Kochetov).
• without -q option the database does not get created at startup. #11604 (giordyb).
• Fix rare crash caused by using Nullable column in prewhere condition. (Probably it is connected with #11572 somehow). #11608 (Nikolai Kochetov).
• Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
• Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. #11644 (Alexey Milovidov).
• Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (Alexey Milovidov).
• Fix syntax hilite in CREATE USER query. #11664 (Alexey Milovidov).
• Fix error which leads to an incorrect state of system.mutations. It may show that whole mutation is already done but the server still has MUTATE_PART tasks in the replication queue and tries to execute them. This fixes #11611. #11681 (alesapin).
• Fix possible Pipeline stuck for selects with parallel FINAL. Fixes #11636. #11682 (Nikolai Kochetov).
• Fix LIMIT n WITH TIES usage together with ORDER BY statement, which contains aliases. #11689 (Anton Popov).
• Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (Alexey Milovidov).
• Fix very rare race condition in SYSTEM SYNC REPLICA. If the replicated table is created and at the same time from the separate connection another client is issuing SYSTEM SYNC REPLICA command on that table (this is unlikely, because another client should be aware that the table is created), it's possible to get nullptr dereference. #11691 (Alexey Milovidov).
• Fix ORDER BY ... WITH FILL over const columns. #11697 (Anton Popov).
• Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (Alexander Tokmakov).
• Parse metadata stored in zookeeper before checking for equality. #11739 (Azat Khuzhin).
• Now replicated fetches will be cancelled during metadata alter. #11744 (alesapin).
• Fixes crash in special generated queries when optimize_arithmetic_operations_in_aggregate_functions = 1. #11756 (Ruslan).
• Fixed Scalar doesn't exist exception when using WITH <scalar subquery> ... in SELECT ... FROM merge_tree_table ... #11621. #11767 (Amos Bird).
• Fix using too many threads for queries. #11788 (Nikolai Kochetov).
• Preserve column alias with optimize_aggregators_of_group_by_keys (optimize_aggregators_of_group_by_keys has been introduced in #11667). #11806 (Azat Khuzhin).
• Fix wrong result of comparison of FixedString with constant String. This fixes #11393. This bug appeared in version 20.4. #11828 (Alexey Milovidov).
• Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (Alexey Milovidov).

Build/Testing/Packaging Improvement

• Add new build for query tests using pytest framework. #10039 (Ivan).
• Fix FreeBSD build. #10150 (Ivan).
• Fix UBSan report in Decimal parse. This fixes #7540. #10512 (Alexey Milovidov).
• Adding fuzzers and preparing for oss-fuzz integration. #10546 (kyprizel).
• Enable ThinLTO for clang builds, continuation of https://github.com/ClickHouse/ClickHouse/pull/10435. #10585 (Amos Bird).
• Increasing timeout when opening a client in tests/queries/0_stateless/helpers/client.py. #10599 (vzakaznikov).
• Fixing hard coded timeouts in new live view tests. #10604 (vzakaznikov).
• Lower memory usage in tests. It may fix the issue that "address sanitizer is out of memory" in stress test. #10617 (Alexey Milovidov).
• Fix capnproto version check for capnp::UnalignedFlatArrayMessageReader. #10618 (Matwey V. Kornilov).
• Added auto-generated machine-readable file with list of stable versions. #10628 (Alexey Milovidov).
• Update LZ4 to the latest dev branch. It may fix the error under UBSan. #10630 (Alexey Milovidov).
• Fix UBSan report in LZ4 library. #10631 (Alexey Milovidov).
• Allow to use lld to link blobs (resources). #10632 (Alexey Milovidov).
• Remove external call to ld (bfd) linker during tzdata processing in compile time. #10634 (alesapin).
• Fix UBSan report (adding zero to nullptr) in HashTable that appeared after migration to clang-10. #10638 (Alexey Milovidov).
• Fix OOM in ASan stress test. #10646 (Alexey Milovidov).
• Fixing and re-enabling 00979_live_view_watch_continuous_aggregates.py test. #10658 (vzakaznikov).
• Update zstd to 1.4.4. It has some minor improvements in performance and compression ratio. If you run replicas with different versions of ClickHouse you may see reasonable error messages Data after merge is not byte-identical to data on another replicas. with explanation. These messages are Ok and you should not worry. #10663 (Alexey Milovidov).
• Volumes and storages refactoring. #10666 (Gleb Novikov).
• Trying to fix tests/queries/0_stateless/01246_insert_into_watch_live_view.py test. #10670 (vzakaznikov).
• Update instruction to install RPM packages. This was suggested by Denis (TG login @ldviolet) and implemented by Arkady Shejn. #10707 (Alexey Milovidov).
• Update cross-builds to use clang-10 compiler. #10724 (Ivan).
• Fix performance test errors. #10766 (Alexey Milovidov).
• Fix performance tests errors, part 2. #10773 (Alexey Milovidov).
• Restore a patch that was accidentially deleted in #10396. #10774 (Alexey Milovidov).
• Suppressions of warnings from libraries was mistakenly declared as public in #10396. #10776 (Alexey Milovidov).
• Enable extra warnings for base, utils, programs. #10779 (Alexey Milovidov).
• GRPC make couldn't find protobuf files, changed make file by adding the right link. #10794 (mnkonkova).
• Add MSan suppression for MariaDB Client library. #10800 (Alexey Milovidov).
• Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in #9968. #10801 (Alexey Milovidov).
• Fix obvious race condition in "Split build smoke test" check. #10820 (Alexey Milovidov).
• Better cooperation with sanitizers. Print information about query_id in the message of sanitizer failure. #10832 (Alexey Milovidov).
• Added more asserts in columns code. #10833 (Alexey Milovidov).
• Prepare to add MSan and UBSan stress tests. #10871 (Alexey Milovidov).
• Database is recreated for every test. This improves separation of tests. #10902 (Alexey Milovidov).
• Added a test for empty external data. #10926 (Alexey Milovidov).
• Return tzdata to build images and as dependency to .deb package. #10929 (alesapin).
• Fix non-deterministic test. #10989 (Alexey Milovidov).
• Wait for odbc-bridge with exponential backoff. Previous wait time of 200 ms was not enough in our CI environment. #10990 (Alexey Milovidov).
• Enable clang-tidy for programs and utils. #10991 (Alexey Milovidov).
• Add ability to run zookeeper in integration tests over tmpfs. #11002 (alesapin).
• Fixing 00979_live_view_watch_continuous_aggregates test. #11024 (vzakaznikov).
• Make system_tables_lazy_load false by default. #11029 (Azat Khuzhin).
• Add performance test for non-constant polygons. #11141 (Alexey Milovidov).
• Create root S3 bucket for tests before any CH instance is started. #11142 (Pavel Kovalenko).
• Enable performance test that was not working. #11158 (Alexey Milovidov).
• Improve build scripts for protobuf & gRPC. #11172 (Vitaly Baranov).
• Split /programs/server into actual program and library. #11186 (Ivan).
• Now parts of linker command for cctz library will not be shuffled with other libraries. #11213 (alesapin).
• Fix several non significant errors in unit tests. #11262 (alesapin).
• Add a test for Join table engine from @donmikel. This closes #9158. #11265 (Alexey Milovidov).
• Repeat test in CI if curl invocation was timed out. It is possible due to system hangups for 10+ seconds that are typical in our CI infrastructure. This fixes #11267. #11268 (Alexey Milovidov).
• Fix potentially flacky test 00731_long_merge_tree_select_opened_files.sh. It does not fail frequently but we have discovered potential race condition in this test while experimenting with ThreadFuzzer: #9814 See link for the example. #11270 (Alexey Milovidov).
• Now clickhouse-test check the server aliveness before tests run. #11285 (alesapin).
• Emit a warning if server was build in debug or with sanitizers. #11304 (Alexey Milovidov).
• Better check for hung queries in clickhouse-test. #11321 (Alexey Milovidov).
• Remove redundant timeout from integration test test_insertion_sync_fails_with_timeout. #11343 (alesapin).
• Add support for unit tests run with UBSan. #11345 (Alexey Milovidov).
• Fix several flaky integration tests. #11355 (alesapin).
• Refactor CMake build files. #11390 (Ivan).
• Leave only unit_tests_dbms in deb build. #11429 (Ilya Yatsishin).
• Increase ccache size for builds in CI. #11450 (alesapin).
• Speed up build by removing old example programs. Also found some orphan functional test. #11486 (Alexey Milovidov).
• Fix missed include for std::move used at line 17. #11525 (Matwey V. Kornilov).
• Added a random sampling of instances where copier is executed. It is needed to avoid Too many simultaneous queries error. Also increased timeout and decreased fault probability. #11573 (Nikita Mikhaylov).
• Removes unused imports from HTTPHandlerFactory. #11660 (Bharat Nallan).
• Don't allow tests with "fail" substring in their names because it makes looking at the tests results in browser less convenient when you type Ctrl+F and search for "fail". #11817 (Alexey Milovidov).
• Added a test to ensure that mutations continue to work after FREEZE query. #11820 (Alexey Milovidov).
• Print compiler build id in crash messages. It will make us slightly more certain about what binary has crashed. Added new function buildId. #11824 (Alexey Milovidov).
• Remove dependency on tzdata: do not fail if /usr/share/zoneinfo directory does not exist. Note that all timezones work in ClickHouse even without tzdata installed in system. #11827 (Alexey Milovidov).

NO CL CATEGORY

• • Not for changelog. #10985 (Alexey Milovidov).

NO CL ENTRY

• NO CL ENTRY: 'Bump numpy from 1.18.3 to 1.18.4 in /docs/tools'. #10648 (dependabot-preview[bot]).
• NO CL ENTRY: '[ImgBot] Optimize images'. #10796 (imgbot[bot]).
• NO CL ENTRY: 'Bump mkdocs from 1.1 to 1.1.1 in /docs/tools'. #10877 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.6 to 0.4.7 in /docs/tools'. #10878 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump mkdocs from 1.1.1 to 1.1.2 in /docs/tools'. #10938 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump protobuf from 3.11.3 to 3.12.0 in /docs/tools'. #10995 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump beautifulsoup4 from 4.9.0 to 4.9.1 in /docs/tools'. #10996 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump soupsieve from 2.0 to 2.0.1 in /docs/tools'. #10997 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.7 to 0.4.9 in /docs/tools'. #11064 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump protobuf from 3.12.0 to 3.12.1 in /docs/tools'. #11093 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump six from 1.14.0 to 1.15.0 in /docs/tools'. #11129 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump protobuf from 3.12.1 to 3.12.2 in /docs/tools'. #11241 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump numpy from 1.18.4 to 1.18.5 in /docs/tools'. #11427 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump livereload from 2.6.1 to 2.6.2 in /docs/tools'. #11502 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump certifi from 2020.4.5.1 to 2020.4.5.2 in /docs/tools'. #11503 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump googletrans from 2.4.0 to 3.0.0 in /docs/tools'. #11675 (dependabot-preview[bot]).
• NO CL ENTRY: 'Bump requests from 2.23.0 to 2.24.0 in /docs/tools'. #11750 (dependabot-preview[bot]).

New Feature: function randomStringUTF8

• Added function randomStringUTF8. #10972 (Andrei Nekrashevich).