ClickHouse/CHANGELOG.md

ClickHouse release 21.1

ClickHouse release v21.1.3.32-stable, 2021-02-03

Bug Fix

• BloomFilter index crash fix. Fixes #19757. #19884 (Maksim Kita).
• Fix crash when pushing down predicates to union distinct subquery. This fixes #19855. #19861 (Amos Bird).
• Fix filtering by UInt8 greater than 127. #19799 (Anton Popov).
• In previous versions, unusual arguments for function arrayEnumerateUniq may cause crash or infinite loop. This closes #19787. #19788 (alexey-milovidov).
• Fixed stack overflow when using accurate comparison of arithmetic type with string type. #19773 (tavplubix).
• Fix crash when nested column name was used in WHERE or PREWHERE. Fixes #19755. #19763 (Nikolai Kochetov).
• Fix a segmentation fault in bitmapAndnot function. Fixes #19668. #19713 (Maksim Kita).
• Some functions with big integers may cause segfault. Big integers is experimental feature. This closes #19667. #19672 (alexey-milovidov).
• Fix wrong result of function neighbor for LowCardinality argument. Fixes #10333. #19617 (Nikolai Kochetov).
• Fix use-after-free of the CompressedWriteBuffer in Connection after disconnect. #19599 (Azat Khuzhin).
• DROP/DETACH TABLE table ON CLUSTER cluster SYNC query might hang, it's fixed. Fixes #19568. #19572 (tavplubix).
• Query CREATE DICTIONARY id expression fix. #19571 (Maksim Kita).
• Fix SIGSEGV with merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read=0/UINT64_MAX. #19528 (Azat Khuzhin).
• Buffer overflow (on memory read) was possible if addMonth function was called with specifically crafted arguments. This fixes #19441. This fixes #19413. #19472 (alexey-milovidov).
• Uninitialized memory read was possible in encrypt/decrypt functions if empty string was passed as IV. This closes #19391. #19397 (alexey-milovidov).
• Fix possible buffer overflow in Uber H3 library. See https://github.com/uber/h3/issues/392. This closes #19219. #19383 (alexey-milovidov).
• Fix system.parts _state column (LOGICAL_ERROR when querying this column, due to incorrect order). #19346 (Azat Khuzhin).
• Fixed possible wrong result or segfault on aggregation when Materialized View and its target table have different structure. Fixes #18063. #19322 (tavplubix).
• Fix error Cannot convert column now64() because it is constant but values of constants are different in source and result. Continuation of #7156. #19316 (Nikolai Kochetov).
• Fix bug when concurrent ALTER and DROP queries may hang while processing ReplicatedMergeTree table. #19237 (alesapin).
• Fixed There is no checkpoint error when inserting data through http interface using Template or CustomSeparated format. Fixes #19021. #19072 (tavplubix).
• Disable constant folding for subqueries on the analysis stage, when the result cannot be calculated. #18446 (Azat Khuzhin).
• Mutation might hang waiting for some non-existent part after MOVE or REPLACE PARTITION or, in rare cases, after DETACH or DROP PARTITION. It's fixed. #15537 (tavplubix).

ClickHouse release v21.1.2.15-stable 2021-01-18

Backward Incompatible Change

• The setting input_format_null_as_default is enabled by default. #17525 (alexey-milovidov).
• Check settings constraints for profile settings from config. Server will fail to start if users.xml contain settings that do not meet corresponding constraints. #18486 (tavplubix).
• Restrict ALTER MODIFY SETTING from changing storage settings that affects data parts (write_final_mark and enable_mixed_granularity_parts). #18306 (Amos Bird).
• Set insert_quorum_parallel to 1 by default. It is significantly more convenient to use than "sequential" quorum inserts. But if you rely to sequential consistency, you should set the setting back to zero. #17567 (alexey-milovidov).
• Remove sumburConsistentHash function. This closes #18120. #18656 (alexey-milovidov).
• Removed aggregate functions timeSeriesGroupSum, timeSeriesGroupRateSum because a friend of mine said they never worked. This fixes #16869. If you have luck using these functions, write a email to clickhouse-feedback@yandex-team.com. #17423 (alexey-milovidov).
• Prohibit toUnixTimestamp(Date()) (before it just returns UInt16 representation of Date). #17376 (Azat Khuzhin).
• Allow using extended integer types (Int128, Int256, UInt256) in avg and avgWeighted functions. Also allow using different types (integer, decimal, floating point) for value and for weight in avgWeighted function. This is a backward-incompatible change: now the avg and avgWeighted functions always return Float64 (as documented). Before this change the return type for Decimal arguments was also Decimal. #15419 (Mike).
• Expression toUUID(N) no longer works. Replace with toUUID('00000000-0000-0000-0000-000000000000'). This change is motivated by non-obvious results of toUUID(N) where N is non zero.
• SSL Certificates with incorrect "key usage" are rejected. In previous versions they are used to work. See #19262.

New Feature

• Implement gRPC protocol in ClickHouse. #15111 (Vitaly Baranov).
• Allow to use multiple zookeeper clusters. #17070 (fastio).
• Implemented REPLACE TABLE and CREATE OR REPLACE TABLE queries. #18521 (tavplubix).
• Implement UNION DISTINCT and treat the plain UNION clause as UNION DISTINCT by default. Add a setting union_default_mode that allows to treat it as UNION ALL or require explicit mode specification. #16338 (flynn).
• Added function accurateCastOrNull. This closes #10290. Add type conversions in x IN (subquery) expressions. This closes #10266. #16724 (Maksim Kita).
• IP Dictionary supports IPv4 / IPv6 types directly. #17571 (vdimir).
• IP Dictionary supports key fetching. Resolves #18241. #18480 (vdimir).
• Add *.zst compression/decompression support for data import and export. It enables using *.zst in file() function and Content-encoding: zstd in HTTP client. This closes #16791 . #17144 (Abi Palagashvili).
• Added mannWitneyUTest, studentTTest and welchTTest aggregate functions. Refactored rankCorr a bit. #16883 (Nikita Mikhaylov).
• Add functions countMatches/countMatchesCaseInsensitive. #17459 (Azat Khuzhin).
• Implement countSubstrings()/countSubstringsCaseInsensitive()/countSubstringsCaseInsensitiveUTF8() (Count the number of substring occurrences). #17347 (Azat Khuzhin).
• Add information about used databases, tables and columns in system.query_log. Add query_kind and normalized_query_hash fields. #17726 (Amos Bird).
• Add a setting optimize_on_insert. When enabled, do the same transformation for INSERTed block of data as if merge was done on this block (e.g. Replacing, Collapsing, Aggregating...). This setting is enabled by default. This can influence Materialized View and MaterializeMySQL behaviour (see detailed description). This closes #10683. #16954 (Kruglov Pavel).
• Kerberos Authenticaiton for HDFS. #16621 (Ilya Golshtein).
• Support SHOW SETTINGS statement to show parameters in system.settings. SHOW CHANGED SETTINGS and LIKE/ILIKE clause are also supported. #18056 (Jianmei Zhang).
• Function position now supports POSITION(needle IN haystack) synax for SQL compatibility. This closes #18701. ... #18779 (Jianmei Zhang).
• Now we have a new storage setting max_partitions_to_read for tables in the MergeTree family. It limits the max number of partitions that can be accessed in one query. A user setting force_max_partition_limit is also added to enforce this constraint. #18712 (Amos Bird).
• Add query_id column to system.part_log for inserted parts. Closes #10097. #18644 (flynn).
• Allow create table as select with columns specification. Example CREATE TABLE t1 (x String) ENGINE = Memory AS SELECT 1;. #18060 (Maksim Kita).
• Added arrayMin, arrayMax, arrayAvg aggregation functions. #18032 (Maksim Kita).
• Implemented ATTACH TABLE name FROM 'path/to/data/' (col1 Type1, ... query. It creates new table with provided structure and attaches table data from provided directory in user_files. #17903 (tavplubix).
• Add mutation support for StorageMemory. This closes #9117. #15127 (flynn).
• Support syntax EXISTS DATABASE name. #18458 (Du Chuan).
• Support builtin function isIPv4String && isIPv6String like MySQL. #18349 (Du Chuan).
• Add a new setting insert_distributed_one_random_shard = 1 to allow insertion into multi-sharded distributed table without any distributed key. #18294 (Amos Bird).
• Add settings min_compress_block_size and max_compress_block_size to MergeTreeSettings, which have higher priority than the global settings and take effect when they are set. close 13890. #17867 (flynn).
• Add support for 64bit roaring bitmaps. #17858 (Andy Yang).
• Extended OPTIMIZE ... DEDUPLICATE syntax to allow explicit (or implicit with asterisk/column transformers) list of columns to check for duplicates on. ... #17846 (Vasily Nemkov).
• Added functions toModifiedJulianDay, fromModifiedJulianDay, toModifiedJulianDayOrNull, and fromModifiedJulianDayOrNull. These functions convert between Proleptic Gregorian calendar date and Modified Julian Day number. #17750 (PHO).
• Add ability to use custom TLD list: added functions firstSignificantSubdomainCustom, cutToFirstSignificantSubdomainCustom. #17748 (Azat Khuzhin).
• Add support for PROXYv1 protocol to wrap native TCP interface. Allow quotas to be keyed by proxy-forwarded IP address (applied for PROXYv1 address and for X-Forwarded-For from HTTP interface). This is useful when you provide access to ClickHouse only via trusted proxy (e.g. CloudFlare) but want to account user resources by their original IP addresses. This fixes #17268. #17707 (alexey-milovidov).
• Now clickhouse-client supports opening EDITOR to edit commands. Alt-Shift-E. #17665 (Amos Bird).
• Add function encodeXMLComponent to escape characters to place string into XML text node or attribute. #17659 (nauta).
• Introduce DETACH TABLE/VIEW ... PERMANENTLY syntax, so that after restarting the table does not reappear back automatically on restart (only by explicit request). The table can still be attached back using the short syntax ATTACH TABLE. Implements #5555. Fixes #13850. #17642 (filimonov).
• Add asynchronous metrics on total amount of rows, bytes and parts in MergeTree tables. This fix #11714. #17639 (flynn).
• Add settings limit and offset for out-of-SQL pagination: #16176 They are useful for building APIs. These two settings will affect SELECT query as if it is added like select * from (your_original_select_query) t limit xxx offset xxx;. #17633 (hexiaoting).
• Provide a new aggregator combinator : -SimpleState to build SimpleAggregateFunction types via query. It's useful for defining MaterializedView of AggregatingMergeTree engine, and will benefit projections too. #16853 (Amos Bird).
• Added queries-file parameter for clickhouse-client and clickhouse-local. #15930 (Maksim Kita).
• Added query parameter for clickhouse-benchmark. #17832 (Maksim Kita).
• EXPLAIN AST now support queries other then SELECT. #18136 (taiyang-li).

Experimental Feature

• Added functions for calculation of minHash and simHash of text n-grams and shingles. They are intended for semi-duplicate search. Also functions bitHammingDistance and tupleHammingDistance are added. #7649 (flynn).
• Add new data type Map. See #1841. First version for Map only supports String type of key and value. #15806 (hexiaoting).
• Implement alternative SQL parser based on ANTLR4 runtime and generated from EBNF grammar. #11298 (Ivan).

Performance Improvement

• New IP Dictionary implementation with lower memory consumption, improved performance for some cases, and fixed bugs. #16804 (vdimir).
• Parallel formatting for data export. #11617 (Nikita Mikhaylov).
• LDAP integration: Added verification_cooldown parameter in LDAP server connection configuration to allow caching of successful "bind" attempts for configurable period of time. #15988 (Denis Glazachev).
• Add --no-system-table option for clickhouse-local to run without system tables. This avoids initialization of DateLUT that may take noticeable amount of time (tens of milliseconds) at startup. #18899 (alexey-milovidov).
• Replace PODArray with PODArrayWithStackMemory in AggregateFunctionWindowFunnelData to improve windowFunnel function performance. #18817 (flynn).
• Don't send empty blocks to shards on synchronous INSERT into Distributed table. This closes #14571. #18775 (alexey-milovidov).
• Optimized read for StorageMemory. #18052 (Maksim Kita).
• Using Dragonbox algorithm for float to string conversion instead of ryu. This improves performance of float to string conversion significantly. #17831 (Maksim Kita).
• Speedup IPv6CIDRToRange implementation. #17569 (vdimir).
• Add remerge_sort_lowered_memory_bytes_ratio setting (If memory usage after remerge does not reduced by this ratio, remerge will be disabled). #17539 (Azat Khuzhin).
• Improve performance of AggregatingMergeTree with SimpleAggregateFunction(String) in PK. #17109 (Azat Khuzhin).
• Now the -If combinator is devirtualized, and count is properly vectorized. It is for this PR. #17043 (Amos Bird).
• Fix performance of reading from Merge tables over huge number of MergeTree tables. Fixes #7748. #16988 (Anton Popov).
• Improved performance of function repeat. #16937 (satanson).
• Slightly improved performance of float parsing. #16809 (Maksim Kita).
• Add possibility to skip merged partitions for OPTIMIZE TABLE ... FINAL. #15939 (Kruglov Pavel).
• Integrate with fast_float from Daniel Lemire to parse floating point numbers. #16787 (Maksim Kita). It is not enabled, because performance its performance is still lower than rough float parser in ClickHouse.
• Fix max_distributed_connections (affects prefer_localhost_replica = 1 and max_threads != max_distributed_connections). #17848 (Azat Khuzhin).
• Adaptive choice of single/multi part upload when sending data to S3. Single part upload is controlled by a new setting max_single_part_upload_size. #17934 (Pavel Kovalenko).
• Support for async tasks in PipelineExecutor. Initial support of async sockets for remote queries. #17868 (Nikolai Kochetov).
• Allow to use optimize_move_to_prewhere optimization with compact parts, when sizes of columns are unknown. #17330 (Anton Popov).

Improvement

• Avoid deadlock when executing INSERT SELECT into itself from a table with TinyLog or Log table engines. This closes #6802. This closes #18691. This closes #16812. This closes #14570. #15260 (alexey-milovidov).
• Support SHOW CREATE VIEW name syntax like MySQL. #18095 (Du Chuan).
• All queries of type Decimal * Float or vice versa are allowed, including aggregate ones (e.g. SELECT sum(decimal_field * 1.1) or SELECT dec_col * float_col), the result type is Float32 or Float64. #18145 (Mike).
• Improved minimal Web UI: add history; add sharing support; avoid race condition of different requests; add request in-flight and ready indicators; add favicon; detect Ctrl+Enter if textarea is not in focus. #17293 #17770 (alexey-milovidov).
• clickhouse-server didn't send close request to ZooKeeper server. #16837 (alesapin).
• Avoid server abnormal termination in case of too low memory limits (max_memory_usage = 1 / max_untracked_memory = 1). #17453 (Azat Khuzhin).
• Fix non-deterministic result of windowFunnel function in case of same timestamp for different events. #18884 (Fuwang Hu).
• Docker: Explicitly set uid / gid of clickhouse user & group to the fixed values (101) in clickhouse-server Docker images. #19096 (filimonov).
• Asynchronous INSERTs to Distributed tables: Two new settings (by analogy with MergeTree family) has been added: - fsync_after_insert - Do fsync for every inserted. Will decreases performance of inserts. - fsync_directories - Do fsync for temporary directory (that is used for async INSERT only) after all operations (writes, renames, etc.). #18864 (Azat Khuzhin).
• SYSTEM KILL command started to work in Docker. This closes #18847. #18848 (alexey-milovidov).
• Expand macros in the zk path when executing FETCH PARTITION. #18839 (fastio).
• Apply ALTER TABLE <replicated_table> ON CLUSTER MODIFY SETTING ... to all replicas. Because we don't replicate such alter commands. #18789 (Amos Bird).
• Allow column transformer EXCEPT to accept a string as regular expression matcher. This resolves #18685 . #18699 (Amos Bird).
• Fix SimpleAggregateFunction in SummingMergeTree. Now it works like AggregateFunction. In previous versions values were summed together regardless to the aggregate function. This fixes #18564 . #8052. #18637 (Amos Bird). Another fix of using SimpleAggregateFunction in SummingMergeTree. This fixes #18676 . #18677 (Amos Bird).
• Fixed assertion error inside allocator in case when last argument of function bar is NaN. Now simple ClickHouse's exception is being thrown. This fixes #17876. #18520 (Nikita Mikhaylov).
• Fix usability issue: no newline after exception message in some tools. #18444 (alexey-milovidov).
• Add ability to modify primary and partition key column type from LowCardinality(Type) to Type and vice versa. Also add an ability to modify primary key column type from EnumX to IntX type. Fixes #5604. #18362 (alesapin).
• Implement untuple field access. #18133. #18309 (hexiaoting).
• Allow to parse Array fields from CSV if it is represented as a string containing array that was serialized as nested CSV. Example: "[""Hello"", ""world"", ""42"""" TV""]" will parse as ['Hello', 'world', '42" TV']. Allow to parse array in CSV in a string without enclosing braces. Example: "'Hello', 'world', '42"" TV'" will parse as ['Hello', 'world', '42" TV']. #18271 (alexey-milovidov).
• Make better adaptive granularity calculation for merge tree wide parts. #18223 (alesapin).
• Now clickhouse install could work on Mac. The problem was that there is no procfs on this platform. #18201 (Nikita Mikhaylov).
• Better hints for SHOW ... query syntax. #18183 (Du Chuan).
• Array aggregation arrayMin, arrayMax, arraySum, arrayAvg support for Int128, Int256, UInt256. #18147 (Maksim Kita).
• Add disk to Set and Join storage settings. #18112 (Grigory Pervakov).
• Access control: Now table function merge() requires current user to have SELECT privilege on each table it receives data from. This PR fixes #16964. #18104 #17983 (Vitaly Baranov).
• Temporary tables are visible in the system tables system.tables and system.columns now only in those session where they have been created. The internal database _temporary_and_external_tables is now hidden in those system tables; temporary tables are shown as tables with empty database with the is_temporary flag set instead. #18014 (Vitaly Baranov).
• Fix clickhouse-client rendering issue when the size of terminal window changes. #18009 (Amos Bird).
• Decrease log verbosity of the events when the client drops the connection from Warning to Information. #18005 (filimonov).
• Forcibly removing empty or bad metadata files from filesystem for DiskS3. S3 is an experimental feature. #17935 (Pavel Kovalenko).
• Access control: allow_introspection_functions=0 prohibits usage of introspection functions but doesn't prohibit giving grants for them anymore (the grantee will need to set allow_introspection_functions=1 for himself to be able to use that grant). Similarly allow_ddl=0 prohibits usage of DDL commands but doesn't prohibit giving grants for them anymore. #17908 (Vitaly Baranov).
• Usability improvement: hints for column names. #17112. #17857 (fastio).
• Add diagnostic information when two merge tables try to read each other's data. #17854 (徐炘).
• Let the possibility to override timeout value for running script using the ClickHouse docker image. #17818 (Guillaume Tassery).
• Check system log tables' engine definition grammar to prevent some configuration errors. Notes that this grammar check is not semantical, that means such mistakes as non-existent columns / expression functions would be not found out util the table is created. #17739 (Du Chuan).
• Removed exception throwing at RabbitMQ table initialization if there was no connection (it will be reconnecting in the background). #17709 (Kseniia Sumarokova).
• Do not ignore server memory limits during Buffer flush. #17646 (Azat Khuzhin).
• Switch to patched version of RocksDB (from ClickHouse-Extras) to fix use-after-free error. #17643 (Nikita Mikhaylov).
• Added an offset to exception message for parallel parsing. This fixes #17457. #17641 (Nikita Mikhaylov).
• Don't throw "Too many parts" error in the middle of INSERT query. #17566 (alexey-milovidov).
• Allow query parameters in UPDATE statement of ALTER query. Fixes #10976. #17563 (alexey-milovidov).
• Query obfuscator: avoid usage of some SQL keywords for identifier names. #17526 (alexey-milovidov).
• Export current max ddl entry executed by DDLWorker via server metric. It's useful to check if DDLWorker hangs somewhere. #17464 (Amos Bird).
• Export asynchronous metrics of all servers current threads. It's useful to track down issues like this. #17463 (Amos Bird).
• Include dynamic columns like MATERIALIZED / ALIAS for wildcard query when settings asterisk_include_materialized_columns and asterisk_include_alias_columns are turned on. #17462 (Ken Chen).
• Allow specifying TTL to remove old entries from system log tables, using the <ttl> attribute in config.xml. #17438 (Du Chuan).
• Now queries coming to the server via MySQL and PostgreSQL protocols have distinctive interface types (which can be seen in the interface column of the tablesystem.query_log): 4 for MySQL, and 5 for PostgreSQL, instead of formerly used 1 which is now used for the native protocol only. #17437 (Vitaly Baranov).
• Fix parsing of SETTINGS clause of the INSERT ... SELECT ... SETTINGS query. #17414 (Azat Khuzhin).
• Correctly account memory in RadixSort. #17412 (Nikita Mikhaylov).
• Add eof check in receiveHello in server to prevent getting Attempt to read after eof exception. #17365 (Kruglov Pavel).
• Avoid possible stack overflow in bigint conversion. Big integers are experimental. #17269 (flynn).
• Now set indices will work with GLOBAL IN. This fixes #17232 , #5576 . #17253 (Amos Bird).
• Add limit for http redirects in request to S3 storage (s3_max_redirects). #17220 (ianton-ru).
• When -OrNull combinator combined -If, -Merge, -MergeState, -State combinators, we should put -OrNull in front. #16935 (flynn).
• Support HTTP proxy and HTTPS S3 endpoint configuration. #16861 (Pavel Kovalenko).
• Added proper authentication using environment, ~/.aws and AssumeRole for S3 client. #16856 (Vladimir Chebotarev).
• Add more OpenTelemetry spans. Add an example of how to export the span data to Zipkin. #16535 (Alexander Kuzmenkov).
• Cache dictionaries: Completely eliminate callbacks and locks for acquiring them. Keys are not divided into "not found" and "expired", but stored in the same map during query. #14958 (Nikita Mikhaylov).
• Fix never worked fsync_part_directory/fsync_after_insert/in_memory_parts_insert_sync (experimental feature). #18845 (Azat Khuzhin).
• Allow using Atomic engine for nested database of MaterializeMySQL engine. #14849 (tavplubix).

Bug Fix

• Fix the issue when server can stop accepting connections in very rare cases. #17542 (Amos Bird, alexey-milovidov).
• Fix index analysis of binary functions with constant argument which leads to wrong query results. This fixes #18364. #18373 (Amos Bird).
• Fix possible wrong index analysis when the types of the index comparison are different. This fixes #17122. #17145 (Amos Bird).
• Disable write with AIO during merges because it can lead to extremely rare data corruption of primary key columns during merge. #18481 (alesapin).
• Restrict merges from wide to compact parts. In case of vertical merge it led to broken result part. #18381 (Anton Popov).
• Fix possible incomplete query result while reading from MergeTree* in case of read backoff (message <Debug> MergeTreeReadPool: Will lower number of threads in logs). Was introduced in #16423. Fixes #18137. #18216 (Nikolai Kochetov).
• Fix use after free bug in rocksdb library. #18862 (sundyli).
• Fix infinite reading from file in ORC format (was introduced in #10580). Fixes #19095. #19134 (Nikolai Kochetov).
• Fix bug in merge tree data writer which can lead to marks with bigger size than fixed granularity size. Fixes #18913. #19123 (alesapin).
• Fix startup bug when clickhouse was not able to read compression codec from LowCardinality(Nullable(...)) and throws exception Attempt to read after EOF. Fixes #18340. #19101 (alesapin).
• Restrict MODIFY TTL queries for MergeTree tables created in old syntax. Previously the query succeeded, but actually it had no effect. #19064 (Anton Popov).
• Make sure groupUniqArray returns correct type for argument of Enum type. This closes #17875. #19019 (alexey-milovidov).
• Fix possible error Expected single dictionary argument for function if use function ignore with LowCardinality argument. Fixes #14275. #19016 (Nikolai Kochetov).
• Fix inserting of LowCardinality column to table with TinyLog engine. Fixes #18629. #19010 (Nikolai Kochetov).
• Join tries to materialize const columns, but our code wants them in other places. #18982 (Nikita Mikhaylov).
• Disable optimize_move_functions_out_of_any because optimization is not always correct. This closes #18051. This closes #18973. #18981 (alexey-milovidov).
• Fix possible exception QueryPipeline stream: different number of columns caused by merging of query plan's Expression steps. Fixes #18190. #18980 (Nikolai Kochetov).
• Fixed very rare deadlock at shutdown. #18977 (tavplubix).
• Fix incorrect behavior when ALTER TABLE ... DROP PART 'part_name' query removes all deduplication blocks for the whole partition. Fixes #18874. #18969 (alesapin).
• Attach partition should reset the mutation. #18804. #18935 (fastio).
• Fix issue with bitmapOrCardinality that may lead to nullptr dereference. This closes #18911. #18912 (sundyli).
• Fix possible hang at shutdown in clickhouse-local. This fixes #18891. #18893 (alexey-milovidov).
• Queries for external databases (MySQL, ODBC, JDBC) were incorrectly rewritten if there was an expression in form of x IN table. This fixes #9756. #18876 (alexey-milovidov).
• Fix *If combinator with unary function and Nullable types. #18806 (Azat Khuzhin).
• Fix the issue that asynchronous distributed INSERTs can be rejected by the server if the setting network_compression_method is globally set to non-default value. This fixes #18741. #18776 (alexey-milovidov).
• Fixed Attempt to read after eof error when trying to CAST NULL from Nullable(String) to Nullable(Decimal(P, S)). Now function CAST returns NULL when it cannot parse decimal from nullable string. Fixes #7690. #18718 (Winter Zhang).
• Fix minor issue with logging. #18717 (sundyli).
• Fix removing of empty parts in ReplicatedMergeTree tables, created with old syntax. Fixes #18582. #18614 (Anton Popov).
• Fix previous bug when date overflow with different values. Strict Date value limit to "2106-02-07", cast date > "2106-02-07" to value 0. #18565 (hexiaoting).
• Add FixedString data type support for replication from MySQL. Replication from MySQL is an experimental feature. This patch fixes #18450 Also fixes #6556. #18553 (awesomeleo).
• Fix possible Pipeline stuck error while using ORDER BY after subquery with RIGHT or FULL join. #18550 (Nikolai Kochetov).
• Fix bug which may lead to ALTER queries hung after corresponding mutation kill. Found by thread fuzzer. #18518 (alesapin).
• Proper support for 12AM in parseDateTimeBestEffort function. This fixes #18402. #18449 (vladimir-golovchenko).
• Fixed value is too short error when executing toType(...) functions (toDate, toUInt32, etc) with argument of type Nullable(String). Now such functions return NULL on parsing errors instead of throwing exception. Fixes #7673. #18445 (tavplubix).
• Fix the unexpected behaviour of SHOW TABLES. #18431 (fastio).
• Fix -SimpleState combinator generates incompatible arugment type and return type. #18404 (Amos Bird).
• Fix possible race condition in concurrent usage of Set or Join tables and selects from system.tables. #18385 (alexey-milovidov).
• Fix filling table system.settings_profile_elements. This PR fixes #18231. #18379 (Vitaly Baranov).
• Fix possible crashes in aggregate functions with combinator Distinct, while using two-level aggregation. Fixes #17682. #18365 (Anton Popov).
• Fixed issue when clickhouse-odbc-bridge process is unreachable by server on machines with dual IPv4/IPv6 stack; Fixed issue when ODBC dictionary updates are performed using malformed queries and/or cause crashes of the odbc-bridge process; Possibly closes #14489. #18278 (Denis Glazachev).
• Access control: SELECT count() FROM table now can be executed if the user has access to at least single column from a table. This PR fixes #10639. #18233 (Vitaly Baranov).
• Access control: SELECT JOIN now requires the SELECT privilege on each of the joined tables. This PR fixes #17654. #18232 (Vitaly Baranov).
• Fix key comparison between Enum and Int types. This fixes #17989. #18214 (Amos Bird).
• Replication from MySQL (experimental feature). Fixes #18186 Fixes #16372 Fix unique key convert issue in MaterializeMySQL database engine. #18211 (Winter Zhang).
• Fix inconsistency for queries with both WITH FILL and WITH TIES #17466. #18188 (hexiaoting).
• Fix inserting a row with default value in case of parsing error in the last column. Fixes #17712. #18182 (Jianmei Zhang).
• Fix Unknown setting profile error on attempt to set settings profile. #18167 (tavplubix).
• Fix error when query MODIFY COLUMN ... REMOVE TTL doesn't actually remove column TTL. #18130 (alesapin).
• Fixed std::out_of_range: basic_string in S3 URL parsing. #18059 (Vladimir Chebotarev).
• Fix comparison of DateTime64 and Date. Fixes #13804 and #11222. ... #18050 (Vasily Nemkov).
• Replication from MySQL (experimental feature): Fixes #15187 Fixes #17912 support convert MySQL prefix index for MaterializeMySQL. #17944 (Winter Zhang).
• When server log rotation was configured using logger.size parameter with numeric value larger than 2^32, the logs were not rotated properly. This is fixed. #17905 (Alexander Kuzmenkov).
• Trivial query optimization was producing wrong result if query contains ARRAY JOIN (so query is actually non trivial). #17887 (sundyli).
• Fix possible segfault in topK aggregate function. This closes #17404. #17845 (Maksim Kita).
• WAL (experimental feature): Do not restore parts from WAL if in_memory_parts_enable_wal is disabled. #17802 (detailyang).
• Exception message about max table size to drop was displayed incorrectly. #17764 (alexey-milovidov).
• Fixed possible segfault when there is not enough space when inserting into Distributed table. #17737 (tavplubix).
• Fixed problem when ClickHouse fails to resume connection to MySQL servers. #17681 (Alexander Kazakov).
• Windows: Fixed Function not implemented error when executing RENAME query in Atomic database with ClickHouse running on Windows Subsystem for Linux. Fixes #17661. #17664 (tavplubix).
• In might be determined incorrectly if cluster is circular- (cross-) replicated or not when executing ON CLUSTER query due to race condition when pool_size > 1. It's fixed. #17640 (tavplubix).
• Fix empty system.stack_trace table when server is running in daemon mode. #17630 (Amos Bird).
• Exception fmt::v7::format_error can be logged in background for MergeTree tables. This fixes #17613. #17615 (alexey-milovidov).
• When clickhouse-client is used in interactive mode with multiline queries, single line comment was erronously extended till the end of query. This fixes #13654. #17565 (alexey-milovidov).
• Fix alter query hang when the corresponding mutation was killed on the different replica. Fixes #16953. #17499 (alesapin).
• Fix issue with memory accounting when mark cache size was underestimated by clickhouse. It may happen when there are a lot of tiny files with marks. #17496 (alesapin).
• Fix ORDER BY with enabled setting optimize_redundant_functions_in_order_by. #17471 (Anton Popov).
• Fix duplicates after DISTINCT which were possible because of incorrect optimization. Fixes #17294. #17296 (li chengxiang). #17439 (Nikolai Kochetov).
• Fixed high CPU usage in background tasks of *MergeTree tables. #17416 (tavplubix).
• Fix possible crash while reading from JOIN table with LowCardinality types. Fixes #17228. #17397 (Nikolai Kochetov).
• Replication from MySQL (experimental feature): Fixes #16835 try fix miss match header with MySQL SHOW statement. #17366 (Winter Zhang).
• Fix nondeterministic functions with predicate optimizer. This fixes #17244. #17273 (Winter Zhang).
• Fix possible Unexpected packet Data received from client error for Distributed queries with LIMIT. #17254 (Azat Khuzhin).
• Fix set index invalidation when there are const columns in the subquery. This fixes #17246. #17249 (Amos Bird).
• clickhouse-copier: Fix for non-partitioned tables #15235. #17248 (Qi Chen).
• Fixed possible not-working mutations for parts stored on S3 disk (experimental feature). #17227 (Pavel Kovalenko).
• Bug fix for funciton fuzzBits, related issue: #16980. #17051 (hexiaoting).
• Fix optimize_distributed_group_by_sharding_key for query with OFFSET only. #16996 (Azat Khuzhin).
• Fix queries from Merge tables over Distributed tables with JOINs. #16993 (Azat Khuzhin).
• Fix order by optimization with monotonic functions. Fixes #16107. #16956 (Anton Popov).
• Fix incorrect comparison of types DateTime64 with different scales. Fixes #16655 ... #16952 (Vasily Nemkov).
• Fix optimization of group by with enabled setting optimize_aggregators_of_group_by_keys and joins. Fixes #12604. #16951 (Anton Popov).
• Minor fix in SHOW ACCESS query. #16866 (tavplubix).
• Fix the behaviour with enabled optimize_trivial_count_query setting with partition predicate. #16767 (Azat Khuzhin).
• Return number of affected rows for INSERT queries via MySQL wire protocol. Previously ClickHouse used to always return 0, it's fixed. Fixes #16605. #16715 (Winter Zhang).
• Fix inconsistent behavior caused by select_sequential_consistency for optimized trivial count query and system tables. #16309 (Hao Chen).
• Throw error when REPLACE column transformer operates on non existing column. #16183 (hexiaoting).
• Throw exception in case of not equi-join ON expression in RIGH|FULL JOIN. #15162 (Artem Zuikov).

Build/Testing/Packaging Improvement

• Add simple integrity check for ClickHouse binary. It allows to detect corruption due to faulty hardware (bit rot on storage media or bit flips in RAM). #18811 (alexey-milovidov).
• Change OpenSSL to BoringSSL. It allows to avoid issues with sanitizers. This fixes #12490. This fixes #17502. This fixes #12952. #18129 (alexey-milovidov).
• Simplify Sys/V init script. It was not working on Ubuntu 12.04 or older. #17428 (alexey-milovidov).
• Multiple improvements in ./clickhouse install script. #17421 (alexey-milovidov).
• Now ClickHouse can pretend to be a fake ZooKeeper. Currently, storage implementation is just stored in-memory hash-table, and server partially support ZooKeeper protocol. #16877 (alesapin).
• Fix dead list watches removal for TestKeeperStorage (a mock for ZooKeeper). #18065 (alesapin).
• Add SYSTEM SUSPEND command for fault injection. It can be used to faciliate failover tests. This closes #15979. #18850 (alexey-milovidov).
• Generate build id when ClickHouse is linked with lld. It's appeared that lld does not generate it by default on my machine. Build id is used for crash reports and introspection. #18808 (alexey-milovidov).
• Fix shellcheck errors in style check. #18566 (Ilya Yatsishin).
• Update timezones info to 2020e. #18531 (alesapin).
• Fix codespell warnings. Split style checks into separate parts. Update style checks docker image. #18463 (Ilya Yatsishin).
• Automated check for leftovers of conflict markers in docs. #18332 (alexey-milovidov).
• Enable Thread Fuzzer for stateless tests flaky check. #18299 (alesapin).
• Do not use non thread-safe function strerror. #18204 (alexey-milovidov).
• Update anchore/scan-action@main workflow action (was moved from master to main). #18192 (Stig Bakken).
• Now clickhouse-test does DROP/CREATE databases with a timeout. #18098 (alesapin).
• Enable experimental support for Pytest framework for stateless tests. #17902 (Ivan).
• Now we use the fresh docker daemon version in integration tests. #17671 (alesapin).
• Send info about official build, memory, cpu and free disk space to Sentry if it is enabled. Sentry is opt-in feature to help ClickHouse developers. This closes #17279. #17543 (alexey-milovidov).
• There was an uninitialized variable in the code of clickhouse-copier. #17363 (Nikita Mikhaylov).
• Fix one MSan report from #17309. #17344 (Nikita Mikhaylov).
• Fix for the issue with IPv6 in Arrow Flight library. See the comments for details. #16664 (Zhanna).
• Add a library that replaces some libc functions to traps that will terminate the process. #16366 (alexey-milovidov).
• Provide diagnostics in server logs in case of stack overflow, send error message to clickhouse-client. This closes #14840. #16346 (alexey-milovidov).
• Now we can run almost all stateless functional tests in parallel. #15236 (alesapin).
• Fix corruption in librdkafka snappy decompression (was a problem only for gcc10 builds, but official builds uses clang already, so at least recent official releases are not affected). #18053 (Azat Khuzhin).
• If server was terminated by OOM killer, print message in log. #13516 (alexey-milovidov).
• PODArray: Avoid call to memcpy with (nullptr, 0) arguments (Fix UBSan report). This fixes #18525. #18526 (alexey-milovidov).
• Minor improvement for path concatenation of zookeeper paths inside DDLWorker. #17767 (Bharat Nallan).
• Allow to reload symbols from debug file. This PR also fixes a build-id issue. #17637 (Amos Bird).
• TestFlows: fixes to LDAP tests that fail due to slow test execution. #18790 (vzakaznikov).
• TestFlows: Merging requirements for AES encryption functions. Updating aes_encryption tests to use new requirements. Updating TestFlows version to 1.6.72. #18221 (vzakaznikov).
• TestFlows: Updating TestFlows version to the latest 1.6.72. Re-generating requirements.py. #18208 (vzakaznikov).
• TestFlows: Updating TestFlows README.md to include "How To Debug Why Test Failed" section. #17808 (vzakaznikov).
• TestFlows: tests for RBAC ACCESS MANAGEMENT privileges. #17804 (MyroTk).
• TestFlows: RBAC tests for SHOW, TRUNCATE, KILL, and OPTIMIZE. - Updates to old tests. - Resolved comments from #https://github.com/ClickHouse/ClickHouse/pull/16977. #17657 (MyroTk).
• TestFlows: Added RBAC tests for ATTACH, CREATE, DROP, and DETACH. #16977 (MyroTk).

Changelog for 2020