thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-26 09:32:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
241b897e4b
ClickHouse/CHANGELOG.md
Alexander Kuzmenkov 7dba89eaa8 changelog fixes
2020-07-07 12:52:27 +03:00

352 KiB
Raw Blame History

ClickHouse release 20.5

ClickHouse release v20.5.2.7-stable 2020-07-02

Backward Incompatible Change

  • Return non-Nullable result from COUNT(DISTINCT), and uniq aggregate functions family. If all passed values are NULL, return zero instead. This improves SQL compatibility. #11661 (alexey-milovidov).
  • Added a check for the case when user-level setting is specified in a wrong place. User-level settings should be specified in users.xml inside <profile> section for specific user profile (or in <default> for default settings). The server won't start with exception message in log. This fixes #9051. If you want to skip the check, you can either move settings to the appropriate place or add <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> to config.xml. #11449 (alexey-milovidov).
  • The setting input_format_with_names_use_header is enabled by default. It will affect parsing of input formats -WithNames and -WithNamesAndTypes. #10937 (alexey-milovidov).
  • Remove experimental_use_processors setting. It is enabled by default. #10924 (Nikolai Kochetov).
  • Update zstd to 1.4.4. It has some minor improvements in performance and compression ratio. If you run replicas with different versions of ClickHouse you may see reasonable error messages Data after merge is not byte-identical to data on another replicas. with explanation. These messages are Ok and you should not worry. This change is backward compatible but we list it here in changelog in case you will wonder about these messages. #10663 (alexey-milovidov).
  • Added a check for meaningless codecs and a setting allow_suspicious_codecs to control this check. This closes #4966. #10645 (alexey-milovidov).
  • Several Kafka setting changes their defaults. See #11388.

New Feature

  • TTL DELETE WHERE and TTL GROUP BY for automatic data coarsening and rollup in tables. #10537 (expl0si0nn).
  • Implementation of PostgreSQL wire protocol. #10242 (Movses).
  • Added system tables for users, roles, grants, settings profiles, quotas, row policies; added commands SHOW USER, SHOW [CURRENT|ENABLED] ROLES, SHOW SETTINGS PROFILES. #10387 (Vitaly Baranov).
  • Support writes in ODBC Table function #10554 (ageraab). #10901 (tavplubix).
  • Add query performance metrics based on Linux perf_events (these metrics are calculated with hardware CPU counters and OS counters). It is optional and requires CAP_SYS_ADMIN to be set on clickhouse binary. #9545 Andrey Skobtsov. #11226 (Alexander Kuzmenkov).
  • Now support NULL and NOT NULL modifiers for data types in CREATE query. #11057 (Павел Потемкин).
  • Add ArrowStream input and output format. #11088 (hcz).
  • Added a new layout direct which loads all the data directly from the source for each query, without storing or caching data. #10622 (Artem Streltsov).
  • Added new complex_key_direct layout to dictionaries, that does not store anything locally during query execution. #10850 (Artem Streltsov).
  • Added support for MySQL style global variables syntax (stub). This is needed for compatibility of MySQL protocol. #11832 (alexey-milovidov).
  • Added syntax highligting to clickhouse-client using replxx. #11422 (Tagir Kuskarov).
  • minMap and maxMap functions were added. #11603 (Ildus Kurbangaliev).
  • Add the system.asynchronous_metric_log table that logs historical metrics from system.asynchronous_metrics. #11588 (Alexander Kuzmenkov).
  • Add functions extractAllGroupsHorizontal(haystack, re) and extractAllGroupsVertical(haystack, re). #11554 (Vasily Nemkov).
  • Add SHOW CLUSTER(S) queries. #11467 (hexiaoting).
  • Add netloc function for extracting network location, similar to urlparse(url), netloc in python. #11356 (Guillaume Tassery).
  • Add 2 more virtual columns for engine=Kafka to access message headers. #11283 (filimonov).
  • Add _timestamp_ms virtual column for Kafka engine (type is Nullable(DateTime64(3))). #11260 (filimonov).
  • Add function fuzzBits that randomly flips bits in a string with given probability. #11237 (Andrei Nekrashevich).
  • Allow comparison of numbers with constant string in comparison operators, IN and VALUES sections. #11647 (alexey-milovidov).
  • Add round_robin load_balancing mode. #11645 (Azat Khuzhin).
  • Add cast_keep_nullable setting. If set CAST(something_nullable AS Type) return Nullable(Type). #11733 (Artem Zuikov).
  • Added column position to system.columns table and column_position to system.parts_columns table. It contains ordinal position of a column in a table starting with 1. This closes #7744. #11655 (alexey-milovidov).
  • ON CLUSTER support for SYSTEM {FLUSH DISTRIBUTED,STOP/START DISTRIBUTED SEND}. #11415 (Azat Khuzhin).
  • Add system.distribution_queue table. #11394 (Azat Khuzhin).
  • Support for all format settings in Kafka, expose some setting on table level, adjust the defaults for better performance. #11388 (filimonov).
  • Add port function (to extract port from URL). #11120 (Azat Khuzhin).
  • Now dictGet* functions accept table names. #11050 (Vitaly Baranov).
  • The clickhouse-format tool is now able to format multiple queries when the -n argument is used. #10852 (Darío).
  • Possibility to configure proxy-resolver for DiskS3. #10744 (Pavel Kovalenko).
  • Make pointInPolygon work with non-constant polygon. PointInPolygon now can take Array(Array(Tuple(..., ...))) as second argument, array of polygon and holes. #10623 (Alexey Ilyukhov) #11421 (Alexey Ilyukhov).
  • Added move_ttl_info to system.parts in order to provide introspection of move TTL functionality. #10591 (Vladimir Chebotarev).
  • Possibility to work with S3 through proxies. #10576 (Pavel Kovalenko).
  • Add NCHAR and NVARCHAR synonims for data types. #11025 (alexey-milovidov).
  • Resolved #7224: added FailedQuery, FailedSelectQuery and FailedInsertQuery metrics to system.events table. #11151 (Nikita Orlov).
  • Add more jemalloc statistics to system.asynchronous_metrics, and ensure that we see up-to-date values for them. #11748 (Alexander Kuzmenkov).
  • Allow to specify default S3 credentials and custom auth headers. #11134 (Grigory Pervakov).
  • Added new functions to import/export DateTime64 as Int64 with various precision: to-/fromUnixTimestamp64Milli/-Micro/-Nano. #10923 (Vasily Nemkov).
  • Allow specifying mongodb:// URI for MongoDB dictionaries. #10915 (Alexander Kuzmenkov).
  • OFFSET keyword can now be used without an affiliated LIMIT clause. #10802 (Guillaume Tassery).
  • Added system.licenses table. This table contains licenses of third-party libraries that are located in contrib directory. This closes #2890. #10795 (alexey-milovidov).
  • New function function toStartOfSecond(DateTime64) -> DateTime64 that nullifies sub-second part of DateTime64 value. #10722 (Vasily Nemkov).
  • Add new input format JSONAsString that accepts a sequence of JSON objects separated by newlines, spaces and/or commas. #10607 (Kruglov Pavel).
  • Allowed to profile memory with finer granularity steps than 4 MiB. Added sampling memory profiler to capture random allocations/deallocations. #10598 (alexey-milovidov).
  • SimpleAggregateFunction now also supports sumMap. #10000 (Ildus Kurbangaliev).
  • Support ALTER RENAME COLUMN for the distributed table engine. Continuation of #10727. Fixes #10747. #10887 (alesapin).

Bug Fix

  • Fix UBSan report in Decimal parse. This fixes #7540. #10512 (alexey-milovidov).
  • Fix potential floating point exception when parsing DateTime64. This fixes #11374. #11875 (alexey-milovidov).
  • Fix rare crash caused by using Nullable column in prewhere condition. #11895 #11608 #11869 (Nikolai Kochetov).
  • Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (alexey-milovidov).
  • Fix wrong result of comparison of FixedString with constant String. This fixes #11393. This bug appeared in version 20.4. #11828 (alexey-milovidov).
  • Fix wrong result for if with NULLs in condition. #11807 (Artem Zuikov).
  • Fix using too many threads for queries. #11788 (Nikolai Kochetov).
  • Fixed Scalar doesn't exist exception when using WITH <scalar subquery> ... in SELECT ... FROM merge_tree_table ... https://github.com/ClickHouse/ClickHouse/issues/11621. #11767 (Amos Bird).
  • Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. #11753 (hexiaoting).
  • Now replicated fetches will be cancelled during metadata alter. #11744 (alesapin).
  • Parse metadata stored in zookeeper before checking for equality. #11739 (Azat Khuzhin).
  • Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (tavplubix).
  • Fix ORDER BY ... WITH FILL over const columns. #11697 (Anton Popov).
  • Fix very rare race condition in SYSTEM SYNC REPLICA. If the replicated table is created and at the same time from the separate connection another client is issuing SYSTEM SYNC REPLICA command on that table (this is unlikely, because another client should be aware that the table is created), it's possible to get nullptr dereference. #11691 (alexey-milovidov).
  • Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (alexey-milovidov).
  • Fix LIMIT n WITH TIES usage together with ORDER BY statement, which contains aliases. #11689 (Anton Popov).
  • Fix possible Pipeline stuck for selects with parallel FINAL. Fixes #11636. #11682 (Nikolai Kochetov).
  • Fix error which leads to an incorrect state of system.mutations. It may show that whole mutation is already done but the server still has MUTATE_PART tasks in the replication queue and tries to execute them. This fixes #11611. #11681 (alesapin).
  • Fix syntax hilite in CREATE USER query. #11664 (alexey-milovidov).
  • Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (alexey-milovidov).
  • Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. #11644 (alexey-milovidov).
  • Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
  • Without -q option the database does not get created at startup. #11604 (giordyb).
  • Fix error Block structure mismatch for queries with sampling reading from Buffer table. #11602 (Nikolai Kochetov).
  • Fix wrong exit code of the clickhouse-client, when exception.code() % 256 == 0. #11601 (filimonov).
  • Fix race conditions in CREATE/DROP of different replicas of ReplicatedMergeTree. Continue to work if the table was not removed completely from ZooKeeper or not created successfully. This fixes #11432. #11592 (alexey-milovidov).
  • Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (alexey-milovidov).
  • Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. #11580 (Nikolai Kochetov).
  • Fixed rare segfault in SHOW CREATE TABLE Fixes #11490. #11579 (tavplubix).
  • All queries in HTTP session have had the same query_id. It is fixed. #11578 (tavplubix).
  • Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
  • Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (alexey-milovidov).
  • Fix shard_num/replica_num for <node> (breaks use_compact_format_in_distributed_parts_names). #11528 (Azat Khuzhin).
  • Fix async INSERT into Distributed for prefer_localhost_replica=0 and w/o internal_replication. #11527 (Azat Khuzhin).
  • Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. #11496 (alexey-milovidov).
  • Fix Pipeline stuck exception for INSERT SELECT FINAL where SELECT (max_threads>1) has multiple streams but INSERT has only one (max_insert_threads==0). #11455 (Azat Khuzhin).
  • Fix wrong result in queries like select count() from t, u. #11454 (Artem Zuikov).
  • Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
  • Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
  • Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. #11420 (alexey-milovidov).
  • Fix crash in JOIN over LowCarinality(T) and Nullable(T). #11380. #11414 (Artem Zuikov).
  • Fix error code for wrong USING key. #11373. #11404 (Artem Zuikov).
  • Fixed geohashesInBox with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov).
  • Better errors for joinGet() functions. #11389 (Artem Zuikov).
  • Fix possible Pipeline stuck error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov).
  • Remove redundant lock during parts send in ReplicatedMergeTree. #11354 (alesapin).
  • Fix support for \G (vertical output) in clickhouse-client in multiline mode. This closes #9933. #11350 (alexey-milovidov).
  • Fix potential segfault when using Lazy database. #11348 (alexey-milovidov).
  • Fix crash in direct selects from Join table engine (without JOIN) and wrong nullability. #11340 (Artem Zuikov).
  • Fix crash in quantilesExactWeightedArray. #11337 (Nikolai Kochetov).
  • Now merges stopped before change metadata in ALTER queries. #11335 (alesapin).
  • Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes #10241. #11330 (Nikolai Kochetov).
  • Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 (Ewout).
  • Fix very rare race condition in ThreadPool. #11314 (alexey-milovidov).
  • Fix insignificant data race in clickhouse-copier. Found by integration tests. #11313 (alexey-milovidov).
  • Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). #11311 (alexey-milovidov).
  • Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes #11286. #11303 (alexey-milovidov).
  • Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. #11296 (alexey-milovidov).
  • Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes https://github.com/ClickHouse/ClickHouse/issues/10586. #11278 (Vitaly Baranov).
  • Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. #11258 (Vitaly Baranov).
  • Fixed a bug when cache dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 (Nikita Mikhaylov).
  • Fix error Block structure mismatch in QueryPipeline while reading from VIEW with constants in inner query. Fixes #11181. #11205 (Nikolai Kochetov).
  • Fix possible exception Invalid status for associated output. #11200 (Nikolai Kochetov).
  • Now primary.idx will be checked if it's defined in CREATE query. #11199 (alesapin).
  • Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. #11185 (Nikolai Kochetov).
  • Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 (Vladimir Chebotarev).
  • If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
  • Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #11145 (filimonov).
  • Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 (Azat Khuzhin).
  • Remove logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
  • Fixed deadlock during server startup after update with changes in structure of system log tables. #11106 (alesapin).
  • Fixed memory leak in registerDiskS3. #11074 (Pavel Kovalenko).
  • Fix error No such name in Block::erase() when JOIN appears with PREWHERE or optimize_move_to_prewhere makes PREWHERE from WHERE. #11051 (Artem Zuikov).
  • Fixes the potential missed data during termination of Kafka engine table. #11048 (filimonov).
  • Fixed parseDateTime64BestEffort argument resolution bugs. #10925. #11038 (Vasily Nemkov).
  • Now it's possible to ADD/DROP and RENAME the same one column in a single ALTER query. Exception message for simultaneous MODIFY and RENAME became more clear. Partially fixes #10669. #11037 (alesapin).
  • Fixed parsing of S3 URLs. #11036 (Vladimir Chebotarev).
  • Fix very rare potential use-after-free error in MergeTree if table was not created successfully. #10986 (alexey-milovidov).
  • Fix metadata (relative path for rename) and data (relative path for symlink) handling for Atomic database. #10980 (Azat Khuzhin).
  • Fix server crash on concurrent ALTER and DROP DATABASE queries with Atomic database engine. #10968 (tavplubix).
  • Fix incorrect raw data size in method getRawData(). #10964 (Igr).
  • Fix incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 (alexey-milovidov).
  • Avoid sending partially written files by the DistributedBlockOutputStream. #10940 (Azat Khuzhin).
  • Fix crash in SELECT count(notNullIn(NULL, [])). #10920 (Nikolai Kochetov).
  • Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #10910 (filimonov).
  • Now it's possible to execute multiple ALTER RENAME like a TO b, c TO a. #10895 (alesapin).
  • Fix possible race which could happen when you get result from aggregate function state from multiple thread for the same column. The only way (which I found) it can happen is when you use finalizeAggregation function while reading from table with Memory engine which stores AggregateFunction state for quanite* function. #10890 (Nikolai Kochetov).
  • Fix backward compatibility with tuples in Distributed tables. #10889 (Anton Popov).
  • Fix SIGSEGV in StringHashTable (if such key does not exist). #10870 (Azat Khuzhin).
  • Fixed WATCH hangs after LiveView table was dropped from database with Atomic engine. #10859 (tavplubix).
  • Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. #10849 (tavplubix).
  • Now constraints are updated if the column participating in CONSTRAINT expression was renamed. Fixes #10844. #10847 (alesapin).
  • Fix potential read of uninitialized memory in cache dictionary. #10834 (alexey-milovidov).
  • Fix columns order after Block::sortColumns() (also add a test that shows that it affects some real use case - Buffer engine). #10826 (Azat Khuzhin).
  • Fix the issue with ODBC bridge when no quoting of identifiers is requested. This fixes #7984. #10821 (alexey-milovidov).
  • Fix UBSan and MSan report in DateLUT. #10798 (alexey-milovidov).
  • Make use of src_type for correct type conversion in key conditions. Fixes #6287. #10791 (Andrew Onyshchuk).
  • Get rid of old libunwind patches. 500aa22791 (r39048012) This allows to disable -fno-omit-frame-pointer in clang builds that improves performance at least by 1% in average. #10761 (Amos Bird).
  • Fix avgWeighted when using floating-point weight over multiple shards. #10758 (Baudouin Giard).
  • Fix parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov).
  • Fix combinator -OrNull and -OrDefault when combined with -State. #10741 (hcz).
  • Fix crash in generateRandom with nested types. Fixes #10583. #10734 (Nikolai Kochetov).
  • Fix data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov).
  • Fix usage of primary key wrapped into a function with 'FINAL' modifier and 'ORDER BY' optimization. #10715 (Anton Popov).
  • Fix possible buffer overflow in function h3EdgeAngle. #10711 (alexey-milovidov).
  • Fix disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes #10674. #10698 (Nikolai Kochetov).
  • Fix atomicity of HTTP insert. This fixes #9666. #10687 (Andrew Onyshchuk).
  • Fix multiple usages of IN operator with the identical set in one query. #10686 (Anton Popov).
  • Fixed bug, which causes http requests stuck on client close when readonly=2 and cancel_http_readonly_queries_on_client_close=1. Fixes #7939, #7019, #7736, #7091. #10684 (tavplubix).
  • Fix order of parameters in AggregateTransform constructor. #10667 (palasonic1).
  • Fix the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes #10655. #10664 (Nikolai Kochetov).
  • Fix possible incorrect number of rows for queries with LIMIT. Fixes #10566, #10709. #10660 (Nikolai Kochetov).
  • Fix bug which locks concurrent alters when table has a lot of parts. #10659 (alesapin).
  • Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 (alexey-milovidov).
  • Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin).
  • Fix optimize_skip_unused_shards with LowCardinality. #10611 (Azat Khuzhin).
  • Fix segfault in StorageBuffer when exception on server startup. Fixes #10550. #10609 (tavplubix).
  • On SYSTEM DROP DNS CACHE query also drop caches, which are used to check if user is allowed to connect from some IP addresses. #10608 (tavplubix).
  • Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table. #10603 (Nikolai Kochetov).
  • Fixed handling condition variable for synchronous mutations. In some cases signals to that condition variable could be lost. #10588 (Vladimir Chebotarev).
  • Fixes possible crash createDictionary() is called before loadStoredObject() has finished. #10587 (Vitaly Baranov).
  • Fix error the BloomFilter false positive must be a double number between 0 and 1 #10551. #10569 (Winter Zhang).
  • Fix SELECT of column ALIAS which default expression type different from column type. #10563 (Azat Khuzhin).
  • Implemented comparison between DateTime64 and String values (just like for DateTime). #10560 (Vasily Nemkov).
  • Fix index corruption, which may accur in some cases after merge compact parts into another compact part. #10531 (Anton Popov).
  • Disable GROUP BY sharding_key optimization by default (optimize_distributed_group_by_sharding_key had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example is if in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. #10516 (Azat Khuzhin).
  • Fixes: #10263 (after that PR dist send via INSERT had been postponing on each INSERT) Fixes: #8756 (that PR breaks distributed sends with all of the following conditions met (unlikely setup for now I guess): internal_replication == false, multiple local shards (activates the hardlinking code) and distributed_storage_policy (makes link(2) fails on EXDEV)). #10486 (Azat Khuzhin).
  • Fixed error with "max_rows_to_sort" limit. #10268 (alexey-milovidov).
  • Get dictionary and check access rights only once per each call of any function reading external dictionaries. #10928 (Vitaly Baranov).

Improvement

  • Apply TTL for old data, after ALTER MODIFY TTL query. This behaviour is controlled by setting materialize_ttl_after_modify, which is enabled by default. #11042 (Anton Popov).
  • When parsing C-style backslash escapes in string literals, VALUES and various text formats (this is an extension to SQL standard that is endemic for ClickHouse and MySQL), keep backslash if unknown escape sequence is found (e.g. \% or \w) that will make usage of LIKE and match regular expressions more convenient (it's enough to write name LIKE 'used\_cars' instead of name LIKE 'used\\_cars') and more compatible at the same time. This fixes #10922. #11208 (alexey-milovidov).
  • When reading Decimal value, cut extra digits after point. This behaviour is more compatible with MySQL and PostgreSQL. This fixes #10202. #11831 (alexey-milovidov).
  • Allow to DROP replicated table if the metadata in ZooKeeper was already removed and does not exist (this is also the case when using TestKeeper for testing and the server was restarted). Allow to RENAME replicated table even if there is an error communicating with ZooKeeper. This fixes #10720. #11652 (alexey-milovidov).
  • Slightly improve diagnostic of reading decimal from string. This closes #10202. #11829 (alexey-milovidov).
  • Fix sleep invocation in signal handler. It was sleeping for less amount of time than expected. #11825 (alexey-milovidov).
  • (Only Linux) OS related performance metrics (for CPU and I/O) will work even without CAP_NET_ADMIN capability. #10544 (Alexander Kazakov).
  • Added hostname as an alias to function hostName. This feature was suggested by Victor Tarnavskiy from Yandex.Metrica. #11821 (alexey-milovidov).
  • Added support for distributed DDL (update/delete/drop partition) on cross replication clusters. #11703 (Nikita Mikhaylov).
  • Emit warning instead of error in server log at startup if we cannot listen one of the listen addresses (e.g. IPv6 is unavailable inside Docker). Note that if server fails to listen all listed addresses, it will refuse to startup as before. This fixes #4406. #11687 (alexey-milovidov).
  • Default user and database creation on docker image starting. #10637 (Paramtamtam).
  • When multiline query is printed to server log, the lines are joined. Make it to work correct in case of multiline string literals, identifiers and single-line comments. This fixes #3853. #11686 (alexey-milovidov).
  • Multiple names are now allowed in commands: CREATE USER, CREATE ROLE, ALTER USER, SHOW CREATE USER, SHOW GRANTS and so on. #11670 (Vitaly Baranov).
  • Clear password from command line in clickhouse-client and clickhouse-benchmark if the user has specified it with explicit value. This prevents password exposure by ps and similar tools. #11665 (alexey-milovidov).
  • Don't use debug info from ELF file if it doesn't correspond to the running binary. It is needed to avoid printing wrong function names and source locations in stack traces. This fixes #7514. #11657 (alexey-milovidov).
  • Return NULL/zero when value is not parsed completely in parseDateTimeBestEffortOrNull/Zero functions. This fixes #7876. #11653 (alexey-milovidov).
  • Skip empty parameters in requested URL. They may appear when you write http://localhost:8123/?&a=b or http://localhost:8123/?a=b&&c=d. This closes #10749. #11651 (alexey-milovidov).
  • Allow using groupArrayArray and groupUniqArrayArray as SimpleAggregateFunction. #11650 (Volodymyr Kuznetsov).
  • Allow comparison with constant strings by implicit conversions when analysing index conditions on other types. This may close #11630. #11648 (alexey-milovidov).
  • https://github.com/ClickHouse/ClickHouse/pull/7572#issuecomment-642815377 Support config default HTTPHandlers. #11628 (Winter Zhang).
  • Make more input formats to work with Kafka engine. Fix the issue with premature flushes. Fix the performance issue when kafka_num_consumers is greater than number of partitions in topic. #11599 (filimonov).
  • Improve multiple_joins_rewriter_version=2 logic. Fix unknown columns error for lambda aliases. #11587 (Artem Zuikov).
  • Better exception message when cannot parse columns declaration list. This closes #10403. #11537 (alexey-milovidov).
  • Improve enable_optimize_predicate_expression=1 logic for VIEW. #11513 (Artem Zuikov).
  • Adding support for PREWHERE in live view tables. #11495 (vzakaznikov).
  • Automatically update DNS cache, which is used to check if user is allowed to connect from an address. #11487 (tavplubix).
  • OPTIMIZE FINAL will force merge even if concurrent merges are performed. This closes #11309 and closes #11322. #11346 (alexey-milovidov).
  • Suppress output of cancelled queries in clickhouse-client. In previous versions result may continue to print in terminal even after you press Ctrl+C to cancel query. This closes #9473. #11342 (alexey-milovidov).
  • Better log messages in while reloading configuration. #11341 (alexey-milovidov).
  • Remove trailing whitespaces from formatted queries in clickhouse-client or clickhouse-format in some cases. #11325 (alexey-milovidov).
  • Add setting "output_format_pretty_max_value_width". If value is longer, it will be cut to avoid output of too large values in terminal. This closes #11140. #11324 (alexey-milovidov).
  • Better exception message in case when there is shortage of memory mappings. This closes #11027. #11316 (alexey-milovidov).
  • Support (U)Int8, (U)Int16, Date in ASOF JOIN. #11301 (Artem Zuikov).
  • Support kafka_client_id parameter for Kafka tables. It also changes the default client.id used by ClickHouse when communicating with Kafka to be more verbose and usable. #11252 (filimonov).
  • Keep the value of DistributedFilesToInsert metric on exceptions. In previous versions, the value was set when we are going to send some files, but it is zero, if there was an exception and some files are still pending. Now it corresponds to the number of pending files in filesystem. #11220 (alexey-milovidov).
  • Add support for multi-word data type names (such as DOUBLE PRECISION and CHAR VARYING) for better SQL compatibility. #11214 (Павел Потемкин).
  • The query log is now enabled by default. #11184 (Ivan Blinkov).
  • Show authentication type in table system.users and while executing SHOW CREATE USER query. #11080 (Vitaly Baranov).
  • Remove data on explicit DROP DATABASE for Memory database engine. Fixes #10557. #11021 (tavplubix).
  • Set thread names for internal threads of rdkafka library. Make logs from rdkafka available in server logs. #10983 (Azat Khuzhin).
  • Support for unicode whitespaces in queries. This helps when queries are copy-pasted from Word or from web page. This fixes #10896. #10903 (alexey-milovidov).
  • Allow large UInt types as the index in function tupleElement. #10874 (hcz).
  • Respect prefer_localhost_replica/load_balancing on INSERT into Distributed. #10867 (Azat Khuzhin).
  • Introduce min_insert_block_size_rows_for_materialized_views, min_insert_block_size_bytes_for_materialized_views settings. This settings are similar to min_insert_block_size_rows and min_insert_block_size_bytes, but applied only for blocks inserted into MATERIALIZED VIEW. It helps to control blocks squashing while pushing to MVs and avoid excessive memory usage. #10858 (Azat Khuzhin).
  • Get rid of exception from replicated queue during server shutdown. Fixes #10819. #10841 (alesapin).
  • Ensure that varSamp, varPop cannot return negative results due to numerical errors and that stddevSamp, stddevPop cannot be calculated from negative variance. This fixes #10532. #10829 (alexey-milovidov).
  • Better DNS exception message. This fixes #10813. #10828 (alexey-milovidov).
  • Change HTTP response code in case of some parse errors to 400 Bad Request. This fix #10636. #10640 (alexey-milovidov).
  • Print a message if clickhouse-client is newer than clickhouse-server. #10627 (alexey-milovidov).
  • Adding support for INSERT INTO [db.]table WATCH query. #10498 (vzakaznikov).
  • Allow to pass quota_key in clickhouse-client. This closes #10227. #10270 (alexey-milovidov).

Performance Improvement

  • Allow multiple replicas to assign merges, mutations, partition drop, move and replace concurrently. This closes #10367. #11639 (alexey-milovidov) #11795 (alexey-milovidov).
  • Optimization of GROUP BY with respect to table sorting key, enabled with optimize_aggregation_in_order setting. #9113 (dimarub2000).
  • Selects with final are executed in parallel. Added setting max_final_threads to limit the number of threads used. #10463 (Nikolai Kochetov).
  • Improve performance for INSERT queries via INSERT SELECT or INSERT with clickhouse-client when small blocks are generated (typical case with parallel parsing). This fixes #11275. Fix the issue that CONSTRAINTs were not working for DEFAULT fields. This fixes #11273. Fix the issue that CONSTRAINTS were ignored for TEMPORARY tables. This fixes #11274. #11276 (alexey-milovidov).
  • Optimization that eliminates min/max/any aggregators of GROUP BY keys in SELECT section, enabled with optimize_aggregators_of_group_by_keys setting. #11667 (xPoSx). #11806 (Azat Khuzhin).
  • New optimization that takes all operations out of any function, enabled with optimize_move_functions_out_of_any #11529 (Ruslan).
  • Improve performance of clickhouse-client in interactive mode when Pretty formats are used. In previous versions, significant amount of time can be spent calculating visible width of UTF-8 string. This closes #11323. #11323 (alexey-milovidov).
  • Improved performance for queries with ORDER BY and small LIMIT (less, then max_block_size). #11171 (Albert Kidrachev).
  • Enable mlock of clickhouse binary by default. It will prevent clickhouse executable from being paged out under high IO load. #11139 (alexey-milovidov).
  • Make queries with sum aggregate function and without GROUP BY keys to run multiple times faster. #10992 (alexey-milovidov).
  • Improving radix sort (used in ORDER BY with simple keys) by removing some redundant data moves. #10981 (Arslan Gumerov).
  • Sort bigger parts of the left table in MergeJoin. Buffer left blocks in memory. Add partial_merge_join_left_table_buffer_bytes setting to manage the left blocks buffers sizes. #10601 (Artem Zuikov).
  • Remove duplicate ORDER BY and DISTINCT from subqueries, this optimization is enabled with optimize_duplicate_order_by_and_distinct #10067 (Mikhail Malafeev).
  • This feature eliminates functions of other keys in GROUP BY section, enabled with optimize_group_by_function_keys #10051 (xPoSx).
  • New optimization that takes arithmetic operations out of aggregate functions, enabled with optimize_arithmetic_operations_in_aggregate_functions #10047 (Ruslan).
  • Use HTTP client for S3 based on Poco instead of curl. This will improve performance and lower memory usage of s3 storage and table functions. #11230 (Pavel Kovalenko).
  • Fix Kafka performance issue related to reschedules based on limits, which were always applied. #11149 (filimonov).
  • Enable percpu_arena:percpu for jemalloc (This will reduce memory fragmentation due to thread pool). #11084 (Azat Khuzhin).
  • Optimize memory usage when reading a response from an S3 HTTP client. #11561 (Pavel Kovalenko).
  • Adjust the default Kafka settings for better performance. #11388 (filimonov).

Experimental Feature

  • Add data type Point (Tuple(Float64, Float64)) and Polygon (Array(Array(Tuple(Float64, Float64))). #10678 (Alexey Ilyukhov).
  • Add's a hasSubstr function that allows for look for subsequences in arrays. Note: this function is likely to be renamed without further notice. #11071 (Ryad Zenine).
  • Added OpenCL support and bitonic sort algorithm, which can be used for sorting integer types of data in single column. Needs to be build with flag -DENABLE_OPENCL=1. For using bitonic sort algorithm instead of others you need to set bitonic_sort for Setting's option special_sort and make sure that OpenCL is available. This feature does not improve performance or anything else, it is only provided as an example and for demonstration purposes. It is likely to be removed in near future if there will be no further development in this direction. #10232 (Ri).

Build/Testing/Packaging Improvement

ClickHouse release v20.4

ClickHouse release v20.4.6.53-stable 2020-06-25

Bug Fix

  • Fix rare crash caused by using Nullable column in prewhere condition. Continuation of #11608. #11869 (Nikolai Kochetov).
  • Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (alexey-milovidov).
  • Fix wrong result of comparison of FixedString with constant String. This fixes #11393. This bug appeared in version 20.4. #11828 (alexey-milovidov).
  • Fix wrong result for if() with NULLs in condition. #11807 (Artem Zuikov).
  • Fix using too many threads for queries. #11788 (Nikolai Kochetov).
  • Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. #11753 (hexiaoting).
  • Now replicated fetches will be cancelled during metadata alter. #11744 (alesapin).
  • Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (tavplubix).
  • Fix ORDER BY ... WITH FILL over const columns. #11697 (Anton Popov).
  • Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (alexey-milovidov).
  • Fix LIMIT n WITH TIES usage together with ORDER BY statement, which contains aliases. #11689 (Anton Popov).
  • Fix error which leads to an incorrect state of system.mutations. It may show that whole mutation is already done but the server still has MUTATE_PART tasks in the replication queue and tries to execute them. This fixes #11611. #11681 (alesapin).
  • Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (alexey-milovidov).
  • Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. #11644 (alexey-milovidov).
  • Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
  • Fix rare crash caused by using Nullable column in prewhere condition. (Probably it is connected with #11572 somehow). #11608 (Nikolai Kochetov).
  • Fix error Block structure mismatch for queries with sampling reading from Buffer table. #11602 (Nikolai Kochetov).
  • Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. #11601 (filimonov).
  • Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (alexey-milovidov).
  • Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. #11580 (Nikolai Kochetov).
  • Fixed rare segfault in SHOW CREATE TABLE Fixes #11490. #11579 (tavplubix).
  • All queries in HTTP session have had the same query_id. It is fixed. #11578 (tavplubix).
  • Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
  • Fix shard_num/replica_num for <node> (breaks use_compact_format_in_distributed_parts_names). #11528 (Azat Khuzhin).
  • Fix race condition which may lead to an exception during table drop. It's a bit tricky and not dangerous at all. If you want an explanation, just notice me in telegram. #11523 (alesapin).
  • Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. #11496 (alexey-milovidov).
  • If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
  • Get rid of old libunwind patches. 500aa22791 (r39048012) This allows to disable -fno-omit-frame-pointer in clang builds that improves performance at least by 1% in average. #10761 (Amos Bird).
  • Fix usage of primary key wrapped into a function with 'FINAL' modifier and 'ORDER BY' optimization. #10715 (Anton Popov).

Build/Testing/Packaging Improvement

ClickHouse release v20.4.5.36-stable 2020-06-10

Bug Fix

  • Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (alexey-milovidov).
  • Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
  • Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
  • Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. #11421 (Alexey Ilyukhov).
  • Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. #11420 (alexey-milovidov).
  • Fixed geohashesInBox with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov).
  • Fix possible Pipeline stuck error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov).
  • Remove redundant lock during parts send in ReplicatedMergeTree. #11354 (alesapin).
  • Fix support for \G (vertical output) in clickhouse-client in multiline mode. This closes #9933. #11350 (alexey-milovidov).
  • Fix potential segfault when using Lazy database. #11348 (alexey-milovidov).
  • Fix crash in quantilesExactWeightedArray. #11337 (Nikolai Kochetov).
  • Now merges stopped before change metadata in ALTER queries. #11335 (alesapin).
  • Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes #10241. #11330 (Nikolai Kochetov).
  • Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 (Ewout).
  • Fix very rare race condition in ThreadPool. #11314 (alexey-milovidov).
  • Fix insignificant data race in clickhouse-copier. Found by integration tests. #11313 (alexey-milovidov).
  • Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). #11311 (alexey-milovidov).
  • Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes #11286. #11303 (alexey-milovidov).
  • Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. #11296 (alexey-milovidov).
  • Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes https://github.com/ClickHouse/ClickHouse/issues/10586. #11278 (Vitaly Baranov).
  • Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. #11258 (Vitaly Baranov).
  • Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 (Nikita Mikhaylov).
  • Fix error Block structure mismatch in QueryPipeline while reading from VIEW with constants in inner query. Fixes #11181. #11205 (Nikolai Kochetov).
  • Fix possible exception Invalid status for associated output. #11200 (Nikolai Kochetov).
  • Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. #11185 (Nikolai Kochetov).
  • Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 (Vladimir Chebotarev).
  • If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
  • Fix Kafka performance issue related to reschedules based on limits, which were always applied. #11149 (filimonov).
  • Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #11145 (filimonov).
  • Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 (Azat Khuzhin).
  • Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin).

Build/Testing/Packaging Improvement

ClickHouse release v20.4.4.18-stable 2020-05-26

No changes compared to v20.4.3.16-stable.

ClickHouse release v20.4.3.16-stable 2020-05-23

Bug Fix

  • Removed logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
  • Fixed memory leak in registerDiskS3. #11074 (Pavel Kovalenko).
  • Fixed the potential missed data during termination of Kafka engine table. #11048 (filimonov).
  • Fixed parseDateTime64BestEffort argument resolution bugs. #11038 (Vasily Nemkov).
  • Fixed very rare potential use-after-free error in MergeTree if table was not created successfully. #10986, #10970 (alexey-milovidov).
  • Fixed metadata (relative path for rename) and data (relative path for symlink) handling for Atomic database. #10980 (Azat Khuzhin).
  • Fixed server crash on concurrent ALTER and DROP DATABASE queries with Atomic database engine. #10968 (tavplubix).
  • Fixed incorrect raw data size in getRawData() method. #10964 (Igr).
  • Fixed incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 (alexey-milovidov).
  • Fixed sending partially written files by the DistributedBlockOutputStream. #10940 (Azat Khuzhin).
  • Fixed crash in SELECT count(notNullIn(NULL, [])). #10920 (Nikolai Kochetov).
  • Fixed the hang which was happening sometimes during DROP of Kafka table engine. (or during server restarts). #10910 (filimonov).
  • Fixed the impossibility of executing multiple ALTER RENAME like a TO b, c TO a. #10895 (alesapin).
  • Fixed possible race which could happen when you get result from aggregate function state from multiple thread for the same column. The only way it can happen is when you use finalizeAggregation function while reading from table with Memory engine which stores AggregateFunction state for quantile* function. #10890 (Nikolai Kochetov).
  • Fixed backward compatibility with tuples in Distributed tables. #10889 (Anton Popov).
  • Fixed SIGSEGV in StringHashTable if such a key does not exist. #10870 (Azat Khuzhin).
  • Fixed WATCH hangs after LiveView table was dropped from database with Atomic engine. #10859 (tavplubix).
  • Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. #10849 (tavplubix).
  • Now constraints are updated if the column participating in CONSTRAINT expression was renamed. Fixes #10844. #10847 (alesapin).
  • Fixed potential read of uninitialized memory in cache-dictionary. #10834 (alexey-milovidov).
  • Fixed columns order after Block::sortColumns(). #10826 (Azat Khuzhin).
  • Fixed the issue with ODBC bridge when no quoting of identifiers is requested. Fixes [#7984] (https://github.com/ClickHouse/ClickHouse/issues/7984). #10821 (alexey-milovidov).
  • Fixed UBSan and MSan report in DateLUT. #10798 (alexey-milovidov).
  • Fixed incorrect type conversion in key conditions. Fixes #6287. #10791 (Andrew Onyshchuk).
  • Fixed parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov).
  • Fixed combinator -OrNull and -OrDefault when combined with -State. #10741 (hcz).
  • Fixed possible buffer overflow in function h3EdgeAngle. #10711 (alexey-milovidov).
  • Fixed bug which locks concurrent alters when table has a lot of parts. #10659 (alesapin).
  • Fixed nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 (alexey-milovidov).
  • Fixed optimize_skip_unused_shards with LowCardinality. #10611 (Azat Khuzhin).
  • Fixed handling condition variable for synchronous mutations. In some cases signals to that condition variable could be lost. #10588 (Vladimir Chebotarev).
  • Fixed possible crash when createDictionary() is called before loadStoredObject() has finished. #10587 (Vitaly Baranov).
  • Fixed SELECT of column ALIAS which default expression type different from column type. #10563 (Azat Khuzhin).
  • Implemented comparison between DateTime64 and String values. #10560 (Vasily Nemkov).
  • Disable GROUP BY sharding_key optimization by default (optimize_distributed_group_by_sharding_key had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example is if in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. #10516 (Azat Khuzhin).
  • Fixed #10263. #10486 (Azat Khuzhin).
  • Added tests about max_rows_to_sort setting. #10268 (alexey-milovidov).
  • Added backward compatibility for create bloom filter index. #10551. #10569 (Winter Zhang).

ClickHouse release v20.4.2.9, 2020-05-12

Backward Incompatible Change

  • System tables (e.g. system.query_log, system.trace_log, system.metric_log) are using compact data part format for parts smaller than 10 MiB in size. Compact data part format is supported since version 20.3. If you are going to downgrade to version less than 20.3, you should manually delete table data for system logs in /var/lib/clickhouse/data/system/.
  • When string comparison involves FixedString and compared arguments are of different sizes, do comparison as if smaller string is padded to the length of the larger. This is intented for SQL compatibility if we imagine that FixedString data type corresponds to SQL CHAR. This closes #9272. #10363 (alexey-milovidov)
  • Make SHOW CREATE TABLE multiline. Now it is more readable and more like MySQL. #10049 (Azat Khuzhin)
  • Added a setting validate_polygons that is used in pointInPolygon function and enabled by default. #9857 (alexey-milovidov)

New Feature

  • Add support for secured connection from ClickHouse to Zookeeper #10184 (Konstantin Lebedev)
  • Support custom HTTP handlers. See ISSUES-5436 for description. #7572 (Winter Zhang)
  • Add MessagePack Input/Output format. #9889 (Kruglov Pavel)
  • Add Regexp input format. #9196 (Kruglov Pavel)
  • Added output format Markdown for embedding tables in markdown documents. #10317 (Kruglov Pavel)
  • Added support for custom settings section in dictionaries. Also fixes issue #2829. #10137 (Artem Streltsov)
  • Added custom settings support in DDL-queries for CREATE DICTIONARY #10465 (Artem Streltsov)
  • Add simple server-wide memory profiler that will collect allocation contexts when server memory usage becomes higher than the next allocation threshold. #10444 (alexey-milovidov)
  • Add setting always_fetch_merged_part which restrict replica to merge parts by itself and always prefer dowloading from other replicas. #10379 (alesapin)
  • Add function JSONExtractKeysAndValuesRaw which extracts raw data from JSON objects #10378 (hcz)
  • Add memory usage from OS to system.asynchronous_metrics. #10361 (alexey-milovidov)
  • Added generic variants for functions least and greatest. Now they work with arbitrary number of arguments of arbitrary types. This fixes #4767 #10318 (alexey-milovidov)
  • Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration: strict_max_lifetime_seconds, which is max_lifetime by default, and query_wait_timeout_milliseconds, which is one minute by default. The first settings is also useful with allow_read_expired_keys settings (to forbid reading very expired keys). #10337 (Nikita Mikhaylov)
  • Add log_queries_min_type to filter which entries will be written to query_log #10053 (Azat Khuzhin)
  • Added function isConstant. This function checks whether its argument is constant expression and returns 1 or 0. It is intended for development, debugging and demonstration purposes. #10198 (alexey-milovidov)
  • add joinGetOrNull to return NULL when key is missing instead of returning the default value. #10094 (Amos Bird)
  • Consider NULL to be equal to NULL in IN operator, if the option transform_null_in is set. #10085 (achimbab)
  • Add ALTER TABLE ... RENAME COLUMN for MergeTree table engines family. #9948 (alesapin)
  • Support parallel distributed INSERT SELECT. #9759 (vxider)
  • Add ability to query Distributed over Distributed (w/o distributed_group_by_no_merge) ... #9923 (Azat Khuzhin)
  • Add function arrayReduceInRanges which aggregates array elements in given ranges. #9598 (hcz)
  • Add Dictionary Status on prometheus exporter. #9622 (Guillaume Tassery)
  • Add function arrayAUC #8698 (taiyang-li)
  • Support DROP VIEW statement for better TPC-H compatibility. #9831 (Amos Bird)
  • Add 'strict_order' option to windowFunnel() #9773 (achimbab)
  • Support DATE and TIMESTAMP SQL operators, e.g. SELECT date '2001-01-01' #9691 (Artem Zuikov)

Experimental Feature

  • Added experimental database engine Atomic. It supports non-blocking DROP and RENAME TABLE queries and atomic EXCHANGE TABLES t1 AND t2 query #7512 (tavplubix)
  • Initial support for ReplicatedMergeTree over S3 (it works in suboptimal way) #10126 (Pavel Kovalenko)

Bug Fix

  • Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table #10603 (Nikolai Kochetov)
  • Fixed bug, which caused HTTP requests to get stuck on client closing connection when readonly=2 and cancel_http_readonly_queries_on_client_close=1. #10684 (tavplubix)
  • Fix segfault in StorageBuffer when exception is thrown on server startup. Fixes #10550 #10609 (tavplubix)
  • The querySYSTEM DROP DNS CACHE now also drops caches used to check if user is allowed to connect from some IP addresses #10608 (tavplubix)
  • Fix usage of multiple IN operators with an identical set in one query. Fixes #10539 #10686 (Anton Popov)
  • Fix crash in generateRandom with nested types. Fixes #10583. #10734 (Nikolai Kochetov)
  • Fix data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov)
  • Fix logic for aggregation_memory_efficient_merge_threads setting. #10667 (palasonic1)
  • Fix disappearing totals. Totals could have being filtered if query had JOIN or subquery with external WHERE condition. Fixes #10674 #10698 (Nikolai Kochetov)
  • Fix the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes #10655 #10664 (Nikolai Kochetov)
  • Fix possible incorrect number of rows for queries with LIMIT. Fixes #10566, #10709 #10660 (Nikolai Kochetov)
  • Fix index corruption, which may occur in some cases after merging compact parts into another compact part. #10531 (Anton Popov)
  • Fix the situation, when mutation finished all parts, but hung up in is_done=0. #10526 (alesapin)
  • Fix overflow at beginning of unix epoch for timezones with fractional offset from UTC. Fixes #9335. #10513 (alexey-milovidov)
  • Better diagnostics for input formats. Fixes #10204 #10418 (tavplubix)
  • Fix numeric overflow in simpleLinearRegression() over large integers #10474 (hcz)
  • Fix use-after-free in Distributed shutdown, avoid waiting for sending all batches #10491 (Azat Khuzhin)
  • Add CA certificates to clickhouse-server docker image #10476 (filimonov)
  • Fix a rare endless loop that might have occurred when using the addressToLine function or AggregateFunctionState columns. #10466 (Alexander Kuzmenkov)
  • Handle zookeeper "no node error" during distributed query #10050 (Daniel Chen)
  • Fix bug when server cannot attach table after column's default was altered. #10441 (alesapin)
  • Implicitly cast the default expression type to the column type for the ALIAS columns #10563 (Azat Khuzhin)
  • Don't remove metadata directory if ATTACH DATABASE fails #10442 (Winter Zhang)
  • Avoid dependency on system tzdata. Fixes loading of Africa/Casablanca timezone on CentOS 8. Fixes #10211 #10425 (alexey-milovidov)
  • Fix some issues if data is inserted with quorum and then gets deleted (DROP PARTITION, TTL, etc.). It led to stuck of INSERTs or false-positive exceptions in SELECTs. Fixes #9946 #10188 (Nikita Mikhaylov)
  • Check the number and type of arguments when creating BloomFilter index #9623 #10431 (Winter Zhang)
  • Prefer fallback_to_stale_replicas over skip_unavailable_shards, otherwise when both settings specified and there are no up-to-date replicas the query will fail (patch from @alex-zaitsev ) #10422 (Azat Khuzhin)
  • Fix the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. Fixes #10226. #10427 (Vadim Plakhtinskiy)
  • Add database name to dictionary name after DETACH/ATTACH. Fixes system.dictionaries table and SYSTEM RELOAD query #10415 (Azat Khuzhin)
  • Fix possible incorrect result for extremes in processors pipeline. #10131 (Nikolai Kochetov)
  • Fix possible segfault when the setting distributed_group_by_no_merge is enabled (introduced in 20.3.7.46 by #10131). #10399 (Nikolai Kochetov)
  • Fix wrong flattening of Array(Tuple(...)) data types. Fixes #10259 #10390 (alexey-milovidov)
  • Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN #9950 (Alexander Kuzmenkov)
  • Fix order of columns after Block::sortColumns() #10826 (Azat Khuzhin)
  • Fix possible Pipeline stuck error in ConcatProcessor which may happen in remote query. #10381 (Nikolai Kochetov)
  • Don't make disk reservations for aggregations. Fixes #9241 #10375 (Azat Khuzhin)
  • Fix wrong behaviour of datetime functions for timezones that has altered between positive and negative offsets from UTC (e.g. Pacific/Kiritimati). Fixes #7202 #10369 (alexey-milovidov)
  • Avoid infinite loop in dictIsIn function. Fixes #515 #10365 (alexey-milovidov)
  • Disable GROUP BY sharding_key optimization by default and fix it for WITH ROLLUP/CUBE/TOTALS #10516 (Azat Khuzhin)
  • Check for error code when checking parts and don't mark part as broken if the error is like "not enough memory". Fixes #6269 #10364 (alexey-milovidov)
  • Show information about not loaded dictionaries in system tables. #10234 (Vitaly Baranov)
  • Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 (alexey-milovidov)
  • Fixed DROP vs OPTIMIZE race in ReplicatedMergeTree. DROP could left some garbage in replica path in ZooKeeper if there was concurrent OPTIMIZE query. #10312 (tavplubix)
  • Fix 'Logical error: CROSS JOIN has expressions' error for queries with comma and names joins mix. Fixes #9910 #10311 (Artem Zuikov)
  • Fix queries with max_bytes_before_external_group_by. #10302 (Artem Zuikov)
  • Fix the issue with limiting maximum recursion depth in parser in certain cases. This fixes #10283 This fix may introduce minor incompatibility: long and deep queries via clickhouse-client may refuse to work, and you should adjust settings max_query_size and max_parser_depth accordingly. #10295 (alexey-milovidov)
  • Allow to use count(*) with multiple JOINs. Fixes #9853 #10291 (Artem Zuikov)
  • Fix error Pipeline stuck with max_rows_to_group_by and group_by_overflow_mode = 'break'. #10279 (Nikolai Kochetov)
  • Fix 'Cannot add column' error while creating range_hashed dictionary using DDL query. Fixes #10093. #10235 (alesapin)
  • Fix rare possible exception Cannot drain connections: cancel first. #10239 (Nikolai Kochetov)
  • Fixed bug where ClickHouse would throw "Unknown function lambda." error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. #10237 (Alexander Kazakov)
  • Fixed reasonably rare segfault in StorageSystemTables that happens when SELECT ... FROM system.tables is run on a database with Lazy engine. #10209 (Alexander Kazakov)
  • Fix possible infinite query execution when the query actually should stop on LIMIT, while reading from infinite source like system.numbers or system.zeros. #10206 (Nikolai Kochetov)
  • Fixed "generateRandom" function for Date type. This fixes #9973. Fix an edge case when dates with year 2106 are inserted to MergeTree tables with old-style partitioning but partitions are named with year 1970. #10218 (alexey-milovidov)
  • Convert types if the table definition of a View does not correspond to the SELECT query. This fixes #10180 and #10022 #10217 (alexey-milovidov)
  • Fix parseDateTimeBestEffort for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10082 #10214 (alexey-milovidov)
  • Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. #10207 (alexey-milovidov)
  • Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes #10092 #10195 (alexey-milovidov)
  • Fix issue with separator appearing in SCRAMBLE for native mysql-connector-java (JDBC) #10140 (BohuTANG)
  • Fix using the current database for an access checking when the database isn't specified. #10192 (Vitaly Baranov)
  • Fix ALTER of tables with compact parts. #10130 (Anton Popov)
  • Add the ability to relax the restriction on non-deterministic functions usage in mutations with allow_nondeterministic_mutations setting. #10186 (filimonov)
  • Fix DROP TABLE invoked for dictionary #10165 (Azat Khuzhin)
  • Convert blocks if structure does not match when doing INSERT into Distributed table #10135 (Azat Khuzhin)
  • The number of rows was logged incorrectly (as sum across all parts) when inserted block is split by parts with partition key. #10138 (alexey-milovidov)
  • Add some arguments check and support identifier arguments for MySQL Database Engine #10077 (Winter Zhang)
  • Fix incorrect index_granularity_bytes check while creating new replica. Fixes #10098. #10121 (alesapin)
  • Fix bug in CHECK TABLE query when table contain skip indices. #10068 (alesapin)
  • Fix Distributed-over-Distributed with the only one shard in a nested table #9997 (Azat Khuzhin)
  • Fix possible rows loss for queries with JOIN and UNION ALL. Fixes #9826, #10113. ... #10099 (Nikolai Kochetov)
  • Fix bug in dictionary when local clickhouse server is used as source. It may caused memory corruption if types in dictionary and source are not compatible. #10071 (alesapin)
  • Fixed replicated tables startup when updating from an old ClickHouse version where /table/replicas/replica_name/metadata node doesn't exist. Fixes #10037. #10095 (alesapin)
  • Fix error Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform. It happened when setting distributed_aggregation_memory_efficient was enabled, and distributed query read aggregating data with mixed single and two-level aggregation from different shards. #10063 (Nikolai Kochetov)
  • Fix deadlock when database with materialized view failed attach at start #10054 (Azat Khuzhin)
  • Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (#8636, #8925). ... #10025 (Alexander Kuzmenkov)
  • Fix wrong results of distributed queries when alias could override qualified column name. Fixes #9672 #9714 #9972 (Artem Zuikov)
  • Fix possible deadlock in SYSTEM RESTART REPLICAS #9955 (tavplubix)
  • Fix the number of threads used for remote query execution (performance regression, since 20.3). This happened when query from Distributed table was executed simultaneously on local and remote shards. Fixes #9965 #9971 (Nikolai Kochetov)
  • Fixed DeleteOnDestroy logic in ATTACH PART which could lead to automatic removal of attached part and added few tests #9410 (Vladimir Chebotarev)
  • Fix a bug with ON CLUSTER DDL queries freezing on server startup. #9927 (Gagan Arneja)
  • Fix bug in which the necessary tables weren't retrieved at one of the processing stages of queries to some databases. Fixes #9699. #9949 (achulkov2)
  • Fix 'Not found column in block' error when JOIN appears with TOTALS. Fixes #9839 #9939 (Artem Zuikov)
  • Fix parsing multiple hosts set in the CREATE USER command #9924 (Vitaly Baranov)
  • Fix TRUNCATE for Join table engine (#9917). #9920 (Amos Bird)
  • Fix race condition between drop and optimize in ReplicatedMergeTree. #9901 (alesapin)
  • Fix DISTINCT for Distributed when optimize_skip_unused_shards is set. #9808 (Azat Khuzhin)
  • Fix "scalar doesn't exist" error in ALTERs (#9878). ... #9904 (Amos Bird)
  • Fix error with qualified names in distributed_product_mode=\'local\'. Fixes #4756 #9891 (Artem Zuikov)
  • For INSERT queries shards now do clamp the settings from the initiator to their constraints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix #9447. #9852 (Vitaly Baranov)
  • Add some retries when commiting offsets to Kafka broker, since it can reject commit if during offsets.commit.timeout.ms there were no enough replicas available for the __consumer_offsets topic #9884 (filimonov)
  • Fix Distributed engine behavior when virtual columns of the underlying table used in WHERE #9847 (Azat Khuzhin)
  • Fixed some cases when timezone of the function argument wasn't used properly. #9574 (Vasily Nemkov)
  • Fix 'Different expressions with the same alias' error when query has PREWHERE and WHERE on distributed table and SET distributed_product_mode = 'local'. #9871 (Artem Zuikov)
  • Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. #9860 (alesapin)
  • Fix calculating grants for introspection functions from the setting allow_introspection_functions. #9840 (Vitaly Baranov)
  • Fix max_distributed_connections (w/ and w/o Processors) #9673 (Azat Khuzhin)
  • Fix possible exception Got 0 in totals chunk, expected 1 on client. It happened for queries with JOIN in case if right joined table had zero rows. Example: select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;. Fixes #9777. ... #9823 (Nikolai Kochetov)
  • Fix 'COMMA to CROSS JOIN rewriter is not enabled or cannot rewrite query' error in case of subqueries with COMMA JOIN out of tables lists (i.e. in WHERE). Fixes #9782 #9830 (Artem Zuikov)
  • Fix server crashing when optimize_skip_unused_shards is set and expression for key can't be converted to its field type #9804 (Azat Khuzhin)
  • Fix empty string handling in splitByString. #9767 (hcz)
  • Fix broken ALTER TABLE DELETE COLUMN query for compact parts. #9779 (alesapin)
  • Fixed missing rows_before_limit_at_least for queries over http (with processors pipeline). Fixes #9730 #9757 (Nikolai Kochetov)
  • Fix excessive memory consumption in ALTER queries (mutations). This fixes #9533 and #9670. #9754 (alesapin)
  • Fix possible permanent "Cannot schedule a task" error. #9154 (Azat Khuzhin)
  • Fix bug in backquoting in external dictionaries DDL. Fixes #9619. #9734 (alesapin)
  • Fixed data race in text_log. It does not correspond to any real bug. #9726 (alexey-milovidov)
  • Fix bug in a replication that doesn't allow replication to work if the user has executed mutations on the previous version. This fixes #9645. #9652 (alesapin)
  • Fixed incorrect internal function names for sumKahan and sumWithOverflow. It led to exception while using this functions in remote queries. #9636 (Azat Khuzhin)
  • Add setting use_compact_format_in_distributed_parts_names which allows to write files for INSERT queries into Distributed table with more compact format. This fixes #9647. #9653 (alesapin)
  • Fix RIGHT and FULL JOIN with LowCardinality in JOIN keys. #9610 (Artem Zuikov)
  • Fix possible exceptions Size of filter doesn't match size of column and Invalid number of rows in Chunk in MergeTreeRangeReader. They could appear while executing PREWHERE in some cases. #9612 (Anton Popov)
  • Allow ALTER ON CLUSTER of Distributed tables with internal replication. This fixes #3268 #9617 (shinoi2)
  • Fix issue when timezone was not preserved if you write a simple arithmetic expression like time + 1 (in contrast to an expression like time + INTERVAL 1 SECOND). This fixes #5743 #9323 (alexey-milovidov)

Improvement

  • Use time zone when comparing DateTime with string literal. This fixes #5206. #10515 (alexey-milovidov)
  • Print verbose diagnostic info if Decimal value cannot be parsed from text input format. #10205 (alexey-milovidov)
  • Add tasks/memory metrics for distributed/buffer schedule pools #10449 (Azat Khuzhin)
  • Display result as soon as it's ready for SELECT DISTINCT queries in clickhouse-local and HTTP interface. This fixes #8951 #9559 (alexey-milovidov)
  • Allow to use SAMPLE OFFSET query instead of cityHash64(PRIMARY KEY) % N == n for splitting in clickhouse-copier. To use this feature, pass --experimental-use-sample-offset 1 as a command line argument. #10414 (Nikita Mikhaylov)
  • Allow to parse BOM in TSV if the first column cannot contain BOM in its value. This fixes #10301 #10424 (alexey-milovidov)
  • Add Avro nested fields insert support #10354 (Andrew Onyshchuk)
  • Allowed to alter column in non-modifying data mode when the same type is specified. #10382 (Vladimir Chebotarev)
  • Auto distributed_group_by_no_merge on GROUP BY sharding key (if optimize_skip_unused_shards is set) #10341 (Azat Khuzhin)
  • Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key #10373 (Azat Khuzhin)
  • Added a setting max_server_memory_usage to limit total memory usage of the server. The metric MemoryTracking is now calculated without a drift. The setting max_memory_usage_for_all_queries is now obsolete and does nothing. This closes #10293. #10362 (alexey-milovidov)
  • Add config option system_tables_lazy_load. If it's set to false, then system tables with logs are loaded at the server startup. Alexander Burmak, Svyatoslav Tkhon Il Pak, #9642 #10359 (alexey-milovidov)
  • Use background thread pool (background_schedule_pool_size) for distributed sends #10263 (Azat Khuzhin)
  • Use background thread pool for background buffer flushes. #10315 (Azat Khuzhin)
  • Support for one special case of removing incompletely written parts. This fixes #9940. #10221 (alexey-milovidov)
  • Use isInjective() over manual list of such functions for GROUP BY optimization. #10342 (Azat Khuzhin)
  • Avoid printing error message in log if client sends RST packet immediately on connect. It is typical behaviour of IPVS balancer with keepalived and VRRP. This fixes #1851 #10274 (alexey-milovidov)
  • Allow to parse +inf for floating point types. This closes #1839 #10272 (alexey-milovidov)
  • Implemented generateRandom table function for Nested types. This closes #9903 #10219 (alexey-milovidov)
  • Provide max_allowed_packed in MySQL compatibility interface that will help some clients to communicate with ClickHouse via MySQL protocol. #10199 (BohuTANG)
  • Allow literals for GLOBAL IN (i.e. SELECT * FROM remote('localhost', system.one) WHERE dummy global in (0)) #10196 (Azat Khuzhin)
  • Fix various small issues in interactive mode of clickhouse-client #10194 (alexey-milovidov)
  • Avoid superfluous dictionaries load (system.tables, DROP/SHOW CREATE TABLE) #10164 (Azat Khuzhin)
  • Update to RWLock: timeout parameter for getLock() + implementation reworked to be phase fair #10073 (Alexander Kazakov)
  • Enhanced compatibility with native mysql-connector-java(JDBC) #10021 (BohuTANG)
  • The function toString is considered monotonic and can be used for index analysis even when applied in tautological cases with String or LowCardinality(String) argument. #10110 (Amos Bird)
  • Add ON CLUSTER clause support to commands {CREATE|DROP} USER/ROLE/ROW POLICY/SETTINGS PROFILE/QUOTA, GRANT. #9811 (Vitaly Baranov)
  • Virtual hosted-style support for S3 URI #9998 (Pavel Kovalenko)
  • Now layout type for dictionaries with no arguments can be specified without round brackets in dictionaries DDL-queries. Fixes #10057. #10064 (alesapin)
  • Add ability to use number ranges with leading zeros in filepath #9989 (Olga Khvostikova)
  • Better memory usage in CROSS JOIN. #10029 (Artem Zuikov)
  • Try to connect to all shards in cluster when getting structure of remote table and skip_unavailable_shards is set. #7278 (nvartolomei)
  • Add total_rows/total_bytes into the system.tables table. #9919 (Azat Khuzhin)
  • System log tables now use polymorpic parts by default. #9905 (Anton Popov)
  • Add type column into system.settings/merge_tree_settings #9909 (Azat Khuzhin)
  • Check for available CPU instructions at server startup as early as possible. #9888 (alexey-milovidov)
  • Remove ORDER BY stage from mutations because we read from a single ordered part in a single thread. Also add check that the rows in mutation are ordered by sorting key and this order is not violated. #9886 (alesapin)
  • Implement operator LIKE for FixedString at left hand side. This is needed to better support TPC-DS queries. #9890 (alexey-milovidov)
  • Add force_optimize_skip_unused_shards_no_nested that will disable force_optimize_skip_unused_shards for nested Distributed table #9812 (Azat Khuzhin)
  • Now columns size is calculated only once for MergeTree data parts. #9827 (alesapin)
  • Evaluate constant expressions for optimize_skip_unused_shards (i.e. SELECT * FROM foo_dist WHERE key=xxHash32(0)) #8846 (Azat Khuzhin)
  • Check for using Date or DateTime column from TTL expressions was removed. #9967 (Vladimir Chebotarev)
  • DiskS3 hard links optimal implementation. #9760 (Pavel Kovalenko)
  • If set multiple_joins_rewriter_version = 2 enables second version of multiple JOIN rewrites that keeps not clashed column names as is. It supports multiple JOINs with USING and allow select * for JOINs with subqueries. #9739 (Artem Zuikov)
  • Implementation of "non-blocking" alter for StorageMergeTree #9606 (alesapin)
  • Add MergeTree full support for DiskS3 #9646 (Pavel Kovalenko)
  • Extend splitByString to support empty strings as separators. #9742 (hcz)
  • Add a timestamp_ns column to system.trace_log. It contains a high-definition timestamp of the trace event, and allows to build timelines of thread profiles ("flame charts"). #9696 (Alexander Kuzmenkov)
  • When the setting send_logs_level is enabled, avoid intermixing of log messages and query progress. #9634 (Azat Khuzhin)
  • Added support of MATERIALIZE TTL IN PARTITION. #9581 (Vladimir Chebotarev)
  • Support complex types inside Avro nested fields #10502 (Andrew Onyshchuk)

Performance Improvement

  • Better insert logic for right table for Partial MergeJoin. #10467 (Artem Zuikov)
  • Improved performance of row-oriented formats (more than 10% for CSV and more than 35% for Avro in case of narrow tables). #10503 (Andrew Onyshchuk)
  • Improved performance of queries with explicitly defined sets at right side of IN operator and tuples on the left side. #10385 (Anton Popov)
  • Use less memory for hash table in HashJoin. #10416 (Artem Zuikov)
  • Special HashJoin over StorageDictionary. Allow rewrite dictGet() functions with JOINs. It's not backward incompatible itself but could uncover #8400 on some installations. #10133 (Artem Zuikov)
  • Enable parallel insert of materialized view when its target table supports. #10052 (vxider)
  • Improved performance of index analysis with monotonic functions. #9607#10026 (Anton Popov)
  • Using SSE2 or SSE4.2 SIMD intrinsics to speed up tokenization in bloom filters. #9968 (Vasily Nemkov)
  • Improved performance of queries with explicitly defined sets at right side of IN operator. This fixes performance regression in version 20.3. #9740 (Anton Popov)
  • Now clickhouse-copier splits each partition in number of pieces and copies them independently. #9075 (Nikita Mikhaylov)
  • Adding more aggregation methods. For example TPC-H query 1 will now pick FixedHashMap<UInt16, AggregateDataPtr> and gets 25% performance gain #9829 (Amos Bird)
  • Use single row counter for multiple streams in pre-limit transform. This helps to avoid uniting pipeline streams in queries with limit but without order by (like select f(x) from (select x from t limit 1000000000)) and use multiple threads for further processing. #9602 (Nikolai Kochetov)

Build/Testing/Packaging Improvement

ClickHouse release v20.3

ClickHouse release v20.3.12.112-lts 2020-06-25

Bug Fix

  • Fix rare crash caused by using Nullable column in prewhere condition. Continuation of #11608. #11869 (Nikolai Kochetov).
  • Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (alexey-milovidov).
  • Fix using too many threads for queries. #11788 (Nikolai Kochetov).
  • Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. #11753 (hexiaoting).
  • Now replicated fetches will be cancelled during metadata alter. #11744 (alesapin).
  • Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (tavplubix).
  • Fix ORDER BY ... WITH FILL over const columns. #11697 (Anton Popov).
  • Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (alexey-milovidov).
  • Fix error which leads to an incorrect state of system.mutations. It may show that whole mutation is already done but the server still has MUTATE_PART tasks in the replication queue and tries to execute them. This fixes #11611. #11681 (alesapin).
  • Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (alexey-milovidov).
  • Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. #11644 (alexey-milovidov).
  • Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
  • Fix rare crash caused by using Nullable column in prewhere condition. (Probably it is connected with #11572 somehow). #11608 (Nikolai Kochetov).
  • Fix error Block structure mismatch for queries with sampling reading from Buffer table. #11602 (Nikolai Kochetov).
  • Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. #11601 (filimonov).
  • Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (alexey-milovidov).
  • Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. #11580 (Nikolai Kochetov).
  • All queries in HTTP session have had the same query_id. It is fixed. #11578 (tavplubix).
  • Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
  • Fix shard_num/replica_num for <node> (breaks use_compact_format_in_distributed_parts_names). #11528 (Azat Khuzhin).
  • Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. #11496 (alexey-milovidov).
  • Fix wrong results of distributed queries when alias could override qualified column name. Fixes #9672 #9714. #9972 (Artem Zuikov).

ClickHouse release v20.3.11.97-lts 2020-06-10

New Feature

  • Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration: strict_max_lifetime_seconds, which is max_lifetime by default and query_wait_timeout_milliseconds, which is one minute by default. The first settings is also useful with allow_read_expired_keys settings (to forbid reading very expired keys). #10337 (Nikita Mikhaylov).

Bug Fix

  • Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (alexey-milovidov).
  • Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
  • Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
  • Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. #11421 (Alexey Ilyukhov).
  • Fix crash in JOIN over LowCarinality(T) and Nullable(T). #11380. #11414 (Artem Zuikov).
  • Fix error code for wrong USING key. #11373. #11404 (Artem Zuikov).
  • Fixed geohashesInBox with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov).
  • Better errors for joinGet() functions. #11389 (Artem Zuikov).
  • Fix possible Pipeline stuck error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov).
  • Remove redundant lock during parts send in ReplicatedMergeTree. #11354 (alesapin).
  • Fix support for \G (vertical output) in clickhouse-client in multiline mode. This closes #9933. #11350 (alexey-milovidov).
  • Fix crash in direct selects from StorageJoin (without JOIN) and wrong nullability. #11340 (Artem Zuikov).
  • Fix crash in quantilesExactWeightedArray. #11337 (Nikolai Kochetov).
  • Now merges stopped before change metadata in ALTER queries. #11335 (alesapin).
  • Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes #10241. #11330 (Nikolai Kochetov).
  • Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 (Ewout).
  • Fix very rare race condition in ThreadPool. #11314 (alexey-milovidov).
  • Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). #11311 (alexey-milovidov).
  • Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes #11286. #11303 (alexey-milovidov).
  • Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. #11296 (alexey-milovidov).
  • Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. #11258 (Vitaly Baranov).
  • Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 (Nikita Mikhaylov).
  • Fix error Block structure mismatch in QueryPipeline while reading from VIEW with constants in inner query. Fixes #11181. #11205 (Nikolai Kochetov).
  • Fix possible exception Invalid status for associated output. #11200 (Nikolai Kochetov).
  • Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. #11185 (Nikolai Kochetov).
  • Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 (Vladimir Chebotarev).
  • If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
  • Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 (Azat Khuzhin).
  • Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin).
  • Introduce commit retry logic to decrease the possibility of getting duplicates from Kafka in rare cases when offset commit was failed. #9884 (filimonov).

Performance Improvement

  • Get dictionary and check access rights only once per each call of any function reading external dictionaries. #10928 (Vitaly Baranov).

Build/Testing/Packaging Improvement

ClickHouse release v20.3.10.75-lts 2020-05-23

Bug Fix

  • Removed logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
  • Fixed parseDateTime64BestEffort argument resolution bugs. #11038 (Vasily Nemkov).
  • Fixed incorrect raw data size in method getRawData(). #10964 (Igr).
  • Fixed incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 (alexey-milovidov).
  • Fixed backward compatibility with tuples in Distributed tables. #10889 (Anton Popov).
  • Fixed SIGSEGV in StringHashTable if such a key does not exist. #10870 (Azat Khuzhin).
  • Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. #10849 (tavplubix).
  • Fixed columns order after Block::sortColumns(). #10826 (Azat Khuzhin).
  • Fixed the issue with ODBC bridge when no quoting of identifiers is requested. Fixes [#7984] (https://github.com/ClickHouse/ClickHouse/issues/7984). #10821 (alexey-milovidov).
  • Fixed UBSan and MSan report in DateLUT. #10798 (alexey-milovidov).
  • Fixed incorrect type conversion in key conditions. Fixes #6287. #10791 (Andrew Onyshchuk)
  • Fixed parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov).
  • Fixed combinator -OrNull and -OrDefault when combined with -State. #10741 (hcz).
  • Fixed crash in generateRandom with nested types. Fixes #10583. #10734 (Nikolai Kochetov).
  • Fixed data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov).
  • Fixed possible buffer overflow in function h3EdgeAngle. #10711 (alexey-milovidov).
  • Fixed disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes #10674. #10698 (Nikolai Kochetov).
  • Fixed multiple usages of IN operator with the identical set in one query. #10686 (Anton Popov).
  • Fixed bug, which causes http requests stuck on client close when readonly=2 and cancel_http_readonly_queries_on_client_close=1. Fixes #7939, #7019, #7736, #7091. #10684 (tavplubix).
  • Fixed order of parameters in AggregateTransform constructor. #10667 (palasonic1).
  • Fixed the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes #10655. #10664 (Nikolai Kochetov).
  • Fixed possible incorrect number of rows for queries with LIMIT. Fixes #10566, #10709. #10660 (Nikolai Kochetov).
  • Fixed a bug which locks concurrent alters when table has a lot of parts. #10659 (alesapin).
  • Fixed a bug when on SYSTEM DROP DNS CACHE query also drop caches, which are used to check if user is allowed to connect from some IP addresses. #10608 (tavplubix).
  • Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table. #10603 (Nikolai Kochetov).
  • Fixed SELECT of column ALIAS which default expression type different from column type. #10563 (Azat Khuzhin).
  • Implemented comparison between DateTime64 and String values. #10560 (Vasily Nemkov).
  • Fixed index corruption, which may accur in some cases after merge compact parts into another compact part. #10531 (Anton Popov).
  • Fixed the situation, when mutation finished all parts, but hung up in is_done=0. #10526 (alesapin).
  • Fixed overflow at beginning of unix epoch for timezones with fractional offset from UTC. This fixes #9335. #10513 (alexey-milovidov).
  • Fixed improper shutdown of Distributed storage. #10491 (Azat Khuzhin).
  • Fixed numeric overflow in simpleLinearRegression over large integers. #10474 (hcz).

Build/Testing/Packaging Improvement

Bug fix

ClickHouse release v20.3.8.53, 2020-04-23

Bug Fix

  • Fixed wrong behaviour of datetime functions for timezones that has altered between positive and negative offsets from UTC (e.g. Pacific/Kiritimati). This fixes #7202 #10369 (alexey-milovidov)
  • Fix possible segfault with distributed_group_by_no_merge enabled (introduced in 20.3.7.46 by #10131). #10399 (Nikolai Kochetov)
  • Fix wrong flattening of Array(Tuple(...)) data types. This fixes #10259 #10390 (alexey-milovidov)
  • Drop disks reservation in Aggregator. This fixes bug in disk space reservation, which may cause big external aggregation to fail even if it could be completed successfully #10375 (Azat Khuzhin)
  • Fixed DROP vs OPTIMIZE race in ReplicatedMergeTree. DROP could left some garbage in replica path in ZooKeeper if there was concurrent OPTIMIZE query. #10312 (tavplubix)
  • Fix bug when server cannot attach table after column default was altered. #10441 (alesapin)
  • Do not remove metadata directory when attach database fails before loading tables. #10442 (Winter Zhang)
  • Fixed several bugs when some data was inserted with quorum, then deleted somehow (DROP PARTITION, TTL) and this leaded to the stuck of INSERTs or false-positive exceptions in SELECTs. This fixes #9946 #10188 (Nikita Mikhaylov)
  • Fix possible Pipeline stuck error in ConcatProcessor which could have happened in remote query. #10381 (Nikolai Kochetov)
  • Fixed wrong behavior in HashTable that caused compilation error when trying to read HashMap from buffer. #10386 (palasonic1)
  • Allow to use count(*) with multiple JOINs. Fixes #9853 #10291 (Artem Zuikov)
  • Prefer fallback_to_stale_replicas over skip_unavailable_shards, otherwise when both settings specified and there are no up-to-date replicas the query will fail (patch from @alex-zaitsev). Fixes: #2564. #10422 (Azat Khuzhin)
  • Fix the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. This fixes #10226. Author: Vadim Plakhtinskiy. #10427 (alexey-milovidov)
  • Check the number and type of arguments when creating BloomFilter index #9623 #10431 (Winter Zhang)

Performance Improvement

  • Improved performance of queries with explicitly defined sets at right side of IN operator and tuples in the left side. This fixes performance regression in version 20.3. #9740, #10385 (Anton Popov)

ClickHouse release v20.3.7.46, 2020-04-17

Bug Fix

  • Fix Logical error: CROSS JOIN has expressions error for queries with comma and names joins mix. #10311 (Artem Zuikov).
  • Fix queries with max_bytes_before_external_group_by. #10302 (Artem Zuikov).
  • Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes #10092. #10195 (alexey-milovidov).
  • Add the ability to relax the restriction on non-deterministic functions usage in mutations with allow_nondeterministic_mutations setting. #10186 (filimonov).

ClickHouse release v20.3.6.40, 2020-04-16

New Feature

  • Added function isConstant. This function checks whether its argument is constant expression and returns 1 or 0. It is intended for development, debugging and demonstration purposes. #10198 (alexey-milovidov).

Bug Fix

  • Fix error Pipeline stuck with max_rows_to_group_by and group_by_overflow_mode = 'break'. #10279 (Nikolai Kochetov).
  • Fix rare possible exception Cannot drain connections: cancel first. #10239 (Nikolai Kochetov).
  • Fixed bug where ClickHouse would throw "Unknown function lambda." error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. #10237 (Alexander Kazakov).
  • Fixed "generateRandom" function for Date type. This fixes #9973. Fix an edge case when dates with year 2106 are inserted to MergeTree tables with old-style partitioning but partitions are named with year 1970. #10218 (alexey-milovidov).
  • Convert types if the table definition of a View does not correspond to the SELECT query. This fixes #10180 and #10022. #10217 (alexey-milovidov).
  • Fix parseDateTimeBestEffort for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10082. #10214 (alexey-milovidov).
  • Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. #10207 (alexey-milovidov).
  • Fix possible inifinite query execution when the query actually should stop on LIMIT, while reading from infinite source like system.numbers or system.zeros. #10206 (Nikolai Kochetov).
  • Fix using the current database for access checking when the database isn't specified. #10192 (Vitaly Baranov).
  • Convert blocks if structure does not match on INSERT into Distributed(). #10135 (Azat Khuzhin).
  • Fix possible incorrect result for extremes in processors pipeline. #10131 (Nikolai Kochetov).
  • Fix some kinds of alters with compact parts. #10130 (Anton Popov).
  • Fix incorrect index_granularity_bytes check while creating new replica. Fixes #10098. #10121 (alesapin).
  • Fix SIGSEGV on INSERT into Distributed table when its structure differs from the underlying tables. #10105 (Azat Khuzhin).
  • Fix possible rows loss for queries with JOIN and UNION ALL. Fixes #9826, #10113. #10099 (Nikolai Kochetov).
  • Fixed replicated tables startup when updating from an old ClickHouse version where /table/replicas/replica_name/metadata node doesn't exist. Fixes #10037. #10095 (alesapin).
  • Add some arguments check and support identifier arguments for MySQL Database Engine. #10077 (Winter Zhang).
  • Fix bug in clickhouse dictionary source from localhost clickhouse server. The bug may lead to memory corruption if types in dictionary and source are not compatible. #10071 (alesapin).
  • Fix bug in CHECK TABLE query when table contain skip indices. #10068 (alesapin).
  • Fix error Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform. It happened when setting distributed_aggregation_memory_efficient was enabled, and distributed query read aggregating data with different level from different shards (mixed single and two level aggregation). #10063 (Nikolai Kochetov).
  • Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (#8636, #8925). #10025 (Alexander Kuzmenkov).
  • Fix the number of threads used for remote query execution (performance regression, since 20.3). This happened when query from Distributed table was executed simultaneously on local and remote shards. Fixes #9965. #9971 (Nikolai Kochetov).
  • Fix bug in which the necessary tables weren't retrieved at one of the processing stages of queries to some databases. Fixes #9699. #9949 (achulkov2).
  • Fix 'Not found column in block' error when JOIN appears with TOTALS. Fixes #9839. #9939 (Artem Zuikov).
  • Fix a bug with ON CLUSTER DDL queries freezing on server startup. #9927 (Gagan Arneja).
  • Fix parsing multiple hosts set in the CREATE USER command, e.g. CREATE USER user6 HOST NAME REGEXP 'lo.?*host', NAME REGEXP 'lo*host'. #9924 (Vitaly Baranov).
  • Fix TRUNCATE for Join table engine (#9917). #9920 (Amos Bird).
  • Fix "scalar doesn't exist" error in ALTERs (#9878). #9904 (Amos Bird).
  • Fix race condition between drop and optimize in ReplicatedMergeTree. #9901 (alesapin).
  • Fix error with qualified names in distributed_product_mode='local'. Fixes #4756. #9891 (Artem Zuikov).
  • Fix calculating grants for introspection functions from the setting 'allow_introspection_functions'. #9840 (Vitaly Baranov).

Build/Testing/Packaging Improvement

ClickHouse release v20.3.5.21, 2020-03-27

Bug Fix

  • Fix 'Different expressions with the same alias' error when query has PREWHERE and WHERE on distributed table and SET distributed_product_mode = 'local'. #9871 (Artem Zuikov).
  • Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. #9860 (alesapin).
  • For INSERT queries shard now clamps the settings got from the initiator to the shard's constaints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix #9447. #9852 (Vitaly Baranov).
  • Fix 'COMMA to CROSS JOIN rewriter is not enabled or cannot rewrite query' error in case of subqueries with COMMA JOIN out of tables lists (i.e. in WHERE). Fixes #9782. #9830 (Artem Zuikov).
  • Fix possible exception Got 0 in totals chunk, expected 1 on client. It happened for queries with JOIN in case if right joined table had zero rows. Example: select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;. Fixes #9777. #9823 (Nikolai Kochetov).
  • Fix SIGSEGV with optimize_skip_unused_shards when type cannot be converted. #9804 (Azat Khuzhin).
  • Fix broken ALTER TABLE DELETE COLUMN query for compact parts. #9779 (alesapin).
  • Fix max_distributed_connections (w/ and w/o Processors). #9673 (Azat Khuzhin).
  • Fixed a few cases when timezone of the function argument wasn't used properly. #9574 (Vasily Nemkov).

Improvement

  • Remove order by stage from mutations because we read from a single ordered part in a single thread. Also add check that the order of rows in mutation is ordered in sorting key order and this order is not violated. #9886 (alesapin).

ClickHouse release v20.3.4.10, 2020-03-20

Bug Fix

  • This release also contains all bug fixes from 20.1.8.41
  • Fix missing rows_before_limit_at_least for queries over http (with processors pipeline). This fixes #9730. #9757 (Nikolai Kochetov)

ClickHouse release v20.3.3.6, 2020-03-17

Bug Fix

  • This release also contains all bug fixes from 20.1.7.38
  • Fix bug in a replication that doesn't allow replication to work if the user has executed mutations on the previous version. This fixes #9645. #9652 (alesapin). It makes version 20.3 backward compatible again.
  • Add setting use_compact_format_in_distributed_parts_names which allows to write files for INSERT queries into Distributed table with more compact format. This fixes #9647. #9653 (alesapin). It makes version 20.3 backward compatible again.

ClickHouse release v20.3.2.1, 2020-03-12

Backward Incompatible Change

  • Fixed the issue file name too long when sending data for Distributed tables for a large number of replicas. Fixed the issue that replica credentials were exposed in the server log. The format of directory name on disk was changed to [shard{shard_index}[_replica{replica_index}]]. #8911 (Mikhail Korotov) After you upgrade to the new version, you will not be able to downgrade without manual intervention, because old server version does not recognize the new directory format. If you want to downgrade, you have to manually rename the corresponding directories to the old format. This change is relevant only if you have used asynchronous INSERTs to Distributed tables. In the version 20.3.3 we will introduce a setting that will allow you to enable the new format gradually.
  • Changed the format of replication log entries for mutation commands. You have to wait for old mutations to process before installing the new version.
  • Implement simple memory profiler that dumps stacktraces to system.trace_log every N bytes over soft allocation limit #8765 (Ivan) #9472 (alexey-milovidov) The column of system.trace_log was renamed from timer_type to trace_type. This will require changes in third-party performance analysis and flamegraph processing tools.
  • Use OS thread id everywhere instead of internal thread number. This fixes #7477 Old clickhouse-client cannot receive logs that are send from the server when the setting send_logs_level is enabled, because the names and types of the structured log messages were changed. On the other hand, different server versions can send logs with different types to each other. When you don't use the send_logs_level setting, you should not care. #8954 (alexey-milovidov)
  • Remove indexHint function #9542 (alexey-milovidov)
  • Remove findClusterIndex, findClusterValue functions. This fixes #8641. If you were using these functions, send an email to clickhouse-feedback@yandex-team.com #9543 (alexey-milovidov)
  • Now it's not allowed to create columns or add columns with SELECT subquery as default expression. #9481 (alesapin)
  • Require aliases for subqueries in JOIN. #9274 (Artem Zuikov)
  • Improved ALTER MODIFY/ADD queries logic. Now you cannot ADD column without type, MODIFY default expression doesn't change type of column and MODIFY type doesn't loose default expression value. Fixes #8669. #9227 (alesapin)
  • Require server to be restarted to apply the changes in logging configuration. This is a temporary workaround to avoid the bug where the server logs to a deleted log file (see #8696). #8707 (Alexander Kuzmenkov)
  • The setting experimental_use_processors is enabled by default. This setting enables usage of the new query pipeline. This is internal refactoring and we expect no visible changes. If you will see any issues, set it to back zero. #8768 (alexey-milovidov)

New Feature

  • Add Avro and AvroConfluent input/output formats #8571 (Andrew Onyshchuk) #8957 (Andrew Onyshchuk) #8717 (alexey-milovidov)
  • Multi-threaded and non-blocking updates of expired keys in cache dictionaries (with optional permission to read old ones). #8303 (Nikita Mikhaylov)
  • Add query ALTER ... MATERIALIZE TTL. It runs mutation that forces to remove expired data by TTL and recalculates meta-information about TTL in all parts. #8775 (Anton Popov)
  • Switch from HashJoin to MergeJoin (on disk) if needed #9082 (Artem Zuikov)
  • Added MOVE PARTITION command for ALTER TABLE #4729 #6168 (Guillaume Tassery)
  • Reloading storage configuration from configuration file on the fly. #8594 (Vladimir Chebotarev)
  • Allowed to change storage_policy to not less rich one. #8107 (Vladimir Chebotarev)
  • Added support for globs/wildcards for S3 storage and table function. #8851 (Vladimir Chebotarev)
  • Implement bitAnd, bitOr, bitXor, bitNot for FixedString(N) datatype. #9091 (Guillaume Tassery)
  • Added function bitCount. This fixes #8702. #8708 (alexey-milovidov) #8749 (ikopylov)
  • Add generateRandom table function to generate random rows with given schema. Allows to populate arbitrary test table with data. #8994 (Ilya Yatsishin)
  • JSONEachRowFormat: support special case when objects enclosed in top-level array. #8860 (Kruglov Pavel)
  • Now it's possible to create a column with DEFAULT expression which depends on a column with default ALIAS expression. #9489 (alesapin)
  • Allow to specify --limit more than the source data size in clickhouse-obfuscator. The data will repeat itself with different random seed. #9155 (alexey-milovidov)
  • Added groupArraySample function (similar to groupArray) with reservior sampling algorithm. #8286 (Amos Bird)
  • Now you can monitor the size of update queue in cache/complex_key_cache dictionaries via system metrics. #9413 (Nikita Mikhaylov)
  • Allow to use CRLF as a line separator in CSV output format with setting output_format_csv_crlf_end_of_line is set to 1 #8934 #8935 #8963 (Mikhail Korotov)
  • Implement more functions of the H3 API: h3GetBaseCell, h3HexAreaM2, h3IndexesAreNeighbors, h3ToChildren, h3ToString and stringToH3 #8938 (Nico Mandery)
  • New setting introduced: max_parser_depth to control maximum stack size and allow large complex queries. This fixes #6681 and #7668. #8647 (Maxim Smirnov)
  • Add a setting force_optimize_skip_unused_shards setting to throw if skipping of unused shards is not possible #8805 (Azat Khuzhin)
  • Allow to configure multiple disks/volumes for storing data for send in Distributed engine #8756 (Azat Khuzhin)
  • Support storage policy (<tmp_policy>) for storing temporary data. #8750 (Azat Khuzhin)
  • Added X-ClickHouse-Exception-Code HTTP header that is set if exception was thrown before sending data. This implements #4971. #8786 (Mikhail Korotov)
  • Added function ifNotFinite. It is just a syntactic sugar: ifNotFinite(x, y) = isFinite(x) ? x : y. #8710 (alexey-milovidov)
  • Added last_successful_update_time column in system.dictionaries table #9394 (Nikita Mikhaylov)
  • Add blockSerializedSize function (size on disk without compression) #8952 (Azat Khuzhin)
  • Add function moduloOrZero #9358 (hcz)
  • Added system tables system.zeros and system.zeros_mt as well as tale functions zeros() and zeros_mt(). Tables (and table functions) contain single column with name zero and type UInt8. This column contains zeros. It is needed for test purposes as the fastest method to generate many rows. This fixes #6604 #9593 (Nikolai Kochetov)

Experimental Feature

  • Add new compact format of parts in MergeTree-family tables in which all columns are stored in one file. It helps to increase performance of small and frequent inserts. The old format (one file per column) is now called wide. Data storing format is controlled by settings min_bytes_for_wide_part and min_rows_for_wide_part. #8290 (Anton Popov)
  • Support for S3 storage for Log, TinyLog and StripeLog tables. #8862 (Pavel Kovalenko)

Bug Fix

  • Fixed inconsistent whitespaces in log messages. #9322 (alexey-milovidov)
  • Fix bug in which arrays of unnamed tuples were flattened as Nested structures on table creation. #8866 (achulkov2)
  • Fixed the issue when "Too many open files" error may happen if there are too many files matching glob pattern in File table or file table function. Now files are opened lazily. This fixes #8857 #8861 (alexey-milovidov)
  • DROP TEMPORARY TABLE now drops only temporary table. #8907 (Vitaly Baranov)
  • Remove outdated partition when we shutdown the server or DETACH/ATTACH a table. #8602 (Guillaume Tassery)
  • For how the default disk calculates the free space from data subdirectory. Fixed the issue when the amount of free space is not calculated correctly if the data directory is mounted to a separate device (rare case). This fixes #7441 #9257 (Mikhail Korotov)
  • Allow comma (cross) join with IN () inside. #9251 (Artem Zuikov)
  • Allow to rewrite CROSS to INNER JOIN if there's [NOT] LIKE operator in WHERE section. #9229 (Artem Zuikov)
  • Fix possible incorrect result after GROUP BY with enabled setting distributed_aggregation_memory_efficient. Fixes #9134. #9289 (Nikolai Kochetov)
  • Found keys were counted as missed in metrics of cache dictionaries. #9411 (Nikita Mikhaylov)
  • Fix replication protocol incompatibility introduced in #8598. #9412 (alesapin)
  • Fixed race condition on queue_task_handle at the startup of ReplicatedMergeTree tables. #9552 (alexey-milovidov)
  • The token NOT didn't work in SHOW TABLES NOT LIKE query #8727 #8940 (alexey-milovidov)
  • Added range check to function h3EdgeLengthM. Without this check, buffer overflow is possible. #8945 (alexey-milovidov)
  • Fixed up a bug in batched calculations of ternary logical OPs on multiple arguments (more than 10). #8718 (Alexander Kazakov)
  • Fix error of PREWHERE optimization, which could lead to segfaults or Inconsistent number of columns got from MergeTreeRangeReader exception. #9024 (Anton Popov)
  • Fix unexpected Timeout exceeded while reading from socket exception, which randomly happens on secure connection before timeout actually exceeded and when query profiler is enabled. Also add connect_timeout_with_failover_secure_ms settings (default 100ms), which is similar to connect_timeout_with_failover_ms, but is used for secure connections (because SSL handshake is slower, than ordinary TCP connection) #9026 (tavplubix)
  • Fix bug with mutations finalization, when mutation may hang in state with parts_to_do=0 and is_done=0. #9022 (alesapin)
  • Use new ANY JOIN logic with partial_merge_join setting. It's possible to make ANY|ALL|SEMI LEFT and ALL INNER joins with partial_merge_join=1 now. #8932 (Artem Zuikov)
  • Shard now clamps the settings got from the initiator to the shard's constaints instead of throwing an exception. This fix allows to send queries to a shard with another constraints. #9447 (Vitaly Baranov)
  • Fixed memory management problem in MergeTreeReadPool. #8791 (Vladimir Chebotarev)
  • Fix toDecimal*OrNull() functions family when called with string e. Fixes #8312 #8764 (Artem Zuikov)
  • Make sure that FORMAT Null sends no data to the client. #8767 (Alexander Kuzmenkov)
  • Fix bug that timestamp in LiveViewBlockInputStream will not updated. LIVE VIEW is an experimental feature. #8644 (vxider) #8625 (vxider)
  • Fixed ALTER MODIFY TTL wrong behavior which did not allow to delete old TTL expressions. #8422 (Vladimir Chebotarev)
  • Fixed UBSan report in MergeTreeIndexSet. This fixes #9250 #9365 (alexey-milovidov)
  • Fixed the behaviour of match and extract functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes #9160 #9163 (alexey-milovidov) #9345 (alexey-milovidov)
  • Avoid throwing from destructor in Apache Avro 3rd-party library. #9066 (Andrew Onyshchuk)
  • Don't commit a batch polled from Kafka partially as it can lead to holes in data. #8876 (filimonov)
  • Fix joinGet with nullable return types. https://github.com/ClickHouse/ClickHouse/issues/8919 #9014 (Amos Bird)
  • Fix data incompatibility when compressed with T64 codec. #9016 (Artem Zuikov) Fix data type ids in T64 compression codec that leads to wrong (de)compression in affected versions. #9033 (Artem Zuikov)
  • Add setting enable_early_constant_folding and disable it in some cases that leads to errors. #9010 (Artem Zuikov)
  • Fix pushdown predicate optimizer with VIEW and enable the test #9011 (Winter Zhang)
  • Fix segfault in Merge tables, that can happen when reading from File storages #9387 (tavplubix)
  • Added a check for storage policy in ATTACH PARTITION FROM, REPLACE PARTITION, MOVE TO TABLE. Otherwise it could make data of part inaccessible after restart and prevent ClickHouse to start. #9383 (Vladimir Chebotarev)
  • Fix alters if there is TTL set for table. #8800 (Anton Popov)
  • Fix race condition that can happen when SYSTEM RELOAD ALL DICTIONARIES is executed while some dictionary is being modified/added/removed. #8801 (Vitaly Baranov)
  • In previous versions Memory database engine use empty data path, so tables are created in path directory (e.g. /var/lib/clickhouse/), not in data directory of database (e.g. /var/lib/clickhouse/db_name). #8753 (tavplubix)
  • Fixed wrong log messages about missing default disk or policy. #9530 (Vladimir Chebotarev)
  • Fix not(has()) for the bloom_filter index of array types. #9407 (achimbab)
  • Allow first column(s) in a table with Log engine be an alias #9231 (Ivan)
  • Fix order of ranges while reading from MergeTree table in one thread. It could lead to exceptions from MergeTreeRangeReader or wrong query results. #9050 (Anton Popov)
  • Make reinterpretAsFixedString to return FixedString instead of String. #9052 (Andrew Onyshchuk)
  • Avoid extremely rare cases when the user can get wrong error message (Success instead of detailed error description). #9457 (alexey-milovidov)
  • Do not crash when using Template format with empty row template. #8785 (Alexander Kuzmenkov)
  • Metadata files for system tables could be created in wrong place #8653 (tavplubix) Fixes #8581.
  • Fix data race on exception_ptr in cache dictionary #8303. #9379 (Nikita Mikhaylov)
  • Do not throw an exception for query ATTACH TABLE IF NOT EXISTS. Previously it was thrown if table already exists, despite the IF NOT EXISTS clause. #8967 (Anton Popov)
  • Fixed missing closing paren in exception message. #8811 (alexey-milovidov)
  • Avoid message Possible deadlock avoided at the startup of clickhouse-client in interactive mode. #9455 (alexey-milovidov)
  • Fixed the issue when padding at the end of base64 encoded value can be malformed. Update base64 library. This fixes #9491, closes #9492 #9500 (alexey-milovidov)
  • Prevent losing data in Kafka in rare cases when exception happens after reading suffix but before commit. Fixes #9378 #9507 (filimonov)
  • Fixed exception in DROP TABLE IF EXISTS #8663 (Nikita Vasilev)
  • Fix crash when a user tries to ALTER MODIFY SETTING for old-formated MergeTree table engines family. #9435 (alesapin)
  • Support for UInt64 numbers that don't fit in Int64 in JSON-related functions. Update SIMDJSON to master. This fixes #9209 #9344 (alexey-milovidov)
  • Fixed execution of inversed predicates when non-strictly monotinic functional index is used. #9223 (Alexander Kazakov)
  • Don't try to fold IN constant in GROUP BY #8868 (Amos Bird)
  • Fix bug in ALTER DELETE mutations which leads to index corruption. This fixes #9019 and #8982. Additionally fix extremely rare race conditions in ReplicatedMergeTree ALTER queries. #9048 (alesapin)
  • When the setting compile_expressions is enabled, you can get unexpected column in LLVMExecutableFunction when we use Nullable type #8910 (Guillaume Tassery)
  • Multiple fixes for Kafka engine: 1) fix duplicates that were appearing during consumer group rebalance. 2) Fix rare 'holes' appeared when data were polled from several partitions with one poll and committed partially (now we always process / commit the whole polled block of messages). 3) Fix flushes by block size (before that only flushing by timeout was working properly). 4) better subscription procedure (with assignment feedback). 5) Make tests work faster (with default intervals and timeouts). Due to the fact that data was not flushed by block size before (as it should according to documentation), that PR may lead to some performance degradation with default settings (due to more often & tinier flushes which are less optimal). If you encounter the performance issue after that change - please increase kafka_max_block_size in the table to the bigger value ( for example CREATE TABLE ...Engine=Kafka ... SETTINGS ... kafka_max_block_size=524288). Fixes #7259 #8917 (filimonov)
  • Fix Parameter out of bound exception in some queries after PREWHERE optimizations. #8914 (Baudouin Giard)
  • Fixed the case of mixed-constness of arguments of function arrayZip. #8705 (alexey-milovidov)
  • When executing CREATE query, fold constant expressions in storage engine arguments. Replace empty database name with current database. Fixes #6508, #3492 #9262 (tavplubix)
  • Now it's not possible to create or add columns with simple cyclic aliases like a DEFAULT b, b DEFAULT a. #9603 (alesapin)
  • Fixed a bug with double move which may corrupt original part. This is relevant if you use ALTER TABLE MOVE #8680 (Vladimir Chebotarev)
  • Allow interval identifier to correctly parse without backticks. Fixed issue when a query cannot be executed even if the interval identifier is enclosed in backticks or double quotes. This fixes #9124. #9142 (alexey-milovidov)
  • Fixed fuzz test and incorrect behaviour of bitTestAll/bitTestAny functions. #9143 (alexey-milovidov)
  • Fix possible crash/wrong number of rows in LIMIT n WITH TIES when there are a lot of rows equal to n'th row. #9464 (tavplubix)
  • Fix mutations with parts written with enabled insert_quorum. #9463 (alesapin)
  • Fix data race at destruction of Poco::HTTPServer. It could happen when server is started and immediately shut down. #9468 (Anton Popov)
  • Fix bug in which a misleading error message was shown when running SHOW CREATE TABLE a_table_that_does_not_exist. #8899 (achulkov2)
  • Fixed Parameters are out of bound exception in some rare cases when we have a constant in the SELECT clause when we have an ORDER BY and a LIMIT clause. #8892 (Guillaume Tassery)
  • Fix mutations finalization, when already done mutation can have status is_done=0. #9217 (alesapin)
  • Prevent from executing ALTER ADD INDEX for MergeTree tables with old syntax, because it doesn't work. #8822 (Mikhail Korotov)
  • During server startup do not access table, which LIVE VIEW depends on, so server will be able to start. Also remove LIVE VIEW dependencies when detaching LIVE VIEW. LIVE VIEW is an experimental feature. #8824 (tavplubix)
  • Fix possible segfault in MergeTreeRangeReader, while executing PREWHERE. #9106 (Anton Popov)
  • Fix possible mismatched checksums with column TTLs. #9451 (Anton Popov)
  • Fixed a bug when parts were not being moved in background by TTL rules in case when there is only one volume. #8672 (Vladimir Chebotarev)
  • Fixed the issue Method createColumn() is not implemented for data type Set. This fixes #7799. #8674 (alexey-milovidov)
  • Now we will try finalize mutations more frequently. #9427 (alesapin)
  • Fix intDiv by minus one constant #9351 (hcz)
  • Fix possible race condition in BlockIO. #9356 (Nikolai Kochetov)
  • Fix bug leading to server termination when trying to use / drop Kafka table created with wrong parameters. #9513 (filimonov)
  • Added workaround if OS returns wrong result for timer_create function. #8837 (alexey-milovidov)
  • Fixed error in usage of min_marks_for_seek parameter. Fixed the error message when there is no sharding key in Distributed table and we try to skip unused shards. #8908 (Azat Khuzhin)

Improvement

  • Implement ALTER MODIFY/DROP queries on top of mutations for ReplicatedMergeTree* engines family. Now ALTERS blocks only at the metadata update stage, and don't block after that. #8701 (alesapin)
  • Add ability to rewrite CROSS to INNER JOINs with WHERE section containing unqialified names. #9512 (Artem Zuikov)
  • Make SHOW TABLES and SHOW DATABASES queries support the WHERE expressions and FROM/IN #9076 (sundyli)
  • Added a setting deduplicate_blocks_in_dependent_materialized_views. #9070 (urykhy)
  • After recent changes MySQL client started to print binary strings in hex thereby making them not readable (#9032). The workaround in ClickHouse is to mark string columns as UTF-8, which is not always, but usually the case. #9079 (Yuriy Baranov)
  • Add support of String and FixedString keys for sumMap #8903 (Baudouin Giard)
  • Support string keys in SummingMergeTree maps #8933 (Baudouin Giard)
  • Signal termination of thread to the thread pool even if the thread has thrown exception #8736 (Ding Xiang Fei)
  • Allow to set query_id in clickhouse-benchmark #9416 (Anton Popov)
  • Don't allow strange expressions in ALTER TABLE ... PARTITION partition query. This addresses #7192 #8835 (alexey-milovidov)
  • The table system.table_engines now provides information about feature support (like supports_ttl or supports_sort_order). #8830 (Max Akhmedov)
  • Enable system.metric_log by default. It will contain rows with values of ProfileEvents, CurrentMetrics collected with "collect_interval_milliseconds" interval (one second by default). The table is very small (usually in order of megabytes) and collecting this data by default is reasonable. #9225 (alexey-milovidov)
  • Initialize query profiler for all threads in a group, e.g. it allows to fully profile insert-queries. Fixes #6964 #8874 (Ivan)
  • Now temporary LIVE VIEW is created by CREATE LIVE VIEW name WITH TIMEOUT [42] ... instead of CREATE TEMPORARY LIVE VIEW ..., because the previous syntax was not consistent with CREATE TEMPORARY TABLE ... #9131 (tavplubix)
  • Add text_log.level configuration parameter to limit entries that goes to system.text_log table #8809 (Azat Khuzhin)
  • Allow to put downloaded part to a disks/volumes according to TTL rules #8598 (Vladimir Chebotarev)
  • For external MySQL dictionaries, allow to mutualize MySQL connection pool to "share" them among dictionaries. This option significantly reduces the number of connections to MySQL servers. #9409 (Clément Rodriguez)
  • Show nearest query execution time for quantiles in clickhouse-benchmark output instead of interpolated values. It's better to show values that correspond to the execution time of some queries. #8712 (alexey-milovidov)
  • Possibility to add key & timestamp for the message when inserting data to Kafka. Fixes #7198 #8969 (filimonov)
  • If server is run from terminal, highlight thread number, query id and log priority by colors. This is for improved readability of correlated log messages for developers. #8961 (alexey-milovidov)
  • Better exception message while loading tables for Ordinary database. #9527 (alexey-milovidov)
  • Implement arraySlice for arrays with aggregate function states. This fixes #9388 #9391 (alexey-milovidov)
  • Allow constant functions and constant arrays to be used on the right side of IN operator. #8813 (Anton Popov)
  • If zookeeper exception has happened while fetching data for system.replicas, display it in a separate column. This implements #9137 #9138 (alexey-milovidov)
  • Atomically remove MergeTree data parts on destroy. #8402 (Vladimir Chebotarev)
  • Support row-level security for Distributed tables. #8926 (Ivan)
  • Now we recognize suffix (like KB, KiB...) in settings values. #8072 (Mikhail Korotov)
  • Prevent out of memory while constructing result of a large JOIN. #8637 (Artem Zuikov)
  • Added names of clusters to suggestions in interactive mode in clickhouse-client. #8709 (alexey-milovidov)
  • Initialize query profiler for all threads in a group, e.g. it allows to fully profile insert-queries #8820 (Ivan)
  • Added column exception_code in system.query_log table. #8770 (Mikhail Korotov)
  • Enabled MySQL compatibility server on port 9004 in the default server configuration file. Fixed password generation command in the example in configuration. #8771 (Yuriy Baranov)
  • Prevent abort on shutdown if the filesystem is readonly. This fixes #9094 #9100 (alexey-milovidov)
  • Better exception message when length is required in HTTP POST query. #9453 (alexey-milovidov)
  • Add _path and _file virtual columns to HDFS and File engines and hdfs and file table functions #8489 (Olga Khvostikova)
  • Fix error Cannot find column while inserting into MATERIALIZED VIEW in case if new column was added to view's internal table. #8766 #8788 (vzakaznikov) #8788 #8806 (Nikolai Kochetov) #8803 (Nikolai Kochetov)
  • Fix progress over native client-server protocol, by send progress after final update (like logs). This may be relevant only to some third-party tools that are using native protocol. #9495 (Azat Khuzhin)
  • Add a system metric tracking the number of client connections using MySQL protocol (#9013). #9015 (Eugene Klimov)
  • From now on, HTTP responses will have X-ClickHouse-Timezone header set to the same timezone value that SELECT timezone() would report. #9493 (Denis Glazachev)

Performance Improvement

  • Improve performance of analysing index with IN #9261 (Anton Popov)
  • Simpler and more efficient code in Logical Functions + code cleanups. A followup to #8718 #8728 (Alexander Kazakov)
  • Overall performance improvement (in range of 5%..200% for affected queries) by ensuring even more strict aliasing with C++20 features. #9304 (Amos Bird)
  • More strict aliasing for inner loops of comparison functions. #9327 (alexey-milovidov)
  • More strict aliasing for inner loops of arithmetic functions. #9325 (alexey-milovidov)
  • A ~3 times faster implementation for ColumnVector::replicate(), via which ColumnConst::convertToFullColumn() is implemented. Also will be useful in tests when materializing constants. #9293 (Alexander Kazakov)
  • Another minor performance improvement to ColumnVector::replicate() (this speeds up the materialize function and higher order functions) an even further improvement to #9293 #9442 (Alexander Kazakov)
  • Improved performance of stochasticLinearRegression aggregate function. This patch is contributed by Intel. #8652 (alexey-milovidov)
  • Improve performance of reinterpretAsFixedString function. #9342 (alexey-milovidov)
  • Do not send blocks to client for Null format in processors pipeline. #8797 (Nikolai Kochetov) #8767 (Alexander Kuzmenkov)

Build/Testing/Packaging Improvement

ClickHouse release v20.1

ClickHouse release v20.1.16.120-stable 2020-60-26

Bug Fix

  • Fix rare crash caused by using Nullable column in prewhere condition. Continuation of #11608. #11869 (Nikolai Kochetov).
  • Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (alexey-milovidov).
  • Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. #11753 (hexiaoting).
  • Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (tavplubix).
  • Fix ORDER BY ... WITH FILL over const columns. #11697 (Anton Popov).
  • Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (alexey-milovidov).
  • Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (alexey-milovidov).
  • Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
  • Fix rare crash caused by using Nullable column in prewhere condition. (Probably it is connected with #11572 somehow). #11608 (Nikolai Kochetov).
  • Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. #11601 (filimonov).
  • Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (alexey-milovidov).
  • Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
  • Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. #11496 (alexey-milovidov).
  • Fix usage of primary key wrapped into a function with 'FINAL' modifier and 'ORDER BY' optimization. #10715 (Anton Popov).

ClickHouse release v20.1.15.109-stable 2020-06-19

Bug Fix

ClickHouse release v20.1.14.107-stable 2020-06-11

Bug Fix

  • Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. #11580 (Nikolai Kochetov).

ClickHouse release v20.1.13.105-stable 2020-06-10

Bug Fix

  • Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (alexey-milovidov).
  • Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
  • Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
  • Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. #11421 (Alexey Ilyukhov).
  • Fixed geohashesInBox with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov).
  • Fix possible Pipeline stuck error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov).
  • Fix crash in quantilesExactWeightedArray. #11337 (Nikolai Kochetov).
  • Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes #10241. #11330 (Nikolai Kochetov).
  • Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 (Ewout).
  • Fix very rare race condition in ThreadPool. #11314 (alexey-milovidov).
  • Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). #11311 (alexey-milovidov).
  • Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes #11286. #11303 (alexey-milovidov).
  • Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. #11296 (alexey-milovidov).
  • Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. #11258 (Vitaly Baranov).
  • Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. #11185 (Nikolai Kochetov).
  • If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
  • Remove logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
  • Fixed parseDateTime64BestEffort argument resolution bugs. #10925. #11038 (Vasily Nemkov).
  • Fix incorrect raw data size in method getRawData(). #10964 (Igr).
  • Fix backward compatibility with tuples in Distributed tables. #10889 (Anton Popov).
  • Fix SIGSEGV in StringHashTable (if such key does not exist). #10870 (Azat Khuzhin).
  • Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. #10849 (tavplubix).
  • Fix columns order after Block::sortColumns() (also add a test that shows that it affects some real use case - Buffer engine). #10826 (Azat Khuzhin).
  • Fix the issue with ODBC bridge when no quoting of identifiers is requested. This fixes #7984. #10821 (alexey-milovidov).
  • Fix UBSan and MSan report in DateLUT. #10798 (alexey-milovidov).
  • Fix parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov).
  • Fix combinator -OrNull and -OrDefault when combined with -State. #10741 (hcz).
  • Fix disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes #10674. #10698 (Nikolai Kochetov).
  • Fix multiple usages of IN operator with the identical set in one query. #10686 (Anton Popov).
  • Fix order of parameters in AggregateTransform constructor. #10667 (palasonic1).
  • Fix the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes #10655. #10664 (Nikolai Kochetov).
  • Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin).
  • Fix error the BloomFilter false positive must be a double number between 0 and 1 #10551. #10569 (Winter Zhang).
  • Fix SELECT of column ALIAS which default expression type different from column type. #10563 (Azat Khuzhin).
    • Implemented comparison between DateTime64 and String values (just like for DateTime). #10560 (Vasily Nemkov).

ClickHouse release v20.1.12.86, 2020-05-26

Bug Fix

  • Fixed incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 (alexey-milovidov).
  • Fixed data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov).
  • Fixed bug, which causes http requests stuck on client close when readonly=2 and cancel_http_readonly_queries_on_client_close=1. Fixes #7939, #7019, #7736, #7091. #10684 (tavplubix).
  • Fixed a bug when on SYSTEM DROP DNS CACHE query also drop caches, which are used to check if user is allowed to connect from some IP addresses. #10608 (tavplubix).
  • Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table. #10603 (Nikolai Kochetov).
  • Fixed the situation when mutation finished all parts, but hung up in is_done=0. #10526 (alesapin).
  • Fixed overflow at beginning of unix epoch for timezones with fractional offset from UTC. This fixes #9335. #10513 (alexey-milovidov).
  • Fixed improper shutdown of Distributed storage. #10491 (Azat Khuzhin).
  • Fixed numeric overflow in simpleLinearRegression over large integers. #10474 (hcz).
  • Fixed removing metadata directory when attach database fails. #10442 (Winter Zhang).
  • Added a check of number and type of arguments when creating BloomFilter index #9623. #10431 (Winter Zhang).
  • Fixed the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. This fixes #10226. #10427 (alexey-milovidov).
  • Prefer fallback_to_stale_replicas over skip_unavailable_shards. #10422 (Azat Khuzhin).
  • Fixed wrong flattening of Array(Tuple(...)) data types. This fixes #10259. #10390 (alexey-milovidov).
  • Fixed wrong behavior in HashTable that caused compilation error when trying to read HashMap from buffer. #10386 (palasonic1).
  • Fixed possible Pipeline stuck error in ConcatProcessor which could have happened in remote query. #10381 (Nikolai Kochetov).
  • Fixed error Pipeline stuck with max_rows_to_group_by and group_by_overflow_mode = 'break'. #10279 (Nikolai Kochetov).
  • Fixed several bugs when some data was inserted with quorum, then deleted somehow (DROP PARTITION, TTL) and this leaded to the stuck of INSERTs or false-positive exceptions in SELECTs. This fixes #9946. #10188 (Nikita Mikhaylov).
  • Fixed incompatibility when versions prior to 18.12.17 are used on remote servers and newer is used on initiating server, and GROUP BY both fixed and non-fixed keys, and when two-level group by method is activated. #3254 (alexey-milovidov).

Build/Testing/Packaging Improvement

  • Added CA certificates to clickhouse-server docker image. #10476 (filimonov).

ClickHouse release v20.1.10.70, 2020-04-17

Bug Fix

  • Fix rare possible exception Cannot drain connections: cancel first. #10239 (Nikolai Kochetov).
  • Fixed bug where ClickHouse would throw 'Unknown function lambda.' error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. #10237 (Alexander Kazakov).
  • Fix parseDateTimeBestEffort for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10082. #10214 (alexey-milovidov).
  • Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. #10207 (alexey-milovidov).
  • Fix possible inifinite query execution when the query actually should stop on LIMIT, while reading from infinite source like system.numbers or system.zeros. #10206 (Nikolai Kochetov).
  • Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes #10092. #10195 (alexey-milovidov).
  • Add the ability to relax the restriction on non-deterministic functions usage in mutations with allow_nondeterministic_mutations setting. #10186 (filimonov).
  • Convert blocks if structure does not match on INSERT into table with Distributed engine. #10135 (Azat Khuzhin).
  • Fix SIGSEGV on INSERT into Distributed table when its structure differs from the underlying tables. #10105 (Azat Khuzhin).
  • Fix possible rows loss for queries with JOIN and UNION ALL. Fixes #9826, #10113. #10099 (Nikolai Kochetov).
  • Add arguments check and support identifier arguments for MySQL Database Engine. #10077 (Winter Zhang).
  • Fix bug in clickhouse dictionary source from localhost clickhouse server. The bug may lead to memory corruption if types in dictionary and source are not compatible. #10071 (alesapin).
  • Fix error Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform. It happened when setting distributed_aggregation_memory_efficient was enabled, and distributed query read aggregating data with different level from different shards (mixed single and two level aggregation). #10063 (Nikolai Kochetov).
  • Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (#8636, #8925). #10025 (Alexander Kuzmenkov).
  • Fix bug in which the necessary tables weren't retrieved at one of the processing stages of queries to some databases. Fixes #9699. #9949 (achulkov2).
  • Fix 'Not found column in block' error when JOIN appears with TOTALS. Fixes #9839. #9939 (Artem Zuikov).
  • Fix a bug with ON CLUSTER DDL queries freezing on server startup. #9927 (Gagan Arneja).
  • Fix TRUNCATE for Join table engine (#9917). #9920 (Amos Bird).
  • Fix 'scalar doesn't exist' error in ALTER queries (#9878). #9904 (Amos Bird).
  • Fix race condition between drop and optimize in ReplicatedMergeTree. #9901 (alesapin).
  • Fixed DeleteOnDestroy logic in ATTACH PART which could lead to automatic removal of attached part and added few tests. #9410 (Vladimir Chebotarev).

Build/Testing/Packaging Improvement

ClickHouse release v20.1.9.54, 2020-03-28

Bug Fix

  • Fix 'Different expressions with the same alias' error when query has PREWHERE and WHERE on distributed table and SET distributed_product_mode = 'local'. #9871 (Artem Zuikov).
  • Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. #9860 (alesapin).
  • For INSERT queries shard now clamps the settings got from the initiator to the shard's constaints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix #9447. #9852 (Vitaly Baranov).
  • Fix possible exception Got 0 in totals chunk, expected 1 on client. It happened for queries with JOIN in case if right joined table had zero rows. Example: select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;. Fixes #9777. #9823 (Nikolai Kochetov).
  • Fix SIGSEGV with optimize_skip_unused_shards when type cannot be converted. #9804 (Azat Khuzhin).
  • Fixed a few cases when timezone of the function argument wasn't used properly. #9574 (Vasily Nemkov).

Improvement

  • Remove ORDER BY stage from mutations because we read from a single ordered part in a single thread. Also add check that the order of rows in mutation is ordered in sorting key order and this order is not violated. #9886 (alesapin).

Build/Testing/Packaging Improvement

  • Clean up duplicated linker flags. Make sure the linker won't look up an unexpected symbol. #9433 (Amos Bird).

ClickHouse release v20.1.8.41, 2020-03-20

Bug Fix

  • Fix possible permanent Cannot schedule a task error (due to unhandled exception in ParallelAggregatingBlockInputStream::Handler::onFinish/onFinishThread). This fixes #6833. #9154 (Azat Khuzhin)
  • Fix excessive memory consumption in ALTER queries (mutations). This fixes #9533 and #9670. #9754 (alesapin)
  • Fix bug in backquoting in external dictionaries DDL. This fixes #9619. #9734 (alesapin)

ClickHouse release v20.1.7.38, 2020-03-18

Bug Fix

  • Fixed incorrect internal function names for sumKahan and sumWithOverflow. I lead to exception while using this functions in remote queries. #9636 (Azat Khuzhin). This issue was in all ClickHouse releases.
  • Allow ALTER ON CLUSTER of Distributed tables with internal replication. This fixes #3268. #9617 (shinoi2). This issue was in all ClickHouse releases.
  • Fix possible exceptions Size of filter doesn't match size of column and Invalid number of rows in Chunk in MergeTreeRangeReader. They could appear while executing PREWHERE in some cases. Fixes #9132. #9612 (Anton Popov)
  • Fixed the issue: timezone was not preserved if you write a simple arithmetic expression like time + 1 (in contrast to an expression like time + INTERVAL 1 SECOND). This fixes #5743. #9323 (alexey-milovidov). This issue was in all ClickHouse releases.
  • Now it's not possible to create or add columns with simple cyclic aliases like a DEFAULT b, b DEFAULT a. #9603 (alesapin)
  • Fixed the issue when padding at the end of base64 encoded value can be malformed. Update base64 library. This fixes #9491, closes #9492 #9500 (alexey-milovidov)
  • Fix data race at destruction of Poco::HTTPServer. It could happen when server is started and immediately shut down. #9468 (Anton Popov)
  • Fix possible crash/wrong number of rows in LIMIT n WITH TIES when there are a lot of rows equal to n'th row. #9464 (tavplubix)
  • Fix possible mismatched checksums with column TTLs. #9451 (Anton Popov)
  • Fix crash when a user tries to ALTER MODIFY SETTING for old-formated MergeTree table engines family. #9435 (alesapin)
  • Now we will try finalize mutations more frequently. #9427 (alesapin)
  • Fix replication protocol incompatibility introduced in #8598. #9412 (alesapin)
  • Fix not(has()) for the bloom_filter index of array types. #9407 (achimbab)
  • Fixed the behaviour of match and extract functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes #9160 #9163 (alexey-milovidov) #9345 (alexey-milovidov)

Build/Testing/Packaging Improvement

ClickHouse release v20.1.6.30, 2020-03-05

Bug Fix

  • Fix data incompatibility when compressed with T64 codec. #9039 (abyss7)
  • Fix order of ranges while reading from MergeTree table in one thread. Fixes #8964. #9050 (CurtizJ)
  • Fix possible segfault in MergeTreeRangeReader, while executing PREWHERE. Fixes #9064. #9106 (CurtizJ)
  • Fix reinterpretAsFixedString to return FixedString instead of String. #9052 (oandrew)
  • Fix joinGet with nullable return types. Fixes #8919 #9014 (amosbird)
  • Fix fuzz test and incorrect behaviour of bitTestAll/bitTestAny functions. #9143 (alexey-milovidov)
  • Fix the behaviour of match and extract functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. Fixes #9160 #9163 (alexey-milovidov)
  • Fixed execution of inversed predicates when non-strictly monotinic functional index is used. Fixes #9034 #9223 (Akazz)
  • Allow to rewrite CROSS to INNER JOIN if there's [NOT] LIKE operator in WHERE section. Fixes #9191 #9229 (4ertus2)
  • Allow first column(s) in a table with Log engine be an alias. #9231 (abyss7)
  • Allow comma join with IN() inside. Fixes #7314. #9251 (4ertus2)
  • Improve ALTER MODIFY/ADD queries logic. Now you cannot ADD column without type, MODIFY default expression doesn't change type of column and MODIFY type doesn't loose default expression value. Fixes #8669. #9227 (alesapin)
  • Fix mutations finalization, when already done mutation can have status is_done=0. #9217 (alesapin)
  • Support "Processors" pipeline for system.numbers and system.numbers_mt. This also fixes the bug when max_execution_time is not respected. #7796 (KochetovNicolai)
  • Fix wrong counting of DictCacheKeysRequestedFound metric. #9411 (nikitamikhaylov)
  • Added a check for storage policy in ATTACH PARTITION FROM, REPLACE PARTITION, MOVE TO TABLE which otherwise could make data of part inaccessible after restart and prevent ClickHouse to start. #9383 (excitoon)
  • Fixed UBSan report in MergeTreeIndexSet. This fixes #9250 #9365 (alexey-milovidov)
  • Fix possible datarace in BlockIO. #9356 (KochetovNicolai)
  • Support for UInt64 numbers that don't fit in Int64 in JSON-related functions. Update SIMDJSON to master. This fixes #9209 #9344 (alexey-milovidov)
  • Fix the issue when the amount of free space is not calculated correctly if the data directory is mounted to a separate device. For default disk calculate the free space from data subdirectory. This fixes #7441 #9257 (millb)
  • Fix the issue when TLS connections may fail with the message OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error and SSL Exception: error:2400006E:random number generator::error retrieving entropy. Update OpenSSL to upstream master. #8956 (alexey-milovidov)
  • When executing CREATE query, fold constant expressions in storage engine arguments. Replace empty database name with current database. Fixes #6508, #3492. Also fix check for local address in ClickHouseDictionarySource. #9262 (tabplubix)
  • Fix segfault in StorageMerge, which can happen when reading from StorageFile. #9387 (tabplubix)
  • Prevent losing data in Kafka in rare cases when exception happens after reading suffix but before commit. Fixes #9378. Related: #7175 #9507 (filimonov)
  • Fix bug leading to server termination when trying to use / drop Kafka table created with wrong parameters. Fixes #9494. Incorporates #9507. #9513 (filimonov)

New Feature

  • Add deduplicate_blocks_in_dependent_materialized_views option to control the behaviour of idempotent inserts into tables with materialized views. This new feature was added to the bugfix release by a special request from Altinity. #9070 (urykhy)

ClickHouse release v20.1.2.4, 2020-01-22

Backward Incompatible Change

  • Make the setting merge_tree_uniform_read_distribution obsolete. The server still recognizes this setting but it has no effect. #8308 (alexey-milovidov)
  • Changed return type of the function greatCircleDistance to Float32 because now the result of calculation is Float32. #7993 (alexey-milovidov)
  • Now it's expected that query parameters are represented in "escaped" format. For example, to pass string a<tab>b you have to write a\tb or a\<tab>b and respectively, a%5Ctb or a%5C%09b in URL. This is needed to add the possibility to pass NULL as \N. This fixes #7488. #8517 (alexey-milovidov)
  • Enable use_minimalistic_part_header_in_zookeeper setting for ReplicatedMergeTree by default. This will significantly reduce amount of data stored in ZooKeeper. This setting is supported since version 19.1 and we already use it in production in multiple services without any issues for more than half a year. Disable this setting if you have a chance to downgrade to versions older than 19.1. #6850 (alexey-milovidov)
  • Data skipping indices are production ready and enabled by default. The settings allow_experimental_data_skipping_indices, allow_experimental_cross_to_join_conversion and allow_experimental_multiple_joins_emulation are now obsolete and do nothing. #7974 (alexey-milovidov)
  • Add new ANY JOIN logic for StorageJoin consistent with JOIN operation. To upgrade without changes in behaviour you need add SETTINGS any_join_distinct_right_table_keys = 1 to Engine Join tables metadata or recreate these tables after upgrade. #8400 (Artem Zuikov)
  • Require server to be restarted to apply the changes in logging configuration. This is a temporary workaround to avoid the bug where the server logs to a deleted log file (see #8696). #8707 (Alexander Kuzmenkov)

New Feature

  • Added information about part paths to system.merges. #8043 (Vladimir Chebotarev)
  • Add ability to execute SYSTEM RELOAD DICTIONARY query in ON CLUSTER mode. #8288 (Guillaume Tassery)
  • Add ability to execute CREATE DICTIONARY queries in ON CLUSTER mode. #8163 (alesapin)
  • Now user's profile in users.xml can inherit multiple profiles. #8343 (Mikhail f. Shiryaev)
  • Added system.stack_trace table that allows to look at stack traces of all server threads. This is useful for developers to introspect server state. This fixes #7576. #8344 (alexey-milovidov)
  • Add DateTime64 datatype with configurable sub-second precision. #7170 (Vasily Nemkov)
  • Add table function clusterAllReplicas which allows to query all the nodes in the cluster. #8493 (kiran sunkari)
  • Add aggregate function categoricalInformationValue which calculates the information value of a discrete feature. #8117 (hcz)
  • Speed up parsing of data files in CSV, TSV and JSONEachRow format by doing it in parallel. #7780 (Alexander Kuzmenkov)
  • Add function bankerRound which performs banker's rounding. #8112 (hcz)
  • Support more languages in embedded dictionary for region names: 'ru', 'en', 'ua', 'uk', 'by', 'kz', 'tr', 'de', 'uz', 'lv', 'lt', 'et', 'pt', 'he', 'vi'. #8189 (alexey-milovidov)
  • Improvements in consistency of ANY JOIN logic. Now t1 ANY LEFT JOIN t2 equals t2 ANY RIGHT JOIN t1. #7665 (Artem Zuikov)
  • Add setting any_join_distinct_right_table_keys which enables old behaviour for ANY INNER JOIN. #7665 (Artem Zuikov)
  • Add new SEMI and ANTI JOIN. Old ANY INNER JOIN behaviour now available as SEMI LEFT JOIN. #7665 (Artem Zuikov)
  • Added Distributed format for File engine and file table function which allows to read from .bin files generated by asynchronous inserts into Distributed table. #8535 (Nikolai Kochetov)
  • Add optional reset column argument for runningAccumulate which allows to reset aggregation results for each new key value. #8326 (Sergey Kononenko)
  • Add ability to use ClickHouse as Prometheus endpoint. #7900 (vdimir)
  • Add section <remote_url_allow_hosts> in config.xml which restricts allowed hosts for remote table engines and table functions URL, S3, HDFS. #7154 (Mikhail Korotov)
  • Added function greatCircleAngle which calculates the distance on a sphere in degrees. #8105 (alexey-milovidov)
  • Changed Earth radius to be consistent with H3 library. #8105 (alexey-milovidov)
  • Added JSONCompactEachRow and JSONCompactEachRowWithNamesAndTypes formats for input and output. #7841 (Mikhail Korotov)
  • Added feature for file-related table engines and table functions (File, S3, URL, HDFS) which allows to read and write gzip files based on additional engine parameter or file extension. #7840 (Andrey Bodrov)
  • Added the randomASCII(length) function, generating a string with a random set of ASCII printable characters. #8401 (BayoNet)
  • Added function JSONExtractArrayRaw which returns an array on unparsed json array elements from JSON string. #8081 (Oleg Matrokhin)
  • Add arrayZip function which allows to combine multiple arrays of equal lengths into one array of tuples. #8149 (Winter Zhang)
  • Add ability to move data between disks according to configured TTL-expressions for *MergeTree table engines family. #8140 (Vladimir Chebotarev)
  • Added new aggregate function avgWeighted which allows to calculate weighted average. #7898 (Andrey Bodrov)
  • Now parallel parsing is enabled by default for TSV, TSKV, CSV and JSONEachRow formats. #7894 (Nikita Mikhaylov)
  • Add several geo functions from H3 library: h3GetResolution, h3EdgeAngle, h3EdgeLength, h3IsValid and h3kRing. #8034 (Konstantin Malanchev)
  • Added support for brotli (br) compression in file-related storages and table functions. This fixes #8156. #8526 (alexey-milovidov)
  • Add groupBit* functions for the SimpleAggregationFunction type. #8485 (Guillaume Tassery)

Bug Fix

  • Fix rename of tables with Distributed engine. Fixes issue #7868. #8306 (tavplubix)
  • Now dictionaries support EXPRESSION for attributes in arbitrary string in non-ClickHouse SQL dialect. #8098 (alesapin)
  • Fix broken INSERT SELECT FROM mysql(...) query. This fixes #8070 and #7960. #8234 (tavplubix)
  • Fix error "Mismatch column sizes" when inserting default Tuple from JSONEachRow. This fixes #5653. #8606 (tavplubix)
  • Now an exception will be thrown in case of using WITH TIES alongside LIMIT BY. Also add ability to use TOP with LIMIT BY. This fixes #7472. #7637 (Nikita Mikhaylov)
  • Fix unintendent dependency from fresh glibc version in clickhouse-odbc-bridge binary. #8046 (Amos Bird)
  • Fix bug in check function of *MergeTree engines family. Now it doesn't fail in case when we have equal amount of rows in last granule and last mark (non-final). #8047 (alesapin)
  • Fix insert into Enum* columns after ALTER query, when underlying numeric type is equal to table specified type. This fixes #7836. #7908 (Anton Popov)
  • Allowed non-constant negative "size" argument for function substring. It was not allowed by mistake. This fixes #4832. #7703 (alexey-milovidov)
  • Fix parsing bug when wrong number of arguments passed to (O|J)DBC table engine. #7709 (alesapin)
  • Using command name of the running clickhouse process when sending logs to syslog. In previous versions, empty string was used instead of command name. #8460 (Michael Nacharov)
  • Fix check of allowed hosts for localhost. This PR fixes the solution provided in #8241. #8342 (Vitaly Baranov)
  • Fix rare crash in argMin and argMax functions for long string arguments, when result is used in runningAccumulate function. This fixes #8325 #8341 (dinosaur)
  • Fix memory overcommit for tables with Buffer engine. #8345 (Azat Khuzhin)
  • Fixed potential bug in functions that can take NULL as one of the arguments and return non-NULL. #8196 (alexey-milovidov)
  • Better metrics calculations in thread pool for background processes for MergeTree table engines. #8194 (Vladimir Chebotarev)
  • Fix function IN inside WHERE statement when row-level table filter is present. Fixes #6687 #8357 (Ivan)
  • Now an exception is thrown if the integral value is not parsed completely for settings values. #7678 (Mikhail Korotov)
  • Fix exception when aggregate function is used in query to distributed table with more than two local shards. #8164 (小路)
  • Now bloom filter can handle zero length arrays and doesn't perform redundant calculations. #8242 (achimbab)
  • Fixed checking if a client host is allowed by matching the client host to host_regexp specified in users.xml. #8241 (Vitaly Baranov)
  • Relax ambiguous column check that leads to false positives in multiple JOIN ON section. #8385 (Artem Zuikov)
  • Fixed possible server crash (std::terminate) when the server cannot send or write data in JSON or XML format with values of String data type (that require UTF-8 validation) or when compressing result data with Brotli algorithm or in some other rare cases. This fixes #7603 #8384 (alexey-milovidov)
  • Fix race condition in StorageDistributedDirectoryMonitor found by CI. This fixes #8364. #8383 (Nikolai Kochetov)
  • Now background merges in *MergeTree table engines family preserve storage policy volume order more accurately. #8549 (Vladimir Chebotarev)
  • Now table engine Kafka works properly with Native format. This fixes #6731 #7337 #8003. #8016 (filimonov)
  • Fixed formats with headers (like CSVWithNames) which were throwing exception about EOF for table engine Kafka. #8016 (filimonov)
  • Fixed a bug with making set from subquery in right part of IN section. This fixes #5767 and #2542. #7755 (Nikita Mikhaylov)
  • Fix possible crash while reading from storage File. #7756 (Nikolai Kochetov)
  • Fixed reading of the files in Parquet format containing columns of type list. #8334 (maxulan)
  • Fix error Not found column for distributed queries with PREWHERE condition dependent on sampling key if max_parallel_replicas > 1. #7913 (Nikolai Kochetov)
  • Fix error Not found column if query used PREWHERE dependent on table's alias and the result set was empty because of primary key condition. #7911 (Nikolai Kochetov)
  • Fixed return type for functions rand and randConstant in case of Nullable argument. Now functions always return UInt32 and never Nullable(UInt32). #8204 (Nikolai Kochetov)
  • Disabled predicate push-down for WITH FILL expression. This fixes #7784. #7789 (Winter Zhang)
  • Fixed incorrect count() result for SummingMergeTree when FINAL section is used. #3280 #7786 (Nikita Mikhaylov)
  • Fix possible incorrect result for constant functions from remote servers. It happened for queries with functions like version(), uptime(), etc. which returns different constant values for different servers. This fixes #7666. #7689 (Nikolai Kochetov)
  • Fix complicated bug in push-down predicate optimization which leads to wrong results. This fixes a lot of issues on push-down predicate optimization. #8503 (Winter Zhang)
  • Fix crash in CREATE TABLE .. AS dictionary query. #8508 (Azat Khuzhin)
  • Several improvements ClickHouse grammar in .g4 file. #8294 (taiyang-li)
  • Fix bug that leads to crashes in JOINs with tables with engine Join. This fixes #7556 #8254 #7915 #8100. #8298 (Artem Zuikov)
  • Fix redundant dictionaries reload on CREATE DATABASE. #7916 (Azat Khuzhin)
  • Limit maximum number of streams for read from StorageFile and StorageHDFS. Fixes https://github.com/ClickHouse/ClickHouse/issues/7650. #7981 (alesapin)
  • Fix bug in ALTER ... MODIFY ... CODEC query, when user specify both default expression and codec. Fixes 8593. #8614 (alesapin)
  • Fix error in background merge of columns with SimpleAggregateFunction(LowCardinality) type. #8613 (Nikolai Kochetov)
  • Fixed type check in function toDateTime64. #8375 (Vasily Nemkov)
  • Now server do not crash on LEFT or FULL JOIN with and Join engine and unsupported join_use_nulls settings. #8479 (Artem Zuikov)
  • Now DROP DICTIONARY IF EXISTS db.dict query doesn't throw exception if db doesn't exist. #8185 (Vitaly Baranov)
  • Fix possible crashes in table functions (file, mysql, remote) caused by usage of reference to removed IStorage object. Fix incorrect parsing of columns specified at insertion into table function. #7762 (tavplubix)
  • Ensure network be up before starting clickhouse-server. This fixes #7507. #8570 (Zhichang Yu)
  • Fix timeouts handling for secure connections, so queries doesn't hang indefenitely. This fixes #8126. #8128 (alexey-milovidov)
  • Fix clickhouse-copier's redundant contention between concurrent workers. #7816 (Ding Xiang Fei)
  • Now mutations doesn't skip attached parts, even if their mutation version were larger than current mutation version. #7812 (Zhichang Yu) #8250 (alesapin)
  • Ignore redundant copies of *MergeTree data parts after move to another disk and server restart. #7810 (Vladimir Chebotarev)
  • Fix crash in FULL JOIN with LowCardinality in JOIN key. #8252 (Artem Zuikov)
  • Forbidden to use column name more than once in insert query like INSERT INTO tbl (x, y, x). This fixes #5465, #7681. #7685 (alesapin)
  • Added fallback for detection the number of physical CPU cores for unknown CPUs (using the number of logical CPU cores). This fixes #5239. #7726 (alexey-milovidov)
  • Fix There's no column error for materialized and alias columns. #8210 (Artem Zuikov)
  • Fixed sever crash when EXISTS query was used without TABLE or DICTIONARY qualifier. Just like EXISTS t. This fixes #8172. This bug was introduced in version 19.17. #8213 (alexey-milovidov)
  • Fix rare bug with error "Sizes of columns doesn't match" that might appear when using SimpleAggregateFunction column. #7790 (Boris Granveaud)
  • Fix bug where user with empty allow_databases got access to all databases (and same for allow_dictionaries). #7793 (DeifyTheGod)
  • Fix client crash when server already disconnected from client. #8071 (Azat Khuzhin)
  • Fix ORDER BY behaviour in case of sorting by primary key prefix and non primary key suffix. #7759 (Anton Popov)
  • Check if qualified column present in the table. This fixes #6836. #7758 (Artem Zuikov)
  • Fixed behavior with ALTER MOVE ran immediately after merge finish moves superpart of specified. Fixes #8103. #8104 (Vladimir Chebotarev)
  • Fix possible server crash while using UNION with different number of columns. Fixes #7279. #7929 (Nikolai Kochetov)
  • Fix size of result substring for function substr with negative size. #8589 (Nikolai Kochetov)
  • Now server does not execute part mutation in MergeTree if there are not enough free threads in background pool. #8588 (tavplubix)
  • Fix a minor typo on formatting UNION ALL AST. #7999 (litao91)
  • Fixed incorrect bloom filter results for negative numbers. This fixes #8317. #8566 (Winter Zhang)
  • Fixed potential buffer overflow in decompress. Malicious user can pass fabricated compressed data that will cause read after buffer. This issue was found by Eldar Zaitov from Yandex information security team. #8404 (alexey-milovidov)
  • Fix incorrect result because of integers overflow in arrayIntersect. #7777 (Nikolai Kochetov)
  • Now OPTIMIZE TABLE query will not wait for offline replicas to perform the operation. #8314 (javi santana)
  • Fixed ALTER TTL parser for Replicated*MergeTree tables. #8318 (Vladimir Chebotarev)
  • Fix communication between server and client, so server read temporary tables info after query failure. #8084 (Azat Khuzhin)
  • Fix bitmapAnd function error when intersecting an aggregated bitmap and a scalar bitmap. #8082 (Yue Huang)
  • Refine the definition of ZXid according to the ZooKeeper Programmer's Guide which fixes bug in clickhouse-cluster-copier. #8088 (Ding Xiang Fei)
  • odbc table function now respects external_table_functions_use_nulls setting. #7506 (Vasily Nemkov)
  • Fixed bug that lead to a rare data race. #8143 (Alexander Kazakov)
  • Now SYSTEM RELOAD DICTIONARY reloads a dictionary completely, ignoring update_field. This fixes #7440. #8037 (Vitaly Baranov)
  • Add ability to check if dictionary exists in create query. #8032 (alesapin)
  • Fix Float* parsing in Values format. This fixes #7817. #7870 (tavplubix)
  • Fix crash when we cannot reserve space in some background operations of *MergeTree table engines family. #7873 (Vladimir Chebotarev)
  • Fix crash of merge operation when table contains SimpleAggregateFunction(LowCardinality) column. This fixes #8515. #8522 (Azat Khuzhin)
  • Restore support of all ICU locales and add the ability to apply collations for constant expressions. Also add language name to system.collations table. #8051 (alesapin)
  • Fix bug when external dictionaries with zero minimal lifetime (LIFETIME(MIN 0 MAX N), LIFETIME(N)) don't update in background. #7983 (alesapin)
  • Fix crash when external dictionary with ClickHouse source has subquery in query. #8351 (Nikolai Kochetov)
  • Fix incorrect parsing of file extension in table with engine URL. This fixes #8157. #8419 (Andrey Bodrov)
  • Fix CHECK TABLE query for *MergeTree tables without key. Fixes #7543. #7979 (alesapin)
  • Fixed conversion of Float64 to MySQL type. #8079 (Yuriy Baranov)
  • Now if table was not completely dropped because of server crash, server will try to restore and load it. #8176 (tavplubix)
  • Fixed crash in table function file while inserting into file that doesn't exist. Now in this case file would be created and then insert would be processed. #8177 (Olga Khvostikova)
  • Fix rare deadlock which can happen when trace_log is in enabled. #7838 (filimonov)
  • Add ability to work with different types besides Date in RangeHashed external dictionary created from DDL query. Fixes 7899. #8275 (alesapin)
  • Fixes crash when now64() is called with result of another function. #8270 (Vasily Nemkov)
  • Fixed bug with detecting client IP for connections through mysql wire protocol. #7743 (Dmitry Muzyka)
  • Fix empty array handling in arraySplit function. This fixes #7708. #7747 (hcz)
  • Fixed the issue when pid-file of another running clickhouse-server may be deleted. #8487 (Weiqing Xu)
  • Fix dictionary reload if it has invalidate_query, which stopped updates and some exception on previous update tries. #8029 (alesapin)
  • Fixed error in function arrayReduce that may lead to "double free" and error in aggregate function combinator Resample that may lead to memory leak. Added aggregate function aggThrow. This function can be used for testing purposes. #8446 (alexey-milovidov)

Improvement

  • Improved logging when working with S3 table engine. #8251 (Grigory Pervakov)
  • Printed help message when no arguments are passed when calling clickhouse-local. This fixes #5335. #8230 (Andrey Nagorny)
  • Add setting mutations_sync which allows to wait ALTER UPDATE/DELETE queries synchronously. #8237 (alesapin)
  • Allow to set up relative user_files_path in config.xml (in the way similar to format_schema_path). #7632 (hcz)
  • Add exception for illegal types for conversion functions with -OrZero postfix. #7880 (Andrey Konyaev)
  • Simplify format of the header of data sending to a shard in a distributed query. #8044 (Vitaly Baranov)
  • Live View table engine refactoring. #8519 (vzakaznikov)
  • Add additional checks for external dictionaries created from DDL-queries. #8127 (alesapin)
  • Fix error Column ... already exists while using FINAL and SAMPLE together, e.g. select count() from table final sample 1/2. Fixes #5186. #7907 (Nikolai Kochetov)
  • Now table the first argument of joinGet function can be table indentifier. #7707 (Amos Bird)
  • Allow using MaterializedView with subqueries above Kafka tables. #8197 (filimonov)
  • Now background moves between disks run it the seprate thread pool. #7670 (Vladimir Chebotarev)
  • SYSTEM RELOAD DICTIONARY now executes synchronously. #8240 (Vitaly Baranov)
  • Stack traces now display physical addresses (offsets in object file) instead of virtual memory addresses (where the object file was loaded). That allows the use of addr2line when binary is position independent and ASLR is active. This fixes #8360. #8387 (alexey-milovidov)
  • Support new syntax for row-level security filters: <table name='table_name'>…</table>. Fixes #5779. #8381 (Ivan)
  • Now cityHash function can work with Decimal and UUID types. Fixes #5184. #7693 (Mikhail Korotov)
  • Removed fixed index granularity (it was 1024) from system logs because it's obsolete after implementation of adaptive granularity. #7698 (alexey-milovidov)
  • Enabled MySQL compatibility server when ClickHouse is compiled without SSL. #7852 (Yuriy Baranov)
  • Now server checksums distributed batches, which gives more verbose errors in case of corrupted data in batch. #7914 (Azat Khuzhin)
  • Support DROP DATABASE, DETACH TABLE, DROP TABLE and ATTACH TABLE for MySQL database engine. #8202 (Winter Zhang)
  • Add authentication in S3 table function and table engine. #7623 (Vladimir Chebotarev)
  • Added check for extra parts of MergeTree at different disks, in order to not allow to miss data parts at undefined disks. #8118 (Vladimir Chebotarev)
  • Enable SSL support for Mac client and server. #8297 (Ivan)
  • Now ClickHouse can work as MySQL federated server (see https://dev.mysql.com/doc/refman/5.7/en/federated-create-server.html). #7717 (Maxim Fedotov)
  • clickhouse-client now only enable bracketed-paste when multiquery is on and multiline is off. This fixes (#7757)[https://github.com/ClickHouse/ClickHouse/issues/7757]. #7761 (Amos Bird)
  • Support Array(Decimal) in if function. #7721 (Artem Zuikov)
  • Support Decimals in arrayDifference, arrayCumSum and arrayCumSumNegative functions. #7724 (Artem Zuikov)
  • Added lifetime column to system.dictionaries table. #6820 #7727 (kekekekule)
  • Improved check for existing parts on different disks for *MergeTree table engines. Addresses #7660. #8440 (Vladimir Chebotarev)
  • Integration with AWS SDK for S3 interactions which allows to use all S3 features out of the box. #8011 (Pavel Kovalenko)
  • Added support for subqueries in Live View tables. #7792 (vzakaznikov)
  • Check for using Date or DateTime column from TTL expressions was removed. #7920 (Vladimir Chebotarev)
  • Information about disk was added to system.detached_parts table. #7833 (Vladimir Chebotarev)
  • Now settings max_(table|partition)_size_to_drop can be changed without a restart. #7779 (Grigory Pervakov)
  • Slightly better usability of error messages. Ask user not to remove the lines below Stack trace:. #7897 (alexey-milovidov)
  • Better reading messages from Kafka engine in various formats after #7935. #8035 (Ivan)
  • Better compatibility with MySQL clients which don't support sha2_password auth plugin. #8036 (Yuriy Baranov)
  • Support more column types in MySQL compatibility server. #7975 (Yuriy Baranov)
  • Implement ORDER BY optimization for Merge, Buffer and Materilized View storages with underlying MergeTree tables. #8130 (Anton Popov)
  • Now we always use POSIX implementation of getrandom to have better compatibility with old kernels (< 3.17). #7940 (Amos Bird)
  • Better check for valid destination in a move TTL rule. #8410 (Vladimir Chebotarev)
  • Better checks for broken insert batches for Distributed table engine. #7933 (Azat Khuzhin)
  • Add column with array of parts name which mutations must process in future to system.mutations table. #8179 (alesapin)
  • Parallel merge sort optimization for processors. #8552 (Nikolai Kochetov)
  • The settings mark_cache_min_lifetime is now obsolete and does nothing. In previous versions, mark cache can grow in memory larger than mark_cache_size to accomodate data within mark_cache_min_lifetime seconds. That was leading to confusion and higher memory usage than expected, that is especially bad on memory constrained systems. If you will see performance degradation after installing this release, you should increase the mark_cache_size. #8484 (alexey-milovidov)
  • Preparation to use tid everywhere. This is needed for #7477. #8276 (alexey-milovidov)

Performance Improvement

  • Performance optimizations in processors pipeline. #7988 (Nikolai Kochetov)
  • Non-blocking updates of expired keys in cache dictionaries (with permission to read old ones). #8303 (Nikita Mikhaylov)
  • Compile ClickHouse without -fno-omit-frame-pointer globally to spare one more register. #8097 (Amos Bird)
  • Speedup greatCircleDistance function and add performance tests for it. #7307 (Olga Khvostikova)
  • Improved performance of function roundDown. #8465 (alexey-milovidov)
  • Improved performance of max, min, argMin, argMax for DateTime64 data type. #8199 (Vasily Nemkov)
  • Improved performance of sorting without a limit or with big limit and external sorting. #8545 (alexey-milovidov)
  • Improved performance of formatting floating point numbers up to 6 times. #8542 (alexey-milovidov)
  • Improved performance of modulo function. #7750 (Amos Bird)
  • Optimized ORDER BY and merging with single column key. #8335 (alexey-milovidov)
  • Better implementation for arrayReduce, -Array and -State combinators. #7710 (Amos Bird)
  • Now PREWHERE should be optimized to be at least as efficient as WHERE. #7769 (Amos Bird)
  • Improve the way round and roundBankers handling negative numbers. #8229 (hcz)
  • Improved decoding performance of DoubleDelta and Gorilla codecs by roughly 30-40%. This fixes #7082. #8019 (Vasily Nemkov)
  • Improved performance of base64 related functions. #8444 (alexey-milovidov)
  • Added a function geoDistance. It is similar to greatCircleDistance but uses approximation to WGS-84 ellipsoid model. The performance of both functions are near the same. #8086 (alexey-milovidov)
  • Faster min and max aggregation functions for Decimal data type. #8144 (Artem Zuikov)
  • Vectorize processing arrayReduce. #7608 (Amos Bird)
  • if chains are now optimized as multiIf. #8355 (kamalov-ruslan)
  • Fix performance regression of Kafka table engine introduced in 19.15. This fixes #7261. #7935 (filimonov)
  • Removed "pie" code generation that gcc from Debian packages occasionally brings by default. #8483 (alexey-milovidov)
  • Parallel parsing data formats #6553 (Nikita Mikhaylov)
  • Enable optimized parser of Values with expressions by default (input_format_values_deduce_templates_of_expressions=1). #8231 (tavplubix)

Build/Testing/Packaging Improvement

Experimental Feature

  • Added experimental setting min_bytes_to_use_mmap_io. It allows to read big files without copying data from kernel to userspace. The setting is disabled by default. Recommended threshold is about 64 MB, because mmap/munmap is slow. #8520 (alexey-milovidov)
  • Reworked quotas as a part of access control system. Added new table system.quotas, new functions currentQuota, currentQuotaKey, new SQL syntax CREATE QUOTA, ALTER QUOTA, DROP QUOTA, SHOW QUOTA. #7257 (Vitaly Baranov)
  • Allow skipping unknown settings with warnings instead of throwing exceptions. #7653 (Vitaly Baranov)
  • Reworked row policies as a part of access control system. Added new table system.row_policies, new function currentRowPolicies(), new SQL syntax CREATE POLICY, ALTER POLICY, DROP POLICY, SHOW CREATE POLICY, SHOW POLICIES. #7808 (Vitaly Baranov)

Security Fix

  • Fixed the possibility of reading directories structure in tables with File table engine. This fixes #8536. #8537 (alexey-milovidov)

Changelog for 2019