mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-05 14:02:21 +00:00
39 KiB
39 KiB
ClickHouse release v20.6.1.4066-prestable FIXME as compared to v20.5.1.3833-prestable
Backward Incompatible Change
clickhouse-local
now uses an unique temporary data directory by default, not the current directory as before. If needed, the data directory can be explicitly specified with the-- --path
option. #11931 (Alexander Kuzmenkov).
New Feature
- Add
Alter table drop replica replica_name
support. This fixes #7080. #10679 (sundyli). - Added new in-memory format of parts in
MergeTree
-family tables, which stores data in memory. Parts are written on disk at first merge. Part will be created in in-memory format if its size in rows or bytes is below thresholdsmin_rows_for_compact_part
andmin_bytes_for_compact_part
. Also optional support of Write-Ahead-Log is available, which is enabled by default and is controlled by settingin_memory_parts_enable_wal
. #10697 (Anton Popov). - Add -Distinct combinator for aggregate functions. #10930 (Sofia Antipushina).
- Support table engine mongo(host:port, database, collection, user, password). #10931 (ageraab).
- Add storage RabbitMQ. #11069 (Kseniia Sumarokova).
- Opt-in settings to send crash reports to the ClickHouse core team via Sentry. #11300 (Ivan Blinkov).
- Add ORCBlockOutputFormat. #11662 (Kruglov Pavel).
max_thread_pool_size
config for changing the maximum number of Threads in Global Thread Pool. #11668 (Bharat Nallan).- Initial implementation of
EXPLAIN
query. Syntax:EXPLAIN SELECT ...
. This fixes #1118. #11873 (Nikolai Kochetov). - Switched paths in S3 metadata to relative which allows to handle S3 blobs more easily. #11892 (Vladimir Chebotarev).
- Avro UUID input/output support. #11954 (Andrew Onyshchuk).
- Added read-only MongoDB table engine. Allows reading flat (primitive types, not nested) fields. #11983 (alesapin).
- Add setting to fields not found in Avro schema. #12007 (Andrew Onyshchuk).
- add function
parseDateTimeBestEffortUS
. #12028 (flynn). - #4006 Support ALTER TABLE ... [ADD|MODIFY] COLUMN ... FIRST. #12073 (Winter Zhang).
- Add a function initializedAggregation to initialize an aggregation based on a single value. #12109 (Guillaume Tassery).
- Support RIGHT and FULL JOIN with
set join_algorithm=partial_merge
. Only ALL strictness is supported (ANY, SEMI, ANTI, ASOF are not). #12118 (Artem Zuikov). - Implementation of PostgreSQL-like ILIKE operator for #11710. #12125 (Mike Kot).
- Allow Nullable types as keys in MergeTree tables. #5319. #12433 (Amos Bird).
Performance Improvement
- Allow to use direct_io and mmap_io for secondary indices if the settings
min_bytes_to_use_direct_io
ormin_bytes_to_use_mmap_io
are configured. #11955 (Alexey Milovidov). - Fix estimation of the number of marks while reading from MergeTree. This is needed to correctly handle the settings
merge_tree_max_rows_to_use_cache
,merge_tree_max_bytes_to_use_cache
,merge_tree_min_rows_for_concurrent_read
,merge_tree_min_bytes_for_concurrent_read
,merge_tree_min_rows_for_seek
,merge_tree_min_bytes_for_seek
. Now settingsmin_bytes_to_use_mmap_io
also applied to read index and compact parts in MergeTree table engines family. #11970 (Alexey Milovidov). - Fix "#10574 Index not used for IN operator with literals", performance regression introduced around v19.3. #12062 (nvartolomei).
- Remove injective functions inside
uniq*()
ifset optimize_injective_functions_inside_uniq=1
. #12337 (Artem Zuikov). - Add order by optimisation that rewrites
ORDER BY x, f(x)
withORDER by x
ifset optimize_redundant_functions_in_order_by = 1
. #12404 (Artem Zuikov). - Lower memory usage for some operations up to 2 times. #12424 (Alexey Milovidov).
Improvement
- Add number of errors to ignore while choosing replicas (
distributed_replica_error_ignore
). #11669 (Azat Khuzhin). - Improved performace of 'ORDER BY' and 'GROUP BY' by prefix of sorting key. #11696 (Anton Popov).
-
- Add
optimize_skip_unused_shards_nesting
(allows control nesting level for shards skipping optimization) - Addforce_skip_optimize_shards_nesting
(allows control nesting level for checking was shards skipped or not) - Deprecateforce_optimize_skip_unused_shards_no_nested
(force_skip_optimize_shards_nesting
should be used instead) - Disableoptimize_skip_unused_shards
if sharding_key has non-deterministic func (i.e.rand()
, note that this does not changes anything for INSERT side). #11715 (Azat Khuzhin).
- Add
- Multiversion metadata for storages without structure locks. #11745 (alesapin).
- Slightly relax the validation of ODBC connection string. If the hostname or username contains only word characters along with
.
and-
, don't put it into curly braces. It is needed, because some ODBC drivers (e.g. PostgreSQL) don't understand when hostname is enclosed in curly braces. #11845 (Alexey Milovidov). - Support parse UUID without separator(separators are always removed in most implementations, this is helpful for users to write data). #11856 (Winter Zhang).
- Support
SIGNED
andUNSIGNED
modifiers of standard integer types (BIGINT
,INT
, ...) for compatibility with MySQL. #11858 (Alexander Tokmakov). - Allow to use
sumWithOverflow
asSimpleAggregateFunction
. Closes #8053. #11865 (Alexander Kuzmenkov). - Add FixedString support in Hashing functions. #11878 (flynn).
- Rewrite code for
optimize_arithmetic_operations_in_aggregate_functions
optimisation. #11899 (Artem Zuikov). - Improve path concatenation and fix double slashed paths using std::filesystem::path instead of std::string in
DatabaseOrdinary.cpp
. #11900 (Bharat Nallan). - Deprecate the old regular style and use the new globalVariable method #11832. #11901 (BohuTANG).
- related to issue 9797. #11923 (flynn).
system.tables
now considers column capacities for Memory and Buffer table engines, which is better approximation for resident memory size. #11935 (Max Akhmedov).- Add CPU frequencies to system.asynchronous_metrics. Make the metric collection period configurable. #11972 (Alexander Kuzmenkov).
- Allow to perform "metadata-only" ALTER of partition key such as adding more elements to Enum data type. This fixes #7513. #11973 (Alexey Milovidov).
- Add replica priority for load_balancing (for manual prioritization of the load balancing). #11995 (Azat Khuzhin).
- Support MySQL engine reading Enums type #3985. #11996 (BohuTANG).
- Implemented single part uploads for DiskS3. #12026 (Vladimir Chebotarev).
- Moved useless S3 logging to TRACE level. #12067 (Vladimir Chebotarev).
- Improves
REVOKE
command: now it requires grant/admin option for only access which will be revoked. For example, to executeREVOKE ALL ON *.* FROM user1
now it doesn't require to have full access rights granted with grant option. Added commandREVOKE ALL FROM user1
- it revokes all granted roles fromuser1
. #12083 (Vitaly Baranov). - Add 'type' column in system.disks. #12115 (ianton-ru).
- Added support for
%g
(two digit ISO year) and%G
(four digit ISO year) substitutions informatDateTime
function. #12136 (vivarum). - Add
KILL QUERY [connection_id]
for the MySQL client/driver to cancel the long query, issue #12038. #12152 (BohuTANG). - This setting allows to chose charset for printing grids (either utf8 or ascii). #12372 (Sabyanin Maxim).
- Write the detail exception message to the client instead of 'MySQL server has gone away'. #12383 (BohuTANG).
- lifetime_rows/lifetime_bytes for Buffer engine. #12421 (Azat Khuzhin).
- Use correct default secure port for clickhouse-benchmark with
--secure
argument. This fixes #11044. #12440 (Alexey Milovidov).
Bug Fix
- Fix unexpected behaviour of queries like
SELECT *, xyz.*
which were success while an error expected. #11753 (hexiaoting). - Fix wrong result for
if()
with NULLs in condition. #11807 (Artem Zuikov). - Fix memory accounting via HTTP interface (can be significant with
wait_end_of_query=1
). #11840 (Azat Khuzhin). - Fix rare crash caused by using
Nullable
column in prewhere condition. Continuation of #11608. #11869 (Nikolai Kochetov). - Fix potential floating point exception when parsing DateTime64. This fixes #11374. #11875 (Alexey Milovidov).
- Fixed bug with no moves when changing storage policy from default one. #11893 (Vladimir Chebotarev).
- Fix rare crash caused by using Nullable column in prewhere condition. Continuation of #11869. #11895 (Nikolai Kochetov).
- Keep aliases for substitutions in query (parametrized queries). This fixes #11914. #11916 (Alexey Milovidov).
- Fix unitialized memory in partitions serialization. #11919 (alesapin).
- Use the correct current database for checking access rights after statement
USE database
. #11920 (Vitaly Baranov). - Fixed
Query parameter was not set
inValues
format. Fixes #11918. #11936 (Alexander Tokmakov). - Fix race condition in extractAllGroups* functions. #11949 (Alexey Milovidov).
- Make mmap IO work again (experimental). Continuation of #8520. #11953 (Alexey Milovidov).
- Fix wrong setting name in log message at server startup. #11997 (Alexey Milovidov).
- Partial revokes work correctly in complex cases as well, for example. #12002 (Vitaly Baranov).
- Fix potential floating point exception. This closes #11378. #12005 (Alexey Milovidov).
- Avoid returning wrong number of geohashes in function
geoHashesInBox
due to accumulation of floating point error. This fixes #11369. #12006 (Alexey Milovidov). - Fix potential array size overflow in generateRandom that may lead to crash. This fixes #11371. #12013 (Alexey Milovidov).
- Fix calculation of access rights when allow_ddl=0. #12015 (Vitaly Baranov).
- When adding floating point number of intervals to date/datetime, the result may be calculated incorrectly. This fixes #11377. #12018 (Alexey Milovidov).
- A query with function
neighbor
as the only returned expression may return empty result if the function is called with offset-9223372036854775808
. This fixes #11367. #12019 (Alexey Milovidov). - Do not mention in changelog, because the bug did not come to release. Fix potential crash when doing ORDER BY multiple columns with specified COLLATE on one of the column when this column is constant. This fixes #11379. The bug was introduced in #11006 in version 20.5. #12020 (Alexey Milovidov).
- Fix wrong result and potential crash when invoking function
if
with arguments of typeFixedString
with different sizes. This fixes #11362. #12021 (Alexey Milovidov). - Fix calculation of access rights when allow_introspection_functions=0. #12031 (Vitaly Baranov).
- Fix crash in JOIN with LowCardinality type with
join_algorithm=partial_merge
. #12035 (Artem Zuikov). - Fix incorrect comparison of tuples with
Nullable
columns. Fixes #11985. #12039 (Nikolai Kochetov). - Fix constraints check if constraint is a constant expression. This fixes #11360. #12042 (Alexey Milovidov).
- Make
topK
aggregate function return Enum for Enum types. This fixes #3740. #12043 (Alexey Milovidov). - Parse tables metadata in parallel when loading database. This fixes slow server startup when there are large number of tables. #12045 (Alexander Tokmakov).
- Fix error
Cannot capture column
for higher-order functions withTuple(LowCardinality)
argument. Fixes #9766. #12055 (Nikolai Kochetov). - Fix error
Expected single dictionary argument for function
for functiondefaultValueOfArgumentType
withLowCardinality
type. Fixes #11808. #12056 (Nikolai Kochetov). - Fix possible crash while using wrong type for
PREWHERE
. Fixes #12053, #12060. #12060 (Nikolai Kochetov). - Fix SIGSEGV in StorageKafka on DROP TABLE. #12075 (Azat Khuzhin).
- Fix unnecessary limiting the number of threads for selects from
VIEW
. Fixes #11937. #12085 (Nikolai Kochetov). - Fix empty
result_rows
andresult_bytes
metrics insystem.quey_log
for selects. Fixes #11595. #12089 (Nikolai Kochetov). - Fix segfault with
-StateResample
combinators. #12092 (Anton Popov). - Fix performance for selects with
UNION
caused by wrong limit for the total number of threads. Fixes #12030. #12103 (Nikolai Kochetov). - Format
Parquet
now properly works withLowCardinality
andLowCardinality(Nullable)
types. Fixes #12086, #8406. #12108 (Nikolai Kochetov). - Fix handling dependency of table with ENGINE=Dictionary on dictionary. This fixes #10994. This fixes #10397. #12116 (Vitaly Baranov).
- Avoid "There is no query" exception for materialized views with joins or with subqueries attached to system logs (system.query_log, metric_log, etc) or to engine=Buffer underlying table. #12120 (filimonov).
- Fix bug which leads to incorrect table metadata in ZooKeepeer for ReplicatedVersionedCollapsingMergeTree tables. Fixes #12093. #12121 (alesapin).
- Normalize "pid" file handling. In previous versions the server may refuse to start if it was killed without proper shutdown and if there is another process that has the same pid as previously runned server. Also pid file may be removed in unsuccessful server startup even if there is another server running. This fixes #3501. #12133 (Alexey Milovidov).
- Fix potential infinite loop in
greatCircleDistance
,geoDistance
. This fixes #12117. #12137 (Alexey Milovidov). - Fix potential overflow in integer division. This fixes #12119. #12140 (Alexey Milovidov).
- Fix bad code in redundant ORDER BY optimization. The bug was introduced in #10067. #12148 (Alexey Milovidov).
- Fix transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. #12151 (Alexey Milovidov).
- Fix wrong logic in ALTER DELETE that leads to deleting of records when condition evaluates to NULL. This fixes #9088. This closes #12106. #12153 (Alexey Milovidov).
- Don't split the dictionary source's table name into schema and table name itself if ODBC connection doesn't support schema. #12165 (Vitaly Baranov).
- Fixed the behaviour when
SummingMergeTree
engine sums up columns from partition key. Added an exception in case of explicit definition of columns to sum which intersects with partition key columns. This fixes #7867. #12173 (Nikita Mikhaylov). - Fix dictGet arguments check during GROUP BY injective functions elimination. #12179 (Azat Khuzhin).
- Cap max_memory_usage* limits to the process resident memory. #12182 (Azat Khuzhin).
- Fixed logical functions for UInt8 values when they are not equal to 0 or 1. #12196 (Alexander Kazakov).
- Fixed the behaviour when during multiple sequential inserts in
StorageFile
header for some special types was written more than once. This fixed #6155. #12197 (Nikita Mikhaylov). - Fixed behaviour on reaching redirect limit in request to S3 storage. #12256 (ianton-ru).
- Not for changelog. Cherry-pick after #12196. #12271 (Alexey Milovidov).
- Implement conversions to the common type for LowCardinality types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes #8212. This fixes #4342. #12275 (Alexey Milovidov).
- The function
arrayFill
worked incorrectly for empty arrays that may lead to crash. This fixes #12263. #12279 (Alexey Milovidov). - Show error after TrieDictionary failed to load. #12290 (Vitaly Baranov).
- Fix typo in setting name. #12292 (Alexey Milovidov).
- Some threads might randomly hang for a few seconds during DNS cache updating. It's fixed. #12296 (Alexander Tokmakov).
- Fix TTL after renaming column, on which depends TTL expression. #12304 (Anton Popov).
- Avoid "bad cast" exception when there is an expression that filters data by virtual columns (like
_table
inMerge
tables) or by "index" columns in system tables such as filtering by database name when querying fromsystem.tables
, and this expression returnsNullable
type. This fixes #12166. #12305 (Alexey Milovidov). - Fix order of columns in
WITH FILL
modifier. Previously order of columns ofORDER BY
statement wasn't respected. #12306 (Anton Popov). - When using codec
Delta
orDoubleDelta
with non fixed width types, exception with codeLOGICAL_ERROR
was returned instead of exception with codeBAD_ARGUMENTS
(we ensure that exceptions with code logical error never happen). This fixes #12110. #12308 (Alexey Milovidov). - Fix very rare race condition in ReplicatedMergeTreeQueue. #12315 (Alexey Milovidov).
- Fix error message and exit codes for
ALTER RENAME COLUMN
queries, whenRENAME
is not allowed. Fixes #12301 and #12303. #12335 (alesapin). - Fix TOTALS/ROLLUP/CUBE for aggregate functions with
-State
andNullable
arguments. This fixes #12163. #12376 (Alexey Milovidov). - Allow to CLEAR column even if there are depending DEFAULT expressions. This fixes #12333. #12378 (Alexey Milovidov).
- Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables. This fixes #11905. #12384 (Alexey Milovidov).
- Additional check for arguments of bloom filter index. This fixes #11408. #12388 (Alexey Milovidov).
- Reverts change introduced in #11079 to resolve #12098. #12397 (Mike Kot).
- Fixed possible segfault if StorageMerge. Closes #12054. #12401 (Alexander Tokmakov).
- Fix overflow when very large LIMIT or OFFSET is specified. This fixes #10470. This fixes #11372. #12427 (Alexey Milovidov).
- Fix UBSan report in base64 if tests were run on server with AVX-512. This fixes #12318. Author: @qoega. #12441 (Alexey Milovidov).
Build/Testing/Packaging Improvement
- Add simple GitHub hook script for the serverless environment. #11605 (alesapin).
- Send logs to client on fatal errors if possible. This will make test results more readable. #11826 (Alexey Milovidov).
- Allow ClickHouse to run on Android. #11894 (Alexey Milovidov).
- Clean up unused header files from
Obfuscator.cpp
andDatabaseAtomic.cpp
. #11922 (Bharat Nallan). - Fix some typos in code. #12003 (Alexey Milovidov).
- Scripts for automated scheduled backporting based on PR labels. #12029 (Ivan).
- Add new type of tests based on Testflows framework. #12090 (vzakaznikov).
- Install
ca-certificates
before the firstapt-get update
in Dockerfile. #12095 (Ivan Blinkov). - Daily check by GitHub CodeQL security analysis tool that looks for CWE. #12101 (Ivan Blinkov).
- Regular check by Anchore Container Analysis security analysis tool that looks for CVE in
clickhouse-server
Docker image. Also confirms thatDockerfile
is buildable. Runs daily onmaster
and on pull-requests toDockerfile
. #12102 (Ivan Blinkov). - Add
UNBUNDLED
flag tosystem.build_options
table. Move skip lists forclickhouse-test
to clickhouse repo. #12107 (alesapin). - Implement AST-based query fuzzing mode for clickhouse-client. See this label for the list of issues we recently found by fuzzing. Most of them were found by this tool, and a couple by SQLancer and
00746_sql_fuzzy.pl
. #12111 (Alexander Kuzmenkov). - Perform an upgrade of system packages in the
clickhouse-server
docker image. #12124 (Ivan Blinkov). - Added a showcase of the minimal Docker image without using any Linux distribution. #12126 (Alexey Milovidov).
- Minor CMake fixes for UNBUNDLED build. #12131 (Matwey V. Kornilov).
- Missed
<atomic>
is required forstd::atomic<>
. #12134 (Matwey V. Kornilov). - Fix warnings from CodeQL.
CodeQL
is another static analyzer that we will use along withclang-tidy
andPVS-Studio
that we use already. #12138 (Alexey Milovidov). - Record additional detail on Dockerfile scan reports. #12159 (Ivan Blinkov).
- Place common docker compose files to integration docker container. #12168 (Ilya Yatsishin).
- Remove verbosity from the binary builds. #12174 (alesapin).
- Remove strange file creation during build in
orc
. #12258 (Nikita Mikhaylov). - Now functional and stress tests will be able to run with old version of
clickhouse-test
script. #12287 (alesapin). - Log sanitizer trap messages from separate thread. This will prevent possible deadlock under thread sanitizer. #12313 (Alexey Milovidov).
- Added S3 HTTPS integration test. #12412 (Pavel Kovalenko).
- Fix UBSan report in HDFS library. This closes #12330. #12453 (Alexey Milovidov).
Other
- Update word break characters to match readline default - all non-alphanumeric characters. ... #11975 (Andrew Onyshchuk).
NO CL CATEGORY
-
- Not for changelog. #12265 (Alexander Kuzmenkov).
- ... #12431 (Tom Bombadil).
NO CL ENTRY
- NO CL ENTRY: 'Bump certifi from 2020.4.5.2 to 2020.6.20 in /docs/tools/translate'. #11853 (dependabot-preview[bot]).
- NO CL ENTRY: 'Merging #10679'. #11896 (Alexander Tokmakov).
- NO CL ENTRY: 'Revert "[experiment] maybe fix warnings in integration tests"'. #12011 (alesapin).
- NO CL ENTRY: 'Bump idna from 2.9 to 2.10 in /docs/tools'. #12024 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools'. #12049 (dependabot-preview[bot]).