mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-05 14:02:21 +00:00
35 KiB
35 KiB
ClickHouse release v20.7.1.4310-prestable FIXME as compared to v20.6.1.4066-prestable
Backward Incompatible Change
- Deprecate special printing of zero Date/DateTime values as
0000-00-00
and0000-00-00 00:00:00
. #12442 (Alexey Milovidov). - Function
modulo
(operator%
) with at least one floating point number as argument will calculate remainder of division directly on floating point numbers without converting both arguments to integers. It makes behaviour compatible with most of DBMS. This also applicable for Date and DateTime data types. Added aliasmod
. This closes #7323. #12585 (Alexey Milovidov).
New Feature
-
- Added support of LDAP authentication for preconfigured users ("Simple Bind" method). #11234 (Denis Glazachev).
- Add mapAdd and mapSubtract functions for working with tuple maps. #11735 (Ildus Kurbangaliev).
- Added
system.crash_log
table into which stack traces for fatal errors are collected. #12316 (Alexey Milovidov). - Add
bayesAB
function for bayesian-ab-testing. #12327 (achimbab). - Integration with COS. #12386 (fastio).
- A function
formatRow
is added to support turning arbitrary expressions into a string via given format. It's useful for manipulating SQL outputs and is quite versatile combined with thecolumns
function. #12574 (Amos Bird). - Add setting
allow_non_metadata_alters
which restricts to executeALTER
queries which modify data on disk. Disabled be default. Closes #11547. #12635 (alesapin). - Add
minMap
andmaxMap
functions support toSimpleAggregateFunction
. #12662 (Ildus Kurbangaliev). - Added http headers
X-ClickHouse-Database
andX-ClickHouse-Format
which may be used to set default database and output format. #12981 (hcz). - Implement user-defined settings. #13013 (Vitaly Baranov).
Performance Improvement
- Optimize PK lookup for queries that match exact PK range. #12277 (Ivan Babrou).
- Attempt to implement streaming optimization in
DiskS3
. #12434 (Vladimir Chebotarev). - Replaces monotonous functions with its argument in ORDER BY if
set optimize_monotonous_functions_in_order_by=1
. #12467 (Artem Zuikov). - Converts String-type arguments of function "if" and "transform" into enum if
set optimize_if_transform_strings_to_enum = 1
. #12515 (Artem Zuikov). - Parallel PK lookup and skipping index stages on parts, as described in #11564. #12589 (Ivan Babrou).
- Push down
LIMIT
step for query plan. #13016 (Nikolai Kochetov). - Slightly improve performance of aggregation by UInt8/UInt16 keys. #13055 (Alexey Milovidov).
- Slightly improve performance of aggregation by UInt8/UInt16 keys. #13091 (Alexey Milovidov).
Improvement
- Adds a new type of polygon dictionary which uses a recursively built grid to reduce the number of polygons which need to be checked for each point. #9278 (achulkov2).
- Allow TabSeparatedRaw as an input format. #12009 (hcz).
- Separated
AWSAuthV4Signer
into different logger, removed "AWSClient: AWSClient". #12320 (Vladimir Chebotarev). -
- Implement
RENAME DATABASE
andRENAME DICTIONARY
forAtomic
database engine - Add implicit{uuid}
macro, which can be used in ZooKeeper path forReplicatedMergeTree
. It works withCREATE ... ON CLUSTER ...
queries. Setshow_table_uuid_in_table_create_query_if_not_nil
totrue
to use it. - MakeReplicatedMergeTree
engine arguments optional,/clickhouse/tables/{uuid}/{shard}/
and{replica}
are used by default. Closes #12135. - Minor fixes. - These changes break backward compatibility ofAtomic
database engine. Previously createdAtomic
databases must be manually converted to new format. #12343 (Alexander Tokmakov).
- Implement
- Now joinGet supports multi-key lookup. #12418 (Amos Bird).
- Rollback insertion errors in
Log
,TinyLog
,StripeLog
engines. In previous versions insertion error lead to inconsisent table state (this works as documented and it is normal for these table engines). This fixes #12402. #12426 (Alexey Milovidov). - Avoid overflow in parsing of DateTime values that will lead to negative unix timestamp in their timezone (for example,
1970-01-01 00:00:00
in Moscow). Saturate to zero instead. This fixes #3470. This fixes #4172. #12443 (Alexey Milovidov). - Add several metrics for requests to S3 storages. #12464 (ianton-ru).
- Changes default value for
multiple_joins_rewriter_version
to 2. It enables new multiple joins rewriter that knows about column names. #12469 (Artem Zuikov). - Allow to set JOIN kind and type in more standad way:
LEFT SEMI JOIN
instead ofSEMI LEFT JOIN
. For now both are correct. #12520 (Artem Zuikov). - Fix explain query format overwrite by default, issue #12432. #12541 (BohuTANG).
- Add SelectedRows and SelectedBytes events. #12638 (ianton-ru).
- Add
current_database
information tosystem.query_log
. #12652 (Amos Bird). - Support truncate table without table keyword. #12653 (Winter Zhang).
- Now exceptions forwarded to the client if an error happened during ALTER or mutation. Closes #11329. #12666 (alesapin).
- Protect from the cases when user may set
background_pool_size
to value lower thannumber_of_free_entries_in_pool_to_execute_mutation
ornumber_of_free_entries_in_pool_to_lower_max_size_of_merge
. In these cases ALTERs won't work or the maximum size of merge will be too limited. Saturate values instead. This closes #10897. #12728 (Alexey Milovidov). - This change makes skipping index efficiency more obvious by showing both skipped and total examined granules. #12754 (Ivan Babrou).
- This change makes skipping index efficiency more obvious by showing total marks before and after skipping indices. #12755 (Ivan Babrou).
- Introduce setting
alter_partition_verbose_result
which outputs information about touched parts for some types ofALTER TABLE ... PARTITION ...
queries (currentlyATTACH
andFREEZE
). Closes #8076. #13017 (alesapin). - Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQueryTimeMicroseconds to system.events. #13028 (ianton-ru).
- break-ing out of a loop because it makes sense to do so. #13058 (Mark Papadakis).
- Keep less amount of logs in ZooKeeper. Avoid too large growth of ZooKeeper nodes in case of offline replicas when having many servers/tables/inserts. #13100 (Alexey Milovidov).
- Add sanity check for MergeTree settings. If the settings are incorrect, the server will refuse to start or to create a table, printing detailed explanation to the user. #13153 (Alexey Milovidov).
- Allow *Map aggregate functions to work on Arrays with NULLs. Fixes #13157. #13225 (Alexey Milovidov).
- Fix assert in parseDateTimeBestEffort. This fixes #12649. #13227 (Alexey Milovidov).
- Fix assert in geohashesInBox. This fixes #12554. #13229 (Alexey Milovidov).
- Now broken parts are also reported when encountered in compact part processing. #13282 (Amos Bird).
- Fix a 'Week'-interval formatting for ATTACH/ALTER/CREATE QUOTA-statements. #13417 (vladimir-golovchenko).
- Updated gitignore-files. #13447 (vladimir-golovchenko).
Bug Fix
- kafka: fix SIGSEGV if there is an message with error in the middle of the batch. #12302 (Azat Khuzhin).
- If MergeTree table does not contain ORDER BY or PARTITION BY, it was possible to request ALTER to CLEAR all the columns and ALTER will stuck. Fixed #7941. #12382 (Alexey Milovidov).
- SystemLog: do not write to ordinary server log under mutex. This can lead to deadlock if
text_log
is enabled. #12452 (Alexey Milovidov). - Fix crash in JOIN with dictionary when we are joining over expression of dictionary key:
t JOIN dict ON expr(dict.id) = t.id
. Disable dictionary join optimisation for this case. #12458 (Artem Zuikov). - Fix SETTINGS parse after FORMAT. #12480 (Azat Khuzhin).
- Fix backwards compatibility in binary format of
AggregateFunction(avg, ...)
values. This fixes #12342. #12486 (Alexey Milovidov). - Fixed performance issue, while reading from compact parts. #12492 (Anton Popov).
- Fixing race condition in live view tables which could cause data duplication. #12519 (vzakaznikov).
- Better exception for function
in
with invalid number of arguments. #12529 (Anton Popov). - Fix bug which lead to broken old parts after
ALTER DELETE
query whenenable_mixed_granularity_parts=1
. Fixes #12536. #12543 (alesapin). - Now ClickHouse will recalculate checksums for parts when file
checksums.txt
is absent. Broken since #9827. #12545 (alesapin). - Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 (Azat Khuzhin).
- Fix race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
- Fix lack of aliases with function
any
. #12593 (Anton Popov). - Fix error
Output of TreeExecutor is not sorted
forOPTIMIZE DEDUPLICATE
. Fixes #11572. #12613 (Nikolai Kochetov). - Fix possible
Pipeline stuck
error for queries with external sorting. Fixes #12617. #12618 (Nikolai Kochetov). - The function
groupArrayMoving*
was not working for distributed queries. It's result was calculated within incorrect data type (without promotion to the largest type). The functiongroupArrayMovingAvg
was returning integer number that was inconsistent with theavg
function. This fixes #12568. #12622 (Alexey Milovidov). - Fix error message about adaptive granularity. #12624 (alesapin).
- Better exception message in disk access storage. #12625 (alesapin).
- Exception
There is no supertype...
can be thrown duringALTER ... UPDATE
in unexpected cases (e.g. when subtracting from UInt64 column). This fixes #7306. This fixes #4165. #12633 (Alexey Milovidov). - CREATE USER IF NOT EXISTS now doesn't throw exception if the user exists. This fixes #12507. #12646 (Vitaly Baranov).
- Add support for function
if
withArray(UUID)
arguments. This fixes #11066. #12648 (Alexey Milovidov). - Fix SIGSEGV in StorageKafka when broker is unavailable (and not only). #12658 (Azat Khuzhin).
- fixes #10572 fix bloom filter index with const expression. #12659 (Winter Zhang).
- fixes #12293 allow push predicate when subquery contains with clause. #12663 (Winter Zhang).
- Fix optimization
optimize_move_functions_out_of_any=1
in case ofany(func(<lambda>))
. #12664 (Artem Zuikov). - Fix memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 (Azat Khuzhin).
- Fix performance with large tuples, which are interpreted as functions in
IN
section. The case when user writeWHERE x IN tuple(1, 2, ...)
instead ofWHERE x IN (1, 2, ...)
for some obscure reason. #12700 (Anton Popov). - Fix CAST(Nullable(String), Enum()). #12745 (Azat Khuzhin).
- Fix rare bug when
ALTER DELETE
andALTER MODIFY COLUMN
queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows incount.txt
and as a consequence incorrect data in part. Also, fix a small bug with simultaneousALTER RENAME COLUMN
andALTER ADD COLUMN
. #12760 (alesapin). - Fix unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
- Fix columns duplication for range hashed dictionary created from DDL query. This fixes #10605. #12857 (alesapin).
- Corrected merge_with_ttl_timeout logic which did not work well when expiration affected more than one partition over one time interval. (Authored by @excitoon). #12982 (Alexander Kazakov).
- Fix
Block structure mismatch
error for queries withUNION
andJOIN
. Fixes #12602. #12989 (Nikolai Kochetov). - Fix crash which was possible for queries with
ORDER BY
tuple and smallLIMIT
. Fixes #12623. #13009 (Nikolai Kochetov). - Add logging when the mutation is not running because of limited disk space or free threads in the background pool. #13068 (alesapin).
- Fix error
Cannot convert column because it is constant but values of constants are different in source and result
for remote queries which use deterministic functions in scope of query, but not deterministic between queries, likenow()
,now64()
,randConstant()
. Fixes #11327. #13075 (Nikolai Kochetov). - Fix wrong index analysis with functions. It could lead to pruning wrong parts, while reading from
MergeTree
tables. Fixes #13060. Fixes #12406. #13081 (Anton Popov). - Fix segfault when mutation killed and the server tries to send the exception to the client. #13169 (alesapin).
- AvroConfluent: Skip Kafka tombstone records AvroConfluent: Support skipping broken records ... #13203 (Andrew Onyshchuk).
- Fix DateTime64 conversion functions with constant argument. #13205 (Azat Khuzhin).
- Fix assert in
arrayElement
function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. #13224 (Alexey Milovidov). - Fix function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. #13226 (Alexey Milovidov).
- Return passed number for numbers with MSB set in roundUpToPowerOfTwoOrZero(). #13234 (Azat Khuzhin).
- Fix assertion in KeyCondition when primary key contains expression with monotonic function and query contains comparison with constant whose type is different. This fixes #12465. #13251 (Alexey Milovidov).
- Fix potentially low performance and slightly incorrect result for
uniqExact
,topK
,sumDistinct
and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes #12491. #13254 (Alexey Milovidov). - The server may crash if user passed specifically crafted arguments to the function
h3ToChildren
. This fixes #13275. #13277 (Alexey Milovidov). - Fix possible error
Totals having transform was already added to pipeline
in case of a query from delayed replica. #13290 (Nikolai Kochetov). - Fix crash in
LEFT ASOF JOIN
withjoin_use_nulls=1
. #13291 (Artem Zuikov). - Throw error on
arrayJoin()
function inJOIN ON
section. #13330 (Artem Zuikov). - Fix segfault when function
groupArrayMovingSum
deserializes empty state. Fixes #13339. #13341 (alesapin). - Fixed the deadlock in textlog. It is a part of #12339. This fixes #12325. #13386 (Nikita Mikhaylov).
- Fix PrettyCompactMonoBlock for clickhouse-local. Fix extremes/totals with PrettyCompactMonoBlock. Fixes #7746. #13394 (Azat Khuzhin).
- Fix queries with constant columns and
ORDER BY
prefix of primary key. #13396 (Anton Popov). - Fix empty output for
Arrow
andParquet
formats in case if query return zero rows. It was done because empty output is not valid for this formats. #13399 (hcz). - Fix
aggregate function any(x) is found inside another aggregate function in query
error withSET optimize_move_functions_out_of_any = 1
and aliases insideany()
. #13419 (Artem Zuikov). - Fix invalid return type for comparison of tuples with
NULL
elements. Fixes #12461. #13420 (Nikolai Kochetov). - Fix error in
parseDateTimeBestEffort
function when unix timestamp was passed as an argument. This fixes #13362. #13441 (Alexey Milovidov).
Build/Testing/Packaging Improvement
- Add compiler option to control that stack frames are not too large. This will help to run the code in fibers with small stack size. #11524 (Alexey Milovidov).
- Rework integrational configuration paths. #12285 (Ilya Yatsishin).
- Add docker image for fast tests. #12294 (alesapin).
- Update fmtlib to master (7.0.1). #12446 (Alexey Milovidov).
- Add PEERDIR(protoc) as protobuf format parses .proto file in runtime. #12475 (Yuriy Chernyshov).
- Now we check that server is able to start after stress tests run. This fixes #12473. #12496 (alesapin).
- Improve performance of TestKeeper. This will speedup tests with heavy usage of Replicated tables. #12505 (Alexey Milovidov).
- Adding RBAC syntax tests in TestFlows. #12642 (vzakaznikov).
- Apply random query mutations (fuzzing) in stress tests. #12734 (Alexander Kuzmenkov).
- Connector-ODBC updated to mysql-connector-odbc-8.0.21. #12739 (Ilya Yatsishin).
- Fix build of test under Mac OS X. This closes #12767. #12772 (Alexey Milovidov).
- increasing timeouts in testflows tests. #12949 (vzakaznikov).
- Add a test for
ALTER TABLE CLEAR COLUMN
query for primary key columns. #12951 (alesapin). - Do not build helper_container image inside integrational tests. Build docker container in CI and use pre-built helper_container in integration tests. #12953 (Ilya Yatsishin).
- Check an ability that we able to restore the backup from an old version to the new version. This closes #8979. #12959 (alesapin).
- Fix MSan error in "rdkafka" library. This closes #12990. Updated
rdkafka
to version 1.5 (master). #12991 (Alexey Milovidov). - Rerun some tests in fast test check. #12992 (alesapin).
- Adding extra xfails for some ldap tests. #13054 (vzakaznikov).
- Added tests for RBAC functionality of
SELECT
privilege in TestFlows. #13061 (Ritaank Tiwari). - Rewrote Function tests to gtest. Removed useless includes from tests. #13073 (Nikita Mikhaylov).
- Improve cache purge in documentation deploy script. #13107 (alesapin).
- Fixing 00960_live_view_watch_events_live.py test. #13108 (vzakaznikov).
- Small fixes to the RBAC SRS. #13152 (vzakaznikov).
- Even more retries in zkutil gtest to prevent test flakiness. #13165 (Alexey Milovidov).
- Add script which set labels for pull requests in GitHub hook. #13183 (alesapin).
- Use
shellcheck
for sh tests linting. #13200 (Alexey Milovidov). - Use
shellcheck
for sh tests linting. #13207 (Alexey Milovidov). - Fix libunwind build in AArch64. This fixes #13204. #13208 (Alexey Milovidov).
- Function
materialize
(the function for ClickHouse testing) will work for NULL as expected - by transforming it to non-constant column. #13212 (Alexey Milovidov). - Now fast test will wait server with retries. #13284 (alesapin).
- Applying LDAP authentication test fixes. #13310 (vzakaznikov).
- Fix timeout error during server restart in the stress test. #13321 (alesapin).
- Remove some of recursive submodules. See #13378. #13379 (Alexey Milovidov).
- Ensure that all the submodules are from proper URLs. Continuation of #13379. This fixes #13378. #13397 (Alexey Milovidov).
Other
- Avoid re-loading completion from the history file after each query (to avoid history overlaps with other client sessions). #13086 (Azat Khuzhin).
NO CL ENTRY
- NO CL ENTRY: 'Bump numpy from 1.18.5 to 1.19.1 in /docs/tools'. #12655 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump urllib3 from 1.25.9 to 1.25.10 in /docs/tools'. #12703 (dependabot-preview[bot]).
- NO CL ENTRY: 'Revert "Refactor joinGet and implement multi-key lookup."'. #12708 (alesapin).
- NO CL ENTRY: 'Revert "Abort on std::out_of_range in debug builds"'. #12752 (Alexey Milovidov).
- NO CL ENTRY: 'Bump protobuf from 3.12.2 to 3.12.4 in /docs/tools'. #13102 (dependabot-preview[bot]).
- NO CL ENTRY: 'Merge #12574'. #13158 (Alexey Milovidov).
- NO CL ENTRY: 'Revert "Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQuer…"'. #13303 (Alexey Milovidov).