mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-05 14:02:21 +00:00
40 KiB
40 KiB
ClickHouse release v20.10.2.20-stable FIXME as compared to v20.9.1.4585-prestable
Backward Incompatible Change
- Add support for nested multiline comments
/* comment /* comment */ */
in SQL. This conforms to the SQL standard. #14655 (Alexey Milovidov). - Change default value of
format_regexp_escaping_rule
setting (it's related toRegexp
format) toRaw
(it means - read whole subpattern as a value) to make the behaviour more like to what users expect. #15426 (Alexey Milovidov). - Make
multiple_joins_rewriter_version
obsolete. Remove first version of joins rewriter. #15472 (Artem Zuikov).
New Feature
- Add new feature: format LineAsString that accepts a sequence of line separated by newlines, spaces and/or commas. #14703 (Nikita Mikhaylov).
- Added
formatReadableQuantity
function. It is useful for reading big numbers by human. #14725 (Artem Hnilov). - Add the ability to remove column properties and table TTLs. Introduced queries
ALTER TABLE MODIFY COLUMN col_name REMOVE what_to_remove
andALTER TABLE REMOVE TTL
. Both operations are lightweight and executed at the metadata level. #14742 (alesapin). - Introduce event_time_microseconds field to
system.text_log
,system.trace_log
,system.query_log
andsystem.query_thread_log
tables. #14760 (Bharat Nallan). - Now we support
WITH <identifier> AS (subquery) ...
to introduce named subqueries in the query context. This closes #2416. This closes #4967. #14771 (Amos Bird). - Allow to omit arguments for Replicated table engine if defaults are specified in config. #14791 (vxider).
- Add table function
null('structure')
. #14797 (vxider). - Added query obfuscation tool. It allows to share more queries for better testing. This closes #15268. #15321 (Alexey Milovidov).
- Added format
RawBLOB
. It is intended for input or output a single value without any escaping and delimiters. This closes #15349. #15364 (Alexey Milovidov). - fix 15350. #15443 (flynn).
- Introduce
enable_global_with_statement
setting which propagates the first select'sWITH
statements to other select queries at the same level, and makes aliases inWITH
statements visible to subqueries. #15451 (Amos Bird). - Add the
reinterpretAsUUID
function that allows to convert a big-endian byte string to UUID. #15480 (Alexander Kuzmenkov). - Add parallel quorum inserts. This closes #15601. #15601 (Latysheva Alexandra).
Performance Improvement
- Improve performance of 256-bit bytes using (u)int64_t as base type for wide integers. Original wide integers use 8-bit types as base. #14859 (Artem Zuikov).
- Only
mlock
code segment when starting clickhouse-server. In previous versions, all mapped regions were locked in memory, including debug info. Debug info is usually splitted to a separate file but if it isn't, it led to +2..3 GiB memory usage. #14929 (Alexey Milovidov). - We used to choose fixed key method to group by one fixed string. It's unnecessary since we have StringHashTable which do the similar packedFix optimization for FixedString columns. And we should use low_cardinality_key_fixed_string if possible. #15034 (Amos Bird).
- Fix
DateTime <op> DateTime
mistakenly choosing the slow generic implementation. This fixes #15153 . #15178 (Amos Bird). - Use one S3 DeleteObjects request instead of multiple DeleteObject in cycle. No any functionality changes, so covered by existing tests like integration/test_log_family_s3. #15238 (ianton-ru).
- Faster 256-bit multiplication. #15418 (Artem Zuikov).
- Improve
quantileTDigest
performance. This fixes #2668. #15542 (Kruglov Pavel). - Explicitly use a temporary disk to store vertical merge temporary data. #15639 (Pervakov Grigorii).
Improvement
- Add
QueryMemoryLimitExceeded
event. This closes #14589. #14647 (fastio). - Fixed the backward-incompatible change by providing the options to build without debug info for functions. #14657 (Mike Kot).
- dynamic reload zookeeper config. #14678 (sundyli).
- Allow parallel execution of distributed DDL. #14684 (Azat Khuzhin).
- Fix potential memory leak caused by zookeeper exists watch. #14693 (hustnn).
- Fixed "Packet payload is not fully read" error in
MaterializeMySQL
database engine. #14696 (BohuTANG). - Fix crash in
bitShiftLeft()
when called with negative big integer. #14697 (Artem Zuikov). - Add
merge_algorithm
to system.merges table to improve merging inspections. #14705 (Amos Bird). - Less unneded code generated by DecimalBinaryOperation template in FunctionBinaryArithmetic. #14743 (Artem Zuikov).
- Now columns can be used to wrap over a list of columns and apply column transformers afterwards. #14775 (Amos Bird).
- Support for disabling persistency for StorageJoin and StorageSet, this feature is controlled by setting
disable_set_and_join_persistency
. And this PR solved issue #6318. #14776 (vxider). - Construct
query_start_time
andquery_start_time_microseconds
from the same timespec. #14831 (Bharat Nallan). - Allow using multi-volume storage configuration in storage Distributed. #14839 (Pavel Kovalenko).
- Show subqueries for
SET
andJOIN
inEXPLAIN
result. #14856 (Nikolai Kochetov). - Provide a
load_balancing_first_offset
query setting to explicitly state what the first replica is. It's used together withFIRST_OR_RANDOM
load balancing strategy, which allows to control replicas workload. #14867 (Amos Bird). - Fixed excessive settings constraint violation when running SELECT with SETTINGS from a distributed table. #14876 (Amos Bird).
- Allow to drop Replicated table if previous drop attempt was failed due to ZooKeeper session expiration. This fixes #11891. #14926 (Alexey Milovidov).
- Avoid deadlock when executing INSERT SELECT into itself from a table with
TinyLog
orLog
table engines. This closes #6802. #14962 (Alexey Milovidov). - Ignore key constraints when doing mutations. Without this pr, it's not possible to do mutations when
force_index_by_date = 1
orforce_primary_key = 1
. #14973 (Amos Bird). - Add option to disable TTL move on data part insert. #15000 (Pavel Kovalenko).
- Enable
Atomic
database engine by default. #15003 (Alexander Tokmakov). - Proper exception message for wrong number of arguments of CAST. This closes #13992. #15029 (Alexey Milovidov).
- Add the ability to specify specialized codecs like
Delta
,T64
, etc. for columns with subtypes. Implements #12551, fixes #11397, fixes #4609. #15089 (alesapin). - Added
optimize
setting toEXPLAIN PLAN
query. If enabled, query plan level optimisations are applied. Enabled by default. #15201 (Nikolai Kochetov). - Do not allow connections to ClickHouse server until all scripts in
/docker-entrypoint-initdb.d/
are executed. #15244 (Aleksei Kozharin). - fix 15264. #15285 (flynn).
- Unfold
{database}
,{table}
and{uuid}
macros inzookeeper_path
on replicated table creation. Do not allowRENAME TABLE
if it may breakzookeeper_path
after server restart. Fixes #6917. #15348 (Alexander Tokmakov). - Add support for "Raw" column format for
Regexp
format. It allows to simply extract subpatterns as a whole without any escaping rules. #15363 (Alexey Milovidov). - Now it's possible to change the type of version column for
VersionedCollapsingMergeTree
withALTER
query. #15442 (alesapin). - Wait for
DROP/DETACH TABLE
to actually finish ifNO DELAY
orSYNC
is specified forAtomic
database. #15448 (Alexander Tokmakov). - Pass through *_for_user settings via Distributed with cluster-secure. #15551 (Azat Khuzhin).
- Use experimental pass manager by default. #15608 (Daniel Kutenin).
- Implement force_data_skipping_indices setting. #15642 (Azat Khuzhin).
- Backported in #16102: Now it's allowed to execute
ALTER ... ON CLUSTER
queries regardless of the<internal_replication>
setting in cluster config. #16075 (alesapin).
Bug Fix
- Fix wrong Decimal multiplication result caused wrong decimal scale of result column. #14603 (Artem Zuikov).
- Stuff the query into ASTFunction's argument list so that we don't break the presumptions of some AST visitors. This fixes #14608. #14611 (Amos Bird).
- Fix bug when
ALTER UPDATE
mutation with Nullable column in assignment expression and constant value (likeUPDATE x = 42
) leads to incorrect value in column or segfault. Fixes #13634, #14045. #14646 (alesapin). - Fixed missed default database name in metadata of materialized view when executing
ALTER ... MODIFY QUERY
. #14664 (Alexander Tokmakov). - Replace column transformer should replace identifiers with cloned ASTs. This fixes #14695 . #14734 (Amos Bird).
- Fix wrong monotonicity detection for shrunk
Int -> Int
cast of signed types. It might lead to incorrect query result. This bug is unveiled in #14513. #14783 (Amos Bird). - Fix unreleased bug for LineAsString Format. #14842 (hexiaoting).
- Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the
from_zk
include option). This fixes #14814. #14843 (Alexander Kuzmenkov). - Fix rare error in
SELECT
queries when the queried column hasDEFAULT
expression which depends on the other column which also hasDEFAULT
and not present in select query and not exists on disk. Partially fixes #14531. #14845 (alesapin). - Fixed bug in parsing MySQL binlog events, which causes
Attempt to read after eof
andPacket payload is not fully read
inMaterializeMySQL
database engine. #14852 (Winter Zhang). - Fixed segfault in CacheDictionary #14837. #14879 (Nikita Mikhaylov).
- Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). #14887 (Azat Khuzhin).
- Fix the issue when some invocations of
extractAllGroups
function may trigger "Memory limit exceeded" error. This fixes #13383. #14889 (Alexey Milovidov). - Fixed
.metadata.tmp File exists
error when usingMaterializeMySQL
database engine. #14898 (Winter Zhang). - Publish CPU frequencies per logical core in
system.asynchronous_metrics
. This fixes #14923. #14924 (Alexander Kuzmenkov). - Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes #14847. #14937 (filimonov).
- Update jemalloc to fix possible issues with percpu arena. #14957 (Azat Khuzhin).
- Now settings
number_of_free_entries_in_pool_to_execute_mutation
andnumber_of_free_entries_in_pool_to_lower_max_size_of_merge
can be equal tobackground_pool_size
. #14975 (alesapin). - Fix crash in RIGHT or FULL JOIN with join_algorith='auto' when memory limit exceeded and we should change HashJoin with MergeJoin. #15002 (Artem Zuikov).
- Fixed
Cannot rename ... errno: 22, strerror: Invalid argument
error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. #15024 (Alexander Tokmakov). - If function
bar
was called with specifically crafted arguments, buffer overflow was possible. This closes #13926. #15028 (Alexey Milovidov). - We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes #14908. #15033 (Amos Bird).
- Backported in #15829: Update jemalloc to fix percpu_arena with affinity mask. #15035 (Azat Khuzhin).
- Fixes
Data compressed with different methods
injoin_algorithm='auto'
. Keep LowCardinality as type for left table join key injoin_algorithm='partial_merge'
. #15088 (Artem Zuikov). - Adjust decimals field size in mysql column definition packet. #15152 (maqroll).
- Fix bug in table engine
Buffer
which doesn't allow to insert data of new structure intoBuffer
afterALTER
query. Fixes #15117. #15192 (alesapin). - Fix instance crash when using joinGet with LowCardinality types. This fixes #15214. #15220 (Amos Bird).
- Fix 'Unknown identifier' in GROUP BY when query has JOIN over Merge table. #15242 (Artem Zuikov).
- Fix MSan report in QueryLog. Uninitialized memory can be used for the field
memory_usage
. #15258 (Alexey Milovidov). - Fix hang of queries with a lot of subqueries to same table of
MySQL
engine. Previously, if there were more than 16 subqueries to sameMySQL
table in query, it hang forever. #15299 (Anton Popov). - Fix rare race condition on server startup when system.logs are enabled. #15300 (alesapin).
- Fix race condition during MergeTree table rename and background cleanup. #15304 (alesapin).
- Fix bug where queries like SELECT toStartOfDay(today()) fail complaining about empty time_zone argument. #15319 (Bharat Nallan).
- Fixed compression in S3 storage. #15376 (Vladimir Chebotarev).
- Fix multiple occurrences of column transformers in a select query. #15378 (Amos Bird).
- fixes #15365 fix attach mysql database engine throw exception(no query context). #15384 (Winter Zhang).
- Report proper error when the second argument of
boundingRatio
aggregate function has a wrong type. #15407 (detailyang). - Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in
ON CLUSTER
. Introduced in #13450. #15477 (alesapin). - Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. #15516 (nvartolomei).
- Fix
Missing columns
errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes #15530. #15532 (alesapin). - Fix bug when
ILIKE
operator stops being case insensitive ifLIKE
with the same pattern was executed. #15536 (alesapin). - Mutation might hang waiting for some non-existent part after
MOVE
orREPLACE PARTITION
or, in rare cases, afterDETACH
orDROP PARTITION
. It's fixed. #15537 (Alexander Tokmakov). - Fix 'Database doesn't exist.' in queries with IN and Distributed table when there's no database on initiator. #15538 (Artem Zuikov).
- Significantly reduce memory usage in AggregatingInOrderTransform/optimize_aggregation_in_order. #15543 (Azat Khuzhin).
- Prevent the possibility of error message
Could not calculate available disk space (statvfs), errno: 4, strerror: Interrupted system call
. This fixes #15541. #15557 (Alexey Milovidov). - Query is finished faster in case of exception. Cancel execution on remote replicas if exception happens. #15578 (Azat Khuzhin).
- Fixed
Element ... is not a constant expression
error when usingJSON*
function result inVALUES
,LIMIT
or right side ofIN
operator. #15589 (Alexander Tokmakov). - Fix the order of destruction for resources in
ReadFromStorage
step of query plan. It might cause crashes in rare cases. Possibly connected with #15610. #15645 (Nikolai Kochetov). - Backported in #16009: Fixed bug with globs in S3 table function, region from URL was not applied to S3 client configuration. #15646 (Vladimir Chebotarev).
- Proper error handling during insert into MergeTree with S3. #15657 (Pavel Kovalenko).
- Backported in #15869: Fix error
Cannot add simple transform to empty Pipe
which happened while reading fromBuffer
table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes #15529. #15662 (Nikolai Kochetov). - Fix race condition in AMQP-CPP. #15667 (alesapin).
- Fix rare race condition in dictionaries and tables from MySQL. #15686 (alesapin).
- Fixed too low default value of
max_replicated_logs_to_keep
setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. #15701 (Alexander Tokmakov). - Fix error
Cannot find column
which may happen at insertion intoMATERIALIZED VIEW
in case if query forMV
containesARRAY JOIN
. #15717 (Nikolai Kochetov). - Backported in #15926: Fix drop of materialized view with inner table in Atomic database (hangs all subsequent DROP TABLE due to hang of the worker thread, due to recursive DROP TABLE for inner table of MV). #15743 (Azat Khuzhin).
- Fix some cases of queries, in which only virtual columns are selected. Previously
Not found column _nothing in block
exception may be thrown. Fixes #12298. #15756 (Anton Popov). - Backported in #15867: Fix
select count()
inaccuracy for MaterializeMySQL. #15767 (Alexander Tokmakov). - Backported in #15902: Fix exception
Block structure mismatch
inSELECT ... ORDER BY DESC
queries which were executed afterALTER MODIFY COLUMN
query. Fixes #15800. #15852 (alesapin). - Backported in #15922: Now exception will be thrown when
ALTER MODIFY COLUMN ... DEFAULT ...
has incompatible default with column type. Fixes #15854. #15858 (alesapin). - Backported in #15917: Fix possible deadlocks in RBAC. #15875 (Vitaly Baranov).
- Backported in #15950: fixes #12513 fix difference expressions with same alias when analyze queries again. #15886 (Winter Zhang).
- Backported in #16169: Fix incorrect empty result for query from
Distributed
table if query hasWHERE
,PREWHERE
andGLOBAL IN
. Fixes #15792. #15933 (Nikolai Kochetov). - Backported in #15969: Fix a crash when database creation fails. #15954 (Winter Zhang).
- Backported in #16023: Fix ambiguity in parsing of settings profiles:
CREATE USER ... SETTINGS profile readonly
is now considered as using a profile namedreadonly
, not a setting namedprofile
with the readonly constraint. This fixes #15628. #15982 (Vitaly Baranov). - Backported in #16217: Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). #15984 (Alexander Tokmakov).
- Backported in #16027: Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. #15987 (filimonov).
- Backported in #16089: Allow to use direct layout for dictionaries with complex keys. #16007 (Anton Popov).
- Backported in #16142: Fix
ALTER MODIFY ... ORDER BY
query hang forReplicatedVersionedCollapsingMergeTree
. This fixes #15980. #16011 (alesapin). - Backported in #16077: Fixes #15780 regression, e.g. indexOf([1, 2, 3], toLowCardinality(1)) now is prohibited but it should not be. #16038 (Mike Kot).
- Backported in #16120: Fix segfault in some cases of wrong aggregation in lambdas. #16082 (Anton Popov).
Build/Testing/Packaging Improvement
- Now ClickHouse uses gcc-10 for the release build. Fixes #11138. #14609 (alesapin).
- Attempt to make performance test more reliable. It is done by remapping the executable memory of the process on the fly with
madvise
to use transparent huge pages - it can lower the number of iTLB misses which is the main source of instabilities in performance tests. #14685 (Alexey Milovidov). -
- In CMake files: - Moved some options' descriptions' parts to comments above. - Replace 0 ->
OFF
, 1 ->ON
inoption
s default values. - Added some descriptions and links to docs to the options. - ReplacedFUZZER
option (there is another optionENABLE_FUZZING
which also enables same functionality). - RemovedENABLE_GTEST_LIBRARY
option as there isENABLE_TESTS
. #14711 (Mike Kot).
- In CMake files: - Moved some options' descriptions' parts to comments above. - Replace 0 ->
- Speed up build a little by removing unused headers. #14714 (Alexey Milovidov).
- Fix build failure in OSX. #14761 (Winter Zhang).
- Attempt to speed up build a little. #14808 (Alexey Milovidov).
- Now we use clang-11 to build ClickHouse in CI. #14846 (alesapin).
- #14809 fix MaterializeMySQL empty transaction unstable test case found in CI. #14854 (Winter Zhang).
- Reformat and cleanup code in all integration test *.py files. #14864 (Bharat Nallan).
- Fixing tests/integration/test_distributed_over_live_view/test.py. #14892 (vzakaznikov).
- Switch from
clang-tidy-10
toclang-tidy-11
. #14922 (Alexey Milovidov). - Convert to python3. This closes #14886. #15007 (Azat Khuzhin).
- Make performance test more stable and representative by splitting test runs and profile runs. #15027 (Alexey Milovidov).
- Maybe fix MSan report in base64 (on servers with AVX-512). This fixes #14006. #15030 (Alexey Milovidov).
- Don't allow any C++ translation unit to build more than 10 minutes or to use more than 10 GB or memory. This fixes #14925. #15060 (Alexey Milovidov).
- Now all test images use
llvm-symbolizer-11
. #15069 (alesapin). - Splitted huge test
test_dictionaries_all_layouts_and_sources
into smaller ones. #15110 (Nikita Mikhaylov). - Added a script to perform hardware benchmark in a single command. #15115 (Alexey Milovidov).
- Fix CMake options forwarding in fast test script. Fixes error in #14711. #15155 (alesapin).
- Improvements in CI docker images: get rid of ZooKeeper and single script for test configs installation. #15215 (alesapin).
- Now we use clang-11 for production ClickHouse build. #15239 (alesapin).
- Allow to run AArch64 version of clickhouse-server without configs. This facilitates #15174. #15266 (Alexey Milovidov).
- Fail early in functional tests if server failed to respond. This closes #15262. #15267 (Alexey Milovidov).
- fix bug for build error: #15272. #15297 (hexiaoting).
- fix bug for building query_db_generator.cpp. #15353 (hexiaoting).
- Allow to build with llvm-11. #15366 (Alexey Milovidov).
- Switch binary builds(Linux, Darwin, AArch64, FreeDSD) to clang-11. #15622 (Ilya Yatsishin).
NO CL ENTRY
- NO CL ENTRY: 'Bump numpy from 1.19.1 to 1.19.2 in /docs/tools'. #14733 (dependabot-preview[bot]).
- NO CL ENTRY: 'Modify the minimum version of the Clang compiler'. #14882 (Simon Liu).
- NO CL ENTRY: 'fix a syntax error bug while using copier'. #14890 (HyaZz).
- NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.9 to 0.4.13 in /docs/tools'. #15067 (dependabot-preview[bot]).
- NO CL ENTRY: 'Revert "Test and doc for PR12771 krb5 + cyrus-sasl + kerberized kafka"'. #15232 (Alexander Tokmakov).
- NO CL ENTRY: 'Revert "Avoid deadlocks in Log/TinyLog"'. #15259 (Alexey Milovidov).
- NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.13 to 0.4.17 in /docs/tools'. #15460 (dependabot-preview[bot]).