mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-05 05:52:05 +00:00
83 KiB
83 KiB
ClickHouse release v20.5.1.3833-prestable FIXME as compared to v20.4.1.3177-prestable
Backward Incompatible Change
- Remove
experimental_use_processors
setting. It is enabled by default. #10924 (Nikolai Kochetov). - The setting
input_format_with_names_use_header
is enabled by default. It will affect parsing of input formats-WithNames
and-WithNamesAndTypes
. #10937 (Alexey Milovidov). - Added a check for the case when user-level setting is specified in a wrong place. User-level settings should be specified in
users.xml
inside<profile>
section for specific user profile (or in<default>
for default settings). The server won't start with exception message in log. This fixes #9051. If you want to skip the check, you can either move settings to the appropriate place or add<skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings>
to config.xml. #11449 (Alexey Milovidov). - Return non-Nullable result from COUNT(DISTINCT), and
uniq
aggregate functions family. If all passed values are NULL, return zero instead. This improves SQL compatibility. #11661 (Alexey Milovidov).
New Feature
- Support Cassandra as external dictionary. #4978 (favstovol).
SimpleAggregateFunction
now also supportssumMap
. #10000 (Ildus Kurbangaliev).- Added OpenCl support and bitonic sort algorithm, which can be used for sorting integer types of data in single column. Needs to be build with flag
-DENABLE_OPENCL=1
. For using bitonic sort algorithm instead of others you need to setbitonic_sort
for Setting's optionspecial_sort
and make sure that OpenCL is available. #10232 (Margarita Konnova [MARK]). - Implementation of PostgreSQL wire protocol. #10242 (Movses Elbakian).
- Selects with final are executed in parallel. Added setting
max_final_threads
to limit the number of threads used. #10463 (Nikolai Kochetov). - Function that extracts from haystack all matching non-overlapping groups with regular expressions, and put those into
Array(Array(String))
column. #10534 (Vasily Nemkov). - Added ability to delete a subset of expired rows, which satisfies the condition in WHERE clause. Added ability to replace expired rows with aggregates of them specified in GROUP BY clause. #10537 (expl0si0nn).
- (Only Linux) ClickHouse server now tries to fallback to ProcfsMetricsProvider when clickhouse binary is not attributed with CAP_NET_ADMIN capability to collect per-query system metrics (for CPU and I/O). #10544 (Alexander Kazakov).
- Allowed to profile memory with finer granularity steps than 4 MiB. Added sampling memory profiler to capture random allocations/deallocations. #10598 (Alexey Milovidov).
- Add new input format
JSONAsString
that accepts a sequence of JSON objects separated by newlines, spaces and/or commas. #10607 (Kruglov Pavel). - Added a new layout
direct
which loads all the data directly from the source for each query, without storing or caching data. #10622 (Artem Streltsov). - Default user and database creation on docker image starting. #10637 (Paramtamtam).
- Add data type Point (Tuple(Float64, Float64)) and Polygon (Array(Array(Tuple(Float64, Float64))). #10678 (Alexey Ilyukhov).
- New function function toStartOfSecond(DateTime64) -> DateTime64 that nullifies sub-second part of DateTime64 value. #10722 (Vasily Nemkov).
- Added a function
randomString
that generates binary string with random bytes (including zero bytes). #10733 (Andrei Nekrashevich). - Added
system.licenses
table. This table contains licenses of third-party libraries that are located incontrib
directory. This closes #2890. #10795 (Alexey Milovidov). - OFFSET keyword can now be used without an affiliated LIMIT clause. #10802 (Guillaume Tassery).
- Added new complex key direct layout to dictionaries, that does not store anything locally during query execution. #10850 (Artem Streltsov).
- Added function
randomFixedString
. #10866 (Andrei Nekrashevich). - Support
ALTER RENAME COLUMN
for the distributed table engine. Continuation of #10727. Fixes #10747. #10887 (alesapin). - Support writes in ODBC Table function #10554 (ageraab). #10901 (Alexander Tokmakov).
- Allow specifying
mongodb://
URI for MongoDB dictionaries. #10915 (Alexander Kuzmenkov). - Added new functions to import/export DateTime64 as Int64 with various precision:
to-/fromUnixTimestamp64Milli/-Micro/-Nano
. #10923 (Vasily Nemkov). - Now support NULL and NOT NULL modifiers for data types in create query. #11057 (Павел Потемкин).
- Add ArrowStream input and output format. #11088 (hcz).
- Default S3 credentials and custom auth headers. #11134 (Pervakov Grigorii).
- Add query performance metrics based on Linux
perf_events
. #9545 Andrey Skobtsov. #11226 (Alexander Kuzmenkov). - Use HTTP client for S3 based on Poco. #11230 (Pavel Kovalenko).
- Add function
fuzzBits
that randomly flips bits in a string with given probability. #11237 (Andrei Nekrashevich). - Add
_timestamp_ms
virtual column for Kafka engine (type isNullable(DateTime64(3))
). #11260 (filimonov). - Add 2 more virtual columns for engine=Kafka to access message headers. #11283 (filimonov).
- Add
netloc
function for extracting network location, similar tourlparse(url)
,netloc
in python. #11356 (Guillaume Tassery). - Added syntax highligting to clickhouse-client using ReplXX. #11422 (Tagir Kuskarov).
- Add SHOW CLUSTER(S) queries. #11467 (hexiaoting).
- Add functions
extractAllGroupsHorizontal(haystack, re)
andextractAllGroupsVertical(haystack, re)
. #11554 (Vasily Nemkov). - Add the
system.asynchronous_metric_log
table that logs historical metrics fromsystem.asynchronous_metrics
. #11588 (Alexander Kuzmenkov). minMap
andmaxMap
functions were added. #11603 (Ildus Kurbangaliev).- Added support for MySQL style global variables syntax (stub). This is needed for compatibility of MySQL protocol. #11832 (Alexey Milovidov).
Performance Improvement
- Optimization of GROUP BY with respect to table sorting key. #9113 (Dmitry Rubashkin).
- New optimization that takes arithmetic operations out of aggregate functions. #10047 (Ruslan).
- This feature eliminates functions of other keys in GROUP BY section. #10051 (Victor Grishanin).
- Add runtime CPU detection to select and dispatch the best function implementation. Add support for codegeneration for multiple targets. This closes #1017. #10058 (DimasKovas).
- Remove duplicate ORDER BY and DISTINCT from subqueries. #10067 (Mikhail Malafeev).
- Sort bigger parts of the left table in MergeJoin. Buffer left blocks in memory. Add
partial_merge_join_left_table_buffer_bytes
setting to manage the left blocks buffers sizes. #10601 (Artem Zuikov). - Get dictionary and check access rights only once per each call of any function reading external dictionaries. #10928 (Vitaly Baranov).
- Improving radix sort by removing some redundant data moves. #10981 (Arslan Gumerov).
- Make queries with
sum
aggregate function and without GROUP BY keys to run multiple times faster. #10992 (Alexey Milovidov). - Enable mlock of clickhouse binary by default. It will prevent clickhouse executable from being paged out under high IO load. #11139 (Alexey Milovidov).
- Improved performance for queries with
ORDER BY
and smallLIMIT
(less, thenmax_block_size
). #11171 (Provet). - Improve performance for INSERT queries via INSERT SELECT or INSERT with clickhouse-client when small blocks are generated (typical case with parallel parsing). This fixes #11275. Fix the issue that CONSTRAINTs were not working for DEFAULT fields. This fixes #11273. Fix the issue that CONSTRAINTS were ignored for TEMPORARY tables. This fixes #11274. #11276 (Alexey Milovidov).
- Improve performance of
clickhouse-client
in interactive mode when Pretty formats are used. In previous versions, significant amount of time can be spent calculating visible width of UTF-8 string. This closes #11323. #11323 (Alexey Milovidov). - New optimization that takes all operations out of "any" function. #11529 (Ruslan).
- Speed up merging in AggregatingMergeTree. This fixes performance regression that was introduced more than a year ago in #4348 (in version 19.3). #11534 (Alexey Milovidov).
- Allow multiple replicas to assign merges, mutations, partition drop, move and replace concurrently. This closes #10367. #11639 (Alexey Milovidov).
- This optimization eliminates min/max/any aggregators of GROUP BY keys in SELECT section. #11667 (Victor Grishanin).
Improvement
- Allow to pass quota_key in clickhouse-client. This closes #10227. #10270 (Alexey Milovidov).
- Added system tables for users, roles, grants, settings profiles, quotas, row policies; added commands SHOW USER, SHOW [CURRENT|ENABLED] ROLES, SHOW SETTINGS PROFILES. #10387 (Vitaly Baranov).
-
- Adding support for
INSERT INTO [db.]table WATCH
query. #10498 (vzakaznikov).
- Adding support for
- Possibility to work with S3 through proxies. #10576 (Pavel Kovalenko).
- Added
move_ttl_info
tosystem.parts
in order to provide introspection of move TTL functionality. #10591 (Vladimir Chebotarev). - Make pointInPolygon work with non-constant polygon. PointInPolygon now can take Array(Array(Tuple(..., ...))) as second argument, array of polygon and holes. #10623 (Alexey Ilyukhov).
- Print a message if clickhouse-client is newer than clickhouse-server. #10627 (Alexey Milovidov).
- Change HTTP response code in case of some parse errors to 400 Bad Request. This fix #10636. #10640 (Alexey Milovidov).
- Added a check for meaningless codecs and a setting
allow_suspicious_codecs
to control this check. This closes #4966. #10645 (Alexey Milovidov). - Adding support for ALTER RENAME COLUMN query to Distributed table engine. #10727 (vzakaznikov).
- Possibility to configure proxy-resolver for DiskS3. #10744 (Pavel Kovalenko).
- Better DNS exception message. This fixes #10813. #10828 (Alexey Milovidov).
- Ensure that
varSamp
,varPop
cannot return negative results due to numerical errors and thatstddevSamp
,stddevPop
cannot be calculated from negative variance. This fixes #10532. #10829 (Alexey Milovidov). - Get rid of exception from replicated queue during server shutdown. Fixes #10819. #10841 (alesapin).
- The
clickhouse-format
tool is now able to format multiple queries when the-n
argument is used. #10852 (Darío). - Provide synonims for some data types. #10856 (Павел Потемкин).
- Introduce
min_insert_block_size_rows_for_materialized_views
,min_insert_block_size_bytes_for_materialized_views
settings. This settings are similar tomin_insert_block_size_rows
andmin_insert_block_size_bytes
, but applied only for blocks inserted intoMATERIALIZED VIEW
. It helps to control blocks squashing while pushing to MVs and avoid excessive memory usage. #10858 (Azat Khuzhin). - Respect prefer_localhost_replica/load_balancing on INSERT into Distributed. #10867 (Azat Khuzhin).
- Allow large UInt types as the index in function
tupleElement
. #10874 (hcz). - Support for unicode whitespaces in queries. This helps when queries are copy-pasted from Word or from web page. This fixes #10896. #10903 (Alexey Milovidov).
- Some additions and cleanup for #10232. #10934 (Artem Zuikov).
- Set thread names for internal threads of rdkafka library. Make logs from rdkafka available in server logs. #10983 (Azat Khuzhin).
- Remove data on explicit
DROP DATABASE
forMemory
database engine. Fixes #10557. #11021 (Alexander Tokmakov). - Add
NCHAR
andNVARCHAR
synonims for data types. #11025 (Alexey Milovidov). - Apply
TTL
for old data, afterALTER MODIFY TTL
query. This behaviour is controlled by settingmaterialize_ttl_after_modify
, which is enabled by default. #11042 (Anton Popov). - Now
dictGet*
functions accept table names. #11050 (Vitaly Baranov). - Show authentication type in table system.users and while executing SHOW CREATE USER query. #11080 (Vitaly Baranov).
- Enable percpu_arena:percpu for jemalloc (This will reduce memory fragmentation due to thread pool). #11084 (Azat Khuzhin).
- Add port() function (to extract port from URL). #11120 (Azat Khuzhin).
- Resolved #7224: added
FailedQuery
,FailedSelectQuery
andFailedInsertQuery
metrics tosystem.events
table. #11151 (Nikita Orlov). - The query log is now enabled by default. #11184 (Ivan Blinkov).
- When parsing C-style backslash escapes in string literals, VALUES and various text formats (this is an extension to SQL standard that is endemic for ClickHouse and MySQL), keep backslash if unknown escape sequence is found (e.g.
\%
or\w
) that will make usage ofLIKE
andmatch
regular expressions more convenient (it's enough to writename LIKE 'used\_cars'
instead ofname LIKE 'used\\_cars'
) and more compatible at the same time. This fixes #10922. #11208 (Alexey Milovidov). - Add support for multi-word data type names (such as
DOUBLE PRECISION
andCHAR VARYING
) for better SQL compatibility. #11214 (Павел Потемкин). - Keep the value of
DistributedFilesToInsert
metric on exceptions. In previous versions, the value was set when we are going to send some files, but it is zero, if there was an exception and some files are still pending. Now it corresponds to the number of pending files in filesystem. #11220 (Alexey Milovidov). - Support kafka_client_id parameter for Kafka tables. It also changes the default
client.id
used by ClickHouse when communicating with Kafka to be more verbose and usable. #11252 (filimonov). - Update librdkafka to version 1.4.2. #11256 (filimonov).
- Support (U)Int8, (U)Int16, Date in ASOF JOIN. #11301 (Artem Zuikov).
- Better exception message in case when there is shortage of memory mappings. This closes #11027. #11316 (Alexey Milovidov).
- Add setting "output_format_pretty_max_value_width". If value is longer, it will be cut to avoid output of too large values in terminal. This closes #11140. #11324 (Alexey Milovidov).
- Remove trailing whitespaces from formatted queries in
clickhouse-client
orclickhouse-format
in some cases. #11325 (Alexey Milovidov). - Better log messages in while reloading configuration. #11341 (Alexey Milovidov).
- Suppress output of cancelled queries in clickhouse-client. In previous versions result may continue to print in terminal even after you press Ctrl+C to cancel query. This closes #9473. #11342 (Alexey Milovidov).
- OPTIMIZE FINAL will force merge even if concurrent merges are performed. This closes #11309 and closes #11322. #11346 (Alexey Milovidov).
- Support for all format settings in Kafka, expose some setting on table level, adjust the defaults for better performance. #11388 (filimonov).
- Add system.distribution_queue table. #11394 (Azat Khuzhin).
- ON CLUSTER support for SYSTEM {FLUSH DISTRIBUTED,STOP/START DISTRIBUTED SEND}. #11415 (Azat Khuzhin).
- Now history file is updated after each query and there is no race condition if multiple clients use one history file. This fixes #9897. #11453 (Tagir Kuskarov).
- Automatically update DNS cache, which is used to check if user is allowed to connect from an address. #11487 (Alexander Tokmakov).
- Adding support for PREWHERE in live view tables. #11495 (vzakaznikov).
- Improve
enable_optimize_predicate_expression=1
logic for VIEW. #11513 (Artem Zuikov). - Better exception message when cannot parse columns declaration list. This closes #10403. #11537 (Alexey Milovidov).
- Optimize memory usage when reading a response from an S3 HTTP client. #11561 (Pavel Kovalenko).
- Improve
multiple_joins_rewriter_version=2
logic. Fix unknown columns error for lambda aliases. #11587 (Artem Zuikov). - Make more input format work with Kafka engine. Fix the issue with premature flushes. Fix the performance issue when
kafka_num_consumers
is greater than number of partitions in topic. #11599 (filimonov). - https://github.com/ClickHouse/ClickHouse/pull/7572#issuecomment-642815377 Support config default HTTPHandlers. #11628 (Winter Zhang).
- Add round_robin load_balancing. #11645 (Azat Khuzhin).
- Allow comparison of numbers with constant string in comparison operators, IN and VALUES sections. #11647 (Alexey Milovidov).
- Allow comparison with constant strings by implicit conversions when analysing index conditions on other types. This may close #11630. #11648 (Alexey Milovidov).
- Allow using
groupArrayArray
andgroupUniqArrayArray
asSimpleAggregateFunction
. #11650 (Volodymyr Kuznetsov). - Skip empty parameters in requested URL. They may appear when you write
http://localhost:8123/?&a=b
orhttp://localhost:8123/?a=b&&c=d
. This closes #10749. #11651 (Alexey Milovidov). - Allow to DROP replicated table if the metadata in ZooKeeper was already removed and does not exist (this is also the case when using TestKeeper for testing and the server was restarted). Allow to RENAME replicated table even if there is an error communicating with ZooKeeper. This fixes #10720. #11652 (Alexey Milovidov).
- Return NULL/zero when value is not parsed completely in parseDateTimeBestEffortOrNull/Zero functions. This fixes #7876. #11653 (Alexey Milovidov).
- Added column
position
tosystem.columns
table andcolumn_position
tosystem.parts_columns
table. It contains ordinal position of a column in a table starting with 1. This closes #7744. #11655 (Alexey Milovidov). - Don't use debug info from ELF file if it doesn't correspond to the running binary. It is needed to avoid printing wrong function names and source locations in stack traces. This fixes #7514. #11657 (Alexey Milovidov).
- Clear password from command line in
clickhouse-client
andclickhouse-benchmark
if the user has specified it with explicit value. This prevents password exposure byps
and similar tools. #11665 (Alexey Milovidov). - Multiple names are now allowed in commands: CREATE USER, CREATE ROLE, ALTER USER, SHOW CREATE USER, SHOW GRANTS and so on. #11670 (Vitaly Baranov).
- When multiline query is printed to server log, the lines are joined. Make it to work correct in case of multiline string literals, identifiers and single-line comments. This fixes #3853. #11686 (Alexey Milovidov).
- Emit warning instead of error in server log at startup if we cannot listen one of the listen addresses (e.g. IPv6 is unavailable inside Docker). Note that if server fails to listen all listed addresses, it will refuse to startup as before. This fixes #4406. #11687 (Alexey Milovidov).
- Added support for distributed
DDL
(update/delete/drop partition) on cross replication clusters. #11703 (Nikita Mikhaylov). - Add
cast_keep_nullable
setting. If setCAST(something_nullable AS Type)
returnNullable(Type)
. #11733 (Artem Zuikov). - Add more
jemalloc
statistics tosystem.asynchronous_metrics
, and ensure that we see up-to-date values for them. #11748 (Alexander Kuzmenkov). - Add ability to set
MATERIALIZED
default type for primary key columns and columns with secondary indices. #11786 (alesapin). - Remove leader election, step 3: remove yielding of leadership; remove sending queries to leader. #11795 (Alexey Milovidov).
- Added
hostname
as an alias to functionhostName
. This feature was suggested by Victor Tarnavskiy from Yandex.Metrica. #11821 (Alexey Milovidov). - Fix sleep invocation in signal handler. It was sleeping for less amount of time than expected. #11825 (Alexey Milovidov).
- Slightly improve diagnostic of reading decimal from string. This closes #10202. #11829 (Alexey Milovidov).
- When reading Decimal value, cut extra digits after point. This behaviour is more compatible with MySQL and PostgreSQL. This fixes #10202. #11831 (Alexey Milovidov).
Bug Fix
- Fixed error with "max_rows_to_sort" limit. #10268 (Alexey Milovidov).
- Fixes: #10263 (after that PR dist send via INSERT had been postponing on each INSERT) Fixes: #8756 (that PR breaks distributed sends with all of the following conditions met (unlikely setup for now I guess):
internal_replication == false
, multiple local shards (activates the hardlinking code) anddistributed_storage_policy
(makeslink(2)
fails onEXDEV
)). #10486 (Azat Khuzhin). - Disable GROUP BY sharding_key optimization by default (
optimize_distributed_group_by_sharding_key
had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example isif
in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. #10516 (Azat Khuzhin). - Fix index corruption, which may accur in some cases after merge compact parts into another compact part. #10531 (Anton Popov).
- Implemented comparison between DateTime64 and String values (just like for DateTime). #10560 (Vasily Nemkov).
- Fix SELECT of column ALIAS which default expression type different from column type. #10563 (Azat Khuzhin).
- Fix error
the BloomFilter false positive must be a double number between 0 and 1
#10551. #10569 (Winter Zhang). - This PR fixes possible crash when
createDictionary()
is called beforeloadStoredObject()
has finished. #10587 (Vitaly Baranov). - Fixed handling condition variable for synchronous mutations. In some cases signals to that condition variable could be lost. #10588 (Vladimir Chebotarev).
- Fixed incorrect scalar results inside inner query of
MATERIALIZED VIEW
in case if this query contained dependent table. #10603 (Nikolai Kochetov). - On
SYSTEM DROP DNS CACHE
query also drop caches, which are used to check if user is allowed to connect from some IP addresses. #10608 (Alexander Tokmakov). - Fix segfault in StorageBuffer when exception on server startup. Fixes #10550. #10609 (Alexander Tokmakov).
- Fix optimize_skip_unused_shards with LowCardinality. #10611 (Azat Khuzhin).
- Fix predicates optimization for distributed queries (
enable_optimize_predicate_expression=1
) for queries withHAVING
section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin). - Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 (Alexey Milovidov).
- Fix bug which locks concurrent alters when table has a lot of parts. #10659 (alesapin).
- Fix possible incorrect number of rows for queries with
LIMIT
. Fixes #10566, #10709. #10660 (Nikolai Kochetov). - Fix the lack of parallel execution of remote queries with
distributed_aggregation_memory_efficient
enabled. Fixes #10655. #10664 (Nikolai Kochetov). - Fix order of parameters in AggregateTransform constructor. #10667 (palasonic1).
- Fixed bug, which causes http requests stuck on client close when
readonly=2
andcancel_http_readonly_queries_on_client_close=1
. Fixes #7939, #7019, #7736, #7091. #10684 (Alexander Tokmakov). - Fix multiple usages of
IN
operator with the identical set in one query. #10686 (Anton Popov). - Fix atomicity of HTTP insert. This fixes #9666. #10687 (Andrew Onyshchuk).
- Fix disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes #10674. #10698 (Nikolai Kochetov).
- Fix possible buffer overflow in function
h3EdgeAngle
. #10711 (Alexey Milovidov). - Fix usage of primary key wrapped into a function with 'FINAL' modifier and 'ORDER BY' optimization. #10715 (Anton Popov).
- Fix data corruption for
LowCardinality(FixedString)
key column inSummingMergeTree
which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov). - Fix crash in
generateRandom
with nested types. Fixes #10583. #10734 (Nikolai Kochetov). - Fix combinator -OrNull and -OrDefault when combined with -State. #10741 (hcz).
- Fix
parallel_view_processing
behavior. Now all insertions intoMATERIALIZED VIEW
without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov). - Fix avgWeighted when using floating-point weight over multiple shards. #10758 (Baudouin Giard).
- Get rid of old libunwind patches.
500aa22791 (r39048012)
This allows to disable-fno-omit-frame-pointer
inclang
builds that improves performance at least by 1% in average. #10761 (Amos Bird). - Make use of
src_type
for correct type conversion in key conditions. Fixes #6287. #10791 (Andrew Onyshchuk). - Fix UBSan and MSan report in DateLUT. #10798 (Alexey Milovidov).
- Fix the issue with ODBC bridge when no quoting of identifiers is requested. This fixes #7984. #10821 (Alexey Milovidov).
- Fix columns order after Block::sortColumns() (also add a test that shows that it affects some real use case - Buffer engine). #10826 (Azat Khuzhin).
- Fix potential read of uninitialized memory in cache dictionary. #10834 (Alexey Milovidov).
- Now constraints are updated if the column participating in
CONSTRAINT
expression was renamed. Fixes #10844. #10847 (alesapin). - Fixed bug in
ReplicatedMergeTree
which might cause someALTER
onOPTIMIZE
query to hang waiting for some replica after it become inactive. #10849 (Alexander Tokmakov). - Fixed
WATCH
hangs afterLiveView
table was dropped from database withAtomic
engine. #10859 (Alexander Tokmakov). - Fix SIGSEGV in StringHashTable (if such key does not exist). #10870 (Azat Khuzhin).
- Fix backward compatibility with tuples in Distributed tables. #10889 (Anton Popov).
- Fix possible race which could happen when you get result from aggregate function state from multiple thread for the same column. The only way (which I found) it can happen is when you use
finalizeAggregation
function while reading from table withMemory
engine which storesAggregateFunction
state forquanite*
function. #10890 (Nikolai Kochetov). - Now it's possible to execute multiple
ALTER RENAME
likea TO b, c TO a
. #10895 (alesapin). - Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #10910 (filimonov).
- Fix crash in
SELECT count(notNullIn(NULL, []))
. #10920 (Nikolai Kochetov). - Avoid sending partially written files by the DistributedBlockOutputStream. #10940 (Azat Khuzhin).
- Fix incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 (Alexey Milovidov).
- Fix incorrect raw data size in method getRawData(). #10964 (Igr).
- Fix server crash on concurrent
ALTER
andDROP DATABASE
queries withAtomic
database engine. #10968 (Alexander Tokmakov). - Fix metadata (relative path for rename) and data (relative path for symlink) handling for Atomic database. #10980 (Azat Khuzhin).
- Fix very rare potential use-after-free error in MergeTree if table was not created successfully. #10986 (Alexey Milovidov).
- Fix memory tracking for two-level GROUP BY when not all rows read from Aggregator (TCP). #11022 (Azat Khuzhin).
- Fixed parsing of S3 URLs. #11036 (Vladimir Chebotarev).
- Now it's possible to
ADD/DROP
andRENAME
the same one column in a singleALTER
query. Exception message for simultaneousMODIFY
andRENAME
became more clear. Partially fixes #10669. #11037 (alesapin). - Fixed parseDateTime64BestEffort argument resolution bugs. #10925. #11038 (Vasily Nemkov).
- Fixes the potential missed data during termination of Kafka engine table. #11048 (filimonov).
- Fix error
No such name in Block::erase()
when JOIN appears with PREWHERE oroptimize_move_to_prewhere
makes PREWHERE from WHERE. #11051 (Artem Zuikov). - Fixed memory leak in registerDiskS3. #11074 (Pavel Kovalenko).
- Fixed deadlock during server startup after update with changes in structure of system log tables. #11106 (alesapin).
- Remove logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
- Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 (Azat Khuzhin).
- Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #11145 (filimonov).
- Fix Kafka performance issue related to reschedules based on limits, which were always applied. #11149 (filimonov).
- If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
- Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 (Vladimir Chebotarev).
- Fix possible error
Cannot capture column
for higher-order functions withArray(Array(LowCardinality))
captured argument. #11185 (Nikolai Kochetov). - Now
primary.idx
will be checked if it's defined inCREATE
query. #11199 (alesapin). - Fix possible exception
Invalid status for associated output
. #11200 (Nikolai Kochetov). - Fix error
Block structure mismatch in QueryPipeline
while reading fromVIEW
with constants in inner query. Fixes #11181. #11205 (Nikolai Kochetov). - Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 (Nikita Mikhaylov).
- Fix crash while reading malformed data in Protobuf format. This fixes #5957, fixes #11203. #11258 (Vitaly Baranov).
- Fix wrong markup in documentation. #11263 (Alexey Milovidov).
- Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes #10586. #11278 (Vitaly Baranov).
- Fix bug when query speed estimation can be incorrect and the limit of
min_execution_speed
may not work or work incorrectly if the query is throttled bymax_network_bandwidth
,max_execution_speed
orpriority
settings. Change the default value oftimeout_before_checking_execution_speed
to non-zero, because otherwise the settingsmin_execution_speed
andmax_execution_speed
have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar inclickhouse-client
. #11296 (Alexey Milovidov). - Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with
empty
ornotEmpty
functions. This fixes #11286. #11303 (Alexey Milovidov). - Fix potential uninitialized memory in conversion. Example:
SELECT toIntervalSecond(now64())
. #11311 (Alexey Milovidov). - Fix insignificant data race in clickhouse-copier. Found by integration tests. #11313 (Alexey Milovidov).
- Fix very rare race condition in ThreadPool. #11314 (Alexey Milovidov).
- Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 (Ewout).
- Make writing to
MATERIALIZED VIEW
with settingparallel_view_processing = 1
parallel again. Fixes #10241. #11330 (Nikolai Kochetov). - Now merges stopped before change metadata in
ALTER
queries. #11335 (alesapin). - Fix crash in
quantilesExactWeightedArray
. #11337 (Nikolai Kochetov). - Fix crash in direct selects from StorageJoin (without JOIN) and wrong nullability. #11340 (Artem Zuikov).
- Fix potential segfault when using
Lazy
database. #11348 (Alexey Milovidov). - Fix support for
\G
(vertical output) in clickhouse-client in multiline mode. This closes #9933. #11350 (Alexey Milovidov). - Remove redundant lock during parts send in ReplicatedMergeTree. #11354 (alesapin).
- Fix possible
Pipeline stuck
error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov). - Better errors for
joinGet()
functions. #11389 (Artem Zuikov). - Fixed geohashesInBox with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov).
- Fix error code for wrong
USING
key. #11373. #11404 (Artem Zuikov). - Fix crash in JOIN over LowCarinality(T) and Nullable(T). #11380. #11414 (Artem Zuikov).
- Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. #11420 (Alexey Milovidov).
- Fix pointInPolygon with nan as point. Fixes #11375. #11421 (Alexey Ilyukhov).
- Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
- Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
- Fix wrong result in queries like
select count() from t, u
. #11454 (Artem Zuikov). - Fix
Pipeline stuck
exception forINSERT SELECT FINAL
whereSELECT
(max_threads
>1) has multiple streams butINSERT
has only one (max_insert_threads
==0). #11455 (Azat Khuzhin). - Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. #11496 (Alexey Milovidov).
- Fix race condition which may lead to an exception during table drop. It's a bit tricky and not dangerous at all. If you want an explanation, just notice me in telegram. #11523 (alesapin).
- Fix async INSERT into Distributed for prefer_localhost_replica=0 and w/o internal_replication. #11527 (Azat Khuzhin).
- Fix shard_num/replica_num for
<node>
(breaks use_compact_format_in_distributed_parts_names). #11528 (Azat Khuzhin). - Fix the error
Data compressed with different methods
that can happen ifmin_bytes_to_use_direct_io
is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (Alexey Milovidov). - Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
- All queries in HTTP session have had the same query_id. It is fixed. #11578 (Alexander Tokmakov).
- Fixed rare segfault in
SHOW CREATE TABLE
Fixes #11490. #11579 (Alexander Tokmakov). - Fix error
Size of offsets doesn't match size of column
for queries withPREWHERE column in (subquery)
andARRAY JOIN
. #11580 (Nikolai Kochetov). - Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (Alexey Milovidov).
- Fix race conditions in CREATE/DROP of different replicas of ReplicatedMergeTree. Continue to work if the table was not removed completely from ZooKeeper or not created successfully. This fixes #11432. #11592 (Alexey Milovidov).
- Fix wrong exit code of the clickhouse-client, when exception.code() % 256 = 0. #11601 (filimonov).
- Fix error
Block structure mismatch
for queries with sampling reading fromBuffer
table. #11602 (Nikolai Kochetov). - without -q option the database does not get created at startup. #11604 (giordyb).
- Fix rare crash caused by using
Nullable
column in prewhere condition. (Probably it is connected with #11572 somehow). #11608 (Nikolai Kochetov). - Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
- Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. #11644 (Alexey Milovidov).
- Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (Alexey Milovidov).
- Fix syntax hilite in CREATE USER query. #11664 (Alexey Milovidov).
- Fix error which leads to an incorrect state of
system.mutations
. It may show that whole mutation is already done but the server still hasMUTATE_PART
tasks in the replication queue and tries to execute them. This fixes #11611. #11681 (alesapin). - Fix possible
Pipeline stuck
for selects with parallelFINAL
. Fixes #11636. #11682 (Nikolai Kochetov). - Fix
LIMIT n WITH TIES
usage together withORDER BY
statement, which contains aliases. #11689 (Anton Popov). - Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (Alexey Milovidov).
- Fix very rare race condition in SYSTEM SYNC REPLICA. If the replicated table is created and at the same time from the separate connection another client is issuing
SYSTEM SYNC REPLICA
command on that table (this is unlikely, because another client should be aware that the table is created), it's possible to get nullptr dereference. #11691 (Alexey Milovidov). - Fix
ORDER BY ... WITH FILL
over const columns. #11697 (Anton Popov). - Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (Alexander Tokmakov).
- Parse metadata stored in zookeeper before checking for equality. #11739 (Azat Khuzhin).
- Now replicated fetches will be cancelled during metadata alter. #11744 (alesapin).
- Fixes crash in special generated queries when
optimize_arithmetic_operations_in_aggregate_functions = 1
. #11756 (Ruslan). - Fixed
Scalar doesn't exist
exception when usingWITH <scalar subquery> ...
inSELECT ... FROM merge_tree_table ...
#11621. #11767 (Amos Bird). - Fix using too many threads for queries. #11788 (Nikolai Kochetov).
- Preserve column alias with optimize_aggregators_of_group_by_keys (
optimize_aggregators_of_group_by_keys
has been introduced in #11667). #11806 (Azat Khuzhin). - Fix wrong result of comparison of FixedString with constant String. This fixes #11393. This bug appeared in version 20.4. #11828 (Alexey Milovidov).
- Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (Alexey Milovidov).
Build/Testing/Packaging Improvement
- Add new build for query tests using pytest framework. #10039 (Ivan).
- Fix FreeBSD build. #10150 (Ivan).
- Fix UBSan report in Decimal parse. This fixes #7540. #10512 (Alexey Milovidov).
- Adding fuzzers and preparing for oss-fuzz integration. #10546 (kyprizel).
- Enable ThinLTO for clang builds, continuation of https://github.com/ClickHouse/ClickHouse/pull/10435. #10585 (Amos Bird).
- Increasing timeout when opening a client in tests/queries/0_stateless/helpers/client.py. #10599 (vzakaznikov).
- Fixing hard coded timeouts in new live view tests. #10604 (vzakaznikov).
- Lower memory usage in tests. It may fix the issue that "address sanitizer is out of memory" in stress test. #10617 (Alexey Milovidov).
- Fix
capnproto
version check forcapnp::UnalignedFlatArrayMessageReader
. #10618 (Matwey V. Kornilov). - Added auto-generated machine-readable file with list of stable versions. #10628 (Alexey Milovidov).
- Update LZ4 to the latest dev branch. It may fix the error under UBSan. #10630 (Alexey Milovidov).
- Fix UBSan report in LZ4 library. #10631 (Alexey Milovidov).
- Allow to use lld to link blobs (resources). #10632 (Alexey Milovidov).
- Remove external call to
ld
(bfd) linker during tzdata processing in compile time. #10634 (alesapin). - Fix UBSan report (adding zero to nullptr) in HashTable that appeared after migration to clang-10. #10638 (Alexey Milovidov).
- Fix OOM in ASan stress test. #10646 (Alexey Milovidov).
- Fixing and re-enabling 00979_live_view_watch_continuous_aggregates.py test. #10658 (vzakaznikov).
- Update zstd to 1.4.4. It has some minor improvements in performance and compression ratio. If you run replicas with different versions of ClickHouse you may see reasonable error messages
Data after merge is not byte-identical to data on another replicas.
with explanation. These messages are Ok and you should not worry. #10663 (Alexey Milovidov). - Volumes and storages refactoring. #10666 (Gleb Novikov).
- Trying to fix tests/queries/0_stateless/01246_insert_into_watch_live_view.py test. #10670 (vzakaznikov).
- Update instruction to install RPM packages. This was suggested by Denis (TG login @ldviolet) and implemented by Arkady Shejn. #10707 (Alexey Milovidov).
- Update cross-builds to use clang-10 compiler. #10724 (Ivan).
- Fix performance test errors. #10766 (Alexey Milovidov).
- Fix performance tests errors, part 2. #10773 (Alexey Milovidov).
- Restore a patch that was accidentially deleted in #10396. #10774 (Alexey Milovidov).
- Suppressions of warnings from libraries was mistakenly declared as public in #10396. #10776 (Alexey Milovidov).
- Enable extra warnings for base, utils, programs. #10779 (Alexey Milovidov).
- GRPC make couldn't find protobuf files, changed make file by adding the right link. #10794 (mnkonkova).
- Add MSan suppression for MariaDB Client library. #10800 (Alexey Milovidov).
- Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in #9968. #10801 (Alexey Milovidov).
- Fix obvious race condition in "Split build smoke test" check. #10820 (Alexey Milovidov).
- Better cooperation with sanitizers. Print information about query_id in the message of sanitizer failure. #10832 (Alexey Milovidov).
- Added more asserts in columns code. #10833 (Alexey Milovidov).
- Prepare to add MSan and UBSan stress tests. #10871 (Alexey Milovidov).
- Database is recreated for every test. This improves separation of tests. #10902 (Alexey Milovidov).
- Added a test for empty external data. #10926 (Alexey Milovidov).
- Return tzdata to build images and as dependency to .deb package. #10929 (alesapin).
- Fix non-deterministic test. #10989 (Alexey Milovidov).
- Wait for odbc-bridge with exponential backoff. Previous wait time of 200 ms was not enough in our CI environment. #10990 (Alexey Milovidov).
- Enable clang-tidy for programs and utils. #10991 (Alexey Milovidov).
- Add ability to run zookeeper in integration tests over tmpfs. #11002 (alesapin).
- Fixing 00979_live_view_watch_continuous_aggregates test. #11024 (vzakaznikov).
- Make
system_tables_lazy_load
false by default. #11029 (Azat Khuzhin). - Add performance test for non-constant polygons. #11141 (Alexey Milovidov).
- Create root S3 bucket for tests before any CH instance is started. #11142 (Pavel Kovalenko).
- Enable performance test that was not working. #11158 (Alexey Milovidov).
- Improve build scripts for protobuf & gRPC. #11172 (Vitaly Baranov).
- Split /programs/server into actual program and library. #11186 (Ivan).
- Now parts of linker command for
cctz
library will not be shuffled with other libraries. #11213 (alesapin). - Fix several non significant errors in unit tests. #11262 (alesapin).
- Add a test for Join table engine from @donmikel. This closes #9158. #11265 (Alexey Milovidov).
- Repeat test in CI if
curl
invocation was timed out. It is possible due to system hangups for 10+ seconds that are typical in our CI infrastructure. This fixes #11267. #11268 (Alexey Milovidov). - Fix potentially flacky test
00731_long_merge_tree_select_opened_files.sh
. It does not fail frequently but we have discovered potential race condition in this test while experimenting with ThreadFuzzer: #9814 See link for the example. #11270 (Alexey Milovidov). - Now clickhouse-test check the server aliveness before tests run. #11285 (alesapin).
- Emit a warning if server was build in debug or with sanitizers. #11304 (Alexey Milovidov).
- Better check for hung queries in clickhouse-test. #11321 (Alexey Milovidov).
- Remove redundant timeout from integration test
test_insertion_sync_fails_with_timeout
. #11343 (alesapin). - Add support for unit tests run with UBSan. #11345 (Alexey Milovidov).
- Fix several flaky integration tests. #11355 (alesapin).
- Refactor CMake build files. #11390 (Ivan).
- Leave only unit_tests_dbms in deb build. #11429 (Ilya Yatsishin).
- Increase ccache size for builds in CI. #11450 (alesapin).
- Speed up build by removing old example programs. Also found some orphan functional test. #11486 (Alexey Milovidov).
- Fix missed include for
std::move
used at line 17. #11525 (Matwey V. Kornilov). - Added a random sampling of instances where copier is executed. It is needed to avoid
Too many simultaneous queries
error. Also increased timeout and decreased fault probability. #11573 (Nikita Mikhaylov). - Removes unused imports from HTTPHandlerFactory. #11660 (Bharat Nallan).
- Don't allow tests with "fail" substring in their names because it makes looking at the tests results in browser less convenient when you type Ctrl+F and search for "fail". #11817 (Alexey Milovidov).
- Added a test to ensure that mutations continue to work after FREEZE query. #11820 (Alexey Milovidov).
- Print compiler build id in crash messages. It will make us slightly more certain about what binary has crashed. Added new function
buildId
. #11824 (Alexey Milovidov). - Remove dependency on
tzdata
: do not fail if/usr/share/zoneinfo
directory does not exist. Note that all timezones work in ClickHouse even without tzdata installed in system. #11827 (Alexey Milovidov).
NO CL CATEGORY
-
- Not for changelog. #10985 (Alexey Milovidov).
NO CL ENTRY
- NO CL ENTRY: 'Bump numpy from 1.18.3 to 1.18.4 in /docs/tools'. #10648 (dependabot-preview[bot]).
- NO CL ENTRY: '[ImgBot] Optimize images'. #10796 (imgbot[bot]).
- NO CL ENTRY: 'Bump mkdocs from 1.1 to 1.1.1 in /docs/tools'. #10877 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.6 to 0.4.7 in /docs/tools'. #10878 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump mkdocs from 1.1.1 to 1.1.2 in /docs/tools'. #10938 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump protobuf from 3.11.3 to 3.12.0 in /docs/tools'. #10995 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump beautifulsoup4 from 4.9.0 to 4.9.1 in /docs/tools'. #10996 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump soupsieve from 2.0 to 2.0.1 in /docs/tools'. #10997 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.7 to 0.4.9 in /docs/tools'. #11064 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump protobuf from 3.12.0 to 3.12.1 in /docs/tools'. #11093 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump six from 1.14.0 to 1.15.0 in /docs/tools'. #11129 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump protobuf from 3.12.1 to 3.12.2 in /docs/tools'. #11241 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump numpy from 1.18.4 to 1.18.5 in /docs/tools'. #11427 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump livereload from 2.6.1 to 2.6.2 in /docs/tools'. #11502 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump certifi from 2020.4.5.1 to 2020.4.5.2 in /docs/tools'. #11503 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump googletrans from 2.4.0 to 3.0.0 in /docs/tools'. #11675 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump requests from 2.23.0 to 2.24.0 in /docs/tools'. #11750 (dependabot-preview[bot]).
New Feature: function randomStringUTF8
- Added function randomStringUTF8. #10972 (Andrei Nekrashevich).