mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-13 09:52:38 +00:00
210 KiB
210 KiB
ClickHouse release v20.4
ClickHouse release v20.4.2.9, 2020-05-12
Backward Incompatible Change
- System tables (e.g. system.query_log, system.trace_log, system.metric_log) are using compact data part format for parts smaller than 10 MiB in size. Compact data part format is supported since version 20.3. If you are going to downgrade to version less than 20.3, you should manually delete table data for system logs in
/var/lib/clickhouse/data/system/
. - When string comparison involves FixedString and compared arguments are of different sizes, do comparison as if smaller string is padded to the length of the larger. This is intented for SQL compatibility if we imagine that FixedString data type corresponds to SQL CHAR. This closes #9272. #10363 (alexey-milovidov)
- Make SHOW CREATE TABLE multiline. Now it is more readable and more like MySQL. #10049 (Azat Khuzhin)
- Added a setting
validate_polygons
that is used inpointInPolygon
function and enabled by default. #9857 (alexey-milovidov)
New Feature
- Add support for secured connection from ClickHouse to Zookeeper #10184 (Konstantin Lebedev)
- Support custom HTTP handlers. See ISSUES-5436 for description. #7572 (Winter Zhang)
- Add MessagePack Input/Output format. #9889 (Kruglov Pavel)
- Add Regexp input format. #9196 (Kruglov Pavel)
- Added output format
Markdown
for embedding tables in markdown documents. #10317 (Kruglov Pavel) - Added support for custom settings section in dictionaries. Also fixes issue #2829. #10137 (Artem Streltsov)
- Added custom settings support in DDL-queries for CREATE DICTIONARY #10465 (Artem Streltsov)
- Add simple server-wide memory profiler that will collect allocation contexts when server memory usage becomes higher than the next allocation threshold. #10444 (alexey-milovidov)
- Add setting
always_fetch_merged_part
which restrict replica to merge parts by itself and always prefer dowloading from other replicas. #10379 (alesapin) - Add function JSONExtractKeysAndValuesRaw which extracts raw data from JSON objects #10378 (hcz)
- Add memory usage from OS to
system.asynchronous_metrics
. #10361 (alexey-milovidov) - Added generic variants for functions
least
andgreatest
. Now they work with arbitrary number of arguments of arbitrary types. This fixes #4767 #10318 (alexey-milovidov) - Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration:
strict_max_lifetime_seconds
, which ismax_lifetime
by default, andquery_wait_timeout_milliseconds
, which is one minute by default. The first settings is also useful withallow_read_expired_keys
settings (to forbid reading very expired keys). #10337 (Nikita Mikhaylov) - Add log_queries_min_type to filter which entries will be written to query_log #10053 (Azat Khuzhin)
- Added function
isConstant
. This function checks whether its argument is constant expression and returns 1 or 0. It is intended for development, debugging and demonstration purposes. #10198 (alexey-milovidov) - add joinGetOrNull to return NULL when key is missing instead of returning the default value. #10094 (Amos Bird)
- Consider
NULL
to be equal toNULL
inIN
operator, if the optiontransform_null_in
is set. #10085 (achimbab) - Add
ALTER TABLE ... RENAME COLUMN
for MergeTree table engines family. #9948 (alesapin) - Support parallel distributed INSERT SELECT. #9759 (vxider)
- Add ability to query Distributed over Distributed (w/o
distributed_group_by_no_merge
) ... #9923 (Azat Khuzhin) - Add function
arrayReduceInRanges
which aggregates array elements in given ranges. #9598 (hcz) - Add Dictionary Status on prometheus exporter. #9622 (Guillaume Tassery)
- Add function arrayAUC #8698 (taiyang-li)
- Support
DROP VIEW
statement for better TPC-H compatibility. #9831 (Amos Bird) - Add 'strict_order' option to windowFunnel() #9773 (achimbab)
- Support
DATE
andTIMESTAMP
SQL operators, e.g.SELECT date '2001-01-01'
#9691 (Artem Zuikov)
Experimental Feature
- Added experimental database engine Atomic. It supports non-blocking
DROP
andRENAME TABLE
queries and atomicEXCHANGE TABLES t1 AND t2
query #7512 (tavplubix) - Initial support for ReplicatedMergeTree over S3 (it works in suboptimal way) #10126 (Pavel Kovalenko)
Bug Fix
- Fixed incorrect scalar results inside inner query of
MATERIALIZED VIEW
in case if this query contained dependent table #10603 (Nikolai Kochetov) - Fixed bug, which caused HTTP requests to get stuck on client closing connection when
readonly=2
andcancel_http_readonly_queries_on_client_close=1
. #10684 (tavplubix) - Fix segfault in StorageBuffer when exception is thrown on server startup. Fixes #10550 #10609 (tavplubix)
- The query
SYSTEM DROP DNS CACHE
now also drops caches used to check if user is allowed to connect from some IP addresses #10608 (tavplubix) - Fix usage of multiple
IN
operators with an identical set in one query. Fixes #10539 #10686 (Anton Popov) - Fix crash in
generateRandom
with nested types. Fixes #10583. #10734 (Nikolai Kochetov) - Fix data corruption for
LowCardinality(FixedString)
key column inSummingMergeTree
which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov) - Fix logic for aggregation_memory_efficient_merge_threads setting. #10667 (palasonic1)
- Fix disappearing totals. Totals could have being filtered if query had
JOIN
or subquery with externalWHERE
condition. Fixes #10674 #10698 (Nikolai Kochetov) - Fix the lack of parallel execution of remote queries with
distributed_aggregation_memory_efficient
enabled. Fixes #10655 #10664 (Nikolai Kochetov) - Fix possible incorrect number of rows for queries with
LIMIT
. Fixes #10566, #10709 #10660 (Nikolai Kochetov) - Fix index corruption, which may occur in some cases after merging compact parts into another compact part. #10531 (Anton Popov)
- Fix the situation, when mutation finished all parts, but hung up in
is_done=0
. #10526 (alesapin) - Fix overflow at beginning of unix epoch for timezones with fractional offset from UTC. Fixes #9335. #10513 (alexey-milovidov)
- Better diagnostics for input formats. Fixes #10204 #10418 (tavplubix)
- Fix numeric overflow in
simpleLinearRegression()
over large integers #10474 (hcz) - Fix use-after-free in Distributed shutdown, avoid waiting for sending all batches #10491 (Azat Khuzhin)
- Add CA certificates to clickhouse-server docker image #10476 (filimonov)
- Fix a rare endless loop that might have occurred when using the
addressToLine
function or AggregateFunctionState columns. #10466 (Alexander Kuzmenkov) - Handle zookeeper "no node error" during distributed query #10050 (Daniel Chen)
- Fix bug when server cannot attach table after column's default was altered. #10441 (alesapin)
- Implicitly cast the default expression type to the column type for the ALIAS columns #10563 (Azat Khuzhin)
- Don't remove metadata directory if
ATTACH DATABASE
fails #10442 (Winter Zhang) - Avoid dependency on system tzdata. Fixes loading of
Africa/Casablanca
timezone on CentOS 8. Fixes #10211 #10425 (alexey-milovidov) - Fix some issues if data is inserted with quorum and then gets deleted (DROP PARTITION, TTL, etc.). It led to stuck of INSERTs or false-positive exceptions in SELECTs. Fixes #9946 #10188 (Nikita Mikhaylov)
- Check the number and type of arguments when creating BloomFilter index #9623 #10431 (Winter Zhang)
- Prefer
fallback_to_stale_replicas
overskip_unavailable_shards
, otherwise when both settings specified and there are no up-to-date replicas the query will fail (patch from @alex-zaitsev ) #10422 (Azat Khuzhin) - Fix the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. Fixes #10226. #10427 (Vadim Plakhtinskiy)
- Add database name to dictionary name after DETACH/ATTACH. Fixes system.dictionaries table and
SYSTEM RELOAD
query #10415 (Azat Khuzhin) - Fix possible incorrect result for extremes in processors pipeline. #10131 (Nikolai Kochetov)
- Fix possible segfault when the setting
distributed_group_by_no_merge
is enabled (introduced in 20.3.7.46 by #10131). #10399 (Nikolai Kochetov) - Fix wrong flattening of
Array(Tuple(...))
data types. Fixes #10259 #10390 (alexey-milovidov) - Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN #9950 (Alexander Kuzmenkov)
- Fix order of columns after Block::sortColumns() #10826 (Azat Khuzhin)
- Fix possible
Pipeline stuck
error inConcatProcessor
which may happen in remote query. #10381 (Nikolai Kochetov) - Don't make disk reservations for aggregations. Fixes #9241 #10375 (Azat Khuzhin)
- Fix wrong behaviour of datetime functions for timezones that has altered between positive and negative offsets from UTC (e.g. Pacific/Kiritimati). Fixes #7202 #10369 (alexey-milovidov)
- Avoid infinite loop in
dictIsIn
function. Fixes #515 #10365 (alexey-milovidov) - Disable GROUP BY sharding_key optimization by default and fix it for WITH ROLLUP/CUBE/TOTALS #10516 (Azat Khuzhin)
- Check for error code when checking parts and don't mark part as broken if the error is like "not enough memory". Fixes #6269 #10364 (alexey-milovidov)
- Show information about not loaded dictionaries in system tables. #10234 (Vitaly Baranov)
- Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 (alexey-milovidov)
- Fixed
DROP
vsOPTIMIZE
race inReplicatedMergeTree
.DROP
could left some garbage in replica path in ZooKeeper if there was concurrentOPTIMIZE
query. #10312 (tavplubix) - Fix 'Logical error: CROSS JOIN has expressions' error for queries with comma and names joins mix. Fixes #9910 #10311 (Artem Zuikov)
- Fix queries with
max_bytes_before_external_group_by
. #10302 (Artem Zuikov) - Fix the issue with limiting maximum recursion depth in parser in certain cases. This fixes #10283 This fix may introduce minor incompatibility: long and deep queries via clickhouse-client may refuse to work, and you should adjust settings
max_query_size
andmax_parser_depth
accordingly. #10295 (alexey-milovidov) - Allow to use
count(*)
with multiple JOINs. Fixes #9853 #10291 (Artem Zuikov) - Fix error
Pipeline stuck
withmax_rows_to_group_by
andgroup_by_overflow_mode = 'break'
. #10279 (Nikolai Kochetov) - Fix 'Cannot add column' error while creating
range_hashed
dictionary using DDL query. Fixes #10093. #10235 (alesapin) - Fix rare possible exception
Cannot drain connections: cancel first
. #10239 (Nikolai Kochetov) - Fixed bug where ClickHouse would throw "Unknown function lambda." error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. #10237 (Alexander Kazakov)
- Fixed reasonably rare segfault in StorageSystemTables that happens when SELECT ... FROM system.tables is run on a database with Lazy engine. #10209 (Alexander Kazakov)
- Fix possible infinite query execution when the query actually should stop on LIMIT, while reading from infinite source like
system.numbers
orsystem.zeros
. #10206 (Nikolai Kochetov) - Fixed "generateRandom" function for Date type. This fixes #9973. Fix an edge case when dates with year 2106 are inserted to MergeTree tables with old-style partitioning but partitions are named with year 1970. #10218 (alexey-milovidov)
- Convert types if the table definition of a View does not correspond to the SELECT query. This fixes #10180 and #10022 #10217 (alexey-milovidov)
- Fix
parseDateTimeBestEffort
for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10082 #10214 (alexey-milovidov) - Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. #10207 (alexey-milovidov)
- Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes #10092 #10195 (alexey-milovidov)
- Fix issue with separator appearing in SCRAMBLE for native mysql-connector-java (JDBC) #10140 (BohuTANG)
- Fix using the current database for an access checking when the database isn't specified. #10192 (Vitaly Baranov)
- Fix ALTER of tables with compact parts. #10130 (Anton Popov)
- Add the ability to relax the restriction on non-deterministic functions usage in mutations with
allow_nondeterministic_mutations
setting. #10186 (filimonov) - Fix
DROP TABLE
invoked for dictionary #10165 (Azat Khuzhin) - Convert blocks if structure does not match when doing
INSERT
into Distributed table #10135 (Azat Khuzhin) - The number of rows was logged incorrectly (as sum across all parts) when inserted block is split by parts with partition key. #10138 (alexey-milovidov)
- Add some arguments check and support identifier arguments for MySQL Database Engine #10077 (Winter Zhang)
- Fix incorrect
index_granularity_bytes
check while creating new replica. Fixes #10098. #10121 (alesapin) - Fix bug in
CHECK TABLE
query when table contain skip indices. #10068 (alesapin) - Fix Distributed-over-Distributed with the only one shard in a nested table #9997 (Azat Khuzhin)
- Fix possible rows loss for queries with
JOIN
andUNION ALL
. Fixes #9826, #10113. ... #10099 (Nikolai Kochetov) - Fix bug in dictionary when local clickhouse server is used as source. It may caused memory corruption if types in dictionary and source are not compatible. #10071 (alesapin)
- Fixed replicated tables startup when updating from an old ClickHouse version where
/table/replicas/replica_name/metadata
node doesn't exist. Fixes #10037. #10095 (alesapin) - Fix error
Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform
. It happened when settingdistributed_aggregation_memory_efficient
was enabled, and distributed query read aggregating data with mixed single and two-level aggregation from different shards. #10063 (Nikolai Kochetov) - Fix deadlock when database with materialized view failed attach at start #10054 (Azat Khuzhin)
- Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (#8636, #8925). ... #10025 (Alexander Kuzmenkov)
- Fix wrong results of distributed queries when alias could override qualified column name. Fixes #9672 #9714 #9972 (Artem Zuikov)
- Fix possible deadlock in
SYSTEM RESTART REPLICAS
#9955 (tavplubix) - Fix the number of threads used for remote query execution (performance regression, since 20.3). This happened when query from
Distributed
table was executed simultaneously on local and remote shards. Fixes #9965 #9971 (Nikolai Kochetov) - Fixed
DeleteOnDestroy
logic inATTACH PART
which could lead to automatic removal of attached part and added few tests #9410 (Vladimir Chebotarev) - Fix a bug with
ON CLUSTER
DDL queries freezing on server startup. #9927 (Gagan Arneja) - Fix bug in which the necessary tables weren't retrieved at one of the processing stages of queries to some databases. Fixes #9699. #9949 (achulkov2)
- Fix 'Not found column in block' error when
JOIN
appears withTOTALS
. Fixes #9839 #9939 (Artem Zuikov) - Fix parsing multiple hosts set in the CREATE USER command #9924 (Vitaly Baranov)
- Fix
TRUNCATE
for Join table engine (#9917). #9920 (Amos Bird) - Fix race condition between drop and optimize in
ReplicatedMergeTree
. #9901 (alesapin) - Fix
DISTINCT
for Distributed whenoptimize_skip_unused_shards
is set. #9808 (Azat Khuzhin) - Fix "scalar doesn't exist" error in ALTERs (#9878). ... #9904 (Amos Bird)
- Fix error with qualified names in
distributed_product_mode=\'local\'
. Fixes #4756 #9891 (Artem Zuikov) - For INSERT queries shards now do clamp the settings from the initiator to their constraints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix #9447. #9852 (Vitaly Baranov)
- Add some retries when commiting offsets to Kafka broker, since it can reject commit if during
offsets.commit.timeout.ms
there were no enough replicas available for the__consumer_offsets
topic #9884 (filimonov) - Fix Distributed engine behavior when virtual columns of the underlying table used in
WHERE
#9847 (Azat Khuzhin) - Fixed some cases when timezone of the function argument wasn't used properly. #9574 (Vasily Nemkov)
- Fix 'Different expressions with the same alias' error when query has PREWHERE and WHERE on distributed table and
SET distributed_product_mode = 'local'
. #9871 (Artem Zuikov) - Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. #9860 (alesapin)
- Fix calculating grants for introspection functions from the setting
allow_introspection_functions
. #9840 (Vitaly Baranov) - Fix max_distributed_connections (w/ and w/o Processors) #9673 (Azat Khuzhin)
- Fix possible exception
Got 0 in totals chunk, expected 1
on client. It happened for queries withJOIN
in case if right joined table had zero rows. Example:select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;
. Fixes #9777. ... #9823 (Nikolai Kochetov) - Fix 'COMMA to CROSS JOIN rewriter is not enabled or cannot rewrite query' error in case of subqueries with COMMA JOIN out of tables lists (i.e. in WHERE). Fixes #9782 #9830 (Artem Zuikov)
- Fix server crashing when
optimize_skip_unused_shards
is set and expression for key can't be converted to its field type #9804 (Azat Khuzhin) - Fix empty string handling in
splitByString
. #9767 (hcz) - Fix broken
ALTER TABLE DELETE COLUMN
query for compact parts. #9779 (alesapin) - Fixed missing
rows_before_limit_at_least
for queries over http (with processors pipeline). Fixes #9730 #9757 (Nikolai Kochetov) - Fix excessive memory consumption in
ALTER
queries (mutations). This fixes #9533 and #9670. #9754 (alesapin) - Fix possible permanent "Cannot schedule a task" error. #9154 (Azat Khuzhin)
- Fix bug in backquoting in external dictionaries DDL. Fixes #9619. #9734 (alesapin)
- Fixed data race in
text_log
. It does not correspond to any real bug. #9726 (alexey-milovidov) - Fix bug in a replication that doesn't allow replication to work if the user has executed mutations on the previous version. This fixes #9645. #9652 (alesapin)
- Fixed incorrect internal function names for
sumKahan
andsumWithOverflow
. It led to exception while using this functions in remote queries. #9636 (Azat Khuzhin) - Add setting
use_compact_format_in_distributed_parts_names
which allows to write files forINSERT
queries intoDistributed
table with more compact format. This fixes #9647. #9653 (alesapin) - Fix RIGHT and FULL JOIN with LowCardinality in JOIN keys. #9610 (Artem Zuikov)
- Fix possible exceptions
Size of filter doesn't match size of column
andInvalid number of rows in Chunk
inMergeTreeRangeReader
. They could appear while executingPREWHERE
in some cases. #9612 (Anton Popov) - Allow
ALTER ON CLUSTER
of Distributed tables with internal replication. This fixes #3268 #9617 (shinoi2) - Fix issue when timezone was not preserved if you write a simple arithmetic expression like
time + 1
(in contrast to an expression liketime + INTERVAL 1 SECOND
). This fixes #5743 #9323 (alexey-milovidov)
Improvement
- Use time zone when comparing DateTime with string literal. This fixes #5206. #10515 (alexey-milovidov)
- Print verbose diagnostic info if Decimal value cannot be parsed from text input format. #10205 (alexey-milovidov)
- Add tasks/memory metrics for distributed/buffer schedule pools #10449 (Azat Khuzhin)
- Display result as soon as it's ready for SELECT DISTINCT queries in clickhouse-local and HTTP interface. This fixes #8951 #9559 (alexey-milovidov)
- Allow to use
SAMPLE OFFSET
query instead ofcityHash64(PRIMARY KEY) % N == n
for splitting inclickhouse-copier
. To use this feature, pass--experimental-use-sample-offset 1
as a command line argument. #10414 (Nikita Mikhaylov) - Allow to parse BOM in TSV if the first column cannot contain BOM in its value. This fixes #10301 #10424 (alexey-milovidov)
- Add Avro nested fields insert support #10354 (Andrew Onyshchuk)
- Allowed to alter column in non-modifying data mode when the same type is specified. #10382 (Vladimir Chebotarev)
- Auto
distributed_group_by_no_merge
on GROUP BY sharding key (ifoptimize_skip_unused_shards
is set) #10341 (Azat Khuzhin) - Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key #10373 (Azat Khuzhin)
- Added a setting
max_server_memory_usage
to limit total memory usage of the server. The metricMemoryTracking
is now calculated without a drift. The settingmax_memory_usage_for_all_queries
is now obsolete and does nothing. This closes #10293. #10362 (alexey-milovidov) - Add config option
system_tables_lazy_load
. If it's set to false, then system tables with logs are loaded at the server startup. Alexander Burmak, Svyatoslav Tkhon Il Pak, #9642 #10359 (alexey-milovidov) - Use background thread pool (background_schedule_pool_size) for distributed sends #10263 (Azat Khuzhin)
- Use background thread pool for background buffer flushes. #10315 (Azat Khuzhin)
- Support for one special case of removing incompletely written parts. This fixes #9940. #10221 (alexey-milovidov)
- Use isInjective() over manual list of such functions for GROUP BY optimization. #10342 (Azat Khuzhin)
- Avoid printing error message in log if client sends RST packet immediately on connect. It is typical behaviour of IPVS balancer with keepalived and VRRP. This fixes #1851 #10274 (alexey-milovidov)
- Allow to parse
+inf
for floating point types. This closes #1839 #10272 (alexey-milovidov) - Implemented
generateRandom
table function for Nested types. This closes #9903 #10219 (alexey-milovidov) - Provide
max_allowed_packed
in MySQL compatibility interface that will help some clients to communicate with ClickHouse via MySQL protocol. #10199 (BohuTANG) - Allow literals for GLOBAL IN (i.e.
SELECT * FROM remote('localhost', system.one) WHERE dummy global in (0)
) #10196 (Azat Khuzhin) - Fix various small issues in interactive mode of clickhouse-client #10194 (alexey-milovidov)
- Avoid superfluous dictionaries load (system.tables, DROP/SHOW CREATE TABLE) #10164 (Azat Khuzhin)
- Update to RWLock: timeout parameter for getLock() + implementation reworked to be phase fair #10073 (Alexander Kazakov)
- Enhanced compatibility with native mysql-connector-java(JDBC) #10021 (BohuTANG)
- The function
toString
is considered monotonic and can be used for index analysis even when applied in tautological cases with String or LowCardinality(String) argument. #10110 (Amos Bird) - Add
ON CLUSTER
clause support to commands{CREATE|DROP} USER/ROLE/ROW POLICY/SETTINGS PROFILE/QUOTA
,GRANT
. #9811 (Vitaly Baranov) - Virtual hosted-style support for S3 URI #9998 (Pavel Kovalenko)
- Now layout type for dictionaries with no arguments can be specified without round brackets in dictionaries DDL-queries. Fixes #10057. #10064 (alesapin)
- Add ability to use number ranges with leading zeros in filepath #9989 (Olga Khvostikova)
- Better memory usage in CROSS JOIN. #10029 (Artem Zuikov)
- Try to connect to all shards in cluster when getting structure of remote table and skip_unavailable_shards is set. #7278 (nvartolomei)
- Add
total_rows
/total_bytes
into thesystem.tables
table. #9919 (Azat Khuzhin) - System log tables now use polymorpic parts by default. #9905 (Anton Popov)
- Add type column into system.settings/merge_tree_settings #9909 (Azat Khuzhin)
- Check for available CPU instructions at server startup as early as possible. #9888 (alexey-milovidov)
- Remove
ORDER BY
stage from mutations because we read from a single ordered part in a single thread. Also add check that the rows in mutation are ordered by sorting key and this order is not violated. #9886 (alesapin) - Implement operator LIKE for FixedString at left hand side. This is needed to better support TPC-DS queries. #9890 (alexey-milovidov)
- Add
force_optimize_skip_unused_shards_no_nested
that will disableforce_optimize_skip_unused_shards
for nested Distributed table #9812 (Azat Khuzhin) - Now columns size is calculated only once for MergeTree data parts. #9827 (alesapin)
- Evaluate constant expressions for
optimize_skip_unused_shards
(i.e.SELECT * FROM foo_dist WHERE key=xxHash32(0)
) #8846 (Azat Khuzhin) - Check for using
Date
orDateTime
column from TTL expressions was removed. #9967 (Vladimir Chebotarev) - DiskS3 hard links optimal implementation. #9760 (Pavel Kovalenko)
- If
set multiple_joins_rewriter_version = 2
enables second version of multiple JOIN rewrites that keeps not clashed column names as is. It supports multiple JOINs withUSING
and allowselect *
for JOINs with subqueries. #9739 (Artem Zuikov) - Implementation of "non-blocking" alter for StorageMergeTree #9606 (alesapin)
- Add MergeTree full support for DiskS3 #9646 (Pavel Kovalenko)
- Extend
splitByString
to support empty strings as separators. #9742 (hcz) - Add a
timestamp_ns
column tosystem.trace_log
. It contains a high-definition timestamp of the trace event, and allows to build timelines of thread profiles ("flame charts"). #9696 (Alexander Kuzmenkov) - When the setting
send_logs_level
is enabled, avoid intermixing of log messages and query progress. #9634 (Azat Khuzhin) - Added support of
MATERIALIZE TTL IN PARTITION
. #9581 (Vladimir Chebotarev) - Support complex types inside Avro nested fields #10502 (Andrew Onyshchuk)
Performance Improvement
- Better insert logic for right table for Partial MergeJoin. #10467 (Artem Zuikov)
- Improved performance of row-oriented formats (more than 10% for CSV and more than 35% for Avro in case of narrow tables). #10503 (Andrew Onyshchuk)
- Improved performance of queries with explicitly defined sets at right side of IN operator and tuples on the left side. #10385 (Anton Popov)
- Use less memory for hash table in HashJoin. #10416 (Artem Zuikov)
- Special HashJoin over StorageDictionary. Allow rewrite
dictGet()
functions with JOINs. It's not backward incompatible itself but could uncover #8400 on some installations. #10133 (Artem Zuikov) - Enable parallel insert of materialized view when its target table supports. #10052 (vxider)
- Improved performance of index analysis with monotonic functions. #9607#10026 (Anton Popov)
- Using SSE2 or SSE4.2 SIMD intrinsics to speed up tokenization in bloom filters. #9968 (Vasily Nemkov)
- Improved performance of queries with explicitly defined sets at right side of
IN
operator. This fixes performance regression in version 20.3. #9740 (Anton Popov) - Now clickhouse-copier splits each partition in number of pieces and copies them independently. #9075 (Nikita Mikhaylov)
- Adding more aggregation methods. For example TPC-H query 1 will now pick
FixedHashMap<UInt16, AggregateDataPtr>
and gets 25% performance gain #9829 (Amos Bird) - Use single row counter for multiple streams in pre-limit transform. This helps to avoid uniting pipeline streams in queries with
limit
but withoutorder by
(likeselect f(x) from (select x from t limit 1000000000)
) and use multiple threads for further processing. #9602 (Nikolai Kochetov)
Build/Testing/Packaging Improvement
- Use a fork of AWS SDK libraries from ClickHouse-Extras #10527 (Pavel Kovalenko)
- Add integration tests for new ALTER RENAME COLUMN query. #10654 (vzakaznikov)
- Fix possible signed integer overflow in invocation of function
now64
with wrong arguments. This fixes #8973 #10511 (alexey-milovidov) - Split fuzzer and sanitizer configurations to make build config compatible with Oss-fuzz. #10494 (kyprizel)
- Fixes for clang-tidy on clang-10. #10420 (alexey-milovidov)
- Display absolute paths in error messages. Otherwise KDevelop fails to navigate to correct file and opens a new file instead. #10434 (alexey-milovidov)
- Added
ASAN_OPTIONS
environment variable to investigate errors in CI stress tests with Address sanitizer. #10440 (Nikita Mikhaylov) - Enable ThinLTO for clang builds (experimental). #10435 (alexey-milovidov)
- Remove accidential dependency on Z3 that may be introduced if the system has Z3 solver installed. #10426 (alexey-milovidov)
- Move integration tests docker files to docker/ directory. #10335 (Ilya Yatsishin)
- Allow to use
clang-10
in CI. It ensures that #10238 is fixed. #10384 (alexey-milovidov) - Update OpenSSL to upstream master. Fixed the issue when TLS connections may fail with the message
OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
andSSL Exception: error:2400006E:random number generator::error retrieving entropy
. The issue was present in version 20.1. #8956 (alexey-milovidov) - Fix clang-10 build. https://github.com/ClickHouse/ClickHouse/issues/10238 #10370 (Amos Bird)
- Add performance test for Parallel INSERT for materialized view. #10345 (vxider)
- Fix flaky test
test_settings_constraints_distributed.test_insert_clamps_settings
. #10346 (Vitaly Baranov) - Add util to test results upload in CI ClickHouse #10330 (Ilya Yatsishin)
- Convert test results to JSONEachRow format in junit_to_html tool #10323 (Ilya Yatsishin)
- Update cctz. #10215 (alexey-milovidov)
- Allow to create HTML report from the purest JUnit XML report. #10247 (Ilya Yatsishin)
- Update the check for minimal compiler version. Fix the root cause of the issue #10250 #10256 (alexey-milovidov)
- Initial support for live view tables over distributed #10179 (vzakaznikov)
- Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in #9968. #10801 (alexey-milovidov)
- clickhouse-docker-util #10151 (filimonov)
- Update pdqsort to recent version #10171 (Ivan)
- Update libdivide to v3.0 #10169 (Ivan)
- Add check with enabled polymorphic parts. #10086 (Anton Popov)
- Add cross-compile build for FreeBSD. This fixes #9465 #9643 (Ivan)
- Add performance test for #6924 #6980 (filimonov)
- Add support of
/dev/null
in theFile
engine for better performance testing #8455 (Amos Bird) - Move all folders inside /dbms one level up #9974 (Ivan)
- Add a test that checks that read from MergeTree with single thread is performed in order. Addition to #9670 #9762 (alexey-milovidov)
- Fix the
00964_live_view_watch_events_heartbeat.py
test to avoid race condition. #9944 (vzakaznikov) - Fix integration test
test_settings_constraints
#9962 (Vitaly Baranov) - Every function in its own file, part 12. #9922 (alexey-milovidov)
- Added performance test for the case of extremely slow analysis of array of tuples. #9872 (alexey-milovidov)
- Update zstd to 1.4.4. It has some minor improvements in performance and compression ratio. If you run replicas with different versions of ClickHouse you may see reasonable error messages
Data after merge is not byte-identical to data on another replicas.
with explanation. These messages are Ok and you should not worry. #10663 (alexey-milovidov) - Fix TSan report in
system.stack_trace
. #9832 (alexey-milovidov) - Removed dependency on
clock_getres
. #9833 (alexey-milovidov) - Added identifier names check with clang-tidy. #9799 (alexey-milovidov)
- Update "builder" docker image. This image is not used in CI but is useful for developers. #9809 (alexey-milovidov)
- Remove old
performance-test
tool that is no longer used in CI.clickhouse-performance-test
is great but now we are using way superior tool that is doing comparison testing with sophisticated statistical formulas to achieve confident results regardless to various changes in environment. #9796 (alexey-milovidov) - Added most of clang-static-analyzer checks. #9765 (alexey-milovidov)
- Update Poco to 1.9.3 in preparation for MongoDB URI support. #6892 (Alexander Kuzmenkov)
- Fix build with
-DUSE_STATIC_LIBRARIES=0 -DENABLE_JEMALLOC=0
#9651 (Artem Zuikov) - For change log script, if merge commit was cherry-picked to release branch, take PR name from commit description. #9708 (Nikolai Kochetov)
- Support
vX.X-conflicts
tag in backport script. #9705 (Nikolai Kochetov) - Fix
auto-label
for backporting script. #9685 (Nikolai Kochetov) - Use libc++ in Darwin cross-build to make it consistent with native build. #9665 (Hui Wang)
- Fix flacky test
01017_uniqCombined_memory_usage
. Continuation of #7236. #9667 (alexey-milovidov) - Fix build for native MacOS Clang compiler #9649 (Ivan)
- Allow to add various glitches around
pthread_mutex_lock
,pthread_mutex_unlock
functions. #9635 (alexey-milovidov) - Add support for
clang-tidy
inpackager
script. #9625 (alexey-milovidov) - Add ability to use unbundled msgpack. #10168 (Azat Khuzhin)
ClickHouse release v20.3
ClickHouse release v20.3.8.53, 2020-04-23
Bug Fix
- Fixed wrong behaviour of datetime functions for timezones that has altered between positive and negative offsets from UTC (e.g. Pacific/Kiritimati). This fixes #7202 #10369 (alexey-milovidov)
- Fix possible segfault with
distributed_group_by_no_merge
enabled (introduced in 20.3.7.46 by #10131). #10399 (Nikolai Kochetov) - Fix wrong flattening of
Array(Tuple(...))
data types. This fixes #10259 #10390 (alexey-milovidov) - Drop disks reservation in Aggregator. This fixes bug in disk space reservation, which may cause big external aggregation to fail even if it could be completed successfully #10375 (Azat Khuzhin)
- Fixed
DROP
vsOPTIMIZE
race inReplicatedMergeTree
.DROP
could left some garbage in replica path in ZooKeeper if there was concurrentOPTIMIZE
query. #10312 (tavplubix) - Fix bug when server cannot attach table after column default was altered. #10441 (alesapin)
- Do not remove metadata directory when attach database fails before loading tables. #10442 (Winter Zhang)
- Fixed several bugs when some data was inserted with quorum, then deleted somehow (DROP PARTITION, TTL) and this leaded to the stuck of INSERTs or false-positive exceptions in SELECTs. This fixes #9946 #10188 (Nikita Mikhaylov)
- Fix possible
Pipeline stuck
error inConcatProcessor
which could have happened in remote query. #10381 (Nikolai Kochetov) - Fixed wrong behavior in HashTable that caused compilation error when trying to read HashMap from buffer. #10386 (palasonic1)
- Allow to use
count(*)
with multiple JOINs. Fixes #9853 #10291 (Artem Zuikov) - Prefer
fallback_to_stale_replicas
overskip_unavailable_shards
, otherwise when both settings specified and there are no up-to-date replicas the query will fail (patch from @alex-zaitsev). Fixes: #2564. #10422 (Azat Khuzhin) - Fix the issue when a query with ARRAY JOIN, ORDER BY and LIMIT may return incomplete result. This fixes #10226. Author: Vadim Plakhtinskiy. #10427 (alexey-milovidov)
- Check the number and type of arguments when creating BloomFilter index #9623 #10431 (Winter Zhang)
Performance Improvement
- Improved performance of queries with explicitly defined sets at right side of
IN
operator and tuples in the left side. This fixes performance regression in version 20.3. #9740, #10385 (Anton Popov)
ClickHouse release v20.3.7.46, 2020-04-17
Bug Fix
- Fix
Logical error: CROSS JOIN has expressions
error for queries with comma and names joins mix. #10311 (Artem Zuikov). - Fix queries with
max_bytes_before_external_group_by
. #10302 (Artem Zuikov). - Fix move-to-prewhere optimization in presense of arrayJoin functions (in certain cases). This fixes #10092. #10195 (alexey-milovidov).
- Add the ability to relax the restriction on non-deterministic functions usage in mutations with
allow_nondeterministic_mutations
setting. #10186 (filimonov).
ClickHouse release v20.3.6.40, 2020-04-16
New Feature
- Added function
isConstant
. This function checks whether its argument is constant expression and returns 1 or 0. It is intended for development, debugging and demonstration purposes. #10198 (alexey-milovidov).
Bug Fix
- Fix error
Pipeline stuck
withmax_rows_to_group_by
andgroup_by_overflow_mode = 'break'
. #10279 (Nikolai Kochetov). - Fix rare possible exception
Cannot drain connections: cancel first
. #10239 (Nikolai Kochetov). - Fixed bug where ClickHouse would throw "Unknown function lambda." error message when user tries to run ALTER UPDATE/DELETE on tables with ENGINE = Replicated*. Check for nondeterministic functions now handles lambda expressions correctly. #10237 (Alexander Kazakov).
- Fixed "generateRandom" function for Date type. This fixes #9973. Fix an edge case when dates with year 2106 are inserted to MergeTree tables with old-style partitioning but partitions are named with year 1970. #10218 (alexey-milovidov).
- Convert types if the table definition of a View does not correspond to the SELECT query. This fixes #10180 and #10022. #10217 (alexey-milovidov).
- Fix
parseDateTimeBestEffort
for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10082. #10214 (alexey-milovidov). - Fix column names of constants inside JOIN that may clash with names of constants outside of JOIN. #10207 (alexey-milovidov).
- Fix possible inifinite query execution when the query actually should stop on LIMIT, while reading from infinite source like
system.numbers
orsystem.zeros
. #10206 (Nikolai Kochetov). - Fix using the current database for access checking when the database isn't specified. #10192 (Vitaly Baranov).
- Convert blocks if structure does not match on INSERT into Distributed(). #10135 (Azat Khuzhin).
- Fix possible incorrect result for extremes in processors pipeline. #10131 (Nikolai Kochetov).
- Fix some kinds of alters with compact parts. #10130 (Anton Popov).
- Fix incorrect
index_granularity_bytes
check while creating new replica. Fixes #10098. #10121 (alesapin). - Fix SIGSEGV on INSERT into Distributed table when its structure differs from the underlying tables. #10105 (Azat Khuzhin).
- Fix possible rows loss for queries with
JOIN
andUNION ALL
. Fixes #9826, #10113. #10099 (Nikolai Kochetov). - Fixed replicated tables startup when updating from an old ClickHouse version where
/table/replicas/replica_name/metadata
node doesn't exist. Fixes #10037. #10095 (alesapin). - Add some arguments check and support identifier arguments for MySQL Database Engine. #10077 (Winter Zhang).
- Fix bug in clickhouse dictionary source from localhost clickhouse server. The bug may lead to memory corruption if types in dictionary and source are not compatible. #10071 (alesapin).
- Fix bug in
CHECK TABLE
query when table contain skip indices. #10068 (alesapin). - Fix error
Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform
. It happened when settingdistributed_aggregation_memory_efficient
was enabled, and distributed query read aggregating data with different level from different shards (mixed single and two level aggregation). #10063 (Nikolai Kochetov). - Fix a segmentation fault that could occur in GROUP BY over string keys containing trailing zero bytes (#8636, #8925). #10025 (Alexander Kuzmenkov).
- Fix the number of threads used for remote query execution (performance regression, since 20.3). This happened when query from
Distributed
table was executed simultaneously on local and remote shards. Fixes #9965. #9971 (Nikolai Kochetov). - Fix bug in which the necessary tables weren't retrieved at one of the processing stages of queries to some databases. Fixes #9699. #9949 (achulkov2).
- Fix 'Not found column in block' error when
JOIN
appears withTOTALS
. Fixes #9839. #9939 (Artem Zuikov). - Fix a bug with
ON CLUSTER
DDL queries freezing on server startup. #9927 (Gagan Arneja). - Fix parsing multiple hosts set in the CREATE USER command, e.g.
CREATE USER user6 HOST NAME REGEXP 'lo.?*host', NAME REGEXP 'lo*host'
. #9924 (Vitaly Baranov). - Fix
TRUNCATE
for Join table engine (#9917). #9920 (Amos Bird). - Fix "scalar doesn't exist" error in ALTERs (#9878). #9904 (Amos Bird).
- Fix race condition between drop and optimize in
ReplicatedMergeTree
. #9901 (alesapin). - Fix error with qualified names in
distributed_product_mode='local'
. Fixes #4756. #9891 (Artem Zuikov). - Fix calculating grants for introspection functions from the setting 'allow_introspection_functions'. #9840 (Vitaly Baranov).
Build/Testing/Packaging Improvement
- Fix integration test
test_settings_constraints
. #9962 (Vitaly Baranov). - Removed dependency on
clock_getres
. #9833 (alexey-milovidov).
ClickHouse release v20.3.5.21, 2020-03-27
Bug Fix
- Fix 'Different expressions with the same alias' error when query has PREWHERE and WHERE on distributed table and
SET distributed_product_mode = 'local'
. #9871 (Artem Zuikov). - Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. #9860 (alesapin).
- For INSERT queries shard now clamps the settings got from the initiator to the shard's constaints instead of throwing an exception. This fix allows to send INSERT queries to a shard with another constraints. This change improves fix #9447. #9852 (Vitaly Baranov).
- Fix 'COMMA to CROSS JOIN rewriter is not enabled or cannot rewrite query' error in case of subqueries with COMMA JOIN out of tables lists (i.e. in WHERE). Fixes #9782. #9830 (Artem Zuikov).
- Fix possible exception
Got 0 in totals chunk, expected 1
on client. It happened for queries withJOIN
in case if right joined table had zero rows. Example:select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;
. Fixes #9777. #9823 (Nikolai Kochetov). - Fix SIGSEGV with optimize_skip_unused_shards when type cannot be converted. #9804 (Azat Khuzhin).
- Fix broken
ALTER TABLE DELETE COLUMN
query for compact parts. #9779 (alesapin). - Fix max_distributed_connections (w/ and w/o Processors). #9673 (Azat Khuzhin).
- Fixed a few cases when timezone of the function argument wasn't used properly. #9574 (Vasily Nemkov).
Improvement
- Remove order by stage from mutations because we read from a single ordered part in a single thread. Also add check that the order of rows in mutation is ordered in sorting key order and this order is not violated. #9886 (alesapin).
ClickHouse release v20.3.4.10, 2020-03-20
Bug Fix
- This release also contains all bug fixes from 20.1.8.41
- Fix missing
rows_before_limit_at_least
for queries over http (with processors pipeline). This fixes #9730. #9757 (Nikolai Kochetov)
ClickHouse release v20.3.3.6, 2020-03-17
Bug Fix
- This release also contains all bug fixes from 20.1.7.38
- Fix bug in a replication that doesn't allow replication to work if the user has executed mutations on the previous version. This fixes #9645. #9652 (alesapin). It makes version 20.3 backward compatible again.
- Add setting
use_compact_format_in_distributed_parts_names
which allows to write files forINSERT
queries intoDistributed
table with more compact format. This fixes #9647. #9653 (alesapin). It makes version 20.3 backward compatible again.
ClickHouse release v20.3.2.1, 2020-03-12
Backward Incompatible Change
- Fixed the issue
file name too long
when sending data forDistributed
tables for a large number of replicas. Fixed the issue that replica credentials were exposed in the server log. The format of directory name on disk was changed to[shard{shard_index}[_replica{replica_index}]]
. #8911 (Mikhail Korotov) After you upgrade to the new version, you will not be able to downgrade without manual intervention, because old server version does not recognize the new directory format. If you want to downgrade, you have to manually rename the corresponding directories to the old format. This change is relevant only if you have used asynchronousINSERT
s toDistributed
tables. In the version 20.3.3 we will introduce a setting that will allow you to enable the new format gradually. - Changed the format of replication log entries for mutation commands. You have to wait for old mutations to process before installing the new version.
- Implement simple memory profiler that dumps stacktraces to
system.trace_log
every N bytes over soft allocation limit #8765 (Ivan) #9472 (alexey-milovidov) The column ofsystem.trace_log
was renamed fromtimer_type
totrace_type
. This will require changes in third-party performance analysis and flamegraph processing tools. - Use OS thread id everywhere instead of internal thread number. This fixes #7477 Old
clickhouse-client
cannot receive logs that are send from the server when the settingsend_logs_level
is enabled, because the names and types of the structured log messages were changed. On the other hand, different server versions can send logs with different types to each other. When you don't use thesend_logs_level
setting, you should not care. #8954 (alexey-milovidov) - Remove
indexHint
function #9542 (alexey-milovidov) - Remove
findClusterIndex
,findClusterValue
functions. This fixes #8641. If you were using these functions, send an email toclickhouse-feedback@yandex-team.com
#9543 (alexey-milovidov) - Now it's not allowed to create columns or add columns with
SELECT
subquery as default expression. #9481 (alesapin) - Require aliases for subqueries in JOIN. #9274 (Artem Zuikov)
- Improved
ALTER MODIFY/ADD
queries logic. Now you cannotADD
column without type,MODIFY
default expression doesn't change type of column andMODIFY
type doesn't loose default expression value. Fixes #8669. #9227 (alesapin) - Require server to be restarted to apply the changes in logging configuration. This is a temporary workaround to avoid the bug where the server logs to a deleted log file (see #8696). #8707 (Alexander Kuzmenkov)
- The setting
experimental_use_processors
is enabled by default. This setting enables usage of the new query pipeline. This is internal refactoring and we expect no visible changes. If you will see any issues, set it to back zero. #8768 (alexey-milovidov)
New Feature
- Add
Avro
andAvroConfluent
input/output formats #8571 (Andrew Onyshchuk) #8957 (Andrew Onyshchuk) #8717 (alexey-milovidov) - Multi-threaded and non-blocking updates of expired keys in
cache
dictionaries (with optional permission to read old ones). #8303 (Nikita Mikhaylov) - Add query
ALTER ... MATERIALIZE TTL
. It runs mutation that forces to remove expired data by TTL and recalculates meta-information about TTL in all parts. #8775 (Anton Popov) - Switch from HashJoin to MergeJoin (on disk) if needed #9082 (Artem Zuikov)
- Added
MOVE PARTITION
command forALTER TABLE
#4729 #6168 (Guillaume Tassery) - Reloading storage configuration from configuration file on the fly. #8594 (Vladimir Chebotarev)
- Allowed to change
storage_policy
to not less rich one. #8107 (Vladimir Chebotarev) - Added support for globs/wildcards for S3 storage and table function. #8851 (Vladimir Chebotarev)
- Implement
bitAnd
,bitOr
,bitXor
,bitNot
forFixedString(N)
datatype. #9091 (Guillaume Tassery) - Added function
bitCount
. This fixes #8702. #8708 (alexey-milovidov) #8749 (ikopylov) - Add
generateRandom
table function to generate random rows with given schema. Allows to populate arbitrary test table with data. #8994 (Ilya Yatsishin) JSONEachRowFormat
: support special case when objects enclosed in top-level array. #8860 (Kruglov Pavel)- Now it's possible to create a column with
DEFAULT
expression which depends on a column with defaultALIAS
expression. #9489 (alesapin) - Allow to specify
--limit
more than the source data size inclickhouse-obfuscator
. The data will repeat itself with different random seed. #9155 (alexey-milovidov) - Added
groupArraySample
function (similar togroupArray
) with reservior sampling algorithm. #8286 (Amos Bird) - Now you can monitor the size of update queue in
cache
/complex_key_cache
dictionaries via system metrics. #9413 (Nikita Mikhaylov) - Allow to use CRLF as a line separator in CSV output format with setting
output_format_csv_crlf_end_of_line
is set to 1 #8934 #8935 #8963 (Mikhail Korotov) - Implement more functions of the H3 API:
h3GetBaseCell
,h3HexAreaM2
,h3IndexesAreNeighbors
,h3ToChildren
,h3ToString
andstringToH3
#8938 (Nico Mandery) - New setting introduced:
max_parser_depth
to control maximum stack size and allow large complex queries. This fixes #6681 and #7668. #8647 (Maxim Smirnov) - Add a setting
force_optimize_skip_unused_shards
setting to throw if skipping of unused shards is not possible #8805 (Azat Khuzhin) - Allow to configure multiple disks/volumes for storing data for send in
Distributed
engine #8756 (Azat Khuzhin) - Support storage policy (
<tmp_policy>
) for storing temporary data. #8750 (Azat Khuzhin) - Added
X-ClickHouse-Exception-Code
HTTP header that is set if exception was thrown before sending data. This implements #4971. #8786 (Mikhail Korotov) - Added function
ifNotFinite
. It is just a syntactic sugar:ifNotFinite(x, y) = isFinite(x) ? x : y
. #8710 (alexey-milovidov) - Added
last_successful_update_time
column insystem.dictionaries
table #9394 (Nikita Mikhaylov) - Add
blockSerializedSize
function (size on disk without compression) #8952 (Azat Khuzhin) - Add function
moduloOrZero
#9358 (hcz) - Added system tables
system.zeros
andsystem.zeros_mt
as well as tale functionszeros()
andzeros_mt()
. Tables (and table functions) contain single column with namezero
and typeUInt8
. This column contains zeros. It is needed for test purposes as the fastest method to generate many rows. This fixes #6604 #9593 (Nikolai Kochetov)
Experimental Feature
- Add new compact format of parts in
MergeTree
-family tables in which all columns are stored in one file. It helps to increase performance of small and frequent inserts. The old format (one file per column) is now called wide. Data storing format is controlled by settingsmin_bytes_for_wide_part
andmin_rows_for_wide_part
. #8290 (Anton Popov) - Support for S3 storage for
Log
,TinyLog
andStripeLog
tables. #8862 (Pavel Kovalenko)
Bug Fix
- Fixed inconsistent whitespaces in log messages. #9322 (alexey-milovidov)
- Fix bug in which arrays of unnamed tuples were flattened as Nested structures on table creation. #8866 (achulkov2)
- Fixed the issue when "Too many open files" error may happen if there are too many files matching glob pattern in
File
table orfile
table function. Now files are opened lazily. This fixes #8857 #8861 (alexey-milovidov) - DROP TEMPORARY TABLE now drops only temporary table. #8907 (Vitaly Baranov)
- Remove outdated partition when we shutdown the server or DETACH/ATTACH a table. #8602 (Guillaume Tassery)
- For how the default disk calculates the free space from
data
subdirectory. Fixed the issue when the amount of free space is not calculated correctly if thedata
directory is mounted to a separate device (rare case). This fixes #7441 #9257 (Mikhail Korotov) - Allow comma (cross) join with IN () inside. #9251 (Artem Zuikov)
- Allow to rewrite CROSS to INNER JOIN if there's [NOT] LIKE operator in WHERE section. #9229 (Artem Zuikov)
- Fix possible incorrect result after
GROUP BY
with enabled settingdistributed_aggregation_memory_efficient
. Fixes #9134. #9289 (Nikolai Kochetov) - Found keys were counted as missed in metrics of cache dictionaries. #9411 (Nikita Mikhaylov)
- Fix replication protocol incompatibility introduced in #8598. #9412 (alesapin)
- Fixed race condition on
queue_task_handle
at the startup ofReplicatedMergeTree
tables. #9552 (alexey-milovidov) - The token
NOT
didn't work inSHOW TABLES NOT LIKE
query #8727 #8940 (alexey-milovidov) - Added range check to function
h3EdgeLengthM
. Without this check, buffer overflow is possible. #8945 (alexey-milovidov) - Fixed up a bug in batched calculations of ternary logical OPs on multiple arguments (more than 10). #8718 (Alexander Kazakov)
- Fix error of PREWHERE optimization, which could lead to segfaults or
Inconsistent number of columns got from MergeTreeRangeReader
exception. #9024 (Anton Popov) - Fix unexpected
Timeout exceeded while reading from socket
exception, which randomly happens on secure connection before timeout actually exceeded and when query profiler is enabled. Also addconnect_timeout_with_failover_secure_ms
settings (default 100ms), which is similar toconnect_timeout_with_failover_ms
, but is used for secure connections (because SSL handshake is slower, than ordinary TCP connection) #9026 (tavplubix) - Fix bug with mutations finalization, when mutation may hang in state with
parts_to_do=0
andis_done=0
. #9022 (alesapin) - Use new ANY JOIN logic with
partial_merge_join
setting. It's possible to makeANY|ALL|SEMI LEFT
andALL INNER
joins withpartial_merge_join=1
now. #8932 (Artem Zuikov) - Shard now clamps the settings got from the initiator to the shard's constaints instead of throwing an exception. This fix allows to send queries to a shard with another constraints. #9447 (Vitaly Baranov)
- Fixed memory management problem in
MergeTreeReadPool
. #8791 (Vladimir Chebotarev) - Fix
toDecimal*OrNull()
functions family when called with stringe
. Fixes #8312 #8764 (Artem Zuikov) - Make sure that
FORMAT Null
sends no data to the client. #8767 (Alexander Kuzmenkov) - Fix bug that timestamp in
LiveViewBlockInputStream
will not updated.LIVE VIEW
is an experimental feature. #8644 (vxider) #8625 (vxider) - Fixed
ALTER MODIFY TTL
wrong behavior which did not allow to delete old TTL expressions. #8422 (Vladimir Chebotarev) - Fixed UBSan report in MergeTreeIndexSet. This fixes #9250 #9365 (alexey-milovidov)
- Fixed the behaviour of
match
andextract
functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes #9160 #9163 (alexey-milovidov) #9345 (alexey-milovidov) - Avoid throwing from destructor in Apache Avro 3rd-party library. #9066 (Andrew Onyshchuk)
- Don't commit a batch polled from
Kafka
partially as it can lead to holes in data. #8876 (filimonov) - Fix
joinGet
with nullable return types. https://github.com/ClickHouse/ClickHouse/issues/8919 #9014 (Amos Bird) - Fix data incompatibility when compressed with
T64
codec. #9016 (Artem Zuikov) Fix data type ids inT64
compression codec that leads to wrong (de)compression in affected versions. #9033 (Artem Zuikov) - Add setting
enable_early_constant_folding
and disable it in some cases that leads to errors. #9010 (Artem Zuikov) - Fix pushdown predicate optimizer with VIEW and enable the test #9011 (Winter Zhang)
- Fix segfault in
Merge
tables, that can happen when reading fromFile
storages #9387 (tavplubix) - Added a check for storage policy in
ATTACH PARTITION FROM
,REPLACE PARTITION
,MOVE TO TABLE
. Otherwise it could make data of part inaccessible after restart and prevent ClickHouse to start. #9383 (Vladimir Chebotarev) - Fix alters if there is TTL set for table. #8800 (Anton Popov)
- Fix race condition that can happen when
SYSTEM RELOAD ALL DICTIONARIES
is executed while some dictionary is being modified/added/removed. #8801 (Vitaly Baranov) - In previous versions
Memory
database engine use empty data path, so tables are created inpath
directory (e.g./var/lib/clickhouse/
), not in data directory of database (e.g./var/lib/clickhouse/db_name
). #8753 (tavplubix) - Fixed wrong log messages about missing default disk or policy. #9530 (Vladimir Chebotarev)
- Fix not(has()) for the bloom_filter index of array types. #9407 (achimbab)
- Allow first column(s) in a table with
Log
engine be an alias #9231 (Ivan) - Fix order of ranges while reading from
MergeTree
table in one thread. It could lead to exceptions fromMergeTreeRangeReader
or wrong query results. #9050 (Anton Popov) - Make
reinterpretAsFixedString
to returnFixedString
instead ofString
. #9052 (Andrew Onyshchuk) - Avoid extremely rare cases when the user can get wrong error message (
Success
instead of detailed error description). #9457 (alexey-milovidov) - Do not crash when using
Template
format with empty row template. #8785 (Alexander Kuzmenkov) - Metadata files for system tables could be created in wrong place #8653 (tavplubix) Fixes #8581.
- Fix data race on exception_ptr in cache dictionary #8303. #9379 (Nikita Mikhaylov)
- Do not throw an exception for query
ATTACH TABLE IF NOT EXISTS
. Previously it was thrown if table already exists, despite theIF NOT EXISTS
clause. #8967 (Anton Popov) - Fixed missing closing paren in exception message. #8811 (alexey-milovidov)
- Avoid message
Possible deadlock avoided
at the startup of clickhouse-client in interactive mode. #9455 (alexey-milovidov) - Fixed the issue when padding at the end of base64 encoded value can be malformed. Update base64 library. This fixes #9491, closes #9492 #9500 (alexey-milovidov)
- Prevent losing data in
Kafka
in rare cases when exception happens after reading suffix but before commit. Fixes #9378 #9507 (filimonov) - Fixed exception in
DROP TABLE IF EXISTS
#8663 (Nikita Vasilev) - Fix crash when a user tries to
ALTER MODIFY SETTING
for old-formatedMergeTree
table engines family. #9435 (alesapin) - Support for UInt64 numbers that don't fit in Int64 in JSON-related functions. Update SIMDJSON to master. This fixes #9209 #9344 (alexey-milovidov)
- Fixed execution of inversed predicates when non-strictly monotinic functional index is used. #9223 (Alexander Kazakov)
- Don't try to fold
IN
constant inGROUP BY
#8868 (Amos Bird) - Fix bug in
ALTER DELETE
mutations which leads to index corruption. This fixes #9019 and #8982. Additionally fix extremely rare race conditions inReplicatedMergeTree
ALTER
queries. #9048 (alesapin) - When the setting
compile_expressions
is enabled, you can getunexpected column
inLLVMExecutableFunction
when we useNullable
type #8910 (Guillaume Tassery) - Multiple fixes for
Kafka
engine: 1) fix duplicates that were appearing during consumer group rebalance. 2) Fix rare 'holes' appeared when data were polled from several partitions with one poll and committed partially (now we always process / commit the whole polled block of messages). 3) Fix flushes by block size (before that only flushing by timeout was working properly). 4) better subscription procedure (with assignment feedback). 5) Make tests work faster (with default intervals and timeouts). Due to the fact that data was not flushed by block size before (as it should according to documentation), that PR may lead to some performance degradation with default settings (due to more often & tinier flushes which are less optimal). If you encounter the performance issue after that change - please increasekafka_max_block_size
in the table to the bigger value ( for exampleCREATE TABLE ...Engine=Kafka ... SETTINGS ... kafka_max_block_size=524288
). Fixes #7259 #8917 (filimonov) - Fix
Parameter out of bound
exception in some queries after PREWHERE optimizations. #8914 (Baudouin Giard) - Fixed the case of mixed-constness of arguments of function
arrayZip
. #8705 (alexey-milovidov) - When executing
CREATE
query, fold constant expressions in storage engine arguments. Replace empty database name with current database. Fixes #6508, #3492 #9262 (tavplubix) - Now it's not possible to create or add columns with simple cyclic aliases like
a DEFAULT b, b DEFAULT a
. #9603 (alesapin) - Fixed a bug with double move which may corrupt original part. This is relevant if you use
ALTER TABLE MOVE
#8680 (Vladimir Chebotarev) - Allow
interval
identifier to correctly parse without backticks. Fixed issue when a query cannot be executed even if theinterval
identifier is enclosed in backticks or double quotes. This fixes #9124. #9142 (alexey-milovidov) - Fixed fuzz test and incorrect behaviour of
bitTestAll
/bitTestAny
functions. #9143 (alexey-milovidov) - Fix possible crash/wrong number of rows in
LIMIT n WITH TIES
when there are a lot of rows equal to n'th row. #9464 (tavplubix) - Fix mutations with parts written with enabled
insert_quorum
. #9463 (alesapin) - Fix data race at destruction of
Poco::HTTPServer
. It could happen when server is started and immediately shut down. #9468 (Anton Popov) - Fix bug in which a misleading error message was shown when running
SHOW CREATE TABLE a_table_that_does_not_exist
. #8899 (achulkov2) - Fixed
Parameters are out of bound
exception in some rare cases when we have a constant in theSELECT
clause when we have anORDER BY
and aLIMIT
clause. #8892 (Guillaume Tassery) - Fix mutations finalization, when already done mutation can have status
is_done=0
. #9217 (alesapin) - Prevent from executing
ALTER ADD INDEX
for MergeTree tables with old syntax, because it doesn't work. #8822 (Mikhail Korotov) - During server startup do not access table, which
LIVE VIEW
depends on, so server will be able to start. Also removeLIVE VIEW
dependencies when detachingLIVE VIEW
.LIVE VIEW
is an experimental feature. #8824 (tavplubix) - Fix possible segfault in
MergeTreeRangeReader
, while executingPREWHERE
. #9106 (Anton Popov) - Fix possible mismatched checksums with column TTLs. #9451 (Anton Popov)
- Fixed a bug when parts were not being moved in background by TTL rules in case when there is only one volume. #8672 (Vladimir Chebotarev)
- Fixed the issue
Method createColumn() is not implemented for data type Set
. This fixes #7799. #8674 (alexey-milovidov) - Now we will try finalize mutations more frequently. #9427 (alesapin)
- Fix
intDiv
by minus one constant #9351 (hcz) - Fix possible race condition in
BlockIO
. #9356 (Nikolai Kochetov) - Fix bug leading to server termination when trying to use / drop
Kafka
table created with wrong parameters. #9513 (filimonov) - Added workaround if OS returns wrong result for
timer_create
function. #8837 (alexey-milovidov) - Fixed error in usage of
min_marks_for_seek
parameter. Fixed the error message when there is no sharding key in Distributed table and we try to skip unused shards. #8908 (Azat Khuzhin)
Improvement
- Implement
ALTER MODIFY/DROP
queries on top of mutations forReplicatedMergeTree*
engines family. NowALTERS
blocks only at the metadata update stage, and don't block after that. #8701 (alesapin) - Add ability to rewrite CROSS to INNER JOINs with
WHERE
section containing unqialified names. #9512 (Artem Zuikov) - Make
SHOW TABLES
andSHOW DATABASES
queries support theWHERE
expressions andFROM
/IN
#9076 (sundyli) - Added a setting
deduplicate_blocks_in_dependent_materialized_views
. #9070 (urykhy) - After recent changes MySQL client started to print binary strings in hex thereby making them not readable (#9032). The workaround in ClickHouse is to mark string columns as UTF-8, which is not always, but usually the case. #9079 (Yuriy Baranov)
- Add support of String and FixedString keys for
sumMap
#8903 (Baudouin Giard) - Support string keys in SummingMergeTree maps #8933 (Baudouin Giard)
- Signal termination of thread to the thread pool even if the thread has thrown exception #8736 (Ding Xiang Fei)
- Allow to set
query_id
inclickhouse-benchmark
#9416 (Anton Popov) - Don't allow strange expressions in
ALTER TABLE ... PARTITION partition
query. This addresses #7192 #8835 (alexey-milovidov) - The table
system.table_engines
now provides information about feature support (likesupports_ttl
orsupports_sort_order
). #8830 (Max Akhmedov) - Enable
system.metric_log
by default. It will contain rows with values of ProfileEvents, CurrentMetrics collected with "collect_interval_milliseconds" interval (one second by default). The table is very small (usually in order of megabytes) and collecting this data by default is reasonable. #9225 (alexey-milovidov) - Initialize query profiler for all threads in a group, e.g. it allows to fully profile insert-queries. Fixes #6964 #8874 (Ivan)
- Now temporary
LIVE VIEW
is created byCREATE LIVE VIEW name WITH TIMEOUT [42] ...
instead ofCREATE TEMPORARY LIVE VIEW ...
, because the previous syntax was not consistent withCREATE TEMPORARY TABLE ...
#9131 (tavplubix) - Add text_log.level configuration parameter to limit entries that goes to
system.text_log
table #8809 (Azat Khuzhin) - Allow to put downloaded part to a disks/volumes according to TTL rules #8598 (Vladimir Chebotarev)
- For external MySQL dictionaries, allow to mutualize MySQL connection pool to "share" them among dictionaries. This option significantly reduces the number of connections to MySQL servers. #9409 (Clément Rodriguez)
- Show nearest query execution time for quantiles in
clickhouse-benchmark
output instead of interpolated values. It's better to show values that correspond to the execution time of some queries. #8712 (alexey-milovidov) - Possibility to add key & timestamp for the message when inserting data to Kafka. Fixes #7198 #8969 (filimonov)
- If server is run from terminal, highlight thread number, query id and log priority by colors. This is for improved readability of correlated log messages for developers. #8961 (alexey-milovidov)
- Better exception message while loading tables for
Ordinary
database. #9527 (alexey-milovidov) - Implement
arraySlice
for arrays with aggregate function states. This fixes #9388 #9391 (alexey-milovidov) - Allow constant functions and constant arrays to be used on the right side of IN operator. #8813 (Anton Popov)
- If zookeeper exception has happened while fetching data for system.replicas, display it in a separate column. This implements #9137 #9138 (alexey-milovidov)
- Atomically remove MergeTree data parts on destroy. #8402 (Vladimir Chebotarev)
- Support row-level security for Distributed tables. #8926 (Ivan)
- Now we recognize suffix (like KB, KiB...) in settings values. #8072 (Mikhail Korotov)
- Prevent out of memory while constructing result of a large JOIN. #8637 (Artem Zuikov)
- Added names of clusters to suggestions in interactive mode in
clickhouse-client
. #8709 (alexey-milovidov) - Initialize query profiler for all threads in a group, e.g. it allows to fully profile insert-queries #8820 (Ivan)
- Added column
exception_code
insystem.query_log
table. #8770 (Mikhail Korotov) - Enabled MySQL compatibility server on port
9004
in the default server configuration file. Fixed password generation command in the example in configuration. #8771 (Yuriy Baranov) - Prevent abort on shutdown if the filesystem is readonly. This fixes #9094 #9100 (alexey-milovidov)
- Better exception message when length is required in HTTP POST query. #9453 (alexey-milovidov)
- Add
_path
and_file
virtual columns toHDFS
andFile
engines andhdfs
andfile
table functions #8489 (Olga Khvostikova) - Fix error
Cannot find column
while inserting intoMATERIALIZED VIEW
in case if new column was added to view's internal table. #8766 #8788 (vzakaznikov) #8788 #8806 (Nikolai Kochetov) #8803 (Nikolai Kochetov) - Fix progress over native client-server protocol, by send progress after final update (like logs). This may be relevant only to some third-party tools that are using native protocol. #9495 (Azat Khuzhin)
- Add a system metric tracking the number of client connections using MySQL protocol (#9013). #9015 (Eugene Klimov)
- From now on, HTTP responses will have
X-ClickHouse-Timezone
header set to the same timezone value thatSELECT timezone()
would report. #9493 (Denis Glazachev)
Performance Improvement
- Improve performance of analysing index with IN #9261 (Anton Popov)
- Simpler and more efficient code in Logical Functions + code cleanups. A followup to #8718 #8728 (Alexander Kazakov)
- Overall performance improvement (in range of 5%..200% for affected queries) by ensuring even more strict aliasing with C++20 features. #9304 (Amos Bird)
- More strict aliasing for inner loops of comparison functions. #9327 (alexey-milovidov)
- More strict aliasing for inner loops of arithmetic functions. #9325 (alexey-milovidov)
- A ~3 times faster implementation for ColumnVector::replicate(), via which ColumnConst::convertToFullColumn() is implemented. Also will be useful in tests when materializing constants. #9293 (Alexander Kazakov)
- Another minor performance improvement to
ColumnVector::replicate()
(this speeds up thematerialize
function and higher order functions) an even further improvement to #9293 #9442 (Alexander Kazakov) - Improved performance of
stochasticLinearRegression
aggregate function. This patch is contributed by Intel. #8652 (alexey-milovidov) - Improve performance of
reinterpretAsFixedString
function. #9342 (alexey-milovidov) - Do not send blocks to client for
Null
format in processors pipeline. #8797 (Nikolai Kochetov) #8767 (Alexander Kuzmenkov)
Build/Testing/Packaging Improvement
- Exception handling now works correctly on Windows Subsystem for Linux. See https://github.com/ClickHouse-Extras/libunwind/pull/3 This fixes #6480 #9564 (sobolevsv)
- Replace
readline
withreplxx
for interactive line editing inclickhouse-client
#8416 (Ivan) - Better build time and less template instantiations in FunctionsComparison. #9324 (alexey-milovidov)
- Added integration with
clang-tidy
in CI. See also #6044 #9566 (alexey-milovidov) - Now we link ClickHouse in CI using
lld
even forgcc
. #9049 (alesapin) - Allow to randomize thread scheduling and insert glitches when
THREAD_FUZZER_*
environment variables are set. This helps testing. #9459 (alexey-milovidov) - Enable secure sockets in stateless tests #9288 (tavplubix)
- Make SPLIT_SHARED_LIBRARIES=OFF more robust #9156 (Azat Khuzhin)
- Make "performance_introspection_and_logging" test reliable to random server stuck. This may happen in CI environment. See also #9515 #9528 (alexey-milovidov)
- Validate XML in style check. #9550 (alexey-milovidov)
- Fixed race condition in test
00738_lock_for_inner_table
. This test relied on sleep. #9555 (alexey-milovidov) - Remove performance tests of type
once
. This is needed to run all performance tests in statistical comparison mode (more reliable). #9557 (alexey-milovidov) - Added performance test for arithmetic functions. #9326 (alexey-milovidov)
- Added performance test for
sumMap
andsumMapWithOverflow
aggregate functions. Follow-up for #8933 #8947 (alexey-milovidov) - Ensure style of ErrorCodes by style check. #9370 (alexey-milovidov)
- Add script for tests history. #8796 (alesapin)
- Add GCC warning
-Wsuggest-override
to locate and fix all places whereoverride
keyword must be used. #8760 (kreuzerkrieg) - Ignore weak symbol under Mac OS X because it must be defined #9538 (Deleted user)
- Normalize running time of some queries in performance tests. This is done in preparation to run all the performance tests in comparison mode. #9565 (alexey-milovidov)
- Fix some tests to support pytest with query tests #9062 (Ivan)
- Enable SSL in build with MSan, so server will not fail at startup when running stateless tests #9531 (tavplubix)
- Fix database substitution in test results #9384 (Ilya Yatsishin)
- Build fixes for miscellaneous platforms #9381 (proller) #8755 (proller) #8631 (proller)
- Added disks section to stateless-with-coverage test docker image #9213 (Pavel Kovalenko)
- Get rid of in-source-tree files when building with GRPC #9588 (Amos Bird)
- Slightly faster build time by removing SessionCleaner from Context. Make the code of SessionCleaner more simple. #9232 (alexey-milovidov)
- Updated checking for hung queries in clickhouse-test script #8858 (Alexander Kazakov)
- Removed some useless files from repository. #8843 (alexey-milovidov)
- Changed type of math perftests from
once
toloop
. #8783 (Nikolai Kochetov) - Add docker image which allows to build interactive code browser HTML report for our codebase. #8781 (alesapin) See Woboq Code Browser
- Suppress some test failures under MSan. #8780 (Alexander Kuzmenkov)
- Speedup "exception while insert" test. This test often time out in debug-with-coverage build. #8711 (alexey-milovidov)
- Updated
libcxx
andlibcxxabi
to master. In preparation to #9304 #9308 (alexey-milovidov) - Fix flacky test
00910_zookeeper_test_alter_compression_codecs
. #9525 (alexey-milovidov) - Clean up duplicated linker flags. Make sure the linker won't look up an unexpected symbol. #9433 (Amos Bird)
- Add
clickhouse-odbc
driver into test images. This allows to test interaction of ClickHouse with ClickHouse via its own ODBC driver. #9348 (filimonov) - Fix several bugs in unit tests. #9047 (alesapin)
- Enable
-Wmissing-include-dirs
GCC warning to eliminate all non-existing includes - mostly as a result of CMake scripting errors #8704 (kreuzerkrieg) - Describe reasons if query profiler cannot work. This is intended for #9049 #9144 (alexey-milovidov)
- Update OpenSSL to upstream master. Fixed the issue when TLS connections may fail with the message
OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
andSSL Exception: error:2400006E:random number generator::error retrieving entropy
. The issue was present in version 20.1. #8956 (alexey-milovidov) - Update Dockerfile for server #8893 (Ilya Mazaev)
- Minor fixes in build-gcc-from-sources script #8774 (Michael Nacharov)
- Replace
numbers
tozeros
in perftests wherenumber
column is not used. This will lead to more clean test results. #9600 (Nikolai Kochetov) - Fix stack overflow issue when using initializer_list in Column constructors. #9367 (Deleted user)
- Upgrade librdkafka to v1.3.0. Enable bundled
rdkafka
andgsasl
libraries on Mac OS X. #9000 (Andrew Onyshchuk) - build fix on GCC 9.2.0 #9306 (vxider)
ClickHouse release v20.1
ClickHouse release v20.1.10.70, 2020-04-17
Bug Fix
- Fix rare possible exception
Cannot drain connections: cancel first
. #10239 (Nikolai Kochetov). - Fixed bug where ClickHouse would throw
'Unknown function lambda.'
error message when user tries to runALTER UPDATE/DELETE
on tables withENGINE = Replicated*
. Check for nondeterministic functions now handles lambda expressions correctly. #10237 (Alexander Kazakov). - Fix
parseDateTimeBestEffort
for strings in RFC-2822 when day of week is Tuesday or Thursday. This fixes #10082. #10214 (alexey-milovidov). - Fix column names of constants inside
JOIN
that may clash with names of constants outside ofJOIN
. #10207 (alexey-milovidov). - Fix possible inifinite query execution when the query actually should stop on LIMIT, while reading from infinite source like
system.numbers
orsystem.zeros
. #10206 (Nikolai Kochetov). - Fix move-to-prewhere optimization in presense of
arrayJoin
functions (in certain cases). This fixes #10092. #10195 (alexey-milovidov). - Add the ability to relax the restriction on non-deterministic functions usage in mutations with
allow_nondeterministic_mutations
setting. #10186 (filimonov). - Convert blocks if structure does not match on
INSERT
into table withDistributed
engine. #10135 (Azat Khuzhin). - Fix
SIGSEGV
onINSERT
intoDistributed
table when its structure differs from the underlying tables. #10105 (Azat Khuzhin). - Fix possible rows loss for queries with
JOIN
andUNION ALL
. Fixes #9826, #10113. #10099 (Nikolai Kochetov). - Add arguments check and support identifier arguments for MySQL Database Engine. #10077 (Winter Zhang).
- Fix bug in clickhouse dictionary source from localhost clickhouse server. The bug may lead to memory corruption if types in dictionary and source are not compatible. #10071 (alesapin).
- Fix error
Cannot clone block with columns because block has 0 columns ... While executing GroupingAggregatedTransform
. It happened when settingdistributed_aggregation_memory_efficient
was enabled, and distributed query read aggregating data with different level from different shards (mixed single and two level aggregation). #10063 (Nikolai Kochetov). - Fix a segmentation fault that could occur in
GROUP BY
over string keys containing trailing zero bytes (#8636, #8925). #10025 (Alexander Kuzmenkov). - Fix bug in which the necessary tables weren't retrieved at one of the processing stages of queries to some databases. Fixes #9699. #9949 (achulkov2).
- Fix
'Not found column in block'
error whenJOIN
appears withTOTALS
. Fixes #9839. #9939 (Artem Zuikov). - Fix a bug with
ON CLUSTER
DDL queries freezing on server startup. #9927 (Gagan Arneja). - Fix
TRUNCATE
for Join table engine (#9917). #9920 (Amos Bird). - Fix
'scalar doesn't exist'
error in ALTER queries (#9878). #9904 (Amos Bird). - Fix race condition between drop and optimize in
ReplicatedMergeTree
. #9901 (alesapin). - Fixed
DeleteOnDestroy
logic inATTACH PART
which could lead to automatic removal of attached part and added few tests. #9410 (Vladimir Chebotarev).
Build/Testing/Packaging Improvement
- Fix unit test
collapsing_sorted_stream
. #9367 (Deleted user).
ClickHouse release v20.1.9.54, 2020-03-28
Bug Fix
- Fix
'Different expressions with the same alias'
error when query hasPREWHERE
andWHERE
on distributed table andSET distributed_product_mode = 'local'
. #9871 (Artem Zuikov). - Fix mutations excessive memory consumption for tables with a composite primary key. This fixes #9850. #9860 (alesapin).
- For INSERT queries shard now clamps the settings got from the initiator to the shard's constaints instead of throwing an exception. This fix allows to send
INSERT
queries to a shard with another constraints. This change improves fix #9447. #9852 (Vitaly Baranov). - Fix possible exception
Got 0 in totals chunk, expected 1
on client. It happened for queries withJOIN
in case if right joined table had zero rows. Example:select * from system.one t1 join system.one t2 on t1.dummy = t2.dummy limit 0 FORMAT TabSeparated;
. Fixes #9777. #9823 (Nikolai Kochetov). - Fix
SIGSEGV
withoptimize_skip_unused_shards
when type cannot be converted. #9804 (Azat Khuzhin). - Fixed a few cases when timezone of the function argument wasn't used properly. #9574 (Vasily Nemkov).
Improvement
- Remove
ORDER BY
stage from mutations because we read from a single ordered part in a single thread. Also add check that the order of rows in mutation is ordered in sorting key order and this order is not violated. #9886 (alesapin).
Build/Testing/Packaging Improvement
- Clean up duplicated linker flags. Make sure the linker won't look up an unexpected symbol. #9433 (Amos Bird).
ClickHouse release v20.1.8.41, 2020-03-20
Bug Fix
- Fix possible permanent
Cannot schedule a task
error (due to unhandled exception inParallelAggregatingBlockInputStream::Handler::onFinish/onFinishThread
). This fixes #6833. #9154 (Azat Khuzhin) - Fix excessive memory consumption in
ALTER
queries (mutations). This fixes #9533 and #9670. #9754 (alesapin) - Fix bug in backquoting in external dictionaries DDL. This fixes #9619. #9734 (alesapin)
ClickHouse release v20.1.7.38, 2020-03-18
Bug Fix
- Fixed incorrect internal function names for
sumKahan
andsumWithOverflow
. I lead to exception while using this functions in remote queries. #9636 (Azat Khuzhin). This issue was in all ClickHouse releases. - Allow
ALTER ON CLUSTER
ofDistributed
tables with internal replication. This fixes #3268. #9617 (shinoi2). This issue was in all ClickHouse releases. - Fix possible exceptions
Size of filter doesn't match size of column
andInvalid number of rows in Chunk
inMergeTreeRangeReader
. They could appear while executingPREWHERE
in some cases. Fixes #9132. #9612 (Anton Popov) - Fixed the issue: timezone was not preserved if you write a simple arithmetic expression like
time + 1
(in contrast to an expression liketime + INTERVAL 1 SECOND
). This fixes #5743. #9323 (alexey-milovidov). This issue was in all ClickHouse releases. - Now it's not possible to create or add columns with simple cyclic aliases like
a DEFAULT b, b DEFAULT a
. #9603 (alesapin) - Fixed the issue when padding at the end of base64 encoded value can be malformed. Update base64 library. This fixes #9491, closes #9492 #9500 (alexey-milovidov)
- Fix data race at destruction of
Poco::HTTPServer
. It could happen when server is started and immediately shut down. #9468 (Anton Popov) - Fix possible crash/wrong number of rows in
LIMIT n WITH TIES
when there are a lot of rows equal to n'th row. #9464 (tavplubix) - Fix possible mismatched checksums with column TTLs. #9451 (Anton Popov)
- Fix crash when a user tries to
ALTER MODIFY SETTING
for old-formatedMergeTree
table engines family. #9435 (alesapin) - Now we will try finalize mutations more frequently. #9427 (alesapin)
- Fix replication protocol incompatibility introduced in #8598. #9412 (alesapin)
- Fix not(has()) for the bloom_filter index of array types. #9407 (achimbab)
- Fixed the behaviour of
match
andextract
functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes #9160 #9163 (alexey-milovidov) #9345 (alexey-milovidov)
Build/Testing/Packaging Improvement
- Exception handling now works correctly on Windows Subsystem for Linux. See https://github.com/ClickHouse-Extras/libunwind/pull/3 This fixes #6480 #9564 (sobolevsv)
ClickHouse release v20.1.6.30, 2020-03-05
Bug Fix
- Fix data incompatibility when compressed with
T64
codec. #9039 (abyss7) - Fix order of ranges while reading from MergeTree table in one thread. Fixes #8964. #9050 (CurtizJ)
- Fix possible segfault in
MergeTreeRangeReader
, while executingPREWHERE
. Fixes #9064. #9106 (CurtizJ) - Fix
reinterpretAsFixedString
to returnFixedString
instead ofString
. #9052 (oandrew) - Fix
joinGet
with nullable return types. Fixes #8919 #9014 (amosbird) - Fix fuzz test and incorrect behaviour of bitTestAll/bitTestAny functions. #9143 (alexey-milovidov)
- Fix the behaviour of match and extract functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. Fixes #9160 #9163 (alexey-milovidov)
- Fixed execution of inversed predicates when non-strictly monotinic functional index is used. Fixes #9034 #9223 (Akazz)
- Allow to rewrite
CROSS
toINNER JOIN
if there's[NOT] LIKE
operator inWHERE
section. Fixes #9191 #9229 (4ertus2) - Allow first column(s) in a table with Log engine be an alias. #9231 (abyss7)
- Allow comma join with
IN()
inside. Fixes #7314. #9251 (4ertus2) - Improve
ALTER MODIFY/ADD
queries logic. Now you cannotADD
column without type,MODIFY
default expression doesn't change type of column andMODIFY
type doesn't loose default expression value. Fixes #8669. #9227 (alesapin) - Fix mutations finalization, when already done mutation can have status is_done=0. #9217 (alesapin)
- Support "Processors" pipeline for system.numbers and system.numbers_mt. This also fixes the bug when
max_execution_time
is not respected. #7796 (KochetovNicolai) - Fix wrong counting of
DictCacheKeysRequestedFound
metric. #9411 (nikitamikhaylov) - Added a check for storage policy in
ATTACH PARTITION FROM
,REPLACE PARTITION
,MOVE TO TABLE
which otherwise could make data of part inaccessible after restart and prevent ClickHouse to start. #9383 (excitoon) - Fixed UBSan report in
MergeTreeIndexSet
. This fixes #9250 #9365 (alexey-milovidov) - Fix possible datarace in BlockIO. #9356 (KochetovNicolai)
- Support for
UInt64
numbers that don't fit in Int64 in JSON-related functions. UpdateSIMDJSON
to master. This fixes #9209 #9344 (alexey-milovidov) - Fix the issue when the amount of free space is not calculated correctly if the data directory is mounted to a separate device. For default disk calculate the free space from data subdirectory. This fixes #7441 #9257 (millb)
- Fix the issue when TLS connections may fail with the message
OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error and SSL Exception: error:2400006E:random number generator::error retrieving entropy.
Update OpenSSL to upstream master. #8956 (alexey-milovidov) - When executing
CREATE
query, fold constant expressions in storage engine arguments. Replace empty database name with current database. Fixes #6508, #3492. Also fix check for local address in ClickHouseDictionarySource. #9262 (tabplubix) - Fix segfault in
StorageMerge
, which can happen when reading from StorageFile. #9387 (tabplubix) - Prevent losing data in
Kafka
in rare cases when exception happens after reading suffix but before commit. Fixes #9378. Related: #7175 #9507 (filimonov) - Fix bug leading to server termination when trying to use / drop
Kafka
table created with wrong parameters. Fixes #9494. Incorporates #9507. #9513 (filimonov)
New Feature
- Add
deduplicate_blocks_in_dependent_materialized_views
option to control the behaviour of idempotent inserts into tables with materialized views. This new feature was added to the bugfix release by a special request from Altinity. #9070 (urykhy)
ClickHouse release v20.1.2.4, 2020-01-22
Backward Incompatible Change
- Make the setting
merge_tree_uniform_read_distribution
obsolete. The server still recognizes this setting but it has no effect. #8308 (alexey-milovidov) - Changed return type of the function
greatCircleDistance
toFloat32
because now the result of calculation isFloat32
. #7993 (alexey-milovidov) - Now it's expected that query parameters are represented in "escaped" format. For example, to pass string
a<tab>b
you have to writea\tb
ora\<tab>b
and respectively,a%5Ctb
ora%5C%09b
in URL. This is needed to add the possibility to pass NULL as\N
. This fixes #7488. #8517 (alexey-milovidov) - Enable
use_minimalistic_part_header_in_zookeeper
setting forReplicatedMergeTree
by default. This will significantly reduce amount of data stored in ZooKeeper. This setting is supported since version 19.1 and we already use it in production in multiple services without any issues for more than half a year. Disable this setting if you have a chance to downgrade to versions older than 19.1. #6850 (alexey-milovidov) - Data skipping indices are production ready and enabled by default. The settings
allow_experimental_data_skipping_indices
,allow_experimental_cross_to_join_conversion
andallow_experimental_multiple_joins_emulation
are now obsolete and do nothing. #7974 (alexey-milovidov) - Add new
ANY JOIN
logic forStorageJoin
consistent withJOIN
operation. To upgrade without changes in behaviour you need addSETTINGS any_join_distinct_right_table_keys = 1
to Engine Join tables metadata or recreate these tables after upgrade. #8400 (Artem Zuikov) - Require server to be restarted to apply the changes in logging configuration. This is a temporary workaround to avoid the bug where the server logs to a deleted log file (see #8696). #8707 (Alexander Kuzmenkov)
New Feature
- Added information about part paths to
system.merges
. #8043 (Vladimir Chebotarev) - Add ability to execute
SYSTEM RELOAD DICTIONARY
query inON CLUSTER
mode. #8288 (Guillaume Tassery) - Add ability to execute
CREATE DICTIONARY
queries inON CLUSTER
mode. #8163 (alesapin) - Now user's profile in
users.xml
can inherit multiple profiles. #8343 (Mikhail f. Shiryaev) - Added
system.stack_trace
table that allows to look at stack traces of all server threads. This is useful for developers to introspect server state. This fixes #7576. #8344 (alexey-milovidov) - Add
DateTime64
datatype with configurable sub-second precision. #7170 (Vasily Nemkov) - Add table function
clusterAllReplicas
which allows to query all the nodes in the cluster. #8493 (kiran sunkari) - Add aggregate function
categoricalInformationValue
which calculates the information value of a discrete feature. #8117 (hcz) - Speed up parsing of data files in
CSV
,TSV
andJSONEachRow
format by doing it in parallel. #7780 (Alexander Kuzmenkov) - Add function
bankerRound
which performs banker's rounding. #8112 (hcz) - Support more languages in embedded dictionary for region names: 'ru', 'en', 'ua', 'uk', 'by', 'kz', 'tr', 'de', 'uz', 'lv', 'lt', 'et', 'pt', 'he', 'vi'. #8189 (alexey-milovidov)
- Improvements in consistency of
ANY JOIN
logic. Nowt1 ANY LEFT JOIN t2
equalst2 ANY RIGHT JOIN t1
. #7665 (Artem Zuikov) - Add setting
any_join_distinct_right_table_keys
which enables old behaviour forANY INNER JOIN
. #7665 (Artem Zuikov) - Add new
SEMI
andANTI JOIN
. OldANY INNER JOIN
behaviour now available asSEMI LEFT JOIN
. #7665 (Artem Zuikov) - Added
Distributed
format forFile
engine andfile
table function which allows to read from.bin
files generated by asynchronous inserts intoDistributed
table. #8535 (Nikolai Kochetov) - Add optional reset column argument for
runningAccumulate
which allows to reset aggregation results for each new key value. #8326 (Sergey Kononenko) - Add ability to use ClickHouse as Prometheus endpoint. #7900 (vdimir)
- Add section
<remote_url_allow_hosts>
inconfig.xml
which restricts allowed hosts for remote table engines and table functionsURL
,S3
,HDFS
. #7154 (Mikhail Korotov) - Added function
greatCircleAngle
which calculates the distance on a sphere in degrees. #8105 (alexey-milovidov) - Changed Earth radius to be consistent with H3 library. #8105 (alexey-milovidov)
- Added
JSONCompactEachRow
andJSONCompactEachRowWithNamesAndTypes
formats for input and output. #7841 (Mikhail Korotov) - Added feature for file-related table engines and table functions (
File
,S3
,URL
,HDFS
) which allows to read and writegzip
files based on additional engine parameter or file extension. #7840 (Andrey Bodrov) - Added the
randomASCII(length)
function, generating a string with a random set of ASCII printable characters. #8401 (BayoNet) - Added function
JSONExtractArrayRaw
which returns an array on unparsed json array elements fromJSON
string. #8081 (Oleg Matrokhin) - Add
arrayZip
function which allows to combine multiple arrays of equal lengths into one array of tuples. #8149 (Winter Zhang) - Add ability to move data between disks according to configured
TTL
-expressions for*MergeTree
table engines family. #8140 (Vladimir Chebotarev) - Added new aggregate function
avgWeighted
which allows to calculate weighted average. #7898 (Andrey Bodrov) - Now parallel parsing is enabled by default for
TSV
,TSKV
,CSV
andJSONEachRow
formats. #7894 (Nikita Mikhaylov) - Add several geo functions from
H3
library:h3GetResolution
,h3EdgeAngle
,h3EdgeLength
,h3IsValid
andh3kRing
. #8034 (Konstantin Malanchev) - Added support for brotli (
br
) compression in file-related storages and table functions. This fixes #8156. #8526 (alexey-milovidov) - Add
groupBit*
functions for theSimpleAggregationFunction
type. #8485 (Guillaume Tassery)
Bug Fix
- Fix rename of tables with
Distributed
engine. Fixes issue #7868. #8306 (tavplubix) - Now dictionaries support
EXPRESSION
for attributes in arbitrary string in non-ClickHouse SQL dialect. #8098 (alesapin) - Fix broken
INSERT SELECT FROM mysql(...)
query. This fixes #8070 and #7960. #8234 (tavplubix) - Fix error "Mismatch column sizes" when inserting default
Tuple
fromJSONEachRow
. This fixes #5653. #8606 (tavplubix) - Now an exception will be thrown in case of using
WITH TIES
alongsideLIMIT BY
. Also add ability to useTOP
withLIMIT BY
. This fixes #7472. #7637 (Nikita Mikhaylov) - Fix unintendent dependency from fresh glibc version in
clickhouse-odbc-bridge
binary. #8046 (Amos Bird) - Fix bug in check function of
*MergeTree
engines family. Now it doesn't fail in case when we have equal amount of rows in last granule and last mark (non-final). #8047 (alesapin) - Fix insert into
Enum*
columns afterALTER
query, when underlying numeric type is equal to table specified type. This fixes #7836. #7908 (Anton Popov) - Allowed non-constant negative "size" argument for function
substring
. It was not allowed by mistake. This fixes #4832. #7703 (alexey-milovidov) - Fix parsing bug when wrong number of arguments passed to
(O|J)DBC
table engine. #7709 (alesapin) - Using command name of the running clickhouse process when sending logs to syslog. In previous versions, empty string was used instead of command name. #8460 (Michael Nacharov)
- Fix check of allowed hosts for
localhost
. This PR fixes the solution provided in #8241. #8342 (Vitaly Baranov) - Fix rare crash in
argMin
andargMax
functions for long string arguments, when result is used inrunningAccumulate
function. This fixes #8325 #8341 (dinosaur) - Fix memory overcommit for tables with
Buffer
engine. #8345 (Azat Khuzhin) - Fixed potential bug in functions that can take
NULL
as one of the arguments and return non-NULL. #8196 (alexey-milovidov) - Better metrics calculations in thread pool for background processes for
MergeTree
table engines. #8194 (Vladimir Chebotarev) - Fix function
IN
insideWHERE
statement when row-level table filter is present. Fixes #6687 #8357 (Ivan) - Now an exception is thrown if the integral value is not parsed completely for settings values. #7678 (Mikhail Korotov)
- Fix exception when aggregate function is used in query to distributed table with more than two local shards. #8164 (小路)
- Now bloom filter can handle zero length arrays and doesn't perform redundant calculations. #8242 (achimbab)
- Fixed checking if a client host is allowed by matching the client host to
host_regexp
specified inusers.xml
. #8241 (Vitaly Baranov) - Relax ambiguous column check that leads to false positives in multiple
JOIN ON
section. #8385 (Artem Zuikov) - Fixed possible server crash (
std::terminate
) when the server cannot send or write data inJSON
orXML
format with values ofString
data type (that requireUTF-8
validation) or when compressing result data with Brotli algorithm or in some other rare cases. This fixes #7603 #8384 (alexey-milovidov) - Fix race condition in
StorageDistributedDirectoryMonitor
found by CI. This fixes #8364. #8383 (Nikolai Kochetov) - Now background merges in
*MergeTree
table engines family preserve storage policy volume order more accurately. #8549 (Vladimir Chebotarev) - Now table engine
Kafka
works properly withNative
format. This fixes #6731 #7337 #8003. #8016 (filimonov) - Fixed formats with headers (like
CSVWithNames
) which were throwing exception about EOF for table engineKafka
. #8016 (filimonov) - Fixed a bug with making set from subquery in right part of
IN
section. This fixes #5767 and #2542. #7755 (Nikita Mikhaylov) - Fix possible crash while reading from storage
File
. #7756 (Nikolai Kochetov) - Fixed reading of the files in
Parquet
format containing columns of typelist
. #8334 (maxulan) - Fix error
Not found column
for distributed queries withPREWHERE
condition dependent on sampling key ifmax_parallel_replicas > 1
. #7913 (Nikolai Kochetov) - Fix error
Not found column
if query usedPREWHERE
dependent on table's alias and the result set was empty because of primary key condition. #7911 (Nikolai Kochetov) - Fixed return type for functions
rand
andrandConstant
in case ofNullable
argument. Now functions always returnUInt32
and neverNullable(UInt32)
. #8204 (Nikolai Kochetov) - Disabled predicate push-down for
WITH FILL
expression. This fixes #7784. #7789 (Winter Zhang) - Fixed incorrect
count()
result forSummingMergeTree
whenFINAL
section is used. #3280 #7786 (Nikita Mikhaylov) - Fix possible incorrect result for constant functions from remote servers. It happened for queries with functions like
version()
,uptime()
, etc. which returns different constant values for different servers. This fixes #7666. #7689 (Nikolai Kochetov) - Fix complicated bug in push-down predicate optimization which leads to wrong results. This fixes a lot of issues on push-down predicate optimization. #8503 (Winter Zhang)
- Fix crash in
CREATE TABLE .. AS dictionary
query. #8508 (Azat Khuzhin) - Several improvements ClickHouse grammar in
.g4
file. #8294 (taiyang-li) - Fix bug that leads to crashes in
JOIN
s with tables with engineJoin
. This fixes #7556 #8254 #7915 #8100. #8298 (Artem Zuikov) - Fix redundant dictionaries reload on
CREATE DATABASE
. #7916 (Azat Khuzhin) - Limit maximum number of streams for read from
StorageFile
andStorageHDFS
. Fixes https://github.com/ClickHouse/ClickHouse/issues/7650. #7981 (alesapin) - Fix bug in
ALTER ... MODIFY ... CODEC
query, when user specify both default expression and codec. Fixes 8593. #8614 (alesapin) - Fix error in background merge of columns with
SimpleAggregateFunction(LowCardinality)
type. #8613 (Nikolai Kochetov) - Fixed type check in function
toDateTime64
. #8375 (Vasily Nemkov) - Now server do not crash on
LEFT
orFULL JOIN
with and Join engine and unsupportedjoin_use_nulls
settings. #8479 (Artem Zuikov) - Now
DROP DICTIONARY IF EXISTS db.dict
query doesn't throw exception ifdb
doesn't exist. #8185 (Vitaly Baranov) - Fix possible crashes in table functions (
file
,mysql
,remote
) caused by usage of reference to removedIStorage
object. Fix incorrect parsing of columns specified at insertion into table function. #7762 (tavplubix) - Ensure network be up before starting
clickhouse-server
. This fixes #7507. #8570 (Zhichang Yu) - Fix timeouts handling for secure connections, so queries doesn't hang indefenitely. This fixes #8126. #8128 (alexey-milovidov)
- Fix
clickhouse-copier
's redundant contention between concurrent workers. #7816 (Ding Xiang Fei) - Now mutations doesn't skip attached parts, even if their mutation version were larger than current mutation version. #7812 (Zhichang Yu) #8250 (alesapin)
- Ignore redundant copies of
*MergeTree
data parts after move to another disk and server restart. #7810 (Vladimir Chebotarev) - Fix crash in
FULL JOIN
withLowCardinality
inJOIN
key. #8252 (Artem Zuikov) - Forbidden to use column name more than once in insert query like
INSERT INTO tbl (x, y, x)
. This fixes #5465, #7681. #7685 (alesapin) - Added fallback for detection the number of physical CPU cores for unknown CPUs (using the number of logical CPU cores). This fixes #5239. #7726 (alexey-milovidov)
- Fix
There's no column
error for materialized and alias columns. #8210 (Artem Zuikov) - Fixed sever crash when
EXISTS
query was used withoutTABLE
orDICTIONARY
qualifier. Just likeEXISTS t
. This fixes #8172. This bug was introduced in version 19.17. #8213 (alexey-milovidov) - Fix rare bug with error
"Sizes of columns doesn't match"
that might appear when usingSimpleAggregateFunction
column. #7790 (Boris Granveaud) - Fix bug where user with empty
allow_databases
got access to all databases (and same forallow_dictionaries
). #7793 (DeifyTheGod) - Fix client crash when server already disconnected from client. #8071 (Azat Khuzhin)
- Fix
ORDER BY
behaviour in case of sorting by primary key prefix and non primary key suffix. #7759 (Anton Popov) - Check if qualified column present in the table. This fixes #6836. #7758 (Artem Zuikov)
- Fixed behavior with
ALTER MOVE
ran immediately after merge finish moves superpart of specified. Fixes #8103. #8104 (Vladimir Chebotarev) - Fix possible server crash while using
UNION
with different number of columns. Fixes #7279. #7929 (Nikolai Kochetov) - Fix size of result substring for function
substr
with negative size. #8589 (Nikolai Kochetov) - Now server does not execute part mutation in
MergeTree
if there are not enough free threads in background pool. #8588 (tavplubix) - Fix a minor typo on formatting
UNION ALL
AST. #7999 (litao91) - Fixed incorrect bloom filter results for negative numbers. This fixes #8317. #8566 (Winter Zhang)
- Fixed potential buffer overflow in decompress. Malicious user can pass fabricated compressed data that will cause read after buffer. This issue was found by Eldar Zaitov from Yandex information security team. #8404 (alexey-milovidov)
- Fix incorrect result because of integers overflow in
arrayIntersect
. #7777 (Nikolai Kochetov) - Now
OPTIMIZE TABLE
query will not wait for offline replicas to perform the operation. #8314 (javi santana) - Fixed
ALTER TTL
parser forReplicated*MergeTree
tables. #8318 (Vladimir Chebotarev) - Fix communication between server and client, so server read temporary tables info after query failure. #8084 (Azat Khuzhin)
- Fix
bitmapAnd
function error when intersecting an aggregated bitmap and a scalar bitmap. #8082 (Yue Huang) - Refine the definition of
ZXid
according to the ZooKeeper Programmer's Guide which fixes bug inclickhouse-cluster-copier
. #8088 (Ding Xiang Fei) odbc
table function now respectsexternal_table_functions_use_nulls
setting. #7506 (Vasily Nemkov)- Fixed bug that lead to a rare data race. #8143 (Alexander Kazakov)
- Now
SYSTEM RELOAD DICTIONARY
reloads a dictionary completely, ignoringupdate_field
. This fixes #7440. #8037 (Vitaly Baranov) - Add ability to check if dictionary exists in create query. #8032 (alesapin)
- Fix
Float*
parsing inValues
format. This fixes #7817. #7870 (tavplubix) - Fix crash when we cannot reserve space in some background operations of
*MergeTree
table engines family. #7873 (Vladimir Chebotarev) - Fix crash of merge operation when table contains
SimpleAggregateFunction(LowCardinality)
column. This fixes #8515. #8522 (Azat Khuzhin) - Restore support of all ICU locales and add the ability to apply collations for constant expressions. Also add language name to
system.collations
table. #8051 (alesapin) - Fix bug when external dictionaries with zero minimal lifetime (
LIFETIME(MIN 0 MAX N)
,LIFETIME(N)
) don't update in background. #7983 (alesapin) - Fix crash when external dictionary with ClickHouse source has subquery in query. #8351 (Nikolai Kochetov)
- Fix incorrect parsing of file extension in table with engine
URL
. This fixes #8157. #8419 (Andrey Bodrov) - Fix
CHECK TABLE
query for*MergeTree
tables without key. Fixes #7543. #7979 (alesapin) - Fixed conversion of
Float64
to MySQL type. #8079 (Yuriy Baranov) - Now if table was not completely dropped because of server crash, server will try to restore and load it. #8176 (tavplubix)
- Fixed crash in table function
file
while inserting into file that doesn't exist. Now in this case file would be created and then insert would be processed. #8177 (Olga Khvostikova) - Fix rare deadlock which can happen when
trace_log
is in enabled. #7838 (filimonov) - Add ability to work with different types besides
Date
inRangeHashed
external dictionary created from DDL query. Fixes 7899. #8275 (alesapin) - Fixes crash when
now64()
is called with result of another function. #8270 (Vasily Nemkov) - Fixed bug with detecting client IP for connections through mysql wire protocol. #7743 (Dmitry Muzyka)
- Fix empty array handling in
arraySplit
function. This fixes #7708. #7747 (hcz) - Fixed the issue when
pid-file
of another runningclickhouse-server
may be deleted. #8487 (Weiqing Xu) - Fix dictionary reload if it has
invalidate_query
, which stopped updates and some exception on previous update tries. #8029 (alesapin) - Fixed error in function
arrayReduce
that may lead to "double free" and error in aggregate function combinatorResample
that may lead to memory leak. Added aggregate functionaggThrow
. This function can be used for testing purposes. #8446 (alexey-milovidov)
Improvement
- Improved logging when working with
S3
table engine. #8251 (Grigory Pervakov) - Printed help message when no arguments are passed when calling
clickhouse-local
. This fixes #5335. #8230 (Andrey Nagorny) - Add setting
mutations_sync
which allows to waitALTER UPDATE/DELETE
queries synchronously. #8237 (alesapin) - Allow to set up relative
user_files_path
inconfig.xml
(in the way similar toformat_schema_path
). #7632 (hcz) - Add exception for illegal types for conversion functions with
-OrZero
postfix. #7880 (Andrey Konyaev) - Simplify format of the header of data sending to a shard in a distributed query. #8044 (Vitaly Baranov)
Live View
table engine refactoring. #8519 (vzakaznikov)- Add additional checks for external dictionaries created from DDL-queries. #8127 (alesapin)
- Fix error
Column ... already exists
while usingFINAL
andSAMPLE
together, e.g.select count() from table final sample 1/2
. Fixes #5186. #7907 (Nikolai Kochetov) - Now table the first argument of
joinGet
function can be table indentifier. #7707 (Amos Bird) - Allow using
MaterializedView
with subqueries aboveKafka
tables. #8197 (filimonov) - Now background moves between disks run it the seprate thread pool. #7670 (Vladimir Chebotarev)
SYSTEM RELOAD DICTIONARY
now executes synchronously. #8240 (Vitaly Baranov)- Stack traces now display physical addresses (offsets in object file) instead of virtual memory addresses (where the object file was loaded). That allows the use of
addr2line
when binary is position independent and ASLR is active. This fixes #8360. #8387 (alexey-milovidov) - Support new syntax for row-level security filters:
<table name='table_name'>…</table>
. Fixes #5779. #8381 (Ivan) - Now
cityHash
function can work withDecimal
andUUID
types. Fixes #5184. #7693 (Mikhail Korotov) - Removed fixed index granularity (it was 1024) from system logs because it's obsolete after implementation of adaptive granularity. #7698 (alexey-milovidov)
- Enabled MySQL compatibility server when ClickHouse is compiled without SSL. #7852 (Yuriy Baranov)
- Now server checksums distributed batches, which gives more verbose errors in case of corrupted data in batch. #7914 (Azat Khuzhin)
- Support
DROP DATABASE
,DETACH TABLE
,DROP TABLE
andATTACH TABLE
forMySQL
database engine. #8202 (Winter Zhang) - Add authentication in S3 table function and table engine. #7623 (Vladimir Chebotarev)
- Added check for extra parts of
MergeTree
at different disks, in order to not allow to miss data parts at undefined disks. #8118 (Vladimir Chebotarev) - Enable SSL support for Mac client and server. #8297 (Ivan)
- Now ClickHouse can work as MySQL federated server (see https://dev.mysql.com/doc/refman/5.7/en/federated-create-server.html). #7717 (Maxim Fedotov)
clickhouse-client
now only enablebracketed-paste
when multiquery is on and multiline is off. This fixes (#7757)[https://github.com/ClickHouse/ClickHouse/issues/7757]. #7761 (Amos Bird)- Support
Array(Decimal)
inif
function. #7721 (Artem Zuikov) - Support Decimals in
arrayDifference
,arrayCumSum
andarrayCumSumNegative
functions. #7724 (Artem Zuikov) - Added
lifetime
column tosystem.dictionaries
table. #6820 #7727 (kekekekule) - Improved check for existing parts on different disks for
*MergeTree
table engines. Addresses #7660. #8440 (Vladimir Chebotarev) - Integration with
AWS SDK
forS3
interactions which allows to use all S3 features out of the box. #8011 (Pavel Kovalenko) - Added support for subqueries in
Live View
tables. #7792 (vzakaznikov) - Check for using
Date
orDateTime
column fromTTL
expressions was removed. #7920 (Vladimir Chebotarev) - Information about disk was added to
system.detached_parts
table. #7833 (Vladimir Chebotarev) - Now settings
max_(table|partition)_size_to_drop
can be changed without a restart. #7779 (Grigory Pervakov) - Slightly better usability of error messages. Ask user not to remove the lines below
Stack trace:
. #7897 (alexey-milovidov) - Better reading messages from
Kafka
engine in various formats after #7935. #8035 (Ivan) - Better compatibility with MySQL clients which don't support
sha2_password
auth plugin. #8036 (Yuriy Baranov) - Support more column types in MySQL compatibility server. #7975 (Yuriy Baranov)
- Implement
ORDER BY
optimization forMerge
,Buffer
andMaterilized View
storages with underlyingMergeTree
tables. #8130 (Anton Popov) - Now we always use POSIX implementation of
getrandom
to have better compatibility with old kernels (< 3.17). #7940 (Amos Bird) - Better check for valid destination in a move TTL rule. #8410 (Vladimir Chebotarev)
- Better checks for broken insert batches for
Distributed
table engine. #7933 (Azat Khuzhin) - Add column with array of parts name which mutations must process in future to
system.mutations
table. #8179 (alesapin) - Parallel merge sort optimization for processors. #8552 (Nikolai Kochetov)
- The settings
mark_cache_min_lifetime
is now obsolete and does nothing. In previous versions, mark cache can grow in memory larger thanmark_cache_size
to accomodate data withinmark_cache_min_lifetime
seconds. That was leading to confusion and higher memory usage than expected, that is especially bad on memory constrained systems. If you will see performance degradation after installing this release, you should increase themark_cache_size
. #8484 (alexey-milovidov) - Preparation to use
tid
everywhere. This is needed for #7477. #8276 (alexey-milovidov)
Performance Improvement
- Performance optimizations in processors pipeline. #7988 (Nikolai Kochetov)
- Non-blocking updates of expired keys in cache dictionaries (with permission to read old ones). #8303 (Nikita Mikhaylov)
- Compile ClickHouse without
-fno-omit-frame-pointer
globally to spare one more register. #8097 (Amos Bird) - Speedup
greatCircleDistance
function and add performance tests for it. #7307 (Olga Khvostikova) - Improved performance of function
roundDown
. #8465 (alexey-milovidov) - Improved performance of
max
,min
,argMin
,argMax
forDateTime64
data type. #8199 (Vasily Nemkov) - Improved performance of sorting without a limit or with big limit and external sorting. #8545 (alexey-milovidov)
- Improved performance of formatting floating point numbers up to 6 times. #8542 (alexey-milovidov)
- Improved performance of
modulo
function. #7750 (Amos Bird) - Optimized
ORDER BY
and merging with single column key. #8335 (alexey-milovidov) - Better implementation for
arrayReduce
,-Array
and-State
combinators. #7710 (Amos Bird) - Now
PREWHERE
should be optimized to be at least as efficient asWHERE
. #7769 (Amos Bird) - Improve the way
round
androundBankers
handling negative numbers. #8229 (hcz) - Improved decoding performance of
DoubleDelta
andGorilla
codecs by roughly 30-40%. This fixes #7082. #8019 (Vasily Nemkov) - Improved performance of
base64
related functions. #8444 (alexey-milovidov) - Added a function
geoDistance
. It is similar togreatCircleDistance
but uses approximation to WGS-84 ellipsoid model. The performance of both functions are near the same. #8086 (alexey-milovidov) - Faster
min
andmax
aggregation functions forDecimal
data type. #8144 (Artem Zuikov) - Vectorize processing
arrayReduce
. #7608 (Amos Bird) if
chains are now optimized asmultiIf
. #8355 (kamalov-ruslan)- Fix performance regression of
Kafka
table engine introduced in 19.15. This fixes #7261. #7935 (filimonov) - Removed "pie" code generation that
gcc
from Debian packages occasionally brings by default. #8483 (alexey-milovidov) - Parallel parsing data formats #6553 (Nikita Mikhaylov)
- Enable optimized parser of
Values
with expressions by default (input_format_values_deduce_templates_of_expressions=1
). #8231 (tavplubix)
Build/Testing/Packaging Improvement
- Build fixes for
ARM
and in minimal mode. #8304 (proller) - Add coverage file flush for
clickhouse-server
when std::atexit is not called. Also slightly improved logging in stateless tests with coverage. #8267 (alesapin) - Update LLVM library in contrib. Avoid using LLVM from OS packages. #8258 (alexey-milovidov)
- Make bundled
curl
build fully quiet. #8232 #8203 (Pavel Kovalenko) - Fix some
MemorySanitizer
warnings. #8235 (Alexander Kuzmenkov) - Use
add_warning
andno_warning
macros inCMakeLists.txt
. #8604 (Ivan) - Add support of Minio S3 Compatible object (https://min.io/) for better integration tests. #7863 #7875 (Pavel Kovalenko)
- Imported
libc
headers to contrib. It allows to make builds more consistent across various systems (only forx86_64-linux-gnu
). #5773 (alexey-milovidov) - Remove
-fPIC
from some libraries. #8464 (alexey-milovidov) - Clean
CMakeLists.txt
for curl. See https://github.com/ClickHouse/ClickHouse/pull/8011#issuecomment-569478910 #8459 (alexey-milovidov) - Silent warnings in
CapNProto
library. #8220 (alexey-milovidov) - Add performance tests for short string optimized hash tables. #7679 (Amos Bird)
- Now ClickHouse will build on
AArch64
even ifMADV_FREE
is not available. This fixes #8027. #8243 (Amos Bird) - Update
zlib-ng
to fix memory sanitizer problems. #7182 #8206 (Alexander Kuzmenkov) - Enable internal MySQL library on non-Linux system, because usage of OS packages is very fragile and usually doesn't work at all. This fixes #5765. #8426 (alexey-milovidov)
- Fixed build on some systems after enabling
libc++
. This supersedes #8374. #8380 (alexey-milovidov) - Make
Field
methods more type-safe to find more errors. #7386 #8209 (Alexander Kuzmenkov) - Added missing files to the
libc-headers
submodule. #8507 (alexey-milovidov) - Fix wrong
JSON
quoting in performance test output. #8497 (Nikolai Kochetov) - Now stack trace is displayed for
std::exception
andPoco::Exception
. In previous versions it was available only forDB::Exception
. This improves diagnostics. #8501 (alexey-milovidov) - Porting
clock_gettime
andclock_nanosleep
for fresh glibc versions. #8054 (Amos Bird) - Enable
part_log
in example config for developers. #8609 (alexey-milovidov) - Fix async nature of reload in
01036_no_superfluous_dict_reload_on_create_database*
. #8111 (Azat Khuzhin) - Fixed codec performance tests. #8615 (Vasily Nemkov)
- Add install scripts for
.tgz
build and documentation for them. #8612 #8591 (alesapin) - Removed old
ZSTD
test (it was created in year 2016 to reproduce the bug that pre 1.0 version of ZSTD has had). This fixes #8618. #8619 (alexey-milovidov) - Fixed build on Mac OS Catalina. #8600 (meo)
- Increased number of rows in codec performance tests to make results noticeable. #8574 (Vasily Nemkov)
- In debug builds, treat
LOGICAL_ERROR
exceptions as assertion failures, so that they are easier to notice. #8475 (Alexander Kuzmenkov) - Make formats-related performance test more deterministic. #8477 (alexey-milovidov)
- Update
lz4
to fix a MemorySanitizer failure. #8181 (Alexander Kuzmenkov) - Suppress a known MemorySanitizer false positive in exception handling. #8182 (Alexander Kuzmenkov)
- Update
gcc
andg++
to version 9 inbuild/docker/build.sh
#7766 (TLightSky) - Add performance test case to test that
PREWHERE
is worse thanWHERE
. #7768 (Amos Bird) - Progress towards fixing one flacky test. #8621 (alexey-milovidov)
- Avoid MemorySanitizer report for data from
libunwind
. #8539 (alexey-milovidov) - Updated
libc++
to the latest version. #8324 (alexey-milovidov) - Build ICU library from sources. This fixes #6460. #8219 (alexey-milovidov)
- Switched from
libressl
toopenssl
. ClickHouse should support TLS 1.3 and SNI after this change. This fixes #8171. #8218 (alexey-milovidov) - Fixed UBSan report when using
chacha20_poly1305
from SSL (happens on connect to https://yandex.ru/). #8214 (alexey-milovidov) - Fix mode of default password file for
.deb
linux distros. #8075 (proller) - Improved expression for getting
clickhouse-server
PID inclickhouse-test
. #8063 (Alexander Kazakov) - Updated contrib/googletest to v1.10.0. #8587 (Alexander Burmak)
- Fixed ThreadSaninitizer report in
base64
library. Also updated this library to the latest version, but it doesn't matter. This fixes #8397. #8403 (alexey-milovidov) - Fix
00600_replace_running_query
for processors. #8272 (Nikolai Kochetov) - Remove support for
tcmalloc
to makeCMakeLists.txt
simpler. #8310 (alexey-milovidov) - Release gcc builds now use
libc++
instead oflibstdc++
. Recentlylibc++
was used only with clang. This will improve consistency of build configurations and portability. #8311 (alexey-milovidov) - Enable ICU library for build with MemorySanitizer. #8222 (alexey-milovidov)
- Suppress warnings from
CapNProto
library. #8224 (alexey-milovidov) - Removed special cases of code for
tcmalloc
, because it's no longer supported. #8225 (alexey-milovidov) - In CI coverage task, kill the server gracefully to allow it to save the coverage report. This fixes incomplete coverage reports we've been seeing lately. #8142 (alesapin)
- Performance tests for all codecs against
Float64
andUInt64
values. #8349 (Vasily Nemkov) termcap
is very much deprecated and lead to various problems (f.g. missing "up" cap and echoing^J
instead of multi line) . Favorterminfo
or bundledncurses
. #7737 (Amos Bird)- Fix
test_storage_s3
integration test. #7734 (Nikolai Kochetov) - Support
StorageFile(<format>, null)
to insert block into given format file without actually write to disk. This is required for performance tests. #8455 (Amos Bird) - Added argument
--print-time
to functional tests which prints execution time per test. #8001 (Nikolai Kochetov) - Added asserts to
KeyCondition
while evaluating RPN. This will fix warning from gcc-9. #8279 (alexey-milovidov) - Dump cmake options in CI builds. #8273 (Alexander Kuzmenkov)
- Don't generate debug info for some fat libraries. #8271 (alexey-milovidov)
- Make
log_to_console.xml
always log to stderr, regardless of is it interactive or not. #8395 (Alexander Kuzmenkov) - Removed some unused features from
clickhouse-performance-test
tool. #8555 (alexey-milovidov) - Now we will also search for
lld-X
with correspondingclang-X
version. #8092 (alesapin) - Parquet build improvement. #8421 (maxulan)
- More GCC warnings #8221 (kreuzerkrieg)
- Package for Arch Linux now allows to run ClickHouse server, and not only client. #8534 (Vladimir Chebotarev)
- Fix test with processors. Tiny performance fixes. #7672 (Nikolai Kochetov)
- Update contrib/protobuf. #8256 (Matwey V. Kornilov)
- In preparation of switching to c++20 as a new year celebration. "May the C++ force be with ClickHouse." #8447 (Amos Bird)
Experimental Feature
- Added experimental setting
min_bytes_to_use_mmap_io
. It allows to read big files without copying data from kernel to userspace. The setting is disabled by default. Recommended threshold is about 64 MB, because mmap/munmap is slow. #8520 (alexey-milovidov) - Reworked quotas as a part of access control system. Added new table
system.quotas
, new functionscurrentQuota
,currentQuotaKey
, new SQL syntaxCREATE QUOTA
,ALTER QUOTA
,DROP QUOTA
,SHOW QUOTA
. #7257 (Vitaly Baranov) - Allow skipping unknown settings with warnings instead of throwing exceptions. #7653 (Vitaly Baranov)
- Reworked row policies as a part of access control system. Added new table
system.row_policies
, new functioncurrentRowPolicies()
, new SQL syntaxCREATE POLICY
,ALTER POLICY
,DROP POLICY
,SHOW CREATE POLICY
,SHOW POLICIES
. #7808 (Vitaly Baranov)
Security Fix
- Fixed the possibility of reading directories structure in tables with
File
table engine. This fixes #8536. #8537 (alexey-milovidov)